OG House
The conversation · No. 01

We turned off our best channel to find out if it was real

A holdout that moved her budget, a scorecard that cost her a senior buyer, and how she told the board growth would look slower.

Measurement and incrementalityTeam and agencies

Dana Reyes ran growth at a top-50 shopping app for six years and has spent fourteen in mobile. Over 45 minutes she walked through three calls that reshaped how her team spent, measured and talked about growth.

1. Switching off the channel attribution loved

The setup. “One network was getting credit for about a third of our installs. Every dashboard said scale it. But our organic installs dipped whenever it spent more, and rose when it paused over a holiday weekend.”

The call. “I asked for a holdout: no spend in two matched regions for four weeks. My CFO loved it, my UA team hated it. I agreed in writing to call it off if purchases in the holdout regions fell more than ten percent.”

What happened. “Purchases barely moved. Most of what that network was credited with would have happened anyway. We moved most of the budget into two channels that held up under the same test, and our cost per incremental purchaser fell by about a third.”

Steal this. Before any channel passes a fifth of your spend, run a four-week regional holdout and agree the stop rule in writing first.

2. Changing what the team was paid on

The setup. “The test was the easy part. My buyers were bonused on installs at a target CPI, so the scorecard still rewarded exactly what we’d just proved was half fiction.”

The call. “We moved the scorecard to incremental purchasers, measured quarterly by holdouts. Two senior buyers pushed back hard. One left. I don’t regret the change, but I regret how fast I made it.”

“If the scorecard rewards the old number, the team will find a way to keep producing it.”
Steal this. Change the scorecard in the same quarter you change the strategy, and give people one full cycle on the old one so the switch isn’t a pay cut.

3. Telling the board growth would look slower

The setup. “Cutting fake installs makes the topline chart dip. I knew the board would see the dip before it saw the savings.”

The call. “I went in first, with the holdout results and a two-quarter view: installs down, purchasers flat, cost per purchaser down. No surprises. The board meeting after the dip was the shortest I ever had.”

Steal this. Pre-announce the dip. A number that drops on schedule is a plan; a number that drops by surprise is a problem.

What she’d do differently

“Run the first test a year earlier. I waited for certainty, and certainty only comes from the test. And tell the team why before the numbers come in, not after.”

If I were starting today. Learn the difference between what happened next to your ads and what happened because of them. Learn it early.
01
Member of the House · No. 01 Dana Reyes Former VP Growth at a top-50 shopping app. Fourteen years in mobile, six running growth through the shift from attribution to incrementality. Dana Reyes’s profile