The Grail Knight from Indiana Jones and the Last Crusade guards incrementality with his life.
Why the search for incrementality is like the search for the grail
So there's this scene in Indiana Jones and the Last Crusade (you know, the one find the Holy Grail???) when Indiana Jones and this wealthy collector who is also a Nazi make it all the way into this room where the Holy Grail is. They're met by a thousand year old knight who was charged with protecting the Grail from…um…everyone??? Anyways, the room doesn't just contain the Grail, it contains lots of fake grails. The final task for whoever sought the Grail was to pick out the true Grail from all the false ones with one catch: if you chose a fake Grail, you die.
"You must choose, but choose wisely. For as the true Grail will bring you life, a false Grail will take it from you."
Long story short, Indiana Jones tricks the Nazi into picking a false Grail and the Nazi ages super fast before turning into a skeleton and dying. Jones picks the true Grail and uses it to save his father from dying.
Swap "incrementality" for "Holy Grail" and what you have is the modern marketer's crusade for proving their worth to CEOs. Incrementality and incrementality testing are all the rage now. Founders and CEOs need to understand how effective their marketing dollars are and if those dollars are best spent on Google Ads or personnel or whatever.
It used to be easy (or so we thought, more on that in a minute). Before the iOS 14 update we were confident in the in-platform attribution. In other words, we typically believed whatever Google Ads told us the ROAS or CPA was. We knew that Google Ads or Meta Ads liked to take credit or partial credit for conversions that didn't happen on their respective platforms, but we agreed that directionally the ROAS or CPA was correct.
iOS 14 hits and we realized that Google and Meta used forecasts to attribute conversions to ads rather than actual customer tracking. They called this "modeled conversions" and used this to fill in the data gaps and predict the conversion probability for users who opted out of data tracking. We scrambled to combat those silly users who wanted to protect their data.
First it was tracking solutions like Triple Whale and Northbeam.
Then it was Meta and Google claiming they fixed their tracking.
Next it was migrating to server-side tracking.
All in an effort to find the Holy Grail. But…
But we chose…poorly.
It's not that incrementality can't be figured out
It's that incrementality is really really really really really hard to pinpoint. At the end of the day, accounting for incrementality is challenging because we never observe the ground truth. While we can observe how many conversions are made, we cannot tell which conversions are incremental and which conversions would have happened anyway or which conversions are cannibalized from other platforms.
I've seen advice like "you should be looking at what your next dollar will bring you” or “Facebook lift tests and Google's causal impact R package using Bayesian structural time-series models, that can be a good starting point.”
And there is some truth to those lines of thinking. It's not that we can't measure incrementality at all. In fact, we should think along the lines of marginal ROAS when met with ad spending decisions. Marketing can follow a predictable diminishing returns curve that we might not account for. Every marketing channel follows an S-shaped response curve with three distinct phases:
First dollars capture the low-hanging fruit
Mid-level spending shows strong but linear returns
Higher spending inevitably hits diminishing returns
But even here we have to be careful not to fall into the trap of thinking all conversions we see in our ad platform on any given day is attributed to the ad spend of that day.
Let's say that we spent $100 on June 1st on Facebook and generated 0 orders on June 1st. Then we spend $400 on June 2nd and generate 100 conversions on June 2nd. But no one who converted on June 2nd saw/clicked the ad from June 2nd, rather they saw/clicked the ad from June 1st.
Revenue | Spend | ROAS | |
Totals | $4,500 | $500 | 9x |
June 2 | $4,500 | $400 | 11.25x |
June 1 | $0 | $100 | $0 |
And we would think that the spend on June 1st was unprofitable. And we would be wrong. Because in Facebook, all conversions are back-attributed to the day of ad delivery, not the day of conversion.
Facebook anchors its default 30 day or 7 day attribution window on ad delivery instead of the occurrence of a conversion
Facebook Ads data is designed to answer one question: "how many conversions has the given ad generated?"
With an X-day attribution window, any conversion that happens before the end of the window will be attributed to our first ads regardless of whatever happens after the ad is served
Meaning that if a customer makes a purchase within the X-day window, the ad they clicked on or viewed first will take credit for that conversion, regardless of whether the reason for the purchase is from another channel.
Incrementality feels like something you can nail down with the right tool, the right test, the right model. But like the Grail room, it's full of false goblets.
The reality is:
We never observe the ground truth. We see conversions, but we don't know which are truly incremental, which would have happened anyway, and which are just cannibalized from another channel.
Attribution windows distort reality. Ad platforms back-attribute conversions to the day of ad delivery, not the day of purchase. That June 1st "unprofitable" spend? It actually drove June 2nd sales, you just wouldn't know unless you looked deeper.
Incrementality is temporal. Customers don't buy on neat timelines. A six-week test means nothing if your consideration cycle is four months.
So where does that leave us?
The Grail Test for Marketers
“You must choose, but choose wisely.” For marketers, that means choosing between:
The False Grails: Expensive MMM models, $10K/month incrementality platforms, and shiny dashboards that promise truth but deliver noise.
The True Grail: Grounded decision-making rooted in marginal ROAS, profit tracking, and common sense about how your customers actually buy.
Incrementality can be estimated, modeled, directional. But it will never be clean. The only "truth" is in the fundamentals:
Profit is trending up (or it's not).
Customers are getting more valuable over time (or they aren't).
Channels feel sustainable at scale (or they don't).
We can't chase the perfect model because unlike the movie, there is no Holy Grail of incrementality or attribution. Rather, we can chase making better, braver decisions with imperfect information.
Until next week,
P.S. Found this helpful? Forward it to another marketer who might benefit. We're all in this together.
P.P.S. Don't forget to download the Growth Marketing OS by clicking the button below.
P.P.P.S. What marketing challenges are you facing in your startup journey? Reply directly to this email with your questions or topics you'd like to see covered in future issues.