Insights
Misconceptions in marketing experimentation and MMM
Testing and MMM aren’t only for big budgets, measurement isn’t a one-time answer, and conflicting data sources aren’t errors. A look at the most common marketing measurement misconceptions.
Sundar Swaminathan
Marketing Science Advisor
Misconceptions in marketing experimentation and MMM
Try to discuss marketing science with smaller companies, and you’ll often hear the same objection:
“Best-in-class practices don’t work for us.”
Sometimes, that’s true. But, more often than not, that objection is rooted in misconceptions that show up as limiting beliefs.
The best way to overcome them is to adopt a different way of thinking about marketing measurement. Think about it as a system you run continuously, not a certainty you buy once. That's why the phrase "Marketing Measurement is a muscle you must build" captures the idea so well. Like any muscle, it gets stronger the more you use it.
With that in mind, let's look at some of the most common misconceptions, why they exist, and why they're worth challenging.
Testing and MMMs are for big companies
We commonly hear these objections:
“Testing and MMM are for big companies with big budgets”
“Testing and MMM require complicated tools”
These two ideas usually go hand in hand: you need big budgets and complicated tools. Both stem from the belief that testing needs to deliver 100% precision and accuracy. Not true. Testing and experimentation have a sliding scale for how much you trust what they say.
Take the example of medicine vs food.
In medicine, the scientific rigor needs to be at the highest level because, if not, there are real (possibly catastrophic) consequences to human health. On the flip side, say you're going to a restaurant, and you want to try out a new dish. You order it, but after a couple of bites, you decide you don’t like it. No one is going to sit there and say, "Hey, you need to take 30 bites to ensure you hit the right sample size."
Testing and experimentation aren’t purely about scientific rigor. Yes, bigger budgets allow you to have a larger sample size and therefore more confidence, but you can still run tests at smaller budgets.
Here's an example:
Say you're a startup and, for two years, you've been placing 85-90% of your marketing budget on Google Search. To run an incrementality test, you could simply turn off Google Search for a short period of time. You'd see a very, very clear pattern in your data that would lead to one of two conclusions: either Google Search was truly incremental, or it wasn’t.
When you have that big of a budget change, you can detect the impact easily.
Now let’s address the second part of the misconception – that you need complicated tools.
Tools like Paramark will absolutely help you measure more accurately. But before implementing an MMM, you need two foundations in place: a culture of experimentation and high-quality data. Without them, even the best tools won't deliver meaningful results. That's the real difference between smaller and larger companies: not the tools they use, but the quality of the data and experimentation culture behind them.
Measurement is static
We commonly hear these objections:
“Incrementality is the ground truth”
“Once you’ve measured, you're done”
“Incrementality tests prove a channel works (or doesn't)”
This misconception likely exists because the discourse around marketing science feels so black and white: “The best way to measure is triangulation,” or “Incrementality test is the gold standard.” All true, but we often don't see the iceberg below: the foundations and systems in place, the ongoing and evergreen system.
When you run an incrementality test, you're capturing a static snapshot of that channel's impact. But that can change, literally, within a week. All you need is some big event:
Pandemic
War in Ukraine / Middle East
Algorithm changes on Meta or LinkedIn
Launch of LLMs and Gen AI
Given how quickly the world changes these days, if you're still referencing an incrementality test from two years ago, you might as well throw out those results. They're likely doing more harm than good.
Incrementality testing needs to be an evergreen activity in your business.
There’s a variation of this misconception that we also hear often: that incrementality tests prove a channel works or that it doesn't. I'd amend that statement by adding two words.
“For you”
“Incrementality tests prove a channel works (or doesn’t) FOR YOU”.
Meta, Google, and LinkedIn – they each have over a billion users. Now, unless your ICP is magically not on those channels, then what the test is really showing is whether your current implementation on those channels is working (or not). It's your job to then dissect why. Here are a few variables for you to consider:
ICP
Creative
Frequency and Reach
Positioning + Branding
So, is your testing showing that a channel isn't working for you? Maybe instead of declaring it irrelevant, it's time to revisit your strategy and figure out exactly why it’s not working.
Data sources shouldn’t conflict
We commonly hear these objections:
“MMM and MTA disagree because one of them is wrong”
“Attribution and incrementality should match”
The assumption that underpins this misconception is that these tools (MMM, attribution, etc.) are views of the truth. Quite simply, they are not. A better way to view them is as parts of a picture that, when put together, give us a better understanding of the whole thing.
First off, an MTA will never show attribution for non-digital channels, while an MMM is designed to better capture the impacts of all channels. So right off the bat you have a tool which, by design, does not look at non-digital channels, and you're comparing that to a tool that can. Sounds like user error.
The next example is attribution versus incrementality. Let's assume you're using Last Click. The attribution is simply telling you, based on the data you've passed it, that the last identifiable click closest to purchase was channel X. Nowhere does the attribution say or even pretend to say that that click was incremental.
Again, this comes down to misunderstanding what the tool is actually designed to do.
The only data source that should never conflict is your internal first-party data. Everything outside of that will always be subject to some bias.
Misconceptions on MMM
We commonly hear these objections:
“Confidence intervals mean the model is uncertain”
“MMM tells you where to spend your next dollar”
“More data = better MMM”
Confidence intervals mean the model is uncertain
Note: We’ll use C.I. below to reflect both Confidence and Credibility Intervals.
Many of these misconceptions start from a misunderstanding of what an MMM can do. Unfortunately, vendors that oversell and overpromise do both the industry and marketing leaders a disservice.
Without getting into too much detail, an MMM is a statistical model that attempts to correlate changes in marketing spend with changes in business outcomes. You can read more about our MMM here.
There’s a word from the sentence above that I want you to focus on: model. An MMM is fundamentally a statistical exercise. And statistics always involve uncertainty because they're trying to estimate reality using limited data. They can get us closer to the truth, but they can never reveal it with absolute certainty.
That's exactly why every good MMM should report C.I.s.
But what does a C.I. actually tell us?
A C.I. doesn't mean the model is unsure of itself. It tells us the range within which we believe the true answer is most likely to fall. The uncertainty exists whether you show it or not. A point estimate doesn't eliminate uncertainty, it simply hides it.
So when an MMM reports an interval rather than a single number, it's not hedging. It's being honest about the limits of statistical modelling.
That’s a feature, not a flaw.
So, if you ever see an MMM output without CIs, please run the other way.
MMM tells you where to spend your next dollar
Another misconception is that the MMM tells you where to spend your next dollar. It’s a reasonable assumption, but the problem is that MMMs will always have a lag because they rely on past data.
That creates another challenge: marketing channels don't behave linearly, they operate on saturation curves.
Unless you've already spent at that level or validated it through a pulse-up incrementality test, you can't reliably predict what that saturation curve will look like. Diminishing returns hit harder and sooner than some people understand.
So yes, using an MMM for forecasting and budget allocation is one of its biggest strengths. But those forecasts should always be interpreted alongside the uncertainty behind the model, not as precise predictions of future performance.
More data = better MMM
A common question I get asked is, "How many points of data do I need?"
The general guideline is two years of data, but volume alone isn’t enough. What's more important is the quality and volatility of that data.
If you've spent the same amount on one channel for the past two years, your MMM will struggle to detect meaningful signals. Nothing has changed, so there's very little for the model to learn from.
On the other hand, if you've got two years of data where spend has gone up, down, and back up again, those fluctuations create signals that an MMM can detect.
More data doesn't necessarily equal a better MMM.
Better data equals a better MMM.
And in this case, "better" doesn't just mean cleaner or higher quality data, it also means data with enough variation to reveal patterns.
So the next time you accidentally misconfigure your paid search budget and your spend spikes, you can simply say, "Great, this will make our MMM better."
If you still feel strongly about some of these limiting beliefs, just remember that marketing measurement is a system you run continuously. The longer you invest in it, the better it becomes.
If you'd like to dive deeper into these misconceptions, this episode of the Brandformance podcast explores many of these ideas in more detail.
While you're here
Find your edge
Book a demo now and you'll get a live, expert-run walk through of how Paramark can help you with incrementality measurement and experimentation.
