How to run marketing experiments?
Sep 1, 2023
Marketing leaders at fast-growing businesses are constantly searching for new avenues of growth. These must contribute to the top line and increase market penetration. Most marketers achieve this by coming up with ideas, executing them, and measuring the impact of those ideas via touch-based attribution.
In our previous posts, we've discussed why this approach doesn't find true incremental impact and how Marketing Mix Modeling is a useful method to estimate incrementality. However, to find causality or to validate new ideas, MMM isn't enough. You need experimentation.
In this blog post, we'll explore the significance of conducting marketing experiments and provide an overview of the available methods. We'll explain why we advocate for geo-lift experiments, as well as the common mistakes to avoid while experimenting.
Why run marketing experiments?
Marketers have three broad options to calculate the return on marketing investment — touch-based attribution, Marketing Mix Modeling, and experimentation.
While touch-based attribution has become very popular, it has foundational problems. It's unable to estimate incrementality and can be relied on only for digital event tracking use cases. It's also not able to support all channels.
Marketing Mix Modeling can estimate incrementality and is a robust alternative to touch-based attribution to measure return on investment. However, it has one gap. It's based on correlation and not causation.
Controlled experiments are the most scientific way to establish causality. In combination with Marketing Mix Modeling, experiments provide a strong foundation to understand and communicate the incremental sales generated through marketing investments.
A controlled experiment is a scientific test done under controlled conditions, meaning that only certain elements are changed or kept constant to measure the effects of those changes. This is done to reduce the potential for bias or error in the results. It also helps to ensure that any changes in the results are due only to the elements that were changed. Controlled experiments can be used to test the effectiveness of different treatments or techniques, or to prove or disprove a scientific theory.
There are two main scenarios where experiments are the only practical choice for marketers:
When you have no historical data on a specific channel or strategy, and
When you have a need to generate very precise and certain answers.
While experimentation has become a standard practice in product and growth teams, the same can't be said about marketing experiments. We think this is a missed opportunity. There are clear and strong reasons for every marketing team to invest in experimentation. In the next few sections, we'll show you how.
How do you identify experiment hypotheses?
The starting point for a successful marketing experiment is to have a clearly defined hypothesis. A hypothesis is an assertion based on research and data and should be specific, measurable, and testable to yield meaningful results.
Developing hypotheses starts with understanding the problem you're trying to solve or the opportunity you're trying to capture. You can use data, user research, and customer feedback to identify these. Insights from Marketing Mix Modeling are a great set of hypotheses that can be validated further with experimentation. From there, you should focus on the levers that are likely to make an impact on the desired outcome by testing different strategies or tactics in controlled experiments.
Generally, we see five common types of hypotheses in marketing experimentation:
Increasing or decreasing spend on a specific channel
Investing in a completely new channel
Testing a new campaign (creative, audience)
Testing a new bidding strategy
Testing a new price/offer
It's also important to consider the cost, time, and effort associated with running an experiment before devising your hypothesis. Experiments require careful planning and often involve collecting data beforehand to measure any changes accurately after implementation.
Finally, hypotheses must be easily measurable so you can draw conclusions from them when analyzing data from your experiment. This means identifying key performance indicators (KPIs) such as impressions, leads, pipeline, or sales that'll help you track progress over time and determine if your experiment was successful or not.
Companies can avoid common pitfalls by developing well-defined hypotheses before experimentation.
How do you execute marketing experiments?
There are three primary methods of experimentation available, each with its own set of advantages and drawbacks.
On-platform testing consists of two flavors: Conversion Lift from Google and Meta, and A/B testing on each marketing platform (e.g., LinkedIn, emails, etc.). In Conversion Lift, the ad platform creates a control and treatment group automatically and serves your ad to the treatment group only. The results are then compared to assess the impact of the variation on conversions. In on-platform A/B testing capabilities, there's limited transparency about how control and treatment groups are set up.
The primary advantage of using either of these capabilities is their ease of implementation. However, a disadvantage is the lack of transparency regarding the experiment setup. Additionally, due to the individualized methodologies of each platform, one must become familiar with N distinct approaches, thereby raising the risk of errors.
Geo-based testing helps marketers test their hypotheses in real-world environments by exposing a particular population to an experiment while keeping other populations as controls. We recommend using a synthetic control approach, which uses machine learning to identify and synthesize multiple regions to form your control and treatment groups. This approach provides the highest accuracy and least uncertainty in the results you can observe.
The biggest advantage of using geo-based testing is that it generally works for any media channel. Another benefit is that it's ad-platform agnostic. The biggest disadvantage of geo-based testing is that it requires you to develop the necessary statistics skillset on your team and collect geo-level data to setup the test.
There are other observational studies such as cohort analysis or the simple act of turning media off and on again. While it may be tempting to use these approaches, they're not an effective replacement for experiments because they don't have a control group. These approaches help you learn about correlation and not causation.
Just like product and growth teams at fast-growing businesses have invested in building or buying product experimentation platforms, we recommend marketing teams should make the same level of investment in marketing experimentation.
What are the common pitfalls of experimentation?
Experimentation is a powerful tool for marketers, but it's important to understand the potential pitfalls that come with it.
The most common mistakes made when running marketing experiments include:
focusing on short-term gains rather than long-term value,
failing to set up proper experimental controls,
using incorrect metrics, and
not properly analyzing the data and drawing actionable conclusions from it.
To get the most out of your marketing experiments, it's essential to look at both long-term than short-term gains. Short-term gains may provide immediate rewards but often fail to yield meaningful and lasting results. Instead of aiming for quick wins, experimenters should take a holistic approach that considers larger trends and patterns. This means running experiments over extended periods to collect enough data points necessary for making reasonable predictions about future behavior or outcomes.
Furthermore, experimenters must ensure that they have clearly defined control and test groups so as to measure progress accurately against predefined goals. This is especially hard in marketing experiments because they're not always in control of the test and control splits. Additionally, it's essential for experimenters not only to collect data but also properly analyze it to draw meaningful insights from their tests and inform future strategies.
Another consideration is picking the right metrics. Marketers need to pick metrics that are measurable, can easily be tracked back to the control and test groups, and be sensitive and timely. Metrics that can only be computed several months or years after an experiment treatment (e.g., renewal rate) aren't ideal candidates for inclusion in an experiment set up. Instead, we recommend picking leading metrics that have causal relationships with those eventual business outcomes (e.g., usage).
Finally, validating and verifying the results of an experiment is key for ensuring accuracy and avoiding costly mistakes such as false positives or wrong assumptions. Frequently re-running experiments, and conducting A/A tests, where the treatment and control are the same, can help ensure against such risks as well.
Validating and verifying experiments is key for accuracy and avoiding costly mistakes such as false positives or wrong assumptions. Similarly, poor segmentation can mean the difference between success and failure. Poorly conducted experiments are worse than no experiments, as they can give a false impression of reliability and accuracy.
How do you get started?
Now that you know the importance of experiments, the source, and type of hypotheses, the various methods of conducting them, and mistakes to avoid, it's time to start experimenting.
Just as we outlined in our post on Marketing Mix Modeling, it's wise to first align on the desired outcome metric. Subsequently, formulate your hypotheses and become conversant with the available methods.
We encourage you to start small and build your confidence gradually. If you need help, please reach out.