Category: Best Practices

(AKA Why “Trust Us, It’s Better” Is the Wrong Way to Ship Measurement Algorithms)

A new algorithm just dropped in the marketing measurement ecosystem.

And I had a very mixed reaction.

On one hand: Heck Yeah. I love seeing teams invest in making geo-testing and incrementality analysis more reliable. This stuff is hard. The world is noisy. Decision-makers want something actionable, not an uncertainty interval that’s wider than a barn door.

On the other hand: Sigh. the announcement was basically “we built a new thing, it’s better than the open source equivalents, trust us.” The claim is that it produces less biased estimates, narrower intervals, and better calibration. But without sharing much about the method or the test suite that backs those claims up.

I’m excited about innovations in incrementality. I’m less excited about black-box measurement claims—especially when teams and budgets depend on them.

Why I care (and why this keeps coming up)

For context: GDP sometimes does UA benchmarking and measurement diagnostics—assessing a customer’s current setup (MMM, incrementality, vendor stack), and then giving them a roadmap for improvement.

These engagements usually include some form of vendor analysis (e.g. helping our customers make build vs buy decisions). And in my experience, a meaningful fraction of MMM / incrementality vendors do not share technical details about how their models work.

That usually triggers three red flags for me:

1. One evolutionary (the “Bill Joy” problem)
2. One practical (the “don’t grade your own homework” problem)
3. One ideological (the “measurement is a shared scientific inheritance” problem)

Let’s talk through them.

Red Flag #1: The World Will Out-Innovate You (aka Joy’s Law + Open Innovation)

Bill Joy’s famous line is:

“No matter who you are, most of the smartest people work for someone else.”

A straightforward consequence of Joy’s law is that most innovation happens elsewhere. And that your job, as a company, should include systematically surveying and learning from the wider ecosystem (as a side-note, it’s not a coincidence that Bill Joy’s work on Unix was a key enabler of the open-source revolution).

The academic literature goes one step further: you should not only systematically survey and learn from the wider ecosystem. You should also give back. Knowledge-sharing can be a rational, value-creating strategy, because it attracts external effort and converts it into your advantage. Here’s two important papers on this subject:

• Lerner & Tirole’s economics work explains how open source participation can be motivated by things like career concerns and reputation—and why that’s not just altruism.
• Their later synthesis makes the case that a lot of open source dynamics are explainable using standard economics (theories of labor and industrial organization), i.e., it’s not “vibes,” it’s incentives.

Note also that firms don’t necessarily have to go “fully open.” There’s a well-studied middle ground: selective revealing. Joachim Henkel’s work on embedded Linux describes firms revealing selectively—sharing meaningful chunks of firm-developed innovation to get external support and ecosystem benefits, while still protecting some competitive IP.

In a later paper, Henkel, Schöberl, and Alexy go further: they describe how customer demand for openness can trigger a positive feedback loop, and eventually openness becomes a new dimension of competition.

That last sentence is the key: once open-source takes hold, and once openness becomes competitive, “closed by default” becomes an evolutionary dead-end. Which means that when a vendor, any vendor, announces a new algorithm but won’t disclose method details or evaluation infrastructure, in an area where there is a credible and growing open-source presence, my spidey-sense starts to tingle. Is this actually an improvement on the state of the art? And, even if it is, is it about to get obsoleted by the open ecosystem?

This question matters a lot because switching costs are real. How can I recommend a vendor to my customers if I suspect that their systems are going to become obsolete in short order?

Red Flag #2: Don’t Grade Your Own Homework (Especially in Measurement)

The second pragmatic red flag is simpler:

When a vendor is the only one who can evaluate the vendor’s product, we’re doing marketing and not science.

A black-box algorithm and a proprietary test suite and self-reported “we’re better” claims is very hard to assess responsibly (especially if, like GDP, you’re advising customers and their budgets). And while a potential customer could, in theory, use a free-trial period to do the comparison themselves, most customers don’t have the ability to do that (and especially don’t have the ability to do that across a suite of potential vendors). Not to mention that having every potential customer do a bakeoff against a suite of vendors is monumentally inefficient.

Acknowledging the importance of verification is a great starting point. But it’s almost meaningless if the details of the verification aren’t fully available.

The PyMC case study: a rare “this is how you do it” moment

This is why I was genuinely happy to see the PyMC Labs team publish an apples-to-apples MMM benchmark comparing PyMC‑Marketing and Google’s Meridian.

What they did right:

• They explicitly set out to create and publish a rigorous technical benchmark on realistic synthetic datasets covering different scales (from “startup” to “enterprise”).
• They aligned model structures and used identical priors and sampling configurations to keep it fair.
• They held a webinar and then published a video walking through the comparison.
• They made the benchmark code publicly available on GitHub for reproducibility. The repo itself is not just a toy notebook. It’s a benchmarking suite with data generation and parameter recovery tooling, and it explicitly supports comparing inference methods/libraries.

Even if you disagree with their modeling choices, you can do something extremely powerful:

You can argue with the algorithms and the code instead of arguing with the vibes.

Nico Neumann deserves a special callout here. His LinkedIn post about the PyMC <-> Meridian comparison generated one of the most informative LinkedIn threads in recent memory.

That’s how a field levels up. And it’s the ecosystem pattern I’d love incrementality vendors to lean into more often:

• Publish methodology details
• Publish test suites
• Publish failure modes
• And compete on product + implementation + support, not secrecy

Red Flag #3: Measurement is a Shared Scientific Inheritance (and Secrecy Slows the Whole Genre)

Here’s the more ideological point:

MMM and incrementality aren’t new. These tools are built on decades of rigorous academic work, plus a growing body of open-source implementations:

• Google’s open-source Meridian.
• Meta’s open-source Robyn.
• The entire PyMC marketing toolkit, which includes both a vanilla MMM class and a multidimensional model.
• Open geo-testing toolkits like GeoLift.
• Bayesian data-science toolkits like ArviZ.
• ….

We are all building on shared foundations. And when vendors keep core methodology opaque, the science progresses slower, trust erodes faster, and customers lose confidence more easily.

A few weeks ago, GDP’s Bill Grosso and Julian Runge wrote an article about the potential pitfalls of observational causal inference modeling—Combating Misinformation in Business Analytics: Experiment, Calibrate, Validate—with a particular focus on Media Mix Modeling. The article originally appeared as a guest post on Eric Seufert’s Mobile Dev Memo and has also been reposted on the GDP blog. It sparked a wide variety of comments on LinkedIn (and an article on Ad Exchanger), and we decided to collect the community response.

But before we get to the responses, let’s quickly summarize the background for writing the article in the first place.

Evidence increasingly reveals that observational causal inference (OCI)—methods that infer cause-and-effect relationships from existing data—often leads to misjudgments about the impact of business strategies. Unlike randomized controlled trials (RCTs), OCI relies on naturally occurring data patterns, which are susceptible to biases and unobserved variables. These inaccuracies can result in flawed conclusions about business effectiveness, risking wasted resources and harm to market position. Accurate insights are vital for guiding investments, pricing, and marketing strategies, making rigorous experimental validation essential in contexts where causality drives financial and strategic outcomes.

Bill and Julian discuss the limitations of OCI in business analytics, citing methods like Media and Marketing Mix Modeling (m/MMM), which often misattribute causality due to issues like endogeneity and omitted variable bias. They advocate for prioritizing experimental approaches, such as A/B tests and RCTs, to establish causal clarity. Additionally, they recommend using experimental results to calibrate observational models, correcting biases and improving accuracy. By integrating experimentation with model calibration, businesses can enhance analytics reliability and make better-informed decisions.

Two Significant Recent Papers that Prompted the Original Article

The pace of academic articles about possible issues with observational causal inference has increased in recent years. In particular, Julian and Bill cited two recent papers. The first paper was Observational Price Variation in Scanner Data Cannot Reproduce Experimental Price Elasticities, by Robert Bray, Robert Evan Sanders, and Ioannis Stamatopoulos. 

The authors analyzed 389,890 randomized in-store supermarket prices across 409 products in 82 test stores and found that experimental price elasticity averaged -0.34, while observational data from 34 control stores suggested an elasticity of about -2.0. This highlights a significant mismatch between observational and experimental estimates of demand elasticity. Observational data suggest that retailer prices are in the elastic range, whereas experimental results indicate pricing in the inelastic range. This discrepancy cannot be attributed to typical factors like estimator properties, price variation processes, or elasticity timeframes. The findings challenge the reliability of observational demand elasticity estimates and raise questions about standard economic models’ applicability to retail pricing. 

Julian and Bill also cited Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement, by Brett R. Gordon, Robert Moakler, and Florian Zettelmeyer, in which the authors evaluate the accuracy of non-experimental methods in estimating the causal effects of digital advertising. Utilizing data from 15 Facebook advertising experiments across 11 brands, the researchers compare experimental results with those derived from observational models, including matching, inverse probability weighting, and regression. The findings reveal that these non-experimental approaches often produce biased estimates, with the direction and magnitude of bias varying across brands and methods. This variability underscores the challenges of relying on observational data for advertising measurement. 

Concurring Commentary from AdExchanger

James Hercher, in his article Learning To Love And Let Go Of Attribution Models on AdExchanger, makes a strong case for our article, saying that while mix models and other attribution approaches have been essential tools for understanding marketing impact, they often fall short in today’s complex media landscape.  

Hercher highlights the limitations of models like media mix modeling (MMM) and multi-touch attribution (MTA), which can misattribute causality due to factors such as endogeneity and unseen variables behind emerging MMM tooling, echoing the concerns we raise in our article.

There are other reasons not to trust the MMM trend as a ‘truthier’ attribution fallback, now that multitouch attribution and user-lever tracking is infeasible.

And that’s because MMM might just become another walled garden platform plaything.

Earlier this year, Google open-sourced its own MMM product, which it calls Meridian. Meta has an open-source MMM solution it calls Robyn, while Amazon’s is still a proprietary product, not open-source.

But platform MMM is the same as platform anything. It’s there to prove the platform succeeded, as much as that your marketing worked. Google’s Meridian, for example, is really good at tying together search, YouTube, TV and Google Ads campaigns.

Hercher argues that these traditional models, while helpful in a simpler media environment, are now less effective at navigating the fragmented, multichannel advertising landscape. Instead, he supports our emphasis on integrating experimentation and validation, underscoring that models should not be relied upon as standalone truths. He agrees that mixing empirical experimentation, such as A/B tests, with calibration of observational data allows marketers to correct for biases and improve attribution accuracy.  

Hercher’s view aligns with our stance that business analytics must move beyond traditional attribution models and embrace an iterative, hybrid approach to better capture the causal effects of marketing strategies and guide informed decision-making. 

Discussion in the Community 

The article also resonated with many of our peers in the community, generating a variety of thoughtful comments and discussions after Bill and Julian posted their article on LinkedIn. 

The team at Haus, a startup marketing science platform that helps companies measure the incremental ROI of online and offline ad spend, had a lot to say about our take on MMM and causal analysis. 

Zach Epstein, Founder and CEO at Haus, agrees in principle and notes that experiments are hard: 

This is a great article that covers a lot of the issues we see day in and day out. What I think is less appreciated, especially in the world of advertising, is how difficult it is to run great experiments. The concept of running experiments alone won’t solve this problem – there’s a tremendous need for increasing access to world class methods and infrastructure.  

Running an experiment is easy. Running an experiment that you’d bet your own money on is extremely difficult. 

Chandler Dutton, who works on Customer Success at Haus, also agrees in principle and points out that, in practice, results do not match models: 

This piece thoughtfully explains not just why I’ve been insistent on every brand I know needing to work with a partner like Haus, but why I joined the team. 

Too often, marketing teams are chasing outputs from observational modeling only and trying to use those to inform multi-million dollar decisions without those models being able to prove causality. Whether for its own sake or for the purpose of calibrating such observational models, experimentation is critical and the only path towards getting really actionable data.  

I’ve seen teams chase their modeled attribution results and not have their resulting investments drive the results the model pointed towards. I’ve also seen teams drive incredible success without really knowing why and what to do next to pour gasoline on the fire. The missing piece is experimentation. I’d recommend all of the growth marketers in my network read this one. 

Olivia Kory, who works on Incrementality Testing at Haus, also agrees we need experiments: 

Dr. Julian Runge and William Grosso just released a very important guest essay in Eric Seufert’s Mobile Dev Memo about the shortcomings of observational causal inference modeling, specifically MMM. In their words: 

Evidence is mounting that observational causal inference (aka MMM) often misinforms about the actual impact of business strategies and actions, and this means we need more experimentation — for baseline evaluation of policies, for validation of observational insights, and for calibration of observational models. 

“When a new drug is tested, RCTs are the gold standard because they eliminate bias and confounding, ensuring that any observed effect is truly caused by the treatment. No one would trust observational data alone to conclude that a new medication is safe and effective. So why should businesses trust OCI techniques when millions of dollars are at stake in digital marketing or product design?” 

Others in the community also reacted strongly to the article. 

Tony Williams, an Economist and Director of Data Science at FlowPlay, agrees and wonders if a greater focus on advanced data science techniques (matching methods, et al) might help: 

Definitely excited to read this since I’ve spent the last few days looking at valid matching methods for multiple variants and have been surprised that the academic literature hasn’t covered this more. I know you’re looking at something different (MMM), but as someone who loves experimentation, there are also times we need other methods.  

Very cool to see this discussion getting brought up! 

Jim Kingsbury, an E-Commerce Marketing Advisor who has worked with Zappos, Allbirds, KiwiCo, and Amazon, agrees and lists vendors who routinely do AB tests to verify models:

Running geo-lift tests to validate – or, if needed, calibrate – the output of an MMM is becoming table stakes. 

This essay is a great reminder of how important this is. 

To marketing leaders out there who are using or evaluating MMM solutions, I recommend asking the vendor about their process to validate what their model claims.  

If the vendor hems & haws in response to this question, I’d recommend finding other vendors who enthusiastically embrace this critical step. 

A few vendors I know who always do this include: 

• SegmentStream 
• WorkMagic 
• LiftLab 

I’m sure there are others who do this and I’m excited for anyone reading this post to share who they are. 

A longer back-and-forth discussing the comparison of MMM to drug trials occurred between Jimmy Marsanico, VP of Product at Prescient AI, and Toma Gulea, Lead Data Scientist at Polar Analytics, debating whether comparing marketing to other verticals makes sense: 

I appreciate the focus on measurement rigor, but the paper’s comparison of marketing measurement to drug trials misses some crucial real-world complexity. In clinical trials, control groups get placebos in isolation. But in digital advertising, ‘control’ users are actively shown alternative ads competing for the same share of wallet. When a control user purchases a competitor’s product after seeing their ad instead of yours, they’re naturally less likely to buy your product – not because your ad wouldn’t have worked, but because they already spent their budget elsewhere. This makes the ‘untreated’ state anything but neutral, potentially leading experiments to undercount true advertising impact. 

While empirical tools like incrementality tests provide valuable data, treating any single approach as ‘table stakes’ oversimplifies the challenge. The most successful brands recognize that marketing measurement is more art than perfect science – they triangulate insights from multiple sources (sometimes by leveraging dynamic and regularly updated MMMs, like that of Prescient AI ) and combine them with strategic thinking and domain expertise. 

After all, isn’t the goal to make better decisions, not just chase methodological purity? 

Toma replied: 

Jimmy M. But that’s actually what you want to test. For example if you were to cut your spending overall on a channel, what you describe would happen (consumers shifting to competitors) and that’s exactly these external factors you want to account for when evaluating the true impact of your ads. Am I wrong? 

Jimmy clarified his take on the differences between marketing and drug trials. 

Toma Gulea 🤔 I’m not quite sure we’re saying different things here. The comparison and example of pharmaceuticals is just inherently different in the approach of test/control groups because during a hold out test, your competitors aren’t holding out (but during a pharmaceutical test you’re not taking a competitor’s drug to treat your symptoms). 

Alluding to your other comment below, you’re right — if you’re going to cut spend on a channel (or double it perhaps), if you’ve only spent the same amount daily on that channel, or campaign, there’s less data (or confidence) in the relationship of spend to revenue at any other spend value — making it a model to help make decisions, not a perfect crystal ball to predict impact of ALL future changes. Thus, my argument against “table stakes” — there is value… but only when used appropriately — that applies to any measurement tool. 

And Toma concluded that there are more parallels than differences: 

The principle is the same: isolating the effect of a treatment (or ad exposure) to estimate causal impact. Your competitor’s ad becoming more effective as a result of a change in ad exposure is absolutely part of the causal impact you want to test. The only difference with Pharma is the elimination of the “Placebo effect”.  

Even in pharma, participants in control groups don’t exist in a vacuum—Consider a holdout test for a new pain medication. Some participants in the control group might be taking over-the-counter pain relievers during the trial, while the treatment participants are not. The intervention still caused participants to stop the alternative medication, leading to a better or worse outcome. 

The only thing an RCT can give you is the impact of the intervention on the outcome in the real world, not the mechanism. 

Your change in ad spend causes a drop in revenue because of competition, then so be it—that’s the real-world outcome you’ll get, just like an outcome for a medication is influenced by the use of an alternative medicine. 

Randomized control trials are often impractical, unfeasible, or too costly, and other methods should be employed. But an RCT is an RCT, and the comparison with pharma is absolutely correct.  

Separately, Toma Gulea made an interesting general observation about the differences between claims and the actual value of using MMMs. 

A typical claim from MMM vendors: “testing the model’s accuracy on a separate holdout period ensures its trustworthiness”.  

This misses the core purpose of an MMM. The real goal isn’t simply to predict revenue based on past marketing spend, but to uncover the causal relationship between channel spending and revenue outcomes. The key question is: ‘What would happen if I spent X?’ In situations where the marketing spend has been stable over time, evaluating accuracy on historical data is meaningless because it doesn’t assess how the model will perform when actual changes occur. When it does, your MMM will break and you will realize it’s useless. 

The right approach requires a causal lens: 

• Start by understanding the business and marketing strategy to identify confounders and latent variables. 
• Then, apply causal methods and gather control and instrumental variables. 
• Avoid the lure of “predictive accuracy”: you can’t observe the true relationship you are trying to model. The goal is to have a useful model! 

Kenneth Wilbur, Professor of Marketing and Analytics at the University of California, San Diego – Rady School of Management, made the interesting point that experimentation was viewed as important in the early papers but somehow dropped out of daily practice:

Some of the original MMM literature in the 1950s pointed out that MMMs obviously needed to be calibrated with experimental variation in spending. 

An Operations-Research Study of Sales Response to Advertising, by M. L. Vidale and H. B. Wolfe (1957), demonstrates the necessity of precise, reproducible data to evaluate advertising effectiveness. Through controlled experiments, the authors identified key parameters—Sales Decay Constant, Saturation Level, and Response Constant—that define sales responses to advertising campaigns. These parameters enable the development of predictive mathematical models to optimize advertising efforts and budget allocations. The study emphasizes that well-designed experiments provide actionable insights for tailoring strategies to maximize return on investment, underlining the critical role of empirical data in refining marketing decisions.

A Media Planning Calculus, by John D. C. Little of MIT and Leonard M. Lodish of the University of Pennsylvania (1969), emphasizes the importance of experimentation in developing an effective marketing or MMM strategy by introducing a structured approach to media planning, known as the Media Planning Calculus, and recommending experimental calibration wherever feasible. The authors advocate for controlled experiments and computational modeling to measure and predict market responses to advertising. By integrating concepts like exposure frequency, forgetting, audience segmentation, and diminishing returns, the study demonstrates how experimentation refines parameter estimations, such as exposure values and response functions. This empirical grounding allows for dynamic optimization of advertising schedules and budgets, significantly improving marketing efficiency.

It is awesome to be reminded of these papers that already called out the necessity for experimental calibration of OCI and m/MMM almost 60 years ago.

We are very excited by the positive reception of our analytics strategy opinion piece. GDP is committed to precise analytics and driving forward best practices in gaming and beyond. 

This article originally appeared as a guest post on Eric Seufert’s Mobile Dev Memo, written by Dr. Julian Runge, an Assistant Professor of Marketing at Northwestern University, and William Grosso, the CEO of Game Data Pros.

Observational Causal Inference (OCI) seeks to identify causal relationships from observational data, when no experimental variation and randomization are present. OCI is used in digital product and marketing analytics to deduce the impact of different strategies on outcomes like sales, customer engagement, and product adoption. OCI commonly models the relationship between variables observed in real-world data.

In marketing, one of the most common applications of OCI is in Media and Marketing Mix Modeling (m/MMM). m/MMM leverages historical sales and marketing data to estimate the effect of various actions across the marketing mix, such as TV, digital ads, promotions, pricing, or product changes, on business outcomes. Hypothetically, m/MMM enables companies to allocate budgets, optimize campaigns, and predict future marketing and product performance. m/MMM typically uses regression-based models to estimate these impacts, assuming that other relevant factors are either controlled for or can be accounted for through statistical methods.

However, MMM and similar observational approaches often fall into the trap of correlating inputs and outputs without guaranteeing that the relationship is truly causal. For instance, if advertising spend spikes during a particular holiday season and sales also rise, an MMM might attribute this increase to advertising, even if it was primarily driven by seasonality or other external factors.

Observational Causal Inference Regularly Fails to Identify True Effects

Despite its widespread use, a growing body of evidence indicates that OCI techniques often stray from correctly identifying true causal effects. This is a critical issue because incorrect inferences can lead to misguided business decisions, resulting in financial losses, inefficient marketing strategies, or misaligned product development efforts.

Gordon et al. (2019) provide a comprehensive critique of marketing measurement models in digital advertising. They highlight that most OCI models are vulnerable to endogeneity (where causality flows in both directions between variables) and omitted variable bias (where missing variables distort the estimated effect of a treatment). These issues are not just theoretical: the study finds that models frequently misattribute causality, leading to incorrect conclusions about the effectiveness of marketing interventions, highlighting a need to run experiments instead.

A more recent study by Gordon, Moakler, and Zettelmeyer (2023) goes a step further, demonstrating that even sophisticated causal inference methods often fail to replicate true treatment effects when compared to results from randomized controlled trials. Their findings call into question the validity of many commonly used business analytics techniques. These methods, despite their complexity, often yield biased estimates when the assumptions underpinning them (e.g., no unobserved confounders) are violated—a common occurrence in business settings.

Beyond the context of digital advertising, a recent working paper by Bray, Sanders and Stamatopoulos (2024) notes that “observational price variation […] cannot reproduce experimental price elasticities.” To contextualize the severity of this problem, consider the context of clinical trials in medicine.

When a new drug is tested, RCTs are the gold standard because they eliminate bias and confounding, ensuring that any observed effect is truly caused by the treatment. No one would trust observational data alone to conclude that a new medication is safe and effective. So why should businesses trust OCI techniques when millions of dollars are at stake in digital marketing or product design?

Indeed, OCI approaches in business often rely on assumptions that are easily violated. For instance, when modeling the effect of a price change on sales, an analyst must assume that no unobserved factors are influencing both the price and sales simultaneously. If a competitor launches a similar product during a promotion period, failing to account for this will likely lead to overestimating the promotion’s effectiveness. Such flawed insights can prompt marketers to double down on a strategy that’s ineffective or even detrimental in reality.

Prescriptive Recommendations from Observational Causal Inference May Be Misinformed

If OCI techniques fail to identify treatment effects correctly, the situation may be even worse when it comes to the policies these models inform and recommend. Business and marketing analytics are not just descriptive—they often are used prescriptively. Managers use them to decide how to allocate millions in ad spend, how to design and when to run promotions, or how to personalize product experiences for users. When these decisions are based on flawed causal inferences, the business consequences could be severe.

A prime example of this issue is in m/MMM, where marketing measurement not only estimates past performance but also directly informs a company’s actions for the next period. Suppose an m/MMM incorrectly estimates that increasing spend on display ads drives sales significantly. The firm may decide to shift more budget to display ads, potentially diverting funds from channels like search or TV, which may actually have a stronger (but underestimated) causal impact. Over time, such misguided actions can lead to suboptimal marketing performance, deteriorating return on investment, and distorted assessments of channel effectiveness. What’s more, as the models fail to accurately inform business strategy, executive confidence in m/MMM techniques can be significantly eroded.

Another context where flawed OCI insights can backfire is in personalized UX design for digital products like apps, games, and social media. Companies often use data-driven models to determine what type of content or features to present to users, aiming to maximize engagement, retention, or conversion. If these models incorrectly infer that a certain feature causes users to stay longer, the company might overinvest in enhancing that feature while neglecting others that have a true impact. Worse, they may even make changes that reduce user satisfaction and drive churn.

The Problem Is Serious – And Its Extent Currently Not Fully Appreciated

Nascent large-scale real-world evidence suggests that, even when OCI is implemented on vast, rich, and granular datasets, the core issue of incorrect estimates remains. Contrary to popular belief, having more data does not solve the fundamental issues of confounding and bias. Gordon et al. (2023) show that increasing the volume of data without experimental validation does not necessarily improve the accuracy of OCI techniques. It may even amplify biases, making analysts more confident in flawed results.

The key point to restate is this: Without experimental validation, OCI is at risk of being incorrect, either in magnitude or in sign. That is, the model may not just fail to measure the size of the effect correctly—it may even get the direction of the effect wrong. A company could end up cutting a channel that is actually highly profitable or investing heavily in a strategy that has a negative impact. Ultimately, this is the worst-case scenario for a company deeply embracing data-driven decision-making.

Mitigation Strategies

Given the limitations and risks associated with OCI, what can companies do to ensure they make decisions informed by sound causal insights? There are several remedial strategies.

The most straightforward solution is to conduct experiments wherever possible. A/B tests, geo-based experiments, and incrementality tests can all help establish causality with high confidence. (For a decision tree guiding your choice of method, please see Figure 1 here.)

For digital products, RCTs are often feasible: for example, testing different versions of a web page or varying the targeting criteria for ads. Running experiments, even on a small scale, can provide ground truth for causal effects, which can then be used to validate or calibrate observational models.

Another approach is bandit algorithms that conduct randomized trials in conjunction with policy learning and execution. Their ability to learn policies “on the go” is the key advantage they bring. This however requires a lot of premeditation and careful planning to leverage them successfully. We want to mention them here, but advise to start with simpler approaches to get started with experimentation.

In reality, running experiments (or bandits) across all business areas is not always practical or possible. To help ensure that OCI models produce accurate estimates for these situations, you can calibrate observational models using experimental results. For example, if a firm has run an A/B test to measure the effect of a discount campaign, the results can be used to validate an m/MMM’s estimates of the same campaign. This process, known as calibrating observational models with experimental benchmarks, helps to adjust for biases in the observational estimates. This article in Harvard Business Review summarizes different ways how calibration can be implemented, emphasizing the need for continuous validation of observational models using RCTs. This iterative process ensures that the models remain grounded in accurate empirical evidence.

In certain instances, you may be highly confident that the assumptions for OCI to produce valid causal estimates are met. An example could be the results of a tried-and-tested attribution model. Calibration and validation of OCI models against such results can also be a sensible strategy.

Another related approach can be to develop a dedicated model that is trained on all available experimental results to provide causal assessments across other business analytics decisions and use cases. In a way, such a model can be framed as a “causal attribution model.”

In some situations, experiments and calibrations may not be feasible due to budget constraints, time limitations, or operational challenges. In such cases, we recommend using well-established business strategies to cross-check and validate policy recommendations derived from OCI. If the models’ inferences are not aligned with these strategies, double- and triple-check. Examples for such strategies are:

• Pricing: Purchase history, geo-location, or value-based pricing models that have been extensively validated in the academic literature
• Advertising Strategies: Focus on smart creative strategies that align with your brand values rather than blindly following model outputs
• Product Development: Prioritize features and functionalities based on proven theories of consumer behavior rather than purely data-driven inferences

By leaning into time-tested strategies, businesses can minimize the risk of adopting flawed policies suggested by potentially biased models.

If in doubt, err on the side of caution and stick with a currently successful strategy rather than implementing ineffective or harmful changes. For recent computational advances in this regard, take a look at the m/MMM package Robyn. It provides the ability to formalize a preference for non-extreme results in addition to experiment calibration in a multi-objective optimization framework.

A Call to Action: Experiment, Calibrate, Validate

In conclusion, while OCI techniques are valuable for exploratory analysis and generating hypotheses, current evidence suggests that relying on them without further validation is risky. In marketing and business analytics, where decisions directly impact revenue, brand equity, and customer experiences, businesses cannot afford to act on misleading insights.

“Combating Misinformation” may be a strong frame for our call to action. However, even misinformation on social media is sometimes shared without the originator knowing the information is false. Similarly, a data scientist who invested weeks of work into OCI-based modeling may deeply believe in the accuracy of their results. These results would however still misinform business decisions with potential to negatively impact share- and stakeholders.

To avoid costly mistakes, companies should treat OCI as a starting point, not the final word.

Wherever possible, run experiments to validate your models and calibrate your estimates. If experimentation is not feasible, be critical of your models’ outputs and always cross-check with established business strategies and internal expertise. Without such safeguards, your business strategy could be built on misinformation, leading to misguided decisions and wasted resources.