Three cures for overfitting
We’ve painted a picture of the problem in uncomfortable detail. It seems only right to spend some time talking about the cures. Here are three:
1. The Bayesian MMM cure and other fancy maths
Bayesian MMM allows you to anchor your estimates on a ‘prior’ and impose some stability on the model. For more information on Bayesian MMM see this previous article from The Wheelhouse from last summer.
The issue is: on what basis should you choose to set your prior? You can set priors based on industry benchmarks, but this is quite hard to do in practice, as you need to control for the scale of your client brand and have access to other information that might not be in the public domain, like profitability.
Another option is to take your prior from an initial round of modelling that is more parsimonious than the final version of the model. So, for example, we might anchor on the average performance of paid social as our prior but split the final report at the level of sub-channel (Instagram, Facebook, X, Reddit, Pinterest etc.) and perhaps split again by format or message.
This is not really a fix at a fundamental level. Just because you are using Bayesian MMM does not make the impact of a tiny little budget pot more measurable. Your model is still a random number generator at heart, but you can think of your new estimates as having an elastic band wrapped around them that keeps them close to the prior. It will take a stronger signal in the data to pull that estimate away from the reference point, and the further away the data wants to take you, the tighter that elastic band will get. It is not a cure for overfitting per se, but it will help stabilise your model and limit some of the outlier ROI estimates.
Another method that claims to be better suited to granular measurement is elastic net regression, which powers Meta’s Robyn package. Once again, it’s not a cure per se. Elastic net – or more specifically the ridge regression component – is something known as a ‘regularised regression’ approach that very slightly biases towards giving a little bit of credit to every variable in the model and slightly reducing the credit given to major factors. In an overfitted model, this approach ensures that the long tail of media lines all get their due. But it feels like cheating as we are ever so slightly resting a thumb on the scale.
Fancy maths can’t really fix overfitting, but as long as the end user truly understands the nature of the fix, it should be fine to proceed.
2. The Triangulation cure
Another potential cure to the granularity problem is to allow MMM to play to its strength of estimating overall incrementality and defer to some other, external technique to inform reporting on a more granular, sub-channel basis.
Three examples to bring this cure to life:
a) If your business is reasonably confident in its online tracking framework, it may be sufficient to model at the topline channel level in MMM and rely on attribution to inform the relative efficiencies within a media channel. This is not such a great solution if a high share of transactions happens offline, however.
b) If you believe effectiveness can be linked to a single golden metric – for example visual attentive seconds – then it is possible to score sub-channels on this basis and simplify your econometrics. The job of the MMM then becomes “to estimate the uplift per 100K attentive seconds” with another source doing the heavy lifting for within-channel estimates. There’s more detail here.
c) Geotesting is the gold standard if you want to answer one exam question really well. It’s a robust way to drill down to one sub-channel and it has no dependency on cookies or back data. Unfortunately, well-run tests can be time-consuming and no advertiser has infinite resources, so this solution does not scale well past a few tests each year.
3. The Culture cure
Of the three cures, this one is a must. Marketing effectiveness professionals need to get better at speaking to our clients candidly about the limitations of the methodology. We need to get better at having awkward conversations where feasibility and high expectations collide. Analysts – those who are closest to the data – need to be empowered to do their job as technical advisors on the solution design; the job is bigger than cranking a handle on a machine that reports consistent ROIs update-to-update.
Summing up
A spirit of openness is essential whether you work in an internal marketing effectiveness team, in an agency, or at an independent shop, such as Ebiquity. If clients are taken on the journey, given a deeper understanding of the trade-offs when choosing between a simpler or a more overfitted model, they go from being passive users of MMM to co-authors of the solution, which is how it should be.
Find out more: Ben Lambert: Overfitting in Econometrics; Richard McElreath: Fitting Over & Under.
Up next: Our next Wheelhouse by is by Ebiquity Director, Tom Loughnan. It’s a companion piece to this article dealing with diminishing returns and saturation effects in MMM models.
