Explore how Jeannette Hallerman, Joan Gonzalez-Conde Cantero, and Raul Villagrasa, part of the Omnianalytics team, improved our data-driven marketing mix model to help marketers optimize media spend and performance.
As a part of the Omnichannel Analytics & Optimization (OMNI) Domain at Vista DnA, our mission is to support marketing teams to optimize spend and performance of paid media channels through data and analytics. We’re always looking for efficient methods to understand the impact of marketing spend and portfolio mix on profitable growth of the business.
During the last year, we’ve leveraged well-established top-down marketing mix modeling methods that have proven to be reliable in determining the impact of advertising efforts on business performance. The MMM model leverages advanced statistical methods to evaluate the patterns in spend and KPI, and delivers actionable outputs upon which our media teams can build hypotheses and optimize media spend across channels. In August of 2022, Vista launched its first media mix test based on guidance from the MMM model output. The results of that first randomized control based test confirmed expectations, and increased confidence in our MMM methodology, and confidence in the future of MMM within Vista’s full funnel marketing performance measurement product suite.
Evolving from Homegrown Framework to Advanced ML Techniques
For our first MMM model, we chose Vector AutoRegression (VAR) model defined by Professor Koen Pauwels, an industry expert on Marketing Mix Modeling. We conducted a POC with Koen Pauwels to test the methodology on a sample of data and found the VAR model viable for our objectives.
2021 saw our next iteration of the MMM methodology at Vista, based on Robyn, an open- source library for automated MMM, using machine learning techniques built by Meta. The new model represents a significant upgrade for Vista,
Some notable improvements include incorporating additional drivers into the model and gaining insight into diminishing returns in response, and ad stock decay rate.
We chose Robyn for many reasons:
- The model output answers the top priority questions from our business stakeholders, mainly the percentage of target metric (Bookings, New Customers) contributed by each driver (Media Channel), and the associated short and long-term return on ad spend (ROAS).
- It uses response curves over the historical spend range by media channel (aka diminishing return curves).
- Its results can be used to construct a budget allocation tool to propose optimal spend levels in paid media channels.
- Robyn also allows us to include lower granularity drivers such as linear vs. CTV or display media split by brand/performance, social, YouTube, and other. It also allows introducing organic and/or contextual factors, such as discounts.
- The open-source library incorporates the same foundation (Prophet) being used at Vista for forecasting use cases in manufacturing and pricing.
- Robyn enables driver calibration with controlled lift experiment results.
- The source code is dynamic and frequently updated, with the ability to report bugs and communicate with developers.
The Limitations and Risks of Vista’s Marketing Mix Model (MMM)
While Vista’s second generation MMM is a significant improvement over our early model, there are still a few limitations. For instance, Robyn uses a regression-based technique, which trains the model on historical trends. This means it only indicates how well a particular channel spend mix has performed during the time range of input data. The model cannot predict or guarantee KPI results in the future due to the ever-changing economic and competitive environment, and many other constantly changing strategies within Vista. New features such as predictive capabilities within Robyn package would help the team bring its recommendations to the next level.
The model requires sufficient prior spend across marketing channels before MMM can construct statistically significant response curves. This limits the application of MMM to markets with little to no historical media spend. As such, marketers should interpret the results cautiously and consider it a starting point for a marketing planning process – not a prescriptive, final answer.
Since all forms of modeling have inherent uncertainty, we present contributions and associated performance metrics within the context of ranges. The model process involves running nearly 10,000 iterations to produce meaningful results. Currently, the team applies business logic and statistical measures (e.g., NRMSE and MAPE) to select the best models. Based upon those top models’ coefficients and hyperparameters, we can build confidence intervals around each individual driver’s contribution as well as response and decay curves. Confidence intervals around response curves and driver’s contribution are a key tool for informing budget allocation, and hypothesis-based testing of MMM driven decisions. They increase the transparency of current model precision by accounting for the uncertainty of potential returns, thus promoting trust amongst stakeholders.
The following figure shows an example of a response curve with a confidence interval for each level of spend, constructed based on the hyper-parameters alpha and gamma of the best models. For more info on those hyperparameters, check the official documentation.
Since the model uses transactional data and partner spent data, the output depends largely on the reliability of the source data. As a result, it is key to integrate automated Q&A checks into the data ingestion pipeline. For this, Vista uses DBT.
Coefficients from the selected 12-month model (aka initial model) are used to estimate channel contributions. Initial model parameters are then fed into a refresh process, generating new coefficients used to estimate channel contributions for each month within the 12 month training period. As a result, there are output metrics from the model that relate to the entire 12-month training period and other metrics specific to each month.
Output for each month: Channel contributions to target metrics for each month, % of target metric. (for example: % of new customers contributed by each channel in each month.) These values are used to estimate channel-level return on ad spend (ROAS), considering channel spend, and average lifetime value of acquired customers.
Output for the entire training period: Diminishing returns (target metric response to spend) for each channel. This output shows how marginal response will keep decreasing with each higher level of spend, until audience saturation is reached.
Ad stock decay rate: (aka., half-life or drag curve) shows the decay of each media effect over time. For example, we saw model output reports that the next day after streaming linear TV, 95.2% of the impact on the day of streaming is still present, and this decay rate is carried forward each day until the impact becomes nominal.
Graph Source: Meta’s online documentation
Impact of the MMM Solution
Robyn’s main output statistics, such as ROAS (return on ad spend) and diminishing returns curves, are helping Vista marketing teams build media mix hypotheses to maximize the effectiveness of spend. We have ongoing conversations with the marketers to understand how they’re leveraging insights from this model. Their inputs are helping us estimate the value of our MMM-driven data product retrospectively.
Some of the key questions we expect our model to answer include:
- What is the overall impact of marketing spend on sales, beyond baseline sales? How much sales would have occurred in the absence of marketing?
- Which media channels drive the greatest short-term and long-term ROAS?
- What is the maximum we can spend in each media channel before we hit diminishing returns?
- What is the optimal budget allocation across paid media?In August of 2022, Vista launched its first media test with a randomized control group, based on the insights from MMM. Test results indicated a significant lift in key performance indicators resulting from increased spend in tested media channels. Learnings from each test are used to shape future media mix, and are leveraged also by the MMM to calibrate future model refreshes.We are seeing interest from other teams within Vista, to integrate MMM with other data products, and expand guidance to additional markets. For example, there is potential to use the estimated contribution of marketing spend from the MMM output, as an input into our pricing demand models.
The Road Ahead
Vista is at the beginning of our journey, exploring and deploying the Robyn MMM solution. In the future, we plan to tailor the model to answer more pertinent business questions: How valuable is a brand investment in driving long-term ROAS? How can we optimize our channel mix to attract more new high-value customers?
Going forward, we plan to explore several qualitative and heuristic approaches to tighten confidence intervals around response curves, and ensure the MMM adequately reflects the full impact of media in all channels both short and long term, thus increasing confidence in and actionability of MMM based spend guidance. From there, the outputs from the MMM model will become the foundation for a data driven “what-if” model that will guide budgeting decisions.
The MMM team would like to thank Leonel Sentana (Head of Marketing Science, Iberia at Meta; co-creator of Robyn library) for collaborating in the review of this document.
Interested in using data and analytics to tackle complex problems? At Vista DnA, we always seek talented individuals to join our team. If you want to make a difference by applying your skills and expertise to real-world challenges, explore our career opportunities today!