Lukas Vermeer is Vista’s Director of Experimentation and worked previously in a similar role at Booking.com. Together with his team – the “Experimentation Hub” – he is driving our cross-organizational effort to help Vista navigate towards our Experimentation North Star.
Vista’s goal is to deliver jaw-dropping customer value to small business owners looking for design and marketing solutions. To confirm that our execution and delivery do indeed result in jaws being dropped, we have a strong desire to test the validity of every assumption and the impact of every change. Delivery is not our goal. Experimentation is also not our goal. Customer value is our goal. Experimentation is a powerful method through which we can connect delivery to value.
I joined Vista earlier this year to help chart this new territory and journey into the data and analytics space. To kickstart the A/B testing momentum and build a culture of democratized online controlled experimentation.
In this post, I will draw a rough outline of the kind of culture that we are trying to build, and how we are taking the first steps towards that vision.
But first: let’s start with why.
Why do we want to test every change and every assumption?
Even with great user research and stellar product development strategy and technical execution, it is impossible to predict exactly how individual changes will impact the customer experience. This is not a failure on our part, it is simply an uncomfortable truth about the nature of building products for humans. To fail to acknowledge, measure and deal with this reality—effectively putting our product delivery before the customer experience—is to fail to be customer obsessed and data-driven.
This truth holds for changes big and small; especially when considering the possibility that any implementation might negatively impact the customer experience—and our business—in unintended ways. Merely striving to deliver jaw dropping customer value is insufficient. Only measuring the impact of some changes is also insufficient. We need to confirm that we are moving in the right direction at every step we make along the way.
Experimentation can support customer obsession in two distinct ways:
-
Informed Risk Taking: we want our teams to take risks, including occasionally big ones, in pursuit of customer value. Experimentation is fundamentally about enabling the taking of risks, knowing that many, many of our ideas will fail. Experimentation also allows teams to mitigate the risks involved with implementing changes. Any change left untested is a potentially unobserved degradation of the customer experience. By measuring the impact of each change on our customers and overall business performance—even when there was no intent to have any impact—we enable distributed but informed risk taking.
-
Increased customer understanding: Experimentation allows teams to inform decisions and future plans. The value of a single test is not limited to the change being tested. Things learnt from one test—for example which types of products small business owners prefer to purchase together—can inform prioritization, product direction and investment decisions in the future. This downstream value is magnified when results and knowledge are shared between teams, allowing teams to learn from experiments carried out elsewhere in the organization. Continuous experimentation also forces teams to think about causality and what motivates small business owners and moves the needle, sharpening their thinking and hypothesis formulation.
Both of these functions will require some amount of standardization and centralization of tooling and processes. At the same time, we strive to maximize the the ability of our teams to operate with freedom within a framework and minimize the amount of centralized decision making, enabling teams to execute experiments as autonomously as possible.
Note that we want to encourage autonomy, not independence. The difference is important. Autonomy means self-directed but can exist within the confines and context of a higher-order system. Independence means not needing or not influenced by others. We want our teams to be autonomous whenever possible, but not independent. Ultimately, we are one Vista team, working together to improve the customer experience.
Building a culture of experimentation
Put simply, we want everyone at Vista to test more—and test more reliably—to deliver more value to small business owners looking for design and marketing solutions. Ultimately, we envision a future where testing is ubiquitous. This will require more than technical capabilities: it will require “a tacit social order [which] shapes attitudes and behaviors in wide-ranging and durable ways”. Our objective is to build such a mature culture of experimentation which is:
-
customer-obsessed both in terms of what we test as well as how we measure success (e.g., front-line teams should receive direct feedback from customer behaviour to inform their work);
-
empowering autonomy by providing freedom within a framework that enables teams to autonomously experiment, decide and be accountable;
-
ubiquitous in nature, supporting decisions across our product suite, customer touchpoints, and value chain (e.g., from front-end applications all the way to the factory floor);
-
trustworthy to ensure that experiment execution and results can be relied on;
-
transparent to ensure that experiments can be reviewed (to assess relevance and reliability) and learnings are disseminated (to increase impact and organisational learning).
Building such a culture does not happen overnight. It requires concerted effort. It needs a team.
Introducing the Experimentation Hub
To build the foundations for this culture of experimentation, we have created a multidisciplinary team spanning multiple departments which is comprised of analysts, developers and data product managers. This Experimentation Hub has a shared objective to evolve the maturity of our culture of experimentation through subject matter expertise, education, documentation, organisational frameworks, and (data) products that scale experimentation capabilities throughout Vista in an efficient, control-inverted and maximally automated way.
The core focus of the Experimentation Hub is not on individual experiments but rather on making the Experimentation Flywheel spin faster. It does this by decreasing the cost and friction involved in experiments—as well as increasing the value of the results. For example by creating standard tools to automate repetitive experimentation related tasks, and by facilitating sharing of results and peer review. Simultaneously, the Experimentation Hub will monitor and guide the quality and reliability of experiments being executed by teams.
We want to build a strong culture of experimentation, so that we can confirm that we are delivering—and continue to deliver—jaw-dropping customer value. With the introduction of Experimentation Hub, we are taking the first steps towards that vision. We will regularly share here highlights from our Experimentation journey—follow Vista Data and Analytics on LinkedIn to stay up-to-date.
Want to help us navigate towards our Experimentation North Star? Explore our career opportunities in Vista Data & Analytics and Digital Product Management.