Skip to main content

Experimentation-driven product development

Experimentation-driven product development is a shift in product development from focusing shipping new products, to focus on shipping features that have an impact on your business. The best way of determining impact is through A/B testing.

The ideal goal with product-driven experimentation is to test every new feature that is developed. This level of experimentation often requires adjusting your existing product process.

If you want to read more about how to make the case for experimentation-driven product development, or the benefits to your culture, you can read our section on experimentation programs.

Platform integration​

Experimentation-driven product development requires a tight integration between your product and the experimentation platform. This is important to keep the incremental cost of running an experiment low, while increasing the ability to run a high volume of experiments. The cost can be cost in terms of effort, and cost in terms of actual money. You want to keep these a low as possible and make sure your platform encourages your team to run experiments.

Given this, many companies choose to build their own experimentation platform. This is a big undertaking, and much harder than it might seem at first. There is also ongoing maintenance costs, as well as the risk in making product decisions on a platform that might have undiscovered bugs.

Reducing costs​

Costs of running experiments can be broken down into a few categories: the cost of data storage, the cost of engineering time, and the cost of the platform itself. GrowthBook is designed to address these costs directly and allow you to run a lot of experiments. GrowthBook is warehouse native and uses any data you already have. It was also designed to be extremely easy to add an experiment (2 lines of code) and have a high-quality developer experience that reduces engineering costs. GrowthBook itself is open-core and extremely economical to run.

Product prioritization changes​

As you become aware of HiPPOs effect on your decision-making process and start to move away from this, you need another way to prioritize projects. There are many prioritization frameworks to help with this (we have a whole section on experimentation prioritization frameworks), but the goal the system you choose should be to encourage building smaller testable features, a high frequency of experiments, and to make sure you have a good mix of ideas and sizes of projects.

This is a big change from the traditional product development process, where we would spend a lot of time trying to predict what features will work, and then spend a lot of time building them. With experimentation-driven product development, we spend less time predicting and guessing and more time testing. This means that we can spend less time building features that don’t work, and more time building features that do work.

HAMM​

HAMM is a framework to help you think about experimentation-focused product development. It stands for Hypothesis, Actions, Measure, and MVP. It is one product framework to help increase your learning rate, and build a culture of experimentation.

  • Hypothesis: What is the hypothesis you are testing? This should be a clear, falsifiable statement of what you are trying to learn. See our section on how to make good hypothesis.
  • Actions: What are the actions you expect your user to take if this hypothesis is true?
  • Measure: What are the metrics that you could measure that would indicate that the user is doing the actions you expect? What might be a counterfactual metric that would indicate that the user is not doing the actions you expect? What are the guardrail metrics that you want to make sure you don't negatively impact?
  • MVP: Given the above, what is the smallest thing you could build to test this hypothesis?

Thinking about the HAMM process at the beginning of a project lays the groundwork for a high-quality experiment. You'll have the hypothesis, the metrics, and the success criteria (OEC). With a smaller MVP or MTP (Minimum Testable Products), you can also increase your experimentation rate and therefore your learning rate.

John Hamm

Why you're not seeing experiment impacts​

Quite often, companies run A/B tests that show positive results, and yet when overall metrics are examined, the impact of these tests is invisible. You might be expecting to see inflection points around the time the experiment was implemented. Here are some reasons why:

  • Confusing statistics - Interpreting results can be confusing without a solid grasp of what the statistics are telling.
  • Bad practices - You may be running experiments that are not valid, or are not measuring the right thing. See the Peeking problem.
  • Lost in the noise - The impact of the experiment may be too small to be visible. This is especially true if you are running a lot of experiments - but this is not bad, just hard to see on a macro level.
  • Optimizing wrong product - You might be experimenting with a section of your product that doesn't represent a large fraction of your overall use. Even if you're successful in these areas of the product, the overall impact will be limited by the small fraction of users that you've affected.
  • Optimizing wrong metric - You might be optimizing for a metric that doesn't matter. For example, you might be optimizing for a metric that is not correlated with revenue, your main KPI. This is especially true if you are optimizing for a proxy metric, such as clicks, instead of the actual metric, such as revenue.

You can read more about this topic on our blog post Why the impact of A/B testing can seem invisible