News publishers increased their subscriber base by 95% on average, between Q1 2019 and Q3 2021, according to the INMA Subscription Benchmarking Service. However, there was a significant disparity between the most and least successful businesses – the upper quartile grew by 262%, the lower quartile by just 66%.
“Become comfortable with experiments that fail”
A new report by FT Strategies focuses on how a structured approach to experimentation can help publishers bridge this gap. The report, The art and science of experimentation for growth, is based on The European GNI Subscriptions Lab – an eight-month program by the Google News Initiative (GNI), the International News Media Association (INMA), and FT Strategies.
“Experimentation has helped us learn what does and doesn’t work with our audience,” says John Kundert, Chief Product Officer, the Financial Times. “It’s been powerful enough for the FT to make the decision to invest in our experimentation and testing capability.”
“One of the most important things to remember is that the more experiments you run, the more your organization becomes a learning organization. And you need to become comfortable with experiments that fail.”John Kundert, Chief Product Officer, Financial Times
“The ability to run robust, impactful and efficient experiments has helped the Financial Times to reach its North Star goal of 1M paying subscribers by 2020 – a year ahead of schedule,” the authors note. “This approach is now guiding the company towards its updated target of 1.5M subscribers.”
A robust experimentation process enables accelerated learning, saves money, promotes reader-centricity, and builds a data-driven culture. The report presents an approach to experimentation developed by the FT over many years. It comprises four steps:
- Choosing the right experiments
- Laying the foundations
- Creating a robust design
- Analyzing success
“Implement a series of small experiments at high velocity”
While many publishers are able to identify different ways to grow subscriber numbers, they need to evaluate their options. “The challenge they face instead is prioritizing these hypotheses,” the authors write. “By thinking about hypotheses in terms of impact (how successful each could be) and risk (how difficult it would be to test each of them), publishers can start to understand which hypotheses to test first.”
The prioritization methodology has helped us to implement a series of small experiments at high velocity. Without this framework, it is easy to cross the fine line between agility and chaos.Philipp Ostrop, Head of Product Development, Lensing Media (Ruhr Nachrichten)
The report shares a prioritization matrix that publishers can use to create a 12-month action plan allocating hypotheses over different quarters in a year.
The next step involves rigorously planning the experiments. “Publishers should invest the time to understand fully the problem they want to solve or the opportunity they want to take,” the authors suggest. “This will increase the likelihood that they explore key risks and assumptions, ultimately leading to better results.”
We have found that laying the foundations is very important. In the past, we didn’t take enough time to clearly define what outcomes our tests should have and what risks might come along.Martin Prinz, Head of Reader Marketing & Sales, Oönachrichten
Publishers should then select an experiment format suited to their specific needs. “Formats vary according to product fidelity (the extent to which an experiment affects the ‘live’ product),” the authors explain, “and reader coverage (the proportion of readers it reaches).”
The next step involves facilitating the design and deciding how to measure impact. For example, a publisher intending to test an approach to increase the proportion of subscribers on an annual package may consider the following hypothesis: “If we make annual subscription the default option, we will increase our proportion of annual subscribers whilst maintaining the same conversion rate, because readers will appreciate the value proposition presented to them.”
The Courier, the second largest regional paper in the UK, set itself the goal of growing its subscriber base eight-fold by 2025. The publisher had not yet found the right balance of content to lock behind the paywall. So it tested increasing the volume of locked content sixfold and starting a weekly review meeting between its Audience and Editorial teams. Subscriptions sales volume grew by 182% over the course of eight weeks after the start of the test.
“An important part of our daily business”
Finally, the authors underline the importance of analyzing the results. “Experiments should only be concluded when statistical significance has been established,” they write. “Publishers should bear in mind that follow-up analysis is often necessary, sometimes months later, for example analyzing annual retention rates in a win back experiment. This enriches findings and informs further experimentation.”
“If a publisher has designed an experiment well, minimizing the variables and progressively testing risk, it is unlikely that a hypothesis is proved or disproved after a single iteration,” they add. “Publishers will need to experiment further because by the term ‘experiment’ we mean ‘a series of iterative experiments’”
Adopting this approach has helped the participants of The European GNI Subscriptions Lab achieve success in a short space of time, according to the report. The publishers identified ways to grow their reader revenues, began to experiment in these areas and started to build the governance structures and processes to support a culture of experimentation.
We now take a more structured approach to experiments; breaking down different ideas into specific small steps makes it easier to quickly see results and to change parameters. Setting these up has become an important part of our daily business.Martin Prinz, Head of Reader Marketing & Sales, Oönachrichten
The full report can be downloaded from FT Strategies:
The art and science of experimentation for growth