Not understanding growth is dangerous

To understand how your product grows, is essential to understanding what is most important to grow it further

I was working for a B2C company that had two main product lines. Combined together it seemed like growth was “fine”.

Digging deeper…

1. Product A: High margin, growth was plateauing

2. Product B: Low margin, growth was strong

We were headed for an unpleasant reality-check.

In short, we had a Retention problem with Product A, and no one saw it coming even though it could have been predicted years in advance.

When broken down into a series of inputs & outputs, it all makes a lot more sense. Problems in Acquisition, Activation, Conversion, longer term Engagement & Retention can all contribute in different ways.

So little time, so much to do

Every executive wants to see revenue growth, faster.

In our constrained reality, there are so many more ideas than time & resources allow. The quarter comes and goes like a flash.

How do we make the most impactful choices?

When I come in to a new company, there’s an essential activity that I never skip out on, and that’s building up an understanding of how the product(s) grow.

By understanding that, we can maximise the time spent on high leverage opportunities.

There’s only a certain distance we can go based on experience-based pattern matching, and heuristic analysis of where the new product might be lacking. Once these easier opportunities are captured, then we need a systematic way to drive growth.

An essential activity is to build a Growth Model, which allows us:

1. To understand mathematically how growth happens

   1. Enabling prediction, and a clear overview of the system

   2. Finding the highest leverage points

2. Gives us a structure to delve deeper to find the hidden relationships

   1. Enabling us to discover the metrics that influence key product outcomes

What we’ll do, in a nutshell

1. First, we build the Macro growth model.

   1. Identifying KPIs that create growth

2. Next, we target an area, and develop the Micro growth model for that area.

   1. Identifying Influencing factors that drive the hard Macro metrics

3. Finally, we can “Move the needle” - in other words, grow faster.

Difference between the Macro & Micro level

Macro level KPIs are somewhat “universal”, in that they hardly vary from product to product.

They have a deterministic relationship to each other, unlike Micro level KPIs which we’ll talk about next.

It’s fairly easy to map these out, and they serve the purpose of aligning your focus with others on what Business & Product Outcomes[1] you’re trying to influence.

Micro level KPIs, what Teresa Torres might call a “Traction Metric”[1], have an influential relationship with the Macro level Product, and Business outcomes. By influential, I mean the Micro level influences the Macro to a degree, but it’s not a 1:1 relationship.

Discovering these connections requires a deep curiosity for what your users find valuable and a persistence to establish a measurable connection in the data.

Example of the intended outcome:

• We have a B2C content creation tool

• You’ve mapped out your Business & Product metrics to understand how you grow.

  • Your conclusion is that improving your Trial Conversion rate is the highest leverage activity, much better than Signup improvements ✅

• You investigate the relationship between certain key actions and Trial Conversion

  • You find that users who export their first file by Day 2 have a 2X higher chance of trial conversion ✅

• You can now Move the Needle

  • Focus on mapping out opportunities in the new user experience to enable more users to export within 2 days ✅

  • The timeframe gives constraints around how you need to approach the problem space ✅

So let’s move on to the focus for Part 1: The Macro perspective

Subscribe now

Part 1: Understanding your growth from a Macro perspective

What does it look like?

You might have seen/read about a “KPI Tree” before.

It’s an interconnected set of metrics, built to model how different parts of a system affect each other.

e.g. Increase conversion rate → New Revenue increases

This enables us to:

1. Understand how revenue is created. Predict future growth → Plug in your numbers, and you’ll have a decent view of future growth of the system, and

2. Understand how value is created. And, how value relates to revenue, once we go deeper into the Micro actions that influence the overall model

Example:

Here’s a visual representation of the “Macro” part of a KPI Tree for a SaaS product with monthly & trial subscriptions:

It doesn’t need to be exhaustive, since the main goal here is to create a high level view, and understand which areas are most promising.

We could break this down further, e.g. see below, but for now the broad overview is most important.

And what it could look like in a spreadsheet:

Once you have it in spreadsheet form, we can do the fun part → use it to see where we have the most leverage.

Setting up your KPI tree

Example:

• We have a generic SaaS product a single monthly plan

Firstly, how does your product make money?

→ It’s the sum of new revenue, plus revenue from existing customers (recurring revenue), minus the churn.

New Revenue

1. New Revenue = New Customers * Average $ / Customer

2. New Customers = New Signups * Signup → Customer Conversion Rate %

3. New Signups = Unique Visitors * Visitor → Signup Conversion Rate %

Recurring revenue

Comes from existing customers, which are existing Yearly + Monthly customers

Churned revenue

I’ll use the massive oversimplification of a flat % churn against the existing revenue base = Existing Revenue * Churn %

This holds only for established products where the growth & churn rates have reached a sort of equilibrium, but for this exercise it’s far easier.

The above, represented visually:

We have the bones of a basic KPI tree!!! 🎉🎉🎉

Going a level deeper

Each of the steps we have listed consist of many substeps.

If we break them down one level further, we’ll have enough detail to start calculating leverage.

How much detail is enough?

It’s possible to go to extreme detail e.g. literally every must-do action, but at first I’d recommend building an overview of the distinct sub-journeys.

So, something like this could make sense for Visitor → Signup:

1. Visitor → Start signup

2. Start Signup → Account Created

3. Account Created → Completed Segmentation Survey (for example, a short survey which asks who they are, what kind of work they hope to accomplish with the product)

So now, in terms of influencing New Revenue, we can put this in a spreadsheet to get what we need.

We’ll use “relative” conversion rates, since that makes more sense for identifying opportunities.

Putting it all together with some example Visitor numbers & Revenue / customer, and we can model the system over time, getting an output like the one below:

Okay cool, but without having an idea of where to focus, this is not useful.

Finding leverage

Where can you make the most impact?

It’s usually where most of the pain/friction between the user & value exists. But we haven’t really mapped out value creation yet. (that comes in Part 2)

We don’t have the time to deep dive into every small part of the journey, so we can use the following exercises to find areas of interest to explore further.

It’s also dependent on how much there is even left to gain, and that’s why mapping out the numbers is crucial. As we’ll see next, some steps are naturally limited in the amount of gains we can even make.

The mathematical

Imagine you could improve any step by a 10% relative increase in conversion rate.

What would the overall uplift on the system be?

Always, 10%*.

(*As long as that doesn’t bring that step to above 100%)

In other “words” → A * B * 1.1 = 1.1 * A * B

The problem is, each step has a WILDLY different opportunity for uplift. In my experience, of users who begin a signup process, roughly 70-90% will complete it.

For a paywall, it might be only 10% that see it, end up buying.

So for these two scenarios, if we add a relative 10% uplift:

1. 10% increase on 90% brings it to 99%.

2. 10% increase on 10%, brings it to 11%.

Which do you think is more realistic?

In reality, 1. will never happen, but 2. is generally achievable.

Simple rule of thumb:

When conversion rates are lower, e.g. <30%, then we’re looking for 10-20% uplift milestones on that converting share.

When conversion rates get very high, the better way to think about it is that you’re trying to convert a % of the remaining non-converters. e.g. for the 90% conversion step, it’s the remaining 10% that we want to capture 10-20% of.

Shown mathematically:

1. 100-90 = 10, then 10*1.1 = 1%, therefore 90% → 91% = 1% uplift on the system

2. 10% → 11% = 10% uplift on the system

This means that a similarly realistic goal for Signup & Paywall conversion would be an order of magnitude different in terms of impact.

The closer you are to the conversion rate ceiling, the ever more marginal the gains.

A 90% conversion rate step can never exceed 11% relative improvement.

A 10% conversion rate step can eventually gain 30-100% over months and years.

This means we’re generally going to be biased to favour low-conversion steps for improvement

The subjective

As mentioned earlier, where’s the pain?

You must form a strong opinion on the flows you’re responsible for, and keep it up to date.

You might have a best-in-class signup experience, but if your Paywalls are still default templates, you might have a strong hunch that the opportunity there is significant.

These judgements inform the mathematical side.

The main sources of this information are:

1. Customer feedback → Customer service, interviews, surveys, session recordings

2. Competitor awareness → Knowing how the strongest rivals approach their UX, and also non-competitors who simply do those types of experiences well

3. Your own product sense → The longer you spend in the business of making lives easier, the easier it is to detect potential problems

The benchmarks?

In my view, these aren’t going to be super helpful → Why? Because every product has subtly different customers, geographies, and definitions about terms.

There can be some value in them, but I think they can take energy from the hard work of gaining an ever-deepening understanding of your unique userbase.

Example Macro Growth Model spreadsheet

Here’s an example spreadsheet of a basic KPI Tree that you can play around with.

SPREADSHEET LINK

Visual representation:

What next?

We’re going to get to the fun part, which is: Diving into the Micro → Identifying influencing factors. What good is knowing I want to drive Trial Conversion, if I don’t know what influences it?

Additional Reading

1. Petra Willie on KPI Trees https://www.petra-wille.com/blog/kpi-trees-how-to-bridge-the-gap-between-customer-behavior-product-metrics-and-company-goals

2. Teresa Torres on Opportunity-Solution Trees, a strongly related topic: https://www.producttalk.org/2023/12/opportunity-solution-trees/
   

Appendix

[1] I use the following terms as defined by Teresa Torres here

1. Business outcomes measure the health of the business and are typically financial metrics like grow revenue, increase market share, or reduce churn. e.g. New Monthly Recurring Revenue

2. Product outcomes typically measure a customer behavior in the product or a customer sentiment about the product. e.g. Signup completion Conversion rate %

3. Traction metrics measure adoption of a single feature. e.g. % of users who make their first export within 1 day of signup

See appendix note below for more

[1 - See Appendix]

You’ve just signed up to a new product with a trial.

You always seem to forget to cancel those, so naturally, you navigate to the “Profile/Settings” area to manage your subscription - you want the “Cancel” action.

Can’t find it.

You suspect they hid it on purpose.

Buried within a nearly impenetrable labyrinth of screens - you find the button of power to prevent you from losing $50 (again).

Now the true test begins.

The… Cancellation Flow.

Step after step of swapped button placements and colours, repeating a very similar rewording of “Are you... really sure you’re sure?”, wearing down your rapidly depleting willpower until you get some unrelated notificat-

*PING*

Ah yes I had to get back to whats-their-name about whats-that-thing.

Let’s take care of that now… done. And what was doing again?

I’ll cancel this later.

(You won’t, but you’ll probably complain to your friends/partner/cat that it was too hard to cancel)

What are they, and why are they used?

One of the most common “growth/retention hacks” is the attempt to intercept the customer on their path to cancellation and change their mind. At this point, if you’ve got nothing to alter their course, there’s a near-100% chance you’ve lost a customer.

You see them almost everywhere in the B2C world, and even some areas of B2B (especially the “Prosumer*” segment).

Because.

They.

🌈✨Work✨🌈

*Prosumer: Targeted at serious, enthusiastic consumers, incorporating professional features but often modified for non-professional use. (wiki)

Why do they work?

1. We hate friction. As is the case with activating new users, the greater the non-valuable friction is, the fewer will be successful with your product. The same is true for cancellation → fewer will complete the action, but it’s important that we keep this friction in good-faith.

2. We love a good deal. If you offer an outgoing customer a juicy deal, it can be hard to say no. 50% off for the same value will cause most to re-think their choice.

3. It’s relatively easy work for the gain. The alternative → add new features, products, or improving the new user experience is a tough job that’s never really done, and can take 3-6 months to build up learnings and wins in a new area.

The moment of the customer initiating the cancellation process is absolutely decisive.

If you wait until AFTER they cancel, the post-cancellation resurrection success rates are in my experience in the (very) low single digits if you’re lucky.

My teams have implemented these flows in a couple of different products - to great effect - and I’ve looked at a LOT of examples so you don’t have to.

There are a few catches though:

1. They aren’t for every product/industry

2. The “addressable canceller” share for Mobile Apple/Google store subscribers is low (because they tend to cancel directly in the respective stores).

3. They shouldn’t be the first tool to reach for, since there’s typically a mountain sized opportunity in improving activation to prevent cancellation actions in the first place (“Growth through Activation” covered in future issues of the newsletter)

What we cover here

1. Common cancellation flow steps, what they’re for and when to use them

2. Good examples to sample from

3. How hard is too hard?

4. Does a cancellation flow make sense for me?

   1. The “Mobile problem”

   2. Is it technically feasible?

1) The common cancellation flow steps

Note that many of the screenshots in this article were sourced from Mobbin, an excellent User Flow sampling tool, which is super useful for inspiration & research.

If you’re interested, you can get 20% off your first billing cycle of the Pro plan here. (Disclosure: I get $10 of Mobbin credit for successful referrals)

The “Survey”

One of the most common steps, and for good reasons.

1. Source of data about why customers cancel. This is the best time to get this information, since emailing them later will have a much lower success rate, and the reason is the most fresh in their mind right now.

   1. Note: If you’re including a “Price” related question, it’s highly likely this will be the most selected one. This can mean a lot of things outside of purely “price”. They subscribed at that price in the first place, so it’s theoretically affordable, but the actual value received over time wasn’t sufficient to justify continuing. (It’s of course possible for the financial situation of someone to change over time, but most of the time I believe it’s more value related)

2. Friction. It’s a process interrupt that can make someone reflect on how they really feel.

3. Versatile. Valuable for most use-cases, not that dependent on industry or target customer.

Tips:

1. Maximum 5-7 options, fewer is better imo, especially if you’re lacking insights yet on the main reasons

2. Allow an “Other” with a text answer for them to give you more information.

3. Keep it optional → In my opinion if you want higher quality data, don’t force them to share. Either skippable or a “Prefer not to say” option

   1. Alternatively, you can go the Netflix route and show this screen after the cancellation is confirmed. You trade “friction” for even higher quality data.

Here’s a decent article from Userpilot’s blog on the topic - Note this is more related to a survey the user can complete in their own time, but I feel the insights they recommend focusing on are valuable.

The “Discount Offer”

Fairly common in the B2C world, where the end user is also the buyer, and decisions are more made on impulse.

It’s a last-ditch effort to save a customer that’s already highly motivated to replace you (or already has and you don’t know it yet).

It’s a reward for the least loyal, and it can be highly effective in the right situation, for the right type of customer.

I’m talking >10% customers saved in a good case.

Problems:

1. Not suitable for all segments (See below)

2. If you’re too aggressive with discounting in general, this can erode the value of your product over time.

3. Tying into 2. above, word might get out about your cancellation/discount policy.

   1. Note: I haven’t personally noticed increases in users intentionally canceling to trigger a discount since implementing the cancellation flows themselves. The thinking is that someone will share the fact there’s a deal, then more start to check even without the original intent to cancel, but I haven’t seen evidence of this on a significant scale)

What kinds of customer segment does this make sense for?

B2C - Highly suitable

Customer is paying out of their own pocket, decisions about their own life can also be highly emotional. We’re a lot more susceptible to impulse buys than a multi-user team.

B2B Prosumer (Individual users but professional use-case)

Fairly suitable, situationally dependent. Often they’re paying out of their own pocket, are more price conscious which can make them work.

I would be more careful about the size and frequency of discount, since if you’re targeting the Prosumer segment, you’re also likely eventually looking up-market to expand into larger accounts.

B2B Startup+

I don’t think it makes a lot of sense. The decision is more of a collective one, and if they’re cancelling, they’ve likely already vetted and implemented an alternative.

The “Value reminder”

A personal favourite of mine, the “Value reminder” is constantly evolving in the world of cancellation flows, and a tactic that can be employed either:

1. Within the flow itself.

2. Before the cancellation starts

We can see it used to great effect in Canva in-flow, who show you how much you’ve used which premium features over time.

I’m a big fan of this implementation, because with some products it can also be confusing what is/isn’t included in the free plan. In this way they give more certainty about what you lose while simultaneously reminding you of the value you’ve gotten. 💡

The second option is working this value display into area where the user would normally initiate cancellation.

A highly effective example of my Wolt account (Food delivery). The core value prop of Wolt+ is saving money on delivery, and that is on clear display here.

Why would I cancel their €40 subscription when I can see saved me already €200 over the past 7 months?

Prompt: What is the core value that you’re delivering, and how might you remind the customer of that as they’re considering their option to cancel?

The “Down-/Sidegrade”

If it’s done well: Appealing to the customer’s lack of awareness of other plan tiers. It could be that they’re subscribed to a plan that has more than they need, and a lower plan holds better value for them.

If it feels there just because: Friction for friction’s sake

I particularly like Asana’s approach (the example below on the most-right), since they can offer a plan that better fits the value delivered.

The “Pause”

Sometimes customers want to cancel due to only having an intermittent need, and in this case, the Pause can make some sense. E.g. a B2C gym membership, and I’m going on holiday for a few weeks, or a B2B Prosumer design tool which is used on a project basis.

If this sounds like your product, then this can definitely be a good choice. Below is an example from Bloom, an investing app where they try to sell the value of the pause with loss aversion → The customer can take a break without having to sell all of their stocks, and can resume in the future without any hassle.

The “Are you sure you’re sure?”

When there’s a double up of very similar steps, it’s hard not to think they’re there purely for friction.

I’d advise against these kinds of patterns when you have nothing substantial to show or give customers.

Duolingo gets a begrudging pass from me on Step 1 here, since this example is of a trial user, and it’s super common for people to cancel immediately simply to avoid getting charged without their consent in case they forget.

Truebill however, a product that is there to help you manage costs from your subscriptions, gets too close to a step duplication in my opinion. Oh, the irony…

2) Good examples to sample from

B2C mobile - Duolingo

1. They start with “Change Plan”, which is a way to introduce Down-/Sidegrade naturally into the flow

2. Survey

3. (If Trial) - Trial Reminder Reminder → Because many users try to cancel ‘cause they know they’ll forget.

4. Are you sure you’re sure? (not a big fan)

5. Done ✅

B2B SMB/Prosumer - Canva

A great example of clear, quick, respectful.

1. Value reminder

2. Survey

3. Done ✅

4. Post-cancellation qualitative feedback/suggestions

A nice collection of cancellation flows to sample from, by Khushi Lunkad on the Figma community→ https://www.figma.com/community/file/1343077677230477273

3) How hard is too hard?

When your flow is named internally “The Iliad Flow” after the Homerian Greek epic about the long arduous Trojan War”, you might just be the bad guys.

The cancellation flow is a part of your product experience, and you should therefore treat outgoing customers as well as your incoming ones.

Because people remember. And people talk.

Too many people think that “churned” means “eternally vanish from this plane of existence”.

Have you ever checked your re-activation numbers before? If you’ve been looking at your customer lifecycle as a flat funnel with a beginning and end, you’re missing the big picture → that many customers re-activate, churn again, re-re-activate, and so on.

Even if it wasn’t a perfect fit for them perpetually, they might need you again in the future.

Or, it might be a perfect fit for someone they know.

We are highly social creatures, and we talk about stuff. We can also be petty and spiteful (speaking completely for myself here). If I feel like I’m getting screwed over, you better BELIEVE if I hear that name mentioned, or someone asked about my day, that I’m going to tell you about it.

If you handle your cancellations with grace, and they had a mostly positive experience, they could still recommend you to connections of theirs in the future.

It’s a different story if you’re holding a (near) monopoly.

There’s a reason why Apple hasn’t added date-range export for Offer Code redemptions in Appstore Connect, YouTube will continue stacking ads until we’ve come full-circle back to the “TV days”, and Amazon pre-2022 required you to recite 1000 digits of pi to cancel.

What are you going to do about it as an individual? Nothing.

Well that’s not entirely true → you can complain to your local consumer authority 😈

Because Consumer Protection

In researching this article, I tried to cancel my Amazon Prime subscription. It was shockingly straightforward. What happened to the Ninja Warrior obstacle course of dark-pattern f*ckery?

I naively theorised that the marginal gains of ever-worsening cancellation flows weren’t worth the long-term negative impact, so they voluntarily made it easier 😂. (Because there’s a reason why products competing in healthy markets treat you well → they have to to survive).

It wasn’t competition pressure, rather, the EU that forced Amazon to shorten and simplify the aforementioned “Iliad Flow”.

Another notable example is Adobe, which the FTC took action against this year. The assessment is fairly damning:

“Adobe trapped customers into year-long subscriptions through hidden early termination fees and numerous cancellation hurdles…”

Just. Don’t. Do it.

LAST MINUTE NOTE: On the October 16 2024 ruling by the FTC ‘Click to cancel’, this could change the game for what is/isn’t allowed, but I haven’t had time to properly digest it. Fortunately, those with actual legal experience have done so, see appendix link below.

[1 - See Appendix]

Are we making it too hard? My rule of thumb

It’s simple: “Would I feel frustrated going through this?”

It will keep you on the right side of karma, and the law*.

*This is not legal advice 👩🏻‍⚖️

4) Does a cancellation flow make sense for my product?

Eventually, I believe most digital products should have one.

Firstly, this should come after significant work has been done into ensuring

Different steps make sense for different industries, as mentioned prior. If you’ve got a B2B team-focused product, then a discount offer doesn’t make sense, but in all cases a survey, and “Value Reminder” design is going to provide value.

If you’re a mobile app using Appstore/Playstore as a significant source of subscriptions, then there is more to consider.

The “Mobile” problem

If Apple Appstore or Google Playstore are a main subscription source for you, then this makes things a little more tricky. All mobile subscriptions need to be canceled in the context of the store, and it’s also the “default” place to go to initiate cancellation for most customers.

In my experience, even if you have a “Settings/Profile” area with a “Manage/Cancel subscription" button, easily over half of mobile cancellations will happen without any interaction with your Settings/Profile area.

This number goes way up when talking about recently initiated trials.

For products with a trial, there’s a huge spike in cancellations in the first 1-5 minutes after trial start (those who don’t want to forget / are just using trial for something temporary), and a vanishingly small fraction of those will turn it back on.

So is it worth it for my mobile app?

It absolutely can be.

But whether it makes sense for you now depends on many factors.

Have you already heavily invested in Onboarding & Activation of new users?

Preventing new customers from wanting to cancel in the first place is the right focus most of the time.

By ensuring they have a great first impression, onboarding, all the way into forming a routine is something I strongly recommend exploring before looking for the growth “hacks”.

Do you have a significant “addressable share” of users?

By “Addressable share” I mean the % of cancelling users who initiate through a touchpoint that you control e.g. settings/account area in your app.

This is useful for understanding how many more users you might be able to funnel into your cancellation flow.

I’d highly recommend checking the following data for the past month:

1. How many cancellations did you get? (Not the # of subscriptions that expired, but rather the # of customers who turned off auto-renew)

2. How many unique users started their cancellation journey from inside your app i.e. tapped on “manage your subscription” or “cancel”.

While you can directly measure the absolute # of “addressable cancellers” by the unique “cancel” taps, knowing the share gives us a baseline for trying to push that number up also.

Is it technically feasible?

If you want to offer a discount step on mobile, it can be a significant effort, especially if you haven’t got a full-stack team who can implement everything for you. In my experience the solutions Apple provide for this are far from ideal, full of manual work (Promotional Offers, Offer Codes), but Google is more flexible since rolling out their v2 subscriptions a couple years ago.

Thankfully, there are now products popping up offering plug & play cancellation solutions e.g., Raaft, ChurnKey to name a couple. I haven’t got first hand experience with these to confirm their limitations, nor are they sponsoring me, but they’re out there.

Putting it together

Let’s take a few scenarios:

1. If only 10% cancel starting in your app, then it’s likely there isn’t a large opportunity to be captured yet → You might experiment with encouraging more to initiate cancel in-app

2. If 40% cancel start in your app, the opportunity to be captured is decent, and we can try to divert more users towards the flow after building it → You might build the flow first, then encourage more to initiate in-app

3. If 70% already cancel in-app, then opportunity is high, but the opportunity to get more users in there might not be as high → You might just build the flow

Summary

1. Cancellation flows can be high impact growth lever, especially for B2C, and especially after you’ve gotten good at activating new users and making them successful

2. For Mobile, there are significant catches when considering a discount step, like the share of users cancelling in-app and the feasibility of implementation

3. Treat your offboarding like any other part of your customer experience, with respect and care

4. Please don’t build the “Iliad Flow”🙏🏻😭

Thanks so much for reading, and I hope you got some valuable insights to take away 💡

What’s next?

Growth + Impact model series → Understanding leverage to drive impact

Perhaps the most important job for anyone making Product or Growth decisions, is deciding “where can we have the most impact?”. You might have seen a super cool design/idea in the wild that you’re excited about, but will it move the needle the most?

I love exploring these questions, and will reveal my findings over the next few episodes, in different levels of depth/complexity.

Appendix

[1]Note on the October 2024 ruling by the FTC ‘Click to cancel’

Source for below: https://advertisinglaw.fkks.com/post/102jly9/click-to-cancel-is-here-as-ftc-announces-final-negative-option-rule.

• Cancellation.  Sellers must provide a simple cancellation mechanism, i.e., one that is “at least as easy to use” and available through the same medium as the mechanism the consumer used to consent to the negative option feature.  The rule specifies the different minimum requirements for this “simple cancellation mechanism” for various media (online, phone and in person).  Notably, for online cancellations, the rule states that “[i]n no event shall a consumer be required to interact with a live or virtual representative (such as a chatbot) to cancel if the consumer did not do so to consent.”  And for in-person cancellations, an online or telephonic method of cancellation must also be available.

One thing is crystal clear → No more “Talk to an agent” to cancel, unless that’s how the customer signed up in the first place.

“At least as easy to use” could be open for wider interpretation, but this is all the more reason to design these flows ethically (without repeat steps, confusing design etc.).

Recap from Part 1

1. Evaluating Paywall AB tests accurately can be difficult due to complexity and sheer number of potential metrics

2. New LTV/User is our #1 success metric for these tests, which is how much revenue we expect to make per user who sees the paywall, over then next X years - (Usually at least 24 months but can be longer)

3. It’s necessary to have an accurate LTV per paying customer in order to calculate this

What we’ll cover in Part 2

Because we have to prep the right data, the next sections are on making sure we have our customer LTVs ready to use in the Main Calculator.

Main points:

1. Estimating LTV for use in our calculations

2. Using the Main Calculator

3. Evaluating success

Choose your own adventure

LTV Preparation:

1. For those with more than 24 months of subscription/retention data AND your billing system can report on this for you → See “Tips on how to find your LTVs” section from Part 1

2. For those with more than 24 months of subscription/retention data, BUT the system doesn’t calculate it for you → Read on to the next section “Estimating customer LTVs with the LTV Calculator (when you have >24 months of data)”

3. For those with fewer than 24 months of data → skip to “Estimating LTVs when you have <24 months of data”

If you already have your customer LTVs ready:

You can skip straight to → “The Main Calculator”

Estimating customer LTVs with the LTV Calculator (when you have >24 months of data)

Firstly, what does it do?

You put in retention & subscription value data, and it gives you 24 month LTV values to use in the Main Paywall AB Test Calculator.

Note: The LTV calculator works up to 36m estimated LTV, but I think in general 24m is a better middle-ground to use for the actual AB test calculator in most cases.

1. First, you need some kind of “Subscription/Cohort Retention” report, which can give you a relative % retention by period after initial subscription period. Essentially it means “What % of customers still have active subscriptions in Month/Year n after their first subscription”

2. Second, you need ARPA/ACV (Average Revenue Per Account / Average Contract Value) for each subscription type - it’s simply the average (mean) value of each paying customer

3. Next, we enter the data into the calculator as below:

Then on the right side of the sheet we have the outputs:

Now you have your customer LTVs ready, you can now skip to “Using the Main Calculator”

Estimating LTVs when you have <24 months of data

Example: You launched paid plans ~3 months ago

Assumptions:

1. You’ve got two subscription lengths, monthly + yearly

2. You know what % of monthly customers have churned from that first monthly cohort in months 1 & 2 after month 0 (first purchase month)

3. You know what % of yearly customers have already canceled / turned off auto-renew for their subscription

Estimating Monthly Plan LTV

• 3 months of data on your first monthly cohort, and the retention of this cohort looks something like the below, dropping to 50% in M1, and 30% in M2

This puts us at a relative 50% loss in M1 (50/100), and 40% loss in M2 (= 20/50), so we can build some rough bounds around what is a likely range of outcomes here.

We’re already seeing a relative slowing from 50→40% churn in M0→M1→M2. Assuming your product is solving an ongoing need, and most of the losses you make in M0/1 are due to lower intent/willingness-to-pay/product-market fit users, then the relative churn will tend to slow down somewhat.[1 - See appendix for more]

Below I outline a couple of potential scenarios based on those first 3 months of data.

Scenario A (optimistic): Churn slows down from 40% to 30, 20, 10%

• Product solves ongoing need

• Retention flattens out at a higher number

Scenario B (pessimistic): Churn stays at 40% relative until M6

• The vast majority of monthly customers don’t need, or don’t want to choose your product in the long-term

• Retention flattens out at a very low number

  

Reality is very likely to lie somewhere between these ranges. Looking to our 24m estimation, let’s assume retention remains relatively flat after 6 months. It’s possible there’s churn to zero, but if the product is designed for ongoing use, that isn’t the most likely outcome.

Assuming flattened retention at 6 months, we get something like the below graph:

The LTV we collect, is effectively the area under the graph here, and you might be surprised at the difference between these two.

Let’s enter the numbers as shown below and see what comes out:

What would you guess Scenario B’s advantage to be?

30% more?

50% more?

.

.

.

It’s 70% more revenue collected over the 24m timeframe.

After repeating this for both, we get:

Scenario A: $29.2

Scenario B: $49.9

A relatively small looking change in assumptions can create a HUGE difference when it comes to retention & LTV. This can be dangerous when making significant decisions like “which paywall design to go with” that may favour one plan or the other.

In my personal opinion, I’d split the difference between Scenarios A & B, something like $35-$40 for monthly, and update the value as each new month of data brings us more clarity. Perhaps even closer to B assuming the worst, since undervaluing shorter terms is a less harmful mistake to make.

The rule of thumb is longer subs will be worth more, but there are certain types of products where they are designed to solve a time-limited use-case, and there might be strategic value in that e.g. an App that supports a weight-loss journey that aims to have you achieve a certain result within 3 months.

Estimating Yearly Plan LTV

Since it’s been less than a year since launching your plans, you don’t have any renewal data. How can we best make an informed guess about this?

1. Cancellation numbers up to this point

   1. Some % of customers will have already turned off auto-renewal, and it’s safe to say that the vast majority (let’s round it up to 100%) of those won’t turn it back on

2. Benchmarks. An amazing resource is Lenny’s Newsletter issue on retention benchmarks. It’s going to heavily depend on where you’re playing, let’s assume B2C consumer SaaS.

   Note: Technically, this is Net Revenue Retention, which is a measure of retained revenue (as opposed to % of customers or accounts retained), which factors in the possibility that some accounts expand after a year, which drives up the overall number, but I think it’s good enough.

3. Intuition & Guesswork

   1. Typically there are cancellation spikes at the beginning and end of the billing period, with relatively even increase in cancellation during the middle.

   2. For yearly plans, this can look like a huge spike during the first day(s), another spike in the last weeks before renewal (especially if customers receive a reminder).

   3. Since this can vary so wildly depending on industry, product, customers, plans etc., there’s no simple formula or calculation for this (as far as I’m aware).

Note: At time of writing, some billing systems are experimenting with features to predict LTV, since they’re realising they can do so with the data from thousands of business customers. You can read how RevenueCat’s new Prediction Explorer works in that way.

Example:

If you’re launching a new product in this space, let’s assume our baseline assumption is somewhere under the “Good” benchmark of 55%, say roughly around 40%.

1. If 30% have canceled at 3 months, 70% is our absolute ceiling with 9 months to go. In this case 40% might be a fair range.

2. If 60% have canceled at 3 months, 40% is our absolute ceiling, so we could be looking more in the area of 20%

Using the Main Calculator

My spreadsheet calculator that I use, to help you make better decisions

This is a more readable version of what I already use live, and while I tried to keep it simple, there’s a necessary level of complexity here that needs explanation.

Features:

1. Calculates the most valuable success metrics for you

2. Includes an LTV calculator sheet to help you estimate Predicted LTVs, which is especially helpful for products where you don’t have 1-2 years of data already

3. Reduces stress headaches (maybe)

Limitations:

1. IMPORTANT: In the interest of simplification, I’ve taken some unscientific shortcuts in how we declare success. My goal here is not perfection, but improvement over most peoples status-quo in [2 - See appendix]

2. Only two versions in the test (Control + Variant)

3. It supports 3 types of subscriptions: Monthly, Trial → Yearly, & Directly yearly*, since this is one of the most popular packaging configurations

   1. Weekly, quarterly lengths aren’t in there to keep the thing a bit more manageable, but in principle this can be extended quite easily.

   2. *Note: Direct Yearly is there for users who have already used trial, since you might be testing paywalls for existing users, not only new ones.

4. It’s far from exhaustive in the metrics it provides, simply because I believe the vast majority are situationally useful at best, distracting and meaningless at worst for the purposes of determining success.

Preparing your data

Test Details:

1. Start & End date of your test

2. # of users in control + variant

Subscription / LTV details

Example for monthly:

1. Average Monthly MRR is the same as the Monthly ARPA (Average Revenue per Account) - the average $ per paying monthly customer from the test

2. Monthly 24m LTV / paying Monthly user is what we’ve derived from the LTV Estimation earlier in the article, OR the LTV Calculator, OR directly from the billing system if we have enough data on that.

3. Monthly Conversions is how many converted during the test, which may or may not have already been corrected for refunds. I know that some billing systems factor in refunds in their “Conversion to Paid” reports, but not necessarily in the results from the AB test. Double-check with your provider to be sure. If they do factor it in, simply set refunds to 0

4. Monthly Refunds is how many customers refunded after purchase

Then repeat for the other sections as needed.

On the other subscription sections:

If you are running an AB test on your signup paywall only, and you only offer monthly + trial into yearly, then you can zero out the “Direct Yearly” section, and similarly if you only have yearly plans, you can zero out “Monthly”.

Example below:

These three sections aren’t special in any way, so theoretically they can be mapped to whatever setup you have, e.g. Weekly + Monthly + Quarterly if you’ve got shorter plans

The Results & Success metrics section

Have you got enough data?

This article is targeted at evaluating the results, but I had to include a disclaimer in the sheet regarding sample sizes.

Here’s a link to my go-to pre-test calculator, which you should use to determine test length based on your traffic.

If you don’t have enough traffic to detect a 10-20% uplift within 3-6 weeks (20% is a BIG improvement), then you should reconsider AB testing as a tool for you in the first place.

One “shortcut” that I won’t fully address here is that the main success metric, 24m LTV, is calculated through weighting & transforming conversion metrics, which takes a step away from the basis of binomial split testing. [2 - See Appendix]

Evaluating our primary success metrics

As mentioned in Part 1, our north star is the LTV/User, since it’s the best answer to the question “Are we going to make more money?”.

However, if you’ve got a simple setup (single plan e.g. monthly only), then you don’t really need the spreadsheet, but Overall Conversion Rate is good enough (Success Metric 2).

Is this difference big enough?

As mentioned before, my guidance here is a simplification, and it depends on how you’ve setup your test.

You’re roughly aiming for at least the Minimum Detectable Effect (MDE), which is a key factor in determining how long you need to run your test.

Example

If you’ve set an MDE of 10%, and hit the required level of traffic:

1. An increase of >10%, it’s highly likely our variant is better

2. An increase of <10%, down to a decrease of <10%, more or less inconclusive

3. A decrease of >10%, it’s highly likely our control is better

As mentioned in Part 1, if you’ve set up your conditions pre-test with a trustworthy calculator (for example from Evan’s Awesome A/B Tools), and run it for long enough to get enough traffic, then I do believe this will be “good enough”, and superior to applying the “correct methods” to the wrong success metric.

Another point worth mentioning is that this is just one tool for de-risking a pricing/packaging change. You should already be going into testing your paywalls with a hypothesis in mind e.g. “We believe we’re underpriced because X”, or “This packaging aligns the value we charge for in a better way for our customers”, and the testing helps to strengthen that decision.

It can be correct to reject a change because it was inconclusive, or implement a change because it was at least inconclusive i.e. not drastically negative. Every product’s specific context will be different.

The “Projected Impact” section

I included a “Projected Impact” section to understand the potential of a variant on a weekly/monthly/annualised scale, but this is only if the measured uplift/drop holds.

Continually measure for a few weeks/months afterwards to be sure.

Note: This is difference in New LTV, so the impact “per month” means during the average month of new conversions, we would expect to gain that extra revenue over a 24 month period from those customers.

The “Calculations” section

How everything is calculated is out in the open here for your interest.

Summary

1. New LTV per user should be our primary success metric, because it answers “Are we going to make more money?”

2. Measuring or Estimating LTV per paying customer is necessary first, and it’s a tameable dark magic that can give us “good enough” numbers

3. If you can dream it, you can achieve it.

I wholeheartedly encourage you leave feedback or questions you have about what I’ve written here, or the spreadsheet itself. 🙏🏻

What’s next?

I’m preparing a series on Cancellation Flows, a growth “hack” that can have a significant impact. I’ll go in depth on how you can retain a lot more customers without burning bridges, or selling your soul!

Subscribe below if you want to receive more actionable guidance on growing your products.

Appendix

[1] Since we’re talking about a new and unproven product, it’s entirely possible that due to poor product-market fit, activation experience or other reasons that this trends to zero after some time. We will still be fairly conservative with our estimations, and as more data is gathered the estimation will be tuned, so there’s no need, nor is it realistic to be perfect with this.

[2] Normally with AB testing we’re evaluating success purely based on the binary 1 or 0 of conversion, but as soon as we say “Plan A conversions are worth 2X more”, it’s taking a step away from the scientific basis of split testing. My opinion on this, is that if you’re using best-practice sample sizes, then we aren’t twisting the science of it too far, and this is still a big improvement on the status quo for how many teams are evaluating their tests.

To get more technical, the correct way to evaluate a non-binomial value (more than just 0 or 1, convert or didn’t) - LTV Revenue in our case - requires a different methodology altogether, requiring you to easily access the data of each user conversion. I’m assuming in most cases that you won’t have easy access to this. The variance, and standard error of the mean for each variant needs to be calculated, and from this the significance can be calculated. You can read more here if you’re interested.

Ever felt unsure or confused in trying to interpret your Paywall AB test results?

Then this article is for you.

I’ll cover:

1. What are the most common sources of confusion

2. Which success metric makes the most sense

3. How to calculate that metric

This should be useful for anyone with a stake in paywalls:

1. Founders

2. Junior-Senior Product/Growth folks

3. Growth-curious or product/growth adjacent roles

Why did I write about this?

A couple of recent events lined up:

• I’ve noticed Founders that I’m mentoring are often unsure which metrics are the best measures of “success” in the context of Paywall AB tests, or have misinterpreted what different metrics truly mean.

• An executive I worked with checked the results on one of our tests and declared “It’s a resounding success, with a 45% uplift!” (Realised LTV was up [1 - See Appendix] , but that’s not a good success metric in most cases)

I suspect these issues are fairly widespread.

The situation: Evaluating Paywall AB tests is confusing and hard

You’ve just stopped your test.

The dashboard has about 25 metrics.

Some of them are “red”, some “grey”, with varying percentage changes and the product you’re using declares some of them to be “statistically significant”, and others not.

Unlike AB testing CTA button (Call to Action) copy on a landing page, answering that key question gets us into deep complexity, due to three main problems.

First problem: Many potential success metrics are thrown at you

What do you pay attention to, when the dashboard lets you filter by 20 metrics? Let’s skip the denial, anger, confusion and go straight to acceptance.

There’s a lot to process here, most of it noise, and we’ll look at what should matter the most to you.

Second problem: Many secondary factors that affect results in less obvious ways

A non-exhaustive list of potential influencing factors beyond the the raw metrics:

1. Your customer retention curves and how they affect the LTVs, and thus the real value per conversion - without weighting our different types of conversions (e.g. monthly vs yearly) by the long-term revenue we gain, conversion data can be highly misleading

   1. Note: This is IF you even have solid data on this yet! We didn’t, but I can show you how to make a well-informed estimate 🦄

2. Your business & product strategy - It could be that monthly customers fit more of a legacy ICP (Ideal Customer Profile) before you pivoted into longer-retaining yearly plan customers, who fit your current ICP. Conversely, maybe customer base growth is actually your target within your strategy, and other metrics are secondary.

3. Refunds - Yeap, it’s a thing! These tend to happen more frequently for trial → yearly subscriptions than monthly, and if you’re not 100% sure they’re automatically factored in to whatever reporting you’re using, it could be a decisive factor that you need to account for. (In my experience trial → yearly conversions can have up to a 10X higher refund rate than monthly conversions 🤯.)

Third problem: Your tools are trying to help, but are only as useful as your understanding of how they work

I feel uneasy declaring outright that they are lying to you. A closer analogy is they’re giving you 1 part signal, 9 parts noise.

The issue stems from AB/Paywall/Billing tools becoming more user-friendly, and doing work on your behalf that would have taken a deeper expertise in the past, they’re taking shortcuts to understanding that is crucial when making such pivotal decisions.

You’re likely dealing with one of the following:

1. A paywall tool that handles your AB testing, and tries to help you a little bit in giving you the numbers and the change, but can’t tell you which of these is the most important for you

2. A paywall tool that handles your AB testing, and goes as far as to give you a list of metrics, magnitude of change & “confidence” or highlights “statistically significant results”.

   Please be very skeptical of these declarations - single metrics are almost always useless alone, and also scanning a long list of 15-20 metrics almost guarantees one will hit “stat-sig” due to the “Multiple Hypothesis Testing” problem

3. Something even less sophisticated e.g. a basic in-house tool, or manual data export where you have to do all the heavy lifting

All 3 of the above examples need varying degrees of “help” to be useful, especially so if you don’t have multiple years of data to go from.

Help I’m here to give you :)

Guidance on how to calculate success

What’s the most important metric?

In most cases, you’re looking to optimise with a laser-focus on New LTV per user

LTV is typically defined as how much revenue we expect to make per new user over a given timeframe - usually 12 to 36 months. (or proceeds i.e. Revenue minus Platform fees e.g. Appstore/Playstore commission).

The “user” in this case, is potential customer who has been exposed to the Paywall test.

Note: Reminder that this is is vastly different from “Realised LTV per user/customer/account” [1].

Why LTV and not Conversion Rate, New MRR or any other metric?

As mentioned earlier, how well your customers retain in each subscription, and thus the amount of revenue you’re able to capture from each one throughout their lifetime as a customer of yours, has massive impact on their relative value to your business.

Who cares about how much money we collect from users in month 0. If you plan on being in business past the next 6-12 months, you should very much care about the long-term picture.

At this point it’s conventional wisdom that yearly customers/revenue are worth more than monthly, but by how much can differ drastically depending on your industry, target customers etc.

In my personal experience, with a yearly plan that’s discounted ~20-40% from the annualised monthly price, the yearly usually has ~2:1 or greater advantage in LTV - in other words, the yearly customer is worth ~2X+ to you.

Whether you need to dive into this complexity depends heavily on your setup:

If you’re in the first two cases (single plan with or without trial), you probably don’t need help. Conversion to paying is your success metric, and then at most you should pay attention to see if those cohorts retain as well.

What if we’re in the 3rd category?

Most SaaS products are, and this is what I want to help you solve.

When multiple plans come into play, we need to get our hands dirty with LTVs, and manual calculations.

The basic formula is as follows:

(Plan A LTV * (Plan A Conversions - Plan A Refunds) + Plan B LTV * (Plan B Conversions - Plan B Refunds))

ALL of the above divided by Users in the variant

Or visualised:

We do this for the control, and each variant we have to calculate which is making us more money per user.

How to determine LTV

If you have a lot of retention data and an easy to use billing system e.g. RevenueCat or similar, you should be able to report on this directly.

If you have a fancy billing system AND plenty of data

Step 1: We need to find “24 month Realised LTV per paying Customer”.

Why 24 months?

1. Anything less is basically useless, since after 12m non-cancelled yearly renewals are counted, but monthly doesn’t have the rest of the second year to catch up.

2. 36m is a long time and very speculative without a deep history of data. For more established products where these numbers are well understood, it makes sense to extend the LTV window, but I’m targeting earlier stage products here.

   1. If you have 36m+ of data, then it’s also fine to increase this timeframe. I’ve worked on products where we had the data, and the plans were fairly stable over time, so 36m was the standard.

Example below is RevenueCat:

1. We want the “Realised LTV per paying Customer” report

2. Filter by Product duration

3. Set Customer Lifetime to 24 months

4. Repeat for all other subscription lengths

Next, you take those LTV numbers, and plug them into the formula with conversions & refunds for each variant in your test.

Example

1. Monthly 24m LTV = $50

2. Yearly 24m LTV = $100

Control:

1. 500 monthly conversions, 25 refunds

2. 300 yearly conversions, 30 refunds

3. 20000 users during the test

Variant:

1. 450 monthly conversions 20 refunds

2. 375 yearly conversions, 40 refunds

3. 20000 users during the test

Control 24m LTV / user = ( $50 * (500-25) + $100 (300 - 30) ) / 20000 = $2.54

Variant 24m LTV / user = ( $50 * (450-20) + $100 (375 - 40) ) / 20000 = $2.75

While conversion to paid is +3% for the Variant here, the actual New 24m LTV performs ~8% better.

If you don’t have a fancy billing system or 24m of data

Don’t worry, I’ve got you covered. 😉 At time of writing, we have ~7 months of data at Linearity, I needed to extrapolate to get usable LTVs.

This LTV calculation/estimation, my personal spreadsheet calculator for AB tests, and a full guided example on how to use it will be covered in Part 2

I will be sharing the spreadsheet I use to determine Paywall AB test success, which does all of these calculations for you, as well as an LTV calculator.

Sneak peak:

Summary & Cheat sheet

1. Determining Paywall AB test success is complex, but with the right support it should be manageable for all who are responsible for driving success there.

2. New LTV per user should be our primary success metric, over and above the dozens of other metrics that we see in our dashboards, except in very simple scenarios.

   1. The formula:

      (Plan A LTV * (Plan A Conversions - Plan A Refunds) + Plan B LTV * (Plan B Conversions - Plan B Refunds))

      ALL of the above divided by Users in the variant

3. It’s necessary to have a fairly accurate LTV per paying customer to calculate this

What’s next?

This is Part 1 of 2 on evaluating Paywall AB tests.

In the next part, I will share my personal spreadsheet calculator, and practical examples on how to use it (+estimating LTV for use in the calculator).

Appendix

[1] Note that Realised LTV is very different to true LTV, and simply means revenue collected, rather the amount of revenue you expect to collect over a defined customer lifetime. By its nature, Realised LTV undervalues monthly subscriptions. If you have multiple subscription lengths, especially up to yearly, Realised LTV is not going to be very helpful.

[2] : Generally, some kind of LTV should be your main goal, but it’s perfectly valid if a business is optimising for certain types of subscriptions/customers, or simply higher growth in paid users at the expense of other metrics.

Glossary

1. LTV = (Customer) Lifetime Value = Average (Mean) revenue collected from each paying customer

2. AB Test = A quantitative experiment, where we split traffic to 2 or more versions of an experience, to measure the difference between them

3. Realised LTV = Revenue collected to date e.g. from test, or cohort state date

4. 24m LTV / user = Expected LTV revenue collected per user over a 24m period, user being defined for example as when “New user who has seen paywall”

5. CTA = Call to action = Shorthand for a button that is calling the user to action, e.g. Subscribe to my newsletter!

Subscribe now