(This originally appeared as an article on LinkedIn. I periodically mirror such articles here as blog posts.)
Following my posts on metrics and companies drifting into autopilot, today I have a story that involves three of my favorite topics:
- data
- risk
- business models
And it all starts with some plain old tech issues:
The setup
On a whim, I recently tried to upgrade a subscription service plan from Basic level to Premium. I went through the workflow, only to get stopped at the end.
“You’re already subscribed.”
Technically, this was true: I was already subscribed.
Then again, isn’t that the point of tiered subscriptions? That you could, y’know, move between tiers? Especially moving up, so I can contribute more to the vaunted Recurring Revenue Stream. The “Forever Transaction,” as some might call it. The thing that really drives that other vaunted metric, Customer Lifetime Value.
What went wrong?
The data angle?
One important metric online merchants track is cart abandonment. That’s when a person starts to buy something and then … leaves. Maybe they were just on your site to compare prices. Maybe they got distracted and pulled away before they could finalize the purchase. When you get those reminder e-mails letting you know that you didn’t complete your transaction (Zappos is good about this), that’s a merchant trying to trim that cart abandonment metric.
A data-minded friend suggested that the service’s cart abandonment metric might (eventually) alert them to the bug, and they’d fix it. Which sounded great at first, until we realized that they probably wouldn’t catch my you-can’t-upgrade situation. Their system had ended the interaction, not me. So it would not have appeared as an abnormal termination.
I had uncovered a dead end in their workflow. The error I encountered probably wasn’t tracked as such, so it left the false impression that everything was OK when it was not.
The lesson: good metric + incomplete data = bad metric
The risk angle?
Operational risk expert Ariane Chapelle notes that key performance indicators (KPIs) and key risk indicators (KRIs) go hand-in-hand. As a KPI drifts out of acceptable range, that usually coincides with an increase in risk. So when your KPIs are fooling you, your KRIs can’t point out the growing problem.
Risk handling starts with performing a risk assessment. In turn, that starts with working through a lot of “what if?” questions. The goal is to surface cases where things may not go according to plan, so you can figure out how to address them. Maybe you make changes so a given problem doesn’t occur. Or you devise a procedure for handling the problem when it crops up, because at least now you’re aware that it can crop up.
And it seems this company hadn’t asked “what if … a customer wants to change plans?” The site could have told me that I was ineligible for an upgrade (for whatever reason), or pointed me to the upgrade workflow (if I had somehow gone down the wrong path), or any number of things. Instead, it dead-ended me.
(I was about to say that a simulation may have uncovered this gap, but given the limited number of possible states and transitions, this could have been worked out on a single sheet of paper.)
How many other customers had hit this same scenario? I don’t know. And the company doesn’t know, either. Not unless they are tracking “subscribers who try to subscribe again.” Which is unlikely.
The lesson: bad metrics lull you into a false sense of security.
Corollary: poorly-maintained metrics are bad metrics.
The business angle?
For this company, Premium costs quite a bit more than Basic. That would have meant more money out of my pocket, hence a significant increase in revenue for the service. Recurring revenue. Remember that trusty Customer Lifetime Value metric?
Recall, I’d considered the upgrade on a whim. The extra friction from the dead-end workflows gave me the time and mental space to ask: “do I really want this?” And my answer was now a firm “no.”
As such, the company didn’t get my (additional) money from me.
The lesson: Be mindful of unexpected friction in your workflows. Especially the workflows that involve someone paying you.
Bottom line
By now, many of you may be wondering: “Which service was this? What site prevented you from making a paid upgrade from Basic to Premium?”
And the answer is … I won’t say.
I didn’t write this piece as a name-and-shame tactic. I did it in the hopes that readers would take a step back to review their company’s metrics and the data pipelines that feed them.
Are you capturing all of the data required for accurate measurement? Or do you have leaks on the way to your dashboards and alert systems? Have you explored a variety of “what if?” questions to uncover scenarios you hadn’t considered before?
It might be worth a second look.
(Oh, and if your company runs a tiered subscription service: you really, really want to check that upgrade workflow. Thanks.)