Skip to content

Lean Analytics – Book Notes

Lean Analytics: Use Data to Build a Better Startup Faster (Lean Series)
Alistair Croll and Benjamin Yoskovitz

Don’t sell what you can make; make what you can sell.

Lean isn’t about being cheap or small, it’s about eliminating waste and moving quickly, which is good for organizations of any size.

The faster your organization iterates through the cycle, the more quickly you’ll find the right product and market. If you measure better, you’re more likely to succeed.

As an entrepreneur, you need to live in a semi-delusional state just to survive the inevitable rollercoaster ride of running your startup.

data-driven learning is the cornerstone of success in startups. It’s how you learn what’s working and iterate toward the right product and market before the money runs out.

Management guru and author Peter Drucker famously observed, “If you can’t measure it, you can’t manage it.”[

Sometimes, growth comes from an aspect of your business you don’t expect. When you think you’ve found a worthwhile idea, decide how to test it quickly, with minimal investment. Define what success looks like beforehand, and know what you’re going to do if your hunch is right.

Analytics is about tracking the metrics that are critical to your business.

In a startup, you don’t always know which metrics are key, because you’re not entirely sure what business you’re in. You’re frequently changing the activity you analyze. You’re still trying to find the right product, or the right target audience. In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out.

A good metric changes the way you behave. This is by far the most important criterion for a metric:

Qualitative versus quantitative metrics

Vanity versus actionable metrics

Exploratory versus reporting metrics

Leading versus lagging metrics

Correlated versus causal metrics

Analysts look at specific metrics that drive the business, called key performance indicators (KPIs).

Quantitative data abhors emotion; qualitative data marinates in it.

Many companies claim they’re data-driven. Unfortunately, while they embrace the data part of that mantra, few focus on the second word: driven. If you have a piece of data on which you cannot act, it’s a vanity metric.

Whenever you look at a metric, ask yourself, “What will I do differently based on this information?” If you can’t answer that question, you probably shouldn’t worry about the metric too much.

A leading metric (sometimes called a leading indicator) tries to predict the future. For example, the current number of prospects in your sales funnel gives you a sense of how many new customers you’ll acquire in the future.

On the other hand, a lagging metric, such as churn (which is the number of customers who leave in a given time period) gives you an indication that there’s a problem — but by the time you’re able to collect the data and identify the problem, it’s too late.

As a leading indicator, customer complaints also give you ammunition to dig into what’s going on, figure out why customers are complaining more, and address those issues.

Now consider account cancellation or product returns. Both are important metrics — but they measure after the fact. They pinpoint problems, but only after it’s too late to avert the loss of a customer.

Ultimately, you need to decide whether the thing you’re tracking helps you make better decisions sooner.

looking at a simple correlation without demanding causality leads to some bad decisions.

Finding a correlation between two metrics is a good thing. Correlations can help you predict what will happen. But finding the cause of something means you can change it.

You prove causality by finding a correlation, then running an experiment in which you control the other variables and measure the difference.

correlation is good. Causality is great. Sometimes, you may have to settle for the former — but you should always be trying to discover the latter.

The response from parents was a surprise. Many of them were using HighScore House only once or twice a week, but they were getting value out of the product. From this, Kyle learned about segmentation and which types of families were more or less interested in what the company was offering. He began to understand that the initial baseline of usage the team had set wasn’t consistent with how engaged customers were using the product. That doesn’t mean the team shouldn’t have taken a guess. Without that initial line in the sand, they would have had no benchmark for learning, and Kyle might not have picked up the phone. But now he really understood his customers. The combination of quantitative and qualitative data was key. As a result of this learning, the team redefined the “active user” threshold to more accurately reflect existing users’ behavior. It was okay for them to adjust a key metric because they truly understood why they were doing it and could justify the change.

A segment is simply a group that shares some common characteristic.

A second kind of analysis, which compares similar groups over time, is cohort analysis. As you build and test your product, you’ll iterate constantly. Users who join you in the first week will have a different experience from those who join later on.

This kind of reporting allows you to see patterns clearly against the lifecycle of a customer, rather than slicing across all customers blindly without accounting for the natural cycle a customer undergoes. Cohort analysis can be done for revenue, churn, viral word of mouth, support costs, or any other metric you care about.

Cohort experiments that compare groups like the one in Table 2-2 are called longitudinal studies, since the data is collected along the natural lifespan of a customer group. By contrast, studies in which different groups of test subjects are given different experiences at the same time are called cross-sectional studies.

Much of Lean Analytics is about finding a meaningful metric, then running experiments to improve it until that metric is good enough for you to move to the next problem or the next stage of your business, as shown in Figure 2-3. Eventually, you’ll find a business model that is sustainable, repeatable, and growing, and learn how to scale it.

Take a look at the top three to five metrics that you track religiously and review daily. Write them down. Now answer these questions about them: How many of those metrics are good metrics? How many do you use to make business decisions, and how many are just vanity metrics? Can you eliminate any that aren’t adding value? Are there others that you’re now thinking about that may be more meaningful? Cross off the bad ones and add new ones to the bottom of your list, and let’s keep going through the book.

If you’re going to survive as a founder, you have to find the intersection of demand (for your product), ability (for you to make it), and desire (for you to care about it).

First, ask yourself: can I do this thing I’m hoping to do, well? This is about your ability to satisfy your market’s need better than your competitors, and it’s a combination of design skill, coding, branding, and myriad other factors. If you identify a real need, you won’t be the only one satisfying it, and you’ll need all the talent you can muster in order to succeed. Do you have a network of friends and contacts who can give you an unfair advantage that improves your odds? Do you have the talent to do the things that matter really well? Never start a company on a level playing field — that’s where everyone else is standing.

Would you work on it even if you weren’t being paid? Is it a problem worth solving, that you’ll brag about to others? Is it something that will take your career in the direction you want, and give you the right reputation within your existing organization? If not, maybe you should keep looking.

Humans do inspiration; machines do validation.

Math is good at optimizing a known system; humans are good at finding a new one. Put another way, change favors local maxima; innovation favors global disruption.

Check your data at the door to be sure it’s valid and useful.

Without a big vision, you’ll lack purpose, and over time you’ll find yourself wandering aimlessly.

Getting paid is, in some ways, the ultimate metric for identifying a sustainable business model. If you make more money from customers than it costs you to acquire them — and you do so consistently — you’re sustainable. You don’t need money from external investors, and you’re growing shareholder equity every day.

The two knobs on this machine are customer lifetime value (CLV) and customer acquisition cost (CAC). Making more money from customers than you spend acquiring them is good, but the equation for success isn’t that simple. You still need to worry about cash flow and growth rate, which are driven by how long it takes a customer to pay off. One way to measure this is time to customer breakeven — that is, how much time it will take to recoup the acquisition cost of a customer.

Having reviewed these frameworks, we needed a model that identified the distinct stages a startup usually goes through, and what the “gating” metrics should be that indicate it’s time to move to the next stage. The five stages we identified are Empathy, Stickiness, Virality, Revenue, and Scale. We believe most startups go through these stages, and in order to move from one to the next they need to achieve certain goals with respect to the metrics they’re tracking.

one of the keys to startup success is achieving real focus and having the discipline to maintain

As noted in Chapter 5, Eric Ries talks about three engines that drive company growth: the sticky engine, the viral engine, and the paid engine. But he cautions that while all successful companies will ultimately use all three engines, it’s better to focus on one engine at a time. For example, you might make your product sticky for its core users, then use that to grow virally, and then use the user base to grow revenue. That’s focus.

Capture everything, but focus on what’s important.

Whatever your current OMTM, expect it to change. And expect that change to reveal the next piece of data you need to build a better business faster.

One thing we’ve noticed about almost all successful founders we’ve met is their ability to work at both a very detailed, and a very abstracted, level within their business. They can worry about the layout of a page or the wording of an email subject one day, and consider the impact of one-time versus monthly recurring sales the next. That’s partly because they’re not only trying to run a business, they’re also trying to discover the best business model.

Sergio Zyman, Coca-Cola’s CMO, said marketing is about selling more stuff to more people more often for more money more efficiently.[

Not all customers are good. Don’t fall victim to customer counting. Instead, optimize for good customers and segment your activities based on the kinds of customer those activities attract.

Even before a year has elapsed, an e-commerce company can look at 90-day repurchase rates and get a sense of which model it’s in. A 90-day repurchase rate of 1% to 15% means you’re in acquisition mode. A 90-day repurchase rate of 15% to 30% means you’re in hybrid mode. A 90-day repurchase rate of over 30% means you’re in loyalty mode.

Accounting for the cost of acquisition in aggregate is fairly easy; it’s more complicated when you have myriad channels driving traffic to your site. The good news is that analytics tools were literally built to do this for you. The reason Google has a free analytics product is because the company makes money from relevant advertising, and wants to make it as easy as possible for you to buy ads and measure their effectiveness.

Shipping time is key, and it’s tightly linked to how effectively the retailer handles logistics. E-commerce companies can most likely achieve significant operational efficiencies just by optimizing their fulfillment and shipping processes. These efficiencies turn into a competitive advantage, because they let you sell to consumers who are more interested in faster, better-quality service than the cheapest price.

It’s vital to know if you’re focused on loyalty or acquisition. This drives your whole marketing strategy and many of the features you build.

While conversion rates, repeat purchases, and transaction sizes are important, the ultimate metric is the product of the three of them: revenue per customer.

The company cares about the following key metrics: Attention How effectively the business attracts visitors. Enrollment How many visitors become free or trial users, if you’re relying on one of these models to market the service. Stickiness How much the customers use the product. Conversion How many of the users become paying customers, and how many of those switch to a higher-paying tier. Revenue per customer How much money a customer brings in within a given time period. Customer acquisition cost How much it costs to get a paying user. Virality How likely customers are to invite others and spread the word, and how long it takes them to do so. Upselling What makes customers increase their spending, and how often that happens. Uptime and reliability How many complaints, problem escalations, or outages the company has. Churn How many users and customers leave in a given time period. Lifetime value How much customers are worth from cradle to grave.

“In early 2010 we were paying $243 to acquire a customer, who only paid us $39 per year,” explained Robert. “Those are horrible economics. Most consumer apps get around the high acquisition costs with some sort of virality, but backup isn’t viral. So we had to pivot [from consumer sales] to go after businesses.”

CLV and CAC are the two essential metrics for a subscription business.

“MRR growth will probably be our top metric until we hit $10M in annual recurring revenue,” said Robert. “I watch churn, but I’m more focused on customer acquisition payback in months, which is how quickly I make my money back on each customer.” Robert’s target for that metric is 12 months or less for any given channel. Customer acquisition payback is a great example of a single number that encompasses many things, since it rolls up marketing efficiency, customer revenue, cash flow, and churn rate.

Before focusing on sophisticated financial metrics, start with revenue. But don’t ignore costs, because profitability is the real key to growth.

your paid engine is humming along nicely, which happens when the CAC is a small fraction of the CLV — a sure sign you’re getting a good return on your investment.

There’s a natural progression of metrics that matter for a business that change over time as the business evolves. The metrics start by tracking questions like “Does anyone care about this at all?” and then get more sophisticated, asking questions like “Can this business actually scale?” As you start to look at more sophisticated metrics, you may realize your business model is fundamentally flawed and unsustainable. Don’t just start from scratch: sometimes what you need is a new market, not a new product, and that market may be closer than you think.

The ultimate metric for engagement is daily use. How many of your customers use your product on a daily basis? If your product isn’t a daily use app, establishing a minimum baseline of engagement takes longer, and the time it takes to iterate through a cycle of learning is longer. It’s also hard to demonstrate enough value, quickly enough, to keep people from churning. Habits are hard to form — and with any new product, you’re creating new habits, which you want to do as quickly and intensely as possible.

That’s an important lesson around business models and Lean Startup — you bring an early version of your product to the market, test its usage, and look for where it’s got the highest engagement among your customers. If there’s a subsection of users who are hooked on your product — your early adopters — figure out what’s common to them, refocus on their needs, and grow from there. Claim your beachhead. It will allow you to iterate much more quickly on a highly engaged segment of the market.

A data-driven approach to measuring engagement should show you not only how sticky your product or service is, but also who stuck and whether your efforts are paying off.