In this #AmazonMechanism post, human error is inevitable. Amazon uses a mechanism called correction of error (CO) which identifies the issue, measures the customer impact, asks questions (the 5 why’s) about how it happened, and generates action items to ensure the error never happens again. (Dave Anderson, Scarlet Ink)
An instrument for decomposing complex problems into simplified solutions, is the strategy of first principles thinking, often otherwise known as reasoning from first principles, and is often leveraged by self-made geniuses such as Elon Musk. In other words, a form of mental reverse engineering.
Techtello best puts it as realigning your mindset to demarcate from conventional wisdom, questioning and validating one’s beliefs. Humans are inherently governed by values and perceptions, belief systems that we learn to reasons with, influencing our minds to apply shortcuts in the form of conclusions learned previously.
Something a product manager should have ingrained in her or his mind, this hypothesis-driven thought process advocates breaking down a complex problem into its fundamental building blocks, down to its pure essence, diving down to the basic truths, and separating facts from assumptions. You then reconstruct your view from the ground up with those validated truths.
In Amazon, we employ as part of Correction of Error (COE) investigations, a mechanism employed to discover the root cause, the five why’s, a tool that helps us discover the essence of a problem, or in this case, the fundamental pillars of a component.
Children inherently think in first principle because they inquisitively question everything, from why do you go to work, why do you need to eat, why do you need to sleep. A fundamental leadership principle of Amazon, the learn and be curious principle, encourages questioning perception and opening yourself to an alternative reality. Breaking the autopilot trap your mind inclines to follow.
In his quest to getting a rocket to Mars, Elon Musk concluded upon first investigation that the cost of buying a rocket is extremely cost prohibitive, over $65m. With that fundamental problem as a barrier of entrant into the space race, Elon embraced physics to employ first principles reasoning:
Breaking down his problem into the fundamental components a rocket, instead of buying a finished rocket, he created his own rocket from raw materials, and that is what resulted in the founding of SpaceX. The company successfully cut the price of launching a rocket by 10x, decomposing a problem into fundamental components, and rebuilding.
It’s Aways Day 1 at Amazon
Another fabric of Amazon’s DNA, injected all the way from Jeff B, is the mantra of Day I. It’s a simple yet profound war-cry, no matter whether Amazon is 25 years or 5 years, whether your organization within Amazon is 2 years or 12 years, everyone acts as if we are in the first day of a brand new startup. You know that feeling you get, that excitement, motivation, and drive to go from zero to 100 in 7 seconds, that is the mentality and culture that is demanded of you.
Why is it important to always be at day-one mentality? It instills a sense of passion combined with energy, with a bearing on customer needs, not stuck in processes and other bureaucracies befitting mature organizations, often referred to as organizations in Day 2.
The outside world can push you into Day 2 if you won’t or can’t embrace powerful trends quickly. If you fight them, you’re probably fighting the future. Embrace them and you have a tailwind.Jeff Bezos
Reading between the lines, new startups with fresh mentalities are nimble enough to sense new trends and pivot as needed, and not be stuck in the past. Ensure we don’t stay stuck in the past, and putting customers first.
Organizations often claim to be data-driven, relying on the utilization of data to base their product and engineering decisions. But not all data is equal. We have what’s called vanity metrics, misleading and misrepresentative, that are often mascaraed as forcing factors for making decisions.
Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies. These metrics are exciting to point to if you want to appear to be improving, but they often aren’t actionable and aren’t related to anything you can control or repeat in a meaningful way. Vanity metrics are most often contrasted against actionable metrics, which is data that helps you make decisions and helps your business reach its goals or grow.Tableau
In Amazon’s sphere, we often refer to vanity metrics as output metrics. A common problem, teams would tend to focus on output metrics and not input metrics. Take the following analogy, as explained in the book, Working Backwards, by Colin Bryar and Bill Carr:
Before you can improve any system . . . you must understand how the inputs affect the outputs of the system. You must be able to change the inputs (and possibly the system) in order to achieve the desired results. This will require a sustained effort, constancy of purpose, and an environment where continual improvement is the operating philosophy.
Working Backwards (2021, Bryar, Carr)
Input metrics are controllable metrics, known in the industry as leading indicators, whereas output metrics are known as lagging indicators.
“The right input metrics get the entire organization focused on the things that matter most. Finding exactly the right one is an iterative process that needs to happen with every input metric.”
Once teams define and refine their input metrics, it is presented in an Amazon cadence, called a Weekly Business Review.
Amazon’s Metrics Lifecycle
Through a continuous improvement cycle, Amazon leverages DMAIC, short for define, measure, Analyze, Improve and Control. This mechanism is leveraged from the industry-famous Six Sigma framework.
1. Identify Input Metrics
What is the metrics that is controllable that subsequently frames and influences the output metrics? Bryar and Carr recall the following anecdote:
One mistake we made at Amazon as we started expanding from books into other categories was choosing input metrics focused around selection, that is, how many items Amazon offered for sale. Each item is described on a “detail page” that includes a description of the item, images, customer reviews, availability (e.g., ships in 24 hours), price, and the “buy” box or button. One of the metrics we initially chose for selection was the number of new detail pages created, on the assumption that more pages meant better selection.Working Backwards (2021, Bryar, Carr) as citied in holistics.io
Once we identified this metric, it had an immediate effect on the actions of the retail teams. They became excessively focused on adding new detail pages—each team added tens, hundreds, even thousands of items to their categories that had not previously been available on Amazon.
(…) We soon saw that an increase in the number of detail pages, while seeming to improve selection, did not produce a rise in sales, the output metric. Analysis showed that the teams, while chasing an increase in the number of items, had sometimes purchased products that were not in high demand.
When we realized that the teams had chosen the wrong input metric—which was revealed via the WBR process—we changed the metric to reflect consumer demand instead. Over multiple WBR meetings, we asked ourselves, “If we work to change this selection metric, as currently defined, will it result in the desired output?” As we gathered more data and observed the business, this particular selection metric evolved over time from
– number of detail pages, which we refined to
– number of detail page views (you don’t get credit for a new detail page if customers don’t view it), which then became
– the percentage of detail page views where the products were in stock (you don’t get credit if you add items but can’t keep them in stock), which was ultimately finalized as
– the percentage of detail page views where the products were in stock and immediately ready for two-day shipping, which ended up being called ‘Fast Track In Stock’.
Instrumentation comes next to help validate your input metrics. From your hypothesis to measurement, you ensure your tooling removes bias in measurement, and set forth a mechanism for how to audit your metrics.
As your measurement matures in the lifecycle, “you develop a comprehensive understanding of the underlying drivers behind the metrics” (2021, Bryar, Carr). The authors also call this, reducing the variance, to ensure the process is predictable and controllable.
Charlie Bell, an SVP in AWS, has a saying: “when you encounter a problem, the probability you’re actually looking at the actual root cause of the problem in the initial 24 hours is pretty close to zero, because it turns out that behind every issue there’s a very interesting story.”Working Backwards (2021, Bryar, Carr) as citied in holistics.io
Here comes the iteration part, where you look at the output metrics and use that as a guide to improve your product feature. You make changes that will lead to improvements in the output metrics. One thing the authors have noted is that as you improve your features, your input metrics end up becoming less useful, in which case it is okay to deprecate them, in favor of more useful metrics.
The final phase is control, a measure to ensure your mechanism is optimally performing and not regressing. This may eventually lead to a complete automation and other improvements.
The Amazon Deck
Within the Weekly Business Reviews (WBRs) at Amazon, a deck consists of the most important metrics in an organization. holistics.io highlight a few notable properties of a deck:
- The deck represents an end-to-end view of the business.This is deliberate — the authors write that “while departments shown on org charts are simple and separate, business activities usually are not. The deck presents a consistent, end-to-end review of the business each week that is designed to follow the customer experience with Amazon. This flow from topic to topic can reveal the interconnectedness of seemingly independent activities.”
- The deck is primarily charts, graphs and data tables. Since there are hundreds of visualizations to review, written notes will bog the meeting down too much. Two notable exceptions to this rule are ‘exception reporting’, as well as the ‘voice of the customer’ anecdotes that customer service is allowed to insert into the metrics deck.
- There is no ideal number of metrics to review. Amazon itself constantly adds, modifies and removes metrics from the WBR deck as business needs evolve.
- Emerging patterns are a key focus. You want trend lines, and you want to know them long before they show up in a quarterly or yearly result.
- Graphs are usually plotted against a comparable prior period. Metrics make sense when compared against prior periods, so that you have a proper apples-to-apples comparison (for instance, you’ll want to compare holiday periods to a prior holiday period, not to a slow period).
- Graphs show two or more timelines, for example, trailing 6-week and trailing 12-months. Small but important issues tend to only show up in shorter trend lines; they tend to be smoothed out in longer ones.
- Anecdotes and exception reporting are woven into the deck. The only exception to the ‘charts, graphs and data tables’ rule are anecdotes and exception reporting. About which, more later.
It doesn’t happen often, but this week marks the acceptance of two new leadership principles into Amazon’s cultural tenets, Strive to be the Earth’s Best Employer and Success and Scale Bring Broad Responsibility.
Amazon’s Leadership Principles, or LPs help its employees hold themselves and each other accountable, through tangible and measurable qualities that guide and lead decision-making, with customers at the forefront.
Let’s dive deeper into the two new principles.
Strive to be Earth’s Best Employer
This is quite a unique LP, unlike others which are more operational-driven, to achieve business results, this LP is about fostering and contributing to a better place to work at. This is often forgotten in environments where metrics and business success take precedence over mental and emotional wellbeing.
This principal ensures you build on your emotional intelligence, becoming an empathy-driven leader where being successful means others around you are also successful.
Success and Scale Bring Broad Responsibility
Beyond our current and immediate environment, the 16th LP advocates for us do be empathetic beyond our immediate team and colleagues, but with our customers, and wider community and planet. This is an altruistic LP to be a good citizen and do good for the world at large, leaving a legacy and footprint.
These two LPs are a great additional beacons to making yourself a better person for others, within your immediate radius and globally.