A Metric Dropped. Now What?

Every product person eventually gets the message: a number that mattered went the wrong way. Conversion is down. Repeat rate slipped. Basket size shrank. The instinct — and the mistake — is to jump straight to “let’s break down the funnel and see what’s broken.”

The funnel teardown is real work, but it’s near the end of the process, not the start. Most metric moves are explained by something cheaper and more mundane than a broken funnel stage, and if you don’t rule those out first, you’ll spend a week diagnosing a problem that doesn’t exist.

What follows is the sequence I actually run. Each step earns the right to the next one. You don’t graduate to the expensive analysis until the cheap explanations are dead.

1. Define the metric before you touch it

A metric is not a number, it’s a ratio, and ratios have two moving parts. Before anything else, write down:

The numerator — what event are we counting?
The denominator — over what population?
The time window — daily, weekly, rolling?

This sounds pedantic until you realize how many “drops” are just a ratio moving because the bottom changed, not the top. Conversion rate can fall while the absolute number of conversions rises — you just added more people to the denominator. If you don’t know exactly what’s on top and bottom, you can’t reason about which one moved, and you’ll chase behavior that never changed.

2. Is the drop even real?

Before you theorize about users, confirm the number reflects reality and not a change in how you measure reality.

A “drop” is very often an instrumentation artifact: an event that stopped firing after a deploy, a property that got renamed, a logging change, a double-count that someone helpfully fixed. Ask what shipped near the drop date on the tracking side, not just the product side.

This is a thirty-minute check that saves you from days of analyzing a ghost. Do it early, every time.

3. Read the shape of the change

Before pulling a single additional cut, look at the shape of the drop over time. The shape encodes the cause.

A step function — a sharp vertical drop on a specific date — means something switched. A release, a config change, a price update, a partner turned on, an outage. Your job is to find what happened on that edge. Within “step,” the edge quality tells you more: a drop-then-recover is a transient (an outage that got fixed, a promo that ended); a drop-to-a-floor-that-stays is a permanent change.

A gradual slope means something is drifting. Nothing flipped; a continuous process is grinding the number down. And “gradual” is where people get lazy and mutter “seasonality,” so it’s worth understanding why a slope appears. There are three families:

Mix shift. The composition of your denominator is changing even though no single segment’s rate moves. If City A converts at 60% and City B at 30%, and you’re steadily acquiring more B-type users, the blended rate slides — purely from reweighting. Every segment can be flat while the aggregate falls. This is the Simpson’s paradox trap.
Degradation. Something is accumulating. Tables grow and queries slow, so latency creeps up and conversion bleeds with it. Catalog data goes stale. Search relevance rots. The cause is a slow ramp, so the effect is a slow ramp.
Cohort turnover. The population is being reshaped by entry and attrition. A big acquisition spike ages through your funnel over weeks; retention thins cohorts so survivors over-represent some behavior. No individual changed — the pool did.

The unifying idea: a gradual change means the cause itself is continuous. Something is reweighting, rotting, or aging at a steady pace, and the metric inherits that pace. A step means something switched. Knowing which one you’re looking at tells you whether to hunt for an event or a process — before you’ve run any real analysis.

4. Concentration and mix: is it everywhere, or somewhere?

A metric is an average over a heterogeneous population, and an aggregate move is very often a localized move hiding inside a flat average. So before you tear the funnel apart, slice the drop across the dimensions that carry variance for your business.

The dimensions aren’t a fixed checklist — you pick the ones that plausibly hold the variance for this metric at this company. For a Q-commerce grocery business, that usually means city, dark store or vendor cluster, category (fresh vs. packaged behave nothing alike), time of day and day of week, courier type, and user segment.

Two things fall out of this one check:

Concentration. If the drop is uniform across every slice, you’re looking at a system-wide cause and the funnel teardown is coming. If it collapsed in three dark stores in one city, you have a local ops problem and the funnel analysis would have wasted a day. Where the drop lives determines what kind of problem it is.

Mix vs. rate. This is the subtle one, and it belongs right here — not earlier (you don’t have the segmented view yet), not later (by then you’ve already assumed it’s a rate problem). When you segment, look at two things side by side: each segment’s rate over time, and each segment’s share of volume over time. If the rates are flat but a low-converting segment grew its share, the blend fell without anything actually breaking. The funnel is fine; the population moved.

The cleanest way to settle it: hold the mix constant and recompute. Apply last period’s segment weights to this period’s segment rates. If the reweighted number matches the old number, it was pure rate. If it matches the new number, it was pure mix. Usually it’s in between, and now you’ve quantified how much each contributed instead of hand-waving.

The decision rule at this node:

Concentrated rate drop in a slice → dig there; it’s local and probably causal.
No slice is low, but the blend fell → suspect mix; check whether a low-rate segment grew or a high-rate segment shrank.
Rates uniformly down, shares stable → genuine system-scale degradation. Now the funnel teardown earns its place.

You don’t advance to stage-by-stage analysis until you’ve cleared both concentration and mix, because either one can fully explain an aggregate move without a single funnel stage breaking.

5. The funnel teardown — finally

By the time you get here, you’ve established the drop is real (not instrumentation), system-wide (not concentrated), and a genuine rate move (not mix). That’s what justifies the expensive analysis. You earned it.

A conversion rate is a product of stage-to-stage rates: session → search → add-to-cart → checkout → payment → fulfilled. A drop in the headline lives in one or a few specific transitions. Two disciplines keep this from becoming a fishing trip:

Chase the step that moved, not the step that’s lowest. Every funnel has a worst-converting step; that’s normal, not your culprit. Diff each transition against baseline. A step that’s been 40% forever and is still 40% isn’t your problem even if it’s the lowest in the funnel. The transition that changed is the one that ships you to the cause.

The located stage names the cause family. Where the bleed sits constrains what could cause it:

Drop at add-to-cart → discovery, search relevance, availability/out-of-stock, or pricing perception. They looked and didn’t want it.
Drop at checkout → payment → friction, fees revealed late, delivery cost or SLA shown at a bad moment, address or slot problems. They wanted it but balked at the terms.
Drop at payment → fulfilled → not a user-intent problem at all. Failed payments, cancellations, out-of-stock-after-order, courier failures. They committed and the system failed them.

So the location of the drop already tells you whether you’re looking at a demand problem, a friction problem, or a fulfillment problem — before you’ve formed a single specific hypothesis.

6. Hypotheses that actually discriminate

Once you’ve localized the stage, you rarely have the deciding data live in front of you. The credible move isn’t to guess a conclusion — it’s to name a small, ranked set of hypotheses, each carrying three things:

The mechanism — why this would cause a drop at this specific stage, not a generic “maybe UX got worse.”
The validation — the specific cut or query that confirms or kills it.
The expected signature — what the data looks like if this hypothesis is true versus if a competing one is.

That third item is the upgrade most people skip. Weak validation looks for evidence consistent with your guess — and almost everything is consistent with something. Strong validation names what the data would look like under each competing hypothesis, so the cut actually discriminates. If two hypotheses predict the same pattern, that cut is useless; you need a different one.

Here’s the discipline made concrete for an add-to-cart drop — people are landing and browsing but adding less:

Traffic-mix (check this first). Before concluding “add-to-cart got worse,” confirm the input population didn’t change. If a campaign dumped low-intent traffic into sessions, the rate falls mechanically — the numerator is fine, the denominator inflated. Confirms if: a low-intent channel ballooned as a share of sessions while the rate within each channel stayed flat. Killed if: channel mix stable and the rate fell within established channels — now it’s a real problem.

Availability / out-of-stock. Items people want show as unavailable — the intent is there, the button isn’t. Confirms if: OOS rate rises on the same edge date and the drop concentrates in exactly the stores and categories where OOS rose. Killed if: OOS flat while add-to-cart fell.

Search / discovery relevance. They can’t find what they came for. Confirms if: the drop lives in search sessions specifically, null-result rate climbed, and it tracks a ranking deploy. Killed if: browse sessions dropped just as hard — then it’s broader than search.

The clean discriminator between the last two: availability is “they found it but couldn’t add it”; relevance is “they couldn’t find it.” The OOS rate separates them. If OOS is flat but null-results are up, it’s relevance; reverse it and it’s availability.

Pricing perception. Prices rose (or a competitor’s fell) and the value proposition shifted. Confirms if: the drop concentrates in price-sensitive segments and elastic categories and lines up with a pricing or promo change. Killed if: it’s uniform across price-sensitive and price-insensitive segments — if the users who never shop on price dropped just as hard, price isn’t the lever.

Run them cheapest-and-most-disqualifying first: traffic-mix, then availability, then relevance, then pricing. Each validation is built so that confirming it and killing it look different in the data. That’s what turns a list of guesses into a decisive sequence.

The spine, in one place

Define the metric — numerator, denominator, window.
Data integrity — is the drop real, or an instrumentation artifact?
Shape — step (an event switched) vs. gradual (a process is drifting).
Concentration + mix — localized to a slice, or is the blend moving because weights shifted rather than rates? Hold mix constant and recompute to settle it.
Funnel teardown — only once concentration and mix are ruled out; localize to the stage that moved, which names the cause family.
Discriminating hypotheses — mechanism + validation + expected signature, ranked cheapest-to-disqualify first.

None of this is about knowing more statistics than the next person. It’s about order — refusing to spend expensive analysis until the cheap explanations are dead, and refusing to accept a hypothesis until you’ve named the data that would prove it wrong. The number moved for a reason. The method is just making sure you find the real one.