Product sense case: how to diagnose a metric drop

“Weekly orders on our app dropped 8% last week. What happened?” Some version of this turns up in a lot of product manager interviews, and it trips people up because it looks open-ended. It isn't. There's a reliable order to work through it, and interviewers mostly want to see whether you follow it calmly or start guessing.

The classic mistake is jumping to a cause. “Maybe a competitor launched a promotion.” Maybe. But you've no evidence yet, and if you're wrong you've spent five minutes on a story. Rule out the boring explanations first, then narrow down where the drop lives and only then explain why.

Pin down the metric before you explain it

Start by making sure you and the interviewer mean the same thing. These questions sound basic but each one can end the case early.

1. How exactly is the metric defined? Orders placed, or orders completed? Unique users, or sessions?
2. Is 8% compared with last week, or the same week last year?
3. Was it sudden (one day) or gradual (over the week)?
4. Is it still falling, flat at the new level or recovering?

A sudden drop on one day points at something that changed on that day: a release, an outage, a tracking change. A gradual slide points at something slower, like a competitor, seasonality or a product change that's hurting retention.

Rule out the data first

Before you touch user behaviour, ask whether the drop is real. A surprising share of metric drops in real companies turn out to be measurement problems: an analytics event renamed in a release, a dashboard filter changed, a logging pipeline delayed.

“Before I look at users, I'd like to check the number is real. Did anything change in how we log orders last week, like a new app version or a change to the dashboard? And do our payment records show the same drop?”

Cross-checking against a separate source (payments, the backend database) is the quickest test. If payments are flat and the dashboard is down, you've found your answer and the interviewer will usually tell you to assume the data is fine and carry on. Either way, you've shown the right instinct.

Internal or external?

Next, split possible causes into things the company did and things that happened to it. Say this split out loud so the interviewer can follow.

1. Internal: a new release, a pricing or fee change, a marketing campaign ending, an experiment rolled out, an outage, a change to search or recommendations.
2. External: seasonality or a holiday, weather, a competitor launch or promotion, an app store change, a news event, a change on a platform you depend on.

Ask the interviewer which of these happened. You're not expected to know. You're expected to ask the right questions in the right order.

Segment until the drop is somewhere specific

This is the heart of the case. An 8% overall drop is rarely 8% everywhere. Usually one segment fell a lot and the rest barely moved. Good cuts to ask for:

1. Platform: iOS, Android, web
2. App version
3. Geography: country, city
4. New versus returning users
5. Acquisition channel
6. Category or type of order

When the interviewer gives you a number, do the quick sum to check it explains the whole drop. This is where a lot of candidates miss easy marks.

Walk the funnel inside that segment

Once you know where, find which step broke. For an ordering app: open the app, search or browse, view a menu, add to basket, check out, pay. Ask for conversion at each step for the affected segment against last week.

If app opens are flat but checkout-to-payment conversion collapsed on the latest Android version, you have a very strong lead: something in the new release broke payments. If opens themselves fell, the problem is upstream: fewer people are arriving at all, which suggests notifications, marketing or the app store rather than the product.

Then, and only then, explain why

Now give a hypothesis that fits all the evidence, say how you'd confirm it and say what you'd do.

“My best guess is the Android release on Tuesday broke something in payment. To confirm, I'd check crash and error logs for the payment screen on that version. If it's confirmed, I'd roll back or push a fix, then check the users who failed to pay and consider a message or credit to win them back.”

Notice the last part. The case is about a product, so finish with what you'd do for users, not just what caused the dip. Interviewers often follow up with “how would you stop this happening again?” Have an answer: staged rollouts, alerts on conversion by version, a release checklist for the payment flow.

What interviewers listen for

1. Do you clarify the metric before diagnosing it?
2. Do you check whether the data is real?
3. Do you segment systematically, or jump around?
4. Do you check that a segment's drop actually adds up to the total?
5. Do you end with an action and a way to prevent it next time?

None of this needs special knowledge. It needs discipline, and discipline under questioning comes from saying it out loud enough times that the order is automatic. Practising with someone who only reveals the Android number when you ask for the platform split is far closer to the real thing than reading a model answer.