Analytics on a very small sample

A job search produces tens of applications, not thousands. At that scale, "your response rate for remote roles is higher" is usually the difference between two responses and one, and it will reverse next month.

Showing it as a finding gives a user a reason to change behaviour based on nothing. The honest version says the sample is too small to tell, which is unsatisfying and correct.

Segmenting makes it worse rather than more insightful. Splitting forty applications by role type, company size and location leaves cells of three or four, and a chart with a visible difference between two of them is a picture of noise.

Descriptive facts are fine: how many applications, to which kinds of roles, how long since each was sent, which are still open. These are counts, not inferences, and they are genuinely useful.

Very large differences can also be reported with appropriate hedging. Twenty applications with no response at all is a signal worth surfacing, even without statistical confidence about the cause.

Set a minimum before any rate is shown at all. Below it, display the counts and say plainly that there is not enough data yet — an empty state that tells the truth is better than a percentage computed from four observations.

• Counts and timelines — always safe
• Where applications are concentrated — descriptive, useful
• Extreme outcomes — worth flagging with hedging
• Small differences between segments — not worth reporting

A user’s own history is too small for trends, but aggregate data across many users can support comparisons: typical response rates for this kind of role, usual time to first response, whether a wait is unusual.

That context answers the question people actually have — "is this normal?" — which their own numbers cannot. Anonymise and aggregate properly, and avoid comparisons that amount to ranking users against each other.

Aggregates about employers are the most valuable thing you can offer, and the most sensitive. A response rate is a statement about a real organisation, so set a minimum sample, show a confidence, and describe behaviour rather than implying intent.

Your analytics only cover applications the system knows about, responses it classified correctly, and outcomes users bothered to record. Every one of those introduces a bias.

Someone who got a job through a referral may never update anything, so your data under-represents the most successful route entirely. State the limitation rather than presenting partial data as complete.

Silence is the largest blind spot and it is structural. Most applications get no reply, so the outcomes you can analyse come only from employers who chose to respond — a non-random population rather than a sample of what happened.

The number worth putting in front of someone is responses per hour spent. It is the one that improves when they change something that matters, and it captures the trade-off automation actually affects.

The number to avoid is applications sent. It is the easiest to compute, it rises whenever someone works harder rather than better, and a dashboard that celebrates it teaches exactly the behaviour that makes outcomes worse.

Resist ranking users against each other in any form. A percentile puts someone having a difficult search at the bottom of a list, which is accurate, useless and the fastest way to make them close the product.

These numbers describe a difficult period in someone’s life. A dashboard leading with a low conversion rate is accurate and makes the product something people avoid.

Frame around what to do: this application is overdue a follow-up, these three postings match better than the ones you have been sending. Keep the raw statistics available for those who want them, and do not put them first.

Do not fill an empty insights panel. The pressure to say something when there is nothing to say is precisely where an honest dashboard starts presenting noise as a finding, and "not enough data yet" is a legitimate thing for a panel to display.