Why Averages Lie and P95 Actually Matters

 

Why Averages Lie and P95 Actually Matters




For a long time, I looked at average latency and felt confident.

Dashboards were green.
Numbers looked healthy.
Nothing seemed broken.

And yet, users were complaining.

Pages felt slow.
Flows felt unreliable.
Support tickets kept coming.

That disconnect taught me an important lesson:

Users do not experience averages.

They experience the slow moments. The frustrating pauses. The one request that took far too long when it mattered.

That is why P95 and P99 matter more than the average.

The Problem With Averages

Average latency tells you how the system behaves most of the time.

But real user experience is shaped by the worst moments, not the common ones.

If 95 out of 100 requests are fast, but 5 are painfully slow, users will remember the slow ones. Those slow requests often happen during peak load, degraded dependencies, or edge cases. Exactly when users need the system most.

From a user’s perspective:

  • A single slow login feels broken

  • A delayed checkout feels risky

  • A hanging form submission feels unreliable

Averages hide these experiences.

What P50, P95, and P99 Really Mean

Here’s a simple way to think about it:

P50
This is the median. Half your requests are faster, half are slower.
This is what demos usually show.

P95
This represents the slowest experience most users will encounter.
This is where frustration starts.

P99
This is where incidents live.
These are the requests that cause retries, complaints, and escalations.

If P95 feels slow, users notice.
If P99 is unstable, teams get paged.

Why Teams Miss This

Most teams do not ignore P95 on purpose.

They miss it because:

  • Averages are easy to explain

  • P95 requires better instrumentation

  • Dashboards default to mean values

  • Performance work often starts too late

It is comfortable to say, “Average latency is fine.”
It is harder to say, “Five percent of our users are having a bad day.”

But that honesty is what leads to better systems.

A Practical Way to Improve What Users Feel

You do not need to optimize everything at once.

Start small and focused.

  1. Pick one critical user journey
    Login, search, checkout, registration. Something users care about.

  2. Measure P95 end to end
    Not just backend. Include client time, network, and dependencies.

  3. Break it into segments
    Frontend, API, database, external services. Find the real bottleneck.

  4. Fix the biggest contributor first
    Do not spread effort thin. Focus where it hurts most.

  5. Re-check P95 after every change
    Performance work is iterative, not one-time.

This approach keeps performance work grounded in real user experience instead of abstract metrics.

The Mindset Shift That Matters

Optimizing averages makes dashboards look good.

Optimizing P95 makes products feel good.

That difference shows up in:

  • Fewer support tickets

  • Higher trust

  • Calmer on-call rotations

  • More predictable releases

Performance is not just a technical concern. It is a product quality signal.

Closing Thought

If you only track averages, you are optimizing for the best-case user.

If you track P95 and P99, you are designing for reality.

And reality is where good engineering actually matters.

Question for you:
Does your team actively track P95 and P99 today, or do averages still drive performance decisions?

___________________________________________________________________________

Written by Sharath Chandra Odepalli

LinkedIn 

Comments

Post a Comment

Popular posts from this blog

Stop Building: Why Less Is More in Software Development

  Stop Building: Why Less Is More in Software Development The biggest waste in software isn't bugs, technical debt, or slow deployments. It's building features nobody uses. Research reveals a sobering truth: 45% of features are never used, 19% are rarely used, and only 36% are used regularly or often. Translation: nearly half of engineering effort creates zero value. Think about that for a moment. Half of all the sprints, all-nighters, code reviews, bug fixes, and deployment stress—wasted on features that users will never click. This isn't a small problem. It's the silent killer of engineering productivity and product success. The Three Forces Driving Waste Force 1: Engineers Love Solving Hard Problems We're wired to seek complexity. When faced with a problem, we instinctively reach for the sophisticated solution. A simple notification system? Why not build a real-time WebSocket infrastructure with Redis pub/sub and horizontal scaling? A basic reporting feature? Obv...
Read more

Clarity: The Most Underrated Skill in Technology

Image
There’s a quiet truth in the engineering world that people only discover after years of writing code, leading teams, and shipping systems: Your value in tech is measured far less by what you build and far more by how clearly you think. This isn’t intuitive. In a field obsessed with new frameworks, new architectures, and now new AI models—clarity feels almost… old-fashioned. But here’s the irony: The more complex the industry becomes, the more clarity becomes a competitive advantage. Complexity Isn’t Hard. Simplicity Is. If you’ve spent long enough in engineering, you’ve seen this pattern: A senior engineer writes code that works—but nobody else wants to maintain it. A team ships a feature—but half the customers don’t understand how to use it. A brilliant system is designed—but onboarding takes three meetings and an internal wiki. We call these “engineering problems,” but they’re actually  clarity problems . The hardest parts of building software aren’t the algorithms, the databases...
Read more

The Transparency Era: What California's New AI Law Means for Tech Builders

Image
  Yesterday marked a pivotal moment in technology regulation - one that signals we've crossed a threshold from which there's no turning back. Governor Gavin Newsom signed  SB 53 , California's first major AI transparency law, requiring the world's largest artificial intelligence companies to publicly disclose their safety protocols and report critical incidents within 15 days. This isn't just another piece of legislation lost in the noise of regulatory bureaucracy. It's a watershed moment that redefines the social contract between technology companies and the public they serve. From Voluntary to Mandatory: The Shift Nobody Saw Coming For years, AI safety has been a gentleman's agreement. Companies like OpenAI, Anthropic, Google, and Meta have made voluntary commitments to safety testing and responsible development. They've published research papers, formed ethics boards, and publicly discussed the importance of alignment and safety. But voluntary commitm...
Read more