Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
41

Huh! as a signal

from Dan Slimmon [alt+shift+b] in programming

We can never predict with certainty what the next system failure will be. But we can predict, because painful experience has taught us, that some or all of the causes of that failure will be surprising. We can use that!
6th Feb 2023

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Dan Slimmon

Lecture: Queueing theory on a cocktail napkin

Queues are everywhere, and they follow mathematical rules. Learn a few of those rules! It'll go a long way to making you a stronger SRE.

25th Jun 2025 • 50 votes
did u ever read so hard u accidentally wrote?

Owning a production Postgres database is never boring. The other day, I’m looking for trouble (as I am wont to do), and I notice this weird curve in the production database metrics: So we’ve got these spikes in WALWrite: the number of processes waiting to write to the write-ahead log (or “WAL”). The write-ahead log … Continue reading did u ever read so hard u accidentally wrote?

14th Mar 2025 • 58 votes
Is ops a bullshit job?

I recently had the pleasure of reading anthropologist David Graeber’s 2018 book, Bullshit Jobs: A Theory. Graeber defines a bullshit job as, a form of paid employment that is so completely pointless, unnecessary, or pernicious that even the employee cannot justify its existence even though, as part of the conditions of employment, the employee feels … Continue reading Is ops a bullshit job?

11th Feb 2025 • 97 votes
Incident SEV scales are a waste of time

Ask an engineering leader about their incident response protocol and they’ll tell you about their severity scale. “The first thing we do is we assign a severity to the incident,” they’ll say, “so the right people will get notified.” And this is sensible. In order to figure out whom to get involved, decision makers need … Continue reading Incident SEV scales are a waste of time

29th Jan 2025 • 83 votes
Putting a meaningful dent in your error backlog

We often don't realize how noisy the errors have gotten until things are already well out of hand. After all, we've got shit to do. Deadlines to hit. By the time we decide to get serious about error management, a huge, impenetrable, meaningless backlog of errors has already accumulated. I call this stuff "slag."

15th Aug 2024 • 127 votes

More in programming

Notes from CoRL 2026

(Tuesday)

an hour ago • 1 votes
The Codebase Rewrite is Back

The economics of software engineering have fundamentally changed.

4 hours ago • 1 votes
How OCaml effects simplified my JSON interpreter

How effect handlers removed control-flow plumbing from query-json

23 hours ago • 1 votes
Unattended reboots with encrypted disks

An introduction to ZKeys: a new service that stores encryption keys off-site and automatically locks access to them if the client machines go offline for a long period of time.

3 days ago • 1 votes
Education Shouldn't Be Held for Ransom

Google told K-12 schools they could turn off AI, but only if they gave up search too. That isn’t integration. It’s holding education for ransom.

6 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in