The Treachery of Postmortems
Postmortems, retrospectives, incident reviews—these are all different names for roughly the same thing. We're surprised by the improbable (an incident), and are looking to gain some insight from that unexpected turn of events. The problem is, as an industry we more often than not mistake capturing and archiving information for developing meaningful insights.
Take the famous painting "The Treachery of Images" created almost a century ago. It is unmistakably a pipe, with a long stem and bright sheen over well polished wood. But underneath in French, it reads "Ceci n'est pas une pipe" ("This is not a pipe"). Can you stuff the chamber on it and smoke it, smell the cooled embers, or even pick it up? No, because it's only the representation of the object, not the object itself.
Much like that pipe, the documents we share and call postmortems are only representative of an attempt at shared learning, the mechanical motions meant to mimic what's needed for learning. To our own detriment, we've conflated generating a postmortem with learning, and in doing so substitute the transcription of events with the meaningful work in understanding what has happened. Even worse, AI summarizations of events are done without human intervention and treated as the insight itself.
Documentation as Performative Insights
Tech loves a solid incident retro document. People want information, answers to the questions surrounding the seemingly impossible: How did this happen? Our post incident docs tend towards generating “action items” in the hopes of ensuring it never happens again or just necessary PR in a public blog post to shore up the worries of a customer base or investors. These are common and understandable, but they fall short of actual learning.
The majority of those docs are fairly formulaic as well, a reason we see frequent use of cookie cutter templates for them. State the facts (that are worth stating), find a few choice parts, and sprinkle in a few comments, maybe drop in some dashboard screen caps for good measure. And there you have it! Your document is ready, you've captured the learning, and once again the world is made right. Right?
Except your retrospective, postmortem, or however you want to name it is merely data capture, which is not the same thing as a learning review.
What's Wrong with Postmortem Docs?
We often confuse the creation of a shareable document as the goal. So what's missing?
A frequent failure mode, even for the best of us, is that we tend towards quick answers when disaster strikes. “Tell me the one thing we could do to make sure this never happens again, so we can address it and completely close that gap.” Unfortunately, that's not how incidents work. The sole output of an incident being a document (that likely contains recommended action items) provides a myopic view of the events.
There's a bias to quickly skim and move on when it's only a doc, or perhaps ignore it altogether. "I'll read it later," you say, filing it in your inbox or browser bookmarks. Weeks go by, you rediscover it, except now it doesn't hold the same weight it might have earlier. The new normal takes hold, informing what is critical and what can be ignored, information that seemed pressing at the time feeling trivial or inconsequential now. A whole host of new problems have arisen, we can't worry about something we've already "solved!"
That doesn't mean writing a doc is inherently wrong. We should write them, focusing on capturing discussions as they happen rather than curating a predisposed view. Allowing postmortems to be editable as well means we're continuously incorporating ideas. We should use them as a means of bringing folks in, adding to them, and allowing differing or disagreeing points of view to be surfaced.
If we're only writing an artifact and stopping, we're failing to make good on better understanding our system. Instead, we can use timelines, interviews, questions, graphs and figures, early suggestions, and everything else that helps develop the doc as a jumping off point.
A Cure for the Common Review
What makes a post incident review discussion so useful? It's the chance to question assumptions and recalibrate mental models, both individual and collective. If we're taking the opinion of a few people as sacrosanct, we're missing out on what others see, in and out of the incident. So the solution for the everyday is walking the untread path.
Writing a postmortem can be a lonely exercise, but there's a remedy: make it a communal activity with individual interviews. If you can spare fifteen minutes or more, let participants walk you through the timeline as they see it. Those gaps, the places where folks differ or question alternative narratives, that's where they have their "aha!" moments. Most importantly let the interviewee talk and avoid correcting their answers (a common pitfall). That may seem counterintuitive but once you've nudged them you've inadvertently closed a door on all the potential misconceptions you might have trouble finding otherwise, insights you can develop as a team together in larger discussions.
Share the responsibility of discussion lead. Time is a luxury and not every incident is going to garner a corresponding meeting, but that doesn't mean all is lost. Some of that roteness comes from the same people writing the reviews every time. Pass the proverbial conch shell to another member of the team, specifically someone not involved in the incident or less experienced in writing. When they get stuck, if the timeline is not quite fitting together or the gaps in the runbook make themselves present, that's a great opportunity to level up. Likewise, if you can switch recitation of events between participants, you can avoid the risk of a soliloquy. It requires a little more courage, the potential to be wrong in front of a room of your colleagues, but that's also a great way to reinforce the blame awareness that's so core to a review.
Another pitfall I've fallen into is accidentally leading the witness, in both interview and discussion. It's tempting in interviews and various discussions to present data to folks to prime them. This sounds reasonable—you want them to be able to recall things, so why not present questions ahead of time? The rub is that priming will root them more in what you think is important. "Walk me through the event" and "What happened next?" are wonderfully simple starting points that will often surprise you as to where they will lead.
You're level setting without even realizing it, giving your team the chance to learn through others' lived experience. This may seem obvious, but the divide between what you read on a digital doc while your code compiles and the additional attention from a shared conversation is light years apart. Those retro docs you see shared externally, that you pore over at the virtual water cooler and say "Would have loved to be a fly on the wall in that room." You have a chance to!
Post Postmortem
Is your team ready for the next incident? Hard to say. Resilience is adapting to the unexpected, and by definition, once you know it, it isn't unexpected. Finding time to invest in these actions are equally fraught with uncertainty. What we can do is avoid fooling ourselves with the treacherous recitative work that only seems like learning.
We're performing these tasks with the hopes they're not performative. If that's a sincere belief, then it also only makes sense for us to extract the most knowledge from these unplanned exercises. The greatest irony in all of this is that our most frequent discussions about incident reviews are the ones conducted elsewhere, the large corporations who fail hard and leave us to speculate past the information presented to the details that are too salacious for public consumption. We don't have to be constrained by that internally though! We're putting so much effort into capturing data, we should make good on realizing everything that comes with it, treating it as a platform to launch these larger discussions and better prepare for the next incident.
Will Gallego
Secretary, Resilience in Software Foundation