The speaker recounts causing a major production outage by running a backfill script that overwhelmed the Facebook API and halted data updates.
We tend to judge others by their actions but ourselves by our intentions, so we should assume good intent from colleagues during incidents.
When a production issue occurs, it's a system failure, not an individual's fault, as responsibility is shared across developers, reviewers, and processes.
Shifting to a blameless culture requires fostering trust, understanding intentions, practicing self-awareness, and owning mistakes without displacing frustration.
Blameless postmortems, originating from aviation and healthcare, focus on investigating root causes to strengthen systems rather than assigning individual blame.
Applying the five-to-one ratio of positive to negative interactions can improve team dynamics, especially by adding positive comments during code reviews.
Pre-mortems are a proactive exercise where teams imagine a project has already failed in order to identify potential risks and edge cases beforehand.
The incident took 20 hours to fully resolve but was a valuable learning experience that exposed system flaws and reinforced a healthy team culture.
The speaker answers audience questions about customer reactions to the outage and shares a story about his worst production break involving a failed form.
29:38Luís Ventura
28:36Noaa Barki
27:52Maish Saidel-Keesing
21:39Johannes Hansen
30:53Brenda Romero
29:55Malin Litwinski
27:27Christian Seifert
42:41Eric Rabinovich
Jobs that call for the skills explored in this talk.
Peter Park System GmbH
München, Germany
smartclip Europe GmbH
Hamburg, Germany
CONTIAMO GMBH
Berlin, Germany
AUTO1 Group SE
Berlin, Germany
