INCIDENT COMMANDER

Andrew Hatch

Learning SRE

Incident Commanders in the wild (and film!)

Basics of incident command

Incident Commander Response Framework

Core skills of Incident Commanders

Managing communication

Working with Incident Commanders

References & Training

>

>

2009

2019

1M acres

2000

173

80M acres

3500

34

SRE HIVE

2M QPS

Outcomes

Checkpoints

?

		Be decisive but flexible, course-correct on the fly when needed
		Stay high-level, avoid diving in to solve technical issues
		Regulate the flow of information, be structured
		Domain knowledge of our systems, processes and stakeholders
		Avoid burnout, delegate yourself

Control emotions, mediate discussions and listen without judgement

Speak clearly, avoid jargon, slang and non-inclusive language.

Control noise, be firm but fair, avoid deference of hierarchy affects

Don't @omnibot the hell out of the main channel.

Check your biases at the door. E.g.

"How could we not notice this?" or "Weren't you paying attention?"

Don't ask for repeated status updates in Slack.

Stay focused on what you are doing, avoid multi-tasking. Be clear on what you need.

Pager Duty Incident Commander
Atlassian Incident Commander
Earthquakes, Forest Fires, and Your Next Production Incident		A Hidalgo
How to be a great Incident Commander		E Binette and B Adele Long
Anticipating, monitoring and managing incidents via Slack		R Obstler, D Bruce, A Rayl and J Soin
Incident Review and Postmortem Best Practices Analysis		G Orosz
Exploring Costs of Coordination During Outages		L Maguire
Learning More from Complex Systems		A Hatch

Thank you!

https://sitecon22.hatchman76.com

@hatchman76

@anhatch