This article explains how we organised and completed three 90-minute crisis exercises today for our own company.
Setup
Our approach to business continuity is to ensure a range of people are able to step into the incident response team. Relying on a dedicated few isn’t practical for our organisation so we train as many as we can.
Team Size
Our incident team (formed on-demand from who is available and best suited at that time) has 4-5 people each dedicated to a particular role but able to switch roles depending on who is available and what access they have to our systems. Our security policy restricts data & systems access on a need-to-know basis so responding to an incident requires collaboration and communication to share information and direction.
Exercise Prep
I spent about 7 hours spread over three days building the exercise in Conducttr. I took a real incident from about 3 years ago and exported the Slack conversations and imported them into a master events list. This gave the exercise a realistic timing and realistic voice. I then created three other event lists – one that started the exercise and set the scene, and two others that could be used on-demand if I felt a team was getting along too well and needed a little extra pressure to stretch them. I also used our new AI assistant to create historic content for a fictional client.
I used our player grid feature to log in as a team and rehearse the exercise – checking that everyone would have enough to do. I used the data from the play through to build the Pulse dashboards so that I’d have a real-time view of the sessions as they played out. A screenshot of Pulse from one of the sessions is shown below. I saved all the necessary files for future reference: the exercise sharefile, the teams, the pulse dashboards and the training objectives.
Registration
I sent calendar invites for three sessions to the different teams. I scheduled the least experienced people to start the day and finished with the most experience. Having teams of similarly experienced people is helpful because it avoids a very capable person dominating the exercise and robbing less experienced people from a learning experience.
In my Conducttr exercise I registered about 20 staff and organised by department; then for each session I organised the incident response team by dragging and dropping from the department teams into the incident team.
The Zoom link was included in the calendar invite so at the allotted time everyone signed in and I started the introduction. Following a short presentation I gave everyone the exercise URL.
Session Structure
I wanted to use the time we were together to deliver some training and refreshers on our service level agreements and legal frameworks; and show how I’d reorganised our incident flowchart & materials. Hence I divided the sessions into a 15 min intro presentation and review of the training objectives; 60 min exercise and a 15 min wash-up/after action review. Note that I used a blended approach to the exercise – using Conducttr to drive the incident narrative, respond to support tickets and support collaboration but told the players to use our real systems to find information.
Observation & Facilitation
I’d designed the exercise so that I could run everything on my own but invited our CTO, Jon, to sit in as an observer to provide additional critique of the teams’ performance and notes for improvements in our procedure and materials. Jon and I were sat in a meeting room together with the Pulse dashboard on a big screen and the facilitator dashboard on my laptop screen. Once my presentation was over, I turned off my camera and muted my audio but we maintained the connection so we could eavesdrop on the conversations as well as follow in Pulse.
Execution
The screenshot below shows the virtual desktop I used. Because of the nature of this exercise and the inexperience of some in the team, I used a channel called “guidance” to add scaffolding and one or two hints, I named the log channel “thoughts” and invited anyone to record insights or observations during the exercise. Jon and I also recorded our observations there. You’ll see the checklist channel for verifying we’d covered all the training objectives and the help desk channel to simulate our usual support ticketing system. Finally I separated our #alerts channel from the main simulated Slack channel just to give it extra significance and focus. Some might argue that I introduced a few artificialities but these were design choices, not constraints.
As the exercise progressed I split the Facilitator Dashboard into three – the player view on the left where I could impersonate stakeholders if necessary, the review panel in the middle where I could see player responses to the injects and on the right the multiple events lists which opened and closed as needed. Notifications that went only to me prompted me at the right time to kick-off an additional MEL.
Wrap-up and Conclusions
At the end of the exercise I exported the session data and the exercise report and saved to a folder. After soliciting feedback from the team about how they thought they did and how immersive/useful they found the exercise, I was able to use the exported report at the basis for the wrap-up session to discuss feedback from Jon and me. Conducttr has a feature called After Action Review which allowed me to comment on any player response live during the exercise so I didn’t miss a thought or have to write it elsewhere.
It was very fulfilling and exciting to watch the three teams in action over the three sessions; and of course to see Conducttr in action. It is extremely reassuring to know that we’d simulated an incident as true to life as possible and the teams had all learned & gained something from the experience.