Surviving a crisis - a view from the trenches

by Stuart Corner

BIRTH AND DEATH OF A CRISIS—AND THE BITS IN BETWEEN

If you’re in the throes of a cybersecurity crisis, you’d certainly be aware of it. But pinpointing the start of a crisis and, just as important, the end of it, is not so easy, according to Eric Pinkerton, aka ‘Pinky’.

For his presentation at AusCERT2021, he was introduced as “Eric Pinkerton from Trustwave.” His LinkedIn profile is rather more colourful. “Eric of House Trustwave, Breaker of blockchain, King of the memes, Lord of the Files, Counter of Monte Carlo, Raider of the lost Archives, Father of PAE-PAL-PFU, Douser of trash fires, Speaker of unpleasant truths.”

And there were unpleasant truths aplenty in his presentation. On identifying the start of a crisis, and triggering a crisis management plan, Pinkerton said: ”You don’t want to rely on Jerry on the help desk having a funny feeling about this. Because if Jerry isn’t working on that particular day, someone else might look at it more subjectively. You want to try to nail down the point at which the crisis management team has to be convened. You want that to happen as soon as it’s possible to say, ‘this is a serious thing,’ but not so often that everyone gets fatigued and jaded by getting constant phone calls at 2am on a Friday night.”

Nor, he says, is the endpoint of a crisis always clear, and failure to recognise this can also have serious consequences. “I’ve been in crisis situations that have dragged on for weeks and weeks because nobody wanted to make the call that it was over.

“It proved very costly for the organisation to have people sitting around, not just financially, but in terms of having very senior people constantly checking in on meetings. They can’t go and do what they normally would be doing. They’re very senior executive people. So that has an impact on the organisation.”

NK, YK,TK, EK: THE FOUR PHASES OF A CRISIS

Pinkerton identified four phases to a crisis: NK, YK,TK, EK. “Nobody knows, you know, they know, everybody knows. … Number three, ‘they know’, is where you have to go and talk to the people affected, or your partners, or a wider circle. Step four, ‘everybody knows’, is when you have journalists phoning you for a comment. That is what you’re trying to avoid.”

A key to successful crisis management, is to resolve the crisis, bypassing stage four, but then, he said there are two other, usually inevitable stages: the blame game and the postmortem, and a further important stage, often not implemented. confident that the attacker they’re trying to evict from the network is not across their communications.”

• Have cyber insurance and knowing what it covers.

“The blame game typically happens before the postmortem. People will speculate before you have a position on exactly what went wrong,” he said. “And then the final piece, which I think is the most commonly overlooked piece, is what I call ‘executing on lessons learned’: making sure the findings and recommendations from the postmortem are followed through. “Otherwise you will find yourself having déjà vu. You will find yourself in a meeting going ‘hang on a minute, this happened before. I’m sure we should have solved this problem’. But nobody followed through or nobody checked that it was done.”

“I was working with an organisation that got hit by ransomware. They were having a dialogue with the attackers, and said, ‘We can’t possibly afford to pay this ransom.’ The attacker said, ‘yes, you can. You can just claim on your insurance.’ And they said, ‘No, our insurance won’t cover this’. So the attacker “It proved very costly for the organisation to have people sitting around, not just financially, but in terms of having very senior people constantly checking in on meetings. They can’t go and do what they normally would be doing. They’re very senior executive people. So that has an impact on the organisation.”

This aspect of crisis response and management was one of many pieces of sound advice offered by Pinkerton in his presentation. Many might seem like crisis management 101. But as was demonstrated by the case study he presented, of a very high profile crisis management fail by global IT company; just because they are basic does not mean they are followed.

Here are some aspects of crisis management he presented that are perhaps most likely to be overlooked.

• Set up an ‘out of band’ communications channel

for those identified as being part of the crisis management team, such as a WhatsApp group, or gmail accounts.

“I’ve worked with organisations that have had reason to question the integrity of the communication platforms they’re using, because the hypothesis is that there might be a state-based actor active within their [Microsoft] Exchange, and they cannot be sent a copy of the insurance certificate that they has exfiltrated from this organisation, highlighted, saying ‘yes, you’re covered, don’t worry’.”

Pinkerton said attackers had been known to take this strategy even further: hacking the insurer, identifying clients that were appropriately insured against ransomware, working their way through these and, finally, targeting the insurer.

• Make sure the crisis response is fully documented

“Organisations, unless they’re very disciplined, will not naturally start scribing what is happening. That is absolutely critical, because after every crisis there will be questions asked: ‘why did that decision get made?’. In hindsight, it’s very easy to say, ‘well, that was the wrong decision’. But if you document the rationale for that decision: ‘what we knew, what we didn’t know, at that particular time was this, and that’s why we all agreed to do that’. If that is recorded in black and white, then your covered.”

Surviving a crisis by Kylie Watson

• Line up external crisis response partners

“If you need to engage an incident response or forensic partner, or external legal counsel, get those ducks in a row before you need to. Because when you phone an incident response company, or a lawyer, at 6pm,on a Friday, which will always be when this stuff happens, and start trying to negotiate commercials and rates and sign NDAs, you will lose hours if not days, and it will cost you a fortune.”

Pinkerton finished his presentation with some case studies of crisis management failures, the bestknown likely being the failure of Australia’s 2016 online census, delivered by IBM.

HEADLESS CHICKENS IN CRISIS

“It went live, there was a denial of service attack. It resulted in IBM running around like headless chickens. There was a 43 hour outage during the period you were supposed to do your census. … The lessons learned were that IBM had mountains of playbooks, but they were completely untested, and they were completely impractical. The ABS had no real incident response process, and they did the whole thing on the fly and ad hoc .

“Support mechanisms weren’t linked to the process. So they didn’t have the phone numbers they could ring, or a process where they had a team ready and waiting ready to go. Escalation thresholds were not clear. So the point at which somebody needed to call a minister or someone needed to give someone an update were lost in the fog of war. The DDoS protections obviously were deemed inadequate. The DDoS attack itself was very, very low volume and should not have caused a problem had they been doing the right thing.

“Crisis communications were absolutely shocking. They were very, very after the fact, and were very short and terse. IBM had employed a subcontractor which they ended up trying to throw under the bus when [prime minister] Malcolm [Turnbull] threw IBM under the bus. It was a massive, massive mess.

“Had they done a tabletop exercise or a crisis simulation many of those things could have been picked up before they were a problem.”