Focus on the functionality

IT-manager in the middle of the crisis

After one month, the e-mail crash is far from resolved. A majority of those affected have got their inbox back, even if it was empty. However, hundreds of people still have no functioning email. – From our perspective, the focus is now on getting the functionality up and running, says IT Manager Sören Ehrnberg.

But the external audit must ascertain what happened and why. It is also unclear whether all the data can be recovered.

AT THE TIME OF the journal’s deadline (October 15), the IT Unit had managed to restore 3,300 active e-mail accounts out of a total of 5,000. The work on restoring the inboxes takes time, which the IT manager finds regrettable. – We completely understand that it is frustrating to lose your email and that a lot of people have had to wait so long. It is very serious and we are doing everything we can to sort this out. It is a difficult balancing act between, on the one hand getting the communication in and out, and on the other hand not jeopardizing the stored data. The risk of turning it back on across the board is that we risk destroying an empty mailbox. It is a very complex and multifaceted computer environment with different email programmes, which complicates the work that to a great extent has to be done manually for each user, says Sören Ehrnberg.

In recent weeks, there has been a lot of pressure on IT support, which has had to handle three times as many cases as usual on a daily basis, and is getting almost 300 calls a day. Most of them involve the email crash. According to Zeljko Ergic, Head of IT Support, close to 100 people are working on the email incident. – And almost 60 percent of them have worked overtime to varying degrees. In addition, staff from our operating partner ATEA and subcontractors such as Microsoft have been extensively involved in the work.

The damaged disks have been sent to the hard disk company IBIS, which will try to save the data on them. As far as the GU Journal can ascertain, the company charges SEK 6,000-7,000 per hour for the work. – But in the context this is not a lot of money, considering the impact this has had on the university. The focus is on correcting it, rather than on what it costs, says Sören Ehrnberg.

According to the IT manager, the crash was due to a hidden bug in the firmware, which damaged the email systems. The entire incident has been reported to the Swedish Data Protection Authority. The incident will be examined by external investigators, according to a decision by the University’s Chair of the Board and Vice-Chancellor. Sören Ehrnberg believes that it will involve two different reviews: a technical audit and another more indepth investigation that focuses on the process and working methods. – What we know currently is that the storage disks failed, both the regular ones and the backups at the same time. We would like to know more about the exact cause. The “black box” will be analysed and the audit will find out what is right or wrong.

Is there a risk of data and email history being lost?

– That risk cannot be ruled out. Currently, we do not know what the prognosis is. We are collecting everything that is available and that we can get our hands on. The best case scenario is that we can recreate everything as it was before the crash, and the worst case scenario is that we can recreate what exists today, says Sören Ehrnberg.

What responsibility does the company ATEA, which manages the University of Gothenburg’s email system, have for what happened, given that they have 24/7 monitoring?

– This is something that the audit will have to take a closer look at. Our external partners are doing everything they can to get your email up and running and save data. But the hardware itself is owned by and located at the University of Gothenburg.

What lessons have you learned from the email crash?

– We could not even imagine this happening, so it is difficult to learn any lessons from it. One challenge is to communicate the right information to the rest of the organisation. Thankfully, we have had around-the-clock help from the Communication Unit. – In March, we had a finalized plan to migrate everyone over to the new Office 365 platform, which is cloud-based and thus more secure. But then the corona pandemic happened and we had to postpone those plans. But I’m pretty sure this would not have happened if we had switched to that solution this spring, says Zeljko Ergic, who emphasizes that the IT Unit adheres to Microsoft’s procedures (best practice) for what to do in a crisis like this. – We try to minimize the impact for our users. We are working in the background to create a recovery mailbox that we are also transferring to the cloud to secure the information. We completely understand that it has created a tremendous amount of irritation and anger. Email is very important. My staff in IT are extremely skilled and have a good understanding of the users’ experience, so we are doing our utmost to solve the problems as soon as possible.

Text: Allan Eriksson Photo: Johan Wingborg

The e-mail crisis has lead to a great amount of questions from the staff.

IT-manager in the middle of the crisis