Continuing Repairs from Power Outage

January 24, 2018

CIS is continuing to recover from this past weekend's power outage. While services are currently online and operational, over the next week, we may experience periodic slowness of online resources, internet connections, and other short system outages.

Three Key Reasons Why This is Happening?

Damaged Equipment

We are continuing to find equipment, particularly networking gear, that has been damaged by the power outage. Replacing the equipment may come with additional isolated outages that may impact an entire building.  CIS staff will be communicating any anticipated outages to key staff in those areas, but some unexpected services disruptions may continue to occur.

Damaged Systems and Software 

Data Centers do not handle catastrophic loss of power well. While some of SPU's online systems came back online with no issue, others did not.  Last night's Banner Information System outage was one of these casualties.  We are continuing to get alerts from systems that did not fully recover after the power outage and we are working to address them as fast as we can.

System Backups

Each day SPU backs up all on campus data systems to a disaster recovery side located at SPU and to an off-site cloud backup.  This process typically happens overnight and only sends the changes from the day before. The outage and recovery process caused a significant amount of change, which requires us to process a complete backup of all systems.  Starting from scratch like this takes 1-2 weeks running 24/7, which means we can't limit the backup process to non-business hours.

While a system is actively being backed you may experience significant slowness.  CIS staff will be contacting key staff we believe may be impacted by this ongoing process.