Multiple Service Events - A Busy Monday
This event has now been resolved, Scroll down for latest updates.
Summary of events
We experienced several events on Monday. IDC remains fully operational.
Planned maintenance of four hours is set to commence at 7:30pm this evening, Tuesday 21 January. We will update this page throughout the day as plans are finalised. During this maintenance time access to Earthlight will be interrupted, however your connection online will remain intact. ** Updated 21/1 09:30 Planned maintenance on hold, scroll down
On Monday morning we failed over to generator. This was in response to loss of main supply indications. It is a routine event. The automated changeover switch failed to show the main supply had restored. Investigation pointed to failing sensor electronics within the switch. Replacement parts were ordered and we manually changed back over to main supply. Later in the morning we received indications of phase issues with supply. We failed over to generator again and further investigations were carried out.
Early afternoon things escalated with one supply phase failing. This affected some air conditioning, reducing capacity. Troubeshooting of this revealed a larger issue to do with supply to campus. Meantime a water radiator decided it was 'time' and cracked, flooding an unoccupied office. We continued to monitor temperatures in Machine and UPS rooms, everything remained within limits. Troubleshooting for the failed phase identified cause and was remedied, restoring full power to the IDC environment.
A campus supply issue remained and was further investigated. It was determined that an underground supply cable has deteriorated, dropping one phase. By this stage it was getting on for 8:00pm. The next step was to start digging to locate the cable fault. The supply company opted to provide a large (XXL) generator to power the entire campus. This was brought onsite and commissioned. All supply was restored approximately 10:00pm.
A replacement changeover switch is due onsite today. The cable fault in the supply to campus will be located and remedied.
We will review failover plans should they be required. Meantime we continue to monitor the situation and will post further updates here.
While outages are never welcome, we remained online throughout and communicated as best we could to keep everyone informed. We also note a similar level of committment and communication from Campus support staff, and this made dealing with issues so much easier. As always if you need anything further please get in touch. Meantime we have a wet carpet smell a few doors down to get rid of.
21/1 7:30am Daily checks all fine. Entire campus on generator supply from lines company.
21/1 9:30am Planned maintenance this evening has been postponed. Supply cable work will be undertaken first. Once completed the changeover switch will be retested.
21/1 4:30pm Fault has been located in the supply cable. Digging has exposed it ready for repairs.
22/1 9:30am Repairs to the supply cable are underway. These will take all of today and go into tomorrow. At this stage looking at cutting back to regular supply late Thursday afternoon.
23/1 9:30am Repair work is proceeding. Wrap-up today. Plan is to then shut down generator supply to campus at 5:00pm and reconnect to repaired supply. IDC will fail over to independent generator backup while the change back takes place. Staff will be in attendance throughout. Once done we will review operation of the changeover switch to satisfy ourselves with its operation.
23/1 6:30pm Change back for supply commenced at 5:00pm. IDC went over to generator and then returned to supply once changeover work was complete. Everything is running and all indications are good. Naturally we will continue to monitor services as part of our routine.