Connectivity and Service Issues

** Scroll down for latest updates **

On Sunday March 24 services were affected following changes performed on the core network upstream. We became aware of some service issues early Sunday morning. Connectivity appeared fine overall, although we did see some sites were offline. Troubleshooting cleared our systems and core network. At this point e-mail was flowing. The most noticable issue being access to webmail.

At this point we reached out to upstream providers and shared our symptoms and observations. This enabled troubleshooting further afield to identify issues. To remedy, parts of the upstream core network required restart for changes to be applied. Once completed all services returned.

We then proceeded with recheck of our services, client site connectivity and connections upstream. All passed. Clients who had been in touch were contacted to advise service restoration and check their side of things. A debrief of the events is being undertaken on Monday. We will advise further here if there is anything to add.

Sunday Event Timeline

0200 - 0500 Network changes performed upstream from Earthlight
0700 Earthlight picked up issues with Webmail and own checkout proceeded. Initially appeared to be a local issue rather than system related
1030 Following wider checking reach out to upstream sharing details
1230 Elements of core upstream network restarted
1330 Restarts complete. Checkout processes underway
1420 Clients contacted to advise services, connectivity restored

The item will be updated further on Monday once we receive debrief proceedings and review our own processes.

** Update Tuesday 3:30pm **

Two things - 1) Overnight this evening there will be a brief outage while a core connection is moved from one location to another. Staff will be onsite to carry out the change and monitor network status. This will follow a changeover in part of the Christchurch handover (which serves north of the Waitaki River) which will involve an outage of up to ten minutes. This will take place between 0200 - 0500.

2) The debrief of weekend work identified some processes which will be improved going forward. The changes themselves had some delays along the way as implementation threw up some issues that had to be worked through. Tonight's change is less involved in a lot of ways, and staff will be checking once the changes have been made to ensure all services are functioning.