We apologize for the inconvenience.

All provisioning requests via the Mail Administration Center (MAC) and Application protocol (APP) are being accepted and processed in normal time. The previous queue of change requests have been cleared.

Customer mailbox access and inbound/outbound mailbox were fully available during this incident.

Incident Summary (Updated 16:47 ET/20:47 UTC)
At about 02:15 ET our network monitoring system alerted us to issues with mailbox provisioning on Cluster B of our Email Service.

Analysis showed that provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) were degraded due to a high load on the provisioning system. Customer mailboxes remained accessible via IMAP, POP, and Webmail. Inbound and outbound mail were fully available.

By 04:07 ET we isolated the cause of the issue as a misconfigured provisioning script. The script put an unusually high load on the provisioning servers that caused normal provisioning requests to be queued.

At 04:24 ET we stopped all provisioning requests to investigate further. We cleared a invalid provisioning requests and restarted the provisioning system at 05:02 ET. All queued and new provisioning requests were then processed normally.

The provisioning queue cleared by 05:19 ET.

Our investigation shows that an incorrectly configured script caused this abnormally high load on the provisioning system. While we normally run such scripts during off-peak hours we are now refining our internal processes to ensure that these types of scripts receive greater scrutiny before being used.

We apologize for any inconvenience to you and your customers.

We identified the cause of the provisioning issue and addressed it. We will continue our investigation.

Customer mailboxes are fully available. Mailbox access was not affected by this issue. Inbound and outbound mail are also working well.

Provisioning System Unavailable (Offline)
Our technical teams continue to investigate a high load on the provisioning system. All provisioning changes will be queued. We have temporarily stopped the provisioning system. Changes via the Mail Administration Center (MAC) and Application protocol (APP) are unavailable at this time.

All mailboxes are available via IMAP, POP and Webmail. Inbound and Outbound mail are also unaffected.

Update 07:07 UTC:
We have identified the cause of the high load on the provisioning system.

Update 07:19 UTC:
The provisioning change request queue is slowly processing. There continues to be a backlog of change requests queued. Change requests submitted will be processed with some delay. We continue to work on addressing the cause of the high load on the provisioning system.

Update 07:56 UTC:
Our Technical teams are investigating the best solution to address the high load on the provisioning systems.

Our Technical Operations team finished the necessary work early. All Email Services are fully available. (The 3-hour scheduled window was announced to end at 11:00 UTC.)

Service Impact for Resellers:
We will use this time to upgrade our core routers to accommodate 10 Gigabit cards. To minimize the impact on your services, traffic will be routed through secondary routers. This action should limit the actual down time to approximately 20 - 30 minutes within the 3 hour window. OpenSRS Email Services via Webmail, IMAP and POP, inbound/outbound mail flow Provisioning via the Mail Administration Center (MAC) and the Application Protocol Interface (API) will be unavailable during that time.

Service Impact for End-users:
Within the window, customer will experience 20 - 30 minutes with no access to their mailbox via Webmail, IMAP, or POP. Their mail will be temporarily remotely queued for delivery during that same period.

All services including IMAP, POP, and Webmail are online. Inbound and outbound mail are flowing. Provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) are online too.

We worked with NetApp (our storage vendor) to fix a number of bugs which revealed themselves when we changed the faulty hardware component. These were addressed and fully tested as resolved. We apologize that we had to extend the window an additional hour, but it was necessary to test and resolve these items.

Update: 08:30 ET (12:30 UTC)

Our Technical Operations, Network Operations and Technical Support teams continued testing services after we changed the status to Online. All services continue to be fully available. All our Network Monitoring tools indicate that services are all up.

We continue to closely monitor all services.

During the window our Technical Operations (Ops) team has been working closely with the NetApp Storage team to resolve a number of minor issues after we replaced the faulty hardware component. There is one minor anomaly that we are investigating and testing. We have been assured that the data is fine and we should be able to move to the next steps shortly.

The next steps are to carefully bring the cluster back up with full monitoring and test each of the service elements (IMAP, POP, Webmail, etc.). Our Network Operations Center and Technical Support teams will do this. Once we have confirmation that all systems are up and tested well, we will let you know. We may not need the full hour, but we want to ensure full testing is completed.

We apologize for the inconvenience to you and your customers.

We are scheduling a 3-hour emergency maintenance window for OpenSRS Email Cluster B on Sunday, August 16, 2009 UTC. Within this window, we will be shutting down the cluster for a period of approximately 2 hours to perform necessary maintenance. The additional hour will be used for testing and quality assurance.

Our Operations team has detected a hardware failure on a redundant component of our NetApp Storage device. Currently there is no impact to your services or the cluster performance. We are 100% online. In order to minimize any risks to our customers we are exercising extreme caution in repairing this component. Thus, we will be performing a complete shutdown of the cluster before proceeding with our work.

Service Impact:
OpenSRS Email Cluster B will be offline. Customers will have no access to their mailboxes via IMAP, POP and Webmail. Provisioning services and inbound/outbound mail will also be unavailable. Inbound mail will be remotely queued for delivery once the window is complete.

We are scheduling a 3 hour window as a precaution, however we do not expect to utilize the whole window.