We apologize for the inconvenience.

We apologize for the inconvenience.

Service Impact:
OpenSRS Email Services Cluster A will be affected briefly during the 3-hour window.

Impact for Resellers:
The Mail Administration Center (MAC) will be temporarily unavailable for under 30 minutes within the window.

Impact for End-users:
The service interruption is expected to be less than an hour. It may be measured in minutes or seconds in some cases. During this window, customers may not have access to their mailboxes via IMAP, POP or Webmail. Inbound mail will be temporarily remotely queued for delivery.

Some customers may experience Webmail rendering issues in some browsers. The temporary workaround is to have users clear their browser cache. We are investigating the javascript discrepancy and we will provide further updates within this post when available.

(IMAP and POP users are unaffected. Inbound/Outbound mailflow is working well.)

Update Thursday, November 4, 2009 16:10 UTC):
Our Development and Operations team identified a bug and rolled back the Webmail code that was causing this issue.

All provisioning requests via the Mail Administration Center (MAC) and Application protocol (APP) are being accepted and processed in normal time. The previous queue of change requests have been cleared.

Customer mailbox access and inbound/outbound mailbox were fully available during this incident.

Incident Summary (Updated 16:47 ET/20:47 UTC)
At about 02:15 ET our network monitoring system alerted us to issues with mailbox provisioning on Cluster B of our Email Service.

Analysis showed that provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) were degraded due to a high load on the provisioning system. Customer mailboxes remained accessible via IMAP, POP, and Webmail. Inbound and outbound mail were fully available.

By 04:07 ET we isolated the cause of the issue as a misconfigured provisioning script. The script put an unusually high load on the provisioning servers that caused normal provisioning requests to be queued.

At 04:24 ET we stopped all provisioning requests to investigate further. We cleared a invalid provisioning requests and restarted the provisioning system at 05:02 ET. All queued and new provisioning requests were then processed normally.

The provisioning queue cleared by 05:19 ET.

Our investigation shows that an incorrectly configured script caused this abnormally high load on the provisioning system. While we normally run such scripts during off-peak hours we are now refining our internal processes to ensure that these types of scripts receive greater scrutiny before being used.

We apologize for any inconvenience to you and your customers.

We identified the cause of the provisioning issue and addressed it. We will continue our investigation.

Customer mailboxes are fully available. Mailbox access was not affected by this issue. Inbound and outbound mail are also working well.

Provisioning System Unavailable (Offline)
Our technical teams continue to investigate a high load on the provisioning system. All provisioning changes will be queued. We have temporarily stopped the provisioning system. Changes via the Mail Administration Center (MAC) and Application protocol (APP) are unavailable at this time.

All mailboxes are available via IMAP, POP and Webmail. Inbound and Outbound mail are also unaffected.

Update 07:07 UTC:
We have identified the cause of the high load on the provisioning system.

Update 07:19 UTC:
The provisioning change request queue is slowly processing. There continues to be a backlog of change requests queued. Change requests submitted will be processed with some delay. We continue to work on addressing the cause of the high load on the provisioning system.

Update 07:56 UTC:
Our Technical teams are investigating the best solution to address the high load on the provisioning systems.