Service Impact:
OpenSRS Email Services - Cluster B will be available during the 4-hour window except calendar services. All mailbox access (IMAP, POP, SMTP, Webmail) and provisioning via the Mail Administration Center (MAC) will remain fully online.

Impact for Resellers and End users:
Calendar services will be unavailable for the duration of the window. All other Open SRS Email Services will remain available.

Impact for End-users:
Calendar services will be unavailable for the duration of the window. All other Open SRS Email Services will remain available. Calendar functionality will remain unchanged. This is an infrastructure change only.

All other Email services were unaffected.

All other email services are working well.

Inbound and outbound mail was unaffected. Mailbox access was fully available.

Incident Summary: (updated Thursday, April 15, 2010:  17:30 ET / 21:30 UTC  )

OpenSRS Email Services Cluster B had a brief provisioning issue from 16:55 ET/20:55 UTC - 17:16/21:16 ET.  Resellers may have experienced provisioning timeouts.

Our Network Monitoring system (Nagios) alerted of the high queue of provisioning jobs. This large queue of provisioning changes resulted in threads to be overworked. This caused the timeouts.  We cleared the provisioning queues. We tested services and confirmed online by 17:16/21:16 ET.

We have an existing project to restructure our provisioning systems. Part of this process is to improve the length of time required for all provisioning tasks.

Inbound/outbound mail and mailbox access (IMAP, POP, and Webmail) were unaffected.

All other email services including inbound/outbound mail and mailbox access are working well.

We will have more details shortly.

We apologize for the inconvenience.

All provisioning requests via the Mail Administration Center (MAC) and Application protocol (APP) are being accepted and processed in normal time. The previous queue of change requests have been cleared.

Customer mailbox access and inbound/outbound mailbox were fully available during this incident.

Incident Summary (Updated 16:47 ET/20:47 UTC)
At about 02:15 ET our network monitoring system alerted us to issues with mailbox provisioning on Cluster B of our Email Service.

Analysis showed that provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) were degraded due to a high load on the provisioning system. Customer mailboxes remained accessible via IMAP, POP, and Webmail. Inbound and outbound mail were fully available.

By 04:07 ET we isolated the cause of the issue as a misconfigured provisioning script. The script put an unusually high load on the provisioning servers that caused normal provisioning requests to be queued.

At 04:24 ET we stopped all provisioning requests to investigate further. We cleared a invalid provisioning requests and restarted the provisioning system at 05:02 ET. All queued and new provisioning requests were then processed normally.

The provisioning queue cleared by 05:19 ET.

Our investigation shows that an incorrectly configured script caused this abnormally high load on the provisioning system. While we normally run such scripts during off-peak hours we are now refining our internal processes to ensure that these types of scripts receive greater scrutiny before being used.

We apologize for any inconvenience to you and your customers.

We identified the cause of the provisioning issue and addressed it. We will continue our investigation.

Customer mailboxes are fully available. Mailbox access was not affected by this issue. Inbound and outbound mail are also working well.