OpenSRS Status

The OpenSRS Email Service - Cluster B maintenance has been completed ahead of schedule. All services are available.

Customers using domain aliases to login to webmail should no longer encounter login issues. We tested and implemented a fix.

All other Email services were unaffected.

This update is related to Incident 12557

Customers who use domain aliases to login to their webmail may encounter login issues. We are implementing a change and will keep you posted.

All other email services are working well.

This update is related to Incident 12557

The issue with provisioning services on Cluster B has cleared. NOC has fully tested and confirmed that services are fully available.

Inbound and outbound mail was unaffected. Mailbox access was fully available.

Incident Summary: (updated Thursday, April 15, 2010:  17:30 ET / 21:30 UTC  )

OpenSRS Email Services Cluster B had a brief provisioning issue from 16:55 ET/20:55 UTC - 17:16/21:16 ET.  Resellers may have experienced provisioning timeouts.

Our Network Monitoring system (Nagios) alerted of the high queue of provisioning jobs. This large queue of provisioning changes resulted in threads to be overworked. This caused the timeouts.  We cleared the provisioning queues. We tested services and confirmed online by 17:16/21:16 ET.

We have an existing project to restructure our provisioning systems. Part of this process is to improve the length of time required for all provisioning tasks.

Inbound/outbound mail and mailbox access (IMAP, POP, and Webmail) were unaffected.

This update is related to Incident 11874

OpenSRS Email Service is fully available however statistics provided in the MAC control panel are not accurate. We are aware of the issue and are working to correct it. We will update Status when the data is corrected.

We apologize for the inconvenience.

OpenSRS Email Services - Cluster B is fully online.

All provisioning requests via the Mail Administration Center (MAC) and Application protocol (APP) are being accepted and processed in normal time. The previous queue of change requests have been cleared.

Customer mailbox access and inbound/outbound mailbox were fully available during this incident.

Incident Summary (Updated 16:47 ET/20:47 UTC)
At about 02:15 ET our network monitoring system alerted us to issues with mailbox provisioning on Cluster B of our Email Service.

Analysis showed that provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) were degraded due to a high load on the provisioning system. Customer mailboxes remained accessible via IMAP, POP, and Webmail. Inbound and outbound mail were fully available.

By 04:07 ET we isolated the cause of the issue as a misconfigured provisioning script. The script put an unusually high load on the provisioning servers that caused normal provisioning requests to be queued.

At 04:24 ET we stopped all provisioning requests to investigate further. We cleared a invalid provisioning requests and restarted the provisioning system at 05:02 ET. All queued and new provisioning requests were then processed normally.

The provisioning queue cleared by 05:19 ET.

Our investigation shows that an incorrectly configured script caused this abnormally high load on the provisioning system. While we normally run such scripts during off-peak hours we are now refining our internal processes to ensure that these types of scripts receive greater scrutiny before being used.

We apologize for any inconvenience to you and your customers.

This update is related to Incident 8469