Email Service Status Archive
Email Cluster B is Online
OpenSRS Email Service is fully available however statistics provided in the MAC control panel are not accurate. We are aware of the issue and are working to correct it. We will update Status when the data is corrected.
We apologize for the inconvenience.
Email Cluster A is Online
OpenSRS Email Service is fully available however statistics provided in the MAC control panel are not accurate. We are aware of the issue and are working to correct it. We will update Status when the data is corrected.
We apologize for the inconvenience.
Email Cluster A is Online
The OpenSRS Email Services maintenance is complete.
Email Cluster A is In Maintenance
There is a 3-hour maintenance for OpenSRS Email Services - Cluster A starting now (04:00 UTC). This time will be used to update the version of firmware (IOS) on a number of switches.
Service Impact:
OpenSRS Email Services Cluster A will be affected briefly during the 3-hour window.
Impact for Resellers:
The Mail Administration Center (MAC) will be temporarily unavailable for under 30 minutes within the window.
Impact for End-users:
The service interruption is expected to be less than an hour. It may be measured in minutes or seconds in some cases. During this window, customers may not have access to their mailboxes via IMAP, POP or Webmail. Inbound mail will be temporarily remotely queued for delivery.
Email Cluster A is Online
All OpenSRS Email services on Cluster A are fully available.
Some customers may experience Webmail rendering issues in some browsers. The temporary workaround is to have users clear their browser cache. We are investigating the javascript discrepancy and we will provide further updates within this post when available.
(IMAP and POP users are unaffected. Inbound/Outbound mailflow is working well.)
Update Thursday, November 4, 2009 16:10 UTC):
Our Development and Operations team identified a bug and rolled back the Webmail code that was causing this issue.
This update is related to Incident 9064
Email Cluster B is Online
OpenSRS Email Services - Cluster B is fully online.
All provisioning requests via the Mail Administration Center (MAC) and Application protocol (APP) are being accepted and processed in normal time. The previous queue of change requests have been cleared.
Customer mailbox access and inbound/outbound mailbox were fully available during this incident.
Incident Summary (Updated 16:47 ET/20:47 UTC)
At about 02:15 ET our network monitoring system alerted us to issues with mailbox provisioning on Cluster B of our Email Service.
Analysis showed that provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) were degraded due to a high load on the provisioning system. Customer mailboxes remained accessible via IMAP, POP, and Webmail. Inbound and outbound mail were fully available.
By 04:07 ET we isolated the cause of the issue as a misconfigured provisioning script. The script put an unusually high load on the provisioning servers that caused normal provisioning requests to be queued.
At 04:24 ET we stopped all provisioning requests to investigate further. We cleared a invalid provisioning requests and restarted the provisioning system at 05:02 ET. All queued and new provisioning requests were then processed normally.
The provisioning queue cleared by 05:19 ET.
Our investigation shows that an incorrectly configured script caused this abnormally high load on the provisioning system. While we normally run such scripts during off-peak hours we are now refining our internal processes to ensure that these types of scripts receive greater scrutiny before being used.
We apologize for any inconvenience to you and your customers.
This update is related to Incident 8469
Email Cluster B is Degraded
We restarted the Provisioning system. All previously issued new and change requests are being processed. There is a backlog of provisioning requests. As it is not prime time business hours, these are processing fast. New change provisioning requests via the Mail Administration Center (MAC) and APP are being accepted for processing.
We identified the cause of the provisioning issue and addressed it. We will continue our investigation.
Customer mailboxes are fully available. Mailbox access was not affected by this issue. Inbound and outbound mail are also working well.
This update is related to Incident 8469
Email Cluster B is Degraded
OpenSRS - Email Services Cluster B provisioning services are offline. All other Email Services including mailbox access and mailflow are fully available and online. Customers can access their mailboxes via IMAP, POP and Webmail. Inbound and outbound mail flow are unaffected.
Provisioning System Unavailable (Offline)
Our technical teams continue to investigate a high load on the provisioning system. All provisioning changes will be queued. We have temporarily stopped the provisioning system. Changes via the Mail Administration Center (MAC) and Application protocol (APP) are unavailable at this time.
This update is related to Incident 8469
Email Cluster B is Degraded
OpenSRS Email Services - Cluster B provisioning services are degraded. Our operations team is investigating a high load of provisioning requests which is causing all new and change requests to be delayed. Provisioning changes will be queued for processing and be completed once this backlog of requests is flushed.
All mailboxes are available via IMAP, POP and Webmail. Inbound and Outbound mail are also unaffected.
Update 07:07 UTC:
We have identified the cause of the high load on the provisioning system.
Update 07:19 UTC:
The provisioning change request queue is slowly processing. There continues to be a backlog of change requests queued. Change requests submitted will be processed with some delay. We continue to work on addressing the cause of the high load on the provisioning system.
Update 07:56 UTC:
Our Technical teams are investigating the best solution to address the high load on the provisioning systems.
This update is related to Incident 8469
Email Cluster A is Online
The OpenSRS Email Services - Cluster A maintenance is complete. Our technical teams finished their work ahead of schedule and confirm all services are online.