Email Cluster B Status Archive
Email Cluster B is Degraded
OpenSRS - Email Services Cluster B provisioning services are offline. All other Email Services including mailbox access and mailflow are fully available and online. Customers can access their mailboxes via IMAP, POP and Webmail. Inbound and outbound mail flow are unaffected.
Provisioning System Unavailable (Offline)
Our technical teams continue to investigate a high load on the provisioning system. All provisioning changes will be queued. We have temporarily stopped the provisioning system. Changes via the Mail Administration Center (MAC) and Application protocol (APP) are unavailable at this time.
This update is related to Incident 8469
Email Cluster B is Degraded
OpenSRS Email Services - Cluster B provisioning services are degraded. Our operations team is investigating a high load of provisioning requests which is causing all new and change requests to be delayed. Provisioning changes will be queued for processing and be completed once this backlog of requests is flushed.
All mailboxes are available via IMAP, POP and Webmail. Inbound and Outbound mail are also unaffected.
Update 07:07 UTC:
We have identified the cause of the high load on the provisioning system.
Update 07:19 UTC:
The provisioning change request queue is slowly processing. There continues to be a backlog of change requests queued. Change requests submitted will be processed with some delay. We continue to work on addressing the cause of the high load on the provisioning system.
Update 07:56 UTC:
Our Technical teams are investigating the best solution to address the high load on the provisioning systems.
This update is related to Incident 8469
Email Cluster B is Online
The OpenSRS Email Services Cluster B maintenance is complete.
Our Technical Operations team finished the necessary work early. All Email Services are fully available. (The 3-hour scheduled window was announced to end at 11:00 UTC.)
Email Cluster B is In Maintenance
There is 3-hour network maintenance at our data center for OpenSRS Email Services Cluster B starting now (08:00 UTC).
Service Impact for Resellers:
We will use this time to upgrade our core routers to accommodate 10 Gigabit cards. To minimize the impact on your services, traffic will be routed through secondary routers. This action should limit the actual down time to approximately 20 - 30 minutes within the 3 hour window. OpenSRS Email Services via Webmail, IMAP and POP, inbound/outbound mail flow Provisioning via the Mail Administration Center (MAC) and the Application Protocol Interface (API) will be unavailable during that time.
Service Impact for End-users:
Within the window, customer will experience 20 - 30 minutes with no access to their mailbox via Webmail, IMAP, or POP. Their mail will be temporarily remotely queued for delivery during that same period.
Email Cluster B is Online
OpenSRS Email Services Cluster B maintenance window is online. We are full monitoring the services and continue our testing.
All services including IMAP, POP, and Webmail are online. Inbound and outbound mail are flowing. Provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) are online too.
We worked with NetApp (our storage vendor) to fix a number of bugs which revealed themselves when we changed the faulty hardware component. These were addressed and fully tested as resolved. We apologize that we had to extend the window an additional hour, but it was necessary to test and resolve these items.
Update: 08:30 ET (12:30 UTC)
Our Technical Operations, Network Operations and Technical Support teams continued testing services after we changed the status to Online. All services continue to be fully available. All our Network Monitoring tools indicate that services are all up.
We continue to closely monitor all services.
Email Cluster B is In Maintenance
We are extending OpenSRS Email Cluster B maintenance window for an additional hour. We will use this time to finalize our testing.
During the window our Technical Operations (Ops) team has been working closely with the NetApp Storage team to resolve a number of minor issues after we replaced the faulty hardware component. There is one minor anomaly that we are investigating and testing. We have been assured that the data is fine and we should be able to move to the next steps shortly.
The next steps are to carefully bring the cluster back up with full monitoring and test each of the service elements (IMAP, POP, Webmail, etc.). Our Network Operations Center and Technical Support teams will do this. Once we have confirmation that all systems are up and tested well, we will let you know. We may not need the full hour, but we want to ensure full testing is completed.
We apologize for the inconvenience to you and your customers.
Email Cluster B is In Maintenance
OpenSRS Email Service - Cluster B has a 3-hour maintenance window starting now. (08:00 UTC).
We are scheduling a 3-hour emergency maintenance window for OpenSRS Email Cluster B on Sunday, August 16, 2009 UTC. Within this window, we will be shutting down the cluster for a period of approximately 2 hours to perform necessary maintenance. The additional hour will be used for testing and quality assurance.
Our Operations team has detected a hardware failure on a redundant component of our NetApp Storage device. Currently there is no impact to your services or the cluster performance. We are 100% online. In order to minimize any risks to our customers we are exercising extreme caution in repairing this component. Thus, we will be performing a complete shutdown of the cluster before proceeding with our work.
Service Impact:
OpenSRS Email Cluster B will be offline. Customers will have no access to their mailboxes via IMAP, POP and Webmail. Provisioning services and inbound/outbound mail will also be unavailable. Inbound mail will be remotely queued for delivery once the window is complete.
We are scheduling a 3 hour window as a precaution, however we do not expect to utilize the whole window.
Email Cluster B is Online
We resolved the issue which was causing provisioning delays and failures for OpenSRS Cluster B. All of your requests are now being processed. Our Ops team identified and addressed it. We continue our investigation, but will provide you with an incident summary by end of business day today.
All other services including mailbox access and mail flow were not affected.
Incident Summary (added 17:09 ET)
Our Network Operations Center (NOC) monitoring system received alerts for the provisioning system at starting at approximately 11:12 ET. Our testing and log evaluation confirmed that there were issues with provisioning related to high connections. We notified you at 11:56 ET. Provisioning requests via the API were unavailable. Changes via the MAC were working. All other email services were functioning.
Operations identified a potential cause of the load and temporarily blocked an IP. The issue did not resolve. After more digging they identified and blocked an additional IP. These IPs were temporarily blocked help us further isolate the source. The high connection load on the provisioning system reduced significantly. Further analysis helped us determine the problem to be database related. Our engineering team successfully resolved it. The temporarily blocked IPs were restored. Service was tested and fully restored by 13:41 ET.
Our DBA team will continue their investigation into the root cause.
This update is related to Incident 7375
Email Cluster B is Degraded
Our Operations team is working to identify the cause of high connections on OpenSRS Cluster B provisioning system. We addressed one issue, but the primary issue continues. At this time, provisioning changes via your API may not work. You can make changes directly in the MAC.
This update is related to Incident 7375
Email Cluster B is Degraded
Our Operations team isolated the issue which was causing provisioning change delays. We took action to address it and are closely monitoring the systems for further issues. We will keep you posted.
Again, all mailbox and mail flow is working well. You can make provisioning changes directly via the MAC.
This update is related to Incident 7375