Email Cluster A Status Archive
Email Cluster A is Online
OpenSRS Email Service is fully available however statistics provided in the MAC control panel are not accurate. We are aware of the issue and are working to correct it. We will update Status when the data is corrected.
We apologize for the inconvenience.
Email Cluster A is Online
The OpenSRS Email Services maintenance is complete.
Email Cluster A is In Maintenance
There is a 3-hour maintenance for OpenSRS Email Services - Cluster A starting now (04:00 UTC). This time will be used to update the version of firmware (IOS) on a number of switches.
Service Impact:
OpenSRS Email Services Cluster A will be affected briefly during the 3-hour window.
Impact for Resellers:
The Mail Administration Center (MAC) will be temporarily unavailable for under 30 minutes within the window.
Impact for End-users:
The service interruption is expected to be less than an hour. It may be measured in minutes or seconds in some cases. During this window, customers may not have access to their mailboxes via IMAP, POP or Webmail. Inbound mail will be temporarily remotely queued for delivery.
Email Cluster A is Online
All OpenSRS Email services on Cluster A are fully available.
Some customers may experience Webmail rendering issues in some browsers. The temporary workaround is to have users clear their browser cache. We are investigating the javascript discrepancy and we will provide further updates within this post when available.
(IMAP and POP users are unaffected. Inbound/Outbound mailflow is working well.)
Update Thursday, November 4, 2009 16:10 UTC):
Our Development and Operations team identified a bug and rolled back the Webmail code that was causing this issue.
This update is related to Incident 9064
Email Cluster A is Online
The OpenSRS Email Services - Cluster A maintenance is complete. Our technical teams finished their work ahead of schedule and confirm all services are online.
Email Cluster A is In Maintenance
There is a three hour network maintenance window at our data center starting now (03:00 UTC).
We will be updating the version of IOS on a number of firewalls, routers and load balancers. As well, we will be upgrading the power supplies in one of our routers. All of these devices are deployed in redundant pairs so in each case the backup unit will be upgraded first followed by the redundant backup unit. This will ensure that your services experience only a brief 5-minute service disruption within the 3-hour window.
Service Impact:
OpenSRS Email Services Cluster A will be available during the 3-hour window except for a brief 5-minute period. During that 5-minute window, the service impact will be:
Impact for Resellers:
The Mail Administration Center (MAC) will be temporarily unavailable.
Impact for End-users:
During this brief time, customers may not have access to their mail via IMAP, POP or Webmail. Inbound mail will be temporarily remotely queued for delivery.
Email Cluster A is Online
OpenSRS Email Cluster A is fully online.
The provisioning queue for Cluster A has cleared. All new changes via the API (Application) or the Mail Administration Center (MAC) are now processing as per normal.
Inbound/Outbound mail flow and mailbox access were not affected by this.
Incident Summary: (added at 17:05ET)
At 07:58 ET we began noticing unusual behaviour with the provisioning system. There were corresponding alerts on our Network Operations Center (NOC) monitoring. Our Operations team began investigating to determine the scope and cause. Application hosts were restarted and this did not resolve the issue. We advised you of a provisioning delay at 08:54ET. All changes via the API were being queued for approximately 45 minutes. There were two types of errors noted: pending and failing. Both types of requests were being queued.
Our Development team identified an issue with the Oracle libraries that provide connectivity to the provisioning database. The back-end hosts were restarted and this resolved the issue. There were approximately 600 provisioning requests queued. We fixed the errored provisioning requests. The provisioning queue cleared. Network Operations Center (NOC) and Technical Support staff fully tested and confirmed that corresponding alerts had stopped on our monitoring system and that all new provisioning requests were successful. The service was fully restored by 09:45ET and we posted it online.
This update is related to Incident 7521
Email Cluster A is Degraded
Our Development and Operations team identified the provisioning problem to be related to behaviour of a particular provisioning device. They restarted it. Your provisioning changes via the API, which were previously queued, are being processed. We are closely monitoring all the provisioning systems. There is a queue of requests which need to be processed.
You can make provisioning changes via the Mail Administration Center (MAC) and API. Please note that there are still delays with changes via the API.
All other Email Cluster A services are working well.
This update is related to Incident 7521
Email Cluster A is Degraded
Our Development team is looking into the issue. Provisioning via the API was working after the maintenance window earlier this morning. Your provisioning requests sent in the last 30 minutes have been queued and are delayed.
Resellers can make provisioning changes such as password changes and creating new accounts via the Mail Administration Center (MAC).
This update is related to Incident 7521
Email Cluster A is Degraded
OpenSRS Cluster A provisioning changes via the APP have been delayed for the last 20 minutes. You can make changes via the Mail Adminstration Center (MAC) Our Operations team is investigating.
All other services for Email including mailflow (inbound and outbound) and mailbox access are working well.
This update is related to Incident 7521