We apologize for the inconvenience.

Service Impact:
OpenSRS Email Services Cluster A will be affected briefly during the 3-hour window.

Impact for Resellers:
The Mail Administration Center (MAC) will be temporarily unavailable for under 30 minutes within the window.

Impact for End-users:
The service interruption is expected to be less than an hour. It may be measured in minutes or seconds in some cases. During this window, customers may not have access to their mailboxes via IMAP, POP or Webmail. Inbound mail will be temporarily remotely queued for delivery.

Some customers may experience Webmail rendering issues in some browsers. The temporary workaround is to have users clear their browser cache. We are investigating the javascript discrepancy and we will provide further updates within this post when available.

(IMAP and POP users are unaffected. Inbound/Outbound mailflow is working well.)

Update Thursday, November 4, 2009 16:10 UTC):
Our Development and Operations team identified a bug and rolled back the Webmail code that was causing this issue.

We will be updating the version of IOS on a number of firewalls, routers and load balancers. As well, we will be upgrading the power supplies in one of our routers. All of these devices are deployed in redundant pairs so in each case the backup unit will be upgraded first followed by the redundant backup unit. This will ensure that your services experience only a brief 5-minute service disruption within the 3-hour window.

Service Impact:
OpenSRS Email Services Cluster A will be available during the 3-hour window except for a brief 5-minute period. During that 5-minute window, the service impact will be:

Impact for Resellers:
The Mail Administration Center (MAC) will be temporarily unavailable.

Impact for End-users:
During this brief time, customers may not have access to their mail via IMAP, POP or Webmail. Inbound mail will be temporarily remotely queued for delivery.

The provisioning queue for Cluster A has cleared. All new changes via the API (Application) or the Mail Administration Center (MAC) are now processing as per normal.

Inbound/Outbound mail flow and mailbox access were not affected by this.

Incident Summary: (added at 17:05ET)
At 07:58 ET we began noticing unusual behaviour with the provisioning system. There were corresponding alerts on our Network Operations Center (NOC) monitoring. Our Operations team began investigating to determine the scope and cause. Application hosts were restarted and this did not resolve the issue. We advised you of a provisioning delay at 08:54ET. All changes via the API were being queued for approximately 45 minutes. There were two types of errors noted: pending and failing. Both types of requests were being queued.

Our Development team identified an issue with the Oracle libraries that provide connectivity to the provisioning database. The back-end hosts were restarted and this resolved the issue. There were approximately 600 provisioning requests queued. We fixed the errored provisioning requests. The provisioning queue cleared. Network Operations Center (NOC) and Technical Support staff fully tested and confirmed that corresponding alerts had stopped on our monitoring system and that all new provisioning requests were successful. The service was fully restored by 09:45ET and we posted it online.

You can make provisioning changes via the Mail Administration Center (MAC) and API. Please note that there are still delays with changes via the API.

All other Email Cluster A services are working well.

Resellers can make provisioning changes such as password changes and creating new accounts via the Mail Administration Center (MAC).

All other services for Email including mailflow (inbound and outbound) and mailbox access are working well.