http://trust.salesforce.com Real-time updates from trust.salesforce.com on alerts, availability and system health Sun, 28 Apr 2013 00:00:00 GMT 2013-04-28T00:00:00Z http://trust.salesforce.com/trust/status On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored. The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST. To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans. Sun, 28 Apr 2013 00:00:00 GMT http://trust.salesforce.com/trust/status 2013-04-28T00:00:00Z