http://trust.salesforce.com Real-time updates from trust.salesforce.com on alerts, availability and system health Wed, 08 May 2013 12:32:00 GMT 2013-05-08T12:32:00Z http://trust.salesforce.com/trust/status The salesforce.com Technology Team has resolved the performance degradation issue affecting the NA8 instance. The problem began at 14:58 UTC and was resolved as of 15:03 UTC. During this period, customers may have experienced slowness or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation. Mon, 08 Apr 2013 15:15:00 GMT http://trust.salesforce.com/trust/status 2013-04-08T15:15:00Z http://trust.salesforce.com/trust/status The salesforce.com Technology Team is working to isolate a performance degradation issue on the NA8 instance. Mon, 08 Apr 2013 15:05:00 GMT http://trust.salesforce.com/trust/status 2013-04-08T15:05:00Z http://trust.salesforce.com/trust/status The problem was identified as an internal process which was having an unanticipated impact. Wed, 08 May 2013 12:32:00 GMT http://trust.salesforce.com/trust/status 2013-05-08T12:32:00Z http://trust.salesforce.com/trust/status On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored. The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST. To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans. Sun, 28 Apr 2013 00:00:00 GMT http://trust.salesforce.com/trust/status 2013-04-28T00:00:00Z http://trust.salesforce.com/trust/status The issue was isolated to a network device error within the application tier. This disrupted the way the application handled traffic to the database tier, which in turn caused high cache load. This high cache load resulted in an increased time to process requests to the Salesforce application. The resulting error was corrected, allowing for normal performance to resume. Actions to Prevent Future Incidents: The salesforce.com Technology Team: -Is investigating corrections to the infrastructure to detect and prevent this issue from happening in the future. Mon, 08 Apr 2013 00:00:00 GMT http://trust.salesforce.com/trust/status 2013-04-08T00:00:00Z