Salesforce.com System Status and Alerts http://trust.salesforce.com Real-time updates from trust.salesforce.com on alerts, availability and system health Sun, 28 Apr 2013 00:00:00 GMT 2013-04-28T00:00:00Z Alert: [ EU1 NA8 NA9 CS7 CS8 CS14 ] The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation. http://trust.salesforce.com/trust/status On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored. The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST. To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans. Sun, 28 Apr 2013 00:00:00 GMT http://trust.salesforce.com/trust/status 2013-04-28T00:00:00Z Alert: [ EU1 ] The salesforce.com Technology Team has resolved a service disruption issue impacting the EU1 instance. The issue began at 12:05 UTC and was resolved by 12:08 UTC. During this time customers on the EU1 instance may have experienced an inability to access the Salesforce application. Thank you for your continued business and patience during this situation. http://trust.salesforce.com/trust/status Root Cause: This issue was caused by an unexpected error while carrying out a routine background database process. Actions to Prevent future incidents: The salesforce.com Technology Team - - Is working to further automate and enhance our database processes to mitigate the risk of similar errors Mon, 15 Apr 2013 16:50:00 GMT http://trust.salesforce.com/trust/status 2013-04-15T16:50:00Z Alert: [ EU1 ] The salesforce.com Technology Team resolved a service disruption affecting the EU1 instance. The Problem began on April 7, 2013 at 17:20 UTC and was resolved at 17:57 UTC. During this time customers on the EU1 instance experienced slow performance or an inability to access the Salesforce application. Thank you for your continued business and patience during this situation. http://trust.salesforce.com/trust/status The problem was identified as a hardware fault within the database tier, the effected database node was isolated and service was restored. Actions to Prevent Future Incidents: The salesforce.com Technology Team: -Is working with our vendor to further improve resiliency and mitigate the impact of such failures. Sun, 07 Apr 2013 00:00:00 GMT http://trust.salesforce.com/trust/status 2013-04-07T00:00:00Z Alert: [ EU1 ] On March 21 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the EU1 instance. The problem began at 10:30 UTC and impact cleared at 10:39 UTC. The problem re-occurred at 13:29 UTC and impact cleared at 18:14 UTC. During these times, customers on the EU1 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation. http://trust.salesforce.com/trust/status The problem was caused by a memory management fault within the EU1 application tier. Impact subsided as traffic volume dropped at the end of the business day. Actions to Prevent Future Incidents: The salesforce.com Technology Team - - Is performing changes to the EU1 application tier to better manage system memory - Is planning hardware upgrades for the EU1 application tier to increase the memory capacity Thu, 21 Mar 2013 23:15:00 GMT http://trust.salesforce.com/trust/status 2013-03-21T23:15:00Z Alert: [ EU1 ] On March 19 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the EU1 instance. The problem began at 14:35 UTC and was resolved by 17:00 UTC. During this time, customers on the EU1 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation. http://trust.salesforce.com/trust/status The problem was caused by a memory management fault within the EU1 application tier. Impact subsided as traffic volume dropped at the end of the business day. Actions to Prevent Future Incidents: The salesforce.com Technology Team - - Is performing changes to the EU1 application tier to better manage system memory - Is planning hardware upgrades for the EU1 application tier to increase the memory capacity Tue, 19 Mar 2013 23:24:00 GMT http://trust.salesforce.com/trust/status 2013-03-19T23:24:00Z