Salesforce.com System Status and Alerts

Alert: [cs1] The salesforce.com Technology Team resolved a performance degradation affecting the CS0, CS1, CS3, and CS12 instances. The problem began on May 7, 2012 at 07:00 UTC and was resolved by 21:07 UTC. The performance degradation was limited to the Change Set setup page within customer's respective Sandbox instance. During this time, customers attempting to access this page would have experienced intermittent errors or page delays.

Tue, 08 May 2012 01:25:00 GMT

The issue was caused by a recent change that exposed an unexpected fault in the way the application tier handles cross instance communication through the internal network tier. This fault caused Change Set requests, from the Sandbox organization to the respective Production organization, to intermittently fail due to connection timeouts.
Actions to Prevent Future Incidents: The salesforce.com Technology Team - - Has implemented changes to the network tier to prevent these Sandbox requests from timing out - Is investigating improvements to make the Salesforce application manage the impacted request more effectively

Alert: [cs1] The salesforce.com Technology Team has resolved a performance degradation issue affecting the database tier on the CS1 instance. The problem began on April 26, 2012 at 00:08 UTC and was resolved by 00:51 UTC April 26, 2012.During this time, customers on the CS1 instance may have experienced slow response times. We apologize for any inconvenience this may have caused you.

Thu, 26 Apr 2012 04:08:00 GMT

Root cause: The problem was caused by a lock contention on the database tier during a routine maintenance. Actions to prevent future incidents: The salesforce.com Technology Team: - is reviewing the procedures to manage maintenance on database nodes to ensure a less impactful shutdown of database nodes. is reviewing additional monitoring regarding lock contention issues.

Status Update: [cs1] CS1 Performance Degradation - Resolved

Thu, 26 Apr 2012 00:59:00 GMT

The Salesforce.com Technology Team has resolved a performance degradation affecting CS1 instance. Please check the status of Trust frequently for updates regarding the issue. We apologize for any inconvenience this may be causing you.

Status Update: [cs1] CS1 Performance Degradation

Thu, 26 Apr 2012 00:40:00 GMT

The Salesforce.com Technology Team is working to isolate a performance degradation affecting CS1 instance. Please check the status of Trust frequently for updates regarding the issue. We apologize for any inconvenience this may be causing you.

Alert: [cs1] On February 24, 2012 the salesforce.com Technology Team resolved a performance degradation impacting the CS1 instance. The problem began at 7:13:00 UTC and was resolved by 14:15:00 UTC. During this time, some customers may have experienced an inability to run Salesforce Object Query Language (SOQL) queries that involve aliases. We apologize for any inconvenience this may have caused.

Fri, 24 Feb 2012 09:05:00 GMT

The problem was caused by an unexpected fault in the application tier. The fault was introduced during a routine application update, which then caused SOQL queries involving aliases to return an error despite being valid queries. The update was rolled back after the issue was discovered, which removed the fault. Actions to Prevent Future Incidents: The salesforce.com Technology Team - - has identified and corrected the fault in the application update. - is updating testing procedures to better identify these types of issues.

Alert: [cs1] On February 24, 2012 the salesforce.com Technology Team resolved a performance degradation impacting the CS1 instance. The problem began at 7:13:00 UTC and was resolved by 14:15:00 UTC. During this time, some customers may have experienced an inability to run Salesforce Object Query Language (SOQL) queries that involve aliases. We apologize for any inconvenience this may have caused.

Thu, 23 Feb 2012 09:05:00 GMT

Status Update: [cs1] Sandbox CS1 salesforce.com Scheduled Upgrade Complete

Sat, 11 Feb 2012 19:42:00 GMT

All upgrade activities for the CS1 instance are now complete and all features for the Spring '12 release have now been made available for your organization.

Status Update: [cs1] Sandbox CS1 salesforce.com Scheduled Upgrade ~ Update

Sat, 11 Feb 2012 18:26:00 GMT

The Spring '12 release is now live for the CS1 instance. The CS1 should be generally available as we continue to perform upgrade activities including feature enablement. We will update Trust once these activities are complete.

Status Update: [cs1] CS1 Scheduled Maintenance Completed

Sun, 22 Jan 2012 06:13:00 GMT

The CS1 scheduled maintenance has successfully completed. Users will now be able to access the salesforce.com service.

Alert: [cs1] On January 18th, 2012 beginning at 08.30 UTC through 13.44 UTC, the salesforce.com Technology team investigated and resolved a performance degradation issue affecting dashboard users on CS1. During this time a small subset customers may have experienced a delay with dashboard functionality on CS1. We apologize for any inconvenience this may have caused you.

Wed, 18 Jan 2012 15:34:00 GMT

An investigation has begun to establish a root cause to the delay with dashboard functionality. Once this investigation has been completed trust.salesforce.com will be updated.

Status Update: [cs1] CS1 - Performance Degradation - Resolved

Wed, 18 Jan 2012 13:53:00 GMT

The salesforce.com Technology Team has resolved the performance degradation issue on the CS1 instance affecting Dashboard users. The problem began at 08.29 UTC and was resolved as of 13.44 UTC. We apologize for any inconvenience this may have caused you.

Status Update: [cs1] CS1 - Performance Degradation

Wed, 18 Jan 2012 13:31:00 GMT

The salesforce.com Technology Team is working to isolate a performance degradation for Dashboard users on the CS1 instance. Please check the status of Trust frequently for updates regarding this issue. We apologize for any inconvenience this may be causing you.

Alert: [cs1] On November 30, 2011 the salesforce.com Technology Team resolved a performance degradation issue on the NA1, NA2, NA5, NA6, AP0, CS0, CS1, CS3 and CS12 instances. The problem began at 15:47 UTC and was resolved as of 16:18 UTC. For the duration of the event, customers may have experienced intermittent inability to access salesforce.com services. We apologize for any inconvenience this may have caused you.

Thu, 01 Dec 2011 07:00:00 GMT

The root cause of the problem was isolated to a configuration issue on our network load balancer. The configuration exposed the device to a vendor bug that resulted in heavy load on its CPU and it stopped accepting new requests. Our service provider detected the problem and automatically routed the traffic through another active network device. Our network architecture which is designed to be resilient for most failure scenarios experienced a double failure. The automatic failover that was performed resulted in excessive load on the network firewalls. The salesforce.com Technology Team had to intervene to restore normal connectivity. During this time, a subset of incoming network requests may have failed, and the subsequent retries may have succeeded. Actions to prevent future incidents: The salesforce.com Technology Team: - has corrected the configuration issue on our network load balancers. - is working with our network vendor to establish better collaboration and testing of device configurations. - will work to implement greater level of monitoring around our network devices.

Alert: [cs1] On November 16th, 2011 starting at 05:14 UTC, the salesforce.com Technology Team became aware of a performance degradation affecting customers on the NA1/NA2/NA5/NA6/AP0/CS0/CS1/CS3/CS12 instances. The issue was cleared at 05:52 UTC. During this time, customers may have experienced an inability to use Sites, intermittent errors when using Single Sign-On, and slow page performance. We apologize for any inconvenience this may have caused you.

Tue, 15 Nov 2011 20:32:00 GMT

The problem is believed to be a flaw in vendor software on our shared networking infrastructure. The affected services were restarted to re-establish normal operations.

Actions to Prevent Future Incidents: - Continuing investigation to be sure we have identified the cause of today's incident - Patches for the vendor software involved are being prepared for production deployment - Additional review of the incident to identify other related infrastructure issues that need to be addressed

Alert: [cs1] The salesforce.com Technology Team is aware of a performance degradation issue that impacted the AP0, NA1, NA2, NA5, NA6, CS0, CS1, CS3 and CS12 instances. The problem began on October 31, 2011 at 19:43 UTC and was resolved by 20:25 UTC. Customers on these instances may have experienced slower than normal performance, and a small subset of customers may have experienced intermittent ability to log into the Salesforce application. We apologize for any inconvenience this may have caused you.

Mon, 31 Oct 2011 19:25:00 GMT

The root cause of this incident was isolated to an issue in our network tier that caused a significant increase in traffic utilization. This increase resulted in contention of resources that degraded performance of inbound customer requests. Actions to prevent future incidents: The salesforce.com Technology Team: - Has implemented a work around in our network tier to mitigate this should it happen again. - Is investigating ways to better detect such increases in utilization before it becomes a problem.

Status Update: [cs1] Winter 12 CS1 Release ~ Complete

Sat, 15 Oct 2011 18:52:00 GMT

The upgrade activities for the Sandbox instance are now complete.

Status Update: [cs1] Winter 12 CS1 Release ~ Update

Sat, 15 Oct 2011 17:50:00 GMT

The Winter '12 release is now live for the CS1 instance. The CS1 should be generally available as we continue to perform upgrade activities. We will update Trust once these activities are complete. NOTE: Refreshing your sandbox may change the instance on which your sandbox resides. For more details please reference the solution "Why does my instance change whenever I refresh my sandbox?" in the Help & Training.

Status Update: [cs1] Winter 12 CS1 & CS2 Release ~ Start

Sat, 15 Oct 2011 17:00:01 GMT

The scheduled upgrade for {CS1 & CS2} is about to begin. During this period the salesforce.com service is expected to be generally available. Your organization should expect to experience a disruption of up to five (5) minutes during this timeframe. We appreciate your patience while we update this instance to Winter '12. NOTE: Refreshing your sandbox may change the instance on which your sandbox resides. For more details please reference the solution "Why does my instance change whenever I refresh my sandbox?" in the Help & Training.