Salesforce.com System Status and Alerts

Alert: [na9] On March 15, 2012 the salesforce.com Technology Team resolved a service disruption affecting the NA9 instance. The problem began at 02:36 UTC and was resolved by 02:59 UTC, to further eliminate the remaining risk of service impact, an emergent maintenance was carried out between 03:08 UTC and 03:10 UTC. During the 2 aforementioned periods, customers may have experienced an inability to access the Salesforce service. We apologize for any inconvenience this may have caused.

Thu, 15 Mar 2012 05:00:00 GMT

The problem was caused by a database locking contention event. Intensive lock contentions were triggered unexpectedly while a routine database maintenance procedure was being conducted. A manual intervention was initiated to eliminate the database contention to restore the service.
Actions to Prevent Future Incidents: The salesforce.com Technology Team - - Is investigating the cause of the contention with the software vendor - Is studying improvements to the routine procedure to reduce the risks of impact further

Status Update: [na9] NA9 Service Disruption - Resolved

Thu, 15 Mar 2012 03:19:00 GMT

The salesforce.com Technology Team has resolved the service disruption issue affecting the NA9 instance. We continue to closely monitor further. We apologize for any inconvenience this may have caused you.

Status Update: [na9] NA9 Service Disruption -Update #1

Thu, 15 Mar 2012 03:14:00 GMT

The salesforce.com Technology Team continues to work on isolating the service disruption issue on the NA9 instance. Please check the status of Trust frequently for updates regarding this issue. We apologize for any inconvenience this may be causing you.

Status Update: [na9] NA9 Service Disruption

Thu, 15 Mar 2012 02:47:00 GMT

The salesforce.com Technology Team is working to isolate a service disruption issue on the NA9 instance. Please check the status of Trust frequently for updates regarding this issue. We apologize for any inconvenience this may be causing you.

Alert: [na9] On March 6, 2012 the salesforce.com Technology Team resolved a performance degradation affecting the NA9 instance. The problem began at 03:07 UTC and was resolved by 03:23 UTC. During this time, a small subset of customers may have experienced an intermittent inability to access the Salesforce service. We apologize for any inconvenience this may have caused.

Tue, 06 Mar 2012 05:00:00 GMT

The problem was caused by a network device failover event. A subset of traffic using the affected network pathway was impacted while the standby device became active. A manual failover was initiated when the primary device became unhealthy. Actions to Prevent Future Incidents: The salesforce.com Technology Team - - Is investigating the cause of the failover with the hardware vendor - Is reviewing improvements to our network architecture to reduce the impact of failovers

Alert: [na9] On February 24, 2012 the salesforce.com Technology Team resolved a performance degradation impacting the NA9 instance. The problem began at 3:33:00 UTC and was resolved by 13:40:00 UTC. During this time, some customers may have experienced an inability to run Salesforce Object Query Language (SOQL) queries that involve aliases. We apologize for any inconvenience this may have caused.

Fri, 24 Feb 2012 09:05:00 GMT

The problem was caused by an unexpected fault in the application tier. The fault was introduced during a routine application update, which then caused SOQL queries involving aliases to return an error despite being valid queries. The update was rolled back after the issue was discovered, which removed the fault. Actions to Prevent Future Incidents: The salesforce.com Technology Team - - has identified and corrected the fault in the application update. - is updating testing procedures to better identify these types of issues.

Alert: [na9] On February 24, 2012 the salesforce.com Technology Team resolved a performance degradation impacting the NA9 instance. The problem began at 3:33:00 UTC and was resolved by 13:40:00 UTC. During this time, some customers may have experienced an inability to run Salesforce Object Query Language (SOQL) queries that involve aliases. We apologize for any inconvenience this may have caused.

Thu, 23 Feb 2012 09:05:00 GMT

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Fri, 10 Feb 2012 08:01:00 GMT

The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue. Actions to Prevent Future Incidents: The salesforce.com Technology Team is - - Improving testing procedures to better identify and address issues like this in the future

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Thu, 09 Feb 2012 08:01:00 GMT

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Wed, 08 Feb 2012 08:01:00 GMT

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Tue, 07 Feb 2012 08:01:00 GMT

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Mon, 06 Feb 2012 08:01:00 GMT

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Sun, 05 Feb 2012 08:01:00 GMT

Status Update: [na9] Spring 12 NA9 Release ~ Complete

Sat, 04 Feb 2012 09:51:00 GMT

All upgrade activities for the NA9 instance are now complete and all features for the Spring '12 release have now been made available for your organization.

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Sat, 04 Feb 2012 08:01:00 GMT

Alert: [na9] On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.

Fri, 03 Feb 2012 08:01:00 GMT

Alert: [na9] On January 23, 2012 beginning at 19:10 UTC through 19:25 UTC, the salesforce.com Technology Team investigated and resolved a performance degradation affecting the NA8, NA9, EU1, CS7, and CS8 instances. During this time a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.

Mon, 23 Jan 2012 23:27:00 GMT

The problem was caused by a fault on a network load balancer device. During the incident period, requests handled by this load balancer may not have completed successfully. Other load balancers in the environment were not affected. The root cause of the load balancer fault is under investigation with the vendor.

Actions to Prevent Future Incidents: The salesforce.com Technology Team is - -Continuing vendor investigation into the load balancer fault

Alert: [na9] On January 17th, 2012 beginning at 18:30 UTC through 18:54 UTC, the salesforce.com Technology team investigated and resolved a performance degradation affecting the NA8/NA9/EU1/CS7/CS8 instances. During this time, a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.

Wed, 18 Jan 2012 00:38:00 GMT

The problem was caused by a low-level device failure in our network tier. This failure was transient in nature and did not trigger an automatic re-route of traffic. The issue was resolved by manually isolating the device and re-routing all traffic to the redundant system. Actions to Prevent Future Incidents: The salesforce.com Technology Team - -has scheduled the replacement of the failed component with the vendor -is improving monitoring and automatic handling of failures of this specific nature

Alert: [na9] On November 16th, 2011 starting at 05:21 UTC, the salesforce.com Technology Team became aware of a performance degradation affecting customers on the NA7/NA8/NA9/NA10/NA11/NA12/EU1/CS7/CS8/CS9/CS10/CS11 instances. The issue was cleared at 05:40 UTC. During this time, customers may have experienced an inability to use Sites, intermittent errors when using Single Sign-On, and slow page performance. We apologize for any inconvenience this may have caused you.

Tue, 15 Nov 2011 21:34:00 GMT

The problem is believed to be a flaw in vendor software on our shared networking infrastructure. The affected services were restarted to re-establish normal operations. Actions to Prevent Future Incidents: - Continuing investigation to be sure we have identified the cause of today's incident - Patches for the vendor software involved are being prepared for production deployment - Additional review of the incident to identify other related infrastructure issues that need to be addressed

Status Update: [na9] NA9 - Scheduled Upgrade Complete

Sat, 08 Oct 2011 09:48:00 GMT

The upgrade activities for the NA9 instance are now complete.