| Save My Instances | What instance am I using? |
North America Instances
|
|
 Instance available |
 Performance issues |
 Service disruption |
 Informational message |
 Status not available |
| * Service Performance History is updated four times daily. Today's date may reflect partial day's data. |
| * In the Service Performance History matrix, each indicator will be yellow for issues that last longer than 10 minutes, and red for disruptions that last longer than 30 minutes. For incidents of less than 10 minutes, please click on the i for more information. |
Performance Degradation
Time: 05/03/13 09:38 PM UTC
Detail: On May 3 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA3 instance. There were four periods of impact during this incident:
From 08:35 UTC to 08:41 UTC
From 09:10 UTC to 09:15 UTC
From 09:41 UTC to 09:44 UTC
From 10:55 UTC to 11:01 UTC
During these times, customers on the NA3 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The salesforce.com Technology Team is investigating the root cause of the incident
Detail: On May 3 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA3 instance. There were four periods of impact during this incident:
From 08:35 UTC to 08:41 UTC
From 09:10 UTC to 09:15 UTC
From 09:41 UTC to 09:44 UTC
From 10:55 UTC to 11:01 UTC
During these times, customers on the NA3 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The salesforce.com Technology Team is investigating the root cause of the incident
Performance Degradation
Time: 05/06/13 11:00 PM UTC
Detail: On May 6 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA3 instance. The problem began at 21:40 UTC and was resolved by 21:43 UTC. During this time, customers on the NA3 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by resource contention on the NA3 database tier. The contention cleared and normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is continuing to work with our database vendor to more efficiently manage resources within the database tier
Detail: On May 6 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA3 instance. The problem began at 21:40 UTC and was resolved by 21:43 UTC. During this time, customers on the NA3 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by resource contention on the NA3 database tier. The contention cleared and normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is continuing to work with our database vendor to more efficiently manage resources within the database tier
Performance Degradation
Time: 04/25/13 11:22 PM UTC
Detail: On April 25 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA2 instance. The problem began at 22:14 UTC and was resolved by 22:16 UTC. During this time, customers on the NA2 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a hardware failure within the NA2 database tier. Application requests were automatically rebalanced around the failure and further customer impact was minimized. During the rebalancing process, database performance was impacted.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is working with our vendor to investigate and resolve the hardware failure
- Is investigating ways to increase system resiliency and further mitigate customer impact
Detail: On April 25 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA2 instance. The problem began at 22:14 UTC and was resolved by 22:16 UTC. During this time, customers on the NA2 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a hardware failure within the NA2 database tier. Application requests were automatically rebalanced around the failure and further customer impact was minimized. During the rebalancing process, database performance was impacted.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is working with our vendor to investigate and resolve the hardware failure
- Is investigating ways to increase system resiliency and further mitigate customer impact
Performance Degradation
Time: 05/21/13 03:27 PM UTC
Detail: The salesforce.com Technology Team has resolved the search performance degradation on the NA2 instance. The problem began at 13:00 UTC and was resolved as of 13:56 UTC. During this period customers on this instance may have experienced an inability to search for changed or newly created content but old content was fine throughout.
Root Cause: The problem was caused by a server hardware fault within the NA2 search tier. The faulty server has been taken offline for an emergency maintenance. The maintenance has completed and the search functionality has been fully restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is investigating more effective methods to ensure that the search application is more resilient to these kind of faults.
Detail: The salesforce.com Technology Team has resolved the search performance degradation on the NA2 instance. The problem began at 13:00 UTC and was resolved as of 13:56 UTC. During this period customers on this instance may have experienced an inability to search for changed or newly created content but old content was fine throughout.
Root Cause: The problem was caused by a server hardware fault within the NA2 search tier. The faulty server has been taken offline for an emergency maintenance. The maintenance has completed and the search functionality has been fully restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is investigating more effective methods to ensure that the search application is more resilient to these kind of faults.
Performance Degradation
Time: 05/13/13 11:45 PM UTC
Detail: On May 13 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA6 instance. There were nine periods of impact during this incident:
From 11:47 UTC to 15:25 UTC
From 15:44 UTC to 15:56 UTC
From 16:23 UTC to 16:41 UTC
From 17:03 UTC to 17:39 UTC
From 17:45 UTC to 17:54 UTC
From 18:26 UTC to 18:53 UTC
From 19:44 UTC to 20:02 UTC
From 20:25 UTC to 20:41 UTC
From 21:17 UTC to 21:30 UTC
During these times, customers on the NA6 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: After a routine change was applied to the NA6 database tier, anomalous behavior was observed with the database tier's balancing of requests across nodes in the cluster. This behavior caused intermittent periods of resource contention and in turn degraded performance of customer requests.
Several attempts were made to restore database stability without requiring customer downtime, but these efforts were not successful. On Tuesday May 14 at 7:00pm PDT, an emergency maintenance was executed to fully restart the NA6 database tier. After this maintenance activity, NA6 performance has been within normal parameters.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is improving database maintenance procedures to reduce the chance of introducing this anomalous behavior
- Is continuing to investigate the anomalous behavior with the database vendor
Detail: On May 13 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA6 instance. There were nine periods of impact during this incident:
From 11:47 UTC to 15:25 UTC
From 15:44 UTC to 15:56 UTC
From 16:23 UTC to 16:41 UTC
From 17:03 UTC to 17:39 UTC
From 17:45 UTC to 17:54 UTC
From 18:26 UTC to 18:53 UTC
From 19:44 UTC to 20:02 UTC
From 20:25 UTC to 20:41 UTC
From 21:17 UTC to 21:30 UTC
During these times, customers on the NA6 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: After a routine change was applied to the NA6 database tier, anomalous behavior was observed with the database tier's balancing of requests across nodes in the cluster. This behavior caused intermittent periods of resource contention and in turn degraded performance of customer requests.
Several attempts were made to restore database stability without requiring customer downtime, but these efforts were not successful. On Tuesday May 14 at 7:00pm PDT, an emergency maintenance was executed to fully restart the NA6 database tier. After this maintenance activity, NA6 performance has been within normal parameters.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is improving database maintenance procedures to reduce the chance of introducing this anomalous behavior
- Is continuing to investigate the anomalous behavior with the database vendor
Performance Degradation
Time: 05/14/13 11:59 PM UTC
Detail: On May 14 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA6 instance. There were multiple periods of impact during this incident:
From 13:33 UTC to 13:40 UTC
From 13:53 UTC to 14:28 UTC
From 14:53 UTC to 15:15 UTC
From 15:53 UTC to 15:59 UTC
From 16:02 UTC to 16:10 UTC
From 16:53 UTC to 17:09 UTC
From 18:03 UTC to 18:18 UTC
From 18:53 UTC to 18:58 UTC
From 19:03 UTC to 19:18 UTC
From 20:04 UTC to 20:20 UTC
From 21:07 UTC to 21:19 UTC
From 22:05 UTC to 22:09 UTC
From 23:05 UTC to 23:14 UTC
During these times, customers on the NA6 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: After a routine change was applied to the NA6 database tier, anomalous behavior was observed with the database tier's balancing of requests across nodes in the cluster. This behavior caused intermittent periods of resource contention and in turn degraded performance of customer requests.
Several attempts were made to restore database stability without requiring customer downtime, but these efforts were not successful. On Tuesday May 14 at 7:00pm PDT, an emergency maintenance was executed to fully restart the NA6 database tier. After this maintenance activity, NA6 performance has been within normal parameters.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is improving database maintenance procedures to reduce the chance of introducing this anomalous behavior
- Is continuing to investigate the anomalous behavior with the database vendor
Detail: On May 14 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA6 instance. There were multiple periods of impact during this incident:
From 13:33 UTC to 13:40 UTC
From 13:53 UTC to 14:28 UTC
From 14:53 UTC to 15:15 UTC
From 15:53 UTC to 15:59 UTC
From 16:02 UTC to 16:10 UTC
From 16:53 UTC to 17:09 UTC
From 18:03 UTC to 18:18 UTC
From 18:53 UTC to 18:58 UTC
From 19:03 UTC to 19:18 UTC
From 20:04 UTC to 20:20 UTC
From 21:07 UTC to 21:19 UTC
From 22:05 UTC to 22:09 UTC
From 23:05 UTC to 23:14 UTC
During these times, customers on the NA6 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: After a routine change was applied to the NA6 database tier, anomalous behavior was observed with the database tier's balancing of requests across nodes in the cluster. This behavior caused intermittent periods of resource contention and in turn degraded performance of customer requests.
Several attempts were made to restore database stability without requiring customer downtime, but these efforts were not successful. On Tuesday May 14 at 7:00pm PDT, an emergency maintenance was executed to fully restart the NA6 database tier. After this maintenance activity, NA6 performance has been within normal parameters.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is improving database maintenance procedures to reduce the chance of introducing this anomalous behavior
- Is continuing to investigate the anomalous behavior with the database vendor
Informational Message
Time: 04/28/13 12:00 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Performance Degradation
Time: 04/24/13 04:09 PM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation issue affecting NA7 and NA10 instances. The problem first began on April 24, 2013 at 14:17 UTC and was resolved at 14:19 UTC. The problem reoccurred at 14:47 UTC and was resolved at 14:49 UTC. During these two periods customers on the NA7 and N10 instances may have experienced slow performance or an inability to access the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a hardware error on the storage tier shared by the affected instances. A malfunctioning network component was identified and disabled. Redundant paths automatically engaged.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- is replacing the hardware and is working with the hardware vendor to identify the cause of the malfunction.
Detail: The salesforce.com Technology Team resolved a performance degradation issue affecting NA7 and NA10 instances. The problem first began on April 24, 2013 at 14:17 UTC and was resolved at 14:19 UTC. The problem reoccurred at 14:47 UTC and was resolved at 14:49 UTC. During these two periods customers on the NA7 and N10 instances may have experienced slow performance or an inability to access the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a hardware error on the storage tier shared by the affected instances. A malfunctioning network component was identified and disabled. Redundant paths automatically engaged.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- is replacing the hardware and is working with the hardware vendor to identify the cause of the malfunction.
Performance Degradation
Time: 04/24/13 04:09 PM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation issue affecting NA7 and NA10 instances. The problem first began on April 24, 2013 at 14:17 UTC and was resolved at 14:19 UTC. The problem reoccurred at 14:47 UTC and was resolved at 14:49 UTC. During these two periods customers on the NA7 and N10 instances may have experienced slow performance or an inability to access the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a hardware error on the storage tier shared by the affected instances. A malfunctioning network component was identified and disabled. Redundant paths automatically engaged.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- is replacing the hardware and is working with the hardware vendor to identify the cause of the malfunction.
Detail: The salesforce.com Technology Team resolved a performance degradation issue affecting NA7 and NA10 instances. The problem first began on April 24, 2013 at 14:17 UTC and was resolved at 14:19 UTC. The problem reoccurred at 14:47 UTC and was resolved at 14:49 UTC. During these two periods customers on the NA7 and N10 instances may have experienced slow performance or an inability to access the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a hardware error on the storage tier shared by the affected instances. A malfunctioning network component was identified and disabled. Redundant paths automatically engaged.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- is replacing the hardware and is working with the hardware vendor to identify the cause of the malfunction.
Performance Degradation
Time: 05/10/13 06:58 PM UTC
Detail: On May 10 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA11 instance. The problem began at 14:10 UTC and was resolved by 14:12 UTC. During this time, customers on the NA11 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The root cause of the incident is under investigation
Detail: On May 10 2013, the salesforce.com Technology Team observed a performance degradation issue affecting the NA11 instance. The problem began at 14:10 UTC and was resolved by 14:12 UTC. During this time, customers on the NA11 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The root cause of the incident is under investigation
Service Disruption
Time: 05/12/13 06:50 PM UTC
Detail: On May 12, 2013, the salesforce.com Technology Team encountered an issue while performing the announced planned maintenance on the NA11 instance. This issue resulted in a delay to the completion of the planned maintenance by one hour and forty-seven minutes, from 11:00 UTC to 12:47 UTC. We thank you for your continued business and patience.
Root Cause: The delay resulted from an unexpected fault within the storage tier. The root cause of the fault is, at this time, still under investigation.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is taking the following actions:
- Working to determine the cause of the storage fault
Detail: On May 12, 2013, the salesforce.com Technology Team encountered an issue while performing the announced planned maintenance on the NA11 instance. This issue resulted in a delay to the completion of the planned maintenance by one hour and forty-seven minutes, from 11:00 UTC to 12:47 UTC. We thank you for your continued business and patience.
Root Cause: The delay resulted from an unexpected fault within the storage tier. The root cause of the fault is, at this time, still under investigation.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is taking the following actions:
- Working to determine the cause of the storage fault
Informational Message
Time: 04/23/13 12:00 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the CS14 instance. The problem began on April 23, 2013 at 14:37 UTC and was resolved at 16:28 UTC. During this time a subset of customers on the CS14 instance may have experienced delays in Dashboard Refreshes, Web-To-Case, Web-To-Lead, and other asynchronous requests. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by an inefficient request within the database tier that impacted the processing of asynchronous requests for customers on the instance. The request was cleared and normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team:
- Is working to further improve the resiliency of asynchronous requests to such faults
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the CS14 instance. The problem began on April 23, 2013 at 14:37 UTC and was resolved at 16:28 UTC. During this time a subset of customers on the CS14 instance may have experienced delays in Dashboard Refreshes, Web-To-Case, Web-To-Lead, and other asynchronous requests. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by an inefficient request within the database tier that impacted the processing of asynchronous requests for customers on the instance. The request was cleared and normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team:
- Is working to further improve the resiliency of asynchronous requests to such faults
Informational Message
Time: 04/28/13 12:00 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Performance Degradation
Time: 05/19/13 02:10 PM UTC
Detail: On May 19 2013, the salesforce.com Technology Team observed a performance degradation affecting the AP1 instance. The problem began at 5:34 UTC and was resolved by 11:43 UTC. During this time, some customers may have experienced slow performance or intermittent errors. Thank you for your continued business and patience during this situation.
Root Cause: The problem was isolated to resource contention within the database tier.
Actions to Prevent future incidents:
The salesforce.com Technology Team -
-Is investigating the root cause of the resource contention with our salesforce.com Technology Teams
Detail: On May 19 2013, the salesforce.com Technology Team observed a performance degradation affecting the AP1 instance. The problem began at 5:34 UTC and was resolved by 11:43 UTC. During this time, some customers may have experienced slow performance or intermittent errors. Thank you for your continued business and patience during this situation.
Root Cause: The problem was isolated to resource contention within the database tier.
Actions to Prevent future incidents:
The salesforce.com Technology Team -
-Is investigating the root cause of the resource contention with our salesforce.com Technology Teams
Informational Message
Time: 04/28/13 12:00 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Performance Degradation
Time: 05/08/13 12:32 PM UTC
Detail: On 8 May 2013, the salesforce.com Technology team became aware of a performance degradation affecting the database tier for the NA8 instance. The problem began at 07:38 UTC and was resolved by 10:19 UTC. During this time, customers on the NA8 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was identified as an internal process which was having an unanticipated impact.
Detail: On 8 May 2013, the salesforce.com Technology team became aware of a performance degradation affecting the database tier for the NA8 instance. The problem began at 07:38 UTC and was resolved by 10:19 UTC. During this time, customers on the NA8 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was identified as an internal process which was having an unanticipated impact.
Informational Message
Time: 04/28/13 12:00 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Informational Message
Time: 04/28/13 12:00 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Performance Degradation
Time: 05/17/13 11:37 PM UTC
Detail: On May 17 2013, the salesforce.com Technology Team observed a performance degradation affecting API transactions on the NA14 instance. The problem began at 19:18 UTC and was resolved by 19:49 UTC. During this time, some customers' API transactions with the Salesforce application may have experienced slow performance or intermittent errors. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by resource exhaustion on the application servers used for API query caching. The exhaustion was addressed and normal API performance was restored.
Actions to Prevent Future Incidents –
The salesforce.com Technology Team –
- Is adding capacity and enhancing the system configuration, to better manage resources and prevent a similar exhaustion in the future
- Is making infrastructure improvements to more quickly detect and be resilient to this type of resource exhaustion
Detail: On May 17 2013, the salesforce.com Technology Team observed a performance degradation affecting API transactions on the NA14 instance. The problem began at 19:18 UTC and was resolved by 19:49 UTC. During this time, some customers' API transactions with the Salesforce application may have experienced slow performance or intermittent errors. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by resource exhaustion on the application servers used for API query caching. The exhaustion was addressed and normal API performance was restored.
Actions to Prevent Future Incidents –
The salesforce.com Technology Team –
- Is adding capacity and enhancing the system configuration, to better manage resources and prevent a similar exhaustion in the future
- Is making infrastructure improvements to more quickly detect and be resilient to this type of resource exhaustion
Service Disruption
Time: 05/04/13 03:20 PM UTC
Detail: The salesforce.com Technology Team investigated a service disruption affecting the CS9 instance. The problem began on May 4th, 2013 at 10:48 UTC and was resolved at 11:57 UTC.
During this time customers on the CS9 instance experienced an inability to access or use the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a connectivity issue within the CS9 database tier. This issue subsequently caused the database to fault and become unresponsive. The connectivity issue was resolved, the database was brought back online and normal Salesforce services were restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team:
- Is investigating the cause of the connectivity issue
- Is working with our vendor to investigate the database fault, as well as methods to increase database resiliency to connectivity issues
Detail: The salesforce.com Technology Team investigated a service disruption affecting the CS9 instance. The problem began on May 4th, 2013 at 10:48 UTC and was resolved at 11:57 UTC.
During this time customers on the CS9 instance experienced an inability to access or use the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by a connectivity issue within the CS9 database tier. This issue subsequently caused the database to fault and become unresponsive. The connectivity issue was resolved, the database was brought back online and normal Salesforce services were restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team:
- Is investigating the cause of the connectivity issue
- Is working with our vendor to investigate the database fault, as well as methods to increase database resiliency to connectivity issues
Performance Degradation
Time: 05/14/13 12:54 AM UTC
Detail: On 14 May 2013, the salesforce.com Technology team became aware of a performance degradation affecting the application tier for the CS9 instance. The problem began at 15:11 UTC and was resolved by 15:55 UTC. During this time, customers on the CS9 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was isolated to a resource contention on the application tier. The contention was cleared, allowing for normal performance to resume.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is reviewing processes which contributed to this resource contention for optimization.
Detail: On 14 May 2013, the salesforce.com Technology team became aware of a performance degradation affecting the application tier for the CS9 instance. The problem began at 15:11 UTC and was resolved by 15:55 UTC. During this time, customers on the CS9 instance may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was isolated to a resource contention on the application tier. The contention was cleared, allowing for normal performance to resume.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is reviewing processes which contributed to this resource contention for optimization.
Performance Degradation
Time: 04/22/13 10:04 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation issue affecting Search on the EU0 instance. The problem began on April 22, 2013 at 09:05 UTC and was resolved at 09:33 UTC. During this time customers on the EU0 instance may have experienced intermittent periods of where they were unable to search for newly created or updated records. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by resource contention on the EU0 Search tier. In response to the resource contention, a subset of Search queries were automatically routed to a secondary Search method which does not perform as well as the primary Search method. The resource contention was cleared and normal Search performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is investigating more effective methods to ensure that the Search application is more resilient to these kind of faults.
Detail: The salesforce.com Technology Team resolved a performance degradation issue affecting Search on the EU0 instance. The problem began on April 22, 2013 at 09:05 UTC and was resolved at 09:33 UTC. During this time customers on the EU0 instance may have experienced intermittent periods of where they were unable to search for newly created or updated records. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by resource contention on the EU0 Search tier. In response to the resource contention, a subset of Search queries were automatically routed to a secondary Search method which does not perform as well as the primary Search method. The resource contention was cleared and normal Search performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is investigating more effective methods to ensure that the Search application is more resilient to these kind of faults.
Performance Degradation
Time: 04/25/13 04:32 AM UTC
Detail: The salesforce.com Technology Team has resolved a performance degradation issue on the EU0 instance. The problem began on April 25, 2013 at 01:14 UTC and was resolved at 01:26 UTC. During this time, customers may have experienced slow performance or intermittent errors while accessing the application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by contention within the database tier. The affected nodes have been removed from cluster and normal performance has been restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology team –
- Is reviewing database configuration and make corrections to ensure resiliency
Detail: The salesforce.com Technology Team has resolved a performance degradation issue on the EU0 instance. The problem began on April 25, 2013 at 01:14 UTC and was resolved at 01:26 UTC. During this time, customers may have experienced slow performance or intermittent errors while accessing the application. Thank you for your continued business and patience during this situation.
Root Cause: The problem was caused by contention within the database tier. The affected nodes have been removed from cluster and normal performance has been restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology team –
- Is reviewing database configuration and make corrections to ensure resiliency
Performance Degradation
Time: 05/02/13 02:59 AM UTC
Detail: The salesforce.com Technology Team has resolved a performance degradation issue affecting the EU0 instance. The performance degradation began on May 2, 2013 at 00:14 UTC and was resolved by 00:19 UTC. During the timeframe, customers may have experienced slow performance or intermittent errors. Thank you for your continued business and patience during this situation.
Root Cause: During a routine database tier maintenance, an unexpected fault was found on a database node. The affected node have been removed from cluster and application requests were automatically rebalanced. Further customer impact was minimized.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is troubleshooting the database node.
- Is reviewing database configuration to investigate why this caused customer impact and make improvements to ensure resiliency
Detail: The salesforce.com Technology Team has resolved a performance degradation issue affecting the EU0 instance. The performance degradation began on May 2, 2013 at 00:14 UTC and was resolved by 00:19 UTC. During the timeframe, customers may have experienced slow performance or intermittent errors. Thank you for your continued business and patience during this situation.
Root Cause: During a routine database tier maintenance, an unexpected fault was found on a database node. The affected node have been removed from cluster and application requests were automatically rebalanced. Further customer impact was minimized.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is troubleshooting the database node.
- Is reviewing database configuration to investigate why this caused customer impact and make improvements to ensure resiliency
Service Disruption
Time: 05/21/13 11:17 AM UTC
Detail: The salesforce.com Technology Team has resolved a service disruption affecting the CS10 instance. The problem began on May 21st, 2013 at 10:08 UTC and was resolved at 10:42 UTC.
During this time customers on the CS10 instance experienced an inability to access or use the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: Root Cause: The problem was isolated to a fault within our third party database application.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is working with our vendor to mitigate further issues
Detail: The salesforce.com Technology Team has resolved a service disruption affecting the CS10 instance. The problem began on May 21st, 2013 at 10:08 UTC and was resolved at 10:42 UTC.
During this time customers on the CS10 instance experienced an inability to access or use the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: Root Cause: The problem was isolated to a fault within our third party database application.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is working with our vendor to mitigate further issues
Informational Message
Time: 04/28/13 12:00 AM UTC
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Detail: The salesforce.com Technology Team resolved a performance degradation affecting the NA8, NA9, EU1, CS7, CS8 and CS14 instances. The problem began on April 28, 2013 at 07:22 UTC and was resolved at 16:09 UTC. During this time customers may have experienced problems receiving emails from the affected instances and failures while accessing external services from the Salesforce.com application. Thank you for your continued business and patience during this situation.
Root Cause: On April 28th at approximately 12:00 AM PST, the salesforce.com Technology team performed a scheduled Load Balancer change that involved upgrading a subset of SSL certificates. This change went through the standard change management process and was executed within the planned maintenance window. While the intended certificate change succeeded as expected, there was an additional change, scheduled for future purposes, that was inadvertently executed on the same load balancers. The monitoring system quickly alerted the salesforce.com team and the unintended change was immediately rolled back. After the rollback, our monitoring system and verification with our support team indicated that full service had been restored.
The broader impact to the outbound proxy service was not immediately evident. A subsequent alert notified the salesforce.com Technology team of additional impact, and the team immediately began troubleshooting efforts. Through these efforts, an incorrect configuration was found on one of the load balancers that was routing outbound proxy traffic to a non-existent destination. Through our investigation, the root cause of the service disruption was identified as a bug in the rollback commands. The salesforce.com Technology team removed the incorrect configuration and the service was fully restored at 9:00 AM PST.
To prevent this from occurring again, the team has identified a number of preventive and mitigating steps, that include faster diagnosis, more expeditious internal and vendor triaging, and improving the review of change plans.
Performance Degradation
Time: 05/06/13 11:59 PM UTC
Detail: On May 6 2013, the salesforce.com Technology Team observed a performance degradation affecting the NA13 instance. The problem began at 15:25 UTC and impact initially cleared at 16:05 UTC. After this initial impact period, 3 additional brief impacts occurred at 16:15 UTC, 16:25 UTC, and 16:28 UTC. During these times, customers may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: A session within the database tier got into a faulted state, and was automatically quiesced. This caused the database to roll back the changes made by the session. During this roll back period, performance was degraded for a subset of customers on the NA13 instance. Once the roll back completed, normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is reviewing the monitoring process and terminated database request for improvements to prevent such impact in the future
- Is continuing to work with our database vendor to better manage resources within the database tier
Detail: On May 6 2013, the salesforce.com Technology Team observed a performance degradation affecting the NA13 instance. The problem began at 15:25 UTC and impact initially cleared at 16:05 UTC. After this initial impact period, 3 additional brief impacts occurred at 16:15 UTC, 16:25 UTC, and 16:28 UTC. During these times, customers may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: A session within the database tier got into a faulted state, and was automatically quiesced. This caused the database to roll back the changes made by the session. During this roll back period, performance was degraded for a subset of customers on the NA13 instance. Once the roll back completed, normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Is reviewing the monitoring process and terminated database request for improvements to prevent such impact in the future
- Is continuing to work with our database vendor to better manage resources within the database tier
Performance Degradation
Time: 05/07/13 06:04 PM UTC
Detail: On May 7 2013 the salesforce.com Technology team became aware of a performance degradation affecting the NA13 instance. The problem began at 14:14 UTC and was resolved by 15:47 UTC, with an additional impact period that began at 16:51 UTC to 16:52 UTC. During this period, customers may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The root cause of the incident is under investigation.
Detail: On May 7 2013 the salesforce.com Technology team became aware of a performance degradation affecting the NA13 instance. The problem began at 14:14 UTC and was resolved by 15:47 UTC, with an additional impact period that began at 16:51 UTC to 16:52 UTC. During this period, customers may have experienced slow performance or intermittent errors while accessing the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The root cause of the incident is under investigation.
Performance Degradation
Time: 05/10/13 10:03 PM UTC
Detail: The salesforce.com Technology Team observed a performance degradation issue affecting the NA13instance. The problem began on May 10, 2013 at 18:25 UTC and was resolved at 18:28 UTC. During this time customers on the NA13 instance may have experienced slow performance or intermittent errors while using the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The issue was isolated to resource contention on the NA13 database tier. The contention cleared and normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team –
- Is investigating more effective methods to manage database resource contention
Detail: The salesforce.com Technology Team observed a performance degradation issue affecting the NA13instance. The problem began on May 10, 2013 at 18:25 UTC and was resolved at 18:28 UTC. During this time customers on the NA13 instance may have experienced slow performance or intermittent errors while using the Salesforce application. Thank you for your continued business and patience during this situation.
Root Cause: The issue was isolated to resource contention on the NA13 database tier. The contention cleared and normal performance was restored.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team –
- Is investigating more effective methods to manage database resource contention
- 1-800-NO-SOFTWARE – 1-800-667-6389
- |
- Privacy Statement
- |
- Security Statement
- |
- Site Map
- USA
- |
- Australia
- |
- Brazil
- |
- China
- |
- Denmark
- |
- Finland
- |
- France
- |
- Germany
- |
- Italy
- |
- Japan
- |
- Korea
- |
- Latin America
- |
- Netherlands
- |
- Russia
- |
- Spain
- |
- Sweden
- |
- Taiwan
- |
- Thailand
- |
- UK
© Copyright 2000-2013 salesforce.com, inc. Web-based Customer Relationship Management (CRM) Software-as-a-Service (SaaS).
All rights reserved Various trademarks held by their respective owners.
Salesforce.com, inc. The Landmark @ One Market, Suite 300, San Francisco, CA, 94105, United States
General Enquiries: 415-901-7000 | Sales: 1-800-NO-SOFTWARE
All rights reserved Various trademarks held by their respective owners.
Salesforce.com, inc. The Landmark @ One Market, Suite 300, San Francisco, CA, 94105, United States
General Enquiries: 415-901-7000 | Sales: 1-800-NO-SOFTWARE