| Updated 2/12/2012 2:40 am PST |
 |
System Status
|
 |
|
|
|
| 02/12/12 |
02/11/12 |
02/10/12 |
02/09/12 |
02/08/12 |
02/07/12 |
02/06/12 |
02/05/12 |
02/04/12 |
02/03/12 |
02/02/12 |
02/01/12 |
01/31/12 |
01/30/12 |
01/29/12 |
01/28/12 |
01/27/12 |
01/26/12 |
01/25/12 |
01/24/12 |
01/23/12 |
01/22/12 |
01/21/12 |
01/20/12 |
01/19/12 |
01/18/12 |
01/17/12 |
01/16/12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75,873,194 |
274,094,846 |
651,537,037 |
703,652,444 |
707,741,476 |
722,055,164 |
684,152,397 |
247,384,962 |
276,664,982 |
647,491,412 |
692,294,350 |
700,839,929 |
729,036,464 |
688,091,761 |
241,174,007 |
270,424,433 |
635,021,658 |
652,766,994 |
661,951,850 |
666,179,361 |
643,819,366 |
222,943,904 |
267,720,548 |
634,813,437 |
675,464,507 |
678,500,169 |
660,607,056 |
566,209,412 |
| 0.204 |
0.215 |
0.252 |
0.245 |
0.243 |
0.235 |
0.235 |
0.195 |
0.212 |
0.239 |
0.237 |
0.252 |
0.242 |
0.239 |
0.198 |
0.205 |
0.226 |
0.222 |
0.230 |
0.238 |
0.242 |
0.193 |
0.204 |
0.245 |
0.228 |
0.251 |
0.242 |
0.237 |
|
|
| |
| |
Informational Report
Informational Message
Detail: No incidents to report.
|
|
| |
| |
| |
What System Are You Using?
The instance your organization uses is indicated in the browser's Address bar, shown
highlighted below.
|
|
| |
Service Performance History is updated four times daily. Today's date may reflect partial day's data.
Salesforce.com service transaction speed. Your on-site page load times may vary based on your
bandwidth and network latency. |
|
|
In the Service Performance History matrix, each indicator will be blue with a white "i"
for issues that last longer than 10 minutes,
and red for disruptions that last longer than 30 minutes.
For incidents of less than 10 minutes, please click on the i for more information.
|
| |
| |
Incident Report
Informational Message
Time: 1/23/12 9:40 pm PST
Detail: The salesforce.com Technology Team resolved a performance degradation issue affecting the search functionality on the AP0 instance. The problem began on January 24, 2012 at 03:07 hrs UTC and was resolved by 05:35 hrs UTC. During the incident, customers may have experienced delays and/or inability to search newly added data. We sincerely apologize for any inconvenience this may have caused you.
Root cause: The root cause of this incident was isolated to a suboptimal search indexing operation that resulted in high free space utilization on the temporary storage. The search application actively uses this temporary storage for its effective normal functioning. As this state developed, the application became unperformant and was unable to process new data indexing requests. Emergency measures were taken to reclaim the temporary storage resources and the search servers were restarted to address the suboptimal operation.
A permanent fix to address the risk of this suboptimal search indexing operation is being scheduled for deployment.
Actions to prevent future incidents:
- The Technology Team has implemented additional alerting for the temporary storage space utilization issues.
- The Technology Team is validating options to implement monitoring around this class of search indexing operation.
- A permanent resolution is under review and is currently being scheduled for deployment.
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/26/12 3:25 pm PST
Detail: On January 26, 2012 the salesforce.com Technology Team resolved a performance degradation affecting the Content Search functionality on the AP1 instance. The problem began at 03:35 UTC and was resolved by 20:20 UTC. During this time, customers may have experienced incomplete results when searching Content. We apologize for any inconvenience this may have caused.
Root cause: The problem was caused by file system errors on the volume containing AP1 Content index data. In the process of repairing the errors, a number of index files were not recoverable. Re-index operations were initiated for customers impacted by this, followed by an instance-wide index validation to ensure no impact remained. Actions to Prevent Future Incidents:
The salesforce.com Technology Team is investigating the cause of the file system errors
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/23/12 3:27 pm PST
Detail: On January 23, 2012 beginning at 19:10 UTC through 19:25 UTC, the salesforce.com Technology Team investigated and resolved a performance degradation affecting the NA8, NA9, EU1, CS7, and CS8 instances. During this time a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by a fault on a network load balancer device. During the incident period, requests handled by this load balancer may not have completed successfully. Other load balancers in the environment were not affected. The root cause of the load balancer fault is under investigation with the vendor. Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
-Continuing vendor investigation into the load balancer fault
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/17/12 4:38 pm PST
Detail: On January 17th, 2012 beginning at 18:30 UTC through 18:54 UTC, the salesforce.com Technology team investigated and resolved a performance degradation affecting the NA8/NA9/EU1/CS7/CS8 instances. During this time, a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by a low-level device failure in our network tier. This failure was transient in nature and did not trigger an automatic re-route of traffic. The issue was resolved by manually isolating the device and re-routing all traffic to the redundant system.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
-has scheduled the replacement of the failed component with the vendor
-is improving monitoring and automatic handling of failures of this specific nature
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/11/12 3:49 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. A small subset of customers were further impacted until February 11, 2012. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/10/12 5:10 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. However, for a small subset of customers, the issue still persists and we are actively continuing our investigation. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/9/12 5:10 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. A small subset of customers were further impacted until February 11, 2012. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/8/12 5:09 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. However, for a small subset of customers, the issue still persists and we are actively continuing our investigation. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/7/12 5:08 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. A small subset of customers were further impacted until February 11, 2012. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/6/12 5:05 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. A small subset of customers were further impacted until February 11, 2012. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/5/12 5:00 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. A small subset of customers were further impacted until February 11, 2012. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/4/12 5:00 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. A small subset of customers were further impacted until February 11, 2012. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/3/12 4:00 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA1 instance. The problem began on February 3, 2012 and was resolved for most customers by February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. A small subset of customers were further impacted until February 11, 2012. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by an application update that resolved the root issue for most customers on NA1 at 02:19 UTC February 10, 2012. Another application update resolved the remaining issues at 10:37 UTC February 11, 2012.
Next Actions:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/1/12 9:38 am PST
Detail: The salesforce.com Technology Team has resolved the performance degradation issue on the NA1 instance. The problem began at 14.40 UTC and was resolved as of 14.49 UTC. During this time customers may have experienced slower than normal login times and performance response times. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by an over-utilization of the storage functionality. As the event occurred manual steps were taken to alleviate the over-utilization. These steps restored normal performance to the instance.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
-Working with storage vendors on a permanent solution.
-Reviewing storage monitoring procedures.
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/16/12 10:40 pm PST
Detail: On January 16, 2012 the salesforce.com Technology Team resolved a performance degradation affecting search functionality on the NA3 instance. The problem began at 00:40 UTC and was resolved by 02:55 UTC. During this time, customers may have experienced slow search performance or incomplete search results. We apologize for any inconvenience this may have caused you.
Root cause: The problem occurred during system maintenance on an NA3 Search indexing host. After moving indexing functions to another host to prepare for this work, a metadata consistency issue was experienced on the Search storage tier. The metadata issue was isolated to a single host in the storage tier. The problem was resolved by directing traffic away from the affected host and re-mounting NA3 Search index volumes.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Revising Search indexing failover procedures to prevent introducing a similar metadata issue in future activities.
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/16/12 7:15 am PST
Detail: On January 16, 2012 beginning at 14:30 UTC and ending at 14:54 UTC, the salesforce.com Technology Team resolved a performance degradation impacting the NA4 instance. During this time, customers may have experienced slow performance or intermittent errors while trying to access or use the Salesforce application. We apologize for any inconvenience this may have caused you.
Root cause: Root Cause: The problem was caused by resource handling on the database tier related to utilization. This caused delays in handling requests. The salesforce.com Technology Team is working with the database vendor to investigate resource utilization improvements.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
-Working with the database vendor to determine a permanent resolution.
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/10/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/9/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/8/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/7/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/6/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/5/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/4/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Service Disruption
Time: 2/3/12 4:39 pm PST
Detail: On February 3, 2012 the salesforce.com Technology Team resolved a service disruption affecting the NA6 instance. The problem began at 22:38 UTC and was resolved by 23:13 UTC. During this time, customers were unable to access the NA6 instance. We apologize for any inconvenience this may have caused.
Root cause: The problem was caused by improper execution of preparation tasks for the Spring '12 release. A database account used by the NA6 application tier was locked out, at which point the instance became unavailable to customers. To restore service, the account was unlocked and the NA6 application tier restarted.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
- Has added validations to release procedures to prevent this scenario
- Is reviewing release preparation procedures for further improvements
Performance Degradation
Time: 2/3/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/10/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/9/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/8/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/7/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/6/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/5/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/4/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/3/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/10/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/9/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/8/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/7/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/6/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/5/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/4/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/3/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/23/12 3:26 pm PST
Detail: On January 23, 2012 beginning at 19:10 UTC through 19:25 UTC, the salesforce.com Technology Team investigated and resolved a performance degradation affecting the NA8, NA9, EU1, CS7, and CS8 instances. During this time a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by a fault on a network load balancer device. During the incident period, requests handled by this load balancer may not have completed successfully. Other load balancers in the environment were not affected. The root cause of the load balancer fault is under investigation with the vendor. Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
-Continuing vendor investigation into the load balancer fault
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/17/12 4:36 pm PST
Detail: On January 17th, 2012 beginning at 18:30 UTC through 18:54 UTC, the salesforce.com Technology team investigated and resolved a performance degradation affecting the NA8/NA9/EU1/CS7/CS8 instances. During this time, a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by a low-level device failure in our network tier. This failure was transient in nature and did not trigger an automatic re-route of traffic. The issue was resolved by manually isolating the device and re-routing all traffic to the redundant system.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
-has scheduled the replacement of the failed component with the vendor
-is improving monitoring and automatic handling of failures of this specific nature
Performance Degradation
Time: 1/17/12 4:35 pm PST
Detail: On January 17, 2012 beginning at 18:23 UTC through 19:51 UTC, the salesforce.com Technology Team investigated and resolved a performance degradation affecting the NA8 instance. During this time, some customers on the NA8 instance experienced slow performance and an inability to log in to the application. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by database index contention which blocked execution of some types of requests. The issue was resolved by terminating individual requests which were not releasing locks in a normal fashion. The root cause of the index contention remains under investigation.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
-Is continuing to investigate internally and with our vendor as to the root cause of the index contention
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/10/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/9/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/8/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/7/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/6/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/5/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/4/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/3/12 12:01 am PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the NA6, NA7, NA8, and NA9 instances. The problem began on February 3, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would have remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem was due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/23/12 3:27 pm PST
Detail: On January 23, 2012 beginning at 19:10 UTC through 19:25 UTC, the salesforce.com Technology Team investigated and resolved a performance degradation affecting the NA8, NA9, EU1, CS7, and CS8 instances. During this time a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by a fault on a network load balancer device. During the incident period, requests handled by this load balancer may not have completed successfully. Other load balancers in the environment were not affected. The root cause of the load balancer fault is under investigation with the vendor. Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
-Continuing vendor investigation into the load balancer fault
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/17/12 4:38 pm PST
Detail: On January 17th, 2012 beginning at 18:30 UTC through 18:54 UTC, the salesforce.com Technology team investigated and resolved a performance degradation affecting the NA8/NA9/EU1/CS7/CS8 instances. During this time, a subset of customers may have experienced intermittent and sporadic connectivity issues while accessing the Salesforce service. We apologize for any inconvenience this may have caused you.
Root cause: The problem was caused by a low-level device failure in our network tier. This failure was transient in nature and did not trigger an automatic re-route of traffic. The issue was resolved by manually isolating the device and re-routing all traffic to the redundant system.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team -
-has scheduled the replacement of the failed component with the vendor
-is improving monitoring and automatic handling of failures of this specific nature
|
|
| |
| |
| |
Incident Report
Informational Message
Time: 2/10/12 3:37 pm PST
Detail: The salesforce.com Technology Team encountered an issue while performing the Spring '12 Release. This issue delayed the completion of critical tasks in this upgrade, therefore an emergency extension of the Release Maintenance Window was declared from 6:31am UTC to 6:36am. We apologize for any inconvenience that this may have caused you.
Root cause: The problem was caused by a premature exit of an automated program for the Spring '12 release upgrade. The automated tasks had to be conducted manually in order to complete the upgrade, thereby causing a delay.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is taking the following actions:
-Investigating the root cause on why the automated program failed
-Reviewing current procedures to run the automated tasks in more efficient ways
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/10/12 7:40 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/9/12 7:38 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/8/12 7:35 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/7/12 7:32 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/6/12 7:27 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/5/12 7:24 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/4/12 7:22 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/3/12 7:19 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/2/12 7:17 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 2/1/12 7:13 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/31/12 7:10 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/30/12 7:07 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/29/12 7:05 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/28/12 7:02 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/27/12 6:48 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/26/12 6:45 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specifically, the impacted Time Based Workflow jobs would remain queued and would not get processed at the expected time. We apologize for any inconvenience this may have caused.
Root cause: The problem is due to an unexpected fault within the new framework for Time Based Workflow introduced in the Spring '12 Release. Temporary workarounds were implemented to improve service for impacted customers, followed by application update that resolved the root issue.
Actions to Prevent Future Incidents:
The salesforce.com Technology Team is -
- Improving testing procedures to better identify and address issues like this in the future
|
|
| |
| |
| |
Incident Report
Performance Degradation
Time: 1/25/12 6:42 pm PST
Detail: On February 10, 2012 the salesforce.com Technology Team resolved a performance degradation affecting Time Based Workflow on the CS0, CS3, CS4, CS7, CS9, CS11, CS12, and CS13 instances. The problem began on January 20, 2012 and was resolved on February 10, 2012. During this time, customers may have experienced delays in Time Based Workflow processing. Specific
| |
Informational message
Status not available
