Slow Response Times
Incident Report for ChangeSprout, Inc
Postmortem

The root cause of this issue was slow response times from a third-party application we integrate with. These slow response times caused some requests to back up and disrupted service for other endpoints as our infrastructure consumed resources that should have been processing web requests. Instead of promptly processing requests, threads were instead waiting on third-party APIs to return results causing requests to queue.

While we did have caching on these third-party API requests, and reasonable timeouts set, those timeouts were not aggressive enough to keep the application working when we were also experiencing moderate traffic.

We’ve now updated the platform to more quickly give up using aggressive timeout settings if certain third-party CRMs are excessively slow. This should make the application more resilient to this type of slow down / request queueing behavior.

Posted Mar 16, 2020 - 23:07 UTC

Resolved
Response times remain stable, and we're rolling out a fix for the underlying issue now.
Posted Mar 16, 2020 - 22:06 UTC
Identified
We've identified the potential root cause of the problem, and we're implementing a fix now. In the meantime, we've provisioned additional servers to mitigate the issue, and response times seem to be back to normal.
Posted Mar 16, 2020 - 15:59 UTC
Investigating
We've received a report of slow response times. We're investigating and will post updates here.
Posted Mar 16, 2020 - 15:29 UTC
This incident affected: ControlShift Platform (US datacenter).