The capacity improvements have stabilized the system. The issue stemmed from a large influx of traffic which overwhelmed a system in an unexpected way. Our fixes this morning had unintended side effects which led to the second incident. We don’t expect further issues but will update the status page if there are any recurrences or regressions.
Posted Mar 23, 2018 - 16:21 PDT
Update
We are working to resolve the issue that is overloading our internal queuing system. Additional capacity and workarounds are being deployed and we’ll provide more details as we continue working to solve the problem.
Posted Mar 23, 2018 - 15:33 PDT
Investigating
We are investigating another issue causing test run issues (delays and expirations). More details to follow.