Summary
Between the evening of 16 September and the morning of 17 September, Siit was intermittently slow and some requests failed. The admin dashboard was most affected. Two short interruptions of a few minutes each occurred on 17 September while our primary database failed over and recovered. The platform has been stable since 10:50 CEST on 17 September.
Impact
16 September, 20:20 to 00:15 CEST (18:20 to 22:15 UTC): intermittent slowness in the admin dashboard, with occasional requests timing out.
17 September, 08:30 to 10:50 CEST (06:30 to 08:50 UTC): sustained slowness. Requests from the admin dashboard, the employee portal and the Slack and Teams integrations could take many seconds or fail with an error.
17 September, 09:19 to 09:31 CEST and 10:36 to 10:47 CEST: two brief interruptions of all write operations while the primary database failed over and returned.
17 September, 10:18 to 10:35 CEST: some Slack/Teams interactions and integration webhooks were not processed on first attempt. They were retried automatically.
No data was lost or corrupted. Requests that failed returned an error to the user and were not partially applied. Automations and integrations that could not run were queued and completed once the platform recovered.
What happened
A recent change introduced real-time refresh of the admin dashboard, so that changes made by other admins or by automations appeared without a manual reload. Under real-world conditions, especially during high traffic / compute periods, this feature generated far more database read traffic than our testing had shown.
How we resolved it
- We rolled back the real-time refresh feature. The admin dashboard has returned to refreshing on navigation and auto/manual pulling for now.
- Restored the primary database and increased read replicas capacity.
- Reduced the number of open database connections from our application servers.
What we are doing next
- Redesigning the real-time refresh feature to batch updates on the server before it is re-enabled
- Tightening the safeguards on our database autoscaling and improving our overall architecture to ensure more resilience in the future
We are sorry for the disruption. If you noticed an impact we have not described here, please contact support@siit.io.
Resolved
Summary
Between the evening of 16 September and the morning of 17 September, Siit was intermittently slow and some requests failed. The admin dashboard was most affected. Two short interruptions of a few minutes each occurred on 17 September while our primary database failed over and recovered. The platform has been stable since 10:50 CEST on 17 September.
Impact
16 September, 20:20 to 00:15 CEST (18:20 to 22:15 UTC): intermittent slowness in the admin dashboard, with occasional requests timing out.
17 September, 08:30 to 10:50 CEST (06:30 to 08:50 UTC): sustained slowness. Requests from the admin dashboard, the employee portal and the Slack and Teams integrations could take many seconds or fail with an error.
17 September, 09:19 to 09:31 CEST and 10:36 to 10:47 CEST: two brief interruptions of all write operations while the primary database failed over and returned.
17 September, 10:18 to 10:35 CEST: some Slack/Teams interactions and integration webhooks were not processed on first attempt. They were retried automatically.
No data was lost or corrupted. Requests that failed returned an error to the user and were not partially applied. Automations and integrations that could not run were queued and completed once the platform recovered.
What happened
A recent change introduced real-time refresh of the admin dashboard, so that changes made by other admins or by automations appeared without a manual reload. Under real-world conditions, especially during high traffic / compute periods, this feature generated far more database read traffic than our testing had shown.
How we resolved it
- We rolled back the real-time refresh feature. The admin dashboard has returned to refreshing on navigation and auto/manual pulling for now.
- Restored the primary database and increased read replicas capacity.
- Reduced the number of open database connections from our application servers.
What we are doing next
- Redesigning the real-time refresh feature to batch updates on the server before it is re-enabled
- Tightening the safeguards on our database autoscaling and improving our overall architecture to ensure more resilience in the future
We are sorry for the disruption. If you noticed an impact we have not described here, please contact support@siit.io.
Monitoring
Our infra seems to be back on tracks, we are monitoring for results and will provide more information once we confirm after a while.
Monitoring
Our jobs are pilling up, we are still applying some changes to fix the underlying issue and we expect to come back to normal in a few minutes. You might notice errors with our agents in the meantime until our queues has fully catched up.
Monitoring
We applied some configuration changes on our database cluster and are monitoring for recovery
Identified
The load and response time have been steadily increasing this morning and we're on it
Monitoring
We have identified the issue, applied a fix and we are seeing a noticeable decrease in response time. We'll keep monitoring for now.
Investigating
We are investigating slower than usual response time and are investigating the reason
