Timetracker Cloud was not available

Incident Report for Everit

Resolved

Incident description

An invalid SQL statement was executed during the worklog synchronization process. This error was logged in a faulty way that caused an Out-Of-Memory error and an outage on our backend side.

Timeline (UTC)

2022.03.28. 23:30 - First monitoring alert received about the outage.
- The automatic recovery and restart process restarted the application several times.
2022.03.29. 05:30 - Investigation started by our technical staff.
2022.03.29. 08:30 - The problematic Atlassian client was identified and separated from our other clients.
- The Timetracker application was availble again.
2022.03.29. 10:00 - The Timetracker application was updated with the bugfix and was available again for all of our clients.

Impact

The clients couldn't use the Timetracker application in that time window.
There were no data loss during the outage.

Posted Mar 29, 2022 - 08:00 CEST