Summary
On Wendnesday 03.05.2023 around 15:45 CET we noticed that customers were unable to work with Plan/ShiftCenter.
16:41 we managed to rollback our changes.
Unexpectedly, however, this did not solve the problem and further investigations became necessary.
17:15 we involved our platform engineering team to downgrade the machine to a previous version.
The machine rollback resolved the issue instantly.

Fault
Any attempt to open a ShiftCenter leads to the error message about application load failed.

Impact
Starting 03.05.2023 15:45 CET until 03.05.2023 18:25 CET all customers using Plan/ShiftCenter were unable to use it.

Detection
The incident was detected by our development team first.

Response
We responded to customers on 03.05.2023 15:45 CET about a partial outage of Plan/ShiftCenter at status.injixo.com and
raise the outage to major outage at 03.05.2023 16:41 CET.

Recovery
We rolled back all code changes made by our team first. As this did not resolved the outage we rolled back all changes made on the machine.

Root cause
We accidentally deployed an insufficiant tested version with security updates that blocked starting the application.

Corrective actions
We developed peer review approval process to ensure, that accidentally deployments will not happend again.

We apologise for the service disruption.