On Wendnesday 03.05.2023 around 15:45 CET we noticed that customers were unable to work with Plan/ShiftCenter.
16:41 we managed to rollback our changes.
Unexpectedly, however, this did not solve the problem and further investigations became necessary.
17:15 we involved our platform engineering team to downgrade the machine to a previous version.
The machine rollback resolved the issue instantly.
Any attempt to open a ShiftCenter leads to the error message about application load failed.
Starting 03.05.2023 15:45 CET until 03.05.2023 18:25 CET all customers using Plan/ShiftCenter were unable to use it.
The incident was detected by our development team first.
We responded to customers on 03.05.2023 15:45 CET about a partial outage of Plan/ShiftCenter at status.injixo.com and
raise the outage to major outage at 03.05.2023 16:41 CET.
We rolled back all code changes made by our team first. As this did not resolved the outage we rolled back all changes made on the machine.
We accidentally deployed an insufficiant tested version with security updates that blocked starting the application.
We developed peer review approval process to ensure, that accidentally deployments will not happend again.
We apologise for the service disruption.