Increased API Errors during Login

Incident Report for injixo

Postmortem

At 7:59 UTC we were informed by our monitoring systems that database performance changed significantly. During investigation we found out, that problems occurred after performing a routine database migration. The WFM service was either rebooting or starting with a previous release. Only a small amount of customers were effected due to our upgrade mechanism.

At 09:28 UTC we applied a fix for the effected database migration. The WFM service operates now as expected again.

We enhanced our monitoring and fallback mechanism to avoid such a situation in future.

We apologise for the service disruption.

Posted Mar 01, 2021 - 10:04 UTC

Resolved

This incident has been resolved.

Posted Mar 01, 2021 - 09:54 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Mar 01, 2021 - 09:28 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Mar 01, 2021 - 09:23 UTC

Investigating

We are currently experiencing a service disruption.

Some users may be experiencing trouble when logging in to our systems.

Our infrastructure team is working to identify the root cause and implement a solution.

We will send an additional update as soon as we found the root cause.

Posted Mar 01, 2021 - 07:59 UTC

This incident affected: WFM, Forecast, Me, Learn, Account, Support, and Login.