Service disruption in Price Engine

Incident Report for Cary Group AB - Statuspage

Postmortem

The reason for the deploy was a downgrade of a package due to a bug introduced in the latest version. There was a breaking change between the new and old version of the package that the team did not realize would be triggered.

The conditions for triggering the breaking change were only present in production.

Reasonable precautions were in place for catching problems before reaching production. The problem was caught and mitigated within seven minutes.

The system has been updated to handle these corner cases in configuration.

Posted Mar 25, 2026 - 12:45 CET

Resolved

On March 25th, 2026 between 10:04 and 10:11 UTC, the price engine API returned 500 Internal Server Error for all request.

The root cause was a breaking change in how a package handles certain aspects of configuration. This was not caught during the automated tests since the test environments does not have the kind of configuration we have in production that triggered the problem.

Timeline:

09:57 A new version of the software is deployed to production.
10:04 The new version fails to correctly handle the configuration in production and starts returning 500 Internal Server Error.
10:05 The team monitors the logs in production during rollout and notice the problem and starts rolling back the new version.
10:07 Automated alarms are triggered in production due to the errors.
10:11 The previous version is fully rolled out and we are back to normal operations.

Posted Mar 25, 2026 - 11:59 CET