On December 8th, we experienced an issue affecting application deployments, resulting in application promotion failures to test and production stages.

The incident was caused by an issue related to connection limits on an internal message mechanism. A high volume of retries from a specific Data Services Execution Job, which lacked retry limits, escalated the problem into a connection exhaustion situation.

To resolve the issue, a configuration change was implemented to increase the connection limit on the internal message mechanism, and deployments recovered quickly thereafter.

To prevent new occurrences, we are creating an action item to limit retries on the offending job to mitigate future connection escalations.