Summary

For approximately 12 hours between 2023-08-15 04:30 UTC and 2023-08-15 16:48 UTC, some MongoDB Atlas Serverless customers in GCP region europe-west1 and AWS region us-east-1 experienced a complete outage. All connections to their serverless instances would have failed during this time period.

Root Cause and Remediation

The connection problem was determined to be caused by a series of cascading events:

• A new software version was rolled out to the Atlas Serverless fleet.
• This software version contained a change that enforced stricter validation of configuration sent to the Atlas data servers from the Atlas control plane.
• This stricter validation uncovered an existing bug where Atlas would sometimes send incompatible configuration when a customer's project contained both a serverless instance and a dedicated cluster. This incompatibility started causing the connection handling proxy processing requests for the affected customers to crash. Because these proxies are shared between a small number of customers, the bug caused downtime for all customers assigned to the affected proxies.

After the root cause was identified, Atlas engineers worked on a patch and deployed the fix at approximately 16:00 UTC. The system fully recovered by 16:48 UTC when the last affected proxy was successfully restarted.

Prevention and Follow-Up

We have identified the root cause, patched the bug and ensured we added additional unit test coverage in the affected code paths. We have also scheduled follow-up work to improve test coverage going forward for cases involving mixed deployments like the issue above.

We apologize to everyone for the inconvenience caused by this outage and appreciate your patience while we worked on the remediation plan.