Network disruption in fra

Summary

On October 27, 2026, from 10:04 to about 10:21 UTC, workloads in private networks in fra, including Kubernetes clusters, lost their established network connections. New connections kept working, but existing ones, for example to databases and caches, stopped responding until they timed out and were rebuilt. For many applications this looked like a complete outage of their backends.

The cause was a failure in the network gateway that routes traffic between private networks. This gateway is currently operated by a third-party provider. We have been planning to move it into our own infrastructure for some time and will complete that migration this quarter.

Impact

• Established connections from private networks in fra stopped working for about 17 minutes. Applications that depend on long-lived connections to databases, caches or storage were unavailable or degraded during that time.
• New connections, public endpoints that do not depend on these backends, and the virtual machines and cluster nodes themselves kept running.
• No data was lost and no customer data was accessed or exposed.

Applications recovered on their own once their connections were rebuilt. If a workload still shows errors from this window, restart it.

Timeline

All times are UTC on October 27, 2026.

• 10:04 The active gateway node fails and restarts. The standby node takes over.
• 10:05 The restarted node takes back the active role before it has received the connection state from the standby node. From this point it drops packets of all connections that were open before.
• 10:05 to 10:20 New connections work. Existing connections hang until the operating systems on the affected machines give up on them, which takes about 15 minutes by default.
• 10:21 Connections are rebuilt and all monitored services are healthy again.

What went wrong

The gateway runs as a redundant pair that is meant to hand over seamlessly, including all open connections. Two things failed together: the active node crashed, and when it came back it took over again without first syncing the connection state from its partner. The handover itself worked, but every connection opened before it was lost.

What we are changing

• Together with the provider we are fixing the state synchronization, so that a restarted node only becomes active once it has the full state, and updating the gateway software.
• We are moving this gateway function into our own infrastructure this quarter, and routing internal platform traffic so that it no longer depends on a stateful gateway.
• We are adding monitoring for the gateway pair and for connection churn, so we see this class of failure directly instead of through its effects.