On July 30, 2025 17:21Z, all production CME Globex live gateways had an issue with the network stack and stopped normalizing data. This lasted until 17:48Z. Failover instances without gaps were brought into the production pool at 17:56Z.
ICE live gateways also experienced the same issue with the network stack at 17:21Z, but continued processing data with degraded performance. Failover instances were activated at 18:03Z.
An issue with the network stacked used in our GLBX and ICE live gateways. We've seen this behavior a handful of times now. We opened a ticket with them back on February 3, however, investigations by the vendor and our own have not led to any clues.
The network stack issue for GLBX meant no packets were being delivered to the socket and customers stopped receiving market data updates. Customers remained connected to the gateways, but would have only received heartbeats. ICE datasets suffered here as well, however packets were still passed through to the socket. ICE customers saw high rates of packet loss and increased latency.
We'll continue to work with the vendor to identify the root cause of the loss of network acceleration. Failover took longer than anticipated, and as a result we've already made changes to speed up this process to reduce the length of the impact.
We're setting up automated incident response to improve response time in the future. We'll also move the ICE live gateways to their own NIC to limit the impact, should the issue reoccur.