Report: Cloud Flare service issues

Damon Billian writes:  Sorry about the service issues today in Amsterdam.  A full explanation from our CEO is below:

We had problems in our Amsterdam datacenter last night which lead to a period of time where traffic through that datacenter was resulting in 502 errors. These error pages were white and displayed a “cloudflare-nginx” footer. Unlike error pages that show our logo, these errors indicate that something has gone wrong in our internal systems. This is an explanation of what we know went wrong and what we’ve done to resolve it.

The core issue was that servers in the AMS datacenter ran out of free memory due to what was effectively a memory leak in new code that was pushed live yesterday. More details to follow. We began receiving the first internal alerts from our monitoring systems around 8:30 GMT. There are some reports of issues beginning about an hour earlier we have seen in the customer service queue and via Twitter. The nature of the problem meant that it may have affected different people at different times and wasn’t picked up by our internal monitors as early as it should have been.

CloudFlare moves around a large amount of internal data in order to make the system work. When you make a change to your DNS settings, for example, it needs to be pushed out to all 14 data centers where the DNS servers then announce it to the world. Similarly, when an attack is seen against one data center, information about that attack is shared back to the core and then pushed out to the edges of our network as quickly as possible.

We are constantly looking for ways to make the movement of this data faster. In the DNS example, one of the most frequent customer service issues we see is that someone makes a change to their DNS and it takes up to 2 minutes to be reflected throughout our infrastructure. That’s much faster than our competitors in the Anycasted DNS space, but we want it to be instant. For attack data, being faster means keeping more sites safer.

For the last week, we’ve been testing code in our San Jose (SJC) datacenter that makes moving thIs sort of data around faster. The challenge is ensuring that, if a datacenter is down when a file gets written, that it gets written again when the file comes back up. We use a custom network file system we created with a backend powered by the key-value store Kyoto Cabinet to make this work. All of our servers also have an in-memory/stateless operating system that is configured remotely and on the fly to meet the current needs at any particular location.

The new code had been running well in SJC for a week, so yesterday we pushed it to AMS. AMS is our busiest datacenter. While we are still investigating the root problem, it appears a configuration error in the new setup caused what we refer to as the “macro list” — which contains all the data on abuse events seen throughout our network — to not have old versions deleted. While the list itself is not a particularly large file (<1GB), it is updated frequently. It is written to a directory allocated by the OS. Since the OS lives in memory, this file also lives in memory. As old versions were not purged, the RAM allocation grew until it began crowding out other processes. This caused servers to begin swapping between disk and memory space which initially degraded performance and then resulted in the errors our monitoring picked up beginning around 8:30 GMT.

The ops team on call saw the alerts and began investigating the problem quickly. AMS see a number of attacks, so their first suspicion was this was a DDoS of some sort that was consuming resources. Quickly they diagnosed that there wasn’t anomalous network traffic. It wasn’t immediately clear what process was consuming the extra RAM. The ops team came to the (correct) conclusion that the problem was within our own stack and likely related to the new code push. They changed the Anycast announcements for the AMS datacenter at approximately 2:30 GMT. This has the effect of immediately dropping that datacenter offline and routing all traffic that would have hit it to the next closest facility (in most cases with AMS Paris or Frankfurt).

Additional ops and dev engineers were contacted to diagnose the problem. They examined San Jose (SJC) and saw the early stages of the same problem. Since at that hour traffic to SJC is relatively light, the problems has not yet begun to affect traffic. The team isolated the issue to the macro list growing beyond its anticipated size limits and therefore consuming excess system memory.

The new rapid update system was still being tested, even though it was running in the production environment, so it was not the primary update mechanism. The process that runs rapid updates was shut down and the macro list memory space was cleared in SJC. This stopped the RAM allocation from continuing to grow. In AMS the systems were tested and we found that the macro list had grown to consume more than 50GB of memory. That was cleared and the rapid update process was shut down. About 30 minutes after AMS had been dropped out of production, the router announcements were again changed and traffic began flowing through the system again.

Throughout the event the only traffic that was affected was that which hit our AMS data center. And visitors to one of the other 13 datacenter would not have seen an error. Going forward we are doing several things:

1) Fixing the bug that caused old versions of the macro list to not be cleared;
2) Reviewing code to see if there are other places data is being written to in-memory storage where we can instead use persistent storage in order to prevent similar issues in the future;
3) Putting caps on in-memory directory allocations to ensure they cannot take up so much RAM that they begin affecting other processes; and
4) Revisiting the RAM monitoring systems we have in place to ensure that when we have RAM limitations in the future we are alerted before they begin to create issues.

I apologize for the problems over night. No downtime is acceptable to us. The good news of this, if there is any, is that this bug arose from our efforts to make the system faster and more resilient. We are hopeful that we will stamp out these problems with the rapid updating system and move it fully into production soon.

Please let us know if you have any additional questions or suggestions on how we could have better handled the incident.