Dangerous Prototypes

Working from China is a constant reminder that the location of internet services is super important. North America and Europe are just a few internet hops from our servers in Hetzner’s carbon neutral, green energy data center in Germany. From China, however, we’re often routed out through Beijing to San Jose, then to New York, London, Amsterdam, and finally Frankfurt. On a good day we can do that with only 2-3% packet loss.

Back in the good old days Gmail’s imap service worked perfectly even though Google was blocked, but eventually email was blocked too. At that point we added a managed mail server at Hetzner in Germany. Despite having our own non-blocked mail server, Hetzner’s data centers are unreachable during peak periods of the day because of peering agreements, congestion, geolocation, and a million other factors not worth arguing about.

The image above is a WinMTR report, a combination of ping and traceroute that helps locate network problems. 219.* is where we exit on China Unicom’s cable in Beijing and connect at San Jose, California. Nearly 50% packet loss, average ping time of 482ms. This is better than normal because San Jose connected directly to Frankfurt when this test was taken, instead of the more typical New York-> London-> Amsterdam route.

So how bad is it exactly? 5 tries to send a simple text email without attachments. “Failed to save draft, try again?” prompts every few seconds. Ability to see mail subject headers, but not download the message text. Want to send or receive an attachment? Better schedule that for late at night or first thing in the morning.

International bandwidth from China

There are three cable landings in China: Beijing, Shanghai, and Guangzhou. Each is dominated by one of the three state-owned ISPs. China Unicom (our provider) is big in the north with international traffic routing through Beijing and around 1TB/s bandwidth. China Telecom is big in the south (e.g. Shenzhen area) with around 2.5TB/s of bandwidth through Guangzhou. There are also a lot of bit players like Great Wall and TopWay that have city and regional backbones that eventually dump onto the major state-owned ISP infrastructure.

Part of our problem is using a northern ISP (Unicom) in the south. All of our traffic is routed to Beijing before leaving China. While we can see Hong Kong from the office and can walk there in about 15 minutes, connections to Hong Kong websites are routed up to Beijing and back for a 3000km+ journey. This isn’t our choice, Unicom has a monopoly in our Huaqiangbei office and nothing else is available.

This was intended to be an epic post using MTR to analyze the optimal routes and geographical locations to stash internet services with the best chance to be reachable from China. That idea bombed because nearly everything changed dramatically day to day. A test from yesterday is different than a test today, which will probably be different than a test tomorrow. Follow below for tests of several major email providers and their accessibility from China.

Continue reading “CHINA STUFF: Email that works…”