CHINA STUFF: Email that works…

mail-dp

Working from China is a constant reminder that the location of internet services is super important. North America and Europe are just a few internet hops from our servers in Hetzner’s carbon neutral, green energy data center in Germany. From China, however, we’re often routed out through Beijing to San Jose, then to New York, London, Amsterdam, and finally Frankfurt. On a good day we can do that with only 2-3% packet loss.

Back in the good old days Gmail’s imap service worked perfectly even though Google was blocked, but eventually email was blocked too. At that point we added a managed mail server at Hetzner in Germany. Despite having our own non-blocked mail server, Hetzner’s data centers are unreachable during peak periods of the day because of peering agreements, congestion, geolocation, and a million other factors not worth arguing about.

The image above is a WinMTR report, a combination of ping and traceroute that helps locate network problems. 219.* is where we exit on China Unicom’s cable in Beijing and connect at San Jose, California. Nearly 50% packet loss, average ping time of 482ms. This is better than normal because San Jose connected directly to Frankfurt when this test was taken, instead of the more typical New York-> London-> Amsterdam route.

So how bad is it exactly? 5 tries to send a simple text email without attachments. “Failed to save draft, try again?” prompts every few seconds. Ability to see mail subject headers, but not download the message text. Want to send or receive an attachment? Better schedule that for late at night or first thing in the morning.

International bandwidth from China

There are three cable landings in China: Beijing, Shanghai, and Guangzhou. Each is dominated by one of the three state-owned ISPs. China Unicom (our provider) is big in the north with international traffic routing through Beijing and around 1TB/s bandwidth. China Telecom is big in the south (e.g. Shenzhen area) with around 2.5TB/s of bandwidth through Guangzhou. There are also a lot of bit players like Great Wall and TopWay that have city and regional backbones that eventually dump onto the major state-owned ISP infrastructure.

Part of our problem is using a northern ISP (Unicom) in the south. All of our traffic is routed to Beijing before leaving China. While we can see Hong Kong from the office and can walk there in about 15 minutes, connections to Hong Kong websites are routed up to Beijing and back for a 3000km+ journey. This isn’t our choice, Unicom has a monopoly in our Huaqiangbei office and nothing else is available.

This was intended to be an epic post using MTR to analyze the optimal routes and geographical locations to stash internet services with the best chance to be reachable from China. That idea bombed because nearly everything changed dramatically day to day. A test from yesterday is different than a test today, which will probably be different than a test tomorrow. Follow below for tests of several major email providers and their accessibility from China.

Rackspace Business Email is a disaster from China

rackspace

Over the past few months we tested a lot of email services. Rackspace is a well respected company with business email hosting for $2 per box per month. Signup from China triggered a review, so we had to call support to complete the order. The Rackspace rep volunteered that they have constant complaints from users in China, not something you want to hear. The MTR report shows why, routing to their imap server is a disaster (30% loss, 361ms average ping).

Microsoft Office365 email works great from China, but is itself a disaster

office365

Probably the most distressing part of daily life in China is using Microsoft’s Bing search engine. It’s a terrible search engine, but it’s always super fast within China. You might even find what you’re looking for if you skip directly to the third page of results, link number 5…

Microsoft’s Office365 imap mail service also works very well inside China, and at $4 per box per month it isn’t very expensive. The MTR report suggests Microsoft is running a server in Hong Kong that connects directly to China Unicom (219.*). The service is fast (21ms ping) and very accessible (very few lost packets).

(Un)fortunately Microsoft’s automatic email migration tool sprayed crap all over the place. Their suggestion was to hire an authorized partner for support. After a while all the subscription plans, conditions, and lack of support started to seem quite sleazy. We canceled when mail migration failed and they demanded 1 year commitments for each test account to help debug it.

ABCHK.net is our email hero

abchk

ABCHK.net is a hosting and email provider located in Hong Kong that specializes in email service for Mainland China. We were super skeptical that we could get a stable connection to Hong Kong, but they provided a test account that blew everyone away. 10 meg attachments? Uploaded and sent in seconds. It works perfectly at all times of the day, and the MTR report shows a direct, clean route to the imap server. No packets to destination lost, 31ms average ping.

For less than $7/month we get five mail boxes and 100GB of storage. Real people answered emails and handled the mail migration from our old server in Germany. After a month we are still extremely thrilled to be able to use email “normally” from inside China. Attachments upload and download super fast, and the server is always 100% reachable.

This applies to Unicom only!

Tests were done on commercial and residential China Unicom 100Mbps fiber connections. China Unicom is not the optimal ISP for Shenzhen though. It makes more sense to be on China Telecom with 2.5x more international bandwidth exiting just an hour north in Guangzhou. In the office Unicom has a monopoly, but we had a Telecom connection installed at home and will run the tests again on Telecom in a few days. Anecdotal evidence from other Telecom users doesn’t seem particularly promising though.

Admittedly this is all niche info, but it was hard won and seemed worth sharing here. At the very least someone in a similar situation might find this on Google. Or on Bing, page 3, result 5.

Join the Conversation

12 Comments

  1. Wow. This reminds me of way back in the 1980’s when you had to route email by hand over UUCP. You looked up hosts on a map and spelled out the route as the email address. Something like: teklabs!chico!harpo!eagle!ihnp4!menlo70!sytek!intelqa!bob

    Glad to hear you find a provider that works!

  2. Ouch – that is a lot of painful effort and aggravation.

    In addition to email issues article also highlights likely location based internet routing issues and possible best ISP.
    Beijing China Unicom
    Shanghai China Mobile??
    Guangzhou China Telecom

    Some quick searching showed seven new cable landings – new access points to the world’s Internet backbone just added in 2015. These obviously are no use in your situation, but might help others, if we can work out the major ISP for each area.
    Apparently the big 3 companies where heavily involved in this expansion, so it may be that each will dominate one or more of these additional areas.
    Chengdu
    Wuhan
    Xi’an
    Shenyang
    Nanjing
    Chongqing
    Zhenzhou

    1. Doesn’t a cable landing have to be at a border? I don’t see how some of those places could be landing points.

      There are a number of places around the coast of China where undersea cables terminate. What Ian referred to as Beijing is probably the landing points at QingDao. There are several landing points around the ZheJiang region. There are FuZhou and ShanTou landings on the south coast. The numerous landings in Hong Kong also relay traffic into China. I don’t know about any land cables that might connect China to the rest of the world.

      1. You are correct. I’m a lazy blogger and glossed over the details. According the the report I read from these guys:

        http://www.asiabriefing.com/store/book/internet-challenges-solutions-when-doing-business-in-china-612

        Beijing = Qindao landing
        Guangzhou = Shantou landing
        Shanghai = Shanghai landing

        I’m sure, like China’s bullet train network, that it is growing like crazy and whatever I read is well out of date. From personal observations over the last six months I can say that for my Unicom connection in Shenzhen all the traffic exits China in Beijing (Qingdao). Should that change some day I will be a supper happy geek :)

        PS don’t waste $14 on the publication linked above, totally not worth it.

  3. yay not blocked, I guess Ian changed spam plugins, previous one banned me after mentioning facebook I think :)

    What you need is a VPN inside china. If ABCHK works great on unicom and bad on Telecom you will need a VPN exit node on your unicom office connection. I imagine there should be no problem connecting to unicom IP from within china, that is assuming your office Unicom link has external IP

    It would be even better if you found a VPN provider in HK that has a good route to Unicom and is not blocked by great firewall. Google tells me ExpressVPN and VPN.AC have servers in HK. To be extra safe I would run another box in EU/US with another layer of VPN on top of commercial VPN, just in case some sleazy VPN provider sniffs traffic.

    In summary tunnel everything instead of migrating mail and suffering bing.

    1. I talked to the ABCHK guys and it seems like they have a few dedicated ADSL-like lines into China that connect to the major ISPs directly. I’ve read about that previously when messing with servers inside China.

      In a previous post I wrote about my experience with AWS China and Aliyun trying exactly what you suggest – an inside-China bridge that takes advantage of the great bandwidth inside China but provides a better international exit point than my ISP. Unfortunately AWS China had like 16K/s to the outside world, and even apt-get update from their own internal apt repositories was only 64K/s.

      To be honest I am doing everything I can to avoid VPN. It is quite fickle to use and maintain, and during major events they just completely obliterate all VPN connections. If we were VPN dependent then we would be out of commission 2-3 weeks at a time, 2-3 times a year.

      Since I don’t depend on any blocked services, I’ve been trying instead to find the best routes out of China and locate our internal server resources there. Originally this post had a write up about this, but I saved it for a dedicated post later. Basically I found two solid cables that we have access to: Beijing to San Jose/San Francisco and Beijing to Osaka/Tokyo. The SJO cable, until a few weeks ago, got a solid 1.5Mbps upload to the DirtyPCBs.com server hosted by Digital Ocean at 200 Paul Street (famous datacenter I guess). Now, however, it is almost unusable. The Osaka cable gets a solid 3Mbps day and night, and is solid as a rock (as of this moment).

      In order to share resources between our team inside china with our team around the rest of the world, we setup an OwnCloud (open source cloud storage for PCs/mobile devices) in a Tokyo datacenter on a tiny server and then attached an Amazon S3 storage bucket also located in AWS Tokyo region. This has been a really good setup so far. I can share screencasts with the team, scan and upload receipts for the US company accountant, etc all with great speed. We also moved all our testing and development servers for the “dev site” to Tokyo and it is so lovely I’ve regrown all the hair I pulled out last year.

      I have also been known to SSH into the Tokyo server and use it as a bridge between servers that have slow or poor connections from China. wget big downloads that fail directly from china and then FTP them in using the strong connection to Japan. At certain times of the day I cannot establish a usable SSH connection to the servers in Germany to update the site, but I can SSH into Tokyo and then SSH from the Tokyo servers into the Germany servers with no problem. With all the ssh keys setup and one of the updated forks of Putty this is all quite seamless. I considered setting up a redmail server there to handle our email as well, but I don’t want to constantly mess around with a mail server (or pay a management company $100+ per month) when I can have a professional handle it for about $1 per mailbox per month.

      I’m coming to view international internet access as a series of roads linking China with the rest of the world. We’ve gotta kinda choose the best route for the situation and time of day, and it’s not always the route the GPS nav computer chooses. That might seem pretty obvious, but coming from the west I have never once had to think about which cables my packets were going through. In Holland the internet “just works” and it just works pretty darn well. As more cables are added and more bandwidth is brought online I’m sure this will all become a highly niche archaic experience like working with punch cards, tapes, modems, floppy disks, zip disks, and CDs.

  4. rasz_pl wrote:

    yay not blocked, I guess Ian changed spam plugins, previous one banned me after mentioning facebook I think :)

    What you need is a VPN inside china. If ABCHK works great on unicom and bad on Telecom you will need a VPN exit node on your unicom office connection. I imagine there should be no problem connecting to unicom IP from within china, that is assuming your office Unicom link has external IP

    It would be even better if you found a VPN provider in HK that has a good route to Unicom and is not blocked by great firewall. Google tells me ExpressVPN and VPN.AC have servers in HK. To be extra safe I would run another box in EU/US with another layer of VPN on top of commercial VPN, just in case some sleazy VPN provider sniffs traffic.

    In summary tunnel everything instead of migrating mail and suffering bing.

    1. I talked to the ABCHK guys and it seems like they have a few dedicated ADSL-like lines into China that connect to the major ISPs directly. I’ve read about that previously when messing with servers inside China.

      In a previous post I wrote about my experience with AWS China and Aliyun trying exactly what you suggest – an inside-China bridge that takes advantage of the great bandwidth inside China but provides a better international exit point than my ISP. Unfortunately AWS China had like 16K/s to the outside world, and even apt-get update from their own internal apt repositories was only 64K/s.

      To be honest I am doing everything I can to avoid VPN. It is quite fickle to use and maintain, and during major events they just completely obliterate all VPN connections. If we were VPN dependent then we would be out of commission 2-3 weeks at a time, 2-3 times a year.

      Since I don’t depend on any blocked services, I’ve been trying instead to find the best routes out of China and locate our internal server resources there. Originally this post had a write up about this, but I saved it for a dedicated post later. Basically I found two solid cables that we have access to: Beijing to San Jose/San Francisco and Beijing to Osaka/Tokyo. The SJO cable, until a few weeks ago, got a solid 1.5Mbps upload to the DirtyPCBs.com server hosted by Digital Ocean at 200 Paul Street (famous datacenter I guess). Now, however, it is almost unusable. The Osaka cable gets a solid 3Mbps day and night, and is solid as a rock (as of this moment).

      In order to share resources between our team inside china with our team around the rest of the world, we setup an OwnCloud (open source cloud storage for PCs/mobile devices) in a Tokyo datacenter on a tiny server and then attached an Amazon S3 storage bucket also located in AWS Tokyo region. This has been a really good setup so far. I can share screencasts with the team, scan and upload receipts for the US company accountant, etc all with great speed. We also moved all our testing and development servers for the “dev site” to Tokyo and it is so lovely I’ve regrown all the hair I pulled out last year.

      I have also been known to SSH into the Tokyo server and use it as a bridge between servers that have slow or poor connections from China. wget big downloads that fail directly from china and then FTP them in using the strong connection to Japan. At certain times of the day I cannot establish a usable SSH connection to the servers in Germany to update the site, but I can SSH into Tokyo and then SSH from the Tokyo servers into the Germany servers with no problem. With all the ssh keys setup and one of the updated forks of Putty this is all quite seamless. I considered setting up a redmail server there to handle our email as well, but I don’t want to constantly mess around with a mail server (or pay a management company $100+ per month) when I can have a professional handle it for about $1 per mailbox per month.

      I’m coming to view international internet access as a series of roads linking China with the rest of the world. We’ve gotta kinda choose the best route for the situation and time of day, and it’s not always the route the GPS nav computer chooses. That might seem pretty obvious, but coming from the west I have never once had to think about which cables my packets were going through. In Holland the internet “just works” and it just works pretty darn well. As more cables are added and more bandwidth is brought online I’m sure this will all become a highly niche archaic experience like working with punch cards, tapes, modems, floppy disks, zip disks, and CDs.

      1. Two high buildings with the border between them, two lasers, two diodes and you are in a new business. :-)

    1. No, your off topic misguided attempt to appear smart and snarkie has been deleted.

      You replied to an order status system email, which are typically in the history of the universe no-reply addresses, and it bounced. You then attributed that to our mail and came here to point out how smart you are and how you figured out that my choice of mail provider actually sucks. Jarred by your abrasiveness I left a reply. Then, in better judgement, deleted the two comments as off topic because a bounced reply to a no-reply system email is not related this post. Or, I guess in your mind this must be a conspiracy to suppress knowledge that my chosen email provider is “hopeless” and I’m just shilling for $7/month email service because China. Or something like that.

      None of which helps your stated goal of getting order support which you can do via a support ticket, as described in the email you replied to that bounced. You could also use the contact form, forum, or even a paypal dispute. However the LEAST likely way to get support is to post a snarky, misinformed comment on a random blog post.

      I have a new theory of my own. John is a Microsoft sock pupet sent to punish me for going against Office365 mail and bing. Yeah that’s it! And Microsoft is going to torture me, more than even Bing manages, by insulting my mail provider. 1337 level unlocked! Thanks for the laugh John.

Leave a comment

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.