This morning I spent over an hour trying to publish a new update of Store Locator Plus to the public WordPress extensions directory. It failed, multiple times. I assumed it was something wrong with our repository so I decided to move on to something else until the remote server was fixed.
My next task was to get the WordPress language translation tools into my dev kit so we can start providing better international support. I decided to fetch the latest language dev kit with subversion via the standard checkout:
svn co http://svn.automattic.com/wordpress-i18n/tools/trunk/
Here are a couple of the many failure messages I received back after about 10 minutes:
svn: PROPFIND of '/wordpress-i18n/tools/trunk': could not connect to server (http://svn.automattic.com)
svn: PROPFIND of '/wordpress-i18n/!svn/vcc/default': could not connect to server (http://svn.automattic.com)
My first assumption was that my virtual machine was having network issues. I reset the network, then shut down & restarted the virtual machine. No luck. I then tried directly from the host. Again no luck. I decided to go to the office and try it from there. Maybe my modem or router at the house was causing issues.
I get to the office and try again. Same problems. Odd, more Googling was in my future. After reading a lot of articles about proxy servers with svn (I don’t use a proxy) and doing all the “svn tricks” I know and that I could dig up online, I stumbled across an interesting post at Stack Overflow. This is what caught my attention:
I had a co-worker test this out on his home connection — he uses Comcast as well. He got the same error as I did. So it appears to be some Comcast-related issue specific to the WordPress svn repository. I was able to checkout other public repositories via http (e.g. from Google Code) just fine.
Huh, that’s interesting. I too could use SVN with a variety of other services. I also was using Comcast at the home office and on one-half of the network at the corporate office. So I decided to try a couple of things.
First, shut off the Comcast connection at the office and force my system to connect via the T1. Guess, what? It worked. The repo was cloned immediately.
Interesting. Second test, log in to our server in Michigan on multiple backbones, NONE of which are on Comcast. Hey, look at that… it worked immediately as well.
Back to the office services. Turn off the T1, turn on just Comcast. Instant fail. Well not instant, it waits for about 5 minutes then fails.
In addition to the failure to pull subversion content directly from the WordPress IP addresses, we have also found several other interesting things about Comcast Business Class Internet. Comcast is billing us for 50Mbps/10Mbps speeds at both my home office and our corporate office locations. We have NEVER been able to get anything close to that at either location. Our download speeds always seem to max out around 20Mbps/4Mbps at home and 26Mbps/6Mbps at the office.
Today, in an effort to understand what may be going on, we ran ShaperProbe from GA Tech. It turns out that at my home office we are dropping so many upload packets that many tools, including ShaperProbe fail. We also learned the Comcast is THROTTLING the incoming bandwidth at the home office to 17Mbps, less than HALF the 50Mbps promised and paid for. This is one method used to ensure all users have some bandwidth when they oversell a neighborhood. Ouch.
Comcast Speed Test Results
After the “network improvement” work that Comcast did this week our Corporate line is now crawling at 1/10th the advertised download rate. We are able to receive our packets from WordPress, but now we can’t get more than a handful of simple transfers going at the same time.
I am still doing research on this issue and will post updates here. However it is very obvious from the initial tests that Comcast is doing some sort of traffic shaping or other network manipulation on their business class services and it is “breaking the Internet”.
I’ll try contacting them, but I am 99.999% certain that whomever I get ahold of will be clueless. They usually are. In fact I bet the first thing they ask me to do is reboot my computer, then turn off the modem. Then they’ll bounce the modem remotely.
We have called Comcast Business services and we are quite shocked to have reached Terry at Business Class Services. She actually emailed us and is escalating the problem to a higher level tech and is going to chase this down for us today, on a Saturday of all days. Wow. That was surprising! Now lets all pray for some results as doing this proxy thing is a pain!
Business Customer Service: (800) 391-3000
Residential Customer Service: (800) 266-2278
In the meantime if you are having the same issues with Comcast please share. Especially if you found a viable workaround to the issue.
Tracking The Issue
Here are some related articles we’ve dug up about this issue:
- Our YouTube video showing the Comcast failure.
- WordPress Forums “subversion error copying my plugin…”
- WordPress Forums “could not connect to server”
- @nacin is Tweeting people as he digs into this.
- The Stack Overflow post that gave us “The Clue”
- An article about Net Nuetrality and Comcast blocking/filtering traffic.
Nobody ever called back or emailed us about this issue. We did receive two automated calls the past 2 days in a row that our service would be offline from midnight until 5AM for “network improvement
work. This morning our WordPress packets are now arriving intact. Slow as can be though. Our 50M/10M line is now clocking in at 5M/5M as noted by Comcast Speed Test.