News & Info

Daily Updates and Tech Chatter

Is Comcast Playing “Big Brother” With Your Internet?

The Symptom

This morning I spent over an hour trying to publish a new update of Store Locator Plus to the public WordPress extensions directory.   It failed, multiple times.  I assumed it was something wrong with our repository so I decided to move on to something else until the remote server was fixed.

My next task was to get the WordPress language translation tools into my dev kit so we can start providing better international support.  I decided to fetch the latest language dev kit with subversion via the standard checkout:

svn co http://svn.automattic.com/wordpress-i18n/tools/trunk/

 

Here are a couple of the many failure messages I received back after about 10 minutes:

svn: PROPFIND of '/wordpress-i18n/tools/trunk': could not connect to server (http://svn.automattic.com)
svn: PROPFIND of '/wordpress-i18n/!svn/vcc/default': could not connect to server (http://svn.automattic.com)

 

My first assumption was that my virtual machine was having network issues.  I reset the network, then shut down & restarted the virtual machine.  No luck.   I then tried directly from the host.  Again no luck.  I decided to go to the office and try it from there.  Maybe my modem or router at the house was causing issues.

A Clue

I get to the office and try again.  Same problems.    Odd, more Googling was in my future.   After reading a lot of articles about proxy servers with svn (I don’t use a proxy) and doing all the “svn tricks” I know and that I could dig up online, I stumbled across an interesting post at Stack Overflow.  This is what caught my attention:

Update

I had a co-worker test this out on his home connection — he uses Comcast as well. He got the same error as I did. So it appears to be some Comcast-related issue specific to the WordPress svn repository. I was able to checkout other public repositories via http (e.g. from Google Code) just fine.

Huh, that’s interesting.  I too could use SVN with a variety of other services.  I also was using Comcast at the home office and on one-half of the network at the corporate office.   So I decided to try a couple of things.

The Test

First, shut off the Comcast connection at the office and force my system to connect via the T1. Guess, what?  It worked.  The repo was cloned immediately.

Interesting.   Second test, log in to our server in Michigan on multiple backbones, NONE of which are on Comcast.   Hey, look at that… it worked immediately as well.

Back to the office services.  Turn off the T1, turn on just Comcast.  Instant fail.   Well not instant, it waits for about 5 minutes then fails.

Bandwidth Caps

In addition to the failure to pull subversion content directly from the WordPress IP addresses, we have also found several other interesting things about Comcast Business Class Internet.   Comcast is billing us for 50Mbps/10Mbps speeds at both my home office and our corporate office locations.  We have NEVER been able to get anything close to that at either location.  Our download speeds always seem to max out around 20Mbps/4Mbps at home and 26Mbps/6Mbps at the office.

Today, in an effort to understand what may be going on, we ran ShaperProbe from GA Tech.   It turns out that at my home office we are dropping so many upload packets that many tools, including ShaperProbe fail.   We also learned the Comcast is THROTTLING the incoming bandwidth at the home office to 17Mbps, less than HALF the 50Mbps promised and paid for.   This is one method used to ensure all users have some bandwidth when they oversell a neighborhood.   Ouch.

Comcast Traffic Shaping Test

Comcast Traffic Shaping Test

Comcast Speed Test Results

After the “network improvement” work that Comcast did this week our Corporate line is now crawling at 1/10th the advertised download rate.    We are able to receive our packets from WordPress, but now we can’t get more than a handful of simple transfers going at the same time.

Comcast Speed Test December 2011

Comcast Speed Test December 2011

 

Comcast Fails

I am still doing research on this issue and will post updates here.  However it is very obvious from the initial tests that Comcast is doing some sort of traffic shaping or other network manipulation on their business class services and it is “breaking the Internet”.

I’ll try contacting them, but I am 99.999% certain that whomever I get ahold of will be clueless.  They usually are.    In fact I bet the first thing they ask me to do is reboot my computer, then turn off the modem.  Then they’ll bounce the modem remotely.

We have called Comcast Business services and we are quite shocked to have reached Terry at Business Class Services.  She actually emailed us and is escalating the problem to a higher level tech and is going to chase this down for us today, on a Saturday of all days.   Wow.  That was surprising!  Now lets all pray for some results as doing this proxy thing is a pain!

Reaching Comcast

http://business.comcast.com/smb/contact

Business Customer Service:  (800) 391-3000

Residential Customer Service: (800) 266-2278

In the meantime if you are having the same issues with Comcast please share.   Especially if you found a viable workaround to the issue.

Tracking The Issue

Here are some related articles we’ve dug up about this issue:

 

Update

 

12/17/2011 03:45 EST
Comcast has found a routing issue and is working on it.  We’ll see what happens.

 

12/17/2011 04:15 EST
Comcast claims the routing is fine.  The problem, they claim, is on the WordPress servers.   I made it clear via email this is not the case.    The service works 100% fine when I switch all routing to/from the wordpress.org or automattic.com domains to go over the Windstream T1 v. Comcast business service.   Looks like they couldn’t find a quick/easy answer and are back to their lame excuses and passing-the-buck.

 

12/18/2011 01:05AM EST
Comcast reps never pinged me back before they left for the day (10PM) as promised.   Not surprised about that.   “Dave”, the level 2 tech, said it is not a Comcast problem and that was that.  Bah.  Time to ratchet it up a notch.

 

12/21/2011 10:03AM EST
Nobody ever called back or emailed us about this issue.   We did receive two automated calls the past 2 days in a row that our service would be offline from midnight until 5AM for “network improvement
work.    This morning our WordPress packets are now arriving intact.   Slow as can be though.  Our 50M/10M line is now clocking in at 5M/5M as noted by Comcast Speed Test.

 

Similar Articles

 

Tags: , , , ,

4 Awesome Comments So Far

Don't be a stranger, join the discussion by leaving your own comment
  1. Steve
    December 18, 2011 at 10:07 PM #

    Seems to be fixed now. Well, at least for me.

    • lcleveland
      December 19, 2011 at 9:43 AM #

      Still down here at corporate & at the home office.

      Comcast is “looking into it” but “can’t find anything wrong with the routing”. We know it is not a routing issue, we can access the servers fine. The problem is certain packets coming back are never getting to us. It is a packet inspection & filtering issue. We’re going to put our packet analysis tool online & post results. Unfortunately it is a TON of data, thousands of lines of data for a few seconds of listening, that we need to sort through.

  2. Simon
    January 18, 2012 at 9:01 PM #

    I’m up here in Vancouver BC and this is really starting to annoy me. I can’t do an svn update!

    I am powerless being in Canada though.

Trackbacks/Pingbacks

  1. Committing to WordPress svn via proxy | James Fishwick - January 5, 2012

    […] After I bit of research, I stumbled across this link: http://www.cybersprocket.com/2011/tips-tricks/is-comcast-playing-big-brother-with-your-internet/. […]