Virgin media - cant connect to http://apt.osmc.tv

ravenstar68 · 18 May 2017 11:54

Hi - I’m going to start off by introducing myself and explaining how I’ve come to this thread.

I don’t have OSMC, but I am interested in this problem I am a Superuser on Virgin Media’s community forum and my attention was drawn to this issue by a thread claiming Virgin Media are blocking traffic,

https://community.virginmedia.com/t5/Networking-and-WiFi/Virgin-Media-is-blocking-and-interfering-with-traffic-to-certain/td-p/3426163

Superusers like myself don’t work for Virgin we have been given that status because of how helpful we’ve been to other users. I like to lay the blame where it lies so if I beleived Virgin Media were at fault I would indeed be posting it there.

Here’s what I find when I test.

DNS for osmc.tv and apt.osmc.tv resolves to the same IP address meaning the sites are hosted on the same web server (unsurprisingly). And both Virgin Media’s and Google’s public DNS gives the same result.

C:\Users\timdu>nslookup apt.osmc.tv
Server: cache1.service.virginmedia.net
Address: 194.168.4.100
Non-authoritative answer:
Name: apt.osmc.tv
Address: 159.253.212.250
```
 C:\Users\timdu>nslookup apt.osmc.tv 8.8.8.8
 Server:  google-public-dns-a.google.com
 Address:  8.8.8.8

 Non-authoritative answer:
 Name:    apt.osmc.tv
 Address:  159.253.212.250
```
When I browse to http://apt.osmc.tv I get a 302 redirect to the actual repository location as well I should
When I browse to http://osmc.tv I get a 302 redirect to https://osmc.tv - again as expected.
4 I downloaded wget for Windows and confirmed what was happening in the browser here are my wget results.

For apt.osmc.tv

    C:\Users\timdu>wget apt.osmc.tv
    SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
    syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
    --2017-05-18 07:23:22--  http://apt.osmc.tv/
    Resolving apt.osmc.tv... 159.253.212.250
    Connecting to apt.osmc.tv|159.253.212.250|:80... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: http://ftp.fau.de/osmc/osmc/apt/ [following]
    --2017-05-18 07:23:22--  http://ftp.fau.de/osmc/osmc/apt/
    Resolving ftp.fau.de... 131.188.12.211
    Connecting to ftp.fau.de|131.188.12.211|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 1364 (1.3K) [text/html]
    Saving to: `index.html'

    100%[==============================================================================>] 
    1,364       --.-K/s   in 0s

    2017-05-18 07:23:22 (27.5 MB/s) - `index.html' saved [1364/1364]

Now for omsc.tv

C:\Users\timdu>wget osmc.tv --no-check-certificate
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
--2017-05-18 12:39:32--  http://osmc.tv/
Resolving osmc.tv... 159.253.212.250
Connecting to osmc.tv|159.253.212.250|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://osmc.tv/ [following]
--2017-05-18 12:39:32--  https://osmc.tv/
Connecting to osmc.tv|159.253.212.250|:443... connected.
WARNING: cannot verify osmc.tv's certificate, issued by `/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html.2'

    [ <=>                                                                           ] 114,311     --.-K/s   in 0.01s

2017-05-18 12:39:33 (9.64 MB/s) - `index.html.2' saved [114311]

Grindax repeated the tests and again he reached the server but he got a different result.

osmc@osmc:~$ wget apt.osmc.tv
converted 'http://apt.osmc.tv' (ANSI_X3.4-1968) -> 'http://apt.osmc.tv' (UTF-8)
--2017-05-18 08:05:22-- http://apt.osmc.tv/
Resolving apt.osmc.tv (apt.osmc.tv)... 159.253.212.250
Connecting to apt.osmc.tv (apt.osmc.tv)|159.253.212.250|:80... connected.
HTTP request sent, awaiting response... 500 Internal server error
2017-05-18 08:05:22 ERROR 500: Internal server error.

And for osmc.tv

osmc@osmc:~$ wget http://osmc.tv
converted 'http://osmc.tv' (ANSI_X3.4-1968) -> 'http://osmc.tv' (UTF-8)
--2017-05-18 08:14:59--  http://osmc.tv/
Resolving osmc.tv (osmc.tv)... 159.253.212.250
Connecting to osmc.tv (osmc.tv)|159.253.212.250|:80... connected.
HTTP request sent, awaiting response... 500 Internal server error
2017-05-18 08:14:59 ERROR 500: Internal server error.

Finally he did a test to the secure version of the site directly

osmc@osmc:~$ wget https://osmc.tv
converted 'https://osmc.tv' (ANSI_X3.4-1968) -> 'https://osmc.tv' (UTF-8)
--2017-05-18 08:15:13--  https://osmc.tv/
Resolving osmc.tv (osmc.tv)... 159.253.212.250
Connecting to osmc.tv (osmc.tv)|159.253.212.250|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html'

index.html                                           [ <=>                                                                                                         ] 111.63K  --.-KB/s   in 0.05s

2017-05-18 08:15:13 (2.00 MB/s) - 'index.html' saved [114311]

So what does this tell us?

In certain circumstances, the 302 redirects in place on your web server are failing and the server is throwing a 500 error instead… The only possible use of proxies I know of on the Virgin Media network are their Web Safe platform, however from past experience where it has caused an issue I’ve seen the DNS lookups actually diverted to the web safe platform. This would be evident in Grindax’s wget results if this were happening.

If I were running the server at osmc.tv, I’d certainly want to be examining my server logs at this point to see if I can figure out what is going on.

HTH

Tim

sam_nazarko · 18 May 2017 11:56

It’s not a server issue. We don’t see the request come from the user, so we don’t serve the 500.

ravenstar68 · 18 May 2017 12:03

But you can see the https request from the same person?

sam_nazarko · 18 May 2017 12:06

HTTPS is terminated and goes through the same VCL chain, so yes.

APT and web are hosted on different servers but they will be treated the same.

ravenstar68 · 18 May 2017 13:02

It would be interesting to find out if everyone having the problem is on a similar part of the Virgin network. Certainly, the tracert earlier in the thread shows the connection Leaving Virgin at the same point mine does (see hop 7). However, the next hop in mine doesn’t respond to an ICMP request.

C:\Users\timdu>tracert apt.osmc.tv

Tracing route to apt.osmc.tv [159.253.212.250]
over a maximum of 30 hops:

  1     4 ms     2 ms     1 ms  192.168.0.1
  2     *        *        *     Request timed out.
  3    17 ms    15 ms    12 ms  perr-core-2b-xe-112-0.network.virginmedia.net [62.255.32.241]
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7    18 ms    22 ms    18 ms  eislou2-ic-2-ae0-0.network.virginmedia.net [62.254.84.62]
  8     *        *        *     Request timed out.
  9    25 ms    23 ms    26 ms  37.220.95.73.srvlist.ukfast.net [37.220.95.73]
 10    27 ms    27 ms    25 ms  s2.stmlabs.com [159.253.212.250]

Trace complete.

C:\Users\timdu>

They all appear to reach the destination IP though. So if the 500 response isn’t coming from your server - how do you account for this?

henryjfry · 18 May 2017 13:09

I can tell you that recent changes on the virgin network are affecting other secure sites, it’s not just osmc.

I was only able to login on the above site through a proxy because through virgin media it tries to send login details through the secure layer which instantly kicks you back to the login screen.

But this is only another site I can remember off hand. Goolge ads syndication links dont work, disqus redirect links don’t work if I can remember any more I’ll add them.

It’s gotten to the stage where I’m thinking about switching to another isp who doesn’t molest their users traffic

sam_nazarko · 18 May 2017 14:14

No idea – but we don’t see the traffic reach us, so I suspect there’s some kind of MITMing going on. We can’t be serving that 500 if we’re not receiving the request.

dillthedog · 18 May 2017 15:05

Depending on the network configuration, an ICMP ping/traceroute should be able to bypass a transparent http proxy, since it’s a different protocol. As a result, even those who are affected by this issue may have no problem running either ping or traceroute.

Second, this problem occurred only for a subset of Virgin Media users, so while @ravenstar68 is unable to reproduce the issue, it simply means that s/he is not in that group.

Third, Grindax’s wget commands contain lines:

Connecting to apt.osmc.tv (apt.osmc.tv)|159.253.212.250|:80... connected.

and

Connecting to osmc.tv (osmc.tv)|159.253.212.250|:80... connected.

showing a successful connection on port 80, yet @sam_nazarko says that the logs indicate no connection occurred. That certainly could be down to an http proxy.

If this were a general issue, I’d be looking to a problem the osmc.tv server but it’s important to emphasise that it’s not a general issue, and apparently confined to a subset of Virgin Media customers.

If it occurs again, there are ways to sniff out transparent proxies. One method is to run nmap osmc.tv. For future reference, this is what I get:

[user@fedora-24-dvm ~]$ nmap osmc.tv

Starting Nmap 7.40 ( https://nmap.org ) at 2017-05-18 16:00 BST
Nmap scan report for osmc.tv (159.253.212.250)
Host is up (0.25s latency).
rDNS record for 159.253.212.250: s2.stmlabs.com
Not shown: 992 closed ports
PORT     STATE    SERVICE
25/tcp   filtered smtp
80/tcp   open     http
111/tcp  open     rpcbind
443/tcp  open     https
2020/tcp open     xinupageserver
7777/tcp open     cbt
8080/tcp open     http-proxy
9102/tcp open     jetdirect

Nmap done: 1 IP address (1 host up) scanned in 22.98 seconds

If it differs, that might indicate suspicious activity. Another method is to see how long the http connection takes, since a transparent proxy is likely to be closer by.

Edit: After looking around, the best tool to measure http connection times seems to be httperf. It’s available in the Debian repo, so can be installed using apt-get. Here’s httperf in action:

osmc@osmc:~$ httperf --server osmc.tv --port 80 --num-conns 5
httperf --client=0/1 --server=osmc.tv --port=80 --uri=/ --send-buffer=4096 --recv-buffer=16384 --num-conns=5 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 5 requests 5 replies 5 test-duration 0.850 s

Connection rate: 5.9 conn/s (170.0 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 147.7 avg 170.0 max 204.1 median 166.5 stddev 21.4
Connection time [ms]: connect 85.8
Connection length [replies/conn]: 1.000

Request rate: 5.9 req/s (170.0 ms/req)
Request size [B]: 60.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 84.2 transfer 0.0
Reply size [B]: header 166.0 content 0.0 footer 0.0 (total 166.0)
Reply status: 1xx=0 2xx=0 3xx=5 4xx=0 5xx=0

CPU time [s]: user 0.22 system 0.63 (user 25.9% system 74.1% total 100.0%)
Net I/O: 1.3 KB/s (0.0*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

The important line, as far as we’re concerned is this one:

Connection time [ms]: connect 85.8

which is the average time to establish an http connection. (The figures above this line are the “lifetime” of the connection, which is the time between a TCP connection being initiated and the time the connection was closed )

I’ve chosen 5 connection attempts as being sufficient to get a reasonable baseline and (hopefully) not incur the wrath of Sam too much. Virgin Media users who have been affected by this issue in the past should obtain a baseline figure while unaffected, to be used as a reference should problems occur again.

ravenstar68 · 18 May 2017 16:14

So I’m assuming a tool like tracetcp which uses the syn/syn-ack handshake to do its job over layer 4 might be useful

C:\Users\timdu>tracetcp osmc.tv

Tracing route to 159.253.212.250 [s2.stmlabs.com] on port 80
Over a maximum of 30 hops.
1       4 ms    4 ms    4 ms    192.168.0.1
2       *       *       *       Request timed out.
3       25 ms   17 ms   18 ms   62.255.32.241   [perr-core-2b-xe-112-0.network.virginmedia.net]
4       *       *       *       Request timed out.
5       *       *       *       Request timed out.
6       *       *       *       Request timed out.
7       20 ms   17 ms   20 ms   62.254.84.62    [eislou2-ic-2-ae0-0.network.virginmedia.net]
8       *       *       *       Request timed out.
9       25 ms   25 ms   28 ms   37.220.95.73    [37.220.95.73.srvlist.ukfast.net]
10      Destination Reached in 27 ms. Connection established to 159.253.212.250
Trace Complete.

Intriguingly, I do get a different result on port 25 compared to yours.

C:\Users\timdu>nmap osmc.tv

Starting Nmap 7.40 ( https://nmap.org ) at 2017-05-18 17:08 GMT Summer Time
Nmap scan report for osmc.tv (159.253.212.250)
Host is up (0.21s latency).
rDNS record for 159.253.212.250: s2.stmlabs.com
Not shown: 992 closed ports
PORT     STATE SERVICE
25/tcp   open  smtp
80/tcp   open  http
111/tcp  open  rpcbind
443/tcp  open  https
2020/tcp open  xinupageserver
7777/tcp open  cbt
8080/tcp open  http-proxy
9102/tcp open  jetdirect

dillthedog · 18 May 2017 16:19

Port 25 is commonly blocked to prevent mail spamming. It’s probably my ISP or someone else in the chain.

ravenstar68 · 19 May 2017 19:43

@sam_nazarko Do you have Varnish as an edge proxy? Repeating my wget test but using the -S switch

C:\Users\timdu>wget osmc.tv -S --ca-certificate c:\cacert.pem
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
--2017-05-19 20:34:45--  http://osmc.tv/
Resolving osmc.tv... 159.253.212.250
Connecting to osmc.tv|159.253.212.250|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 302 Found
  Date: Fri, 19 May 2017 19:32:42 GMT
  Server: Varnish
  X-Varnish: 12354300
  Location: https://osmc.tv/
  Content-Length: 0
  Connection: keep-alive
Location: https://osmc.tv/ [following]
--2017-05-19 20:34:45--  https://osmc.tv/
Connecting to osmc.tv|159.253.212.250|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.10.2
  Date: Fri, 19 May 2017 19:32:42 GMT
  Content-Type: text/html; charset=utf-8
  Connection: close
  cache-control: public, max-age=0
  etag: W/"1be87-91E/525gGnDdM6k5TtCmXw"
  vary: Accept-Encoding
  Age: 5
  Accept-Ranges: bytes
Length: unspecified [text/html]

Which could explain why you don’t see the http requests hit your actual server, because Varnish is serving the redirection code. Or in the problem cases the 500 error.

Tim

sam_nazarko · 19 May 2017 20:01

Yes, we use Varnish in the chain, but as explained, all traffic goes through Varnish, including HTTPS. So it doesn’t really make sense that HTTPS works for them but HTTP doesn’t. If Varnish was serving this error it would do for both paths.

All Varnish does for HTTP is do a basic synth.
HTTPS goes through Varnish post-termination, and we adjust vcl_hash() to avoid mixed content and add X-Forwarded-Proto so applications are aware we’re serving via HTTPS. Otherwise you’re hitting the same VCL rules.

I checked the varnishncsa log, as that’s where APT redirection is handled.
The firewall also shows no record of connections on port 80 from the IPs the affected users are posting with.

I really can’t see how this is a problem with our server, but if you have further suggestions I’m happy to look in to them. I’m also not sure why it’s only affecting a small number (two?) Virgin Media users. I am on VM myself and do not experience issues.

Zissou · 19 May 2017 22:46

This could possibly not be the same problem but i could not reach http://apt.osmc.tv/ either. Checked the router and everything i could think of. So i switched ISP and then it worked. Changed from one swedish provider to another, so not a virgin customer.

sam_nazarko · 19 May 2017 23:53

Not sure what you mean by switched ISP.

From your post history I suspect you may be using a VPN. Can you try disabling your VPN and let me know if you still have a problem?

Sam

Zissou · 20 May 2017 06:21

I changed ISP. Ditched ‘Universal’ and signed with ‘Bahnhof’. Started to work perfectly then. I am not using a VPN.

ravenstar68 · 20 May 2017 12:21

To be honest I’m stumped. At this point, the only test we haven’t done is for Grindax to take his hub out of modem mode and put it back into router mode. This would give him a different IP address until he changes it back. I’ve asked for this a couple of times yet at present, he’s declined to do this.

The fact that he can connect via VPN does not prove that Virgin is at fault, as connections will again come from a different IP address. For me, things that point against this being a Virgin Media issue are as follows.

Only a very small subset of Virgin Media users are affected. While I’m not cognisant in how popular OSMC is, with Virgin Media having around 5 million customers, I would have expected more people in this thread since February.
While Virgin Media do have proxies in place for their Web Safe platform, These use DNS redirection to point traffic to the proxies, the proxies themselves should handle the onward connection. However the DNS lookups would not point directly to your site, they would point to a Virgin Media IP address. I’ve witnessed this first hand when it was interfering with rowing machine software on windows.
Virgin Media do not make a habit of deliberately interfering with normal web traffic, unless:
They have a specific court order instructing them to do this.
A site is on the IWF list
The site is flagged up by Virus Safe or Child Safe

In all cases, Virgin Media provide redirection to a web page that details the reason a site is blocked.

In many cases where users have claimed that Virgin Media is blocking them, it’s been determined that in fact, the problem was at the site end. However, I’m not at this point ignoring the fact that your logs fail to show the connections. I do wonder if this could be down to something at your hosting provider’s side of the connection.

dillthedog · 21 May 2017 10:32

I haven’t forgotten about this issue but haven’t had enough time to look at it until now.

On the balance of probability, I’m still inclined to think that it’s a localised problem within the Virgin Media network since it seems that the only people reporting the issue are with VM.

If either @henryjfry or @jenkar is still with us, I’d be grateful if one (or both) of you would run a few tests. You’ll need to install two programs: tcptraceroute and httping. They’re both available in the Debian repo.

First, tcptraceroute. This is probably similar to the program used by @ravenstar68 in post #44 but it’s a Linux version, so might work differently. It will act in a similar way to a normal traceroute but will use the TCP protocol, instead of either UDP or ICMP, and will try to inititiate a TCP connection on ports 80 and 443 at each step along the route to osmc.tv. Each network node should not have either port open and the connections should therefore fail until we reach osmc.tv, which has both ports open. If anything along the way shows an open port, that’s a red flag. As a result of this post, I’ve set the SYN packet length to 60, the max TTL to 64 and the don’t-fragment bit to on in attempt to make it look more like a “legitimate” TCP SYN. Here are the commands to run:

tcptraceroute -S -n -F -m 64 -l 60 osmc.tv 443

This is our baseline traceroute, since traffic to port 443 was not affected. Then run:

tcptraceroute -S -n -F -m 64 -l 60 osmc.tv 80

@ravenstar68’s tracetcp got a load of timeouts which looks like VM have some kind of counter-measures in place, so we’ll have to see if this version fares any better.

Next is httping. This attempts to connect to the http(s) port and issues a HTTP HEAD request. I’m not sure it’ll show anything but, then again, we won’t know until we try. Again, we’ll get a baseline on port 443:

httping -s -S -r -l -g https://osmc.tv -c 5

and on port 80:

httping -Z -s -S -r -g http://osmc.tv -c 5

On port 80 I get a “302 Found”, whereas you’ll probably get a “500 Found”, but I’m more interested in the timings.