Outbound network issues

stephr1 · 26 October 2017 22:12

I’m going to start here but suspect I may end up on a Debian forum…

RPI 3 + OSMC + tvh…having problems with the RPI 3 getting out on the network (cannot even download updates for either Debian or the TVHeadEnd server I have 3 other computers on the same subnet (home network) and all of them can access the internet, so I believe my router is good. I can successfully ping the RPI 3 from other computers on the subnet and I can ping from the RPI 3 to the other computers.

I’m not sure how much this might be “0 vector point”, but a couple of months ago I had a Debian update fail (tho, was able to get Debian updates after that until somewhat recently). Also, when ssh’ing into the RPI 3, response time from the RPI 3 seems to have a bigger delay (5-10 secs vs 2-3 secs). One last item is that I used to be able to access the RPI 3 (over ssh) from the other computers on the subnet. Now I can only access it by my desktop.

My sense is something may have gotten messed up in my iptables, but the searching I’ve done and the testing I made as a result of that seems to show the iptables are OK, at least, at a basic level.

Networking/security is not a strong suit of mine so any help here is appreciated.

Thx in advance…

ActionA · 26 October 2017 22:32

To get a better understanding of the problem you are experiencing we need more information from you. The best way to get this information is for you to upload logs that demonstrate your problem. You can learn more about how to submit a useful support request here.

Thanks for your understanding. We hope that we can help you get up and running again shortly.

sam_nazarko · 26 October 2017 23:34

Post those iptables if you don’t mind.

Do things work when you run:

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

stephr1 · 27 October 2017 01:06

Sam,

Truly appreciate the help.

Here are the results of executing the iptables commands you suggested, followed by a “ping 8.8.8.8”. The (^C) was me entering a cntrl-C when I saw no response after 10-15 secs. The results of an “iptables -S” command is at the bottom.

BTW - response time for a ping on the subnet for all systems is < 1ms. The same for my other systems which can access the web.

Again…thx and cheers…

iptables -F

PING 8.8.8.8 (8.8.8.8): 56 data bytes
(^C)
— 8.8.8.8 ping statistics —
5 packets transmitted, 0 packets received, 100% packet loss

iptables -X

PING 8.8.8.8 (8.8.8.8): 56 data bytes
(^C)
— 8.8.8.8 ping statistics —
3 packets transmitted, 0 packets received, 100% packet loss
iptables -t nat -F

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=57 time=12.832 ms
64 bytes from 8.8.8.8: seq=1 ttl=57 time=11.265 ms
(^C)
— 8.8.8.8 ping statistics —
13 packets transmitted, 2 packets received, 84% packet loss
round-trip min/avg/max = 11.265/12.048/12.832 ms

iptables -t nat -X

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=3 ttl=57 time=55.776 ms
64 bytes from 8.8.8.8: seq=4 ttl=57 time=11.804 ms
64 bytes from 8.8.8.8: seq=5 ttl=57 time=13.147 ms
64 bytes from 8.8.8.8: seq=6 ttl=57 time=12.271 ms
64 bytes from 8.8.8.8: seq=7 ttl=57 time=12.704 ms
(^C)
— 8.8.8.8 ping statistics —
11 packets transmitted, 5 packets received, 54% packet loss
round-trip min/avg/max = 11.804/21.140/55.776 ms

iptables -t mangle -F

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=13 ttl=57 time=11.733 ms
64 bytes from 8.8.8.8: seq=14 ttl=57 time=11.320 ms
64 bytes from 8.8.8.8: seq=15 ttl=57 time=41.101 ms
64 bytes from 8.8.8.8: seq=16 ttl=57 time=11.897 ms
64 bytes from 8.8.8.8: seq=17 ttl=57 time=11.900 ms
(^C)
— 8.8.8.8 ping statistics —
39 packets transmitted, 5 packets received, 87% packet loss
round-trip min/avg/max = 11.320/17.590/41.101 ms

iptables -t mangle -X

PING 8.8.8.8 (8.8.8.8): 56 data bytes
(^C)
— 8.8.8.8 ping statistics —
21 packets transmitted, 0 packets received, 100% packet loss(^C)

iptables -P INPUT ACCEPT

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=6 ttl=57 time=12.938 ms
(^C)
— 8.8.8.8 ping statistics —
37 packets transmitted, 1 packets received, 97% packet loss
round-trip min/avg/max = 12.938/12.938/12.938 ms

iptables -P FORWARD ACCEPT
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=19 ttl=57 time=11.160 ms
(^C)
— 8.8.8.8 ping statistics —
44 packets transmitted, 1 packets received, 97% packet loss
round-trip min/avg/max = 11.160/11.160/11.160 ms

iptables -P OUTPUT ACCEPT

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=22 ttl=57 time=10.899 ms
64 bytes from 8.8.8.8: seq=23 ttl=57 time=11.709 ms
64 bytes from 8.8.8.8: seq=24 ttl=57 time=11.634 ms
64 bytes from 8.8.8.8: seq=25 ttl=57 time=11.026 ms
64 bytes from 8.8.8.8: seq=52 ttl=57 time=11.478 ms
64 bytes from 8.8.8.8: seq=53 ttl=57 time=12.354 ms
64 bytes from 8.8.8.8: seq=54 ttl=57 time=12.555 ms
64 bytes from 8.8.8.8: seq=55 ttl=57 time=11.671 ms
(^C)
— 8.8.8.8 ping statistics —
71 packets transmitted, 8 packets received, 88% packet loss
round-trip min/avg/max = 10.899/11.665/12.555 ms

iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

fzinken · 27 October 2017 01:26

So clearly your iptables rules are borked. Suggest you at least remove all of them as a staring point if not doing a full clean install.

What are you trying to say here? You ping to 8.8.8.8 show <12ms I doubt that you get that much faster on other systems

stephr1 · 27 October 2017 02:43

NOTE: Out of curiosity I tried a system updatre. It’s extremely slow and got a bunch of errors. Not clear it’s even going to update.

Have thought about a clean install (and maybe still have to…but hope not). Should I literally delete the iptables (not sure of the actual file) so they can be rebuilt from scratch or is a iptables -F command enuf?

I just tried the -F command option again and then issued an iptables -L and got this:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Then I issued a ping 8.8.8.8 again and got these results:

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=10 ttl=57 time=13.169 ms
64 bytes from 8.8.8.8: seq=11 ttl=57 time=11.370 ms
64 bytes from 8.8.8.8: seq=12 ttl=57 time=11.653 ms
64 bytes from 8.8.8.8: seq=13 ttl=57 time=11.567 ms
64 bytes from 8.8.8.8: seq=51 ttl=57 time=11.914 ms
64 bytes from 8.8.8.8: seq=52 ttl=57 time=11.803 ms
64 bytes from 8.8.8.8: seq=53 ttl=57 time=11.848 ms
64 bytes from 8.8.8.8: seq=92 ttl=57 time=35.564 ms
(^C)
— 8.8.8.8 ping statistics —
140 packets transmitted, 8 packets received, 92% packet loss
round-trip min/avg/max = 11.135/14.129/35.564 ms

It took >10 secs to get the 1st 4 results and then >10secs until the next 4. Then I had to hit cntrl-C.

My bad on the response time. It is ~10+ msecs out to a web address and <1 mSec on my subnet

Thx and cheers…

fzinken · 27 October 2017 02:54

You don’t know how you installed the firewall?
I guess you installed it as a systemd service, or?

The packet loss could indicate a network issue, how are you connected? Do you also see packet loss against your router IP?

stephr1 · 27 October 2017 05:10

I never installed the firewall. I had left it as it was when OSMC got installed. Never directly (or intentionally) modified it after that.

I have 2 other Win systems and 1 Fedora system on my subnet. They all see 100% success (0% packet loss) so I would assume my router is doing fine. There are also a receiver, Dish Network DVR and cell phones (thru wireless) that go thru the router w/out problems. This seems more specific to the RPI 3/OSMC system.

fzinken · 27 October 2017 05:29

By default OSMC has no iptables rules installed. So suggest you reboot and execute sudo iptables -L that would indicate if any rules are installed.

So my question was that if you see also packetloss when you ping the router from OSMC

stephr1 · 27 October 2017 16:06

Reboot and results of iptables -L:
(Same as previous posting)

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Misunderstood about pinging the router directly. Essentially the same as going out onto the web:

PING xxx.xxx.xxx.x (xxx.xxx.xxx.x): 56 data bytes
(10+ secs time pass…)
64 bytes from xxx.xxx.xxx.x: seq=25 ttl=64 time=0.971 ms
(Another 10+ secs time pass…)
64 bytes from xxx.xxx.xxx.x: seq=69 ttl=64 time=0.911 ms
64 bytes from xxx.xxx.xxx.x: seq=70 ttl=64 time=0.911 ms
(yet another…)
64 bytes from xxx.xxx.xxx.x: seq=114 ttl=64 time=0.922 ms
64 bytes from xxx.xxx.xxx.x: seq=115 ttl=64 time=0.909 ms
(10+ secs…)
(^C)
— xxx.xxx.xxx.x ping statistics —
145 packets transmitted, 6 packets received, 95% packet loss

It looks like data xmit times are good, but there seems to be severe data loss and delays between packet transmissions. I ping’ed the router from my Win 10 sys and it gave me 4 rapid replies of 32 bytes <1 mSec, 0% data loss.

Could this be less a network issue and more of some background program sucking up bandwidth? Not sure that would explain the packet losses unless the processor(s) so overwhelmed it cannot process the data fast enuf.

I did a “ps aux | less” command and any program that’s active takes up less than 1% of mem and very few of them take up CPU time. The ones that do are ypically <1% with 1 or 2 progs @ 1-2%. One program of note, kodi.bin, is @ 22% CPU, 18.7% mem in the snap shot. A 2nd snapshot ~1 min later showed kodi.bin @ 20% and 18.7%. Is that anything to be concerned about?

Thx and cheers…

yknivag · 27 October 2017 16:40

On the RPi the network adapter is on the USB bus so if you have drives or active USB TV sticks attached then the path to the network may be congested.

Try installing and running iotop in one session whilst running ping in another.

It could also be due to some strange routing if you have a non-existent primary route and traffic is having to wait to fall back.

Can you post the output of route?

Regarding cpu usage the %wa can be more important for network issues on the pi if you have usb drives attached as that is the indicator of cycles waiting for io.

dillthedog · 27 October 2017 17:34

You really should provide a full log using grab-logs -A and post the URL it returns.
I wouldn’t be so sure about the router being in the clear. In circumstances like this it’s better not to assume that anything is ok. You need to reboot the router.
I assume that you’re connecting via cable but I can’t actually find anything in this thread with this basic information. If it’s via cable, you need to change the cable or, if that’s not possible, physically disconnect and reconnect the cable at both ends. Also swap ports, so that the Pi is on a different physical port.
I note that many pings to the router are also being lost. You need to rule out a problem with the network stack by also pinging 127.0.0.1 (aka localhost).
Do you have anything else in your network, such as switches?

stephr1 · 28 October 2017 02:04

Appreciate you jumping in to help. To answer your questions:

I just executed a grab-logs -A and I’ve been waiting ~30 secs now with no response. Does it need to be executed as root? When I hit cntrl-C to get back control, I don’t get a url, I get the following:

Traceback (most recent call last):
File “/usr/bin/grab-logs”, line 840, in
m.launch_process()
File “/usr/bin/grab-logs”, line 656, in launch_process
self.dispatch_logs()
File “/usr/bin/grab-logs”, line 772, in dispatch_logs
line = f.readline()

Any suggestions?
Have rebooted the router a least a couple of times just to make sure. No change.
This RPI 3 is copper. I have some units like cell phone and laptop thru wi-fi that work fine. Fairly certain I’ve recently checked all you suggest, but just to be sure I swapped ports as well as a cable from a known working system.
Pinged 127.0.0.1 and same problems…~10 secs then a couple of responses, ~10 more secs and couple more responses…etc. A cntrl-C brings the system back with packets received = <10% of packets sent. (94% packet loss). Based on what you say, is this significant?
Yes. I do have switches throughout my network. The RPI 3 shares a switch with 2 or 3 other systems that have no network issues. Sometime during the lesser RPI 3 problems I installed another switch at the RPI 3 “node” to get some more ports. I removed that today, rebooted the RPI 3. Problems still there.

Summary…in the whole system/network/subnet the RPI 3 is the only unit having problems getting out onto the web

Again, thx for the input.

dillthedog:

You really should provide a full log using grab-logs -A and post the URL it returns.

I wouldn’t be so sure about the router being in the clear. In circumstances like this it’s better not to assume that anything is ok. You need to reboot the router.

I assume that you’re connecting via cable but I can’t actually find anything in this thread with this basic information. If it’s via cable, you need to change the cable or, if that’s not possible, physically disconnect and reconnect the cable at both ends. Also swap ports, so that the Pi is on a different physical port.

I note that many pings to the router are also being lost. You need to rule out a problem with the network stack by also pinging 127.0.0.1 (aka localhost).

Do you have anything else in your network, such as switches?

fzinken · 28 October 2017 02:23

Run grab-logs -C -A you will than get a file in /boot/uploadlog.txt (or something like that) which you than can upload from your PC to paste.osmc.io

dillthedog · 28 October 2017 09:34

Yeah, sorry about grab-logs -A request when you have network problems. The sentiment was correct even if the execution was clearly flawed. File that one under “D’oh!”. I’ve slapped my head a few times but I just know I’ll repeat the same mistake in the future. I guess it’s a case of old dog, new tricks…

Point 4 (localhost ping) seems to be significant. Perhaps something in the log will give us a clue.

fzinken · 28 October 2017 10:04

Would have not thought that the connection is that even log upload not working

stephr1 · 28 October 2017 16:06

No problem on the grab-logs command. Here’s the link to the log output file: http://paste.osmc.io/folelenawi.vhdl

I look forward to whatever input/suggestions you can offer.

Thx and cheers…

stephr1 · 28 October 2017 16:14

iotop isn’t installed on the system. And since apt-get install can’t get out onto the web…

I’ll try route when I get the chance and post results.

Thx and cheers…

Tom_Doyle · 28 October 2017 17:29

Hi,

Oct 27 18:27:51 htpc_sys ovpn-Windscribe-US-East[357]: SIGUSR1[soft,no-push-reply] received, process restarting
Oct 27 18:28:06 htpc_sys ovpn-Windscribe-Finland[461]: RESOLVE: Cannot resolve host address: fi.windscribe.com: No address associated with hostname
Oct 27 18:28:13 htpc_sys ovpn-Windscribe-US-East[357]: RESOLVE: Cannot resolve host address: us-east.windscribe.com: No address associated with hostname
Oct 27 18:28:26 htpc_sys ovpn-Windscribe-Finland[461]: RESOLVE: Cannot resolve host address: fi.windscribe.com: No address associated with hostname
Oct 27 18:28:33 htpc_sys ovpn-Windscribe-US-East[357]: RESOLVE: Cannot resolve host address: us-east.windscribe.com: No address associated with hostname

This repeated several times on startup in the log. Doesn’t look like openvpn connection is starting up correctly.

If you stop/disable openvpn are you still able to reproduce the network issues, you have been facing?

Thanks Tom.

stephr1 · 28 October 2017 20:31

Tom,

Thanks for the suggestion. Had not thought of it. I killed openvpn (verified it was inactive/disabled), rebooted (verified again) and, unfortunately, no change. Boooooo

Must be something else.

Cheers…