Hi,
I’ve been struggling with ethernet transmission problems that I’ve been trying to fault find for weeks now. Anyway…
Is anyone able to please confirm or deny if the stats returned by iperf3 on a pi2 which has KODI running (but not actively playing/streaming anything) are meaningful? I think I’m seeing gaps in network transmission on pi’s running OSMC/Kodi but not vanilla raspbian, and it would be helpful to know if I’m chasing ghosts or not.
Many thanks.
Mark W.
I am not aware of anything like that. OSMC and Raspbian differ in a number of ways and the issue might have nothing to do with Kodi itself.
If you can provide some more details, perhaps we can help. The more information, the better, including details of network configuration and how you ran the tests.
Sure.
All networking is via gigabit ethernet, no (relevant) wifi. I have a pair of PI 2s configured with OSMC, and a pair of PI 1s configured with Raspbian. My distribution is as follows:-
Pair of QNAP NAS serving mostly a couple of laptops from one and mostly media from the other, connected to…
Netgear GS110TP (internet router connects here too)
Netgear GS728TP (WAP connects here too)
one connection goes to Netgear GS105PE and on to a PI (plus DVD player and TV)
One conneection goes to a 2nd GS110TP then on to a Netgear GS108PE and another PI (plus DVD player, amp and TV)
This setup was working faultlessly for about a year, until a few weeks ago.
My primary testing has been with the first pi with only 3 switches to traverse. I have iperf3 loaded on all PIs. I’ve been SSH into a pair of PIs then using iperf3. I have a Raspian PI plugged into the GS110TP closest to thre NAS’, which I’ve been using to initiate tests as e.g. iperf3 -t600 -c192.168.1.180 -u (or without -u). I’ve been swapping back an forth between the “normal” OSMC PI and the other Raspbian PI, and using that as a server with iperf3 -s.
Switches and cabling are therefore the same.
iperf3 is reporting essentially perfect transmission with the Raspian PI, however when I swap to the OSMC PI I’m seeing significant dropped packets.
The actual symptom that kicked this off is KODI replay on the OSMC PI had started fairly regularly stopping and returning to the KODI menu. MKV media on NAS being shared via OS level NFS v3 mount on the OSMC PIs. I was trying to ascertain if the issue was the NAS, the network or the PIs.
Regards,
Mark W.
I suspect that not too many people who visit this forum are using 28-port smart switches. That’s quite a rig you have there!
Just to kick off proceedings, I see that the switches are all power-over-ethernet (PoE) devices. To clear up one point, are you using PoE to power the RPis?
If I remember correctly there were others reporting issues with gigabit and managed switches. Try to use the search function maybe you find something
Enable Flow Control if you can (should be able to on a Smart Switch); or try adding smsc95xx.turbo_mode=Y to the end of the line in /boot/cmdline.txt then rebooting.
Sam, I’ve just tried both of these against iperf3 test scenario and can see no appreciable difference with the throughput drop outs with the osmc PI as the receiver.
With both sender and receiver as 100BaseT would it really make that much difference anyway?
Regards
Mark W.
It would be good to actually see the iperf results.
Please find below 60 second example of TCP/IP transmission for Raspbian PI to ODMS PI. This is pretty representative, and has flow control turned on on the switches and smsc95xx.turbo_mode=Y.
iperf3 -t60 -c192.168.1.176
Connecting to host 192.168.1.176, port 5201
[ 4] local 192.168.1.173 port 60970 connected to 192.168.1.176 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.09 MBytes 9.14 Mbits/sec 6 1.41 KBytes
[ 4] 1.00-2.04 sec 2.50 MBytes 20.2 Mbits/sec 26 110 KBytes
[ 4] 2.04-3.00 sec 6.25 MBytes 54.3 Mbits/sec 0 132 KBytes
[ 4] 3.00-4.16 sec 7.50 MBytes 54.3 Mbits/sec 0 132 KBytes
[ 4] 4.16-5.13 sec 6.25 MBytes 54.2 Mbits/sec 0 132 KBytes
[ 4] 5.13-6.09 sec 6.25 MBytes 54.7 Mbits/sec 0 132 KBytes
[ 4] 6.09-7.05 sec 6.25 MBytes 54.9 Mbits/sec 0 132 KBytes
[ 4] 7.05-8.00 sec 6.25 MBytes 54.8 Mbits/sec 0 132 KBytes
[ 4] 8.00-9.00 sec 175 KBytes 1.43 Mbits/sec 2 1.41 KBytes
[ 4] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 4] 10.00-11.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 4] 11.00-12.16 sec 6.25 MBytes 45.1 Mbits/sec 94 133 KBytes
[ 4] 12.16-13.12 sec 6.25 MBytes 54.8 Mbits/sec 0 133 KBytes
[ 4] 13.12-14.07 sec 6.25 MBytes 54.9 Mbits/sec 0 133 KBytes
[ 4] 14.07-15.03 sec 6.25 MBytes 55.0 Mbits/sec 0 133 KBytes
[ 4] 15.03-16.18 sec 7.50 MBytes 54.7 Mbits/sec 0 133 KBytes
[ 4] 16.18-17.13 sec 6.25 MBytes 54.7 Mbits/sec 0 133 KBytes
[ 4] 17.13-18.09 sec 6.25 MBytes 54.6 Mbits/sec 0 133 KBytes
[ 4] 18.09-19.05 sec 6.25 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 19.05-20.00 sec 6.25 MBytes 55.0 Mbits/sec 0 202 KBytes
[ 4] 20.00-21.15 sec 7.50 MBytes 54.7 Mbits/sec 0 202 KBytes
[ 4] 21.15-22.11 sec 6.25 MBytes 54.9 Mbits/sec 0 202 KBytes
[ 4] 22.11-23.07 sec 6.25 MBytes 54.7 Mbits/sec 0 202 KBytes
[ 4] 23.07-24.03 sec 6.25 MBytes 54.6 Mbits/sec 0 202 KBytes
[ 4] 24.03-25.17 sec 7.50 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 25.17-26.13 sec 6.25 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 26.13-27.08 sec 6.25 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 27.08-28.04 sec 6.25 MBytes 54.7 Mbits/sec 0 202 KBytes
[ 4] 28.04-29.00 sec 6.25 MBytes 54.7 Mbits/sec 0 202 KBytes
[ 4] 29.00-30.15 sec 7.50 MBytes 54.9 Mbits/sec 0 202 KBytes
[ 4] 30.15-31.10 sec 6.25 MBytes 54.7 Mbits/sec 0 202 KBytes
[ 4] 31.10-32.06 sec 6.25 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 32.06-33.02 sec 6.25 MBytes 54.9 Mbits/sec 0 202 KBytes
[ 4] 33.02-34.16 sec 7.50 MBytes 54.9 Mbits/sec 0 202 KBytes
[ 4] 34.16-35.12 sec 6.25 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 35.12-36.08 sec 6.25 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 36.08-37.03 sec 6.25 MBytes 54.7 Mbits/sec 0 202 KBytes
[ 4] 37.03-38.18 sec 7.50 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 38.18-39.14 sec 6.25 MBytes 54.8 Mbits/sec 0 202 KBytes
[ 4] 39.14-40.09 sec 6.25 MBytes 55.1 Mbits/sec 0 202 KBytes
[ 4] 40.09-41.04 sec 6.25 MBytes 55.0 Mbits/sec 0 202 KBytes
[ 4] 41.04-42.18 sec 7.50 MBytes 55.3 Mbits/sec 0 202 KBytes
[ 4] 42.18-43.14 sec 6.25 MBytes 55.0 Mbits/sec 0 202 KBytes
[ 4] 43.14-44.09 sec 6.25 MBytes 54.9 Mbits/sec 0 202 KBytes
[ 4] 44.09-45.04 sec 6.25 MBytes 55.2 Mbits/sec 0 202 KBytes
[ 4] 45.04-46.18 sec 7.50 MBytes 55.2 Mbits/sec 0 202 KBytes
[ 4] 46.18-47.13 sec 6.25 MBytes 55.0 Mbits/sec 0 202 KBytes
[ 4] 47.13-48.09 sec 6.25 MBytes 55.0 Mbits/sec 0 202 KBytes
[ 4] 48.09-49.04 sec 6.25 MBytes 54.9 Mbits/sec 0 202 KBytes
[ 4] 49.04-50.19 sec 7.50 MBytes 55.0 Mbits/sec 0 202 KBytes
[ 4] 50.19-51.14 sec 6.25 MBytes 55.2 Mbits/sec 0 202 KBytes
[ 4] 51.14-52.09 sec 6.25 MBytes 55.2 Mbits/sec 0 202 KBytes
[ 4] 52.09-53.00 sec 4.41 MBytes 40.5 Mbits/sec 1 1.41 KBytes
[ 4] 53.00-54.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 4] 54.00-55.08 sec 5.00 MBytes 38.7 Mbits/sec 130 148 KBytes
[ 4] 55.08-56.04 sec 6.25 MBytes 54.7 Mbits/sec 0 148 KBytes
[ 4] 56.04-57.00 sec 6.25 MBytes 54.6 Mbits/sec 0 148 KBytes
[ 4] 57.00-58.15 sec 7.50 MBytes 54.8 Mbits/sec 0 148 KBytes
[ 4] 58.15-59.11 sec 6.25 MBytes 54.7 Mbits/sec 0 148 KBytes
[ 4] 59.11-60.07 sec 6.25 MBytes 54.8 Mbits/sec 0 148 KBytes
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.07 sec 352 MBytes 49.1 Mbits/sec 261 sender
[ 4] 0.00-60.07 sec 352 MBytes 49.1 Mbits/sec receiver
I’ll follow up with a Raspbian to Raspbian exampler soon.
Best regards.
Martk W.
Does dmesg show any dropped kevents in this duration?
There’s a network issue here for sure.
Can you try avoiding the secondary switch?
There’s a stable 4.9 kernel (I believe Raspbian recently migrated to this version) coming for OSMC soon. It would be good if you could test iperf again on this kernel.
Can you also confirm you are running the same iperf commands, and same version of iperf on Raspian? I believe iperf2 is default in Debian Jessie (which OSMC and Raspbian are both based on), but assume you have chosen iperf3 for a reason.
Also interested in window size.
Here’s a close mirror of your test: a Raspbian Pi 1 client to OSMC Pi 3 server, only this one goes through two crappy unmanaged switches and traverses subnets:
pi@raspberrypi:~ $ iperf3 -c 192.168.11.88 -t 60
Connecting to host 192.168.11.88, port 5201
[ 4] local 192.168.8.31 port 41694 connected to 192.168.11.88 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.02 sec 6.08 MBytes 49.9 Mbits/sec 0 281 KBytes
[ 4] 1.02-2.12 sec 7.50 MBytes 57.2 Mbits/sec 0 337 KBytes
[ 4] 2.12-3.02 sec 6.25 MBytes 58.4 Mbits/sec 0 337 KBytes
[ 4] 3.02-4.11 sec 7.50 MBytes 57.7 Mbits/sec 0 358 KBytes
[ 4] 4.11-5.02 sec 6.25 MBytes 58.0 Mbits/sec 0 400 KBytes
[ 4] 5.02-6.11 sec 7.50 MBytes 57.6 Mbits/sec 0 400 KBytes
[ 4] 6.11-7.18 sec 7.50 MBytes 58.9 Mbits/sec 0 441 KBytes
[ 4] 7.18-8.04 sec 5.00 MBytes 48.4 Mbits/sec 0 441 KBytes
[ 4] 8.04-9.10 sec 7.50 MBytes 59.3 Mbits/sec 0 441 KBytes
[ 4] 9.10-10.17 sec 7.50 MBytes 58.9 Mbits/sec 0 441 KBytes
[ 4] 10.17-11.07 sec 6.25 MBytes 58.7 Mbits/sec 0 441 KBytes
[ 4] 11.07-12.14 sec 7.50 MBytes 58.8 Mbits/sec 0 441 KBytes
[ 4] 12.14-13.01 sec 6.25 MBytes 59.9 Mbits/sec 0 672 KBytes
[ 4] 13.01-14.09 sec 7.50 MBytes 58.6 Mbits/sec 0 672 KBytes
[ 4] 14.09-15.15 sec 7.50 MBytes 59.2 Mbits/sec 2 471 KBytes
[ 4] 15.15-16.03 sec 6.25 MBytes 59.2 Mbits/sec 0 471 KBytes
[ 4] 16.03-17.11 sec 7.50 MBytes 58.4 Mbits/sec 0 471 KBytes
[ 4] 17.11-18.03 sec 6.25 MBytes 57.4 Mbits/sec 0 471 KBytes
[ 4] 18.03-19.10 sec 7.50 MBytes 58.4 Mbits/sec 0 471 KBytes
[ 4] 19.10-20.15 sec 7.50 MBytes 60.0 Mbits/sec 0 471 KBytes
[ 4] 20.15-21.15 sec 6.45 MBytes 54.2 Mbits/sec 2 414 KBytes
[ 4] 21.15-22.04 sec 6.25 MBytes 58.8 Mbits/sec 0 458 KBytes
[ 4] 22.04-23.10 sec 7.50 MBytes 59.3 Mbits/sec 0 458 KBytes
[ 4] 23.10-24.17 sec 7.50 MBytes 59.0 Mbits/sec 0 458 KBytes
[ 4] 24.17-25.04 sec 6.25 MBytes 60.1 Mbits/sec 0 492 KBytes
[ 4] 25.04-26.11 sec 7.50 MBytes 58.8 Mbits/sec 0 492 KBytes
[ 4] 26.11-27.17 sec 7.50 MBytes 59.7 Mbits/sec 0 492 KBytes
[ 4] 27.17-28.00 sec 6.25 MBytes 62.8 Mbits/sec 2 331 KBytes
[ 4] 28.00-29.00 sec 7.13 MBytes 59.8 Mbits/sec 2 272 KBytes
[ 4] 29.00-30.08 sec 8.23 MBytes 64.1 Mbits/sec 0 296 KBytes
[ 4] 30.08-31.01 sec 7.45 MBytes 66.7 Mbits/sec 2 225 KBytes
[ 4] 31.01-32.03 sec 7.68 MBytes 63.6 Mbits/sec 0 245 KBytes
[ 4] 32.03-33.06 sec 8.14 MBytes 66.0 Mbits/sec 0 263 KBytes
[ 4] 33.06-34.10 sec 7.40 MBytes 59.9 Mbits/sec 0 283 KBytes
[ 4] 34.10-35.05 sec 7.50 MBytes 66.0 Mbits/sec 0 290 KBytes
[ 4] 35.05-36.08 sec 8.29 MBytes 67.6 Mbits/sec 2 233 KBytes
[ 4] 36.08-37.03 sec 7.25 MBytes 64.1 Mbits/sec 0 264 KBytes
[ 4] 37.03-38.00 sec 7.58 MBytes 65.1 Mbits/sec 0 284 KBytes
[ 4] 38.00-39.02 sec 7.89 MBytes 65.3 Mbits/sec 2 182 KBytes
[ 4] 39.02-40.01 sec 7.39 MBytes 62.2 Mbits/sec 0 230 KBytes
[ 4] 40.01-41.00 sec 7.31 MBytes 62.0 Mbits/sec 0 246 KBytes
[ 4] 41.00-42.01 sec 8.35 MBytes 69.5 Mbits/sec 0 266 KBytes
[ 4] 42.01-43.11 sec 8.74 MBytes 66.5 Mbits/sec 0 287 KBytes
[ 4] 43.11-44.00 sec 7.11 MBytes 67.0 Mbits/sec 0 303 KBytes
[ 4] 44.00-45.01 sec 7.91 MBytes 66.0 Mbits/sec 2 230 KBytes
[ 4] 45.01-46.01 sec 7.66 MBytes 64.5 Mbits/sec 0 270 KBytes
[ 4] 46.01-47.10 sec 8.65 MBytes 66.3 Mbits/sec 2 272 KBytes
[ 4] 47.10-48.05 sec 7.47 MBytes 65.7 Mbits/sec 0 230 KBytes
[ 4] 48.05-49.00 sec 7.15 MBytes 63.2 Mbits/sec 0 245 KBytes
[ 4] 49.00-50.01 sec 7.63 MBytes 63.7 Mbits/sec 0 264 KBytes
[ 4] 50.01-51.00 sec 7.84 MBytes 66.3 Mbits/sec 0 281 KBytes
[ 4] 51.00-52.00 sec 7.52 MBytes 63.0 Mbits/sec 2 219 KBytes
[ 4] 52.00-53.01 sec 7.94 MBytes 66.1 Mbits/sec 0 255 KBytes
[ 4] 53.01-54.00 sec 7.68 MBytes 64.8 Mbits/sec 0 277 KBytes
[ 4] 54.00-55.03 sec 8.17 MBytes 66.9 Mbits/sec 0 288 KBytes
[ 4] 55.03-56.01 sec 7.05 MBytes 60.4 Mbits/sec 1 214 KBytes
[ 4] 56.01-57.13 sec 8.73 MBytes 65.0 Mbits/sec 0 235 KBytes
[ 4] 57.13-58.01 sec 7.01 MBytes 66.6 Mbits/sec 0 253 KBytes
[ 4] 58.01-59.12 sec 8.44 MBytes 64.2 Mbits/sec 0 273 KBytes
[ 4] 59.12-60.01 sec 7.07 MBytes 66.8 Mbits/sec 0 290 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.01 sec 441 MBytes 61.7 Mbits/sec 21 sender
[ 4] 0.00-60.01 sec 440 MBytes 61.6 Mbits/sec receiver
iperf Done.
This suggests, at least to me, that there’s nothing inherently wrong with either Raspbian or OSMC.
You never experience a drop in your connection which is good.
Those speeds are fine for playing back HD video.
For the record, mine’s still on 4.4. I guess I’ll upgrade it one day…
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.4.50+ #970 Mon Feb 20 19:12:50 GMT 2017 armv6l GNU/Linux
Did you see the 4.9 thread?
For the avoidance of doubt, I was simply pointing out that my Raspbian kernel is still on 4.4, though I doubt if it has a bearing on the iperf3 figures.
And, yes, I saw the Development thread, if that’s what you mean.
Okay. I think Raspbian has since moved to 4.9
An update to 4.9 is now in the development thread.
Yout iperf3 results have some of the characteristics of network buffer congestion (“bufferbloat”), though there are some anomalies:
It would be interesting if you ran it for a few minutes longer to see if it settles into some kind of cyclical pattern. If you do that, you might want to change the reporting interval to something like 3 seconds. (Please also use the </> symbol to format your data, as it’s difficult to read otherwise.)
I’d also be interested to see how UDP fares on your network with and without -b (bandwidth) being specified.
Hmmm. iperf3 measurements seem particularly bad this evening. Wonder if that’s temperature related? Been quite warm this afternoon. This is all still Raspbian PI1 to OSMC PI2 as before.
UDP:-
Accepted connection from 192.168.1.173, port 60994
[ 5] local 192.168.1.176 port 5201 connected to 192.168.1.173 port 37601
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 120 KBytes 983 Kbits/sec 0.172 ms 0/15 (0%)
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 0.175 ms 0/16 (0%)
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 0.194 ms 0/16 (0%)
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 0.204 ms 0/16 (0%)
[ 5] 4.00-5.00 sec 120 KBytes 983 Kbits/sec 0.269 ms 1/16 (6.2%)
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 0.241 ms 0/16 (0%)
[ 5] 6.00-7.00 sec 80.0 KBytes 655 Kbits/sec 0.218 ms 0/10 (0%)
[ 5] 7.00-8.00 sec 64.0 KBytes 524 Kbits/sec 0.206 ms 14/22 (64%)
[ 5] 8.00-9.00 sec 128 KBytes 1.05 Mbits/sec 0.188 ms 0/16 (0%)
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 0.192 ms 0/16 (0%)
[ 5] 10.00-11.00 sec 128 KBytes 1.05 Mbits/sec 0.215 ms 0/16 (0%)
[ 5] 11.00-12.00 sec 128 KBytes 1.05 Mbits/sec 0.185 ms 0/16 (0%)
[ 5] 12.00-13.00 sec 128 KBytes 1.05 Mbits/sec 0.188 ms 0/16 (0%)
[ 5] 13.00-14.00 sec 128 KBytes 1.05 Mbits/sec 0.195 ms 0/16 (0%)
[ 5] 14.00-15.00 sec 128 KBytes 1.05 Mbits/sec 0.187 ms 0/16 (0%)
[ 5] 15.00-16.00 sec 120 KBytes 983 Kbits/sec 0.265 ms 1/16 (6.2%)
[ 5] 16.00-17.00 sec 128 KBytes 1.05 Mbits/sec 0.218 ms 0/16 (0%)
[ 5] 17.00-18.00 sec 120 KBytes 983 Kbits/sec 0.287 ms 1/16 (6.2%)
[ 5] 18.00-19.00 sec 128 KBytes 1.05 Mbits/sec 0.232 ms 0/16 (0%)
[ 5] 19.00-20.00 sec 128 KBytes 1.05 Mbits/sec 0.203 ms 0/16 (0%)
[ 5] 20.00-21.00 sec 128 KBytes 1.05 Mbits/sec 0.207 ms 0/16 (0%)
[ 5] 21.00-22.00 sec 128 KBytes 1.05 Mbits/sec 0.209 ms 0/16 (0%)
[ 5] 22.00-23.00 sec 128 KBytes 1.05 Mbits/sec 0.211 ms 0/16 (0%)
[ 5] 23.00-24.00 sec 128 KBytes 1.05 Mbits/sec 0.257 ms 0/16 (0%)
[ 5] 24.00-25.00 sec 120 KBytes 983 Kbits/sec 0.270 ms 1/16 (6.2%)
[ 5] 25.00-26.00 sec 112 KBytes 917 Kbits/sec 0.356 ms 1/15 (6.7%)
[ 5] 26.00-27.00 sec 128 KBytes 1.05 Mbits/sec 0.266 ms 1/17 (5.9%)
[ 5] 27.00-28.00 sec 128 KBytes 1.05 Mbits/sec 0.215 ms 0/16 (0%)
[ 5] 28.00-29.00 sec 120 KBytes 983 Kbits/sec 0.244 ms 1/16 (6.2%)
[ 5] 29.00-30.00 sec 128 KBytes 1.05 Mbits/sec 0.207 ms 0/16 (0%)
[ 5] 30.00-30.04 sec 0.00 Bytes 0.00 bits/sec 0.207 ms 0/0 (nan%)
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-30.04 sec 3.74 MBytes 1.05 Mbits/sec 0.207 ms 21/479 (4.4%)
UDP with bandwidth limit raised to 100Mbs:-
Accepted connection from 192.168.1.173, port 60988
[ 5] local 192.168.1.176 port 5201 connected to 192.168.1.173 port 40496
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datag rams
[ 5] 0.00-1.00 sec 8.01 MBytes 67.2 Mbits/sec 0.542 ms 161/1186 (14%)
[ 5] 1.00-2.00 sec 9.86 MBytes 82.7 Mbits/sec 0.537 ms 108/1370 (7.9%)
[ 5] 2.00-3.00 sec 9.89 MBytes 83.0 Mbits/sec 0.449 ms 110/1376 (8%)
[ 5] 3.00-4.00 sec 9.15 MBytes 76.7 Mbits/sec 0.643 ms 201/1372 (15%)
[ 5] 4.00-5.00 sec 9.58 MBytes 80.3 Mbits/sec 0.525 ms 146/1372 (11%)
[ 5] 5.00-6.00 sec 9.49 MBytes 79.6 Mbits/sec 0.434 ms 151/1366 (11%)
[ 5] 6.00-7.00 sec 9.65 MBytes 80.9 Mbits/sec 0.598 ms 141/1376 (10%)
[ 5] 7.00-8.00 sec 9.39 MBytes 78.8 Mbits/sec 0.529 ms 177/1379 (13%)
[ 5] 8.00-9.00 sec 9.52 MBytes 79.9 Mbits/sec 0.658 ms 163/1382 (12%)
[ 5] 9.00-10.00 sec 8.08 MBytes 67.8 Mbits/sec 0.504 ms 348/1382 (25%)
[ 5] 10.00-11.00 sec 9.48 MBytes 79.6 Mbits/sec 0.408 ms 168/1382 (12%)
[ 5] 11.00-12.00 sec 9.62 MBytes 80.7 Mbits/sec 0.597 ms 157/1388 (11%)
[ 5] 12.00-13.00 sec 8.02 MBytes 67.3 Mbits/sec 0.408 ms 347/1374 (25%)
[ 5] 13.00-14.00 sec 9.83 MBytes 82.4 Mbits/sec 0.609 ms 122/1380 (8.8%)
[ 5] 14.00-15.00 sec 8.28 MBytes 69.5 Mbits/sec 0.322 ms 321/1381 (23%)
[ 5] 15.00-16.00 sec 6.35 MBytes 53.3 Mbits/sec 0.525 ms 566/1379 (41%)
[ 5] 16.00-17.00 sec 6.90 MBytes 57.9 Mbits/sec 0.450 ms 496/1379 (36%)
[ 5] 17.00-18.00 sec 7.74 MBytes 64.9 Mbits/sec 0.483 ms 387/1378 (28%)
[ 5] 18.00-19.00 sec 9.82 MBytes 82.4 Mbits/sec 0.513 ms 115/1372 (8.4%)
[ 5] 19.00-20.00 sec 9.13 MBytes 76.6 Mbits/sec 0.408 ms 213/1382 (15%)
[ 5] 20.00-21.00 sec 7.64 MBytes 64.1 Mbits/sec 0.384 ms 405/1383 (29%)
[ 5] 21.00-22.00 sec 6.86 MBytes 57.5 Mbits/sec 0.267 ms 512/1390 (37%)
[ 5] 22.00-23.00 sec 6.57 MBytes 55.1 Mbits/sec 0.419 ms 549/1390 (39%)
[ 5] 23.00-24.00 sec 6.51 MBytes 54.6 Mbits/sec 0.438 ms 556/1389 (40%)
[ 5] 24.00-25.00 sec 6.67 MBytes 56.0 Mbits/sec 0.516 ms 528/1382 (38%)
[ 5] 25.00-26.00 sec 6.79 MBytes 56.9 Mbits/sec 0.396 ms 505/1374 (37%)
[ 5] 26.00-27.00 sec 6.43 MBytes 53.9 Mbits/sec 0.380 ms 550/1373 (40%)
[ 5] 27.00-28.00 sec 5.94 MBytes 49.8 Mbits/sec 0.399 ms 512/1272 (40%)
[ 5] 28.00-29.00 sec 0.00 Bytes 0.00 bits/sec 0.399 ms 0/0 (nan%)
[ 5] 29.00-30.00 sec 6.57 MBytes 55.2 Mbits/sec 0.605 ms 2018/2859 (71%)
[ 5] 30.00-30.04 sec 360 KBytes 82.1 Mbits/sec 0.491 ms 5/50 (10%)
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datag rams
[ 5] 0.00-30.04 sec 322 MBytes 89.9 Mbits/sec 0.491 ms 10738/41218 (26%) )
Nothing showing up in dmesg - no surpises as clearly looks like a network fault.
I’ll have to work at elimenating switches due to topology. Any suggestions as to whether this is switch or cable related?
PS. I will try with Raspbian to Raspbian though probably not 'til tomorrow.
Best regards,
Mark W.
Tell a lie. Immediate repeat using Raspbian to Raspbian.
UDP:-
Accepted connection from 192.168.1.173, port 50120
[ 5] local 192.168.1.180 port 5201 connected to 192.168.1.173 port 39936
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 120 KBytes 983 Kbits/sec 0.413 ms 0/15 (0%)
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 0.500 ms 0/16 (0%)
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 0.513 ms 0/16 (0%)
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 0.525 ms 0/16 (0%)
[ 5] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec 0.528 ms 0/16 (0%)
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 0.515 ms 0/16 (0%)
[ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 0.522 ms 0/16 (0%)
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 0.514 ms 0/16 (0%)
[ 5] 8.00-9.00 sec 128 KBytes 1.05 Mbits/sec 0.522 ms 0/16 (0%)
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 0.514 ms 0/16 (0%)
[ 5] 10.00-11.00 sec 128 KBytes 1.05 Mbits/sec 0.505 ms 0/16 (0%)
[ 5] 11.00-12.00 sec 128 KBytes 1.05 Mbits/sec 0.507 ms 0/16 (0%)
[ 5] 12.00-13.00 sec 128 KBytes 1.05 Mbits/sec 0.516 ms 0/16 (0%)
[ 5] 13.00-14.00 sec 128 KBytes 1.05 Mbits/sec 0.515 ms 0/16 (0%)
[ 5] 14.00-15.00 sec 128 KBytes 1.05 Mbits/sec 0.516 ms 0/16 (0%)
[ 5] 15.00-16.00 sec 128 KBytes 1.05 Mbits/sec 0.513 ms 0/16 (0%)
[ 5] 16.00-17.00 sec 128 KBytes 1.05 Mbits/sec 0.514 ms 0/16 (0%)
[ 5] 17.00-18.00 sec 128 KBytes 1.05 Mbits/sec 0.512 ms 0/16 (0%)
[ 5] 18.00-19.00 sec 128 KBytes 1.05 Mbits/sec 0.509 ms 0/16 (0%)
[ 5] 19.00-20.00 sec 128 KBytes 1.05 Mbits/sec 0.507 ms 0/16 (0%)
[ 5] 20.00-21.00 sec 128 KBytes 1.05 Mbits/sec 0.515 ms 0/16 (0%)
[ 5] 21.00-22.00 sec 128 KBytes 1.05 Mbits/sec 0.511 ms 0/16 (0%)
[ 5] 22.00-23.00 sec 128 KBytes 1.05 Mbits/sec 0.515 ms 0/16 (0%)
[ 5] 23.00-24.00 sec 128 KBytes 1.05 Mbits/sec 0.525 ms 0/16 (0%)
[ 5] 24.00-25.00 sec 128 KBytes 1.05 Mbits/sec 0.531 ms 0/16 (0%)
[ 5] 25.00-26.00 sec 128 KBytes 1.05 Mbits/sec 0.524 ms 0/16 (0%)
[ 5] 26.00-27.00 sec 128 KBytes 1.05 Mbits/sec 0.518 ms 0/16 (0%)
[ 5] 27.00-28.00 sec 128 KBytes 1.05 Mbits/sec 0.526 ms 0/16 (0%)
[ 5] 28.00-29.00 sec 128 KBytes 1.05 Mbits/sec 0.527 ms 0/16 (0%)
[ 5] 29.00-30.00 sec 128 KBytes 1.05 Mbits/sec 0.516 ms 0/16 (0%)
[ 5] 30.00-30.03 sec 0.00 Bytes 0.00 bits/sec 0.516 ms 0/0 (nan%)
…and with bandwidth set to 100mMbs:-
Accepted connection from 192.168.1.173, port 50122
[ 5] local 192.168.1.180 port 5201 connected to 192.168.1.173 port 56782
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 9.17 MBytes 76.8 Mbits/sec 0.995 ms 9/1183 (0.76%)
[ 5] 1.00-2.01 sec 10.5 MBytes 88.0 Mbits/sec 1.035 ms 21/1368 (1.5%)
[ 5] 2.01-3.00 sec 10.6 MBytes 89.1 Mbits/sec 0.877 ms 6/1358 (0.44%)
[ 5] 3.00-4.00 sec 10.4 MBytes 87.4 Mbits/sec 0.753 ms 37/1370 (2.7%)
[ 5] 4.00-5.00 sec 10.4 MBytes 87.6 Mbits/sec 0.873 ms 33/1370 (2.4%)
[ 5] 5.00-6.00 sec 10.3 MBytes 86.5 Mbits/sec 0.740 ms 62/1381 (4.5%)
[ 5] 6.00-7.00 sec 10.4 MBytes 87.3 Mbits/sec 0.816 ms 52/1384 (3.8%)
[ 5] 7.00-8.00 sec 10.2 MBytes 85.8 Mbits/sec 0.784 ms 77/1387 (5.6%)
[ 5] 8.00-9.00 sec 10.5 MBytes 88.3 Mbits/sec 0.811 ms 30/1377 (2.2%)
[ 5] 9.00-10.00 sec 10.4 MBytes 87.2 Mbits/sec 0.957 ms 53/1384 (3.8%)
[ 5] 10.00-11.00 sec 10.5 MBytes 88.1 Mbits/sec 0.818 ms 42/1387 (3%)
[ 5] 11.00-12.00 sec 10.5 MBytes 87.7 Mbits/sec 1.027 ms 42/1382 (3%)
[ 5] 12.00-13.00 sec 10.5 MBytes 88.0 Mbits/sec 0.765 ms 46/1385 (3.3%)
[ 5] 13.00-14.00 sec 10.6 MBytes 88.6 Mbits/sec 0.945 ms 20/1375 (1.5%)
[ 5] 14.00-15.00 sec 10.6 MBytes 88.4 Mbits/sec 0.990 ms 26/1377 (1.9%)
[ 5] 15.00-16.00 sec 10.5 MBytes 87.9 Mbits/sec 0.999 ms 32/1371 (2.3%)
[ 5] 16.00-17.00 sec 10.5 MBytes 88.0 Mbits/sec 0.922 ms 41/1383 (3%)
[ 5] 17.00-18.00 sec 10.6 MBytes 88.5 Mbits/sec 0.972 ms 36/1387 (2.6%)
[ 5] 18.00-19.01 sec 10.6 MBytes 88.5 Mbits/sec 1.040 ms 34/1389 (2.4%)
[ 5] 19.01-20.00 sec 10.2 MBytes 86.3 Mbits/sec 1.039 ms 70/1382 (5.1%)
[ 5] 20.00-21.00 sec 10.5 MBytes 88.4 Mbits/sec 0.770 ms 42/1388 (3%)
[ 5] 21.00-22.00 sec 10.5 MBytes 87.8 Mbits/sec 0.787 ms 44/1385 (3.2%)
[ 5] 22.00-23.00 sec 10.4 MBytes 87.4 Mbits/sec 0.795 ms 53/1385 (3.8%)
[ 5] 23.00-24.00 sec 10.6 MBytes 88.5 Mbits/sec 0.738 ms 37/1388 (2.7%)
[ 5] 24.00-25.00 sec 10.4 MBytes 86.9 Mbits/sec 0.791 ms 55/1381 (4%)
[ 5] 25.00-26.00 sec 10.5 MBytes 87.7 Mbits/sec 0.791 ms 36/1375 (2.6%)
[ 5] 26.00-27.00 sec 10.2 MBytes 85.4 Mbits/sec 0.850 ms 90/1394 (6.5%)
[ 5] 27.00-28.00 sec 10.5 MBytes 88.3 Mbits/sec 0.743 ms 47/1394 (3.4%)
[ 5] 28.00-29.00 sec 10.5 MBytes 87.7 Mbits/sec 0.970 ms 46/1390 (3.3%)
[ 5] 29.00-30.01 sec 10.5 MBytes 87.7 Mbits/sec 1.457 ms 44/1392 (3.2%)
[ 5] 30.01-30.03 sec 224 KBytes 82.2 Mbits/sec 0.876 ms 11/39 (28%)
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-30.03 sec 323 MBytes 90.1 Mbits/sec 0.876 ms 1274/41291 (3.1%)
Nothing in dmesg again.
Best regards,
Mark W.
I’ve taken the liberty of formatting the figures in the first post using the </> symbol for ease of reading.
The first UDP figures at 1 Mbps are very bad. Even a Pi can handle that kind of traffic. They also seem to point away from buffer congestion being the issue, at least in this case. The 100 Mbps figures are very erratic but get worse as time goes by, so I guess could be buffer-related.
The Pi-to-Pi figures show that at least one segment of your LAN is working well. This should at least help you to narrow down the source of the problem.
Edit:
Impossible to say. Swapping cables methodically is an easy thing to do, though less convenient when they’re buried in walls. 
Maybe time to invest in a LONG piece of cat 5e and crimp some connectors to it.
Edit 2: There’s still the remaining question of whether you’re using PoE to power the RPi 2s. Even if the switches can supply sufficient peak-load current, the splitters would need to be up to spec, as well.