Vero4k+ network throughput mystery

ripley · 22 December 2020 20:46

tl;dr Very low network throughput on 4k+ via wired connection with no discernable cause

This began when I started looking into playback of some 4k movies. Some would play find but others would stutter and freeze. When examining the bit rates I noticed that it was the higher rates, usually above 70Mb/s, where I was seeing the most issue. I initially thought perhaps it was a protocol issue so I changed from CIFS to NFS to connect to my FreeNAS server but that had no effect.

After browsing the forum I saw the repeated recommendation to test with iperf. My initial tests were abysmal, I was getting super low throughput often dropping into single digits. This initially led me to think I had a cabling issue or possibly a problem with the small 5 port switch that the 4k+ is plugged into. I grabbed a new cable and a laptop and tested to make sure I could get full 930MB/s or more throughput and took that to the where the 4k+ is. Connected to that same 5 port switch is also an older NUC running Ubuntu LTS 20.04. I installed iperf on that for testing as well. Testing with the laptop connected to the switch got me full throughput, so I swapped the cable and saw no change from the 4k+. Since I now had iperf on the NUC I also tested from there and was able to get full throughput to my storage server. That indicated that the switch and the cabling from my wall plate back to my server room was also working. To rule out an issue with the switch anyway I ran cable from the wall plate directly to the 4k+ but was still getting terrible speeds.

For additional testing I also put the laptop in the server room and connected it to my main switch and started iperf listening on it to see if it was an issue with my storage server. From the NUC, full speed, from the 4k+ same issue. I actually have two FreeNAS servers and testing to both of them from the 4k+ yields the same results.

What is really odd is that every now and then I’ll get one iperf run that shows full throughput but rerunning the test immediately after, the performance drops off a cliff. These three runs were done back to back:

osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  5] local 172.16.11.36 port 48466 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   933 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   938 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   939 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   939 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   939 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   939 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   938 Mbits/sec
[  5]  10.00-11.00  sec   112 MBytes   937 Mbits/sec
[  5]  11.00-12.00  sec   112 MBytes   939 Mbits/sec
[  5]  12.00-13.00  sec   112 MBytes   939 Mbits/sec
[  5]  13.00-14.00  sec   112 MBytes   939 Mbits/sec
[  5]  14.00-15.00  sec   112 MBytes   939 Mbits/sec
[  5]  15.00-16.00  sec   112 MBytes   939 Mbits/sec
[  5]  16.00-17.00  sec   112 MBytes   938 Mbits/sec
[  5]  17.00-18.00  sec   112 MBytes   936 Mbits/sec
[  5]  18.00-19.00  sec   112 MBytes   940 Mbits/sec
[  5]  19.00-20.00  sec   112 MBytes   938 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.19 GBytes   939 Mbits/sec    0             sender
[  5]   0.00-20.00  sec  2.18 GBytes   938 Mbits/sec                  receiver

iperf Done.
osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.25
Connecting to host 172.16.11.25, port 5201
Reverse mode, remote host 172.16.11.25 is sending
[  5] local 172.16.11.36 port 55513 connected to 172.16.11.25 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  12.8 MBytes   108 Mbits/sec
[  5]   1.00-2.00   sec   482 KBytes  3.95 Mbits/sec
[  5]   2.00-3.00   sec   143 KBytes  1.17 Mbits/sec
[  5]   3.00-4.00   sec   559 KBytes  4.58 Mbits/sec
[  5]   4.00-5.00   sec   387 KBytes  3.17 Mbits/sec
[  5]   5.00-6.00   sec   522 KBytes  4.27 Mbits/sec
[  5]   6.00-7.00   sec   551 KBytes  4.52 Mbits/sec
[  5]   7.00-8.00   sec  1.08 MBytes  9.06 Mbits/sec
[  5]   8.00-9.00   sec   370 KBytes  3.03 Mbits/sec
[  5]   9.00-10.00  sec   318 KBytes  2.61 Mbits/sec
[  5]  10.00-11.00  sec   283 KBytes  2.32 Mbits/sec
[  5]  11.00-12.00  sec  1012 KBytes  8.29 Mbits/sec
[  5]  12.00-13.00  sec   573 KBytes  4.69 Mbits/sec
[  5]  13.00-14.00  sec   411 KBytes  3.37 Mbits/sec
[  5]  14.00-15.00  sec   290 KBytes  2.37 Mbits/sec
[  5]  15.00-16.00  sec   426 KBytes  3.49 Mbits/sec
[  5]  16.00-17.00  sec   115 KBytes   938 Kbits/sec
[  5]  17.00-18.00  sec   393 KBytes  3.22 Mbits/sec
[  5]  18.00-19.00  sec   635 KBytes  5.20 Mbits/sec
[  5]  19.00-20.00  sec   378 KBytes  3.09 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  22.8 MBytes  9.57 Mbits/sec  158             sender
[  5]   0.00-20.00  sec  21.6 MBytes  9.04 Mbits/sec                  receiver

iperf Done.
osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  5] local 172.16.11.36 port 48470 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   510 KBytes  4.18 Mbits/sec
[  5]   1.00-2.00   sec   112 KBytes   915 Kbits/sec
[  5]   2.00-3.00   sec   157 KBytes  1.29 Mbits/sec
[  5]   3.00-4.00   sec   257 KBytes  2.11 Mbits/sec
[  5]   4.00-5.00   sec   352 KBytes  2.89 Mbits/sec
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   6.00-7.00   sec   235 KBytes  1.92 Mbits/sec
[  5]   7.00-8.00   sec  84.8 KBytes   695 Kbits/sec
[  5]   8.00-9.00   sec  56.6 KBytes   463 Kbits/sec
[  5]   9.00-10.00  sec  28.3 KBytes   232 Kbits/sec
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  11.00-12.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  12.00-13.00  sec  28.3 KBytes   232 Kbits/sec
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  15.00-16.00  sec  42.4 KBytes   348 Kbits/sec
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  17.00-18.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  18.00-19.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  19.00-20.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  1.85 MBytes   777 Kbits/sec   17             sender
[  5]   0.00-20.00  sec  1.82 MBytes   763 Kbits/sec                  receiver

iperf Done.

.40 is my primary FreeNAS server and .25 is the one I’m working to decomm but is still serving NFS shares to my VMware cluster. Both are connected with dual GB NICs in LAGG to my Cisco 3560 and are definitely capable of sustained gigabit throughput in iperf. Testing from every other source, the NUC, my desktop, multiple laptops, VMs all show results as expected in the 930+ MB/s range.

I also have an older Vero 4k with a USB gigabit NIC that I pulled out of storage for testing. Plugged into the same 5 port switch and testing to the same FreeNAS servers yields results like this:

osmc@vero:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  4] local 172.16.11.37 port 46056 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  34.7 MBytes   291 Mbits/sec
[  4]   1.00-2.00   sec  35.2 MBytes   295 Mbits/sec
[  4]   2.00-3.00   sec  34.9 MBytes   292 Mbits/sec
[  4]   3.00-4.00   sec  35.1 MBytes   295 Mbits/sec
[  4]   4.00-5.00   sec  34.4 MBytes   289 Mbits/sec
[  4]   5.00-6.00   sec  34.8 MBytes   292 Mbits/sec
[  4]   6.00-7.00   sec  34.1 MBytes   286 Mbits/sec
[  4]   7.00-8.00   sec  17.9 MBytes   150 Mbits/sec
[  4]   8.00-9.00   sec  26.5 MBytes   222 Mbits/sec
[  4]   9.00-10.00  sec  35.2 MBytes   295 Mbits/sec
[  4]  10.00-11.00  sec  35.2 MBytes   295 Mbits/sec
[  4]  11.00-12.00  sec  34.5 MBytes   289 Mbits/sec
[  4]  12.00-13.00  sec  29.4 MBytes   246 Mbits/sec
[  4]  13.00-14.00  sec  23.6 MBytes   198 Mbits/sec
[  4]  14.00-15.00  sec  35.1 MBytes   294 Mbits/sec
[  4]  15.00-16.00  sec  35.1 MBytes   294 Mbits/sec
[  4]  16.00-17.00  sec  34.9 MBytes   293 Mbits/sec
[  4]  17.00-18.00  sec  27.0 MBytes   226 Mbits/sec
[  4]  18.00-19.00  sec  33.8 MBytes   284 Mbits/sec
[  4]  19.00-20.00  sec  26.6 MBytes   223 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-20.00  sec   638 MBytes   268 Mbits/sec  4344             sender
[  4]   0.00-20.00  sec   638 MBytes   268 Mbits/sec                  receiver

iperf Done.
osmc@vero:~$ iperf3 -R -t 20 -c 172.16.11.25
Connecting to host 172.16.11.25, port 5201
Reverse mode, remote host 172.16.11.25 is sending
[  4] local 172.16.11.37 port 50182 connected to 172.16.11.25 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  35.2 MBytes   295 Mbits/sec
[  4]   1.00-2.00   sec  35.8 MBytes   300 Mbits/sec
[  4]   2.00-3.00   sec  35.4 MBytes   297 Mbits/sec
[  4]   3.00-4.00   sec  35.6 MBytes   299 Mbits/sec
[  4]   4.00-5.00   sec  35.4 MBytes   297 Mbits/sec
[  4]   5.00-6.00   sec  35.5 MBytes   298 Mbits/sec
[  4]   6.00-7.00   sec  35.6 MBytes   299 Mbits/sec
[  4]   7.00-8.00   sec  35.5 MBytes   298 Mbits/sec
[  4]   8.00-9.00   sec  27.2 MBytes   228 Mbits/sec
[  4]   9.00-10.00  sec  35.8 MBytes   300 Mbits/sec
[  4]  10.00-11.00  sec  35.9 MBytes   301 Mbits/sec
[  4]  11.00-12.00  sec  35.3 MBytes   296 Mbits/sec
[  4]  12.00-13.00  sec  35.5 MBytes   298 Mbits/sec
[  4]  13.00-14.00  sec  35.4 MBytes   297 Mbits/sec
[  4]  14.00-15.00  sec  34.3 MBytes   288 Mbits/sec
[  4]  15.00-16.00  sec  35.7 MBytes   299 Mbits/sec
[  4]  16.00-17.00  sec  35.5 MBytes   298 Mbits/sec
[  4]  17.00-18.00  sec  35.8 MBytes   300 Mbits/sec
[  4]  18.00-19.00  sec  35.7 MBytes   300 Mbits/sec
[  4]  19.00-20.00  sec  35.8 MBytes   300 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-20.00  sec   702 MBytes   294 Mbits/sec  5662             sender
[  4]   0.00-20.00  sec   702 MBytes   294 Mbits/sec                  receiver

iperf Done.

All right around the 300Mb/s range I’d expect using the USB NIC.

For the final bit of oddness, if I run iperf from the 4k+ to the NUC on the same 5 port switch I can consistently get full gigabit throughput on run after run after run.

In case it helps here are system logs - https://paste.osmc.tv/uluvuwogom

I believe I’ve tested and validated every piece of hardware and software in my setup and the only common issue is the 4k+, except when it talks to the NUC on the same local switch. I am truly baffled. The only thing I haven’t yet tested is reinstalling the OS on the 4k+ which I will even though I can’t think of a reason why it would matter.

Is there anything else that I’ve missed here?

dillthedog · 22 December 2020 21:12

I’d suggest you look at your LAGG configuration. What happens if you remove one of the cables?

ripley · 22 December 2020 21:19

I’m not sure why the LAGG would matter, testing from the 4k+ to a laptop with a single wired connection shows similar results. And testing from my older 4k to the server with LAGG doesn’t have any issues, getting the max throughput it can via its USB NIC.

For thoroughness I did try it but the results did not change.

Results with one cable unplugged:

osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  5] local 172.16.11.36 port 48501 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.23   sec   128 KBytes   855 Kbits/sec
[  5]   1.23-2.00   sec   332 KBytes  3.51 Mbits/sec
[  5]   2.00-3.00   sec  79.2 KBytes   648 Kbits/sec
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   4.00-5.00   sec   194 KBytes  1.59 Mbits/sec
[  5]   5.00-6.00   sec   362 KBytes  2.97 Mbits/sec
[  5]   6.00-7.00   sec  1008 KBytes  8.26 Mbits/sec
[  5]   7.00-8.00   sec   228 KBytes  1.86 Mbits/sec
[  5]   8.00-9.00   sec   310 KBytes  2.54 Mbits/sec
[  5]   9.00-10.00  sec   375 KBytes  3.07 Mbits/sec
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  11.00-12.00  sec  2.83 KBytes  23.2 Kbits/sec
[  5]  12.00-13.00  sec  25.5 KBytes   209 Kbits/sec
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  14.00-15.00  sec   277 KBytes  2.27 Mbits/sec
[  5]  15.00-16.00  sec   214 KBytes  1.75 Mbits/sec
[  5]  16.00-17.00  sec  50.9 KBytes   417 Kbits/sec
[  5]  17.00-18.00  sec  84.8 KBytes   695 Kbits/sec
[  5]  18.00-19.00  sec  50.9 KBytes   417 Kbits/sec
[  5]  19.00-20.00  sec   157 KBytes  1.29 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  3.82 MBytes  1.60 Mbits/sec   27             sender
[  5]   0.00-20.00  sec  3.79 MBytes  1.59 Mbits/sec                  receiver

iperf Done.

dillthedog · 22 December 2020 21:25

What happens if you now use the other cable?

ripley · 22 December 2020 21:31

Which other cable do you mean?

dillthedog · 22 December 2020 21:36

LAGG can make use of the MAC address but this seems to rule that out:

Do you remember when you bought your Vero4K+?

ripley · 22 December 2020 21:37

Order #29988 (June 17, 2020)

dillthedog · 22 December 2020 21:56

There was an issue with an early batch of Vero4K+ devices but that was 2+ years ago, so definitely not related to your problem.

I’m not sure if you’re tried this yet:

Vero4K+ <-> 5-port switch <-> Linux laptop

If that gives crappy results, change both cables and ports. If it’s still bad on a repeatable basis, and the laptop is still looking good to the server(s) that would suggest to me it’s the Vero4K+.

ripley · 22 December 2020 21:58

I was incorrect. That order was one I placed for a friend, it’s not mine. I can’t find the email receipt for my purchase, just for my original 4k, Order #18010 (June 6, 2018). If memory serves it was would have been Dec 2018 or Jan 2019 when I bought my 4k+.

dillthedog · 22 December 2020 22:08

Perhaps @sam_nazarko can help you figure out if yours is one of the affected batch.

I did a quick search and found this post External Gigabit Adapter on Vero 4k+? - #53 by sam_nazarko where Sam says:

All devices shipped are run against an extensive test suite. As of the end of September, this also tests Ethernet performance.

bmillham · 22 December 2020 22:39

Have you run ifconfig on the Vero to see if there are a lot of dropped packets? (I suspect that there will be)

ripley · 22 December 2020 22:50

Thanks for all the help @dillthedog. After reading over the thread you linked I plugged the USB NIC into my 4K+ to test and it works much better than the onboard.

osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.25
Connecting to host 172.16.11.25, port 5201
Reverse mode, remote host 172.16.11.25 is sending
[  5] local 172.16.11.109 port 49435 connected to 172.16.11.25 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  33.8 MBytes   284 Mbits/sec
[  5]   1.00-2.00   sec  35.6 MBytes   299 Mbits/sec
[  5]   2.00-3.00   sec  35.6 MBytes   299 Mbits/sec
[  5]   3.00-4.00   sec  27.2 MBytes   228 Mbits/sec
[  5]   4.00-5.00   sec  35.4 MBytes   297 Mbits/sec
[  5]   5.00-6.00   sec  35.3 MBytes   296 Mbits/sec
[  5]   6.00-7.00   sec  35.0 MBytes   294 Mbits/sec
[  5]   7.00-8.00   sec  35.1 MBytes   294 Mbits/sec
[  5]   8.00-9.00   sec  35.3 MBytes   296 Mbits/sec
[  5]   9.00-10.00  sec  35.1 MBytes   295 Mbits/sec
[  5]  10.00-11.00  sec  35.4 MBytes   297 Mbits/sec
[  5]  11.00-12.00  sec  35.3 MBytes   296 Mbits/sec
[  5]  12.00-13.00  sec  35.3 MBytes   296 Mbits/sec
[  5]  13.00-14.00  sec  35.4 MBytes   297 Mbits/sec
[  5]  14.00-15.00  sec  35.3 MBytes   296 Mbits/sec
[  5]  15.00-16.00  sec  35.4 MBytes   297 Mbits/sec
[  5]  16.00-17.00  sec  35.3 MBytes   296 Mbits/sec
[  5]  17.00-18.00  sec  27.0 MBytes   226 Mbits/sec
[  5]  18.00-19.00  sec  35.2 MBytes   296 Mbits/sec
[  5]  19.00-20.00  sec  35.1 MBytes   294 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   689 MBytes   289 Mbits/sec  5013             sender
[  5]   0.00-20.00  sec   688 MBytes   289 Mbits/sec                  receiver

iperf Done.
osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  5] local 172.16.11.109 port 38439 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  26.4 MBytes   221 Mbits/sec
[  5]   1.00-2.00   sec  34.9 MBytes   292 Mbits/sec
[  5]   2.00-3.00   sec  34.8 MBytes   292 Mbits/sec
[  5]   3.00-4.00   sec  34.7 MBytes   291 Mbits/sec
[  5]   4.00-5.00   sec  34.5 MBytes   289 Mbits/sec
[  5]   5.00-6.00   sec  34.8 MBytes   292 Mbits/sec
[  5]   6.00-7.00   sec  34.5 MBytes   290 Mbits/sec
[  5]   7.00-8.00   sec  34.8 MBytes   292 Mbits/sec
[  5]   8.00-9.00   sec  34.9 MBytes   292 Mbits/sec
[  5]   9.00-10.00  sec  35.1 MBytes   294 Mbits/sec
[  5]  10.00-11.00  sec  35.1 MBytes   294 Mbits/sec
[  5]  11.00-12.00  sec  34.3 MBytes   288 Mbits/sec
[  5]  12.00-13.00  sec  34.9 MBytes   293 Mbits/sec
[  5]  13.00-14.00  sec  34.8 MBytes   292 Mbits/sec
[  5]  14.00-15.00  sec  34.4 MBytes   288 Mbits/sec
[  5]  15.00-16.00  sec  34.7 MBytes   291 Mbits/sec
[  5]  16.00-17.00  sec  34.8 MBytes   292 Mbits/sec
[  5]  17.00-18.00  sec  34.8 MBytes   292 Mbits/sec
[  5]  18.00-19.00  sec  26.1 MBytes   219 Mbits/sec
[  5]  19.00-20.00  sec  34.8 MBytes   292 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   678 MBytes   284 Mbits/sec  3917             sender
[  5]   0.00-20.00  sec   678 MBytes   284 Mbits/sec                  receiver

iperf Done.

@sam_nazarko let me know what info I can provide that might help you find my order. It would be under the same email address as the other two in this thread if that helps. Although it’s probably out of warranty as it’s most likely been over a year.

fzinken · 23 December 2020 00:30

Install ethtool (sudo apt-get install ethtool) and then check with ethtool eth0

Your logs are a bit confusing. How long was the Vero running when you created the logs?
As it also seems you have a NTP issue (maybe related to the network issue).
When the value iperf value drops to the bad numbers does it also get better again after a while or require reboot?

ripley · 23 December 2020 00:56

Here’s the output of ethtool:

osmc@vero4k:~$ ethtool eth0
Settings for eth0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
        Current message level: 0x0000003d (61)
                               drv link timer ifdown ifup
        Link detected: yes

Here’s a new set of logs - https://paste.osmc.tv/azofivilap

This was the uptime when I grabbed them:

osmc@vero4k:~$ uptime
 17:43:05 up  1:06,  1 user,  load average: 2.68, 2.64, 2.55

As for getting the speed back up I’ve not been able to get the speed up after any specific process, rebooting or otherwise.

This was immediately after a reboot:

osmc@vero4k:~$ uptime
 17:45:55 up 0 min,  1 user,  load average: 1.76, 0.50, 0.17
osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  5] local 172.16.11.36 port 59619 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  12.8 MBytes   108 Mbits/sec
[  5]   1.00-2.00   sec  1.99 MBytes  16.7 Mbits/sec
[  5]   2.00-3.00   sec  4.64 MBytes  38.9 Mbits/sec
[  5]   3.00-4.00   sec  2.67 MBytes  22.4 Mbits/sec
[  5]   4.00-5.00   sec  97.6 KBytes   799 Kbits/sec
[  5]   5.00-6.00   sec   592 KBytes  4.85 Mbits/sec
[  5]   6.00-7.00   sec  1.56 MBytes  13.1 Mbits/sec
[  5]   7.00-8.00   sec  2.00 MBytes  16.8 Mbits/sec
[  5]   8.00-9.00   sec  1.36 MBytes  11.4 Mbits/sec
[  5]   9.00-10.27  sec   276 KBytes  1.78 Mbits/sec
[  5]  10.27-11.00  sec   403 KBytes  4.51 Mbits/sec
[  5]  11.00-12.00  sec  28.3 KBytes   232 Kbits/sec
[  5]  12.00-13.00  sec  1.87 MBytes  15.7 Mbits/sec
[  5]  13.00-14.00  sec   255 KBytes  2.09 Mbits/sec
[  5]  14.00-15.00  sec   277 KBytes  2.27 Mbits/sec
[  5]  15.00-16.00  sec   440 KBytes  3.60 Mbits/sec
[  5]  16.00-17.00  sec  2.32 MBytes  19.5 Mbits/sec
[  5]  17.00-18.00  sec   199 KBytes  1.63 Mbits/sec
[  5]  18.00-19.00  sec  2.06 MBytes  17.3 Mbits/sec
[  5]  19.00-20.00  sec  1022 KBytes  8.37 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-19.62  sec  37.0 MBytes  15.8 Mbits/sec   56             sender
[  5]   0.00-20.00  sec  36.8 MBytes  15.4 Mbits/sec                  receiver

iperf Done.

What’s odd is that a little bit ago during testing I had consistent speeds to my FreeNAS box and was able to play back high bit rate 4k, for a little while. It’s now not working again and I have no idea what changed.

fzinken · 23 December 2020 01:05

Ok, the dropped packages in that log look reasonable
Also ethtool look ok
You always run iperf3 -R which is correct to test the Vero, but would be interesting to also see a test without -R to see if it is better.

ripley · 23 December 2020 01:22

Here’s back to back runs with and without -R to my FreeNAS server:

osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  5] local 172.16.11.36 port 59671 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   745 KBytes  6.10 Mbits/sec
[  5]   1.00-2.00   sec   294 KBytes  2.41 Mbits/sec
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   3.00-4.00   sec  2.83 KBytes  23.2 Kbits/sec
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   5.00-6.00   sec  25.5 KBytes   209 Kbits/sec
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  11.00-12.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  13.00-14.00  sec   525 KBytes  4.30 Mbits/sec
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  15.00-16.00  sec   109 KBytes   892 Kbits/sec
[  5]  16.00-17.00  sec   389 KBytes  3.19 Mbits/sec
[  5]  17.00-18.00  sec   609 KBytes  4.99 Mbits/sec
[  5]  18.00-19.00  sec   180 KBytes  1.47 Mbits/sec
[  5]  19.00-20.00  sec  31.1 KBytes   255 Kbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.89 MBytes  1.21 Mbits/sec   21             sender
[  5]   0.00-20.00  sec  2.84 MBytes  1.19 Mbits/sec                  receiver

iperf Done.
osmc@vero4k:~$ iperf3 -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
[  5] local 172.16.11.36 port 59673 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.02 MBytes  16.9 Mbits/sec   57   2.83 KBytes
[  5]   1.00-2.00   sec  1.09 MBytes  9.11 Mbits/sec   30   1.41 KBytes
[  5]   2.00-3.00   sec   421 KBytes  3.45 Mbits/sec   19   2.83 KBytes
[  5]   3.00-4.00   sec   492 KBytes  4.03 Mbits/sec   14   2.83 KBytes
[  5]   4.00-5.00   sec  2.09 MBytes  17.6 Mbits/sec   51   2.83 KBytes
[  5]   5.00-6.00   sec   639 KBytes  5.24 Mbits/sec   22   9.90 KBytes
[  5]   6.00-7.00   sec  1.24 MBytes  10.4 Mbits/sec   29   2.83 KBytes
[  5]   7.00-8.00   sec  1.33 MBytes  11.1 Mbits/sec   38   2.83 KBytes
[  5]   8.00-9.00   sec   150 KBytes  1.23 Mbits/sec    9   1.41 KBytes
[  5]   9.00-10.00  sec  1.68 MBytes  14.1 Mbits/sec   37   2.83 KBytes
[  5]  10.00-11.00  sec   269 KBytes  2.20 Mbits/sec   17   2.83 KBytes
[  5]  11.00-12.00  sec  1.34 MBytes  11.3 Mbits/sec   32   5.66 KBytes
[  5]  12.00-13.00  sec  1.38 MBytes  11.6 Mbits/sec   43   2.83 KBytes
[  5]  13.00-14.00  sec  2.13 MBytes  17.9 Mbits/sec   52   2.83 KBytes
[  5]  14.00-15.00  sec   554 KBytes  4.54 Mbits/sec   21   2.83 KBytes
[  5]  15.00-16.00  sec  1.76 MBytes  14.8 Mbits/sec   42   2.83 KBytes
[  5]  16.00-17.00  sec  1.37 MBytes  11.5 Mbits/sec   47   2.83 KBytes
[  5]  17.00-18.00  sec  1.13 MBytes  9.46 Mbits/sec   48   1.41 KBytes
[  5]  18.00-19.00  sec   724 KBytes  5.93 Mbits/sec   23   2.83 KBytes
[  5]  19.00-20.00  sec   954 KBytes  7.82 Mbits/sec   33   2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  22.7 MBytes  9.50 Mbits/sec  664             sender
[  5]   0.00-20.32  sec  22.6 MBytes  9.33 Mbits/sec                  receiver

iperf Done.

Here’s another to a laptop connected to my core switch:

osmc@vero4k:~$ iperf3 -t 20 -c 172.16.11.133
Connecting to host 172.16.11.133, port 5201
[  5] local 172.16.11.36 port 58762 connected to 172.16.11.133 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.33 MBytes  19.6 Mbits/sec   84   1.43 KBytes
[  5]   1.00-2.00   sec   699 KBytes  5.72 Mbits/sec   43   4.28 KBytes
[  5]   2.00-3.00   sec   321 KBytes  2.63 Mbits/sec   24   2.85 KBytes
[  5]   3.00-4.00   sec  1015 KBytes  8.32 Mbits/sec   46   5.70 KBytes
[  5]   4.00-5.00   sec   264 KBytes  2.16 Mbits/sec   16   2.85 KBytes
[  5]   5.00-6.00   sec   778 KBytes  6.38 Mbits/sec   28   5.70 KBytes
[  5]   6.00-7.00   sec   403 KBytes  3.30 Mbits/sec   22   2.85 KBytes
[  5]   7.00-8.00   sec  1017 KBytes  8.33 Mbits/sec   53   2.85 KBytes
[  5]   8.00-9.00   sec  1.43 MBytes  12.0 Mbits/sec   50   2.85 KBytes
[  5]   9.00-10.00  sec   449 KBytes  3.68 Mbits/sec   26   2.85 KBytes
[  5]  10.00-11.00  sec   184 KBytes  1.51 Mbits/sec   13   4.28 KBytes
[  5]  11.00-12.00  sec   583 KBytes  4.78 Mbits/sec   29   2.85 KBytes
[  5]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec    1   1.43 KBytes
[  5]  13.00-14.00  sec  1.34 MBytes  11.2 Mbits/sec   37   4.28 KBytes
[  5]  14.00-15.00  sec   394 KBytes  3.22 Mbits/sec   23   1.43 KBytes
[  5]  15.00-16.00  sec   205 KBytes  1.68 Mbits/sec   13   2.85 KBytes
[  5]  16.00-17.00  sec  1.19 MBytes  9.99 Mbits/sec   36   1.43 KBytes
[  5]  17.00-18.00  sec   472 KBytes  3.87 Mbits/sec   25   5.70 KBytes
[  5]  18.00-19.00  sec   870 KBytes  7.13 Mbits/sec   46   4.28 KBytes
[  5]  19.00-20.00  sec  1.48 MBytes  12.4 Mbits/sec   61   2.85 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  15.2 MBytes  6.39 Mbits/sec  676             sender
[  5]   0.00-20.00  sec  15.1 MBytes  6.35 Mbits/sec                  receiver

iperf Done.
osmc@vero4k:~$ iperf3 -t 20 -R -c 172.16.11.133
Connecting to host 172.16.11.133, port 5201
Reverse mode, remote host 172.16.11.133 is sending
[  5] local 172.16.11.36 port 58765 connected to 172.16.11.133 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   106 MBytes   888 Mbits/sec
[  5]   1.00-2.00   sec  19.3 MBytes   162 Mbits/sec
[  5]   2.00-3.00   sec   198 KBytes  1.62 Mbits/sec
[  5]   3.00-4.00   sec  1.45 MBytes  12.1 Mbits/sec
[  5]   4.00-5.00   sec   951 KBytes  7.79 Mbits/sec
[  5]   5.00-6.00   sec   861 KBytes  7.05 Mbits/sec
[  5]   6.00-7.00   sec  39.9 KBytes   327 Kbits/sec
[  5]   7.00-8.00   sec  25.7 KBytes   210 Kbits/sec
[  5]   8.00-9.00   sec   130 KBytes  1.06 Mbits/sec
[  5]   9.00-10.00  sec  4.28 KBytes  35.0 Kbits/sec
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  11.00-12.00  sec  0.00 Bytes  0.00 bits/sec
[  5]  12.00-13.00  sec   478 KBytes  3.91 Mbits/sec
[  5]  13.00-14.00  sec   379 KBytes  3.11 Mbits/sec
[  5]  14.00-15.00  sec   546 KBytes  4.47 Mbits/sec
[  5]  15.00-16.00  sec  25.7 KBytes   210 Kbits/sec
[  5]  16.00-17.00  sec  62.7 KBytes   514 Kbits/sec
[  5]  17.00-18.00  sec   549 KBytes  4.50 Mbits/sec
[  5]  18.00-19.00  sec  11.4 KBytes  93.4 Kbits/sec
[  5]  19.00-20.00  sec   646 KBytes  5.29 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-20.00  sec   132 MBytes  55.3 Mbits/sec                  sender
[  5]   0.00-20.00  sec   132 MBytes  55.2 Mbits/sec                  receiver

iperf Done.

fzinken · 23 December 2020 01:29

Just for testing would be interesting if it is stable if you force 100Mbit
ethtool -s eth0 speed 100 duplex full autoneg off

Also as your problem is in both direction it is not the 2018 early Vero4k batch problem which was only in receiving direction.

ripley · 23 December 2020 01:47

For comparison I ran the same test using the USB NIC:

osmc@vero4k:~$ iperf3 -R -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
Reverse mode, remote host 172.16.11.40 is sending
[  5] local 172.16.11.109 port 43926 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  34.3 MBytes   288 Mbits/sec
[  5]   1.00-2.00   sec  34.4 MBytes   289 Mbits/sec
[  5]   2.00-3.00   sec  34.6 MBytes   290 Mbits/sec
[  5]   3.00-4.00   sec  26.1 MBytes   219 Mbits/sec
[  5]   4.00-5.00   sec  34.2 MBytes   287 Mbits/sec
[  5]   5.00-6.00   sec  34.4 MBytes   289 Mbits/sec
[  5]   6.00-7.00   sec  34.5 MBytes   289 Mbits/sec
[  5]   7.00-8.00   sec  34.7 MBytes   291 Mbits/sec
[  5]   8.00-9.00   sec  34.6 MBytes   290 Mbits/sec
[  5]   9.00-10.00  sec  22.1 MBytes   185 Mbits/sec
[  5]  10.00-11.00  sec  30.8 MBytes   258 Mbits/sec
[  5]  11.00-12.00  sec  26.2 MBytes   220 Mbits/sec
[  5]  12.00-13.00  sec  26.4 MBytes   221 Mbits/sec
[  5]  13.00-14.00  sec  34.6 MBytes   290 Mbits/sec
[  5]  14.00-15.00  sec  26.3 MBytes   220 Mbits/sec
[  5]  15.00-16.00  sec  26.3 MBytes   221 Mbits/sec
[  5]  16.00-17.00  sec  17.8 MBytes   149 Mbits/sec
[  5]  17.00-18.00  sec  32.8 MBytes   275 Mbits/sec
[  5]  18.00-19.00  sec  28.5 MBytes   239 Mbits/sec
[  5]  19.00-20.00  sec  34.5 MBytes   290 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   608 MBytes   255 Mbits/sec  3964             sender
[  5]   0.00-20.00  sec   608 MBytes   255 Mbits/sec                  receiver

iperf Done.
osmc@vero4k:~$ iperf3 -t 20 -c 172.16.11.40
Connecting to host 172.16.11.40, port 5201
[  5] local 172.16.11.109 port 43928 connected to 172.16.11.40 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  22.6 MBytes   190 Mbits/sec    0    161 KBytes
[  5]   1.00-2.00   sec  22.0 MBytes   185 Mbits/sec    0    161 KBytes
[  5]   2.00-3.00   sec  22.2 MBytes   186 Mbits/sec    0    161 KBytes
[  5]   3.00-4.00   sec  22.1 MBytes   185 Mbits/sec    0    163 KBytes
[  5]   4.00-5.00   sec  22.2 MBytes   186 Mbits/sec    0    177 KBytes
[  5]   5.00-6.00   sec  21.9 MBytes   184 Mbits/sec    0    177 KBytes
[  5]   6.00-7.00   sec  22.3 MBytes   187 Mbits/sec    0    178 KBytes
[  5]   7.00-8.00   sec  22.1 MBytes   185 Mbits/sec    0    178 KBytes
[  5]   8.00-9.00   sec  22.0 MBytes   184 Mbits/sec    0    178 KBytes
[  5]   9.00-10.00  sec  22.1 MBytes   186 Mbits/sec    0    178 KBytes
[  5]  10.00-11.00  sec  22.2 MBytes   186 Mbits/sec    0    178 KBytes
[  5]  11.00-12.00  sec  22.1 MBytes   185 Mbits/sec    0    178 KBytes
[  5]  12.00-13.00  sec  22.0 MBytes   185 Mbits/sec    0    178 KBytes
[  5]  13.00-14.00  sec  22.1 MBytes   186 Mbits/sec    0    178 KBytes
[  5]  14.00-15.00  sec  21.9 MBytes   184 Mbits/sec    0    178 KBytes
[  5]  15.00-16.00  sec  22.3 MBytes   187 Mbits/sec    0    178 KBytes
[  5]  16.00-17.00  sec  22.1 MBytes   185 Mbits/sec    0    178 KBytes
[  5]  17.00-18.00  sec  22.0 MBytes   185 Mbits/sec    0    178 KBytes
[  5]  18.00-19.00  sec  22.0 MBytes   184 Mbits/sec    0    178 KBytes
[  5]  19.00-20.00  sec  22.2 MBytes   186 Mbits/sec    0    178 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   442 MBytes   186 Mbits/sec    0             sender
[  5]   0.00-20.01  sec   442 MBytes   185 Mbits/sec                  receiver

iperf Done.

sam_nazarko · 23 December 2020 03:01

Hi @ripley

I checked your order, and it was placed on November 3, 2018.
This would have been after the Ethernet issues that affected some Vero 4K + devices.

I’ll admit that I’ve somewhat skim-read the thread, but are you able to achieve a 900Mbps speed with iperf? It looks like you were. Affected devices weren’t able to get anywhere near that – usually with upload (TX) speeds in the order of mere killobits.

If you’re able to sometimes get the right speeds, I’d suspect that the issue isn’t hardware related, or at least not a Vero 4K + issue.

Can you test iperf in both directions? Is only one direction impeded?

fzinken · 23 December 2020 03:27

Both Directions, so seems to be different to the Vero4k+ early batch issue.