Problems with LiveTV stuttering on Vero 4k+

Help and Support Vero 4K / Vero 4K +
21

This does not affect recordings which playback perfectly. Here are a couple of points i’d like to re-iterate from my initial post

  • I have used TVHeadend on this server for the past 2-3 years without these stuttering problems on other clients.

  • I am also running kodi on an PRi3 in another room, with latest stable LibreELEC v8.2.5 - PVR live playback is flawless on this machine

The same tvheadend server was previously running LibreELEC and kodi and this managed to cope much better with livetv. No stuttering at all.

I bought the vero4k+ as i wanted to be able to playback 4k video files. However I had thought that the ‘Basics’ of livetv playback would work just as well as with my old setup.

I suppose i could go back to my old setup for livetv and just have the vero4k+ for playback of 4k files, but i just cant see why the vero4k+ shouldnt handle these streams correctly.

22

Another thought. In my old setup, and on my rpi3 i had the adjust refresh rate set to off and had display at 1080p 50hz

On the vero 4k+ i have it set at 1080p 50hz with adjust refresh rate set to always. So that i can benefit from the 4k playback capabilities. Could this be a contributory factor? I.e would it cope better with dropped frames.

23

Having a look at the networking layer, I seem to be getting dropped packets on the vero4k+ side.

This is the tvheadend server - Server has been up 3 days, and zero dropped TX packets.

james@mediapc:~$ netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
ens35     1500 19088039      0     19 0      202308915      0      0      0 BMRU
lo       65536   359067      0      0 0        359067      0      0      0 LRU

Here is the vero4k+. I rebooted before running further tests last night. During those tests I could see I had 2 stutters. This seems to tally with the 2 dropped RX packets I can see below:

osmc@vero4k:~$ netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500 11919104      0      2 0        904139      0      0      0 BMdRU
lo        4096     1888      0      0 0          1888      0      0      0 LRU

And on the rpi3, which has been running for 40 days, with 0 dropped packets.

eth0      Link encap:Ethernet  HWaddr B8:27:EB:15:25:73
          inet addr:192.168.0.12  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41502746 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22350387 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3035816658 (2.8 GiB)  TX bytes:1856999481 (1.7 GiB)

(LibreELEC has a trimmed down netstat which doesn’t have the -i flag - so I used ifconfig instead)

Should I try tweaking some sysctl parameters on the vero4k+, such as net.core.rmem_default? I’ve never really needed to tweak such parameters on other machines, so I’d appreciate some advice.

I’m currently on the video improvements testing kernel 121

24

Have you checked the throughput with iperf3?

25

Yes, I mentioned this in my original post.

After seeing the various posts from other users about problems with the gigabit network card on some units, this was one of my first suspicions.

Anyhow, here’s some output to help illustrate.

tvheadend server → vero4k+

james@mediapc:~$ iperf3 -c 192.168.0.147
Connecting to host 192.168.0.147, port 5201
[  4] local 192.168.0.6 port 53766 connected to 192.168.0.147 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.01   sec   100 MBytes   835 Mbits/sec    0    495 KBytes
[  4]   1.01-2.01   sec   111 MBytes   933 Mbits/sec    2    397 KBytes
[  4]   2.01-3.00   sec   111 MBytes   933 Mbits/sec    0    440 KBytes
[  4]   3.00-4.01   sec   105 MBytes   877 Mbits/sec    0    465 KBytes
[  4]   4.01-5.00   sec   107 MBytes   903 Mbits/sec    0    485 KBytes
[  4]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    491 KBytes
[  4]   6.00-7.00   sec   112 MBytes   942 Mbits/sec    0    491 KBytes
[  4]   7.00-8.00   sec   111 MBytes   932 Mbits/sec    0    495 KBytes
[  4]   8.00-9.00   sec   113 MBytes   941 Mbits/sec    0    495 KBytes
[  4]   9.00-10.00  sec   104 MBytes   873 Mbits/sec    0    499 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec    2             sender
[  4]   0.00-10.00  sec  1.06 GBytes   910 Mbits/sec                  receiver

iperf Done.

vero4k+ → tvheadend server

osmc@vero4k:~$ iperf3 -c 192.168.0.6
Connecting to host 192.168.0.6, port 5201
[  4] local 192.168.0.147 port 34644 connected to 192.168.0.6 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   112 MBytes   935 Mbits/sec    0   2.24 MBytes
[  4]   1.00-2.00   sec   111 MBytes   933 Mbits/sec    0   2.13 MBytes
[  4]   2.00-3.00   sec   111 MBytes   934 Mbits/sec    0   2.32 MBytes
[  4]   3.00-4.00   sec   112 MBytes   939 Mbits/sec    0   2.48 MBytes
[  4]   4.00-5.00   sec   111 MBytes   933 Mbits/sec    0   2.61 MBytes
[  4]   5.00-6.00   sec   112 MBytes   937 Mbits/sec    0   2.71 MBytes
[  4]   6.00-7.00   sec   111 MBytes   931 Mbits/sec    1   2.00 MBytes
[  4]   7.00-8.00   sec   111 MBytes   934 Mbits/sec    1   1.48 MBytes
[  4]   8.00-9.00   sec   111 MBytes   934 Mbits/sec    0   1.57 MBytes
[  4]   9.00-10.00  sec   111 MBytes   933 Mbits/sec    0   1.64 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec    2             sender
[  4]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec                  receiver

iperf Done.

I’ve previously ran these tests for half an hour, just to be sure, and have never seen any major drops in the speed. It’s always similar to the above.

The two machines are plugged into the same gigabit switch.

The Rpi3 in the other room is hooked upto this same switch via powerline adapters, and as mentioned, isn’t showing any dropped packets in past 40 days it’s been powered on.

26

Ok, that results look good and considering the duration that you have it run I would also say the Retries are OK.

27

It would be interesting to know if you got the stutters on a Pi running OSMC, but I suspect that may be tricky.

There are some Ethernet improvements in staging which improve RX performance

I’d appreciate it if you could test this and provide feedback before we potentially release this as an update to other users. To test this update:

  1. Login via the command line
  2. Edit the file /etc/apt/sources.list
  3. Add the following line: deb http://apt.osmc.tv stretch-devel main
  4. Run the following commands to update: sudo apt-get update && sudo apt-get dist-upgrade && reboot
  5. Your system should have have received the update.

Please see if the issue is resolved.

I also recommend you edit /etc/apt/sources.list again and remove the line that you added after updating. This will return you to the normal update channel.

28

As mentioned, I’m already on the stretch-devel channel as I’ve also been testing the video improvements kernel - see my initial post.

I’ve had this problem on various kernel versions (119, 119 plus video improvements, 121), and it still persists on the 121 kernel.

I checked for updates in stretch-devel a couple of days ago, and have just double-checked again, and can confirm I’m on the latest versions.

osmc@vero4k:~$ sudo apt-get update
Ign:1 http://ftp.debian.org/debian stretch InRelease
Hit:3 http://security.debian.org stretch/updates InRelease
Get:5 http://ftp.debian.org/debian stretch-updates InRelease [91.0 kB]
Hit:6 http://ftp.debian.org/debian stretch Release
Get:2 http://ftp.fau.de/osmc/osmc/apt stretch-devel InRelease [4,697 B]
Hit:4 http://ftp.fau.de/osmc/osmc/apt stretch InRelease
Get:8 http://ftp.fau.de/osmc/osmc/apt stretch-devel/main arm64 Packages [14.6 kB]
Get:9 http://ftp.fau.de/osmc/osmc/apt stretch-devel/main armhf Packages [25.0 kB]
Fetched 135 kB in 2s (58.8 kB/s)
Reading package lists... Done

osmc@vero4k:~$ sudo apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  vero364-image-3.14.29-117-osmc:arm64
Use 'sudo apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
osmc@vero4k:~$ uname -a
Linux vero4k 3.14.29-121-osmc #1 SMP Sat Sep 29 05:20:26 UTC 2018 aarch64 GNU/Linux

At some point I may move the RPi3 from LibreELEC to OSMC, but was planning to do this after Leia is released, since that machine is nice and stable currently.

29

Is Vero doing anything when you get the stutter? High CPU use may have caused it.

I will try reproduce the problem. It looks like a very occasional packet drop.

30

The machine looks fairly calm. Here is the cpu usage whilst playing livetv

top - 13:16:30 up  2:17,  1 user,  load average: 1.14, 1.26, 1.30
Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  0.4 sy,  0.0 ni, 98.1 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  1831348 total,  1079120 free,   374900 used,   377328 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1346712 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  801 osmc      20   0  854736 306128  29004 S  12.6 16.7  30:37.51 kodi.bin
 5534 osmc      20   0    7136   1564   1040 R   0.6  0.1   0:00.23 top
    7 root      20   0       0      0      0 S   0.3  0.0   0:19.96 rcu_sched
  361 avahi     20   0    5196   1456   1148 S   0.3  0.1   0:01.75 avahi-daemon
 5408 root      20   0       0      0      0 S   0.3  0.0   0:01.41 kworker/1:2
    1 root      20   0   26376   3624   2564 S   0.0  0.2   0:01.42 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:02.06 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.29 kworker/0:0H

I’m considering whether to try a different network cable to rule that out. It currently has a cat5e cable that I found in one of my drawers, and it looked fairly cheap. I think I may try swapping it over tonight with the one going out to the RPi3. (then if the RPi3 starts dropping packets, we’ll know this was the problem).

FTR: Dropped packets are still occurring. I’m not in the house but my wife is using the machine. It’s only been powered on for just over 2 hours

osmc@vero4k:~$ uptime
 13:17:25 up  2:18,  1 user,  load average: 1.12, 1.23, 1.28

(was restarted as we’ve had a new boiler installed today, and they needed to switch off our electrics).

So far it’s dropped 4 packets - this is the kind of frequency at which I’m encountering the stutters, so I’m reasonably confident that the dropped packets are to blame.

osmc@vero4k:~$ ifconfig
eth0: flags=-28605<UP,BROADCAST,RUNNING,MULTICAST,DYNAMIC>  mtu 1500
        inet 192.168.0.147  netmask 255.255.255.0  broadcast 192.168.0.255
        ether c4:4e:ac:28:57:a4  txqueuelen 1000  (Ethernet)
        RX packets 5506490  bytes 8129975541 (7.5 GiB)
        RX errors 0  dropped 4  overruns 0  frame 0
        TX packets 349964  bytes 25555120 (24.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 40

In the meantime, if there is any advice on how to further trace why these packets are being dropped, then this would help (not an issue I’ve ever needed to debug before). From what I just read, it depends on your network card drivers as to what detail is available, and how to extract that detail?

31

Finding out if/why a server is dropping packets looks like a good approach. I might need to build you a kernel with debugging symbols so that we can get more details however.

Sam

33

If you can’t reproduce the dropped frames in your environment, then I’d suggest that I firstly try swapping over the network cables to rule that out as a cause.

If I’m still getting dropped packets using the same network cable which works fine with the RPi3, then we can debug further?

34

I checked and I’m not getting dropped packets.
I have a few ‘borderline’ cables (intentionally badly crimped etc) for testing some corner cases.

Sounds like a plan.

35

I have observed this afternoon that my system load is always high - like always above 1 even though the machine is idle

load average: 1.48, 1.28, 1.24

On my tvheadend server this is pretty low:

load average: 0.00, 0.03, 0.00

On on my RPI3 it’s reasonable, also running kodi in an idle state

load average: 0.16, 0.07, 0.08

At the moment there is no video playing so I’ve no idea why it’s so high, and can’t see anything unreasonable in top.

top shows that kodi is usually around 10-20% when Idle (same on RPi3)
I’ve installed and used iotop but can’t see any processes running away with I/O in their either.

What is ‘normal’ load for an idle Vero4k+ ?

36

That seems fine.

Sam

37

Thanks for the confirmation, guess im used to seeing <1 from other systems i work with. I just stopped kodi and can see it flatline at ~1.0. So i guess that 1.0 must be the baseline on this distro.

38

Sorry to jump in but have you tried other stream settings on the tvheadend side.

htsp vs matroska vs pass

I have defaulted to matroska stream.

Configuration >> Stream >> stream profiles

39

If using kodi pvr.hts plugin, i think youll find it always uses htsp for live tv streaming no matter which stream profile you set as default. Play live tv and look in the status tab in the web ui and you’ll see what i mean. You can manually specify a profile to use in pvr.hts addon settings, but you can only use a htsp-based profile. As that feature is mainly so u can have different streaming priorities per device.

40

So…i changed the network cables over last night. Watched a 4k film and then left a hd channel playing over night. no dropped packets but could still see buffer timeout errors in the kodi log, so those could have been a red herring.

Will continue to monitor as ive not yet visually seen a stutter, but out of the house most this weekend.

If theres still stuttering then i may try upgrading tvheadend from 4.2.5->4.2.6 incase there are any fixes in there of intetest.

41

Keep us posted.

Sam