Buffer/Cache Issues When Skipping 4K UHD x265 HDR file

lmoerenhout · 1 April 2020 09:56

Hi all, hope everyone is doing okay at the moment!

Using a Vero 4K+ as my player/client. Just picked up a new Raspberry Pi 4 to try set up an NFS server that is gigabit capable.

Server details:

NFS Server
Ethernet cable through the gigabit port
ExFAT formatted drive (plugged into the USB 3.0 port)
Mounted on Vero 4K+ with fstab

Video playback seems to work for the most part, the only issue I’m experiencing is when I try to skip/seek forward in a large UHD x265 HDR file and the video comes to a halt with a cache overlay appearing. I’ve spent ages searching the forum, and people seem to say they have seamless instantaneous playback.

I’ve run iperf3 tests both ways, shown below:

osmc@LACH-VERO:~$ iperf3 -R -c 192.168.1.7
Connecting to host 192.168.1.7, port 5201
Reverse mode, remote host 192.168.1.7 is sending
[ 4] local 192.168.1.6 port 44604 connected to 192.168.1.7 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 108 MBytes 905 Mbits/sec
[ 4] 1.00-2.00 sec 111 MBytes 932 Mbits/sec
[ 4] 2.00-3.00 sec 111 MBytes 934 Mbits/sec
[ 4] 3.00-4.00 sec 112 MBytes 937 Mbits/sec
[ 4] 4.00-5.00 sec 112 MBytes 939 Mbits/sec
[ 4] 5.00-6.00 sec 112 MBytes 940 Mbits/sec
[ 4] 6.00-7.00 sec 112 MBytes 940 Mbits/sec
[ 4] 7.00-8.00 sec 112 MBytes 940 Mbits/sec
[ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec
[ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec receiver

iperf Done.
osmc@LACH-VERO:~$ iperf3 -c 192.168.1.7
Connecting to host 192.168.1.7, port 5201
[ 4] local 192.168.1.6 port 44606 connected to 192.168.1.7 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 104 MBytes 869 Mbits/sec 0 2.84 MBytes
[ 4] 1.00-2.02 sec 104 MBytes 854 Mbits/sec 0 2.84 MBytes
[ 4] 2.02-3.00 sec 107 MBytes 918 Mbits/sec 0 2.05 MBytes
[ 4] 3.00-4.00 sec 102 MBytes 861 Mbits/sec 0 2.13 MBytes
[ 4] 4.00-5.00 sec 103 MBytes 863 Mbits/sec 0 2.17 MBytes
[ 4] 5.00-6.00 sec 109 MBytes 917 Mbits/sec 5 1.68 MBytes
[ 4] 6.00-7.00 sec 98.8 MBytes 828 Mbits/sec 0 600 KBytes
[ 4] 7.00-8.00 sec 106 MBytes 892 Mbits/sec 0 631 KBytes
[ 4] 8.00-9.00 sec 110 MBytes 918 Mbits/sec 0 682 KBytes
[ 4] 9.00-10.01 sec 105 MBytes 876 Mbits/sec 0 491 KBytes

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.01 sec 1.02 GBytes 880 Mbits/sec 5 sender
[ 4] 0.00-10.01 sec 1.02 GBytes 880 Mbits/sec receiver

iperf Done.

I’ve also tried dd to read a dummy file:

osmc@LACH-VERO:/mnt/LACH_SERVER$ dd if=tempfile of=/dev/null status=progress
1064387072 bytes (1.1 GB, 1015 MiB) copied, 20.0181 s, 53.2 MB/s
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 20.2814 s, 52.9 MB/s

All of these outputs seem to suggest I shouldn’t have any issues, but i’m still learning so happy to keep chugging away to understand what’s going on.

(Log can be found here), I’ve booted the device, started playblack, let it run for a couple seconds then tried to skip 50 minutes.

Please let me know if there’s anything else I can provide to assist. Thanks heaps!

fzinken · 1 April 2020 10:10

While this should be more than enough for any playback the 880 Mbit is not a super healthy number and might indicate issues in the network. Maybe try a longer test as explained in the iperf3 thread.

While it should not impact your seeking it anyhow is recommended that you set the GUI to 1080 not 4k.

You write “cache overlay” I assume you mean the message “buffering…”?

lmoerenhout · 1 April 2020 10:13

Thanks for the speedy reply, I’ll run the longer iperf3 now.

Sorry if it wasn’t clear. The video freezes, the screen goes dim, i downward arrow appears along with the text “cache 0%”. Does this sound familiar? I can provide a screenshot if necessary.

lmoerenhout · 1 April 2020 10:26

Longer iperf3 tests are below. As a side note, I tried to take a screenshot to show you the cache on-screen message, and when trying to saving it to the NFS Server it was as if the server had coming to a screaming halt. Even on my laptop I couldn’t navigate anywhere. The moment I stopped video playback, everything resumed to normal.

osmc@LACH-VERO:~$ iperf3 -t 60 -R -c 192.168.1.7
Connecting to host 192.168.1.7, port 5201
Reverse mode, remote host 192.168.1.7 is sending
[ 4] local 192.168.1.6 port 60547 connected to 192.168.1.7 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 103 MBytes 861 Mbits/sec
[ 4] 1.00-2.00 sec 112 MBytes 937 Mbits/sec
[ 4] 2.00-3.00 sec 112 MBytes 940 Mbits/sec
[ 4] 3.00-4.00 sec 112 MBytes 939 Mbits/sec
[ 4] 4.00-5.00 sec 112 MBytes 940 Mbits/sec
[ 4] 5.00-6.00 sec 112 MBytes 941 Mbits/sec
…This was continuous for the rest of the test

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 6.55 GBytes 938 Mbits/sec 36 sender
[ 4] 0.00-60.00 sec 6.55 GBytes 938 Mbits/sec receiver

iperf Done.
osmc@LACH-VERO:~$ iperf3 -t 60 -c 192.168.1.7
Connecting to host 192.168.1.7, port 5201
[ 4] local 192.168.1.6 port 60549 connected to 192.168.1.7 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 107 MBytes 896 Mbits/sec 0 2.84 MBytes
[ 4] 1.00-2.01 sec 96.2 MBytes 804 Mbits/sec 0 1.99 MBytes
[ 4] 2.01-3.00 sec 100 MBytes 841 Mbits/sec 0 2.01 MBytes
[ 4] 3.00-4.00 sec 102 MBytes 861 Mbits/sec 0 2.22 MBytes
[ 4] 4.00-5.00 sec 105 MBytes 880 Mbits/sec 0 2.39 MBytes
[ 4] 5.00-6.00 sec 102 MBytes 862 Mbits/sec 0 2.45 MBytes
[ 4] 6.00-7.00 sec 102 MBytes 856 Mbits/sec 0 2.61 MBytes
[ 4] 7.00-8.01 sec 109 MBytes 909 Mbits/sec 0 2.71 MBytes
[ 4] 8.01-9.02 sec 101 MBytes 841 Mbits/sec 0 1.34 MBytes
[ 4] 9.02-10.00 sec 105 MBytes 895 Mbits/sec 0 1024 KBytes
[ 4] 10.00-11.00 sec 110 MBytes 923 Mbits/sec 0 1.06 MBytes
[ 4] 11.00-12.02 sec 108 MBytes 883 Mbits/sec 0 1.10 MBytes
[ 4] 12.02-13.01 sec 97.5 MBytes 828 Mbits/sec 0 386 KBytes
[ 4] 13.01-14.00 sec 101 MBytes 858 Mbits/sec 0 297 KBytes
[ 4] 14.00-15.01 sec 98.8 MBytes 822 Mbits/sec 0 325 KBytes
[ 4] 15.01-16.01 sec 98.8 MBytes 827 Mbits/sec 0 341 KBytes
[ 4] 16.01-17.01 sec 97.5 MBytes 819 Mbits/sec 0 249 KBytes
[ 4] 17.01-18.01 sec 98.8 MBytes 833 Mbits/sec 0 230 KBytes
[ 4] 18.01-19.00 sec 112 MBytes 944 Mbits/sec 0 351 KBytes
[ 4] 19.00-20.00 sec 110 MBytes 922 Mbits/sec 0 447 KBytes
[ 4] 20.00-21.01 sec 110 MBytes 915 Mbits/sec 0 527 KBytes
[ 4] 21.01-22.01 sec 98.8 MBytes 828 Mbits/sec 0 551 KBytes
[ 4] 22.01-23.01 sec 98.8 MBytes 830 Mbits/sec 0 403 KBytes
[ 4] 23.01-24.00 sec 105 MBytes 886 Mbits/sec 0 491 KBytes
[ 4] 24.00-25.00 sec 108 MBytes 907 Mbits/sec 0 522 KBytes
[ 4] 25.00-26.00 sec 110 MBytes 922 Mbits/sec 0 540 KBytes
[ 4] 26.00-27.01 sec 104 MBytes 867 Mbits/sec 0 581 KBytes
[ 4] 27.01-28.01 sec 98.8 MBytes 828 Mbits/sec 0 407 KBytes
[ 4] 28.01-29.01 sec 105 MBytes 874 Mbits/sec 0 354 KBytes
[ 4] 29.01-30.00 sec 97.5 MBytes 828 Mbits/sec 0 358 KBytes
[ 4] 30.00-31.03 sec 102 MBytes 837 Mbits/sec 0 263 KBytes
[ 4] 31.03-32.01 sec 98.8 MBytes 845 Mbits/sec 0 311 KBytes
[ 4] 32.01-33.00 sec 102 MBytes 864 Mbits/sec 0 214 KBytes
[ 4] 33.00-34.00 sec 98.8 MBytes 828 Mbits/sec 0 267 KBytes
[ 4] 34.00-35.00 sec 110 MBytes 922 Mbits/sec 0 373 KBytes
[ 4] 35.00-36.00 sec 109 MBytes 917 Mbits/sec 0 452 KBytes
[ 4] 36.00-37.00 sec 104 MBytes 868 Mbits/sec 0 460 KBytes
[ 4] 37.00-38.00 sec 106 MBytes 888 Mbits/sec 0 496 KBytes
[ 4] 38.00-39.00 sec 110 MBytes 923 Mbits/sec 0 568 KBytes
[ 4] 39.00-40.00 sec 110 MBytes 921 Mbits/sec 0 632 KBytes
[ 4] 40.00-41.00 sec 110 MBytes 924 Mbits/sec 0 689 KBytes
[ 4] 41.00-42.00 sec 105 MBytes 882 Mbits/sec 0 740 KBytes
[ 4] 42.00-43.00 sec 98.8 MBytes 827 Mbits/sec 0 740 KBytes
[ 4] 43.00-44.01 sec 98.8 MBytes 825 Mbits/sec 0 740 KBytes
[ 4] 44.01-45.01 sec 100 MBytes 835 Mbits/sec 0 745 KBytes
[ 4] 45.01-46.01 sec 97.5 MBytes 820 Mbits/sec 0 368 KBytes
[ 4] 46.01-47.00 sec 102 MBytes 864 Mbits/sec 0 296 KBytes
[ 4] 47.00-48.00 sec 107 MBytes 898 Mbits/sec 0 386 KBytes
[ 4] 48.00-49.00 sec 110 MBytes 927 Mbits/sec 0 475 KBytes
[ 4] 49.00-50.00 sec 110 MBytes 921 Mbits/sec 0 550 KBytes
[ 4] 50.00-51.00 sec 110 MBytes 923 Mbits/sec 0 615 KBytes
[ 4] 51.00-52.00 sec 110 MBytes 922 Mbits/sec 0 675 KBytes
[ 4] 52.00-53.00 sec 109 MBytes 919 Mbits/sec 0 728 KBytes
[ 4] 53.00-54.00 sec 102 MBytes 859 Mbits/sec 0 759 KBytes
[ 4] 54.00-55.01 sec 101 MBytes 841 Mbits/sec 0 272 KBytes
[ 4] 55.01-56.01 sec 98.8 MBytes 831 Mbits/sec 0 298 KBytes
[ 4] 56.01-57.00 sec 105 MBytes 883 Mbits/sec 0 240 KBytes
[ 4] 57.00-58.00 sec 110 MBytes 918 Mbits/sec 0 365 KBytes
[ 4] 58.00-59.00 sec 110 MBytes 923 Mbits/sec 0 458 KBytes
[ 4] 59.00-60.00 sec 110 MBytes 923 Mbits/sec 0 535 KBytes

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 6.11 GBytes 874 Mbits/sec 0 sender
[ 4] 0.00-60.00 sec 6.10 GBytes 874 Mbits/sec receiver

iperf Done.

fzinken · 1 April 2020 10:40

What are the logs on the NAS say?

lmoerenhout · 1 April 2020 10:53

Sorry, I might need a little bit of guidance. It’s just an NFS server running on a Raspberry pi, what logs should I be looking for?

For reference, I used the following commands to set up my NFS server, if you have any suggestions I’m more than happy to hear them!

sudo mkdir /mnt/LACH_SERVER

sudo nano /etc/exports
/mnt/LACH_SERVER 192.168.1.0/24(rw,all_squash,async,no_subtree_check,anonuid=1000,anongid=1000)

sudo nano /etc/fstab
/dev/sda1 /mnt/LACH_SERVER auto defaults,user 0 1

sudo exportfs -ra

I then mounted it to Vero 4K+ as below:
sudo nano /etc/fstab
192.168.1.7:/mnt/LACH_SERVER /mnt/LACH_SERVER nfs noauto,x-systemd.automount 0 0

fzinken · 1 April 2020 11:04

I am not a NFS Server expert. You might check what is in the journalctl. But for real logging you might need to enable logging for the nfs server.

Ok, so then you might want to check CPU and IO’s during that time. Because the Pi using a extfat drive connected via USB might get at it’s limits in that scenario.

You could also try just a test with copying a file with dd to /dev/null on the Vero to determine the speed.

lmoerenhout · 1 April 2020 11:13

Thanks for all your information, really appreciate the assistance. I’d previously sent through the below dd test which was run on the vero, is that of any help?

osmc@LACH-VERO:/mnt/LACH_SERVER$ dd if=tempfile of=/dev/null status=progress
1064387072 bytes (1.1 GB, 1015 MiB) copied, 20.0181 s, 53.2 MB/s
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 20.2814 s, 52.9 MB/s

mtv · 1 April 2020 11:14

perhaps this is something to look into:

understood from this that @Tom_Doyle and @bmillham created a setup for an NFS server on OSMC, perhaps it could work also on your PI4, just check it out.

lmoerenhout · 1 April 2020 11:16

Thanks mate, I’ll definitely have a look into that one!

mtv · 1 April 2020 11:20

see also on your PI, when watching the movie on the vero, to see waht CPU and IO are doing, that’s what @fzinken mentioned I guess to see if the PI is able to perform those activities

fzinken · 1 April 2020 11:21

That looks quite promising while I still would check the CPU and IO usage when you skip forward.

Tom_Doyle · 1 April 2020 11:26

Hi @mtv,

Thanks for bigging up the post, but I should point out that I don’t believe the script will work for automounted shares in this case as I don’t think raspbian automounts usb shares under /media; I may be wrong but I think its mounts them under under a child directory in the user’s (pi) home directory. But thanks again anyway.

@lmoerenhout,

Could you please post your /etc/exports from the pi.

Thanks Tom.

lmoerenhout · 1 April 2020 11:29

Hi Tom, /etc/exports on the pi are:
/mnt/LACH_SERVER 192.168.1.0/24(rw,all_squash,async,no_subtree_check,anonuid=1000,anongid=1000)

Tom_Doyle · 1 April 2020 11:46

Hi,

That looks fine. Is this happening for multiple UHD x265 files or just the one showing in the log?

Also a bit a longshot, but tweaking the readfactor for video cache may could. Please create on vero4k+ /home/osmc/.kodi/userdata/advancedsettings.xml containing:

<advancedsettings.xml>
	<cache>
		<readfactor>6.0</readfactor>
	</cache>
<advancedsettings.xml>

Thanks Tom.

Smurphy · 1 April 2020 11:52

The ability to fast forward etc. always depends on the balance of Source, Network and Target.
If one of these elements is slow, it will determine the overall speed of data delivery.

I had the same issue with a QNap NAS using a N1600 Atom CPU + 2GB ram + 8x2Tb configured as Raid5.
That was not enough to provide HD Data correctly. The CPU was not always at 100%, but as soon as another process wanted the CPU’s attention, transfers started to suffer. I had fine-tuned the speed and I/O ability taking the Raid5 into account. The problem was I/O on Disk + network were not in sync and caused the first level of latencies. After that came the CPU, and last but not least the caching capability. 2GB of Ram is not much. I could fast forward some seconds then it got stuck. The system did not deliver data fast enough for playing around. Especially since we have 5 media players in house.

In fact, depending on the entire data delivery chain, it can work but does not have to.
Currently, I have a NAS with Raid 5 over 3+1 setup with 4Tb 7200HDD with a Xeon E3-1220L V2 @ 2.30GHz + 16Gb Ram and the 2 network interface on the Nas configured on NAS and Switch side in Bonding-configuration providing 2Gbps of data total (actually, max stream speed on 1Gbps, but if one physical lane is full, it will use the other one).
And this setup provides all I need. However, if I fast forward too fast (16x) for too long, I starts stuttering again (Buffers -> Disk and sometimes network speed)

IMHO, the raspberry PI will eventually provide you enough speed for regular playback, but for nothing else. I don’t know if the Network and USB Bus has enough buffers to keep interrupt requests to the CPU low.
Check on your PI if /proc/interrupts goes out of the roof when streaming data.

lmoerenhout · 1 April 2020 12:07

Thanks so much for everyone’s assistance, I think I might’ve learnt quite a bit from the process!

I followed @fzinken’s great advice, and elected to use nmon on the pi.

The below image shows the Pi 4 at Idle:

And the second is when I attempt the 50 minute skip.

So it seems the CPU is the bottleneck here.

I ran a few extra tests, and didn’t encounter issues betwen 10second and 30 minute skips, but from 40 onwards it bottomed out.

I guess this opens the door for overclocking and the like, but for now I can live with this resolution. Thanks to all for the prompt feedback, great community!

Stay safe everyone!

fzinken · 1 April 2020 12:12

Well you might want to consider formatting the drive as ext2 instead of exfat which might solve your issue

lmoerenhout · 1 April 2020 12:16

Ohh that’s a cool suggestion! Is there a reason I would want to use ext2 ahead of ext4? Just had a quick google, and it seems ext4 is the more modern, but i’m curious if there are benefits to ext2 that i’m missing!

fzinken · 1 April 2020 12:39

That is correct, but ext4 by default is a journaling filesystem which takes unnecessary performance on a fairly steady harddisk.

But to be honest both ext2 or ext4 will be fine.