Freezing/stuttering/buffering issue when playing (variable) high bitrate file over local network

[logan893]

Hardware: Raspberry Pi 3, with 2.4 A power supply.
Software: OSMC 2016.06-02

Freshly installed, yesterday, on a SanDisk Mobile Ultra microSDHC 16 GB UHS-I (Class 10).

I’m having issues while playing certain movie files with very high variable bit rate video and audio via SMB. One particular file I attempt to play has (according to MediaInfo 0.7.87) roughly 5 Mbps audio (TrueHD) (although OSMC overlay says closer to 9 Mbps), and h264 (x264, AVC) video at an average 20.1 Mbps (max 30 according to encoding parameters). I’ve enabled hardware decoding, which when there is data in the buffer makes the video and audio flow perfectly. (With software decoding it’s not working as well, but that’s another issue I may address in a separate post.)

The issue is that the buffer is not refilled fast enough. Slowly it is depleted (vq and aq drop down to 0%), video freezes, buffer starts to refill (vq and aq increase), and only once the buffer has refilled to 90+% (vq and aq) playback starts again.

The buffering happens with varying frequency. To me it appears to vary somewhat depending on the current measured bit rate of the video file. Often playback stops for buffering with between 30-90 second intervals.

Using bcmstat.sh I see the RX rate is between 1.5-3.5 MB/s, with the occasional rare spike close to 5 MB/s. Testing file transfer independently from the servers on my network, I was able to saturate the 100 Mbps line to the RPi, so it’s not a network issue. It is also not an issue with the share providing the video, as I am able to stream the same file to my Windows PC with no impact on the concurrent behavior on the Raspberry Pi. It continues to get stuck buffering as often as before.

Saturating the RPi network interface via SCP:

Time ARM Core H264 Core Temp (Max) IRQ/s RX B/s TX B/s
======== ======= ======= ======= =============== ====== ========== ==========
21:36:35 1200Mhz 400Mhz 300Mhz 61.22C (62.30C) 11,118 11,587,196 248,669
21:36:37 1200Mhz 400Mhz 300Mhz 61.22C (62.30C) 10,912 11,955,859 215,072
21:36:39 1200Mhz 400Mhz 300Mhz 61.22C (62.30C) 11,017 11,946,929 224,658
21:36:41 1200Mhz 400Mhz 300Mhz 60.69C (62.30C) 10,831 10,648,593 269,694

Maximum transfer speed during playback, with stuttering, default settings:

Time ARM Core H264 Core Temp (Max) IRQ/s RX B/s TX B/s
======== ======= ======= ======= =============== ====== ========== ==========
22:28:49 1200Mhz 400Mhz 300Mhz 58.53C (60.15C) 3,353 2,571,254 83,588
22:28:51 1200Mhz 400Mhz 300Mhz 58.53C (60.15C) 3,677 2,764,906 91,712
22:28:53 1200Mhz 400Mhz 300Mhz 58.00C (60.15C) 5,645 4,791,837 155,941
22:28:55 1199Mhz 400Mhz 300Mhz 58.53C (60.15C) 2,645 1,726,276 61,153
22:28:57 1200Mhz 400Mhz 300Mhz 58.53C (60.15C) 3,256 2,313,383 76,666
22:28:59 1200Mhz 400Mhz 299Mhz 58.00C (60.15C) 2,725 1,901,256 62,162
22:29:01 1200Mhz 400Mhz 300Mhz 58.00C (60.15C) 3,474 2,538,904 84,728

I attempted to change readbufferfactor in the advancedsettings.xml, to no avail. This seem to be described also in other threads. I experience the same buffering freezes with readbufferfactor=4.0, readbufferfactor=8.0, and even readbufferfactor=80.0, and I believe this is to be expected when buffermode is not set explicitly to 1.
http://paste.osmc.io/enahusixof

I did further tests with buffermode=1 together with an increased cachemembuffersize (between 30-100 MB) and high readbufferfactor (between 4-16). Playback generally takes slightly longer to start, but buffers better and only stutter once in a while. bcmstat.sh shows spikes of 5-9 MB/s (depending somewhat on readbufferfactor). It still seem to be too slow sometimes, especially during first start of playback, or during seek, where it can take many seconds before playback gets going. And with these settings I also sometimes receive “cache full” popups, which according to some research is more of an annoyance than a real problem itself.

Can I tune the SMB download speed in the “non-cache” scenario (buffermode not explicitly set, defaults to 0 I think) to pull video faster and avoid playback freezing?

Edit:
List of things to test:

• 1. Play video from NFS share mounted via fstab (ActionA) - Works like a charm! Update: Playback via SMB if mounted (e.g. via fstab) also works basically just as well.
• 2. Add smsc95xx.turbo_mode=Y to /boot/cmdline.txt (DBMandrake) - Works well, also

Both seem like workarounds, and not proper fixes to the problem.

[DBMandrake]

@logan893

Testing file transfer independently from the servers on my network, I was able to saturate the 100 Mbps line to the RPi, so it’s not a network issue.

How exactly did you perform this test ?

What type of switch is the Pi connected to and does it have flow control enabled on the switch port ?

Have you tried the Ethernet turbo cmdline.txt option discussed in this thread ?

Bad video performance on Pi 3

Is your Cisco switch a managed switch ? (EG has a web interface or configuration utility) If so check to see if flow control is enabled on the ports. If not try enabling it. Alternatively try adding smsc95xx.turbo_mode=Y to the end of the line in /boot/cmdline.txt then rebooting.

[logan893]

@DBMandrake

How exactly did you perform this test ?

What type of switch is the Pi connected to and does it have flow control enabled on the switch port ?

Have you tried the Ethernet turbo cmdline.txt option discussed in this thread ?

Basic throughput test was performed using scp to /dev/null of the same video file from my FreeNAS box.

Also the fact that enabling buffering, and increasing readbufferfactor somewhat, indicates that the network and servers can deliver more than fast enough.

I have two dumb switches between RPi3 and my server running VMware ESXi. Server runs one Windows VM and one FreeNAS VM. Both have SMB shares, from which I’ve tried to play the same file with the exact same result. I have only unmanaged switches, no flow control capability anywhere in the network.

I will have to try smsc95xx.turbo_mode tonight. Thanks for the suggestion.

[ActionA]

Tried using NFS?

[DBMandrake]

@logan893

I have only unmanaged switches, no flow control capability anywhere in the network.

Actually, unmanaged switches almost universally have ethernet flow control enabled. (With no way to turn it off, obviously) Managed switches usually default to flow control off.

However flow control on is what you want for a Raspberry Pi.

[logan893]

@ActionA

Tried using NFS?

Not yet. I’ll set it up tonight as an additional test.

@DBMandrake

Actually, unmanaged switches almost universally have ethernet flow control enabled. (With no way to turn it off, obviously) Managed switches usually default to flow control off.

However flow control on is what you want for a Raspberry Pi.

I see. The network interface on ESXi facing LAN has Flow Control autonegotiation enabled. If both switches between ESXi and RPi3 support it (and they likely do), it should then be enabled.

$ ethtool --show-pause vmnic2
Pause parameters for vmnic2:
Autonegotiate: on
RX: off
TX: off

[logan893]

Where can I read more about what smsc95xx.turbo_mode does, and other smsc95xx settings?

[logan893]

Test 1: NFS
Default settings (restored advancedsettings.xml), playing from NFS share (mounted via fstab) instead of SMB.
Plays perfectly. Starts quick.
Video buffer staying consistently high, vq mostly over 80%, never see it drop below 50%.

Tried with NFS mount settings used by DBMandrake noatime,noauto,x-systemd.automount,async,nfsvers=3,rsize=8192,nolock,nofail,local_lock=all,soft,retrans=2,tcp grabbed from this thread:

Consistently interrupted playback

This doesn’t sound like a libNFS (Kodi built in NFS) issue to me, as you are having the problem both with Kodi’s libnfs and Kernel mounts - I see a pause of about a second randomly a few times during a TV episode if I use the Kodi built in NFS client however I do not have any issues if I use a kernel NFS mount. My NFS server is just the standard built (bsd) nfsd on a Mac Mini. Can you post your exact fstab line (with IP’s changed if you prefer) including all the mount options you are using ? T…

Burst buffering during start:

Time ARM Core H264 Core Temp (Max) IRQ/s RX B/s TX B/s
======== ======= ======= ======= =============== ====== ========== ==========
18:17:10 1200Mhz 400Mhz 300Mhz 60.15C (60.15C) 11,429 11,132,203 566,759
18:17:12 1200Mhz 400Mhz 300Mhz 60.69C (60.69C) 9,405 8,994,566 458,580
18:17:15 1199Mhz 400Mhz 300Mhz 60.15C (60.69C) 4,666 4,193,502 218,826
18:17:17 1200Mhz 400Mhz 300Mhz 60.69C (60.69C) 3,479 3,310,421 170,895
18:17:19 1200Mhz 400Mhz 300Mhz 60.15C (60.69C) 3,547 3,238,123 165,136
18:17:21 1200Mhz 400Mhz 300Mhz 60.15C (60.69C) 1,405 1,081,141 59,262

Network transfer speeds are generally higher, mostly 2.5-4 MB/s with brief spikes going higher, and now keeps up with the high bit rate video.

Time ARM Core H264 Core Temp (Max) IRQ/s RX B/s TX B/s
======== ======= ======= ======= =============== ====== ========== ==========
18:23:18 1199Mhz 400Mhz 300Mhz 60.15C (60.69C) 3,667 3,367,887 173,031
18:23:20 1200Mhz 400Mhz 300Mhz 59.61C (60.69C) 3,388 3,055,703 160,232
18:23:22 1199Mhz 400Mhz 300Mhz 60.15C (60.69C) 4,065 3,930,100 199,125
18:23:24 1200Mhz 400Mhz 300Mhz 60.15C (60.69C) 4,634 4,382,782 225,104
18:23:26 1200Mhz 399Mhz 300Mhz 60.15C (60.69C) 5,228 4,914,440 250,440
18:23:28 1200Mhz 400Mhz 300Mhz 60.15C (60.69C) 3,646 3,309,712 169,875
18:23:30 1200Mhz 399Mhz 300Mhz 60.69C (60.69C) 5,184 4,891,087 254,166
18:23:32 1200Mhz 400Mhz 299Mhz 60.69C (60.69C) 2,465 2,154,692 112,107

Seeking is much quicker also, does not freeze for several seconds as before, and results in bursts of data over the network.

Time ARM Core H264 Core Temp (Max) IRQ/s RX B/s TX B/s
======== ======= ======= ======= =============== ====== ========== ==========
18:26:52 1200Mhz 400Mhz 300Mhz 60.69C (61.22C) 2,863 2,490,168 132,120
18:26:54 1200Mhz 400Mhz 300Mhz 61.22C (61.22C) 6,429 6,467,222 322,788
18:26:56 1200Mhz 400Mhz 300Mhz 61.76C (61.76C) 11,642 11,766,781 592,580
18:26:58 1200Mhz 400Mhz 300Mhz 62.30C (62.30C) 8,974 8,682,299 445,486
18:27:00 1200Mhz 400Mhz 300Mhz 62.30C (62.30C) 2,882 2,549,225 134,262
18:27:02 600Mhz 250Mhz 250Mhz 60.69C (62.30C) 2,657 2,587,990 131,635
18:27:04 1200Mhz 400Mhz 300Mhz 61.22C (62.30C) 3,226 3,119,204 157,819

[ActionA]

Using native protocols between Linux systems will generally always result in better performance than having to deal with the excessive overhead when using samba.

[logan893]

@ActionA

Using native protocols between Linux systems will generally always result in better performance than having to deal with the excessive overhead when using samba.

This is often true, however playback from SMB share worked reasonably well (except for the several seconds of buffering before playback initially starts, and after seek) when I enabled buffering for SMB with buffermode=1.

I’ve now also tried playback from SMB share mounted via an fstab entry, and this works basically as well as the mounted NFS share. No playback freezes, no buffering before playback starts, including after seek. Buffer (vq) seem to be reach a slightly lower saturation (more often around 70-90% than with NFS).

To me this seems like a Kodi problem with built-in local network playback support. Should I reach out to Kodi support in this matter?

[ActionA]

@logan893

Should I reach out to Kodi support in this matter?

If you like. It’s quite well known that system level mounts usually perform better than kodi’s implementation.

[logan893]

I posted the same question to the Kodi forum.

http://forum.kodi.tv/showthread.php?tid=285288

[sam_nazarko]

I have replied on the Kodi forum.