I bought a vero4k, but I have had this issue on the Rpi3 I installed a couple of days ago.
I will start a movie from my NAS (I tried connecting the NAS with NFS or SMB, issue stays the same. The movie will start to play, but if I open top over ssh, the memory usage of kodi starts increasing until it gets killed by the kernel.
Well just take one of the public available demos and just reencode it with your handbrake settings and share that.
But to be honest considering the amount of people playing files without issues so there must something wrong with encodes.
The thing is that I always use the presets in handbrake and most encoded files do work.
I re-encoded one of those files to mkv with a different video and audio codec, I re-ripped the disc with the same new settings and the problem remains.
Until yesterday, I wanted to debug it further, played the mkv file on kodi 18, it worked. The m4v file worked. It also plays on my osmc now, unmodified.
I hope it stays like this, but it’s one of the weirdest issues I’ve seen.