I think I solved it!
It’s the lirc_rpi-module that get’s loaded with OSMC by default. I removed it from the config.txt and no stutter. I loaded the module in OpenELEC and now have the same stutter.
Anyone PLEASE test this out! Remove the dtoverlay line from the config.txt and reboot.
Come to think of it, that line was the first thing I noticed when I switched to OSMC but I totally forgot about it.