So I tried a Class 10 SD card (Samsung EVO 16GB). I have not had a chance to test extensively, but I was listening for an hour or so yesterday and still noticed a couple of gaps.
But today I just saw this post about “micro dropouts”. It sounds kind of similar to what I am experiencing. In that thread, @popcornmix suggested setting buffermode=1 (Kodi page). I will try this when I get home and see if it helps my setup too.