Sometimes when there is nothing spoken for a longer time, a nee frame drop may occur as well when the next subtitle appears… seems like the stuff rendering the subtitle offloads after a certain amount of time as well.
Could you add a feature (with on/off switch in settings) that always loads the module for subtitles and always keeps it active? The drops for SRT seem to be purely a result of Kodi loading it during playback (and possibly after longer time of no subs during playback). I think you mentioned something like that before.