Do a little print in the GLX sync method please and compare the deltas - then we see if the “gpu clock” is going wrong. If that’s the case we know the issue.
Edit: Please turn on Sync Playback to Display and turn on Debug logging, then post me a debuglog after you have seen some of those unsmooth operations. We log those issues, here: https://github.com/xbmc/xbmc/blob/master/xbmc/video/videosync/VideoSyncGLX.cpp#L216
It’s smooth. It’s not just not buttery smooth… I think this is just an unstable hardware clock/pll on the nvidia gpu and not any misconfiguration. (I can only guess since I’m not a Nvidia engineer) but the effect is very noticeable. (I have a properly calibrated Pioneer Plasma display that do a 3x multiply on the 23.976/24hz refreshrates, but otherwise nothing else (no picture “enhancements” enabled). On some sort of microlevel, so to speak, the display has to sync to the incoming data, the more stable it is, the more stable the picture will be also, and I guess it’s extra noticeable because of the inherent low fps of film.
You know what, when the picture is smooth, the movie also gets better, the microstutters really do detract a little from the movie in some subconscious way.
As I mentioned earlier: Even my gtx970 doesn’t do 23.976hz (it’s 23.973hz) out of the box. It is supposed to, cause why else include a 23.xxx hz mode? (I haven’t tried toying around with custom resolutions since this is my gaming rig so it doesn’t matter.)
As always,ymmv. (We’re pretty OT too…)
I’ve managed to get rid of virtually all the jitters now by remembering to turn back on the standard Motion Enhancer (default) setting on my Sony TV.
The high setting gives you the awful “Soap Opera effect” or as I like to call it lately the 'Hobbit Vision effect" 
Thanks for the 23.976Hz and 24Hz Modeline settings Fritsch.
nvidia 304.125 drivers with generic ubuntu kernel 4.0.8 on kodibuntu on the atv seems to show 23.97091hz instead of 23.976, just like Fritsch mentioned.
Kodi runs fine but reporting 235MB ram total, instead of 242MB on CB2.0. Free ram (with Isengard) reports about 18MB ram less available (but total ram was less to begin with). This is with literallly no tweaking.
CrystalHD compiled fine but haven’t tested playback since I haven’t built Kodi with CHD support.