As the VNC server is not part of OSMC do I assume you have followed this guide?
Maybe you best continue your troubleshooting in that thread.