You should try a custom lircd.conf file. There are directions on this forum on how to do this. You might even be able to google a working lircd.conf and drop that right in.
It would be easy to edit the standard one that ships with OSMC to find the code that toggles full screen and change it to enter. However, I have no idea where it is. I write my own because it’s eaiser to change functions on the remote.