To play bluray menu, you have to install Java Runtime Environment. On myOSMC → AppStore.
Then you can look at these 3 “Bluray Playback” topics if it don’t work. Just read and apply what you think is good.
https://discourse.osmc.tv/t/bluray-iso-playback/616/45
https://discourse.osmc.tv/t/commercial-bluray-playback-from-external-bd-drive/84608
https://discourse.osmc.tv/t/bluray-playback/83657/48