(The following is true for the default OSMC skin - for Estuary and other skins, you’ll need to reach out to their developers/users for specifics.)
‘Settings’ > ‘Interface’ > ‘Skin’ > ’ - Configure skin’ to bring up the skin settings, then ensure ‘Home’ is selected on the left. If you see ‘Enable menu customisation’ on the right then select it and follow the prompts to install the Skin Shortcuts script, otherwise go straight to ‘Customise main menu’.
You can then add a new menu item with ‘Add’ at the bottom of the screen, and re-order items with the arrows on the left.
To actually run RetroPi you’ll need to type in a custom action for your new menu item. ‘Change action’ > ‘Custom action >’ (the last item in the list). The action you’ll want will vary depending on exactly what you’re launching and how (sorry, I’m not familiar with RetroPi). If it’s a python script then
RunScript(/path/to/script.py)
If it’s launched via a single command then something of the lines of
System.Exec(/path/to/command)
(or possibly System.ExecWait - which waits for the command to finish).
A list of the built-in functions that can be used for a custom action is here.