How to make CEC fully work the way I want?

You can remap any keys to be the volume controls. The fast forward and rewind buttons are often used to do this.

Long press isn’t supported with CEC. See here. Some TVs hack around it, but it’s not passed through to OSMC. Repeat presses can be configured in Peripherals → Input.

What’s the question?

The first thing I would do is enable debug logging and check if the buttons you are pressing on your remote are being received by OSMC. If they are, you can remap them with keymap editor. If they are not, then you can’t, as the TV isn’t forwarding the button.

The dongle can’t be reprogrammed to accept new remote codes. There is a built in IR sensor which can be programmed with irrecord (LIRC)