Managed to solve this myself after a while. Starting with a fresh install of osmc (2018-06 i think).
TL;DR: Once you have a2dp-app-osmc installed, have paired and connected to your headphones, set your device to a2dp_sink via pactl. Despite profiles being a component of bluetooth connection, Profile switching is handled by the audio controller, in OSMC its Pulse Audio
pactl list sources short
get the device # of your headphones from the output and put in place in the following command.
pactl set-card-profile # a2dp_sink
Longer version, Roughly everything that I did to get headphones working with the right profile
Install the a2dp software (this may be unnecessary with newer builds of osmc)
sudo apt update && apt install a2dp-app-osmc
I’m not sure if the internal BT device on the raspberry Pi is now fully functional so I disabled the internal bluetooth controller and used a BT Dongle.
dtoverlay=pi3-disable-bt into the file
Reboot the pi.
Ok so now we need to pair the headset, (im sort of going from memory here so this may not be perfect)
commands from here are all within the bluetoothctl application
power off just to give it the ol clean start state
A whole heap of stuff will start showing up, when you see a line that looks something like your headphones you can run
[NEW] Device XX:XX:XX:XX:XX:XX XX-XX-XX-XX-XX-XX
[CHG] Device XX:XX:XX:XX:XX:XX Name: Bose Headphones
[bluetooth] # scan off
From here, our device will be
XX:XX:XX:XX:XX:XX, note that bluetoothctl has autocompletion on tab, so you can just type in the first few hex entries when refering to a device or command, press tab and it will write out the rest.
My headphones had problems connecting before trusting and there seems to be no default-agent defined so I trusted and unblocked the device before pairing and connecting
[bluetooth] # trust XX:XX:XX:XX:XX:XX
[bluetooth] # unblock XX:XX:XX:XX:XX:XX
it should output if it was succesfull or not for each command but just to check
[bluetooth] # info XX:XX:XX:XX:XX:XX
this will give a fair bit of info including a list of profiles available, but also show the status of trusting and blocking.
Now pair and connect
[bluetooth] # pair XX:XX:XX:XX:XX:XX
[bluetooth] # connect XX:XX:XX:XX:XX:XX
if connected the bluetooth component will now show up as your device name
[Bose Headphones] #
at this point you are connected/paired/trusted and unblocked, and if like me your headphones are screaming “Call From [Beep]” because they are stuck in Hands Free Profile (HFP)/Headset profile (HSP).
Bluetoothctl wont actually handle the switching of the profiles so we can quit out of here for now. But at this point we are connected and paired and that is good!
[bluetooth] # quit
Pulse audio is handling what service profile our bluetooth device is acting as so lets list our sources. The full output of listing our sources is unnecessary at this point so lets just get the cut down version.
pactl list sources short
You should see something along the lines of
<N> BlueZ5... XX_XX_XX_XX...
<N> is our source/sink number and
XX_XX_XX_XX... is the same mac address of our bluetooth headset.
So now we are going to set that audio device to an a2dp output device.
pactl set-card-profile <N> a2dp_sink
The setting is persistent so you shouldn’t need to do it again unless you remove the device or switch profiles manually.
I realize this is probably obvious to a lot of people but I had problems find clear answers about switching profiles so thought i would elaborate.
[Edit]: pactl commands need to be run as the active user i.e. no sudo prefix