You probably need to edit the file while the transmission service is shut down otherwise it overwrites your changes on exit.
Before editing the file issue this command:
sudo systemctl stop transmission.service
Edit your file then reboot. See if that helps. BTW you can also stop/disable transmission via the Services section of OSMC settings.