Transmission doesn’t know what the ‘smb://’ URI is. That is done by Kodi’s VFS layer automatically, but doesn’t work in other applications.
Instead, you should look at adding your network share to /etc/fstab and mounting it to a directory. Then you can specify the mount directory. To prevent a race condition, you may wish to set up the mount as a systemd unit. This will allow you to use a drop in with transmission.service and ensure that the share is mounted before Transmission starts.
What firmware is your router using? If you can enable nfs server on the router, that would be the best way to mount the drive in OSMC. It’s possible with samba but not as fast. Also, from what I’m reading, it’s much easier using nfs. A lot less hoops to jump through.