System PATH broken when running a non-interactive sudo command

[LxP]

I’m trying to use Ansible to administer several OSMC installations in unison, along with many other separate Debian-based systems.

For those who haven’t heard of Ansible, it’s a system configuration management tool whose primary method of getting things done is to copy Python scripts onto a remote computer and execute them to achieve some desired system state. In my case, I’m successfully using it to centrally manage Kodi add-ons and settings.

I’d also like to use it to manage Debian package states, but when Ansible connects to an OSMC device, the system PATH is mangled and things fail.

I’ve confirmed that this isn’t an Ansible issue. I’ve found two possible culprits in the way OSMC is set up though:

1. The secure_path default in /etc/sudoers doesn’t seem to be honoured:

   $ ssh osmc@XXX bash -c '"sudo grep secure_path /etc/sudoers ; echo $PATH ; sudo bash -c echo\ $PATH"'
   ...
   Defaults	secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
   /usr/local/bin:/usr/bin:/bin:/usr/games
   /usr/local/bin:/usr/bin:/bin:/usr/games
   

2. There’s a big difference in the PATH depending on whether the shell is run interactively or not:

   $ ssh osmc@XXX
   ...
   osmc@osmc:~$ echo $PATH
   /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin:/opt/vc/bin
   

   $ ssh osmc@XXX bash -c '"echo $PATH"'
   ...
   /usr/local/bin:/usr/bin:/bin:/usr/games
   

On every other Debian-based system I have access to, the PATH is not different based on whether the shell is login, non-login, interactive or non-interactive. In the case of Point #2, it seems that having PATH setup in /etc/profile and /etc/profile.d/*.sh could be part of the issue, as these files are only read and executed for “login” shells. Shells started by non-interactive SSH commands are not login shells.

I don’t have an elegant solution for this issue at the moment. What are your thoughts?

[LxP]

A Stack Overflow answer strongly suggests that things like PATH setup should occur in a bashrc file (such as /etc/bash.bashrc) since “it is sourced by non-interactive non-login shells” (whereas profile and friends are not).

Accordingly, perhaps an elegant solution is the introduction of a /etc/bash.bashrc.d directory (similar in purpose to /etc/profile.d), whose scripts execute in all cases (including non-interactive non-login shells, where a complete PATH is still needed).

If this solution were adopted, /etc/profile.d would be pared down to only contain scripts that should run for login shells (add-paste.sh and generate-locale.sh would probably stay there, for instance).

[DBMandrake]

Not an easy one to answer this, however a couple of points:

@LxP

The secure_path default in /etc/sudoers doesn’t seem to be honoured:

Have you looked at the overrides in /etc/sudoers.d/ ?

We don’t modify the default sudoers file for among other reasons being that it is too risky - a lot of the functionality of osmc relies on passwordless sudo to root so if sudo is broken even for a moment you’re screwed and can’t fix it because you now can’t sudo to root to edit /etc/sudoers. (Been there done that )

I don’t quite recall why we decided to disable secure_path, but I know it was for a good reason. I think it was because we weren’t able to override secure_path from /etc/sudoers.d/ yet we don’t want to modify /etc/sudoers either. So the next best solution was to pass the users path via sudo.

Technically this is less secure but keep in mind that we have passwordless sudo to root enabled by default - this is not a server operating system.

@LxP

On every other Debian-based system I have access to, the PATH is not different based on whether the shell is login, non-login, interactive or non-interactive. In the case of Point #2, it seems that having PATH setup in /etc/profile and /etc/profile.d/*.sh could be part of the issue, as these files are only read and executed for “login” shells. Shells started by non-interactive SSH commands are not login shells.

This is not something that has been considered to be honest, however I would suggest that in general it’s a bad idea to rely on certain directories being in the path, especially for a non-interactive SSH session.

So despite the path being different to other Debian systems I would suggest the bug is in making assumptions about what might or might not be in the path. The path is certainly not standardised across different distributions of Linux either.

@LxP

A Stack Overflow answer strongly suggests that things like PATH setup should occur in a bashrc file (such as /etc/bash.bashrc) since “it is sourced by non-interactive non-login shells” (whereas profile and friends are not).

Accordingly, perhaps an elegant solution is the introduction of a /etc/bash.bashrc.d directory (similar in purpose to /etc/profile.d), whose scripts execute in all cases (including non-interactive non-login shells, where a complete PATH is still needed).

This problem with this is .bashrc is only for bash - we also have /bin/sh (ash, I think) which is used in a lot of scripts - so now you would have different paths for sh and bash, which I don’t think is a good idea either.

Perhaps @sam_nazarko has some thoughts on this.

[EqUaTe]

So as I’m quite up with my sysadmin things, I thought I’d take a quick look at what might be going on here.

First off, interactive and non-interactive paths being different is pretty common. When remoting onto systems you should never assume these two things:

1. The path is the same as what you expect no matter how you get on
2. Any aliases or equivalents you use exist there

Second, @DBMandrake has pointed you at the exact cause of secure_path being ignored:

root@akira:~# cat /etc/sudoers.d/osmc-no-secure-path 
Defaults        !secure_path

It looks like this behaviour is to preserve the standard user path, but as it is different between interactive and non-interactive sessions, you are encountering this issue.

What I would suggest you do for now, is modify the above file and place the interactive path there.
That way sudo will work as normal within OSMC, but you should also be able to use it when calling sudo via non-interactive ssh.
An alternative would be to explicitly define secure_path for a new user, though I’ve not looked into how you might go about this or if it’s even possible.

Hope this helps!

[LxP]

@DBMandrake

Have you looked at the overrides in /etc/sudoers.d/ ?

I’d failed to notice that directory (and the relevant comment in /etc/sudoers) until now. That explains why the secure_path setting in /etc/sudoers isn’t honoured. Thanks for the heads-up.

@DBMandrake

This is not something that has been considered to be honest, however I would suggest that in general it’s a bad idea to rely on certain directories being in the path, especially for a non-interactive SSH session.

So despite the path being different to other Debian systems I would suggest the bug is in making assumptions about what might or might not be in the path. The path is certainly not standardised across different distributions of Linux either.

I agree that making assumptions about the PATH is bad in most cases, but in this case we’re talking about really stock-standard PATH entries like /sbin being missing. Technically it’s not me making the assumption about /sbin being there either—it’s a core Debian tool like dpkg:

osmc@osmc:~$ which ldconfig start-stop-daemon
/sbin/ldconfig
/sbin/start-stop-daemon

osmc@osmc:~$ env PATH=/usr/local/bin:/usr/bin:/bin:/usr/games sudo apt-get -y install vim
...
After this operation, 27.9 MB of additional disk space will be used.
dpkg: warning: 'ldconfig' not found in PATH or not executable
dpkg: warning: 'start-stop-daemon' not found in PATH or not executable
dpkg: error: 2 expected programs not found in PATH or not executable
Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin
E: Sub-process /usr/bin/dpkg returned an error code (2)

which itself states that the PATH “should” already contain /sbin among other things. I feel that if I were to complain to Debian about the fact that their thousands of packages make this assumption, their response would be “fix your system PATH.” What are your thoughts on this particular example?

@EqUaTe

First off, interactive and non-interactive paths being different is pretty common.

I didn’t know this. I can accept that statement on face value, but I’m struggling to picture a scenario where a set of unique PATHs for each mode would actually be useful. Why would anyone set up a system like this?

I’d understand things like aliases being missing in a non-interactive session, as aliases are basically just “interactive candy” (for lack of a better term). Not actual executables though; I can’t imagine a situation where something non-interactive should fail to run just because I didn’t spawn an interactive shell before running it.

@EqUaTe

What I would suggest you do for now, is modify the above file and place the interactive path there.

Thanks for the tip. Sounds like the simplest solution, and that’ll be easy enough to script into my Ansible configuration too. I’m mindful that @DBMandrake has hinted above at secure_path overrides not being possible from /etc/sudoers.d, but I’ll give it a crack anyway. If that fails, I’ll just find a way to inject a custom PATH into the affected tasks of my Ansible configuration.

@EqUaTe

An alternative would be to explicitly define secure_path for a new user, though I’ve not looked into how you might go about this or if it’s even possible.

I’ll avoid this if possible, because setting up a separate user account for all I’m trying to achieve seems excessive unless absolutely necessary. Thanks for offering multiple solutions nonetheless.

@EqUaTe

Hope this helps!

Any response that contributes something at all is always helpful. Thanks to everyone for your time.

[bong101]

Hi all

I’ve exactly the same problem with OSMC and Ansible than LxP

ansible@toto:/projets/ansible$ ansible-playbook titi.yml --ask-sudo-pass
SUDO password:

PLAY [titi] ******************************************************

GATHERING FACTS ***************************************************************
ok: [titi]

TASK: [Update + Upgrade] ******************************************************
failed: [titi] => {“failed”: true}
stdout: Reading package lists…
Building dependency tree…
Reading state information…
Reading extended state information…
Initializing package states…
Building tag database…
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
dpkg: warning: ‘ldconfig’ not found in PATH or not executable
dpkg: warning: ‘start-stop-daemon’ not found in PATH or not executable
dpkg: error: 2 expected programs not found in PATH or not executable
Note: root’s PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin
Reading package lists…
Building dependency tree…
Reading state information…
Reading extended state information…
Initializing package states…
Building tag database…

msg: ‘/usr/bin/aptitude full-upgrade’ failed: E: Sub-process /usr/bin/dpkg returned an error code (2)
Failed to perform requested operation on package. Trying to recover:
dpkg: warning: ‘ldconfig’ not found in PATH or not executable
dpkg: warning: ‘start-stop-daemon’ not found in PATH or not executable
dpkg: error: 2 expected programs not found in PATH or not executable
Note: root’s PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin

FATAL: all hosts have already failed – aborting

SSH Connection work fine…path seems to be ok in .profile and /etc/sudoers…

So any idea ?
Thanks in advance

[LxP]

The workaround I’m using is to explicitly set the PATH in your apt task:

- name: upgrade all installed packages
  apt: update_cache=yes upgrade=yes
  environment:
    PATH: /bin:/sbin:/usr/bin:/usr/sbin

I seem to have shut down the conversation with my previous post, sadly.

[Carsten_Bleser_Rasmu]

Hej All
I had the same problem. I did a durty trick.
I copied the missing start-stop-daemon from one machine to another. Like this.

Good machine.
cd /sbin
scp start-stop-daemon smartguy@badmachine:/tmp

ssh smartguy@badmachine
Now badmachine
sudo -i
cd /sbin
cp /tmp/start-stop-daemon .
chmod 755 start-stop-daemon
apt install -f

…
And now it works

[lunk.rat]

Let’s strike this converation back up. @LxP this post helped me a lot. I am exactly like you, managing several Raspbian and Debian systems with Ansible. I write a playbook for my new OSMC install, and of course run into this immediately. Like you, I immediately looked at /etc/sudoers and failed to look at /etc/sudoers.d

Your recommendation to export PATH in the play helped to get me on my way, thank you!

I still think this alteration should be changed in OSMC so that it behaves more like default Raspbian. Let me know how I can help with that effort.

[sam_nazarko]

I’m happy to review a PR to improve this.

Sam