Have you ever had problems using ssh to run commands on a remote machine? Ever wondered, eg, why the PATH on the remote machine wasn’t getting set properly?

Some time ago (2007-ish) I ran into this problem while debugging svn+ssh URLs on Subversion. Since I no longer use Subversion, and because I buried the nuggets of gold that I found in an excessively prolix description of my methods of deduction, I’ve decided to rewrite this page to put the really useful bits at the top, and move the long, verbose version to the end.

I’m going to assume that you’re running Bash on the remote machine. All of the tests that I orginally did were to determine the behavior of Bash. sh, zsh, csh and others might be susceptible to the same analysis, but the details will surely differ.

Short version

To begin to debug PATH (and other environment variable) problems, do this:

  ssh <remote> env | sort | less

This will run env on the remote machine and sort and display the results. Is the PATH correct? If not, you are probably setting it in your .bash_login or .profile on the remote machine, but this won’t work! Why? Because Bash doesn’t source those files when run by sshd. (To find out where this is “documented”, skip to the long version of this story, below.

An incredibly useful tool for debugging this is to set (and export) environment variables in the remote machine’s Bash startup files to see which files are actually getting sourced when Bash is run by sshd. So, put

  export DOTBASHRC=1

into the remote machine’s .bashrc,

  export DOTBASH_LOGIN=1

into .bash_login, and

  export DOTPROFILE=1

into .profile.

Now try running

  ssh <remote> env | sort | less

What did you get? When I did this, I got this:

  DOTBASHRC=1

Moral: Set all of your important environment variables in the .bashrc file, and source it from .bash_login (or .profile, if you use that). This way the variables get set, no matter what.

That was easy, right?

Now read about the painful process I went through while trying to figure all this out!

Long (ago) version

Situation: I was setting up a Mac mini (OSX 10.4) to host Subversion repos, transitioning from FreeBSD.

In the commands that follow, osxbox represents the name of the OSX machine, and bsdbox the FreeBSD machine.

After building Subversion on OSX, and installing a separate (and newer) version (of Subversion), I wondered which one I would get using the svn+ssh scheme:

  svn ls svn+ssh://osxbox/home/svn

This was the result:

  bash: line 1: svnserve: command not found
  svn: Connection closed unexpectedly

svnserve is the protocol server that gets run on the remote end when you use svn+ssh.

Running

   ssh osxbox env

yielded this:

  PATH=/usr/bin:/bin:/usr/sbin:/sbin

The older Subversion was in /usr/local/bin; the newer one in /usr/local/subversion-1.4/bin. Neither was on this PATH.

I use Bash – both on the OSX server-to-be and on the FreeBSD box I’m ssh’ing from. I had to figure out how to debug which Bash startup files get sourced when logging in via ssh. I spent a long time reading the ssh, sshd, ssh_config, and sshd_config man pages, but they said nothing about shell startup files. I knew I didn’t have a .ssh/environment file – which is a way to set enviroment variables on the remote machine – and I wasn’t passing environment variables over the ssh link. So where were they getting set?

This used to work on FreeBSD. Somehow on FreeBSD Bash’s PATH was set to include /usr/local/bin, which is where the Subversion binaries – in particular svnserve – live. In contrast, on OSX, the PATH is rather minimal.

Why the difference? The Mac runs a newer version of sshd than my FreeBSD box. But the sshd man pages weren’t significantly different. I didn’t think that the sshd on FreeBSD was doing something special with the PATH.

After some digging, I found an interesting difference between FreeBSD and OSX: FreeBSD has login classes, and the standard login class sets a PATH that includes /usr/local/bin – which is where “add-on” software, such as Subversion, gets installed. OSX lacks login classes, so the PATH needs to be set by a shell startup script. On FreeBSD running

  ssh bsdbox env

yields

  PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:\
       /usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:\
       /home/david/bin

Conclusion: On OSX we need to set the PATH ourselves. The big question is: Where? When you ssh into a remote machine where you run Bash, which startup scripts get sourced? Any guesses? Here are a few possibilities:

  /etc/profile
  $HOME/.profile
  $HOME/.bashrc
  $HOME/.bash_login
  $HOME/.bash_profile

I read the Bash man page (which is really confusing). Depending on whether the shell is a login shell, an interactive shell, or both, it sources different files. So what kind of Bash are we getting over ssh, when we request command execution rather than a login? (Remember, svn+ssh runs svnserve on the remote end, rather than running a login shell.)

I tried this (after first failing because I used double-quotes – sigh):

  ssh osxbox 'echo $BASH $0 $*'

and it yielded

  /bin/bash bash

which tells us the complete path to bash, and how bash was invoked. (Well, sort of. We can’t see bash’s options, but we can see its parameters – none, in this case.) ssh’ing into the FreeBSD box things are similar, but bash is in a different place:

  ssh bsdbox 'echo $BASH $0 $*'

yields

  /usr/local/bin/bash bash

When I first tried this (with the wrong quotes) I got a confusing answer:

  ssh osxbox "echo $BASH $0 $*"

gives

  /usr/local/bin/bash -bash

which is confusing for two reasons:

and that’s how I knew my quotes were wrong. So now I knew how Bash was getting run. Here’s what the man page says:

  A  login shell is one whose first character of argument zero is a -, or
  one started with the --login option.

Our shell is “bash” and not “-bash”, but we don’t know if it’s being given -l or --login – we can’t see that. So it could be a login shell.

  An interactive shell is one started without  non-option  arguments  and
  without the -c option whose standard input and error are both connected
  to terminals (as determined by isatty(3)), or one started with  the  -i
  option.   PS1 is set and $- includes i if bash is interactive, allowing
  a shell script or a startup file to test this state.

Again, we can’t tell what option arguments were given to bash, and how to tell if we are connected to terminals? What does $- contain?

  ssh osxbox 'echo $BASH $0 $* $-'

gives

  /bin/bash bash hBc

Hmm. No “i” in there, so bash thinks it’s not interactive. The “h” means “hashall” and the “B” means “brace expansion”. There is no mention of “c”, but I think it has to do with being passed commands via -c:

  bash -c 'echo $BASH $0 $* $-'

yields

  /bin/bash bash hBc

on OSX. We seem to be getting a non-interactive shell, but we can’t be sure it’s not a login shell. It probably isn’t. It shouldn’t be. And remember that if I run

  echo "$BASH $0 $* $-"

in an interactive, login shell I get

  /bin/bash -bash  himBH

The “i” in the options means interactive; the “-bash” means login.

Aside: One reason, perhaps, for preferring csh over bash is that it’s trivially easy to make this distinction in csh: it sets the variable $loginsh for login shells.

Let’s assume our shell is a non-interactive, non-login shell. What scripts would it source? Here is what the man page says:

  When bash is started non-interactively, to  run  a  shell  script,  for
  example, it looks for the variable BASH_ENV in the environment, expands
  its value if it appears there, and uses the expanded value as the  name
  of  a  file to read and execute.  Bash behaves as if the following com-
  mand were executed:
         if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
  but the value of the PATH variable is not used to search for  the  file
  name.

BASH_ENV is unset, so nothing happens here. It would seem that it doesn’t source any files. Ah, but there is one last piece, a gross hack:

  Bash attempts to determine when it is being run  by  the  remote  shell
  daemon,  usually  rshd.  If bash determines it is being run by rshd, it
  reads and executes commands from ~/.bashrc, if that file exists and  is
  readable.  It will not do this if invoked as sh.  The --norc option may
  be used to inhibit this behavior, and the --rcfile option may  be  used
  to  force  another  file to be read, but rshd does not generally invoke
  the shell with those options or allow them to be specified.

Our shell is being started by sshd rather than rshd, but Bash might be able to figure this out.

In order to be absolutely sure of what was going on, I finally lit on a simple but powerful idea: set and export a variable in each startup script so I can tell if it has been sourced. .bashrc will set DOTBASHRC=1; .profile will set DOTPROFILE=1, etc.

With this set up, I ran

  ssh macosx env

and got this:

  DOTBASHRC=1

Ah, empiricism!

Problem solved. In order to get PATH set the way you want it for remote ssh command execution, you have to set it in your .bashrc on the remote machine.

Phew!

(Extra credit homework: What does csh do?)