SSH_AUTH_SOCK environment variable not available to Rex

[robrwo]

Describe the bug

I ran into a bug where SSH ForwardAgent was enabled for a server, and I could run from the command-line

ssh -A server "cd /dir && git pull origin main"

but when running that git command in Rex (including using the SCM checkout function) did not work, and received a "Permission denied (public key)" error. See https://groups.google.com/g/rex-users/c/5FavxuXgeb0

The problem was caused by the .ssh/config having a Hostname parameter for the server that specified the IP address. Removing the Hostname fixed the issue.

Upon further investigation, I created a Rex task

task "PrintEnv" => sub {
     my $out = run "printenv";
     say $out if $out;
}

and compared using rex -H server PrintEnv with ssh -A server printenv

A notable difference is that the SSH_AUTH_SOCK environment variable was not set when run through Rex.

When removing the Hostname and re-running the Rex task, SSH_AUTH_SOCK was present. (This seems to have nothing to do with the .ssh/known_hosts although removing it the first time prompts SSH to accept the host fingerprint.)

Expected behavior

Git and other tools being run through Rex would have had their authentication forwarded.

How to reproduce it

For a .ssh/config of a particular server

1. Include a Hostname with the IP address of the server
2. Ensure ForwardAgent is "yes" for the server
3. Run a Rex command on the server that requires forwarded authentication, e.g. git pull

Code example

No response

Additional context

No response

Rex version

1.14.3

Perl version

5.34.0

Operating system running rex

Linux (Ubuntu 22.04.1)

Operating system managed by rex

Linux (Ubuntu 18.04.6, Debian 6.1.85-1)

How rex was installed?

cpan client

[ferki]

Thanks for your patience, @robrwo, and for posting a summary and follow-up to our debugging via Mastodon at the time!

The underlying issue for this is already known, and it can manifest in multiple varations.

The main details are:

• Rex historically used Net::SSH to make ssh connections, because that was the only one working on Windows at the time
• Net::SSH uses libssh2 under the hood, so nothing gets picked up from the system/user ssh config
• so Rex has to attempt its own parsing
• which is guaranteed to not be fully SSH-compatible, and the generic problem (=fully conform SSH config parsing with pure Perl) is out of scope for us anyway
• since the connection layer is pluggable, we support using Net::OpenSSH since a long time
• Net::OpenSSH uses the ssh binary under the hood, so completely compatible with system SSH, including reusing any system/user ssh config
• Rex does its own ssh config parsing hardcoded unconditionally, and makes sure to pass the same parameters regardless of using Net::SSH or Net::OpenSSH to connect (=aiming for same user experience with both)

Now that Net::OpenSSH is a hard dependency, even on Windows, the proposed solution is to ask ssh itself what it thinks about its own config, and use that info instead of the current parsing logic.

I have some a workaround hack on top of the openssh_config_skip branch, and iterating towards proof-of-concept results in the use_ssh_config branch. If you need it with Net::OpenSSH, try to rebase those on top of current default branch, and I'm happy to hear any test feedback. Please be aware those testing branches may be subject to change without further notice.

Unfortunately progress is slow, because a proper implementation driven by tests uncovered some nasty corner cases and shortcomings of the built-in parser, and I have to make sure not to break backwards compatibility too.

I plan to create a separate tracking issue for this, and close the related already open issues in favor of tracking progress in a single place.