Known_hosts.yml headaches

Hi, I’m a bit stuck with the new known_hosts.yml settings. I’ve been trying for hours now with all kinds of combinations for the name and key, but I can’t seem to get it to work.

I run my own gitlab server. Couple of questions:

  1. The example shows two kinds of settings for the gitlab name: one with ssh-ed25519 and the other ssh-rsa. What is the difference between them, are they both necessary?
  2. What should I use as name, does this matter? In the example it’s gitlab.com, should I use my private gitlab server domain name there e.g. gitlab.mydomain.com? Or the full repo url name as set per site in the wordpress_sites.yml, like: git@gitlab.mydomain.com:myname/site1.com.git?
  3. In the example, the keys all start with gitlab.com, is this important and what should it be in my case? And should it match with what is used as the name?
  4. The actual key itself, which one should it be? Is it the same key I use to connect from my local machine to my gitlab server, as what’s in the public id_rsa file? Or something else?
  5. After each change in known_hosts.yml (while I try to get it to work), is it necessary to commit/push the changes as well as reprovision the server before I can try to deploy a site again, or not? Also should the current known hosts be cleared on the gitlab server after each change in known_hosts.yml?

I’m sure it’s me who is doing something wrong here, but is there a way to bypass/disable this method entirely for now and just deploy the old way? I was actually expecting (hoping) the default setting: repo_accept_hostkey: true to bypass all this, but unfortunately nope.

1 Like

What exactly is your error message?

The solution might differ depending on the point of failure, whether it’s Composer, known_hosts, or sshd role.

The names in known_hosts.yml should be host names, and the values will likely be in your own ~/.ssh/known_hosts file.

2 Likes

repo_accept_hostkey defaults to true which does the same thing as Trellis always did. If you’re having a problem with that setting now, it seems like you would have before anyway?

We just now allow you to either change that, or add more keys manually.

Trellis’ known_hosts just calls http://docs.ansible.com/ansible/known_hosts_module.html so you can find the docs there. The name must match the hostname of the key. That’s the only requirement.

This is all standard SSH known hosts stuff. There’s a lot of resources for this online. But in general:

you add a server’s SSH public key to the other server you want to allow access

So that means you add your gitlab server’s public key to your Trellis servers known hosts.

You’d need to reprovision your server after updating this, but nothing else should be necessary.

3 Likes

To undo the effects of roots/trellis#751, in group_vars/all/known_hosts.yml . . .

  • keep the default repo_accept_hostkey: true
  • set known_hosts to an empty list, e.g., known_hosts: []

The other responses above are 100% correct, but here’s another angle. I’m making this broad and verbose for the sake of others who may come across this topic with similar questions.

Be clear on the difference between an SSH client’s key pair and an SSH server’s key pair.

Client key pair – establishes client identity. When you SSH to a server, your local machine is acting as the SSH client. The server establishes your identity by verifying a match between your private key and the public key the server has stored for you in its authorized_keys file. You added this public key to the server either when you created the server, or if you listed the key in users.

Host key pair – establishes host identity. The client (your local machine) establishes the server’s identity by verifying a match between the host’s private key and the host’s public key your machine has stored in known_hosts. You added this public key to known_hosts when you made your very first connection to the server, when your CLI probably gave you a prompt such as this:

The authenticity of host '12.34.56.78 (12.34.56.78)' can't be established.
ED25519 key fingerprint is SHA256:...<snip>
Are you sure you want to continue connecting (yes/no)?

When your server acts as an SSH client. The deploy process causes your server to reach out to a git host and potentially to a composer package host. In that context, your server is acting as an SSH client and the git host or composer package host is acting as the SSH host.

For your server (now the SSH client) to establish the identity of these latter hosts (SSH servers), it needs those hosts’ keys in its own known_hosts file. When repo_accept_hostkey: true, your server will automatically accept the git host’s host key into your server’s known_hosts, without just failing or prompting Are you sure you want to continue connecting (yes/no)?.

If the deploy’s composer install process reaches out to a host not already in your server’s known_hosts, the composer install will fail. Before roots/trellis#751 you would have had to SSH into your server and add those known_hosts entries manually. The new Trellis known_hosts allows you to specify those host keys right in your config, automating the process.

What should the host key entry look like? Find examples of how host keys should look by checking your local machine known_hosts, either by opening and looking, or by using this command:
ssh-keyscan -H gitlab.mydomain.com

If you are on a machine that you trust has an uncompromised connection to a host, you can use this command to retrieve the host’s key:
ssh-keyscan github.com
You’ll see that this output matches what Trellis lists for GitHub in known_hosts.yml:

github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==

The github.com host returns just one key of the rsa type, so known_hosts.yml lists only that one key.

The gitlab.com host returns keys of three types: rsa, ecdsa, and ed25519. In a regular SSH connection (vs. this example ssh-keyscan) there are a number of factors affecting which key a host will offer. One factor is the HostKeyAlgorithms requested by the SSH client. Trellis configures your server’s SSH client to not accept the insecure ecdsa type key from a host, which is why that type is missing from the key types preloaded for gitlab.com.

Your questions.

  1. Both types of host keys are listed to be ready for either type that gitlab.com might send your server, depending on whether your server has been provisioned with the latest sshd role and its updated HostKeyAlgorithms (affecting which key types the gitlab.com server may send). You’ll see that HostKeys can be of various types, and which is selected can be affected by [HostKeyAlgorithms] (http://manpages.ubuntu.com/manpages/xenial/en/man5/ssh_config.5.html) settings. Both keys are listed to avoid the error/warning that you would get if only one key were listed and the server offered the other type.

  2. The name is the domain name involved in the expected connection, no protocol or git@. If the connection will be to git@gitlab.mydomain.com:myname/site1.com.git then the key name should be gitlab.mydomain.com.

  3. See the section above titled “What should the host key entry look like?” You should use the string you find in your own machine’s known_hosts or the string from the ssh-keyscan command.

  4. This is where I suspect you may be confusing the client key pair with the host key pair. See the section above titled “What should the host key entry look like?”

  5. The host keys involved are typically for hosts like GitHub or Bitbucket and will almost never change. If you do have to add/remove a host key, it only matters that the change appears in the Trellis known_hosts.yml file on your local machine. Although you should commit and push the change eventually, just for tracking, it will not affect your provision/deploy whether you’ve pushed that commit to the repo you list in wordpress_sites.

    For remote servers, only deploying adds a new key. So you don’t need to reprovision after adjusting the Trellis file known_hosts.yml. Just deploy. For local dev Vagrant VM, do a regular
    vagrant provision or run only the affected role by running:
    ANSIBLE_TAGS=wordpress-install vagrant provision

    If you wish to remove a host key from your server’s known_hosts, the known_hosts module docs point out that you may add state: absent for a particular key. If you use state: absent with no key, it will remove all keys for the host with that name.

known_hosts:
  - name: github.com  # all keys for github.com will be removed -- because `key` is missing
    state: absent
  - name: gitlab.com  # this key for gitlab.com will be kept
    key: gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
  - name: gitlab.com    # this particular key  for gitlab.com will be removed
    key: gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9
    state: absent
7 Likes

Amazing detailed explanation and instructions! Thank you very much! I realize I still have to put in some effort to truly understand all aspects of ssh keys, key types and known hosts etc., but my known_hosts.yml works like a champ now (including the setting repo_accept_hostkey: false).

What immediately resulted in success was your mention of the ssh-keyscan command. This gave me the correct key to use. Before I had been trying (among other things) to copy-paste from my local known_hosts file, which is hashed, so that really led nowhere. A suggestion, to maybe place a small comment of this ssh-keyscan command in the known_hosts.yml? Might help some other people as well.

2 Likes