Problem when deploying to remote server (staging environment)

[FaabScheeren]

Hello,

I have a problem deploy my local project to a remote server (Digitalocean droplet). When running trellis provision staging I get the following error message.

It seems to be a SSH problem. When creating the droplet I added my SSH key and connecting with ssh <servername> works without problem. I already tried to solve it by removing the keys for the host in the known_host file.

To set up the project I followed this tutorial. He first uses the command ansible-playbook server.yml -e env=staging and after it ./bin/deploy.sh staging <url>. The documentation describes to use trellis provision <environment> and trellis deploy <environment>. Don’t know if this has any affect on the project.

Error message

TASK [connection : Warn about change in host keys] *****************************
task path: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/connection/tasks/main.yml:20
fatal: [128.199.53.64]: FAILED! => {
    "msg": "The conditional check ''REMOTE HOST IDENTIFICATION HAS CHANGED' in connection_status.stdout' failed. The error was: error while evaluating conditional ('REMOTE HOST IDENTIFICATION HAS CHANGED' in connection_status.stdout): 'dict object' has no attribute 'stdout'. 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/connection/tasks/main.yml': line 20, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Warn about change in host keys\n  ^ here\n"
}

[strarsis]

Does manually connecting to the server (from the same terminal as ansible is used) with SSH, removing the old host key fingerprint (a suitable command should be printed with the error message) and re-adding the new host key fingerprint (re-connect manually again and confirming saving the host key fingerprint) fix the issue?
Afterwards ansible should be able to connect without the error.

In the more distant past I also encountered a similar issue, where ansible updated the SSH host key algorithm, resulting in a different SSH host key and the same issues, see this discussion:

Ansible connection role error while provisioning server TASK [connection : Warn about change in host keys]

Although this appears to be an issue with Ansible, you should be able to work around this issue: Manually connect via SSH to the target server. You should see a remote host key identification warning, you can use the command proposed in the warning message to get rid of the non-matching, stored host key. Manually connect via SSH again to the target server. Accept (let ssh store) the new host key. Manually connect a third time via SSH, now the SSH connection should just work. Trellis/Ansible s…

[FaabScheeren]

Thanks for your reply, @strarsis!

Saddly that didn’t work. Connecting to the server manually works without any problem. I removed the key’s for the server in the known_hosts file. After I removed the keys, tried to connect to the server manually. It asks to add the key fingerprint to known_hosts.

Running trellis provision staging still gives the same message. Also created the test-connection.yml as you suggested. Which works fine.

[strarsis]

Are there more than just one fingerprint for the affected server in known_hosts file?

When the host key algorithm is changed, a different fingerprint can result from this, existing fingerprints for that same server can cause issues, when only some are removed.

Unable to re-provision on DigitalOcean UNREACHABLE!

I had a similar issue some years ago - and the reason had something to do with changed SSH host key algorithms. Maybe this also has something to do with your issue?

[FaabScheeren]

I checked the known_hosts file and there were indeed two fingerprints for the server. I removed both of them and added the following lines to the ssh config file. Then manually connected to the server. Again it asks for to add the fingerprint to the known_hosts file, accepted it. Now there is just one fingerprint.

After trying to provision still get the same error message.

[FaabScheeren]

I ran the provision with --verbose option. I get also a few other warnings seems nothing to do with the problem but maybe I overlook it.

Starting galaxy role install process
- composer (1.9.0) is already installed, skipping.
- ntp (2.3.1) is already installed, skipping.
- logrotate (v0.0.5) is already installed, skipping.
- swapfile (v2.0.36) is already installed, skipping.
- mailpit (v1.0.0) is already installed, skipping.

Running command => ansible-playbook server.yml -vvvv -e env=staging
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/vars/__init__ 2.py) as it seems to be invalid: module
'ansible.plugins.vars.__init__ 2' has no attribute 'VarsModule'
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/connection/__init__ 2.py) as it seems to be invalid:
module 'ansible.plugins.connection.__init__ 2' has no attribute 'Connection'
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/shell/__init__ 2.py) as it seems to be invalid: module
'ansible.plugins.shell.__init__ 2' has no attribute 'ShellModule'
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/become/__init__ 2.py) as it seems to be invalid:
module 'ansible.plugins.become.__init__ 2' has no attribute 'BecomeModule'
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/callback/__init__ 2.py) as it seems to be invalid:
module 'ansible.plugins.callback.__init__ 2' has no attribute 'CallbackModule'
ansible-playbook [core 2.15.9]
  config file = /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/ansible.cfg
  configured module search path = ['/Users/fabioscheeren/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/fabioscheeren/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/bin/ansible-playbook
  python version = 3.9.6 (default, Dec  7 2023, 05:42:47) [Clang 15.0.0 (clang-1500.1.0.2.5)] (/Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/bin/python3)
  jinja version = 3.1.3
  libyaml = True
Using /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/development as it did not pass its verify_file() method
script declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/development as it did not pass its verify_file() method
auto declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/development as it did not pass its verify_file() method
Parsed /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/development inventory source with ini plugin
setting up inventory plugins
host_list declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/production as it did not pass its verify_file() method
script declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/production as it did not pass its verify_file() method
auto declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/production as it did not pass its verify_file() method
Parsed /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/production inventory source with ini plugin
setting up inventory plugins
host_list declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/staging as it did not pass its verify_file() method
script declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/staging as it did not pass its verify_file() method
auto declined parsing /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/staging as it did not pass its verify_file() method
Parsed /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts/staging inventory source with ini plugin
redirecting (type: modules) ansible.builtin.ipify_facts to community.general.ipify_facts
Loading collection community.general from /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible_collections/community/general
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/common/tasks/disable_challenge_sites.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/common/tasks/reload_nginx.yml
redirecting (type: modules) ansible.builtin.mount to ansible.posix.mount
Loading collection ansible.posix from /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible_collections/ansible/posix
redirecting (type: modules) ansible.builtin.sysctl to ansible.posix.sysctl
redirecting (type: modules) ansible.builtin.sysctl to ansible.posix.sysctl
redirecting (type: modules) ansible.builtin.timezone to community.general.timezone
redirecting (type: modules) ansible.builtin.authorized_key to ansible.posix.authorized_key
redirecting (type: modules) ansible.builtin.authorized_key to ansible.posix.authorized_key
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/users/tasks/connection-warnings.yml
redirecting (type: modules) ansible.builtin.mysql_user to community.mysql.mysql_user
Loading collection community.mysql from /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible_collections/community/mysql
redirecting (type: modules) ansible.builtin.mysql_user to community.mysql.mysql_user
redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db
redirecting (type: modules) ansible.builtin.sysctl to ansible.posix.sysctl
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/letsencrypt/tasks/setup.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/letsencrypt/tasks/nginx.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/common/tasks/reload_nginx.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/letsencrypt/tasks/certificates.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/wordpress-setup/tasks/database.yml
redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db
redirecting (type: modules) ansible.builtin.mysql_user to community.mysql.mysql_user
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/wordpress-setup/tasks/self-signed-certificate.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/wordpress-setup/tasks/nginx-client-cert.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/wordpress-setup/tasks/nginx-includes.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/wordpress-setup/tasks/nginx.yml
statically imported: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/common/tasks/disable_challenge_sites.yml
Loading callback plugin default of type stdout, v2.0 from /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin vars of type old, v2.0 from /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/lib/trellis/plugins/callback/vars 2.py
Loading callback plugin vars of type old, v2.0 from /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/lib/trellis/plugins/callback/vars.py
Skipping callback 'default 2', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal 2', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline 2', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: server.yml ***********************************************************
Positional arguments: server.yml
verbosity: 4
connection: smart
timeout: 10
force_handlers: True
become_method: sudo
tags: ('all',)
inventory: ('/Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/hosts',)
extra_vars: (' env=staging',)
forks: 5
3 plays in server.yml

PLAY [Ensure necessary variables are defined] **********************************
Trying secret FileVaultSecret(filename='/Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.vault_pass') for vault_id=default

TASK [Ensure environment is defined] *******************************************
task path: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/variable-check.yml:8
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/test/__init__ 2.py) as it seems to be invalid: module
'ansible.plugins.test.7350187003863887113___init__ 2' has no attribute
'TestModule'
[WARNING]: Skipping test plugins in ansible.builtin.__init__ 2'; an error
occurred while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.core 2'; an error occurred
while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.files 2'; an error occurred
while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.mathstuff 2'; an error
occurred while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.uri 2'; an error occurred
while loading: 'NoneType' object has no attribute 'tests'
skipping: [localhost] => {
    "changed": false,
    "false_condition": "env is not defined",
    "skip_reason": "Conditional result was False"
}

PLAY [Test Connection and Determine Remote User] *******************************
Trying secret FileVaultSecret(filename='/Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.vault_pass') for vault_id=default

TASK [connection : Require manual definition of remote-user] *******************
task path: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/connection/tasks/main.yml:2
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/filter/__init__ 2.py) as it seems to be invalid:
module 'ansible.plugins.filter.4068811383039920102___init__ 2' has no attribute
'FilterModule'
[WARNING]: Skipping filter plugins in ansible.builtin.__init__ 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.core 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.encryption 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.mathstuff 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.urls 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.urlsplit 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/test/__init__ 2.py) as it seems to be invalid: module
'ansible.plugins.test.7350187003863887113___init__ 2' has no attribute
'TestModule'
[WARNING]: Skipping test plugins in ansible.builtin.__init__ 2'; an error
occurred while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.core 2'; an error occurred
while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.files 2'; an error occurred
while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.mathstuff 2'; an error
occurred while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping test plugins in ansible.builtin.uri 2'; an error occurred
while loading: 'NoneType' object has no attribute 'tests'
[WARNING]: Skipping plugin (/Users/fabioscheeren/Documents/Documenten - MacBook
Pro van Fabio/projects/keurslager-
vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-
packages/ansible/plugins/filter/__init__ 2.py) as it seems to be invalid:
module 'ansible.plugins.filter.4068811383039920102___init__ 2' has no attribute
'FilterModule'
[WARNING]: Skipping filter plugins in ansible.builtin.__init__ 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.core 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.encryption 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.mathstuff 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.urls 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
[WARNING]: Skipping filter plugins in ansible.builtin.urlsplit 2'; an error
occurred while loading: 'NoneType' object has no attribute 'filters'
skipping: [128.199.53.64] => {
    "changed": false,
    "false_condition": "dynamic_user | default(true) and ansible_user is not defined and cli_ask_pass | default(false)",
    "skip_reason": "Conditional result was False"
}

TASK [connection : Check whether Ansible can connect as root] ******************
task path: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/connection/tasks/main.yml:9
Using module file /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible/modules/command.py
Pipelining is enabled.
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: fabioscheeren
<localhost> EXEC /bin/sh -c '/Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/.trellis/virtualenv/bin/python3 && sleep 0'
ok: [128.199.53.64 -> localhost] => {
    "changed": false,
    "failed_when_result": false,
    "module_stderr": "/bin/sh: /Users/fabioscheeren/Documents/Documenten: No such file or directory\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 127
}

TASK [connection : Warn about change in host keys] *****************************
task path: /Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/connection/tasks/main.yml:20
fatal: [128.199.53.64]: FAILED! => {
    "msg": "The conditional check ''REMOTE HOST IDENTIFICATION HAS CHANGED' in connection_status.stdout' failed. The error was: error while evaluating conditional ('REMOTE HOST IDENTIFICATION HAS CHANGED' in connection_status.stdout): 'dict object' has no attribute 'stdout'. 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/Users/fabioscheeren/Documents/Documenten - MacBook Pro van Fabio/projects/keurslager-vanmelick.nl/trellis/roles/connection/tasks/main.yml': line 20, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Warn about change in host keys\n  ^ here\n"
}

PLAY RECAP *********************************************************************
128.199.53.64              : ok=1    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
localhost                  : ok=0    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

exit status 2

[jbiggs1981]

I had this exact issue a week or so ago. Are you sure you’ve cleared out your known hosts file?

I used the following command:

ssh-keygen -f "/home/user/.ssh/known_hosts" -R "[<host>]:<port>"

Do you maybe need to restart ssh-agent after this? You shouldn’t do but might be worth a shot. I’m fairly sure ssh-agent has nothing to do with known_hosts but still might be worth a shot.