Roots Discourse

External IP resolution fails on all my existing projects

Hi there,

I updated one of my projects to the latest Trellis version (1.3.0) and created a new project also with the latest version (1.3.0) and on both projects I start receiving this error when (re)provisioning my remote servers:

TASK [common : Retrieve SSH client IP] ******************************************************************************************************************
System info:
Ansible 2.8.0; Darwin
Trellis Head
---------------------------------------------------

No valid or no response from url https://api.ipify.org/ within 10 seconds
(timeout)
fatal: [178.62.254.13]: FAILED! => {"changed": false}

TASK [common : Fail when unable to retrieve SSH client IP] **********************************************************************************************
System info:
Ansible 2.8.0; Darwin
Trellis Head
---------------------------------------------------
External IP resolution failed. Check that your DNS servers are working. Try to disable DNSCrypt if you are using it.
fatal: [178.62.254.13]: FAILED! => {"changed": false}

So I checked my other projects and I’m getting this error on ALL my projects now?
I recently updated to Catalina 10.15.3 and to Vagrant 2.2.7 and Ansible 2.8.0.

The api.ipify.org website returns my correct IP address on my host machine, but when I curl it:

curl http://api.ipify.org

it returns my IP but with a percentage sign at the end?

xxx.xxx.xxx.xxx%

Right now, provisioning my servers only works when I follow these instructions.

Anyone else seeing this?
Does MacOS Catalina have DNSCrypt installed somewhere?

Thanks!

Nobody else ran in this issue recently? Hmm then it must be network or device related I guess… What kind of 3rd party software could be causing this?

Getting the same issue. Also MacOS 10.15. Absible 2.8.4.

Thought maybe it was pi-hole, but even disconnecting from that and using a different connection got the same error.

So after a bit of digging, I found a solution here: No response from Ipify

Yeah those are the instructions I was referring to in my post.
Weird issue though… Are you using something like a VPN or other DNS software? Because I’m not?

I tried VPN and I tried without. Without diving deeper into it, I feel like maybe its the change from bash to zsh in 10.15. Probably wrong, but it’s a thought at the moment.

That’s what I thought for a moment, but then I realized that I already switched to zsh before upgrading to Catalina and had no issues related to this?

You can try to set ssh_client_ip_lookup: false. It will skip that check which might allow the provisioning to continue.

I don’t think this is related to the % sign at the end since the error message indicates a timeout?

The ipify_facts module uses curl http://api.ipify.org?format=json anyway as well.

Ok, I just successfully re-provisioned my remote server on the exact same project with the exact same Ansible 2.8.0 & Vagrant 2.2.7 versions but on MacOS Mojave 10.14.6 without any errors.

So it’s definitely a Catalina thing I guess? Does Catalina somehow mess with the DNS lookups?

It shouldn’t… Catalina causing DNS issues would be something we’d hear about I’d imagine. Though it definitely has bugs so maybe something more specific to your setup :slightly_frowning_face:

Yeah probably, but what? I just synced all our essential local software version with my colleague and now we both have the same versions of:

MacOS 10.15.3
Vagrant 2.2.7
Ansible 2.8.0
Python 3.7.6
PHP 7.3.11
Trellis 1.2.0

We both destroyed and recreated our local VM, tried to re-provision the remote and no errors on my colleague’s machine and the same error again on my machine.

The only difference in the playbook log I can detect is this warning:

[WARNING]: Could not find aptitude. Using apt-get instead

I have no VPN software installed, I don’t have any custom DNS or proxy settings in my Network preferences and I don’t have any weird things in my host file…

We can both connect to the remote server as admin and web with our own ssh keys.

What else could be causing this difference?
Could it be that my SSH key is blocked on the remote once the lookup has failed once?

My full verbose error log:

TASK [common : ipify_facts] ***************************************************************************************************************************
task path: ~/path/to/my/project/trellis/roles/common/tasks/main.yml:152
Using module file /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ansible/modules/net_tools/ipify_facts.py
Pipelining is enabled.
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: Username
<localhost> EXEC /bin/sh -c '/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 && sleep 0'
System info:
  Ansible 2.8.0; Darwin
  Trellis 1.2.0: October 11th, 2019
---------------------------------------------------
No valid or no response from url https://api.ipify.org/ within 10 seconds
(timeout)
fatal: [198.199.125.53]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_url": "https://api.ipify.org/",
            "timeout": 10,
            "validate_certs": true
        }
    }
}

TASK [common : fail] **********************************************************************************************************************************
task path: ~/path/to/my/project/trellis/roles/common/tasks/main.yml:158
System info:
  Ansible 2.8.0; Darwin
  Trellis 1.2.0: October 11th, 2019
---------------------------------------------------
External IP resolution failed. Check that your DNS servers are working. Try
to disable DNSCrypt if you are using it.
fatal: [198.199.125.53]: FAILED! => {
    "changed": false
}

Ok I got a bit further, when I disable the validate_certs in the ipify_facts task: in /roles/common/tasks/main.yml:

- name: Retrieve SSH client IP
  block:
    - ipify_facts:
        validate_certs: no
      delegate_to: localhost
      become: no
      when: env != 'development' and ssh_client_ip_lookup | default(true)
      tags: [fail2ban, ferm]
  rescue:
    - fail:
        msg: "External IP resolution failed. Check that your DNS servers are working. Try to disable DNSCrypt if you are using it."

I can re-provision without any errors?
Which certificates is ipify_facts trying to validate anyway? Don’t understand…

The default API url is https://api.ipify.org/

Ansible says:

validate_certs:
    description:
      - When set to C(NO), SSL certificates will not be validated.
    type: bool
    default: yes
    version_added: "2.4"

The certificates should be valid so it’s strange that it’s apparently not for you?

But which certificates? From the ipify.org domain?
On the same network, but different computer it does validate correctly… :thinking:

Yes, for api.ipify.org since it’s connecting to it via https