Are you getting any 504 or 502 errors by chance?
I think this issue might be related to my recent issue: 504 Time-out - upstream timed out
If I were you, I’d download the Sendgrid Wordpress Plugin. So far using the API keys seems to fix the issue. I’ll probably stick with this fix rather than debug the port issue.