That’s exactly what I did. The problem is WordPress (via redirect_canonical I think) does a 301 redirect when it sees that the port doesn’t match.
Tried adding port_in_redirect off; to nginx server block but it didn’t seem to help…
And the funny thing is I already strip off the port from HTTP Host header in my .vcl file. Should I not be doing that? (Oh wait… This only matters if the request URL from browser contains a port number. Nevermind…)