HTTP/2: Only one/two simultaneous requests? (Particular server/mystery issue)

trellis
1

On one particular Trellis server (recent Ubuntu and nginx) only one or two requests are made at a time. This is very noticeable when viewing the media library in thumbnail view with caching disabled (Chrome dev tools network tab). Only one or two images are downloaded at a time. Disabling HTTP/2 and using HTTP1.1 results in many more images downloaded and faster overall download. As this only happens with this particular nginx over HTTP/2 I suspect a subtle configuration issue being the underlying issue, e.g. TLS related.

  • Server has good uplink (simple speedtest showed speeds far beyond what could be a reason for this issue).
  • Client has good internet connection (fiber, symmetric, speedtest shows full expected speed up and down).
  • Different client hardware, internet connections and OS and browsers all experience the same issue.
  • There appears to be one exception though: One internet connection uses a proxy with CONNECT (mandatory) and here the issue does not appear to occur (many concurrent downloads).
  • Issue occurs with static files (most notable when many static files are downloaded), PHP-specific processing time can be ruled out.
  • Forcing HTTP/1.1 instead of HTTP/2 results in many more requests being finished and much faster than with HTTP/2.
  • Qualys SSL Labs gives a A+.
  • nginx debug log proved valuable (nginx in package comes with debug enabled in build). (use debug keyword in site error_log; service nginx stop; service nginx-debug start). In nginx debug log are lots of TLS errors logged for HTTP/2 connections:
SSL_get_error: 2
SSL_do_handshake: -1
  • Affected nginx version and build configuration (nginx -V):
nginx version: nginx/1.27.1
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.27.1/debian/debuild-base/nginx-1.27.1=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

Do you experience the same with current Ubuntu + nginx (latest package update)? When you load the media library page with thumbnail view, with cache disabled (in Chrome Developer Tools, network tab, disable cache option), do the images load all in parallel, or do they load one or two at a time and individually with slow download speed?

2 Likes
2

How is the SSL certificate handled on this server? By Trellis and Lets Encrypt or something else?

3

For the SSL certificate normal Let’s Encrypt is used.

This is really strange: When I use a “normal” mobile or fiber internet connection, the issue with only one/two concurrent connection(s) occurs - when I use a rather locked down WiFi hotspot with a HTTP CONNECT proxy, the issue does not occur.

Apparently there are multiple “flavors” of HTTPS certificate?

4

If this is caused by some obscure TLS issue in nginx/OpenSSL library internally used by it, as the nginx configuration looks fine/stock, I plan to try the following:

  • Renew the Let’s Encrypt issued certificates completely.
  • Renew the Diffie-Hellman key file. (was removed)

Renewing all certificates did not fix the issue.

5 
reusable connection: 1
[...]
*379 SSL_read: 538
*379 SSL_read: -1
*379 SSL_get_error: 2
*379 reusable connection: 0

reusable connection can be 1 (true) or 0 (false).
After the SSL-specific issues it changes from 1 true to 0 (false), there is also only one or two concurrent HTTP/2 request.

6

So for further testing I added a test site and after the provisioning, the issue went away!
Just forcing the re-creation of the certificates alone did not solve the issue - a domain had to be added.

I suspect some software state was changed that caused the issue, which persisted beyond nginx restarts and reboots, and it had to be reset/cleaned up.

1 Like