First rule out DNS issues, how fast does dig look up the IP address of the site? Use ping and check for network issues (short ping times, variations).
Then use a tool like wget or curl and check the response time reported by these, are those really that large? Do they vary a lot?
(Besides profiling) I achieved significant performance improvements by using an object cache (Redis based in that case):
PHP 8 also improved performance compared to PHP 7.x.