Got it working.
If anyone is interested, you have to change the web_user in users.yml to www-data which is the default user that nginx uses. Then your web_user will have the appropriate permissions and will be able to delete the cache either with a recursive delete PHP script or simply a shell command using rm.
After you change the user to www-data you will not only have to reprovision the server but also spin up an entirely new Ubuntu instance (if you are using Digital Ocean, you can just rebuild the droplet with the same image).
This way you can set the expiry time of the server cache way beyond the “micro” oriented cache and clear it whenever needed.