, 8 min read

From Hiawatha to NGINX

Since mid of August I switched from Hiawatha web-server to NGINX web-server. I initially intended to use OpenLiteSpeed web-server. See Installing OpenLiteSpeed on Arch Linux, but installation and configuration of OpenLiteSpeed turned out to be complicated. I had previously experimented and used Lighttpd.

1. Motivation

The author and maintainer of Hiawatha, Hugo Leisink, on 18-Feb-2019 stated on his weblog:

Many times, I wondered whether I should keep going on with the project or not, but somehow I always found a reason to continue. But not this time. Recently, a serious issue was found in the Hiawatha webserver and the fact that I didn't care much, made me realize that it's time to stop.

Clearly, Hiawatha will never support HTTP/2 or HTTP/3 ... new features will be based on what I need, not on what is needed for a webserver in general.

Over the years he did not change his opinion on that. So it clearly was time to find a web-server which is fully maintained and offers below functionality:

  1. Brotli compression
  2. HTTP/3 and QUIC
  3. URL rewriting
  4. Built-in caching like Varnish

In Set-Up Hiawatha Web-Server I compared the size of various web-servers.

web-server #header #C source LOC
Hiawatha 11.3 155 136 206,878
NGINX 1.25 136 259 229,625

2. NGINX installation

Installing NGINX is pretty simple as it is contained in the Extra-repository of Arch Linux. For installing the Brotli extension you need to install the NGINX source code, then download the Brotli module, and compile the module. Below comment from the original GitHub repository is important:

You will need to use exactly the same ./configure arguments as your Nginx configuration and append --with-compat --add-dynamic-module=/path/to/ngx_brotli to the end, otherwise you will get a "module is not binary compatible" error on startup. You can run nginx -V to get the configuration arguments for your Nginx installation. Then

$ cd nginx-1.25
$ ./configure --with-compat --add-dynamic-module=/path/to/ngx_brotli
$ make modules

A concrete example for installing brotli-1.0rc. Switch to root user and go to directory /usr/src/nginx. In below configure the majority of the command line is the from nginx -V.

./configure --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=stderr --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-cc-opt='-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/nginx-mainline/src=/usr/src/debug/nginx-mainline -flto=auto' --with-ld-opt='-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto' --with-compat --with-debug --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-pcre-jit --with-stream --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-compat --add-dynamic-module=/tmp/ngx_brotli-1.0.0rc

Now make modules and copy the two files

cd objs
cp -p ngx_http_brotli_filter_module.so /usr/lib/nginx/
cp -p ngx_http_brotli_static_module.so /usr/lib/nginx/

3. Brotli compilation for Arch Linux

Go to /tmp directory.

git clone https://github.com/google/ngx_brotli.git
cd ngx_brotli
git submodule update --init

Go to NGINX source code: cd /usr/src/nginx, switch to root user.

./configure `nginx -V 2>&1 | perl -ne 's/ --with-cc-opt='.+'//g; print $1 if /configure arguments: (.+)$/;'` --with-compat --add-dynamic-module=/tmp/ngx_brotli
make
cd objs
cp -p ngx_http_brotli_filter_module.so /usr/lib/nginx/
cp -p ngx_http_brotli_static_module.so /usr/lib/nginx/
systemctl restart nginx

4. NGINX configuration

For the special case w.r.t. body-size I had already written on this here: nginx: 413 Request Entity Too Large - File Upload Issue. The general structure of a NGINX configuration is like below:

some global configuration;
http {
    server A {
        list 80;
    }
    server B {
        listen 443;
    }
}

All the rewriting rules for port 80 and 443 are the same, just copied from the top server to the bottom server config.

#user http;
worker_processes  1;

error_log  /var/log/nginx/error.log;

load_module /usr/lib/nginx/ngx_http_brotli_filter_module.so;
load_module /usr/lib/nginx/ngx_http_brotli_static_module.so;


events {
    worker_connections  1024;
}


http {
    root   /srv/http;
    index  index.html;
    client_max_body_size 15900M;

    http2 on;
    gzip  on;
    brotli on;
    brotli_comp_level 10;
    brotli_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
    gzip_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;

    fastcgi_cache_path /var/cache/nginx/ keys_zone=nginxpc:720m inactive=720m;
    fastcgi_cache_key "$request_method$request_uri";
    fastcgi_cache nginxpc;
    fastcgi_cache_valid 720m;


    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';
    log_format hiawatha_format '$remote_addr|$time_local|$status|$bytes_sent|$request|$http_referer|$http_user_agent|$host:$server_port|$https';
    access_log  /var/log/nginx/access.log hiawatha_format;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    http3 on;
    http3_hq on;
    types_hash_max_size 4096;

    server {
        listen       80;
        server_name  localhost;

        rewrite "^(/*)$" "/blog" redirect;
        rewrite "^/aux/search.php$" "/rewrite/sndsaaze/public/aux/search.php" last;
        rewrite "^/(404\.html|feed\.xml|sitemap\.html|sitemap\.xml)$" "/rewrite/sndsaaze/public/index.php?/$1" last;
        rewrite "^/(aux|blog|music|gallery)($|/.*)"  "/rewrite/sndsaaze/public/index.php?/$1$2" last;

        #charset koi8-r;

        error_page  404              /rewrite/sndsaaze/public/index.php?/404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

        location ~ \.php$ {
            try_files $fastcgi_script_name =404;

            # default fastcgi_params
            include fastcgi_params;

            # fastcgi settings
            fastcgi_pass			unix:/run/php-fpm/php-fpm.sock;
            fastcgi_buffers			8 16k;
            fastcgi_buffer_size		32k;

            # fastcgi params
            fastcgi_param DOCUMENT_ROOT	$realpath_root;
            fastcgi_param SCRIPT_FILENAME	$realpath_root$fastcgi_script_name;
        }

        location ~ ^/ttyd(.*)$ {
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_pass http://eklausmeier.goip.de:7681/$1;
        }
    }


    server {
        listen       443 quic;
        listen       443 ssl;
        server_name  localhost;

        rewrite "^(/*)$" "/blog" redirect;
        rewrite "^/aux/search.php$" "/rewrite/sndsaaze/public/aux/search.php" last;
        rewrite "^/(404\.html|feed\.xml|sitemap\.html|sitemap\.xml)$" "/rewrite/sndsaaze/public/index.php?/$1" last;
        rewrite "^/(aux|blog|music|gallery)($|/.*)"  "/rewrite/sndsaaze/public/index.php?/$1$2" last;

        ssl_certificate      /etc/hiawatha/eklausmeier.goip.de.pem;
        ssl_certificate_key  /etc/hiawatha/eklausmeier.goip.de.pem;

        # From https://blog.qualys.com/product-tech/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";

        location / {
            # used to advertise the availability of HTTP/3
            add_header Alt-Svc 'h3=":443"; ma=86400';
        }

        location ~ \.php$ {
            try_files $fastcgi_script_name =404;

            # default fastcgi_params
            include fastcgi_params;

            # fastcgi settings
            fastcgi_pass			unix:/run/php-fpm/php-fpm.sock;
            fastcgi_buffers			8 16k;
            fastcgi_buffer_size		32k;

            # fastcgi params
            fastcgi_param DOCUMENT_ROOT	$realpath_root;
            fastcgi_param SCRIPT_FILENAME	$realpath_root$fastcgi_script_name;
        }

        location ~ ^/ttyd(.*)$ {
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_pass http://eklausmeier.goip.de:7681/$1;
        }
    }

}

It is a common error to forget:

brotli_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
gzip_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;

See the examples below, where this configuration has been forgotten:

  1. Analysis of Performance of Demo Open E-Mobility Site
  2. Performance Remarks on PublicoMag Website

Method to check for compression:

curl -D - -H "Accept-Encoding: gzip,deflate,br" --write-out "%{size_download}\n" -o /tmp/prism-css.br http://localhost/jscss/prism.css

5. Caching

By using fastcgi_cache with a quite large retention interval of 720 minutes (=12 hours), I keep already generated pages in the cache for a long time.

If you want to delete specific entries in the cache, the critical line is:

fastcgi_cache_key "$request_method$request_uri";

You can compute the file-name of the cache file by specifying request-method and URL like so:

printf "GET/blog" | md5sum

or

printf "GET/blog/2023/08-29-from-hiawatha-to-nginx" | md5sum

This will print the file-name which you can delete with rm. In our case in the directory /var/cache/nginx. See How to Setup FastCGI Caching with Nginx on your VPS.

6. Deployment

With this very aggressive caching in place, the deployment of my blog changed. Previously, I generated all static files with

php saaze -mortb /tmp/build

and then deployed via

blogdeploy -p

Above deployment script essentially just removes the previous directories and replaces them with the newly generated ones. I did this for the "staging environment" and "production", i.e., on my work-PC and on the self-hosting PC.

Now I rarely generate all static files and use the dynamic mode of Simplified Saaze, i.e., Simplified Saaze generates the HTML file whenever it is actually accessed. Once it is generated then NGINX caches it. So, essentially the generation of the static files is deferred to the actual access time:

$ ls -l /var/cache/nginx | wc
    350    3143   26826