1. Member of 512KB club
2. Member of 250KB club

This 1MB club has 781 members as of today.

Becoming member there is by using:

git init
echo "---\npageurl: eklausmeier.goip.de\nsize: 133.1\n---\n" > eklausmeier.goip.de.md
git format-patch -1

The resulting file is then sent via e-mail to patches@btxx.org.

A renewed check of my blog on https://tools.pingdom.com shows:

1. Performance grade: 93
2. Page size: 102.5 KB
3. Load time: 220ms from Frankfurt
4. Requests: 7

Content and requests by content type:

Waterfall diagram:

1. Original: WordPress version, publicomag.com
2. Modified: Simplified Saaze version of PublicoMag

The original website https://www.publicomag.com is hosted on Cloudflare. It uses WordPress.

1. Comparison. For the comparison I use the website tools.pingdom.com, which provides various metrics to evaluate the performance of a website:

1. Page size
2. Number of requests
3. Load time
4. Concrete tips to improve performance
5. Waterfall diagram of requests
6. Breakdown of content types

The first few tests in Pingdom were conducted for Europe/Frankfurt, as I host all stuff on below machine in my living room not far from Frankfurt.

The post in question is Inspiration als Energiequelle: Neues vom grünen Hauptmann von Köpenick. The version using Simplified Saaze is here. This post contains 5 images and 13 comments. All images are served directly from https://www.publicomag.com. I.e., no side has any advantage in that respect. I had already blogged on this here: Performance Remarks on PublicoMag Website.

The results are thus:

Original (WordPress)	Modified (Simplified Saaze)

The results for the original website, based on WordPress, are indeed worse on every dimension: page size, load time, number of requests. In comparison to the modified version using Simplified Saaze the ratio is roughly:

1. Page size is almost than 3:1
2. Load time is almost 8:1
3. Number of requests is more than 5:1

So Simplified Saaze is better in all dimensions by a factor. Load time is particularly striking. This is quite noteworthy as the Simplified Saaze version is entirely self-hosted, i.e., upload to the internet is limited to 50 MBit/s!

The recommendations for the original website are therefore not overly surprising:

The missing compression is clearly an oversight on the web-server part.

The breakdown of the content type for the original WordPress website is:

One can clearly see that half of the page size are images, one third is JavaScript, fonts and CSS each have roughly 8%, the actual HTML content is just 2%.

I uploaded the Simplified Saaze version to Netlify, which provides CDN functionality. I measured again the WordPress post requested from San Francisco, and the Simplified Saaze version from San Francisco. The measurements are pretty similar to the Frankfurt results.

Original (WordPress) San Francisco	Modified (Simplified Saaze) San Francisco

Surprisingly, the Simplified Saaze version on Cloudflare has loading time of 5.24 seconds from San Francisco. Vercel is in line with Netlify and has load times of 385 ms.

For comparison I also hosted the Simplified Saaze version on https://www.lima-city.de. Load times are 248 ms for Frankfurt. Load times are 943 ms for San Francisco.

2. Modified website. The breakdown of the modified site, based on Simplified Saaze, is as below.

Actual loading of the modified site will roughly follow below waterfall diagram. This waterfall diagram shows that a major part of the loading time is spent in loading Google's Playfair fonts. This is quite surprising. The other fonts from Google load in record time.

3. Known limitations. Alexander Wendt wrote about some general limitations with the used technical solution so far:

Trotzdem sind wir zuversichtlich, demnächst das eine oder andere technische Problem hoffentlich befriedigend zu lösen. Generell braucht Publico eine schrittweise Erneuerung seiner technischen Plattform, die in ihren Grundzügen von 2017 stammt.

4. Low powered devices. Dan Luu noted in How web bloat impacts users with slow devices that many so-called modern websites are more or less unusable on older or low-powered devices. Some quotes:

If you've never used a low-end device like this, the general experience is that many sites are unusable on the device and loading anything resource intensive (an app or a huge website) can cause crashes.

Software developers underestimate the impacts low-pwered devices have, when loading websites:

People seem to really underestimate the dynamic range in wealth and income across the world.

It offers below features:

1. Responsive with media breaks for large and small screens, and for printing.
2. Top menu with submenus.
3. Two column using CSS grid, "Holy Grail Layout".
4. Multiple blogs:
   • Each category has its own blog by using filtering.
   • Each author has its own blog by using filtering.
   • Aggregate blog, i.e., the combination of the above.
5. Using the <!--more--> tag to showcase the initial content of a blog post.
6. Sitemap in HTML and XML, RSS feed.
7. WebAssembly based search using pagefind.
8. No cookies, therefore no annoying cookie banner required.

The theme looks like this:

This theme is modeled after the blog from Alexander Wendt. That blog is powered by WordPress and hosted on Cloudflare. I have written on this PublicoMag website: Performance Remarks on PublicoMag Website. Alexander Wendt started this blog in October 2017. The number of posts per year are given in below table. Year 2024 is not complete. As time passes the year 2024 will have more and more posts.

Number of comments were counted like this (varying 2017 to 2024):

perl -ne 'if (/^(\d+) Kommentare <\/h5>/) { $s+=$1; printf("%d\t%d\t%s\n",$1,$s,$ARGV); }' 2017*

1. Installation

There are two parts in the installation.

1. Install the theme including content and the Simplified Saaze static site generator using composer:

$ composer create-project eklausme/saaze-wendt
Creating a "eklausme/saaze-wendt" project at "./saaze-wendt"
Installing eklausme/saaze-wendt (v1.0)
  - Downloading eklausme/saaze-wendt (v1.0)
  - Installing eklausme/saaze-wendt (v1.0): Extracting archive
Created project in /tmp/T/saaze-wendt
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking eklausme/saaze (v2.2)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Downloading eklausme/saaze (v2.2)
  - Installing eklausme/saaze (v2.2): Extracting archive
Generating optimized autoload files
No security vulnerability advisories found.
        real 3.08s
        user 0.48s
        sys 0
        swapped 0
        total space 0

2. The Simplified Saaze installation is described in Simplified Saaze. It documents how to check for PHP version, check for yaml-parsing, FFI, MD4C extension, etc.

Once everything is installed, just run php saaze -mor.

2. Downloading all WordPress content

We need a list or URLs available.

Below approach did not work: We use the month list in WordPress.

for i in `seq 2018 2023`; do for j in `seq -w 01 12`; do curl https://www.publicomag.com/$i/$j/ > m$i-$j.html; done; done

Special cases for 2017 and 2024:

curl https://www.publicomag.com/2017/10/ -o m2017-10.html
curl https://www.publicomag.com/2017/11/ -o m2017-11.html
curl https://www.publicomag.com/2017/12/ -o m2017-12.html
...
curl https://www.publicomag.com/2024/03/ -o m2024-03.html

It turned out that the month-lists lack links. To be exact: It lacks more than 466 URLs.

This approach fetches all links:

$ curl https://www.publicomag.com/ -o wendt-p1.html
$ time ( for i in `seq 2 124`; do
    curl https://www.publicomag.com/page/$i/ -o wendt-p${i}.html;
  done )

This creates 124 files:

$ ls -alFt | head
total 25580
drwxr-xr-x 2 klm klm   4096 Apr  2 11:34 ./
drwxr-xr-x 4 klm klm   4096 Apr  2 11:33 ../
-rw-r--r-- 1 klm klm 208194 Apr  2 11:28 wendt-p1.html
-rw-r--r-- 1 klm klm 187908 Apr  2 11:27 wendt-p124.html
-rw-r--r-- 1 klm klm 203575 Apr  2 11:27 wendt-p123.html
-rw-r--r-- 1 klm klm 206497 Apr  2 11:27 wendt-p122.html
-rw-r--r-- 1 klm klm 207572 Apr  2 11:27 wendt-p121.html
-rw-r--r-- 1 klm klm 207970 Apr  2 11:27 wendt-p120.html
-rw-r--r-- 1 klm klm 206010 Apr  2 11:27 wendt-p119.html
...

List of URLs:

perl -ne 'print $1."\n" if /<h2 class="post-title"><a href="([^"]+)"/' wendt-p*.html > allURL

Downloading all posts uses below Perl script blogwendtcurl:

#!/bin/perl -W
# Download content from www.publicomag.com (Alexander Wendt) given a list of URLs
# Elmar Klausmeier, 05-Mar-2024

use strict;
my $fn;
my @F;

while (<>) {
    chomp;
    @F = split('/');
    $F[5] =~ s/a%cc%88/ä/;
    $fn = $F[3] . '-' . $F[4] . '-' . $F[5] . '.html';
    printf $fn . "\n";
    `curl $_ -o $fn`;
}

This creates a list of HTML files:

$ ls -alFt | head
total 175856
drwxr-xr-x 3 klm klm   4096 Mar  7 19:16 ../
drwxr-xr-x 2 klm klm  69632 Mar  5 19:53 ./
-rw-r--r-- 1 klm klm 203580 Mar  5 19:53 2024-03-18471.html
-rw-r--r-- 1 klm klm 252784 Mar  5 19:53 2024-03-wenn-die-zukunft-ans-fenster-des-gruenen-hauses-klopft.html
-rw-r--r-- 1 klm klm 203765 Mar  5 19:53 2024-03-zeller-der-woche-niedere-gruende.html
-rw-r--r-- 1 klm klm 203337 Mar  5 19:53 2024-02-zeller-der-woche-widerstaendler.html
-rw-r--r-- 1 klm klm 231904 Mar  5 19:52 2024-02-das-nie-wieder-deutschland-und-seine-millionen-fuer-judenhasser.html
...

3. Analyzing content types

1. Fonts.

1. Logo: Shadows Into Light Two, original uses image instead. Another contender could be Croissant One.
2. Text: Playfair Display

2. Categories. Categories over all posts are as follows:

$ perl -ne 'print $1."\n" if / hentry category-([-\w]+)/' *.html | sort | uniq -c | sort -rn
    595 spreu-weizen
    486 politik-gesellschaft
    122 medien-kritik
     28 fake-news
      3 hausbesuch
      1 film

Different, i.e., multiple, categories can be attributed to a single post. However, the majority of posts only has a single category attached.

In the above list there is no categoriy "alte-weise". I added this category.

We want to convert images in "Alte-Weise" to text. That way loading those pages should be way quicker. Therefore we need to download those images and convert them with tesseract.

3. URLs. Below Perl one-liners produces a list of URLs for the images.

perl -ne 'print "$1$2\n" if (/^<meta property="og:image"\s+content="(https:\/\/www\.publicomag\.com\/wp-content\/uploads\/\d+\/\d+\/)(Alte-Weis[^"]+|AlteWeise[^"]+|AlteuWeise[^"]+|auw-[^" ]+|aub_[^"]+|auw_[^"]+|AuW_[^"]+|AW_[^"]+|OW[^"]+)"/)' *.html | sort > ../allAlte-WeiseURL

Downloading these images:

perl -ane 'chomp; @F=split(/\//); `curl $_ -o $F[7]`' ../allAlte-WeiseURL
curl https://www.publicomag.com/wp-content/uploads/2023/01/Alte-Weise_C.Wright-Mills-1011x715.jpg -o Alte-Weise_Wright_Mills-scaled.jpg

4. JavaScript. A huge number of JavaScript libraries are loaded. We will get rid of them all.

1. Google Analytics
2. JQuery Minimal
3. JQuery Migrate
4. WordPress User Avatar
5. Buzzblog Hercules Likes
6. Borlabs Cookies Prioritize
7. WordPress GDPR Compliance
8. Comment Reply
9. Contact Form
10. JQuery Easing for Buzzblog
11. JQuery MagnificPopup for Buzzblog
12. JQuery Plugins for Buzzblog
13. JQuery JustifiedGallery for Buzzblog
14. Buzzblog Bootstrap
15. Owl Carussel for Buzzblog
16. Buzzblog AnimatedHeader
17. Shariff
18. MailPoet
19. Akismet
20. Borlabs Cookies Minimal

4. Reducing number of images

An easy target is the logo: this was replaced with plain text. This saves one roundtrip to the web-server.

1. For the category "alte-weise" the entire image with text is converted to two elements:

1. An image
2. The actual text

The image is scanned with tesseract.

That way the text can be searched via Pagefind. Also, the required bandwidth is reduced.

Old:

New:

The new approach is to use a blockquote, where the CSS puts an image on top:

blockquote blockquote {
    background: transparent no-repeat top/30% url('/img/Alte-Weise-Kopf.svg');
    text-align:center;
    padding-left:2rem;
    padding-right:2rem;
    padding-top:12rem;
    padding-bottom:1rem;
    background-color:#b6c7c8; border-radius:2.5rem
}

The actual text in Markdown is then:

>> „Zweifel ist nicht das Gegenteil, sondern ein Element des Glaubens.“
>>
>> Paul Tillich

That way the ordinary blockquote in Markdown (single >) is left free to be used for citations.

Obviously, entering the text in >> is way easier than producing an image for each epigram.

2. Care was taken to reduce the number of images needed for the social media icons.

Old:

New:

That reduces loading eight images. However, you need to load some font glyphs.

<a style="background-color:SkyBlue; color:white" href="https://telegram.me/share/url?url=<?=$urlEncoded?>&text=<?=$titleEncoded?>"
   title="Teilen auf Telegram" target=_blank>&nbsp;<span class=symbols>&#x01fbb0;</span>&nbsp;Telegram&nbsp;</a>

In particular this symbol U+1fbb0 is %F0%9F%AE%B0 when URL encoded:

@import url('https://fonts.googleapis.com/css2?family=Noto+Sans+Symbols+2&text=%F0%9F%97%8F%F0%9F%AE%B0%F0%9F%96%82%F0%9F%96%A8');

Similarly, symbol U+1f5cf is %F0%9F%97%8F when URL encoded.

5. Converting WordPress HTML to Markdown

Perl script blogwendtmd is used to convert a single HTML file to Markdown.

$ time ( for i in *.html; do blogwendtmd $i; done )
        real 94.95s
        user 136.51s
        sys 0
        swapped 0
        total space 0

The long runtime is exclusively for running tesseract, i.e., the conversion from image to text. Once all WordPress posts are converted to Markdown, this script no longer needs to be run, obviously.

blogwendtmd is 180 lines of Perl code.

Listing of all authors and their corresponding directories.

$ perl -ne 'print $1."\n" if /\/author\/([^\/]+)\//' 2*.html | sort -u
alexander
archi-bechlenberg
bernd-zeller
cora-stephan
david-berger
hansjoerg-mueller
joerg-friedrich
matthias-matussek
redaktion
samuel-horn
wolfram-ackner

Each of these authors have a separate index beneath /author/.

Generating all yearly overviews:

for i in *; do ( echo $i; cd $i; blogwendtdate -gy$i *.md > index.md ) done

Perl script blogwendtdate generates a Markdown file, which contains all articles for the corresponding year. This script first has to store all posts for one year in a hash, sort it according to date in the frontmatter.

my @L;	# list of posts in a year, in the beginning not necessarily sorted

sub markdownfile(@) {
    my $f = $_[0];
    my ($flag,$title,$date,$draft) = (0,"","",0);
    open(F,"<$f") || die("Cannot open $f");
    while (<F>) {
        if (/^\-\-\-\s*$/) {
            last if (++$flag >= 2);
        . . .
    }
    if ($draft == 0  &&  length($title) > 0  &&  length($date) > 0) {
        push(@L, sprintf("%s: [%s](%s%s)",$date,$title,$prefix,substr($f,0,-3)) );
    }
    close(F) || die("Cannot close $f");
}

while (<@ARGV>) {
    #printf("ARGV=|%s|\n",$_);
    next if (substr($_,-8) eq "index.md");
    markdownfile($_);
}

for (sort @L) {
    printf("%d. %s\n",++$cnt,$_);
}

Many HTML errors were corrected, which were reported by Nu Html Checker. See for example das-magische-sprechen-schafft-macht-fuer-den-augenblick.

6. Handling comments

The Publico blog contains comments, where readers have left their thoughts. In Perl script blogwendtmd we detect comments by checking for <h5> tags for the beginning, and pinglist for the end of all comments.

if (/^<ul class="pinglist">/) { $flag = 0; next; }
elsif (/<h5 class="comments-h">/) {
    ...
    $flag = 1;
}
next if ($flag == 0);

We refrained from integrating the commenting system HashOver. It is not difficult, as we have already demonstrated in the Lemire theme. However, for a political blog a comment system is rather "dangerous", as it can attract rather unwelcoming writings. Under German law the hoster of these comments becomes liable. Essentially, you therefore must check every comment manually:

... da die Kommentare alle gesichtet werden müssen und die Redaktion nach wie vor aus dem Gründer Alexander Wendt und einer Teilzeitredakteurin besteht, können sie nicht umgehend online gehen.

In light of the high volume of comments HashOver should most probably be added.

7. Running static site generator

In serial mode it takes less than 3 seconds to build 19 collections without comments. With comments it takes less than 6 seconds to process 23 thousand pages, see below. This build time can be almost halved by using parallelisation with -p16.

$ time php saaze -morb /tmp/build
Building static site in /tmp/build...
    execute(): filePath=./content/alexander.yml, nSIentries=770, totalPages=39, entries_per_page=20
    execute(): filePath=./content/alte-weise.yml, nSIentries=131, totalPages=7, entries_per_page=20
    execute(): filePath=./content/archi-bechlenberg.yml, nSIentries=5, totalPages=1, entries_per_page=20
    execute(): filePath=./content/bernd-zeller.yml, nSIentries=332, totalPages=17, entries_per_page=20
    execute(): filePath=./content/cora-stephan.yml, nSIentries=1, totalPages=1, entries_per_page=20
    execute(): filePath=./content/david-berger.yml, nSIentries=1, totalPages=1, entries_per_page=20
    execute(): filePath=./content/fake-news.yml, nSIentries=28, totalPages=2, entries_per_page=20
    execute(): filePath=./content/film.yml, nSIentries=1, totalPages=1, entries_per_page=20
    execute(): filePath=./content/hansjoerg-mueller.yml, nSIentries=2, totalPages=1, entries_per_page=20
    execute(): filePath=./content/hausbesuch.yml, nSIentries=2, totalPages=1, entries_per_page=20
    execute(): filePath=./content/joerg-friedrich.yml, nSIentries=2, totalPages=1, entries_per_page=20
    execute(): filePath=./content/mag.yml, nSIentries=1235, totalPages=62, entries_per_page=20
    execute(): filePath=./content/matthias-matussek.yml, nSIentries=1, totalPages=1, entries_per_page=20
    execute(): filePath=./content/medien-kritik.yml, nSIentries=123, totalPages=7, entries_per_page=20
    execute(): filePath=./content/politik-gesellschaft.yml, nSIentries=486, totalPages=25, entries_per_page=20
    execute(): filePath=./content/redaktion.yml, nSIentries=112, totalPages=6, entries_per_page=20
    execute(): filePath=./content/samuel-horn.yml, nSIentries=3, totalPages=1, entries_per_page=20
    execute(): filePath=./content/spreu-weizen.yml, nSIentries=596, totalPages=30, entries_per_page=20
    execute(): filePath=./content/wolfram-ackner.yml, nSIentries=6, totalPages=1, entries_per_page=20
Finished creating 19 collections, 19 with index, and 1248 entries (2.58 secs / 809.47MB)
#collections=19, parseEntry=0.7290/23712-19, md2html=1.1983, toHtml=1.2839/23712, renderEntry=0.1562/1248, renderCollection=0.0403/224, content=23712/0
    real 5.16s
    user 4.36s
    sys 0
    swapped 0
    total space 0

Running pagefind, i.e., indexing al keywords for the WebAssembly based search functionality:

$ time pagefind -s . --exclude-selectors aside --exclude-selectors footer --force-language=de

Running Pagefind v1.0.4
Running from: "/tmp/buildwendt"
Source:       ""
Output:       "pagefind"

[Walking source directory]
Found 1473 files matching **/*.{html}

[Parsing files]
Did not find a data-pagefind-body element on the site.
↳ Indexing all <body> elements on the site.

[Reading languages]
Discovered 1 language: de

[Building search indexes]
Total:
  Indexed 1 language
  Indexed 1473 pages
  Indexed 133261 words
  Indexed 0 filters
  Indexed 0 sorts

Finished in 19.644 seconds
        real 19.87s
        user 18.28s
        sys 0
        swapped 0
        total space 0

It would take 11 seconds without comments, i.e., indexing 77168 words.

8. Collections

There are quite a number of collections at play in this theme. The most important one being mag (short for magazine). This directory contains all the blog posts. All the other collections are just symbolic links to mag, i.e., they do not contain additional content.

total 96
drwxr-xr-x  4 klm klm 4096 Apr 27 17:11 ./
drwxr-xr-x  7 klm klm 4096 May 13 13:00 ../
lrwxrwxrwx  1 klm klm    3 Mar 26 21:48 alexander -> mag/
-rw-r--r--  1 klm klm  273 Apr  2 18:56 alexander.yml
lrwxrwxrwx  1 klm klm    3 Apr 27 17:11 alte-weise -> mag/
-rw-r--r--  1 klm klm  225 Apr 27 17:10 alte-weise.yml
lrwxrwxrwx  1 klm klm    3 Mar 31 17:22 archi-bechlenberg -> mag/
-rw-r--r--  1 klm klm  495 Apr  2 18:58 archi-bechlenberg.yml
lrwxrwxrwx  1 klm klm    3 Mar 31 17:17 bernd-zeller -> mag/
-rw-r--r--  1 klm klm  213 Apr  2 18:01 bernd-zeller.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 15:18 cora-stephan -> mag/
-rw-r--r--  1 klm klm  707 Apr  2 19:01 cora-stephan.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 15:17 david-berger -> mag/
-rw-r--r--  1 klm klm  761 Apr  2 19:06 david-berger.yml
drwxr-xr-x  2 klm klm 4096 Apr  2 16:24 error/
-rw-r--r--  1 klm klm   88 Apr  2 16:21 error.not_used_yml
lrwxrwxrwx  1 klm klm    3 Apr  2 19:25 fake-news -> mag/
-rw-r--r--  1 klm klm  216 Apr  2 19:42 fake-news.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 19:25 film -> mag/
-rw-r--r--  1 klm klm  201 Apr  2 19:43 film.yml
lrwxrwxrwx  1 klm klm    3 Mar 31 17:22 hansjoerg-mueller -> mag/
-rw-r--r--  1 klm klm  318 Apr  2 18:56 hansjoerg-mueller.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 19:25 hausbesuch -> mag/
-rw-r--r--  1 klm klm  219 Apr  2 19:42 hausbesuch.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 15:18 joerg-friedrich -> mag/
-rw-r--r--  1 klm klm  222 Apr  2 18:01 joerg-friedrich.yml
drwxr-xr-x 10 klm klm 4096 May 12 20:56 mag/
-rw-r--r--  1 klm klm  110 Apr  1 22:25 mag.yml
lrwxrwxrwx  1 klm klm    3 Mar 31 17:22 matthias-matussek -> mag/
-rw-r--r--  1 klm klm  228 Apr  2 18:02 matthias-matussek.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 19:25 medien-kritik -> mag/
-rw-r--r--  1 klm klm  234 Apr  2 19:27 medien-kritik.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 17:47 politik-gesellschaft -> mag/
-rw-r--r--  1 klm klm  255 Apr  2 17:59 politik-gesellschaft.yml
lrwxrwxrwx  1 klm klm    3 Mar 31 17:16 redaktion -> mag/
-rw-r--r--  1 klm klm  202 Apr  2 18:03 redaktion.yml
lrwxrwxrwx  1 klm klm    3 Mar 31 17:21 samuel-horn -> mag/
-rw-r--r--  1 klm klm  259 Apr  2 19:03 samuel-horn.yml
lrwxrwxrwx  1 klm klm    3 Apr  2 19:25 spreu-weizen -> mag/
-rw-r--r--  1 klm klm  231 Apr  2 19:27 spreu-weizen.yml
lrwxrwxrwx  1 klm klm    3 Mar 31 17:22 wolfram-ackner -> mag/
-rw-r--r--  1 klm klm  542 Apr  2 19:05 wolfram-ackner.yml

The collection yaml files look like this. First mag.yml:

title: Publico
sort_field: date
sort_direction: desc
index_route: /
entry_route: /{slug}
more: true
rss: true

Now alexander.yml, which filters for author:

title: Publico - Autor Alexander Wendt
subtitle: "Alexander Wendt ist Herausgeber von Publico."
sort_field: date
sort_direction: desc
index_route: /author/alexander
entry: false
entry_route: /{slug}
more: true
filter: return ($entry->data['author'] === 'Alexander Wendt');

Similarly, alte-weise.yml, which filters for categories:

title: Publico - Alte & Weise
sort_field: date
sort_direction: desc
index_route: /alte-weise
entry: false
entry_route: /{slug}
more: true
filter: return (array_search('alte-weise',$entry->data['categories']) !== false);

Except mag.yml, all other yaml files set rss: false.

9. Templates

This theme uses the following PHP template files:

1. bottom-layout.php: commonalities for the bottom part
2. entry.php: template for the entry, i.e., the usual blog post
3. error.php: 404 page, or other error conditions
4. head.php: HTML for the first few lines for all HTML files
5. index.php: template for the index, i.e., the listing of posts
6. overview.php: HTML sitemap
7. rss.php: RSS feed
8. sitemap.php: XML sitemap
9. top-layout.php: commonalities for the top part

I use the following hierarchy of PHP files for my entry-template, i.e., the template for a blog post:

The following hierarchy is used for the index-template, i.e., the template for showing a reverse-date sorted list of blog posts:

In C this is just gmtime(). gmtime accepts time_t and produces struct tm:

struct tm *gmtime(const time_t *timep);

On mainframe, however, it is sometimes a little inconvienent to call a C routine from COBOL. It is easier to just code the short algorithm in COBOL.

2. Approach. P.J. Plauger's book "The Standard C Library" contains the source code for gmtime() and localtime(). This code is then translated to COBOL.

The C code is as below.

/* Convert UNIX timestamp to triple (year,month,day)
   Elmar Klausmeier, 01-Apr-2024
*/

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <limits.h>

// From P.J.Plauger: The Standard C Library, Prentice Hall, 1992

static const int daytab[2][12] = {
    { 0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335 },	// leap year
    { 0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334 }
};

int daysTo (int year, int mon) {	// compute extra days to start of month
    int days;

    if (year > 0) days = (year - 1) / 4;	// correct for leap year: 1801-2099
    else if (year <= -4) days = 1 + (4 - year) / 4;
    else days = 0;

    return days + daytab[year&03 || (year==0)][mon];
}


struct tm *timeTotm (struct tm *t, time_t secsarg, int isdst) {	// convert scalar time to time struct
    int year, mon;
    const int *pm;
    long i, days;
    time_t secs;
    static struct tm ts;

    secsarg += ((70 * 365LU) + 17) * 86400;	// 70 years including 17 leap days since 1900
    if (t == NULL) t = &ts;
    t->tm_isdst = isdst;

    for (secs=secsarg; ; secs=secsarg+3600) {	// loop to correct for DST (not used here)
        days = secs / 86400;
        t->tm_wday = (days + 1) % 7;
        for (year = days / 365; days < (i=daysTo(year,0)+365L*year); --year)
            ;	// correct guess and recheck
        days -= i;
        t->tm_year = year;
        t->tm_yday = days;

        pm = daytab[year&03 || (year==0)];
        for (mon=12; days<pm[--mon]; )
            ;
        t->tm_mon = mon;
        t->tm_mday = days - pm[mon] + 1;

        secs %= 86400;
        t->tm_hour = secs / 3600;
        secs %= 3600;
        t->tm_min = secs / 60;
        t->tm_sec = secs % 60;

        //if (t->tm_isdst >= 0  ||  (t->tm_isdst = IsDST(t)) <= 0) return t;
        return t;
    }
}


int main (int argc, char *argv[]) {
    struct tm t;
    long secs;

    if (argc <= 1) return 0;
    secs = atol(argv[1]);

    timeTotm(&t, secs, 0);
    printf("timeTotm(): year=%d, mon=%d, day=%d, hr=%d, min=%d, sec=%d\n",
        1900+t.tm_year, 1+t.tm_mon, t.tm_mday, t.tm_hour, t.tm_min, t.tm_sec);

    return 0;
}

3. COBOL solution. Below is the COBOL code which was translated from above C code.

Fun fact: GNU Cobol crashed on some intermediate result, see cobc crashes on illegal COBOL source code file. This bug was fixed within a few hours by Simon Sobisch!

Below source code is compiled without problems.

000010 IDENTIFICATION DIVISION.
000020 PROGRAM-ID.   Timestamp2date.
000030 AUTHOR.       Elmar Klausmeier.
000040 DATE-WRITTEN. 02-May-2024.
000050
000060 DATA DIVISION.
000070 WORKING-STORAGE SECTION.
000080*
000090 01 year    PIC S9(18) comp-5.
000100 01 mon     PIC S9(18) comp-5.
000110 01 days    PIC S9(18) comp-5.
000120*
000130* Local helper variables
000140 01 i       PIC S9(18) comp-5.
000150 01 idays   PIC S9(18) comp-5.
000160 01 daysTo  PIC S9(18) comp-5.
000170 01 yearMod4  PIC S9(9) comp-5.
000180 01 leapIx  PIC S9(9) comp-5.
000190 01 daysP1  PIC S9(18) comp-5.
000200*
000210 01 secs    PIC S9(18) comp-5.
000220 01 secsarg PIC S9(18) comp-5.
000230*
000240*
000250* struct tm:
000260*    int tm_sec;    // Seconds          [0, 60]
000270*    int tm_min;    // Minutes          [0, 59]
000280*    int tm_hour;   // Hour             [0, 23]
000290*    int tm_mday;   // Day of the month [1, 31]
000300*    int tm_mon;    // Month            [0, 11]  (January = 0)
000310*    int tm_year;   // Year minus 1900
000320*    int tm_wday;   // Day of the week  [0, 6]   (Sunday = 0)
000330*    int tm_yday;   // Day of the year  [0, 365] (Jan/01 = 0)
000340*    int tm_isdst;  // Daylight savings flag
000350 01 tm_sec  PIC S9(9).
000360 01 tm_min  PIC S9(9).
000370 01 tm_hour PIC S9(9).
000380 01 tm_mday PIC S9(9).
000390*   range: 1-12
000400 01 tm_mon  PIC S9(9).
000410 01 tm_year PIC S9(9).
000420 01 tm_wday PIC S9(9).
000430 01 tm_yday PIC S9(9).
000440*
000450*
000460 01 daytabInit.
000470*   Number of days for leap year
000480    05 daytab-1-1 pic s9(9) comp-5 value 0.
000490    05 daytab-1-2 pic s9(9) comp-5 value 31.
000500    05 daytab-1-3 pic s9(9) comp-5 value 60.
000510    05 daytab-1-4 pic s9(9) comp-5 value 91.
000520    05 daytab-1-5 pic s9(9) comp-5 value 121.
000530    05 daytab-1-6 pic s9(9) comp-5 value 152.
000540    05 daytab-1-7 pic s9(9) comp-5 value 182.
000550    05 daytab-1-8 pic s9(9) comp-5 value 213.
000560    05 daytab-1-9 pic s9(9) comp-5 value 244.
000570    05 daytab-1-10 pic s9(9) comp-5 value 274.
000580    05 daytab-1-11 pic s9(9) comp-5 value 305.
000590    05 daytab-1-12 pic s9(9) comp-5 value 335.
000600*   Number of days for non-leap year
000610    05 daytab-2-1 pic s9(9) comp-5 value 0.
000620    05 daytab-2-2 pic s9(9) comp-5 value 31.
000630    05 daytab-2-3 pic s9(9) comp-5 value 59.
000640    05 daytab-2-4 pic s9(9) comp-5 value 90.
000650    05 daytab-2-5 pic s9(9) comp-5 value 120.
000660    05 daytab-2-6 pic s9(9) comp-5 value 151.
000670    05 daytab-2-7 pic s9(9) comp-5 value 181.
000680    05 daytab-2-8 pic s9(9) comp-5 value 212.
000690    05 daytab-2-9 pic s9(9) comp-5 value 243.
000700    05 daytab-2-10 pic s9(9) comp-5 value 273.
000710    05 daytab-2-11 pic s9(9) comp-5 value 304.
000720    05 daytab-2-12 pic s9(9) comp-5 value 334.
000730 01 daytabArr redefines daytabInit.
000740    05 filler     occurs 2 times.
000750       10 filler     occurs 12 times.
000760          15 daytab  pic s9(9) comp-5.
000770*
000780*
000790
000800 PROCEDURE DIVISION.
000810******************************************************************
000820* A100-main
000830******************************************************************
000840* Function:
000850*
000860******************************************************************
000870 A100-main SECTION.
000880 A100-main-P.
000890
000900*    initialize daytabArr.
000910*    move daytabInit to daytabArr
000920*    perform varying leapIx from 1 by 1 until leapIx > 2
000930*        perform varying mon from 1 by 1 until mon > 12
000940*            display 'daytab(' leapIx ',' mon ') = '
000950*                daytab(leapIx, mon)
000960*        end-perform
000970*    end-perform.
000980
000990     ACCEPT secsarg FROM ARGUMENT-VALUE
001000     perform v910-timeToTm
001010     display '        tm_sec  = ' tm_sec
001020     display '        tm_min  = ' tm_min
001030     display '        tm_hour = ' tm_hour
001040     display '        tm_mday = ' tm_mday
001050     display '        tm_mon  = ' tm_mon
001060     display '        tm_year = ' tm_year
001070     display '        tm_wday = ' tm_wday
001080     display '        tm_yday = ' tm_yday
001090
001100     STOP RUN.
001110
001120
001130* Convert UNIX timestamp to triple (year,month,day)
001140* Converted from C program
001150* From P.J.Plauger: The Standard C Library, Prentice Hall, 1992
001160
001170******************************************************************
001180* V900-daysTo
001190******************************************************************
001200* Function: compute daysTo given year and mon
001210*	          compute extra days to start of month
001220******************************************************************
001230 V900-daysTo SECTION.
001240 V900-daysTo-P.
001250
001260* correct for leap year: 1801-2099
001270     evaluate true
001280         when year > 0
001290             compute idays = (year - 1) / 4
001300         when year <= -4
001310             compute idays = 1 + (4 - year) / 4
001320         when other
001330             move zero to idays
001340     end-evaluate
001350
001360     compute yearMod4 = function mod(year,4)
001370     if yearMod4 not= zero or year = zero then
001380         move 2 to leapIx
001390     else
001400         move 1 to leapIx
001410     end-if
001420     compute daysTo = idays + daytab(leapIx, mon)
001430
001440     CONTINUE.
001450 END-V900-daysTo.
001460     EXIT.
001470
001480
001490******************************************************************
001500* V910-timeToTm
001510******************************************************************
001520* Function: compute tmT from secsarg (seconds since 01-Jan-1970)
001530*	          convert scalar time to time struct
001540******************************************************************
001550 V910-timeToTm SECTION.
001560 V910-timeToTm-P.
001570
001580* 70 years including 17 leap days since 1900
001590     compute secsarg = secsarg + ((70 * 365) + 17) * 86400;
001600     move secsarg to secs
001610
001620     compute days = secs / 86400
001630     add 1 to days giving daysP1
001640     compute tm_wday = function mod(daysP1, 7)
001650
001660     compute year = days / 365
001670     move 1 to mon
001680     perform until 1 = 0
001690         perform v900-daysTo
001700         compute i = daysTo + 365 * year
001710         if days >= i then
001720*            exit perform
001730             go to v910-endloop
001740         end-if
001750*        correct guess and recheck
001760         subtract 1 from year
001770     end-perform.
001780 v910-endloop.
001790
001800     subtract i from days
001810     move year to tm_year
001820     move days to tm_yday
001830
001840     compute yearMod4 = function mod(year,4)
001850     if yearMod4 not= zero or year = zero then
001860         move 2 to leapIx
001870     else
001880         move 1 to leapIx
001890     end-if
001900     move 12 to mon
001910     perform until days >= daytab(leapIx, mon)
001920         subtract 1 from mon
001930     end-perform
001940     move mon to tm_mon
001950     compute tm_mday = days - daytab(leapIx, mon) + 1
001960
001970     compute secs = function mod(secs,86400)
001980     compute tm_hour = secs / 3600;
001990     compute secs = function mod(secs,3600)
002000     compute tm_min = secs / 60;
002010     compute tm_sec = function mod(secs, 60)
002020
002030     CONTINUE.
002040 END-V910-timeToTm.
002050     EXIT.
002060
002070

4. Example output. Here are two examples. First example is Fri May 03 2024 14:16:01 GMT+0000. See https://www.unixtimestamp.com.

$ ./cobts2date 1714745761
        tm_sec  = +000000001
        tm_min  = +000000016
        tm_hour = +000000014
        tm_mday = +000000003
        tm_mon  = +000000005
        tm_year = +000000124
        tm_wday = +000000005
        tm_yday = +000000123

Second example is Thu Dec 31 1964 22:59:59 GMT+0000.

$ ./cobts2date -157770001
        tm_sec  = +000000059
        tm_min  = +000000059
        tm_hour = +000000022
        tm_mday = +000000031
        tm_mon  = +000000012
        tm_year = +000000064
        tm_wday = +000000004
        tm_yday = +000000365

For a list of leap years see Schaltjahr.

1. HTTP/1 and HTTP/1.1,
2. HTTP/2,
3. HTTP/3 ("QUIC"),
4. FastCGI, therefore PHP-FPM,
5. Reverse proxy,
6. Builtin mruby, though, that crashes.

In benchmarks it ranks at the top constantly. See Web Framework Benchmarks.

It works way faster than NGINX or Apache. It shines for static web content.

2. Building. The already existing AUR packages for H2O do not work. I.e., they generate a binary which crashes. Below PKGBUILD produces a H2O binary.

pkgname=h2o-master-git
pkgver=1.0
pkgrel=1
arch=('i686' 'x86_64')
pkgdesc="H2O: the optimized HTTP/1.x, HTTP/2, HTTP/3 server"
provides=(h2o)
url="https://h2o.examp1e.net"
source=("git+https://github.com/h2o/h2o.git?commit=master?signed/" h2o.service)
sha256sums=('SKIP' 734e9d045dd5568665762d48e4077208c3da8c68f87510aaa9559d495dd680fd)


build() {
    cd "$srcdir"/h2o
    cmake -DCMAKE_INSTALL_PREFIX=/usr .
    make
}

package() {
    cd "$srcdir"/h2o
    install -Dm 644 LICENSE "$pkgdir"/usr/share/licenses/$pkgname/LICENSE
    install -Dm 644 README.md "$pkgdir"/usr/share/doc/h2o/README.md
    install -Dm 644 "$srcdir"/h2o.service "$pkgdir"/usr/lib/systemd/system/h2o.service
    install -Dm 644 examples/h2o/h2o.conf "$pkgdir/etc/h2o.conf"
    make DESTDIR="$pkgdir" install
}

Compiling on AMD Ryzen 7 5700G, max clock 4.673 GHz, 64 GB RAM, finishes in less than two minutes.

$ time makepkg -f
...
==> Tidying install...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Copying source files needed for debug symbols...
  -> Compressing man and info pages...
==> Checking for packaging issues...
==> Creating package "h2o-master-git"...
  -> Generating .PKGINFO file...
  -> Generating .BUILDINFO file...
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: h2o-master-git 1.0-1 (Fri 12 Apr 2024 09:48:36 PM CEST)
        real 92.42s
        user 447.76s
        sys 0
        swapped 0
        total space 0

3. Configuration. Below is a working configuration in file /etc/h2o.conf. The configuration accomplishes the following:

1. it serves http and https,
2. it compresses via gzip and brotli,
3. it is started user root, then switches to user http,
4. Log format is similar to the Hiawatha log-format,
5. PHP files are handled by php-fpm.

The entire configuration file is a YAML file.

listen: 80
listen: &ssl_listen
  port: 443
  ssl:
    certificate-file:    /etc/letsencrypt/live/eklausmeier.goip.de/fullchain.pem
    key-file:  /etc/letsencrypt/live/eklausmeier.goip.de/privkey.pem
    minimum-version: TLSv1.2
    cipher-preference: server
    cipher-suite: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
    # Oldest compatible clients: Firefox 27, Chrome 30, IE 11 on Windows 7, Edge, Opera 17, Safari 9, Android 5.0, and Java 8
    # see: https://wiki.mozilla.org/Security/Server_Side_TLS

# The following three lines enable HTTP/3
listen:
  <<: *ssl_listen
  type: quic
header.set: "Alt-Svc: h3-25=\":443\""

user: http
#pid-file: /var/run/h2o/h2o.pid
#crash-handler: /usr/local/bin/h2obacktrace
access-log:
  path: /var/log/h2o/access.log
  format: "%h|%{%Y/%m/%d:%T %z}t|%s|%b|%r|%{referer}i|%{user-agent}i|%V:%p|"
error-log: /var/log/h2o/error.log
compress: [ br, gzip ]
#file.dirlisting: ON

file.custom-handler:
  extension: .php
  fastcgi.connect:
    port: /run/php-fpm/php-fpm.sock
    type: unix

hosts:
  0:
    paths:
      /jpilot/favicon.ico:
        file.file: /home/klm/php/saaze-jpilot/public/favicon.ico
      /jpilot/img:
        file.dir: /home/klm/php/saaze-jpilot/public/img
      /jpilot/jpilot.css:
        file.file: /home/klm/php/saaze-jpilot/public/jpilot.css
      /koehntopp/assets:
        file.dir: /home/klm/php/saaze-koehntopp/public/assets
      /koehntopp/jscss:
        file.dir: /home/klm/php/saaze-koehntopp/public/jscss
      /lemire/jscss:
        file.dir: /home/klm/php/saaze-lemire/public/jscss
      /mobility/img:
        file.dir: /home/klm/php/saaze-mobility/public/img
      /nukeklaus/img:
        file.dir: /home/klm/php/saaze-nukeklaus/public/img
      /nukeklaus/jscss:
        file.dir: /home/klm/php/saaze-nukeklaus/public/jscss
      /panorama/img:
        file.dir: /home/klm/php/saaze-panorama/public/img
      /paternoster/paternoster.css:
        file.file: /home/klm/php/saaze-paternoster/public/paternoster.css
      /saaze-example/blogklm.css:
        file.file: /home/klm/php/saaze-example/public/blogklm.css
      /vonhoff/img:
        file.dir: /home/klm/php/saaze-vonhoff/public/img
      /wendt/pagefind:
        file.dir: /home/klm/php/saaze-wendt/public/pagefind
      /:
        file.dir: /srv/http
        redirect:
          status: 301
          internal: YES
          url: /index.php?
      /p:
        mruby.handler: |
          Proc.new do |env|
            [200, {'content-type' => 'text/plain'}, ["Hello world"]]
          end

As already mentioned at the top: mruby doesn't work. Once you access /p the entire web-server crashes.

H2O does not offer URL rewriting out of the box. The above path-configurations operate on a prefix match schema. I.e., if the URL in question starts with the string provided, this is considered a match. The string after the match is appended to the part in file.dir.

4. Discussion. While alternatives to Apache and NGINX are highly welcome, the current state of H2O leaves many questions unanswered.

1. The builtin brotli compression is "stone old": it is seven years behind the official Google Brotli repository, which contains a number of serious fixes.
2. The builtin mruby software is two years behind, offering mruby version 3.1 instead of 3.3.
3. mruby crashes once called.
4. In the hosts part the hostname seems to have no effect.

I tried to replace the old mruby dependency with the current 3.3 version. The build of H2O then failed.

While embodying software packages directly into the H2O GitHub repo makes building the software easier, it risks that the included software rots. That's exactly what is happening here.

Fun fact: I noticed H2O when reading about the LWAN web-server written by L. Pereira. Both, Kazuho Oku and L. Pereira, work at Fastly.

Also see H2O Tutorial.

In case someone wants to analyze why mruby crashes, here is the result of where in gdb:

Core was generated by `h2o -c h2o.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000062085dc9ae9b in mrb_str_hash (mrb=<optimized out>, str=...) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/string.c:1673
1673        hval ^= (uint32_t)*bp++;
[Current thread is 1 (Thread 0x7002156006c0 (LWP 18088))]
(gdb) where
#0  0x000062085dc9ae9b in mrb_str_hash (mrb=<optimized out>, str=...) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/string.c:1673
#1  0x000062085dc8cb6c in obj_hash_code (h=0x7001d0028660, key=..., mrb=0x1a0) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/hash.c:325
#2  ib_it_init (mrb=mrb@entry=0x7001d00015a0, it=it@entry=0x7002155fe550, h=h@entry=0x7001d0028660, key=...) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/hash.c:645
#3  0x000062085dc8cd3a in ib_init (ib_byte_size=<optimized out>, ib_bit=<optimized out>, h=0x7001d0028660, mrb=<optimized out>) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/hash.c:151
#4  ht_init (mrb=mrb@entry=0x7001d00015a0, h=h@entry=0x7001d0028660, size=size@entry=17, ea=0x7001d0047700, ea_capa=ea_capa@entry=25, ht=ht@entry=0x0, ib_bit=<optimized out>) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/hash.c:793
#5  0x000062085dc8d11a in ar_set (mrb=0x7001d00015a0, h=0x7001d0028660, key=..., val=...) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/hash.c:536
#6  0x000062085dc8c2e6 in h_set (val=..., key=..., h=0x7001d0028660, mrb=0x7001d00015a0) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/hash.c:169
#7  mrb_hash_set (mrb=0x7001d00015a0, hash=..., key=..., val=...) at /usr/src/debug/h2o-master-git/h2o/deps/mruby/src/hash.c:1245
#8  0x000062085dc67938 in iterate_headers_callback (shared_ctx=shared_ctx@entry=0x7001d0001540, pool=pool@entry=0x7001d0076958, header=header@entry=0x7002155fe8d0, cb_data=cb_data@entry=0x7001d0028660) at /usr/src/debug/h2o-master-git/h2o/lib/handler/mruby.c:748
#9  0x000062085dc67e4c in h2o_mruby_iterate_native_headers (shared_ctx=shared_ctx@entry=0x7001d0001540, pool=<optimized out>, headers=<optimized out>, cb=cb@entry=0x62085dc678a0 <iterate_headers_callback>, cb_data=cb_data@entry=0x7001d0028660)
    at /usr/src/debug/h2o-master-git/h2o/lib/handler/mruby.c:727
#10 0x000062085dc6a76e in build_env (generator=0x7001d006cbe0) at /usr/src/debug/h2o-master-git/h2o/lib/handler/mruby.c:836
#11 on_req (_handler=<optimized out>, req=<optimized out>) at /usr/src/debug/h2o-master-git/h2o/lib/handler/mruby.c:974
#12 0x000062085dbc603a in call_handlers (req=0x7001d00765d8, handler=0x62085f2d5ef0) at /usr/src/debug/h2o-master-git/h2o/lib/core/request.c:165
#13 0x000062085dbeeb89 in handle_incoming_request (conn=<optimized out>) at /usr/src/debug/h2o-master-git/h2o/lib/http1.c:714
#14 0x000062085dba6293 in run_socket (sock=0x7001d009b660) at /usr/src/debug/h2o-master-git/h2o/lib/common/socket/evloop.c.h:834
#15 run_pending (loop=loop@entry=0x7001d0000b70) at /usr/src/debug/h2o-master-git/h2o/lib/common/socket/evloop.c.h:876
#16 0x000062085dba6300 in h2o_evloop_run (loop=0x7001d0000b70, max_wait=<optimized out>) at /usr/src/debug/h2o-master-git/h2o/lib/common/socket/evloop.c.h:925
#17 0x000062085dc5da1b in run_loop (_thread_index=<optimized out>) at /usr/src/debug/h2o-master-git/h2o/src/main.c:4210
#18 0x000070022b8a955a in ?? () from /usr/lib/libc.so.6
#19 0x000070022b926a3c in ?? () from /usr/lib/libc.so.6

$ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-u: processes                       254204
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  8192
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 254204
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15: rt cpu time (microseconds)   unlimited

The line for the "core file size" must be greater than zero.

In Arch Linux that alone doesn't help. core files are written to this directory:

$ coredumpctl info
          PID: 16354 (h2o)
           UID: 33 (http)
           GID: 33 (http)
        Signal: 11 (SEGV)
     Timestamp: Wed 2024-04-10 20:02:12 CEST (2h 3min ago)
  Command Line: h2o
    Executable: /usr/bin/h2o
 Control Group: /user.slice/user-1000.slice/user@1000.service/tmux-spawn-3fc3de1b-6e2d-43bf-ad3d-bf55b4ce3a1a.scope
          Unit: user@1000.service
     User Unit: tmux-spawn-3fc3de1b-6e2d-43bf-ad3d-bf55b4ce3a1a.scope
         Slice: user-1000.slice
     Owner UID: 1000 (klm)
       Boot ID: 8b9d5dcffc3a4669b0c7fa244db334be
    Machine ID: 814e9c58b1e34999a682767020267eb0
      Hostname: chieftec
       Storage: /var/lib/systemd/coredump/core.h2o.33.8b9d5dcffc3a4669b0c7fa244db334be.16354.1712772132000000.zst (inaccessible)
       Message: Process 16354 (h2o) of user 33 dumped core.

                Stack trace of thread 16363:
                #0  0x0000777802fe7bb3 n/a (libcrypto.so.53 + 0xd0bb3)
                #1  0x00007778030efd5b SSL_CTX_flush_sessions (libssl.so.56 + 0x24d5b)
                #2  0x00005d994cc02023 cache_cleanup_thread (h2o + 0x12a023)
                #3  0x0000777802c7755a n/a (libc.so.6 + 0x8b55a)
                #4  0x0000777802cf4a3c n/a (libc.so.6 + 0x108a3c)

The command coredumpctl list enlists the core's so far:

$ coredumpctl list
TIME                           PID UID GID SIG     COREFILE     EXE          SIZE
Sat 2024-04-06 17:55:20 CEST 24746  33  33 SIGSEGV inaccessible /usr/bin/h2o    -
Sat 2024-04-06 18:49:20 CEST 26982  33  33 SIGSEGV inaccessible /usr/bin/h2o    -
Sat 2024-04-06 18:50:04 CEST 27178  33  33 SIGSEGV inaccessible /usr/bin/h2o    -

You can start debugging with coredumpctl debug. That will call gdb.

The location and name of the core file can be changed by tampering with

$ cat /proc/sys/kernel/core_pattern
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

More information is here: Core dump file is not generated, coredumpctl, systemd-coredump.

From the CSS Naked Day:

The idea behind CSS Naked Day is to promote web standards. Plain and simple. This includes proper use of HTML, semantic markup, a good hierarchy structure, and of course, a good old play on words. In the words of 2006, it’s time to show off your <body> for what it really is.

The importance of CSS is illustrated by this humorous tweet:

1. The 50 hour window

The logic to enable or disable CSS is given by below PHP routine on CSS Naked Day:

<?php
function is_naked_day($d) {
    $start = date('U', mktime(-14, 0, 0, 04, $d, date('Y')));
    $end = date('U', mktime(36, 0, 0, 04, $d, date('Y')));
    $z = date('Z') * -1;
    $now = time() + $z;
    if ( $now >= $start && $now <= $end ) {
        return true;
    }
    return false;
}
?>

Running this with php -a and unixtimestamp.com for the year 2024 gives the following interval:

1. Start: 08-Apr-2024 12:00 CET
2. End: 10-Apr-2024 14:00 CET

The rationale is:

CSS Naked Day lasts for one international day. Technically speaking, it will be April 9 somewhere in the world for 50 hours. This is to ensure that everyone’s website will be publicly nude for the entire world to see at any given time during April 9.

2. Required changes in templates

For this blog I use the static site generator Simplified Saaze. All templates of this generator are written in PHP. So deactivating CSS is a pretty simple if statement.

I use the following hierarchy of PHP files for my entry-template, i.e., the template for a blog post:

The following hierarchy is used for the index-template, i.e., the template for showing a reverse-date sorted list of blog posts:

3. Changes in <head> section

File head.php does not contain any CSS. File top-layout.php handles the majority of the HTML <head> section, and the beginning of the <body> section.

I use prism.js for syntax highlighting. This in turn uses CSS, which is surrounded by a simple if:

<?php $NO_CSS = getenv('NO_CSS') ? true : false; ?>
<?php if (isset($entry['prismjs']) && ! $NO_CSS) { ?>
    <link href=/jscss/prism.css rel=stylesheet>
<?php } ?>

If I generate all the static HTML files, I use the environment variable NO_CSS. In case of dynamic generation I simply set $NO_CSS explicitly in top-layout.php, i.e., $NO_CSS=true;.

I have a separate CSS file, called blogklm.css, which I also surround with an if:

<?php if (! $NO_CSS) echo "<style>\n" ?>
<?php if (! $NO_CSS) require SAAZE_PATH . "/public/jscss/blogklm.css" ?>
<?php if (! $NO_CSS) echo "</style>\n" ?>

For galleries and Markmap I had a conditional anyway. This needed an additional clause:

<?php if (!isset($pagination) && ! $NO_CSS) {
    if (isset($entry['gallery_css'])) echo $entry['gallery_css'];
    if (isset($entry['markmap_css'])) echo $entry['markmap_css'];
} ?>

I use Pagefind for searching within this blog. Pagefind in turn needs CSS, which is surrounded by an if:

<?php if (! $NO_CSS) { ?>
<link href="/pagefind/pagefind-ui.css" rel="stylesheet">
<script src="/pagefind/pagefind-ui.js"></script>
<script>
    window.addEventListener('DOMContentLoaded', (event) => {
        new PagefindUI({ element: "#search", showSubResults: true });
    });
</script>
<?php } ?>

4. Changes in <body> section

Still in top-layout.php. Finally, I explicitly mention that I stripped all CSS, so visitors are not surprised to find a new layout:

<?php if ($NO_CSS) echo "<h2> &nbsp; &nbsp; &nbsp; &nbsp; <a href=\"https://css-naked-day.github.io\">April 9 is CSS Naked Day!</a></h2>\n"; ?>

5. History and evolution of CSS Naked Day

Below text is copied from CSS Naked Day and the Missing Wikipedia Page:

The event dates back to 2006, when Dustin Diaz, an American web developer, advertised the first CSS Naked Day in order “to promote web standards.”

During the first two years (2006 and 2007), CSS Naked Day was held on April 5, when in 2008, the date was changed to April 9.

Until 2009, the event was organized by Diaz. From 2010 to 2014, Taylor Satula, an American web designer, ...

From the first CSS Naked Day in 2006, which had 763 recorded participants, engagement went up to 2,160 participants in 2008. After another strong participation in 2009 (1,266 recorded participants), fewer people and sites are documented to have taken part.

In recent years (2020–2023), only a fraction of these participants is known, usually including a few dozen individuals and their sites. While there are no reliable ways to measure participation, it seems clear that while CSS Naked Day is still being observed, that is only the case for a small minority of people in the field. ...

In the months following the 2015 edition, and until today, Basmaison and Meiert have kept maintaining the site and promoting the event together.

The usual omnipresent Wikipedia trolls and naysayer blocked this wiki entry.

gcc -fomit-frame-pointer

Unfortunately the ELF itself does not contain a flag, which tells you that. But looking at the assembler code can give you the answer.

First disassemble the code with

objdump -d

Check the disassembly for below pairs directly after any C function:

push   %rbp
mov    %rsp,%rbp

These are the instructions to set up the frame pointer on 64 bit Linux x86 systems.

Example:

0000000000001380 <zif_md4c_toHtml>:
    1380:       55                      push   %rbp
    1381:       48 89 e5                mov    %rsp,%rbp

A good heuristic is then

objdump -d $binary | grep -c "mov.*%rsp,.*%rbp"

Double check with

objdump -d $binary | grep -C1 "mov.*%rsp,.*%rbp"

This heuristic is not fool proof, as individual C routines can be augmented with

__attribute__((optimize("omit-frame-pointer"))

In the intense debate about making -fno-omit-frame-pointer the default in Fedora, see this comment from L. A. F. Pereira in Python 3.11 performance with frame pointers.

See How can I tell whether a binary is compiled with frame pointers or not on Linux?, which discusses the case for 32 bit x86 Linux systems.

Code with framepointers will always contain the both of the two instructions push %ebp and mov %esp, %ebp. ... For those working with x86_64, the registers to look for are the 64-bit equivalents: %rbp and %rsp - the concept is the same though!

The post The Return of the Frame Pointers by Brendan Gregg triggered this task.

As of today, 18-Mar-2024, Arch Linux still does not ship binaries with frame pointer support. For example:

$ objdump -d /bin/zsh | grep -c "mov.*%rsp,.*%rbp"
10

The PHP binary fails the heuristic:

$ objdump -d /bin/php | grep -c "mov.*%rsp,.*%rbp"
173

But looking at the actuall disassembly shows something like this:

000000000021aff2 <php_info_print_box_end@@Base>:
  21aff2:       f3 0f 1e fa             endbr64
  21aff6:       48 8d 05 43 9b 1e 01    lea    0x11e9b43(%rip),%rax        # 1404b40 <sapi_module@@Base>

I.e., no frame pointer handling.

$ lastb
a        ssh:notty    209.97.163.130   Tue Mar  5 13:07 - 13:07  (00:00)
sftpuser ssh:notty    93.123.39.2      Tue Mar  5 13:05 - 13:05  (00:00)
sftpuser ssh:notty    93.123.39.2      Tue Mar  5 13:05 - 13:05  (00:00)
hzp      ssh:notty    43.156.241.167   Mon Mar  4 18:19 - 18:19  (00:00)
hzp      ssh:notty    43.156.241.167   Mon Mar  4 18:19 - 18:19  (00:00)
root     ssh:notty    8.219.249.208    Mon Mar  4 18:17 - 18:17  (00:00)
mheydary ssh:notty    118.178.132.93   Mon Mar  4 12:35 - 12:35  (00:00)
mheydary ssh:notty    118.178.132.93   Mon Mar  4 12:34 - 12:34  (00:00)
ftp1user ssh:notty    143.255.140.241  Mon Mar  4 12:34 - 12:34  (00:00)
ftp1user ssh:notty    143.255.140.241  Mon Mar  4 12:34 - 12:34  (00:00)
panisa   ssh:notty    139.224.200.60   Mon Mar  4 11:13 - 11:13  (00:00)
panisa   ssh:notty    139.224.200.60   Mon Mar  4 11:13 - 11:13  (00:00)
sina     ssh:notty    129.226.158.202  Mon Mar  4 10:45 - 10:45  (00:00)
sina     ssh:notty    129.226.158.202  Mon Mar  4 10:44 - 10:44  (00:00)
hadoop   ssh:notty    129.226.152.121  Mon Mar  4 10:43 - 10:43  (00:00)

In 2020 I used fail2ban. Since 2021 I use SSHGuard. It uses way less resources. See Analysis And Usage of SSHGuard.

I ran a quick analysis which country is the most aggressive penetrator.

1. Collecting IP addresses. SSHGuard filters the offending intruder via ipset.

$ ipset list > i1

This collects all IP addresses.

Now I run these IP numbers through geoiplookup:

$ for i in `perl -ne 'print $1."\n" if /^(\d+\.\d+\.\d+\.\d+)\s+/' i1`; do geoiplookup $i >> i3; done

The resulting list looks like this:

$ head i3
GeoIP Country Edition: CN, China
GeoIP Country Edition: HK, Hong Kong
GeoIP Country Edition: US, United States
GeoIP Country Edition: US, United States
GeoIP Country Edition: KR, Korea, Republic of
GeoIP Country Edition: PE, Peru
GeoIP Country Edition: CA, Canada
GeoIP Country Edition: CN, China
GeoIP Country Edition: KR, Korea, Republic of
GeoIP Country Edition: KE, Kenya

2. Sorting according frequency.

cut -d: -f2 i3 | sort | uniq -c | sort -rn

The top 20 offenders are:

   4228  CN, China
   3175  US, United States
   2142  SG, Singapore
   1596  KR, Korea, Republic of
   1042  DE, Germany
    980  IN, India
    755  HK, Hong Kong
    661  BR, Brazil
    566  RU, Russian Federation
    522  VN, Vietnam
    471  ID, Indonesia
    453  JP, Japan
    403  FR, France
    396  NL, Netherlands
    354  GB, United Kingdom
    313  IR, Iran, Islamic Republic of
    307  CA, Canada
    279  TW, Taiwan
    236  AU, Australia
    173  TH, Thailand

Graphically this looks like this:

1. Installing IBM COBOL for Linux on Arch Linux in 2021
2. Testing COBOLworx gcc-cobol #2 in 2023

Initially I succeeded in installing the IBM compiler in 2021. The IBM compiler compared very favorably against the GNU Cobol compiler, see Comparing GnuCOBOL to IBM COBOL. But in 2023 this installation procedure failed. So, no IBM COBOL on Arch Linux.

Richard Nelson from IBM contacted me today and mentioned that IBM COBOL should also run on Arch Linux. So I tried to install the latest version 1.2.0.2 again. Version 1.2 is particularly appealing as it supports 64 bit. IBM COBOL compilers were notorious with lacking 64 bit support, see Memory Limitations with IBM Enterprise COBOL Compiler.

My current Arch Linux setup is as given in below table.

1. Download. Software package is here: IBM COBOL for Linux on x86. IBM now uses this annoying two-factor authorization procedure, click through all these hoops. This 2FA makes it essentially impossible to write an AUR package, which downloads the IBM file within the PKGBUILD.

The file in question is IBM_COBOL_V1.2.0_LINUX_EVAL.x86-64.240110.tar.gz. Its size is 116 MB.

$ tar ztvf IBM_COBOL_V1.2.0_LINUX_EVAL.x86-64.240110.tar.gz
drwxr-sr-x root/root         0 2023-06-06 01:05 images/
drwxr-sr-x root/root         0 2024-01-10 16:16 images/rhel/
-rw-rw-r-- root/root  26210268 2024-01-10 16:16 images/rhel/cobol.rte.1.2.0-1.2.0.2-231215.x86_64.rpm
-rw-rw-r-- root/root   2331592 2024-01-10 16:16 images/rhel/cobol.dbg.1.2.0-1.2.0.2-231215.x86_64.rpm
-rw-rw-r-- root/root   3055224 2024-01-10 16:16 images/rhel/cobol.cmp.license-eval.1.2.0-1.2.0.2-231215.x86_64.rpm
-rw-rw-r-- root/root  11199076 2024-01-10 16:16 images/rhel/cobol.cmp.1.2.0-1.2.0.2-231215.x86_64.rpm
drwxr-sr-x root/root         0 2024-01-10 16:17 images/sles/
-rw-r--r-- root/root  22295780 2024-01-10 16:17 images/sles/cobol.rte.1.2.0-1.2.0.2-231215.x86_64.rpm
-rw-r--r-- root/root   1975984 2024-01-10 16:17 images/sles/cobol.dbg.1.2.0-1.2.0.2-231215.x86_64.rpm
-rw-r--r-- root/root   2999760 2024-01-10 16:17 images/sles/cobol.cmp.license-eval.1.2.0-1.2.0.2-231215.x86_64.rpm
-rw-r--r-- root/root   9095804 2024-01-10 16:17 images/sles/cobol.cmp.1.2.0-1.2.0.2-231215.x86_64.rpm
drwxr-sr-x root/root         0 2024-01-10 16:17 images/ubuntu/
-rw-r--r-- root/root   1957512 2024-01-10 16:17 images/ubuntu/cobol.dbg.1.2.0_1.2.0.2-231215_amd64.deb
-rw-r--r-- root/root   2992220 2024-01-10 16:17 images/ubuntu/cobol.cmp.license-eval.1.2.0_1.2.0.2-231215_amd64.deb
-rw-r--r-- root/root  10125300 2024-01-10 16:17 images/ubuntu/cobol.cmp.1.2.0_1.2.0.2-231215_amd64.deb
-rw-r--r-- root/root  22514248 2024-01-10 16:17 images/ubuntu/cobol.rte.1.2.0_1.2.0.2-231215_amd64.deb
-rwxr-xr-x root/root      6763 2024-01-10 16:32 install
-rw-r--r-- root/root    820691 2023-06-06 01:05 install.pdf
-rwxr-xr-x root/root   2694559 2023-06-06 01:12 LicenseAgreement.pdf
-rwxr-xr-x root/root    285651 2023-06-06 01:12 LicenseInformation.pdf
-rwxr-xr-x root/root     57001 2023-06-06 01:12 notices
-rw-r--r-- root/root    311858 2023-06-06 01:14 quickstart.fr_FR.pdf
-rw-r--r-- root/root    311477 2023-06-06 01:14 quickstart.ja_JP.pdf
-rw-r--r-- root/root    281309 2023-06-06 01:14 quickstart.pdf
-rwxr-xr-x root/root      2932 2023-06-06 01:12 README

2. Unpacking the Ubuntu part. We will extract the Ubuntu part, highlighted above.

$ tar zxf IBM_COBOL_V1.2.0_LINUX_EVAL.x86-64.240110.tar.gz images/ubuntu/

Change to images/ubuntu directory and run the below loop, which first unpacks the deb-files with ar, then unpacks the resulting tar.xz data file with tar Jx:

for i in *.deb; do ar xf $i; tar Jxf data.tar.xz; done

This creates a subdirectory opt with 188 entries.

Move the resulting opt or opt/ibm to the "real" /opt and chown -R root:root all the files.

Installation size is 135 MB.

3. Checking the installation. See, whether all libraries are in place.

$ ldd /opt/ibm/cobol/1.2.0/bin/cob2
        linux-vdso.so.1 (0x00007ffebab8a000)
        librt.so.1 => /usr/lib/librt.so.1 (0x000070366a43e000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x000070366a439000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x000070366a434000)
        libc.so.6 => /usr/lib/libc.so.6 (0x000070366a252000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x000070366a47a000)

$ ldd /opt/ibm/cobol/1.2.0/bin/cob3
        not a dynamic executable

$ ldd cob3_64
        linux-vdso.so.1 (0x00007ffeddbf9000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007a35119f5000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007a35119f0000)
        libicuuc_64r.so => /opt/ibm/cobol/1.2.0/usr/bin/./../../../rte/usr/lib/libicuuc_64r.so (0x00007a3510600000)
        libcob2_64r.so => /opt/ibm/cobol/1.2.0/usr/bin/./../../../rte/usr/lib/libcob2_64r.so (0x00007a3510000000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007a3511904000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007a3511720000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007a350fc00000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007a35116fb000)
        libicudata_64r.so => /opt/ibm/cobol/1.2.0/usr/bin/./../../../rte/usr/lib/libicudata_64r.so (0x00007a350da00000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007a35116f6000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007a3511a31000)
        libicui18n_64r.so => /opt/ibm/cobol/1.2.0/usr/bin/./../../../rte/usr/lib/libicui18n_64r.so (0x00007a350d200000)
        libdfp_64r.so => /opt/ibm/cobol/1.2.0/usr/bin/./../../../rte/usr/lib/libdfp_64r.so (0x00007a350ca00000)

For convenience add the bin-directory to the PATH:

$ export PATH=$PATH:/opt/ibm/cobol/1.2.0/bin

Up to this point, running the compiler would report a license problem. The actual compiler is cob2.

Here is an example, once the license is setup correctly:

$ cob2 hello1.cob
IBM COBOL for Linux 1.2.0 compile started
End of compilation 1,  program HELLO1,  no statements flagged.

4. Getting a 60 day trial license. Richard Nelson sent me a new file libxlcmpev_64r.so. With this new library file the compiler works flawlessly.

$ license_check
Evaluation (Trial/Eval/TnB) license
Current date    Sat, 02 Mar 2024 17:54:00 GMT
Activation date Thu, 29 Feb 2024 00:00:01 GMT
Expire date     Mon, 29 Apr 2024 23:59:59 GMT
Days left       58

Thanks Richard!

Also, Richard mentioned the install shell script in the original tar file, see line 18. I didn't make use of that! My fault. Once I knew that this libxlcmpev_64r.so is problematic, and looking at the install script:

...
extendTrial="$reldir/cobol/$version/usr/bin/xlcmp xlcbl && rm $reldir/cobol/$version/usr/bin/xlcmp"
eval $extendTrial
...

Generating the license now goes like this, as user root:

/opt/ibm/cobol/1.2.0/usr/bin/xlcmp xlcbl

This generates a new 1.2.0/usr/lib/libxlcmpev_64r.so. This provides a valid 60 day license.

$ license_check
Evaluation (Trial/Eval/TnB) license
Current date    Sat, 02 Mar 2024 18:14:24 GMT
Activation date Sat, 02 Mar 2024 00:00:01 GMT
Expire date     Wed, 01 May 2024 23:59:59 GMT
Days left       60

1. Runtimes in serial mode. In the following we use PHP with no JIT. So far runtimes for this very blog are as below:

$ time php saaze -mortb /tmp/build
Building static site in /tmp/build...
    execute(): filePath=./content/aux.yml, nSIentries=7, totalPages=1, entries_per_page=20
    execute(): filePath=./content/blog.yml, nSIentries=452, totalPages=23, entries_per_page=20
    execute(): filePath=./content/gallery.yml, nSIentries=7, totalPages=1, entries_per_page=20
    execute(): filePath=./content/music.yml, nSIentries=69, totalPages=4, entries_per_page=20
    execute(): filePath=./content/error.yml, nSIentries=0, totalPages=0, entries_per_page=20
Finished creating 5 collections, 4 with index, and 561 entries (0.25 secs / 24.46MB)
#collections=5, parseEntry=0.0103/563-5, md2html=0.0201, MathParser=0.0141/561, renderEntry=0.1573/561, renderCollection=0.0058/33, content=561/0, excerpt=0/0
    real 0.28s
    user 0.16s
    sys 0
    swapped 0
    total space 0

It can be seen that the renderEntry() function uses 0.1573 seconds from overall 0.25 seconds, i.e., more than 60%. These 561 calls will now be parallelized. The rest stays serial.

For the Lemire blog we have:

$ time php saaze -rb /tmp/buildLemire
Building static site in /tmp/buildLemire...
        execute(): filePath=/home/klm/php/saaze-lemire/content/blog.yml, nSIentries=2771, totalPages=139, entries_per_page=20
Finished creating 1 collections, 1 with index, and 4483 entries (1.01 secs / 97.18MB)
#collections=1, parseEntry=0.0702/4483-1, md2html=0.1003, MathParser=0.0594/4483, renderEntry=0.4121/4483, renderCollection=0.0225/140, content=4483/0, excerpt=0/0
        real 1.03s
        user 0.64s
        sys 0
        swapped 0
        total space 0

In this case the output template processing is 0.4121 seconds from overall 1.01 seconds, that's 40%. This shows that the Lemire templates are easier. No wonder, they do not use categories and tags, and many other gimmicks, which I used in this blog. But still, 40% of the runtime is spent on output rendering.

In Performance Comparison Saaze vs. Hugo vs. Zola I wrote:

It would be quite easy to use threads in Saaze, i.e., so-called entries and the chunks of collections could easily be processed in parallel.

It is even easier to parallelize the generation of the output files when the PHP templating is in place. We will see that parallelizing can be done in less than 20 lines of PHP code.

2. Runtimes in serial mode with JIT enabled. Below are the runtime with JIT and OPCache enabled for PHP.

time php saaze -mortb /tmp/build
Building static site in /tmp/build...
        execute(): filePath=./content/aux.yml, nSIentries=7, totalPages=1, entries_per_page=20
        execute(): filePath=./content/blog.yml, nSIentries=453, totalPages=23, entries_per_page=20
        execute(): filePath=./content/gallery.yml, nSIentries=7, totalPages=1, entries_per_page=20
        execute(): filePath=./content/music.yml, nSIentries=69, totalPages=4, entries_per_page=20
        execute(): filePath=./content/error.yml, nSIentries=0, totalPages=0, entries_per_page=20
Finished creating 5 collections, 4 with index, and 562 entries (0.16 secs / 20.36MB)
#collections=5, parseEntry=0.0104/564-5, md2html=0.0219, MathParser=0.0203/562, renderEntry=0.0521/562, renderCollection=0.0022/33, content=562/0, excerpt=0/0
        real 0.19s
        user 0.11s
        sys 0
        swapped 0
        total space 0

The previous massive renderEntry() part in runtime shrank from 0.1573 seconds to 0.0521 seconds. I think this is mainly due to the OPCache, which now avoids recompiling and reparsing the PHP output template.

For the Lemire blog with JIT enabled we have:

time php saaze -rb /tmp/buildLemire
Building static site in /tmp/buildLemire...
        execute(): filePath=/home/klm/php/saaze-lemire/content/blog.yml, nSIentries=2771, totalPages=139, entries_per_page=20
Finished creating 1 collections, 1 with index, and 4483 entries (0.62 secs / 96.24MB)
#collections=1, parseEntry=0.0655/4483-1, md2html=0.0974, MathParser=0.0586/4483, renderEntry=0.0707/4483, renderCollection=0.0110/140, content=4483/0, excerpt=0/0
        real 0.65s
        user 0.40s
        sys 0
        swapped 0
        total space 0

Similar picture to the above: the renderEntry() part dropped from 0.4121 seconds to 0.0707 seconds. That's massive.

3. Unix forks in PHP. As a preliminary introduction to pcntl_fork() in PHP, look at below simple PHP code.

<?php
    for ($i=1; $i<=4; ++$i) {
        if (($pid = pcntl_fork())) {
            printf("i=%d, pid=%d\n",$i,$pid);
            sleep(1);
            exit(0);
        }

Running this script:

$ php forktst.php
i=1, pid=15082
i=2, pid=15083
i=3, pid=15084
i=4, pid=15085

The fork and join method of parallelization is easy to use, but it has the disadvantage that communicating results from the children to the parent is "difficult". Communicating data from the parent to its children is "easy": everything is copied over.

4. Implementation in BuildCommand.php. The command-line version of Simplified Saaze calls buildAllStatic(). This routine iterates through all collections, and for each collection it iterates through all entries.

1. Function getEntries() reads Markdown files into memory and converts them to HTML by using MD4C, all in memory
2. Function buildEntry() uses the entry in question and writes the HTML to disk by processing it through our PHP templates.

PHP function buildEntry() is essentially:

private function buildEntry(Collection $collection, Entry $entry, string $dest) : void {
    ...
    file_put_contents($entryDir, $this->templateManager->renderEntry($entry);
}

buildEntry() is now encapsulated within beginParallel() and endParallel(). That's it.

foreach ($collections as $collection) {
    $entries    = $collection->getEntries();	# finally calls getContentAndExcerpt() and sorts
    $nentries   = count($entries);
    $nSIentries = count($collection->entriesSansIndex);
    $entries_per_page = $collection->data['entries_per_page'] ?? \Saaze\Config::$H['global_config_entries_per_page'];
    $totalPages = ceil($nSIentries / $entries_per_page);
    printf("\texecute(): filePath=%s, nSIentries=%d, totalPages=%d, entries_per_page=%d\n",$collection->filePath,$nSIentries,$totalPages,$entries_per_page);

    $this->beginParallel($nentries,$aprocs);
    $i = 0;
    foreach ($entries as $entry) {
        if ($this->nprocs > 0  &&  ($i++ % $this->nprocs) != $this->procnr) continue;	// distribute work among nprocs processes
        if ($entry->data['entry'] ?? true) {
            $this->buildEntry($collection, $entry, $dest);
            $entryCount++;
        }
    }
    $this->endParallel();

    if ($tags) {	// populate cat_and_tag[][] array
        foreach ($entries as $entry) {
            if ($entry->data['entry'] ?? true)
                $this->build_cat_and_tag($entry,$collection->draftOverride);
        }
    }

    ++$totalCollection;
    if ($this->buildCollectionIndex($collection, 0, $dest)) $collectionCount++;

    for ($page=1; $page <= $totalPages; $page++)
        $this->buildCollectionIndex($collection, $page, $dest);
}

The two PHP functions for fork and join are thus:

protected function beginParallel(int $nentries, int $aprocs) : void {
    $this->pid = 0;
    $this->procnr = 0;
    $this->nprocs = 1;
    if ($nentries < 128) return;	// too few entries to warrant forking
    $this->nprocs = $aprocs;	// aprocs = allowed procs, specified on commmand-line
    for ($this->procnr=0; $this->procnr<$this->nprocs; ++$this->procnr)
        if (($this->pid = pcntl_fork())) return;	// child returns to work
}

protected function endParallel() : void {
    if ($this->pid) exit(0);	// exit child process; pid=0 is parent
}

This fork and join via pcntl_fork() does not work on Microsoft Windows.

5. Benchmarking. How much of an improvement do we get by this? For this very blog with 561 entries, the runtimes can be more than halved. This is in line with the 60% runtime used by the output template processing. It should be noted that this blog is comprised of five collections:

1. aux: 7 entries
2. blog: 452 entries, only these are parallelized!
3. gallery: 7 entries
4. music: 69 entries
5. error: 1 entry

The parallelization kicks in only for at least 128 entries. I.e., only the blog-part is parallelized, the music-part and the other parts are not.

Another benchmark is the Lemire blog converted to Simplified Saaze, see Example Theme for Simplified Saaze: Lemire.

Command-lines are:

time php saaze -p16 -mortb /tmp/build
time php saaze -p16 -rb /tmp/buildLemire

Then we are varying the parameter -p. All output is to /tmp, which is a RAM disk in Arch Linux. Obviously, I do not want to measure disk read or write speed. I want to measure the processing speed of Simplified Saaze.

Timings are from time, taking real time.

Overall, with just 20 lines of PHP we can halve the runtime. For JIT enabled, the drop in runtime is not so pronounced, but also almost halved.

The very good performance of JIT, which we can see here, is in line with the findings in Phoronix: PHP 8.0 JIT Is Offering Very Compelling Performance Ahead Of Its Alpha.

They are vaguely documented by GitHub here: Get feeds.

For example, for my saaze GitHub repository the feed for the commits is:

<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" xml:lang="en-US">
  <id>tag:github.com,2008:/eklausme/saaze/commits/master</id>
  <link type="text/html" rel="alternate" href="https://github.com/eklausme/saaze/commits/master"/>
  <link type="application/atom+xml" rel="self" href="https://github.com/eklausme/saaze/commits/master.atom"/>
  <title>Recent Commits to saaze:master</title>
  <updated>2024-02-17T12:58:12Z</updated>
  <entry>
    <id>tag:github.com,2008:Grit::Commit/48560c8bb5535cfaacdf2fc1be153c43448051d5</id>
    <link type="text/html" rel="alternate" href="https://github.com/eklausme/saaze/commit/48560c8bb5535cfaacdf2fc1be153c43448051d5"/>
    <title>
        Reduced CPU overhead in composer
    </title>
    <updated>2024-02-17T12:58:12Z</updated>
    <media:thumbnail height="30" width="30" url="https://avatars.githubusercontent.com/u/1020520?s=30&amp;v=4"/>
    <author>
      <name>eklausme</name>
      <uri>https://github.com/eklausme</uri>
    </author>
    <content type="html">
      &lt;pre style=&#39;white-space:pre-wrap;width:81ex&#39;&gt;Reduced CPU overhead in composer&lt;/pre&gt;
    </content>
  </entry>
  ...
</feed>

The above output was produced by below command-line:

curl https://github.com/eklausme/saaze/commits.atom

Previous profiling measurements with XHProf and PHPSPY indicated that the handling of the return value from MD4C via FFI::String takes some time. So I changed FFI to a "real" PHP extension. I measured again. Result: No difference between FFI and PHP extension. So the profiling measurements were misleading.

Also the following claim in the PHP manual is downright false:

it makes no sense to use the FFI extension for speed; however, it may make sense to use it to reduce memory consumption.

Nevertheless, writing a PHP extension was a good exercise.

Literature on writing PHP extension are here:

1. Sara Golemon: Extending and Embedding PHP, Sams Publishing, 2006, xx+410 p.
2. PHP Internals: Zend extensions
3. https://github.com/dstogov/php-extension

The PHP extension code is in GitHub: php-md4c.

1. Walk through the C code. For this simple extension there is no need for a separate header file. The extension starts with basic includes for PHP, for the phpinfo(), and for MD4C:

// MD4C extension for PHP: Markdown to HTML conversion

#ifdef HAVE_CONFIG_H
#include "config.h"
#endif

#include <php.h>
#include <ext/standard/info.h>
#include <md4c-html.h>

The following code is directly from the FFI part php_md4c_toHtml.c:

struct membuffer {
    char* data;
    size_t asize;	// allocated size = max usable size
    size_t size;	// current size
};

The following routines are also almost the same as in the FFI case, except that memory allocation is using safe_pemalloc() instead of native malloc(). In our case this doesn't make any difference.

static void membuf_init(struct membuffer* buf, MD_SIZE new_asize) {
    buf->size = 0;
    buf->asize = new_asize;
    if ((buf->data = safe_pemalloc(buf->asize,sizeof(char),0,1)) == NULL)
        php_error_docref(NULL, E_ERROR, "php-md4c.c: membuf_init: safe_pemalloc() failed with asize=%ld.\n",(long)buf->asize);
}

Next routine uses safe_perealloc() instead of realloc().

static void membuf_grow(struct membuffer* buf, size_t new_asize) {
    buf->data = safe_perealloc(buf->data, sizeof(char*), new_asize, 0, 1);
    if (buf->data == NULL)
        php_error_docref(NULL, E_ERROR, "php-md4c.c: membuf_grow: realloc() failed, new_asize=%ld.\n",(long)new_asize);
    buf->asize = new_asize;
}

The rest is identical to FFI.

static void membuf_append(struct membuffer* buf, const char* data, MD_SIZE size) {
    if (buf->asize < buf->size + size)
        membuf_grow(buf, buf->size + buf->size / 2 + size);
    memcpy(buf->data + buf->size, data, size);
    buf->size += size;
}

static void process_output(const MD_CHAR* text, MD_SIZE size, void* userdata) {
    membuf_append((struct membuffer*) userdata, text, size);
}

static struct membuffer mbuf = { NULL, 0, 0 };

Now we come to something PHP specific. We encapsulate the C function into PHP_FUNCTION. Furthermore the arguments of the routine are parsed with ZEND_PARSE_PARAMETERS_START(1, 2). This routine must have at least one argument. It might have an optional second argument. That is what is meant by (1,2). The return string is allocated via estrndup(). In the FFI case we just return a pointer to a string.

/* {{{ string md4c_toHtml( string $markdown, [ int $flag ] )
 */
PHP_FUNCTION(md4c_toHtml) {	// return HTML string
    char *markdown;
    size_t markdown_len;
    int ret;
    long flag = MD_DIALECT_GITHUB | MD_FLAG_NOINDENTEDCODEBLOCKS;

    ZEND_PARSE_PARAMETERS_START(1, 2)
        Z_PARAM_STRING(markdown, markdown_len)
        Z_PARAM_OPTIONAL Z_PARAM_LONG(flag)
    ZEND_PARSE_PARAMETERS_END();

    if (mbuf.asize == 0) membuf_init(&mbuf,16777216);	// =16MB

    mbuf.size = 0;	// prepare for next call
    ret = md_html(markdown, markdown_len, process_output,
        &mbuf, (MD_SIZE)flag, 0);
    membuf_append(&mbuf,"\0",1); // make it a null-terminated C string, so PHP can deduce length
    if (ret < 0) {
        RETVAL_STRINGL("<br>- - - Error in Markdown - - -<br>\n",sizeof("<br>- - - Error in Markdown - - -<br>\n"));
    } else {
        RETVAL_STRING(estrndup(mbuf.data,mbuf.size));
    }
}
/* }}}*/

The following two PHP extension specific functions are just for initialization and shutdown. The following diagram from PHP internals shows the sequence of initialization and shutdown.

Init: Do nothing.

/* {{{ PHP_MINIT_FUNCTION
 */
PHP_MINIT_FUNCTION(md4c) {	// module initialization
    //REGISTER_INI_ENTRIES();
    //php_printf("In PHP_MINIT_FUNCTION(md4c): module initialization\n");

    return SUCCESS;
}
/* }}} */

Shutdown: Do nothing.

/* {{{ PHP_MSHUTDOWN_FUNCTION
 */
PHP_MSHUTDOWN_FUNCTION(md4c) {	// module shutdown
    if (mbuf.data) pefree(mbuf.data,1);
    return SUCCESS;
}
/* }}} */

The following function prints out information when called via phpinfo().

/* {{{ PHP_MINFO_FUNCTION
 */
PHP_MINFO_FUNCTION(md4c) {
    php_info_print_table_start();
    php_info_print_table_row(2, "MD4C", "enabled");
    php_info_print_table_row(2, "PHP-MD4C version", "1.0");
    php_info_print_table_row(2, "MD4C version", "0.5.2");
    php_info_print_table_end();
}
/* }}} */

The output looks like this:

Below describes the argument list.

/* {{{ arginfo
 */
ZEND_BEGIN_ARG_INFO(arginfo_md4c_test, 0)
ZEND_END_ARG_INFO()

ZEND_BEGIN_ARG_INFO(arginfo_md4c_toHtml, 1)
    ZEND_ARG_INFO(0, str)
    ZEND_ARG_INFO_WITH_DEFAULT_VALUE(0, flag, "MD_DIALECT_GITHUB | MD_FLAG_NOINDENTEDCODEBLOCKS")
ZEND_END_ARG_INFO()
/* }}} */

/* {{{ php_md4c_functions[]
 */
static const zend_function_entry php_md4c_functions[] = {
    PHP_FE(md4c_toHtml,	arginfo_md4c_toHtml)
    PHP_FE_END
};
/* }}} */

The zend_module_entry is somewhat classical. All the above is configured here.

/* {{{ md4c_module_entry
 */
zend_module_entry md4c_module_entry = {
    STANDARD_MODULE_HEADER,
    "md4c",						// Extension name
    php_md4c_functions,			// zend_function_entry
    NULL,	//PHP_MINIT(md4c),	// PHP_MINIT - Module initialization
    PHP_MSHUTDOWN(md4c),		// PHP_MSHUTDOWN - Module shutdown
    NULL,						// PHP_RINIT - Request initialization
    NULL,						// PHP_RSHUTDOWN - Request shutdown
    PHP_MINFO(md4c),			// PHP_MINFO - Module info
    "1.0",						// Version
    STANDARD_MODULE_PROPERTIES
};
/* }}} */

This seemingly innocent looking statement is important: Without it you will get PHP Startup: Unable to load dynamic library.

#ifdef COMPILE_DL_TEST
# ifdef ZTS
ZEND_TSRMLS_CACHE_DEFINE()
# endif
#endif
ZEND_GET_MODULE(md4c)

2. M4 config file. PHP extension require a config.m4 file.

dnl config.m4 for php-md4c extension

PHP_ARG_WITH(md4c, [whether to enable MD4C support],
[  --with-md4c[[=DIR]]       Enable MD4C support.
                          DIR is the path to MD4C install prefix])

if test "$PHP_YAML" != "no"; then

    AC_MSG_CHECKING([for md4c headers])
    for i in "$PHP_MD4C" "$prefix" /usr /usr/local; do
        if test -r "$i/include/md4c-html.h"; then
            PHP_MD4C_DIR=$i
            AC_MSG_RESULT([found in $i])
            break
        fi
    done
    if test -z "$PHP_MD4C_DIR"; then
        AC_MSG_RESULT([not found])
        AC_MSG_ERROR([Please install md4c])
    fi

    PHP_ADD_INCLUDE($PHP_MD4C_DIR/include)
    dnl recommended flags for compilation with gcc
    dnl CFLAGS="$CFLAGS -Wall -fno-strict-aliasing"

    export OLD_CPPFLAGS="$CPPFLAGS"
    export CPPFLAGS="$CPPFLAGS $INCLUDES -DHAVE_MD4C"
    AC_CHECK_HEADERS([md4c.h md4c-html.h], [], AC_MSG_ERROR(['md4c.h' header not found]))
    #AC_CHECK_HEADER([md4c-html.h], [], AC_MSG_ERROR(['md4c-html.h' header not found]))
    PHP_SUBST(MD4C_SHARED_LIBADD)

    PHP_ADD_LIBRARY_WITH_PATH(md4c, $PHP_MD4C_DIR/$PHP_LIBDIR, MD4C_SHARED_LIBADD)
    PHP_ADD_LIBRARY_WITH_PATH(md4c-html, $PHP_MD4C_DIR/$PHP_LIBDIR, MD4C_SHARED_LIBADD)
    export CPPFLAGS="$OLD_CPPFLAGS"

    PHP_SUBST(MD4C_SHARED_LIBADD)
    AC_DEFINE(HAVE_MD4C, 1, [ ])
    PHP_NEW_EXTENSION(md4c, md4c.c, $ext_shared)
fi

3. Compiling. Run

phpize
./configure
make

Symbols are as follows:

$ nm md4c.so
0000000000002160 r arginfo_md4c_test
0000000000003d00 d arginfo_md4c_toHtml
                 w __cxa_finalize@GLIBC_2.2.5
00000000000040a0 d __dso_handle
0000000000003dc0 d _DYNAMIC
                 U _emalloc
                 U _emalloc_64
                 U _estrndup
00000000000016c8 t _fini
                 U free@GLIBC_2.2.5
00000000000016c0 T get_module
0000000000003fe8 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
00000000000021c8 r __GNU_EH_FRAME_HDR
0000000000001000 t _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000004180 b mbuf
00000000000040c0 D md4c_module_entry
                 U md_html
                 U memcpy@GLIBC_2.14
                 U php_error_docref
                 U php_info_print_table_end
                 U php_info_print_table_row
                 U php_info_print_table_start
0000000000003d60 d php_md4c_functions
                 U php_printf
0000000000001640 t process_output
0000000000001234 t process_output.cold
                 U _safe_malloc
                 U _safe_realloc
                 U __stack_chk_fail@GLIBC_2.4
                 U strlen@GLIBC_2.2.5
0000000000004168 d __TMC_END__
                 U zend_parse_arg_long_slow
                 U zend_parse_arg_str_slow
                 U zend_wrong_parameter_error
                 U zend_wrong_parameters_count_error
                 U zend_wrong_parameters_none_error
. . .
0000000000001380 T zif_md4c_toHtml
00000000000011cf t zif_md4c_toHtml.cold
0000000000001175 T zm_info_md4c
0000000000001350 T zm_shutdown_md4c
00000000000016b0 T zm_startup_md4c

4. Installing on Arch Linux. Copy the md4c.so library to /usr/lib/php/modules as root:

cp modules/md4c.so /usr/lib/php/modules

Finally activate the extension in php.ini:

extension=md4c

5. Notes on Windows. On Linux we use the installed MD4C library. As noted in Installing Simplified Saaze on Windows 10 #2 it is advisable to amalgamate all MD4C source files into a single file for easier compilation.

pacman -S certbot-nginx

Updating my domains is like this:

certbot --nginx -d eklausmeier.goip.de,klm.ddns.net,eklausmeier.mywire.org,klmport.no-ip.org,klm.no-ip.org

Its output is roughly

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for eklausmeier.goip.de and 4 more domains

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/eklausmeier.goip.de/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/eklausmeier.goip.de/privkey.pem
This certificate expires on 2024-05-19.
These files will be updated when the certificate renews.

Add the first two files in /etc/nginx/nginx.conf:

ssl_certificate      /etc/letsencrypt/live/eklausmeier.goip.de/fullchain.pem;
ssl_certificate_key  /etc/letsencrypt/live/eklausmeier.goip.de/privkey.pem;

Check with nginx -t. If all is OK, then restart with systemctl restart nginx.

Final check is with Qualys SSL Labs:

On my old WordPress blog, https://eklausmeier.wordpress.com, I had 79 subscribers. From their e-mail names, I would suspect that some of them were not really interested in my actual content but were a little bit spammy. Nevertheless, many seemed to be legitimate.

There are a lot of professional newsletter services on the market. For example:

1. https://www.mailjet.com
2. https://buttondown.email
3. https://mailchimp.com
4. https://omnisend.com

There are many more.

These solutions should be differentiated from mailing list software.

2. Data model. Initially, I thought of a single file used to store all information. Something like this: Handling of subscription-file: Read into a PHP hash table, change whatever needs change, and if there is a change required, e.g., new subscriber, then move the old file, and write a new file from the hash.

However, this file needs some protection using flock() to guard against simultaneous writing to it. After some thought it seems more advantageous to use a simple SQLite file, i.e., a database, which already handles concurrency out of the box.

A single database table suffices. Henceforth this table is called subscription.

State diagram for status is as below.

Create script for SQLite is like this:

drop table subscription;

create table subscription (
    email       text primary key,
    firstname   text,
    lastname    text,
    registration    date not null,
    IP          text not null,
    status      int not null,
    token       text not null
);

The following SQL statements will be used:

1. During sending out the newsletter: select email, firstname from subscription where status=1
2. New subscriber: insert into subscription (email,firstname,lastname,registration,IP,status,token) values (...)
3. Checking correct token: select token from subscription where email=:m
4. Updating status column: update subscription set status=:s where email=:m

The following columns could be added to better cope with malicious users.

3. Sketch of solution. Here are considerations and requirements for a simple newsletter software.

1. Programming this application in PHP is preferred as this can be installed on many hosting providers, which offer PHP, e-mail, DNS, etc.
2. Have one single database table, called subscription, see above.
3. Periodically reads incoming e-mails for new subscribers or unsubscription requests.
4. New subscribers add an entry to the subscription table.
5. Subscription requests will generate a random token, which is sent to the e-mail address.
6. Unsubscribe requests set the status column to inactive in the subscription table.
7. During deployment of a new post on the static site, or by manual start, send an e-mail to all recipients on the subscription table, which are active.
8. The IP address of the registrating web client is stored. With this we can defend against flooding of e-mail addresses, which all bounce. For example, this IP address can then be blocked in the firewall of the web-server.

The token does not need to be overly confidential. Its purpose is to defend against funny/stupid/malicious actors, who want to unsubscribe people against their will.

Handling of e-mails: For reading e-mail you can use imap_headers(), for sending imap_mail(). Also see Sending email using PhpMailer with Gmail XOAUTH2, and Gmail Email Inbox using PHP with IMAP.

Subscribing to the mailing list works with an empty e-mail that states Subscribe in the subject line. For unsubscribing you send Unsubscribe in the subject line and the token in the body part. These two operations are also supported by a simple web-form, which essentially asks for the e-mail address and the token from the user and then sends the confirmation e-mail and sets the status in the subscription table.

Reading e-mails is done every 20 minutes, e.g., controlled by cron. The reading process then analyses the subject field for Subscribe and Unsubscribe. This process also checks for any bounces. In case of a bounce the status flag is set to either bounced or bounced during registration. No distinction is made betweeen hard or soft bounces.

A subscription request makes an entry in the subscription table and sets the status column to in-limbo. The sender receives an e-mail, which he must confirm by e-mail or web form. Once the confirming e-mail is received or the web form is used to confirm then the status column is set to active. If a new subscription request is made with an already existing e-mail address then a new token is generated and sent, and the status remains its previous status, e.g., it might remain active or in-limbo.

If a malicious user subscribes to multiple e-mail addresses, which he does not own, then all these e-mail addresses are set to in-limbo. If the legitimate user now wants to subscribe, he can do so without fuss, because new tokens are sent out for any subscription requests. This prevents that e-mail addresses are blocked, which are not confirmed.

4. Web forms. The HTML form for processing subscribe and unsubscribe requests looks very simple:

First name: Arnold
Last name:   Schwarzenegger
E-mail address: Arnold.Schwarzenegger@Terminator.org
Token:   uIYkEk+ylks= (only required for unsubscribe)
    Subscribe       Unsubscribe       Resend token

Changing your e-mail address is done by subscribing to the new address, and then unsubscribing from the old one.

If you have lost or deleted the token for unsubscribing, then simply subscribe again with the same e-mail address. A token will be sent to you, which you then can use for unsubscribing.

While the e-mail address is mandatory, the first and last name are optional.

The actual e-mailing can be done with below simple HTML form:

Greeting:   Firstname will be taken
Content:   Your content
                    Send

The following e-mails are sent depending on the circumstances:

1. Once a user has entered his name and e-mail on the HTML form, he will be sent an e-mail to confirm his e-mail address with the generated token.
2. If the user has unsubscribed from the mailing list, he will receive a confirmation e-mail, which confirms that he has unsubscribed. If the token is wrong then no e-mail will be sent.
3. The actual content is sent to all members stored in the subscription table, which are active. I.e., this is the whole purpose of maintaining this e-mail list.

5. Effort estimation. I expect the whole code for this to be no more than 1kLines of PHP code. I expect the following PHP programs/files:

1. Handling the web form.
2. Run through cron and checking for new subscription or unsubscription requests. Checking for bounces.
3. Configurations for user-id, password, and hostname for e-mail host.
4. Sending an e-mail to each recipient in the subscription table, either by using a web form, or via command-line, taking a text file as input.

Possible problems ahead due to hosting limitations:

1. If you want to use Google Mail as mail provider you will encounter their limit of 500 mails per day.
2. Yahoo seems to have a limit of 500 mails per day.
3. Outlook also has a 500 mails per day limit.
4. IONOS imposes a 500 mails per hour limit.
5. Hetzner similarly restricts to 500 mails per hour.
6. Amazon SES has a limit of 200 mails per day

To counter above limits somewhat, you can split your e-mails into batches, i.e., send 500 e-mails the first hour, then another 500 mails the next hour. For this you need an additional table, which stores the batch-number, and the message text to be sent. Obviously, you will not actually send 500 e-mails, but rather 450 or so, to cope for the confirmation mails for new subscribers or unsubscribers.

I am quite surprised that a Google search didn't reveal any program, which already does something similar. The most resembling is this phpList.

Voraussetzungen: Es sei

ein beliebiges komplexes Polynom mit Koeffizienten $a_i\in\mathbb{C}$ und Nullstellen $\lambda_i\in\mathbb{C}$. Weiter sei

mit der Vereinbarung $a_{n+1}=a_{n+2}=\cdots=0$.

Behauptung: $\mathop{\rm Re}\nolimits \lambda_i<0$ genau dann, wenn

Für $a_0>0$ also $\Delta_i>0$, $i=1,\ldots,n$.

Beweis: Siehe das Buch von Gantmacher, Felix Ruvimovich (1908--1964), Gantmacher (1986), §16.6, "Matrizentheorie", Springer-Verlag, Berlin Heidelberg New York Tokyo, Übersetzung aus dem Russischen von Helmut Boseck, Dietmar Soyka und Klaus Stengert, 1986, 654 S.     ☐

Der obige Satz ist ein Spezialfall des allgemeinen Satzes von Routh/Hurwitz, der es erlaubt die genaue Anzahl der Nullstellen mit echt negativen Realteil genau anzugeben. Der folgende Satz von Liénard/Chipart aus dem Jahre 1914 hat gegenüber dem Stabilitäskriterium von Routh/Hurwitz den Vorteil, nur etwa halb so viele Minoren auf ihr Vorzeichen zu untersuchen.

2. Satz: Stabilitätskriterium von Liénard/Chipart nach Chipart, A.H., Liénard, Alfred-Marie (1869--1958).

Behauptung: $\mathop{\rm Re}\nolimits \lambda_i<0$ ist äquivalent zu einer der folgenden 4 Aussagen:

(1)     $a_n>0$, $a_{n-2}>0$, $\ldots$; $\Delta_1>0$, $\Delta_3>0$, $\ldots$,

(2)     $a_n>0$, $a_{n-2}>0$, $\ldots$; $\Delta_2>0$, $\Delta_4>0$, $\ldots$,

(3)     $a_n>0$, $a_{n-1}>0$, $a_{n-3}>0$, $\ldots$; $\Delta_1>0$, $\Delta_3>0$, $\ldots$,

(4)     $a_n>0$, $a_{n-1}>0$, $a_{n-3}>0$, $\ldots$; $\Delta_2>0$, $\Delta_4>0$, $\ldots.$

Beweis: Siehe erneut Gantmacher (1986), §16.13.     ☐

Für die Überprüfung eines vorgelegten Polynoms wählt man dann zweckmässigerweise von den vier Bedingungen diejenige, sodaß $\Delta_{n-1}$ oder $\Delta_n$ die geringere Zeilenzahl hat.

1. Satz: Formel von Faà di Bruno Es hänge $w$ von $u$ ab, $u$ ist hierbei Funktion von $x$. Es sei $D_x^k u$ die $k$-te Ableitung von $u$ nach $x$. Dann gilt

Beweis: Siehe Knuth, Donald Ervin (*1938), The Art of Computer Programming, Volume 1 -- Fundamental Algorithms, Addison-Wesley Publishing Company, Reading (Massachusetts) Menlo Park (California) London Sydney Manila, 1972, second printing, xxi+634 S. Siehe McEliece im o.a. Buch von Knuth, McEliece, Robert James. Bezeichnet $c(n,j,k_1,k_2,\ldots)$ den Bruchterm, so rechnet man durch Differenzieren

Hierbei ist es von Vorteil unendlich viele $k_i$ anzunehmen, obwohl $k_{n+1}=k_{n+2}=\cdots=0$. Im Induktionsschritt sind $k_1+\cdots+k_n=j$ und $k_1+2k_2+\cdots+nk_n=n$ Invarianten. Man kann nun $n! / k_1! (1!)^{k_1} k_2! (2!)^{k_2}\ldots$ kürzen und gelangt dann zu $k_1+2k_2+\cdots=n+1$. Man vgl. auch Bourbaki und Schwartz.     ☐

Diese Darstellungen für $f$ lassen sich für vektorwertige Funktionen entsprechend verallgemeinern. Wie im Eindimensionalen liegt auch hier wieder das Schwergewicht auf der Gewinnung von Restgliedformeln, oder mit den Worten von Mangoldt und Knopp: (Mangoldt, Hans Carl Friedrich von (1854--1925, Knopp, Konrad Hermann Theodor (1882--1957))

Ausdrücklich sei noch einmal betont, daß der wesentliche Inhalt des Taylorschen Satzes nicht darin besteht, daß ein Ansatz der Form

überhaupt gemacht werden kann. Das ist vielmehr unter der alleinigen Voraussetzung, daß $f^{(n)}(x_0)$ existiert, für jedes seinem Betrage nach hinreichend kleines $h$ unter allen Umständen möglich. $\ldots$ $R_n$ ist lediglich eine abkürzende Bezeichnung für die Differenz der linken Seite und der Summe dieser $(n+1)$ ersten Summanden der rechten Seite. Das Schwergewicht des Problems und damit der allein wesentliche Inhalt des Taylorschen Satzes liegt ausschließlich in den Aussagen, die über dieses Restglied gemacht werden können.

1. Defintion: (Multiindizes) Für $\alpha=(\alpha_1,\ldots,\alpha_n)\in\mathbb{N}^n$ sei die Ordnung eines Multiindex und die Multifakultät definiert zu

Ist $f$ eine $\left|\alpha\right|$-mal stetig differenzierbare Funktion, so sei die ^{Multiableitung} gesetzt zu

insbesondere $D_i^{\alpha_i}=D_i\ldots D_i$ ($i$ mal). Die ^{Multipotenz} für einen Vektor $x$ ist

Nach dem Satz von H.A. Schwarz, Schwarz, Hermann Armandus (1843--1921), ist die Reihenfolge des Differenzierens nach verschiedenen Variablen unerheblich, bei genügend glatter Funktion $f$.

2. Lemma: Es gilt

Beweis: Durch Induktion nach $n$, wenn man die Binomische Formel voraussetzt. Man rechnet über Induktion nach $k$, wenn man dies nicht benutzen will. Für $n=1$ ist die Behauptung klar. Für den Induktionsschluß klammert man $[x_1+(x_2+\cdots+x_n)]^k$.     ☐

Entsprechend gilt

also

Generalvoraussetzung: $f\colon U\subset\mathbb{R}^n\rightarrow\mathbb{R}$ sei $k$-mal stetig differenzierbar auf der offenen Menge $U$. Es sei $x\in U$ und $h\in\mathbb{R}^n$ derart, daß $x+th\in U$, $\forall t\in[0,1]$. Es sei $g\colon[0,1]\rightarrow\mathbb{R}$, mit $g(t):=f(x+th)$.

3. Hilfssatz: Die Funktion $g$ ist $k$-mal stetig differenzierbar und

Beweis: Induktion nach der Ordnung des Multiindex, also nach $k$. Für $k=1$ ist nach der Kettenregel

Induktionsschluß von $(k-1)\rightarrow k$:

Anwenden der Kettenregel und des Lemmas liefert

    ☐

4. Satz: Satz von Taylor, Taylor, Brook (1685--1731). Es sei $f$ jetzt sogar $(k+1)$-mal stetig differenzierbar. Dann existiert ein $\theta\in[0,1]$, so daß

Beweis: $g$ ist wie $f$ mindestens $(k+1)$-mal stetig differenzierbar. Nach der Taylorformel für eine Veränderliche existiert ein $\theta\in[0,1]$, so daß

Einsetzen der im Hilfssatz ermittelten Formeln liefert unmittelbar das Ergebnis.     ☐

5. Corollar: Es sei $f$ mindestens $k$-mal stetig differenzierbar und es sei $h$ hinreichend klein. Dann gilt

dabei steht $o(\left|h\right|^k)$ als Abkürzung für eine Funktion $\varphi$ mit $\varphi(0)=0$ und

Beweis: Nach dem vorhergehenden Satz gibt es ein von $h$ abhängiges $\theta\in[0,1]$, mit

wobei

Wegen der vorausgesetzten Stetigkeit von $D^\alpha f$ verschwindet $r_\alpha(\cdot)$ bei 0, also $\displaystyle\lim_{h\to0} r_\alpha(h)=0$. Setzt man

so folgt $\displaystyle\lim_{h\to0} {\varphi(h) / \left|h\right|^k} = 0$, d.h. $\varphi(h)=o(\left|h\right|^k)$, denn

    ☐

Der Satz von Taylor im $\mathbb{R}^m$ entsteht durch komponentenweise Anwendung der vorherigen Resultate. Man benötigt allerdings $m$ möglicherweise verschiedene Zwischenstellen.

6. Beispiel: Es sei $f\colon\mathbb{R}\rightarrow\mathbb{R}^3$ mit $f(t):=(\sin t,{\mskip 3mu}\cos t,{\mskip 3mu}t)$. Dann ist $f'(t)=(\cos t,{\mskip 3mu}-\sin t,{\mskip 3mu}1)$ und wenn man nur eine einzige Zwischenstelle zulässt erhält man den Widerspruch

Aus $\cos\xi=0=\sin\xi$ folgt $\cos^2\xi+\sin^2\xi=0$.

Literatur: Otto Forster (*1937): Analysis 2.

1. Satz: Voraussetzungen: Es seien $a_{ij}(\lambda)$ differenzierbare Funktionen. Es sei

ferner

insbesondere $\displaystyle{ \alpha_i^j = (-1)^{i+j} A_{1\ldots\widehat\imath\ldots n}^{1\ldots\widehat\jmath\ldots n}. }$

Behauptung:

Beweis: Entwickelt man $A(\lambda)$ nach dem Laplaceschen Entwicklungssatz nach der $i$-ten Zeile, so erkennt man $\partial A/\partial(a_{ij}) = \alpha_i^j$. Anwenden der Kettenregel liefert die mittleren Identitäten. Die letzte Identität ist nur eine Umsortierung der vorherigen (Laplacescher Entwicklungssatz rückwärts gelesen).     ☐

Man vgl. auch Bourbaki (1976): "Éléments de mathématique: Fonctions d'une variable réelle -- Théorie élémentaire", Hermann, Paris, 1976, 54+38+69+46+55+31+38 S. = 331 S.

2. Die Jacobimatrizen einiger Matrizenfunktionen, wie Spur, Determinante, Matrizenprodukt.

Es sei $y=f(x_{11},\ldots,x_{1n},x_{21},\ldots,x_{2n},\ldots,x_{m1},\ldots,x_{mn})$ eine reelle Funktion in $mn$ Veränderlichen, also $y=f(X)$. Es bezeichne

Im Falle $X=(x_1,\ldots,x_n)$ ist ${{dy\over dX}=\nabla y}$.

3. Satz: (1)     $\displaystyle{{d{\mskip 5mu}ax\over dx} = a}$,     $\displaystyle{{d{\mskip 5mu}x^\top Ax\over dx} = 2Ax}$,     ($A=A^\top$).

(2)     $\displaystyle{{d{\mskip 5mu}\ln\det X\over dX} = (X^\top)^{-1}}$,     $\displaystyle{{d{\mskip 5mu}\det X\over dX} = (\det X)^{-1} (X)^{-1}}$.

(3)     $\def\tr{\mathop{\rm tr}}\displaystyle{{d{\mskip 5mu}\tr X^{-1}A\over dX} = -(X^{-1} A X^{-1})^\top}$.

Beweis: (1) ist klar. Bei (2) beachte man

entsprechend

Zu (3): Es gelten

    ☐

1. Sei $f$ eine geeignet gewählte holomorphe Funktion. Dann definiert man für eine quadratische Matrix $A$ die Matrixfunktion $f(A)$ zu

Wegen dem Satz von Cauchy, Cauchy, Augustin Louis (1789--1857), hängt $f(A)$ nicht von der Wahl der Kurve $\Gamma$ ab. Offensichtlich ist $S^{-1}f(A)S=f(S^{-1}AS)$, für jede invertierbare $(n\times n)$-Matrix $S$. Ohne Einschränkung kann man deshalb $A$ bei den weiteren Überlegungen als Jordanmatrix, Jordan, Camille (1838--1922), voraussetzen. Also $A = J = \mathop{\rm diag}(J_\nu)_{\nu=1}^k$, wobei $J_\nu$ Jordanblock ist. Es ist

Viele Behauptungen reduzieren sich damit also sogar lediglich auf die Betrachtung eines einzelnen Jordanblockes $J_\nu$, mit $J_\nu=\lambda_0\delta_{xy}+\left(\delta_{x+1,y}\right)_{x,y=1}^m$.

2. Sei nun $J$ Jordan-Block der Größe $k\times k$ zum Eigenwert $\lambda_0$. Dann gilt

Insbesondere für die spezielle Funktion $f(\lambda):=\lambda^n$ ergibt sich

wobei $\lambda^{-j}:=0$, für $j\in\mathbb{N}$.

3. Diese Darstellungen finden ihre Begründung durch den folgenden Satz, obwohl für den Fall $\lambda^n$ die Darstellung auch leicht direkt unter Benutzung von $J^n = (\lambda I + N)^n$, mit geeignetem Nilpotenzblock $N$ und der binomischen Formel bewiesen werden kann. Man braucht dann nicht den ganzen Weg über Matrizenfunktionen zu gehen. Möchte man die Integraldarstellung stärker berücksichtigen rechnet man wie folgend. Allgemein ist $f(A)=(1/2\pi i)\int_\Gamma f(z)(Iz-A)^{-1}dz$. Entwicklung des Cauchy-Kernes liefert

Dann berechnet man das Residuum durch Vertauschen von Integration und Summation zu

4. Satz: Es ist

Sind $f$ und $g$ holomorph auf (möglicherweise verschiedenen) Umgebungen des Spektrums von $A$, so gilt

Beweis: Es genügt, w.o. bemerkt, sich auf ein einziges Jordankästchen $J$ der Größe $m\times m$ zu beschränken. Es sei $\Gamma$ ein positiv orientierter Kreis um $\lambda_0$. Es ist

wobei $N = (\delta_{x+1,y})_{x,y}^m$, also $N^m=0\in\mathbb{C}^{m\times m}$ ist. Wegen $\int_\Gamma d\lambda/(\lambda-\lambda_0)=2\pi i$, und $\int_\Gamma (\lambda-\lambda_0)^k d\lambda=0$, für $k\in\mathbb{Z}\setminus\{-1\}$ gilt offensichtlich ${1\over2\pi i}\int_\Gamma (I\lambda-J)d\lambda=I$ und

Die additive Linearität ist klar. Für die multiplikative Aussage schließt man: Ist $f(\lambda)=\sum_{k=0}^\infty (\lambda-\lambda_0)^k f_k$ und $g(\lambda)=\sum_{k=0}^\infty (\lambda-\lambda_0)^k g_k$, so ist $f(\lambda)g(\lambda)=\sum_{k=0}^\infty (\lambda-\lambda_0)^k h_k$, mit $h_k=\sum_{i=0}^k f_i g_{k-i}$. Folglich

    ☐

Mit der Darstellung für $J^n$ ergibt sich leicht der folgende Sachverhalt.

5. Satz: Sei $J$ eine beliebige Jordanmatrix. Dann gelten:

(1) $J^n\to0$ genau dann, wenn $\left|\lambda\right| < 1$.

(2) $\sup_{n=1}^\infty|J^n|\le\rm const$ genau dann, wenn $\left|\lambda\right| \le 1$ und zu Eigenwerten vom Betrage 1 nur lineare Elementarteiler gehören, also die Jordanblöcke zum Eigenwert 1 stets von der Größe $(1\times 1)$ sind.

Wegen $A=XJY$, $Y=X^{-1}$ und damit $A^n=XJ^nY$ und wegen $\left|A^n\right|\le\left|X\right|\cdot\left|J^n\right|\cdot\left|Y\right|$, erhält man daher für eine beliebige quadratische Matrix $A$ den folgenden Satz.

6. Satz: Seien $\lambda_i$ für $i=1,\ldots k$, die Eigenwerte der Matrix $A$. Dann gelten

(1) $\def\mapright#1{\mathop{\longrightarrow}\limits^{#1}}|A|\mapright{n\to\infty}0$ genau dann, wenn $|\lambda_i|<1$, für alle $i=1,\ldots,k$, und

(2) $|A^n|$ beschränkt für alle $n\in\mathbb{N}$ genau dann, wenn $|\lambda_i|\le1$ und zu Eigenwerten vom Betrage 1, nur $(1\times 1)$-Jordanblöcke korrespondieren.

7. Bemerkung: Es gelten die Äquivalenzen

Beweis: Zu: $\sum_{n=0}^\infty A^n=(I-A)^{-1}$, falls $\rho(A)<1$. Ist $\lambda$ Eigenwert von $A$, so ist $(1-\lambda)$ Eigenwert von $(I-A)$. Wegen $|\lambda|<1$, ist $(I-A)$ invertierbar. Weiter

Somit gilt für alle $n\in\mathbb{N}$

und damit folgt wegen $A^n\to0$, die Behauptung. Die Rückrichtung $\rho(A)<1$, falls $\sum A^n = (I-A)^{-1}$ ist klar aufgrund der notwendigen Konvergenzbedingung für die Reihe. Die restlichen Äquivalenzen ergeben sich u.a. mit Hilfe des vorhergehenden Satzes und sind offensichtlich.     ☐

8. Eine andere Anwendung für die Darstellung von $J^n$, ist die Lösungsdarstellung für homogene, lineare Differenzengleichungen mit konstanten Koeffizienten. Durch Übergang von der Begleitmatrix zur Jordanmatrix erkennt man dann recht schnell die Lösungsdarstellung für die Differenzengleichung. Es ist

also

9. Satz: Voraussetzung: Es habe $L(\lambda)=\lambda^\ell+a_{\ell-1}\lambda^{\ell-1}+\cdots+a_0 \in \mathbb{C}$ die Faktorisierung

Behauptung: Der Lösungsraum der homogenen, linearen Differenzengleichung $a_{m+\ell}+a_{\ell-1}x_{m+\ell-1}+\cdots+a_0x_m=0$ hat die Dimension $\ell$ und wird aufgespannt von

wobei $\mathop{\rm grad} p_\nu=\eta_\nu-1$, $\nu=1,\ldots,k$. Der Fall $\mathop{\rm grad} p_\nu=0$ bedeutet dabei Konstante.

Beweis: Sei $u_m:=(x_{m-1+\ell},\ldots,x_m)\in\mathbb{C}^\ell$. Die Lösung der Differenzengleichung $L(E)x_m=0$ lautet $u_m = C_1^m u_0 = X J^m Y u_0$, wobei $Y=X^{-1}$ die Matrix der Linksjordanvektoren und $X$ die Matrix der Rechtsjordanvektoren ist. Die Multiplikation von links mit $X$ und von rechts mit $Y$ bewirkt eine Vermischung der einzelnen Jordankästchen. Nach Ausklammern von gemeinsamen Faktoren stehen vor $\mu_\nu$ Summen von Binomialkoeffizienten $m\choose\rho_\nu$, $0\le\rho_\nu<\eta_\nu$, $\nu=1,\ldots,k$, also Polynome in $m$. Da $C_1$ stets nicht-derogatorisch ist -- betrachte Minor $(C_1)_{1,\ldots,n-1}^{2,\ldots,n}$ -- beträgt der Grad von $p_\nu$ genau $\eta_\nu-1$, wegen $\mathop{\rm grad}{m\choose\eta_\nu-1}=\eta_\nu-1$. Aufgrund von $\sum\eta_\nu=\ell$ hat man insgesamt $\ell$ freie Parameter. Noch zu zeigen: die lineare Unabhängigkeit der angegebenen Lösung.     ☐

10. Corollar: Die Folgen $(m^i{\mskip 3mu}\mu_\nu^m)$, $i=0,\ldots,\eta_\nu-1$, für $\nu=1,\ldots,\ell$, bilden eine Basis für den Lösungsraum der Differenzengleichung.

2. Homomorphismus in obere Dreiecksmatrizen

1. Es gibt auch einen anderen Zugang zu holomorphen Matrixfunktionen, siehe den Artikel der beiden Autoren Yasuhiko Ikebe und Toshiyuki Inagaki, Ikebe/Inagaki (1986), "An Elementary Approach to the Functional Calculus for Matrices", The American Mathematical Monthly, Vol 93, No 3, May 1986, pp.390--392

Sei $f$ in einer Umgebung von $\{\lambda_1,\ldots,\lambda_r\}$ genügend oft differenzierbar. Für ein festes $n\in\mathbb{N}$ setzt man

Für $f(z)=z$ ergibt sich

d.h. also ein einfacher Jordanblock der Größe $n\times n$ zum Eigenwert $\lambda$. Mit $J$ sei stets ein solcher Jordanblock gemeint. Ist $f(z)\equiv c=\rm const$, so ist $f^*(z)=cI$.

Die Abbildung $*\colon f\rightarrow f^*$ ist ein Homomorphismus der Algebra der analytischen Funktionen in einer Umgebung von $\{\lambda_1,\ldots,\lambda_r\}$ in die kommutative Algebra der oberen Dreiecksmatrizen.

2. Satz: (Homomorphiesatz) Es gelten

(1)     $(f+g)^* = f^* + g^*$,     Additivität,

(2)     $(cf)^* = c{\mskip 3mu}f^*$, $c\in\mathbb{C}$ fest,     Homogenität,

(3)     $(fg)^* = f^* {\mskip 3mu} g^* = g^* {\mskip 3mu} f^*$,     Multiplikation und Kommutativität,

(4)     $(f/g)^* = f^* {\mskip 3mu} (g^*)^{-1} = (g^*)^{-1}{\mskip 3mu} f^*$, falls $g^*(z)\ne0$,     Quotientenbildung und Kommutativität,

(5)     $(1/g)^* = (g^*)^{-1}$, falls $g^*(z)\ne0$,     Inversenbildung.

Durch wiederholtes Anwenden von (1), (2) und (4) ergibt sich sofort

3. Corollar: Sei $f$ eine rationale Funktion ohne Pol in $\lambda$ und sei $f=p/q$ die vollständig gekürzte Darstellung, mit also teilerfremden Polynomen $p$ und $q$. Dann gilt

Aber auch für Potenzreihen rechnet man wie erwartet. Dies zeigt die

4. Folgerung: Sei $f(\lambda)=a_0+a_1z+\cdots{\mskip 3mu}$ eine Potenzreihe mit Konvergenzradius echt größer als $\left|\lambda\right|$. Dann gilt $ f^*(\lambda) = a_0I+a_1J+\cdots{\mskip 3mu}. $

Zu einer vorgegebenen festen quadratischen Matrix $A$ sei die (bis auf Permutation eindeutige) Jordannormalform $X^{-1}AX=\mathop{\rm diag}\left(J_1,\ldots,J_m\right)$ betrachtet. Hierbei ist $X$ (Matrix der Rechtsjordanketten) invertierbar. $J_i$ bezeichnet einen einfachen Jordanblock zum Eigenwert $\mu_i$, $i=1,\ldots,m$. Die $\mu_i$ müssen nicht notwendig verschieden sein. Ist $f$ eine analytische Funktion in der Umgebung von $\{\mu_1,\ldots,\mu_m\}$, so definiert man $f(A)$ durch

Das Corollar und die Folgerung zeigen, daß $f(A)$ übereinstimmt mit dem, was man gängigerweise erwartet, zumindestens für rationale Funktionen und für Potenzreihen. Direkt aus der Definition folgt

5. Satz: Identitätssatz. Seien $\lambda_1,\ldots,\lambda_r$ die verschiedenen Eigenwerte von $A$. Die Funktionen $f$ und $g$ seinen analytisch in einer Umgebung von $\{\lambda_1,\ldots,\lambda_r\}$. Dann gilt Gleichheit $f(A)=g(A)$ genau dann, wenn die Ableitungen an den Eigenwerten bis zu entsprechender Ordnung übereinstimmen, also

wobei $m_k$ die Größe des größten Jordanblockes zum Eigenwert $\lambda_k$ bezeichnet.

Die oben als Definition für $f(A)$ benutzte Integralformel lässt sich nun, da Funktionen von Matrizen jetzt anders definiert wurden, auch beweisen.

6. Satz: Integraldarstellung für $f(A)$. Sei $\Gamma$ eine einfache geschlossene Kurve, die in ihrem Inneren die sämtlichen Eigenwerte von $A$ umschließt. Sei $f$ holomorph auf $\Gamma$ und im Inneren von $\Gamma$. Dann gilt

Beweis: Wie üblich reduziert sich der Beweis auf die Betrachtung eines einzelnen Jordanblockes $J$ der Größe $n\times n$. Man rechnet

Beim Übergang von der ersten Zeile zur zweiten Zeile wurde benutzt %

und beim Übergang von der $2^{\rm ten}$ zur $3^{\rm ten}$, daß die Inverse von $I\tau-J$ halt so aussieht.     ☐

Der vorletzte Satz (Identitätssatz für Matrixfunktionen) zeigt, daß für eine feste Matrix $A$, die Matrixfunktion als Matrixpolynom darstellbar ist, da eine Übereinstimmung nur an endlich vielen Ableitungen gefordert ist. Sind die $m_k$ ($k=1,\ldots,r$) bekannt, so kann für eine feste Matrix $A\in\mathbb{C}^{n\times n}$ ein Ansatz der Form $g(\lambda) = a_{n-1}\lambda^{n-1} + \cdots + a_1\lambda + a_0$ gemacht werden, und man erhält die $a_i$ als Lösung einer Hermiteschen Interpolationsaufgabe. Sind alle Eigenwerte verschieden, also $m_k=1$ ($k=1,\ldots,r$), so liegt eine gewöhnliche Interpolationsaufgabe zugrunde. Die Lösung geschieht beispielsweise mit Newtonschen Differenzen oder der Lagrangeschen Formel, u.U. auch über die Cramersche Regel. Zu überprüfen ist, ob $f(\lambda)$ für die Eigenwerte $\lambda_1,\ldots,\lambda_r$ auch tatsächlich definiert ist. Probleme treten z.B. auf bei $f(\lambda)=\sqrt\lambda$, $f(\lambda)=\ln\lambda$, für $\lambda\notin\mathbb{R}^+$. Für $A\in\mathbb{C}^{1\times1}$ entartet die Aussage des Identitätssatzes in eine leere Aussage, nämlich $f=g\iff f=g$.

Einfache Folgerungen direkt aus der Definition von Matrizenfunktionen sind nun die folgenden Ergebnisse.

7. Satz: Satz von Cayley/Hamilton, 1.te Fassung, Cayley, Arthur (1821--1895), Hamilton, William Rowan (1805--1865). Das charakteristische Polynom $\chi(z)=\det(Iz-A)$ für $A\in\mathbb{C}^{n\times n}$ annulliert als Matrixpolynom aufgefaßt $A$, also es gilt $\chi(A)=0\in\mathbb{C}^{n\times n}$.

Beweis: Nach Charles A. McCarthy (1975): "The Cayley-Hamilton Theorem", The American Mathematical Monthly, April 1975, Vol 82, No 4, pp.390--391. Die Inverse von $Iz-A$ ist $\left[\det(Iz-A)\right]^{-1} M_{\mu\nu}(z)$, wobei $\mathop{\rm grad} M_{\mu\nu}\le n-1$. Die Integraldarstellung von $\chi(A)$ liefert

nach dem Cauchyschen Integralsatz ($\int_\Gamma f=0$, $f$ holomorph).     ☐

8. Definition: Zu einer Matrix $A\in\mathbb{C}^{n\times n}$ mit Eigenwerten $\lambda_1,\ldots,\lambda_r$ ($r\le n$) und Jordannormalform $A\sim\mathop{\rm diag}(J_1,\ldots,J_s)$ ($r\le s\le n$) heißt

das Minimalpolynom zu $A$. Hierbei ist $m_\nu$ die Ordnung des größten Jordanblocks zum Eigenwert $\lambda_\nu$. Zu $A={1{\mskip 3mu}0\choose0{\mskip 3mu}2}$ ist $\hat\chi(z)=(z-1)(z-2)$, zu $A=I$ ist $\hat\chi(z)=z-1$ unabhängig von $n$, und zu $A=\mathop{\rm diag}[{1{\mskip 3mu}1\choose0{\mskip 3mu}1},1,1,1]$ ist $\hat\chi(z)=(z-1)^2$ das Minimalpolynom.

9. Ähnliche Matrizen haben die gleiche Jordannormalform bis auf Umnumerierung von Jordanblöcken, daher das gleiche Minimalpolynom und auch das gleiche charakteristische Polynom. Offensichtlich verschwindet jeder Faktor $(J-\lambda_\nu I)^k$ ($\forall k\ge m_\nu$) für jeden Jordanblock $J$ zum Eigenwert $\lambda_\nu$, also $\hat\chi(A)=0\in\mathbb{C}^{n\times n}$, aber $(J-\lambda_\nu I)^k\ne0$ ($\forall k<m_\nu$). Damit ist $\hat\chi(z)$ ein Polynom minimalen Grades, welches $A$ annulliert. Aufgrund des führenden Leitkoeffizienten gleich 1 ist $\hat\chi(z)$ sogar eindeutig bestimmt. Da $\hat\chi$ stets Teiler von $\chi(z)=\det(Iz-A)$ ist, folgt

10. Satz: Satz von Cayley/Hamilton, 2.te Fassung, Cayley, Arthur (1821--1895), Hamilton, William Rowan (1805--1865). $\chi(A)=0\in\mathbb{C}^{n\times n}$. In Worten: Die Matrix $A$ annulliert ihr eigenes charakteristisches Polynom.

1. Satz: Satz von Rouché, Rouché, Eugéne (1832--1910).

Voraussetzung: $f$ und $g$ seien meromorph; $Z_f,Z_g$ und $P_f,P_g$ seien die Anzahl der Nullstellen bzw. Pole von $f,g$ innerhalb $\Gamma$, entsprechend ihrer Vielfachheit.

Behauptung: Gilt $\mathopen|f+g\mathclose|<\mathopen|f\mathclose|+\mathopen|g\mathclose|<\infty$ auf $\Gamma$, so folgt $Z_f-P_f=Z_g-P_g$ innerhalb von $\Gamma$.

Beweis: Nach Conway, John B., Conway (1978) "Functions of One Complex Variable", Springer-Verlag, New York Heidelberg Berlin, Second Edition, 1978, xiii+317 S. und Irving Leonard Glicksberg: "A Remark on Rouché's Theorem", The American Mathematical Monthly, March 1976, Vol 83, No 3, pp.186--187.

Aufgrund der strikten Dreiecksungleichung haben $f$ und $g$ keine Pole oder Nullstellen auf $\Gamma$. Weiter ist also

Die meromorphe Funktion $\lambda=f/g$ bildet $\Gamma$ auf $\Omega=\mathbb{C}\setminus\left[0,\infty\right[$ ab, da andernfalls für positive reelle $\lambda(z)$ gelten müsste $\lambda(z)+1<\lambda(z)+1$. Sei $\ell$ ein Zweig des Logarithmus auf $\Omega$. $\ell(f/g)$ ist eine Stammfunktion von $(f/g)^{-1}\cdot(f/g)'$. Somit

    ☐

Bei mehrfacher Umlaufung von $\Gamma$ ist die Aussage entsprechend zu modifizieren. Nach Glicksberg (1976) gilt der Sachverhalt allgemeiner in kommutativen, halbeinfachen Banachalgebren mit Einselement. Bekannter ist die schwächere Aussage: Aus $\mathopen|f+g\mathclose|<\mathopen|f\mathclose|<\infty$ auf $\Gamma$, folgt $Z_f=Z_g$ innerhalb $\Gamma$.

2. Beispiel: Für $p(z)=z^n+a_1z^{n-1}+\cdots+a_n$ gilt

Also

für $\mathopen|z\mathclose|\ge R$, $R$ geeignet groß. Der Satz von Rouché sagt, daß die Polynome $p(z)$ und $z^n$ gleichviele Nullstellen innerhalb der Kreisscheibe mit Radius $R$ haben. Dies ist der Fundamentalsatz der Algebra.

Der nächste Satz besagt: Wenn sich die Koeffizienten zweier Polynome wenig unterscheiden, so differieren auch die Nullstellen nur wenig. Erinnert sei daran, daß eine Implikation wahr sein kann, falls die Prämisse falsch ist.

3. Satz: (Stetigkeit der Wurzeln von Polynomen) Voraussetzungen: Es seien $p(\lambda):=\lambda^n+a_{n-1}\lambda^{n-1}+\cdots+a_1\lambda+a_0$ und $q(\mu):=\mu^n+b_{n-1}\mu^{n-1}+\cdots+b_1\mu+b_0$ zwei komplexe Polynome mit den Nullstellen $\lambda_1,\ldots,\lambda_n$ für $p$ und $\mu_1,\ldots,\mu_n$ für $q$. Die Koeffizienten $a_i$ und $b_i$ sind beliebige komplexe Zahlen.

Behauptung: $\forall\varepsilon>0: \exists\delta>0:\mskip 5mu$ $\left|a_i-b_i\right|<\delta{\mskip 3mu}\Longrightarrow{\mskip 3mu}\left|\lambda_i-\mu_i\right|< \varepsilon$, bei geeigneter Numerierung der Nullstellen $\lambda_i$ und $\mu_i$.

Beweis: Nach Ortega, James McDonough, Ortega (1972): "Numerical Analysis---A Second Course", Academic Press, New York and London, 1972, xiii+201 S.

Es seien $\gamma_1,\ldots,\gamma_k$ ($k\ge1$) die verschiedenen Wurzeln von $p$. Sei $\varepsilon$ kleiner gewählt als der kleinste halbe Abstand zwischen allen verschiedenen Nullstellen, also

Um $\gamma_i$ seien Scheiben $D_i$ mit Radius kleiner $\varepsilon$ gelegt, also

$p$ verschwindet auf keiner der $k$ Scheibenränder, also $p(z)\ne0$, $\forall z\in\partial D_i$, $\forall i=1,\ldots,k$. Aufgrund der Stetigkeit von $p$ und der Kompaktheit der Ränder, nimmt $p$ jeweils das Minimum und Maximum an. Es gibt also Zahlen $m_i$ [$i=1,\ldots,k$, die Minima halt], sodaß

Weiter sei

das Maximum von Polynom“resten” auf den jeweiligen Scheibenrändern und sei nun $\delta$ so klein gewählt, daß

Der obige Satz von Rouché ist nun anwendbar und sagt, daß $p$ und $q$ auf den vollen Scheiben die gleiche Anzahl von Nullstellen besitzen. M.a.W. die Nullstellen sind also nicht “weggelaufen”, sondern haben sich nur jeweils innerhalb der Scheiben bewegt.     ☐

Der Satz sagt nicht, daß die Wurzeln reell bleiben, sofern sie reell waren, bei Variation der Koeffizienten. Eine solche Aussage gilt so nicht. Hierzu bräuchte man stärkere Voraussetzungen.

4. Corollar: Die Eigenwerte einer Matrix hängen stetig von sämtlichen Matrixelementen ab.

Beweis: Die Eigenwerte der Matrix sind die Nullstellen des charakteristischen Polynomes. Die Koeffizienten des charakteristischen Polynoms hängen als Determinantenfunktion stetig von den Matrixelementen ab. Die Verkettung stetiger Funktionen ist wiederum stetig.     ☐

Das obige Corollar gilt nicht unbedingt für die Eigenvektoren.

5. Beispiel: Siehe Ortega (1972): Die Matrix nach J.W. Givens

hat die Eigenwerte $1\pm\varepsilon$ und die beiden Eigenvektoren

welche offensichtlich gegen keinerlei Grenzwert streben ($\varepsilon\to0$), jedoch $A(\varepsilon)\to{1{\mskip 3mu}0\choose 0{\mskip 3mu}1}$ und dies obwohl die Eigenräume jeweils eindimensional und gut separiert sind.

6. Folgerung: Der Nullstellengrad eines Polynomes ist lokal konstant.

Als ein Teilergebnis für Eigenvektoren erhält man

7. Satz: Voraussetzungen: Sei $\lambda$ ein einfacher Eigenwert von $A\in\mathbb{C}^{n\times n}$ und $x\ne0$ der zu $\lambda$ gehörige Eigenvektor. Weiter sei $E_\nu\in\mathbb{C}^{n\times n}$ beliebig aber derart, daß $\lambda(E_\nu)\to\lambda$, falls $E_\nu\to0$, wobei $\lambda(E_\nu)$ ein zu $A+E_\nu$ korrespondierender Eigenwert ist. Die $\left|E_\nu\right|$ seien so klein, daß $\lambda(E_\nu)$ ebenfalls einfacher Eigenwert ist und $A+E_\nu-\lambda(E_\nu)I$ den Rang $(n-1)$ hat, für alle $\nu$.

Behauptung: $\def\mapright#1{\mathop{\longrightarrow}\limits^{#1}}\displaystyle\lambda(E_\nu)\mapright{\nu\to\infty}\lambda$ und $\displaystyle x(E_\nu)\mapright{\nu\to\infty}x$, falls $\displaystyle E_\nu\to0$.

Beweis: Weil $\lambda$ einfacher Eigenwert ist, folgt durch Betrachtung einer Jordannormalform von $A$, daß $A-\lambda I$ den Rang $(n-1)$ hat. Somit gibt es Indizes $i$ und $j$, sodaß

($\delta_{km}$ Kronecker-Delta) Die Koeffizienten Matrix vor $x_m$ ist invertierbar. Sei o.B.d.A. angenommen $x_j=1$,

Sei nun $\lambda(E_\nu)$ der Eigenwert von $A+E_\nu$, sodaß $\lambda(E_\nu)\to\lambda$, für $E_\nu\to0$; man beachte hier die stetige Abhängigkeit nach obigen Satz. Nach der Folgerung ist die Nullstellenordnung lokal konstant. Nun ist die Matrix $A+E-\lambda(E_\nu)I$ nach Streichen der $i$-ten Zeile und $j$-ten Spalte ebenfalls eine invertierbare $(n-1)\times(n-1)$ Matrix. Somit besitzt das lineare Gleichungssytem

genau eine Lösung $x_m(E_\nu)$ ($m\ne j$). Diese eindeutig bestimmte Lösung ist eine stetige Funktion in Abhängigkeit von $E_\nu$ (Cramersche Regel).     ☐

Wenn also die Folge der Matrizen $(E_\nu)$ so beschaffen ist, daß $A+E_\nu-\lambda_\nu I$ stets den Rang $(n-1)$ hat, so überträgt sich die stetige Abhängigkeit der Eigenwerte von den Matrixelementen auf eine stetige Abhängigkeit der Eigenvektoren von den Matrixelementen. Falls $(E_\nu)$ nicht der obigen Rangeinschränkung unterliegt, so liefert der Satz keine Information.

$\tr\adj AB$ ist das Skalarprodukt für zwei quadratische Matrizen $A,B\in\mathbb{C}^{n\times n}$. Deswegen gilt: $\forall B:\tr\adj AB=0$ $\Rightarrow$ $A=0$ (Nichtausgeartetheit des Skalarproduktes/Anisotropie). Aus dem Rieszschen Darstellungssatz, Riesz, Friedrich (1880--1956), folgt die Äquivalenz: $g$ ist eine Linearform genau dann, wenn $\exists B:$ $g=\tr BA$ für alle $A$. Weiterhin gilt

2. Satz: Die folgenden beiden Aussagen sind äquivalent:

(1) $g\colon\mathbb{C}^{n\times n}\to\mathbb{C}$ ist (komplexes) Vielfaches der Spurfunktion.

(2) $g\colon\mathbb{C}^{n\times n}\to\mathbb{C}$ ist eine Linearform, also $g(\lambda A+\mu B)=\lambda g(A)+\mu g(B)$ und es gilt $g(AB)=g(BA)$, für alle $\lambda,\mu\in\mathbb{C}$ und alle $A,B\in\mathbb{C}^{n\times n}$.

Beweis: “(1)$\Rightarrow$(2)”: Dies sind einfache Rechenregeln für die Spurfunktion.

“(2)$\Rightarrow$(1)”: siehe Nicolas Bourbaki (1970)*1970+2A: "Éléments de mathématique: Algèebre", Hermann, Paris, 1970, 167+210+258S. = 635S. Für $n=1$ ist dies klar. Für $n\ge2$ sei $A=E_{ij}$ und $B=E_{jk}$ mit $i\ne k$. Hierbei ist $E_{\rho\tau}$ diejenige Matrix, welche an der Stelle $(\rho,\tau)$ eine 1 enthält und sonst nur Nullen. Für derartige Matrizen bestätigt man leicht $E_{ik} E_{j\ell} = 0$, falls $k\ne j$ und $E_{ik} E_{k\ell} = E_{i\ell}$. Damit gilt $g(E_{ik})=0$ $(i\ne k)$ und mit $A=E_{ij}$ und $B=E_{ji}$ ergibt sich $g(E_{ii})=g(E_{jj})$. Da die $E_{\rho\tau}$ eine Basis von $\mathbb{C}^{n\times n}$ bilden, folgt $g(A)=\lambda\tr A$ $\forall A$, mit geeignetem, festem $\lambda$.     ☐

Der Satz zeigt, daß es Linearformen auf der Algebra $\mathbb{C}^{n\times n}$, die gegenüber Vertauschungen invariant sind, nicht viele gibt. Durch Normierung, etwa $g(E_{11})=1$ oder $g(I)=n$, ist die Spurfunktion eindeutig bestimmt.

3. Lemma: $\forall C,D\in\mathbb{C}^{n\times n}$: $\mathop{\rm Re}\nolimits \tr CD\le{1\over2}\left(\tr C\adj C+\tr D\adj D\right)$.

Beweis: Siehe Sha, Hu-yun (1986): "Estimation of the Eigenvalues of $AB$ for $A>0$, $B>0$", Linear Algebra and Its Applications, Vol 73, January 1986, pp.147--150. Es ist $\mathop{\rm Re}\nolimits \tr CD=\mathop{\rm Re}\nolimits \sum_{i,k}c_{ik}d_{ki}={1\over2}\sum_{i,k}\bigl( c_{ik}d_{ki}+\overline{c_{ik}d_{ki}}\bigr)$, und weiter ist ${1\over2}\bigl(\tr C\adj C+\tr D\adj D\bigr)={1\over2}\sum_{i,k}\bigl( c_{ik}\overline{c_{ik}}+d_{ik}\overline{d_{ik}}\bigr)= {1\over2}\sum_{i,k}\bigl(c_{ik}\overline{c_{ik}}+d_{ki}\overline{d_{ki}}\bigr)$. In abkürzender Schreibweise sei $c_{ik}=e+fi$ und $d_{ki}=g+hi$. Damit hat man

also $c_{ik}d_{ki}+\overline{c_{ik}d_{ki}} \ge c_{ik}\overline{c_{ik}}+d_{ki}\overline{d_{ki}}$, somit ${1\over2}\sum_{i,k}\left(c_{ik}d_{ki}+\overline{c_{ik}d_{ki}}\right) \ge {1\over2}\sum_{i,k}\left(c_{ik}\overline{c_{ik}}+d_{ki}\overline{d_{ki}}\right)$.     ☐

Ist eine hermitesche Matrix $A$ invertierbar, so ist die Inverse $A^{-1}$ ebenfalls hermitesch, da $AB=I=\adj B\adj A=\adj BA=A\adj B$, also $B=\adj B$, weil eine invertierbare Matrix stets mit seiner Inversen kommutiert. Genauso gilt: die Inverse eine normalen Matrix ist normal. ($A=UD\adj U\Rightarrow A^{-1}=(UD\adj A)^{-1}=(\adj U)^{-1} D^{-1} U^{-1} =UD^{-1}\adj U$.) Daraus ergibt sich sofort: die Inverse einer positiv definiten Matrix ist wieder positiv definit. Entsprechend ist die Inverse einer negativ definiten Matrix selbst wieder negativ definit. Es zeigt sich nun, daß das Produkt zweier positiv definiter Matrizen zumindestens wieder positve Eigenwerte besitzt.

4. Satz: Voraussetzungen: Es seien $A\succ0$, $B\succ0$ zwei positiv definite (hermitesche) Matrizen aus $\mathbb{C}^{n\times n}$ mit Eigenwerten $0<\mu_1\le\cdots\le\mu_n$ bzw. $0<\nu_1\le\cdots\le\nu_n$.

Behauptung: (1) $AB$ hat nur positive reelle Eigenwerte $0<\lambda_1\le\cdots\le\lambda_n$.

(2)     $\displaystyle{{2\over\sum_i\mu_i^{-2}+\sum_i\nu_i^{-2}} \le \tr AB \le {1\over2}\left(\sum_i\mu_i^2+\sum_i\nu_i^2\right)}.$

Da alle Eigenwerte $\lambda_i$ von $AB$ echt positiv sind, gilt insbesondere als Vergröberung

Beweis: Siehe Sha, Hu-yun (1986): Zu $A$ existiert $P$ mit $A=P\adj P$. Wegen $B\succ0$ also $P^{-1}B(\adj P)^{-1}\succ0$, daher existiert eine unitäre Matrix $U$, sodaß

mit entsprechenden Eigenwerten $x_i>0$. Nun ist

Die $x_i$ sind die Eigenwerte von $AB$, da

Nach dem selben Muster setzt man $B=Q\adj Q$, $Q^{-1}A^{-1}(\adj Q)^{-1}= V\mathop{\rm diag}(y_1,\ldots,y_n)\adj V$. Also

Die $y_i$ sind zugleich Eigenwerte von $(AB)^{-1}$, denn

    ☐

5. Beispiel: Für $A={1{\mskip 3mu}0\choose0{\mskip 3mu}3}$, $B={2,{\mskip 3mu}-1\choose-1,{\mskip 3mu}2}$, $AB={2,{\mskip 3mu}-1\choose-3,{\mskip 3mu}6}$ lauten die Eigenwerte $1\le3$, $3\le5$ und $3\le5$, insbesondere muß $AB$ nicht hermitesch sein.

Während die Jordansche Normalform für jede komplexe Matrix eine Fast-Diagonalgestalt ermöglicht [genauer $(0,1)$-Bandmatrixform mit Eigenwerten als Diagonalelementen], so erlaubt das nachfolgende Lemma von Schur eine Triagonalgestalt, allerdings auf vollständig unitärer Basis. Genau wie die Jordansche Normalform, gilt die Schursche Normalform nicht für reelle Matrizen in reeller Form, falls das charakteristische Polynom über $\mathbb{R}$ nicht zerfällt. Es entstehen dann $(2\times2)$ reelle Blöcke. Doch spielen hier und im weiteren reelle Matrizen keine bedeutende Rolle.

1. Satz: Satz über eine Schursche Normalform, Schur, Issai (10.01.1875--10.01.1941). $\forall A\in\mathbb{C}^{n\times n}$: $\exists U$ unitär:

mit $\lambda_i$ Eigenwerte von $A$.

Beweis: Sei $\lambda_1$ Eigenwert von $A$ und $x_1$ normierter zugehöriger Eigenvektor $\def\iadj#1{#1^*}\iadj x1 x_1 = 1$. Es existieren linear unabhängige, paarweise unitäre (orthogonale) $y_2,\ldots,y_n\in\mathbb{C}^n$, sodaß $X_1:=(x_1,y_2,\ldots,y_n)$ unitär ist (Basisergänzungssatz, Schmidtsches Orthogonalisierungsverfahren). Schmidt, Erhard (1876--1959). Also $\iadj X1 X_1 = I$, somit $\iadj x1 y_i = 0$ $(i=2,\ldots,n)$, daher

$A_1\in\mathbb{C}^{(n-1)\times(n-1)}$ enthält außer $\lambda_1$, aufgrund der Ähnlichkeitstransformation, genau die gleichen Eigenwerte wie $A$. Man verfährt jetzt erneut wie oben: Zum Eigenwert $\lambda_2$ von $A_1$ (und auch $A$) gehört ein normierter Eigenvektor $x_2$, $A x_2 = \lambda_2 x_2$, mit $\iadj x2 x_2 = 1$. Man ergänzt wieder zu einem paarweise orthogonalen Vektorsystem $x_2,z_3,\ldots,z_n\in\mathbb{C}^{n-1}$, entsprechend

und somit

Da unitäre Matrizen eine multiplikative, nicht-abelsche Gruppe (sogar kompakte Gruppe) bilden, insbesondere abgeschlossen sind, folgt nach nochmaliger $(n-2)$-facher Wiederholung die behauptete Darstellung.     ☐

Aus dem Lemma von Schur folgt übrigens sofort der Dimensionssatz

wenn man bei nicht quadratischen Matrizen, $A$ zu einer quadratischen Matrix aus $\mathbb{C}^{(m\lor n)\times(m\lor n)}$ durch Nullauffüllung ergänzt.

2. $A$ heißt normal, falls $\adj AA=A\adj A$, also $A$ und $\adj A$ kommutieren. Beispielsweise sind hermitesche, schiefhermitesche und (komplexe) Vielfache unitärer Matrizen normal

“Kleine” und spezielle normale Matrizen lassen sich leicht klassifizieren, wie man durch elementare Rechnung leicht nachweist.

3. Lemma: (1) Normale $(2\times2)$ Matrizen sind entweder hermitesch oder komplexe Vielfache unitärer Matrizen.

(2) Eine Dreiecksmatrix ist genau dann normal, wenn sie eine Diagonalmatrix ist.

Die Art einer Diagonalisierbarkeit bestimmt eindeutig Normalität, Hermitizität und Unitärheit.

4. Satz: (1) $A$ normal $\iff$ $A$ unitär diagonalisierbar.

(2) $A$ hermitesch $\iff$ $A$ unitär reell-diagonalisierbar.

(3) $A$ schiefhermitesch $\iff$ $A$ unitär imaginär-diagonalisierbar.

(4) $A$ unitär $\iff$ $A$ unitär unimodular-diagonalisierbar.

Beweis: zu (1): “$\Rightarrow$”: Anwendung des vorstehenden Lemmas auf eine Schursche Normalform von $A$.

“$\Leftarrow$”: Mit $A=UD\adj U$, Diagonalmatrix $D$ und unitärem $U$ ($\adj UU=I$) rechnet man

zu (2): “$\Rightarrow$”: $A=\adj A$ $\Rightarrow$ $\adj xAx=\lambda\left<x,x\right> =\adj x\adj Ax=\adj{(Ax)}x=\overline\lambda\left<x,x\right>$, also $\lambda=\overline\lambda$.

“$\Leftarrow$”: $Ax_i=\lambda x_i=\overline\lambda x_i=\adj Ax_i$ $\forall i$, also stimmen $A$ und $\adj A$ auf einer Eigenbasis $x_1,\ldots,x_n$ überein, also $A=\adj A$ in jeder Basis.

zu (3): “$\Rightarrow$”: $A=-\adj A$, also $A\adj A=-A^2=\adj AA$, daher $A$ normal. $\adj xAx=\lambda \adj xx=-\adj x\adj Ax=-\overline\lambda \adj xx$, somit $\lambda=-\overline\lambda$, folglich $\lambda\in i\mathbb{P}$.

“$\Leftarrow$”: Mit $A=UD\adj U$ und Diagonalmatrix $D=-\adj D$ ist $-\adj A=-U\overline D\adj U=UD\adj U=A$.

zu (4): “$\Rightarrow$”: Wegen $\adj AA=I$ ist $A$ invertierbar. Für ein Eigenelement $(\lambda,x)$ zu $A$, also $Ax=\lambda x$, ergibt sich $\adj Ax=\overline\lambda x=A^{-1}x={1\over\lambda}x$, somit $\lambda\overline\lambda=1=\left|\lambda\right|$, für unitäre Matrizen $A$ sind sämtliche Eigenwerte daher unimodular.

“$\Leftarrow$”: Eine unimodulare Diagonalmatrix ist unitär. Unitäre Matrizen bilden eine (nicht-abelsche) Gruppe.     ☐

Wegen $AX=XD$ ist $X$ die Matrix der Rechtseigenvektoren und $X^{-1}$ wegen $X^{-1}A=DX^{-1}$ die Matrix der Linkseigenvektoren. Eine Umformulierung von (1) des Satzes ist: Das Minimalpolynom einer Matrix besteht genau dann nur aus einfachen Nullstellen, wenn die Matrix normal ist. Natürlich gilt nicht notwendig, daß diagonalähnliche Matrizen hermitesch, unitär oder normal sind, wie $B={1{\mskip 3mu}2\choose0{\mskip 3mu}3}$ zeigt ($BB^\top\ne B^\top B$). Ist $A$ hermitesch, so ist $\adj AA=A^2$ positiv semidefinit und positiv definit genau dann, wenn $A$ invertierbar ist, da alle Eigenwerte von $A^2$ nichtnegativ (bzw. positiv) sind. Der Rang einer schiefsymmetrischen Matrix ist wegen $\left|A\right|=(-1)^n\left|A\right|$, immer gerade. Dies hätte man auch mit Hilfe von (3) erkennen können, da die Determinante einer Diagonalmatrix das Produkt der Diagonalelemente ist.

Während die Schursche Normalform eine beliebige Matrix unitär zu triangulieren vermochte, so kann man sogar jede beliebige Matrix $A$ “unitär-diagonalisieren”, wenn man darauf verzichtet auf beiden Seiten der Matrix $A$ die gleiche unitäre Matrix $U$ bzw. $\adj U$ zu verlangen.

5. Proposition: $\forall A\in\mathbb{C}^{n\times n}$: $\exists U,V$ unitär: $A=UDV$, mit $D=\mathop{\rm diag}\sqrt{\lambda_i}$, mit $\lambda_i$ Eigenwerte von $\adj AA$.

Beweis: Die Matrix $\adj AA$ ist hermitesch, also $\adj AA = W\hat DW^\top$, mit unitärem $W$ und reeller Diagonalmatrix $\hat D=\mathop{\rm diag}\lambda_i$. Es ist

Setze $D=\mathop{\rm diag}\sqrt{\lambda_i}$. Dann ist $D^{-1} \adj W \adj A AWD^{-1}=I$, also $U:=AWD^{-1}$ unitär. $V:=W^{-1}$ ist ebenfalls unitär und es gilt $UDV=AWD^{-1}DW^{-1}=A$.     ☐

Zur Notation siehe Das äußere Produkt und Determinanten.

Für positiv definite (hermitesche) Matrizen erkennt man auch gleich die Existenz einer beliebigen Wurzel, also $\root r \of A$. Insbesondere für eine reelle symmetrische Matrix $A$ mit lauter nicht-negativen Eigenwerten ($\Longleftrightarrow$ positiv semidefinit) gilt: $\exists Q:$ $QQ=A$. Ist $A$ nicht quadratisch, so kann man durch Ergänzen von Nullspalten oder Nullzeilen quadratische Form erreichen und man erhält

6. Satz: Singulärwertzerlegung. $\forall A\in\mathbb{C}^{m\times n}$: $\exists U\in\mathbb{C}^{m\times m}$ unitär, $V\in\mathbb{C}^{n\times n}$ unitär: $A=UDV$, mit $D\in\mathbb{C}^{m\times n}$: $D=\mathop{\rm row}(\mathop{\rm diag}\sqrt{\lambda_i},0) \lor D=\mathop{\rm col}(\mathop{\rm diag}\sqrt{\lambda_i},0)$, mit $\lambda_i$ Eigenwerte von $\adj AA$.

Die Quadratwurzeln der Eigenwerte von $\adj AA$ heißen singuläre Werte, die Zerlegung $A=UDV$ (w.o.) eine Singulärwertzerlegung. An ihr liest man die Pseudoinverse unmittelbar ab: $A^+=UD^+V$, wobei $D^+$ aus $D$ entsteht, indem man alle Nichtnull-Werte invertiert und die Nullen belässt.

7. Satz: Hurwitz-Kriterium, Adolf Hurwitz (1859--1919). Voraussetzungen: $A$ sei hermitesch und $A\succ0$, $A\succeq0$, $A\prec0$, $A\preceq0$ kennzeichne positive, positive Semi-, negative, negative Semidefinitheit. $r$ laufe stets von 1 bis $n$ und der Multiindex $i=(i_1,\ldots,i_r)$ sei stets in natürlicher Reihenfolge angeordnet, also $i_1<\cdots<i_r$. Genauso die Multiindizes $k$ und $\ell$.

Behauptung:

Beweis: Sei $A=XDX^{-1}$ mit othogonalem $X$, also $X^{-1}=X^\top$, und $D$ sei die Diagonalmatrix der Eigenwerte. Es ist

also können nicht sämtliche $X_i^\ell$ verschwinden. Aus $A=XDX^{-1}=X(XD)^\top$ folgt

da $D_k^\ell=0$, für $k\ne\ell$. An dieser Darstellung von $A_i^i$ als Summe von Quadraten liest man nun alles ab. Für den Fall einer hermiteschen Matrix führt man die Überlegungen genauso mit unitärem $X$ $(\adj X=X^{-1})$ unter Beachtung von $\overline{\det C}=\det\overline C$.     ☐

8. Bemerkung: Kann man eine Matrix nicht unitär diagonalisieren oder treten nicht-lineare Elementarteiler auf, so hat man nicht mehr die Darstellung als Summe von Quadraten (Summe von Beträgen) und man kann dann nicht mehr so einfach entscheiden, ob alle Eigenwerte positiv oder dergleichen sind. Beispielsweise für die Begleitmatrix zu $(\lambda-1)(\lambda-2)(\lambda-3)= \lambda^3-6\lambda^2+11\lambda-6$ verschwinden die ersten beiden Hauptminoren. Ist man nur an dem Vorzeichenverhalten einer Form $\adj xAx$ interessiert, so kann man das Hurwitz-Kriterium anwenden auf die hermitesche Matrix ${1\over2}(\adj A + A)$.

Es gilt zwar $A\succeq0{\mskip 5mu}\Rightarrow{\mskip 5mu}A_{1\ldots r}^{1\ldots r}\ge0 \land a_{ii}\ge0$, jedoch die Rückrichtung stimmt nicht, wie man erkennt anhand der Matrix

mit Eigenwerten 0 (zweifach), $(+1)$ und $(-1)$.

2. Beispiel: $f(x_1,x_2)=x_1^2+x_2^2$ oder $f(x_1,x_2)=x_1^3+x_2^3$ sind symmetrische Polynome, da Vertauschungen der Rollen von $x_1$ gegen $x_2$ nichts ändert.

3. Besonders wichtig sind die sogenannten elementarsymmetrischen Polynome

Das Polynom $s_i$ heißt $i$-tes elementarsymmetrisches Polynom. Die $s_i$ üben gewisse Basisfunktionen im Raum der symmetrischen Polynome aus.

4. Satz: (Hauptsatz über elementarsymmetrische Polynome) Zu jedem symmetrischen $n$-stelligen Polynom $f(x_1,\ldots,x_n)$ existiert genau ein Polynom $F(x_1,\ldots,x_n)$, sodaß $f(x_1,\ldots,x_n)=F(s_1,\ldots,s_n)$, $\forall x_1,\ldots,x_n$.

Beweis: Siehe László Rédei (1967), Algebra I, László Rédei (15 November 1900 – 21 November 1980). Existenz: $f(x_1,\ldots,x_n)$ sei bzgl. der Potenzen lexikographisch geordnet und es sei $q=ax_1^{k_1}\ldots x_n^{k_n}$ der lexikographisch letzte Term, $k_1\ge\cdots\ge k_n$. Betrachtet man

so erkennt man, daß dieser Ausdruck als führenden Koeffizienten den Term

besitzt, welcher offensichtlich gleich $q$ ist. Also enthält

nur Terme, die lexikographisch vor $q$ kommen, man beachte $(*)$. $f_1(x_1,\ldots,x_n)$ ist symmetrisch und man wiederholt das Verfahren, welches irgendwann abbricht, da es nur endlich viele Terme der Form $b x_1^{\ell_1} \ldots x_n^{\ell_n}$ ($\ell_1\ge\cdots\ge\ell_n$) gibt.

Eindeutigkeit: Ist $f=F_1=F_2$, so ist $F_1-F_2$ identisch gleich Null, also das Nullpolynom.     ☐

5. Bemerkung: $f$ ist symmetrisch, $F$ ist i.d.R. nicht symmetrisch, wie $x_1^2+x-2^2=s_1^2-2s_2$, oder $x_1^3+x_2^3=s_1^3-3s_1s_2$ zeigen; $s_1=x_1+x_2$, $s_2=x_1x_2$. Die Symmetrie von $f$ verlagert sich also in die Symmetrie der Basen der Polynome.

6. Definition: Es seien $f(x)=a_0x^m+\cdots+a_m$ und $g(x)=b_0x^n+\cdots+b_n$ zwei Polynome. Dann nennt man die Determinante

die Resultante von $f(x)$ und $g(x)$, für $m,n\ge1$, $a_0\ne0$, $b_0\ne0$.

7. Es sei $u$ eine ^{gemeinsame Nullstelle}, also $f(u)=0$ und $g(u)=0$. Dann gilt

Dieses homogene Gleichungssystem hat den nicht-trivialen Lösungsvektor

Daher: falls eine gemeinsame Nullstelle $u$ vorliegt, so ist $R=0$. Es gilt sogar: wenn $R=0$, dann liegt eine gemeinsame Nullstelle vor.

8. Lemma: Es sei $d(x)=\mathop{\rm ggT}(f(x),g(x))$, wobei $\deg f(x)=m\ge1$, $\deg g(x)=n\ge1$. Dann gilt

Beweis: “$\Rightarrow$”: Offensichtlich ist $f(x)=d(x)f_1(x)$ und $g(x)=-d(x)g_1(x)$ mit zwei Polynomen $f_1(x)$ und $g_1(x)$ mit allen oben gewünschten Eigenschaften.

“$\Leftarrow$”: siehe Rédei, L., Rédei (1967).     ☐

9. Satz: Für die Resultante $R$ gilt: $f(x)F(x)+g(x)G(x)=R$, wobei $\deg F(x)<n$, $\deg G(x)<m$.

Beweis: Addiere für $j=1,2,\ldots,m+n-1$ die $j$-te Spalte multipliziert mit $x^{m+n-j}$ zur letzten ($m$-ten) Spalte von $R$, welche zu

wird. Entwickeln nach der letzten Spalte und dann Ausklammern von $f(x)$ bzw. $g(x)$, liefert die angegebene Darstellung.     ☐

Mit Hilfe des Lemmas folgt, daß $R$ genau dann verschwindet, falls $f(x)$ und $g(x)$ einen gemeinsamen Faktor besitzen.

10. Satz: Ist $f(x)=a_0(x-y_1)\ldots(x-y_m)$ und $g(x)=b_0(x-z_1)\ldots(x-z_n)$, $m,n\ge1$, so hat man für die Resultante die drei Darstellungen

11. Die Newton-Identitäten. Newton, Sir Isaac (1643--1727), Urbain Le Verrier (1811--1877). Zur Matrix $A\in\mathbb{C}^{n\times n}$ mit den Eigenwerten $\lambda_i$, sei $p_k=\sum\lambda_i^k=\tr A^k$ und das charakteristische Polynom sei $f(x)=x^n+c_1x^{n-1}+\cdots+c_n=(x-\lambda_1)\ldots(x-\lambda_n)$. Nach der Produktregel ist

und durch Polynomdivision verifiziert man

Summation liefert $f'(x)=nx^{n-1}+(p_1+nc_1)x^{n-2}+(p_2+c_1p_1+nc_2)x^{n-3} +\cdots+(p_{n-1}+c_1p_{n-2}+\cdots+nc_{n-1})$. Koeffizientenvergleich mit $f'(x)=nx^{n-1}+(n-2)x^{n-2}+\cdots+2c_{n-2}+c_{n-1}$ ergibt

und $\lambda_1^k f(\lambda_1) + \cdots + \lambda_n^k f(\lambda_n) = 0$ ergibt

Es gibt eine Fülle von Möglichkeiten Determinanten einzuführen. Ein Weg ist, über das äußere Produkt zu gehen. Die folgenden Ausführungen erfolgen in enger Anlehnung an das Buch Matrizenrechnung von Wolfgang Gröbner (1966).

Es sei $K$ ein beliebiger Körper. Jeden Vektor eines $n$-dimensionalen Vektorraumes über $K$ kann man darstellen als Linearkombination der Basisvektoren (im weiteren Einheiten genannt)

Das äußere Produkt (Zeichen $\land$) wird zunächst für die Einheiten erklärt:

insbesondere

Im $\mathbb{C}^3$ kann dem äußeren Produkt eine anschauliche Bedeutung beigelegt werden. Identifiziert man

liegen also die Einheiten höherer Stufe wieder im ursprünglichen Vektorraume, so gilt in diesem Falle für das äußere Produkt, welches man auch vektorielles Produkt nennt (Schreibweise: $a\times b$)

Die Verallgemeinerung auf das äußere Produkt von Vektoren höherer Stufe geschieht nach der Regel

entsprechend

Unter einem Vektor $k$-ter Stufe versteht man allgemein eine Linearform in den $n\choose k$ Einheiten $k$-ter Stufe. Summe, Differenz und inneres Produkt solcher Vektoren sind nach den üblichen Regeln der Algebra erklärt. Man darf also Vektoren derselben Stufe beliebig mit Skalaren multiplizieren und addieren.

1. Satz: Sind $a_1,a_2,\ldots,a_k$ Vektoren 1-ter Stufe, so ändert sich das äußere Produkt nicht, wenn man zu einem dieser Vektoren, etwa $a_1$, ein lineares Kompositum der übrigen Vektoren addiert:

Der Beweis ergibt sich durch direktes Ausmultiplizieren der rechten Seite. Bis auf den ersten Summand verschwinden alle weiteren Summanden, da bei allen anderen Produkten, außer dem ersten, stets zwei gleiche Vektoren miteinander äußerlich multipliziert werden.

$ \def\multisub#1#2{{\textstyle\mskip-3mu{\scriptstyle1\atop\scriptstyle#2_1}{\scriptstyle2\atop\scriptstyle#2_2}{\scriptstyle\ldots\atop\scriptstyle\ldots}{\scriptstyle#1\atop\scriptstyle#2_#1}}} \def\multisup#1#2{{\textstyle\mskip-3mu{\scriptstyle#2_1\atop\scriptstyle1}{\scriptstyle#2_2\atop\scriptstyle2}{\scriptstyle\ldots\atop\scriptstyle\ldots}{\scriptstyle#2_{#1}\atop\scriptstyle#1}}} \def\multisubsup#1#2#3{{\textstyle\mskip-3mu{\scriptstyle#3_1\atop\scriptstyle#2_1}{\scriptstyle#3_2\atop\scriptstyle#2_2}{\scriptstyle\ldots\atop\scriptstyle\ldots}{\scriptstyle#3_{#1}\atop\scriptstyle#2_{#1}}}} \def\diag{\mathop{\rm diag}} \def\tridiag{\mathop{\rm tridiag}} \def\col{\mathop{\rm col}} \def\row{\mathop{\rm row}} \def\dcol{\mathop{\rm col\vphantom {dg}}} \def\drow{\mathop{\rm row\vphantom {dg}}} \def\rank{\mathop{\rm rank}} \def\grad{\mathop{\rm grad}} \def\adj#1{#1^*} \def\iadj#1{#1^*} \def\tr{\mathop{\rm tr}} \def\mapright#1{\mathop{\longrightarrow}\limits^{#1}} \def\fracstrut{} $

2. Definition einer Determinante

1. Während das Produkt von $k$ Vektoren erster Stufe insgesamt $n\choose k$ Komponenten hat, so hat insbesondere das Produkt von $n$ Vektoren nur noch eine Komponente. Diese Komponente heißt eine Determinante.

Alle Glieder, welche das Produkt von zwei Einheiten mit gleichem Index enthalten, verschwinden. Für die Determinante schreibt man

wobei $\pm=(-1)^{i_1+\ldots+i_n}$.

2. Beispiel: $n=2$: Es ist

$n=3$: Hier berechnet man $\left|A\right|$ zu

Aufgrund der hohen Anzahl der Summanden, nämlich $n!$ (jeder Summand ist $n$-faches Produkt), benutzt man zur eigentlichen Berechnung von Determinanten i.d.R. ab $n\ge3$ Determinantenregeln.

3. Mit Hilfe von Determinanten lassen sich auch die Produkte von weniger als $n$ Vektoren genauer ausschreiben. Das Produkt von

ist

wo $\left|a_i,b_k\right|=\left|{a_i,a_k\atop b_i,b_k}\right|$ bedeutet. Für einen weiteren dritten Vektor

gilt

mit

3. Eigenschaften einer Determinante

1. Bemerkung: Es gelten:

(1) Die Determinante einer quadratischen Matrix $A=(a_{ik})$

ist eine homogene, lineare Funktion der Elemente einer jeden Zeile und einer jeden Spalte.

(2) Eine Determinante ändert ihr Vorzeichen, wenn man zwei Zeilen oder zwei Spalten miteinander vertauscht.

(3) Eigenschaft (1) und (2) sind für eine Determinante charakteristisch. Bis auf eine Normierung durch einen Skalar, gibt es keine weiteren multilinearen, alternierenden Formen dieser Art.

2. Satz: Ist $\Phi\colon \mathop{\rm GL}(n,K)\rightarrow K^\times$ eine Abbildung mit $\Phi(AB)=\Phi(A)\Phi(B)$, für alle $A,B\in \mathop{\rm GL}(n,K)$, dann gibt es $\varphi\colon K^\times\rightarrow K^\times$, mit $\varphi(\alpha\beta)=\varphi(\alpha)\varphi(\beta)$, für alle $\alpha,\beta\in K^\times$ und es ist $\Phi(A)=\varphi(\det A)$, für alle $A\in \mathop{\rm GL}(n,K)$.

Beweis: siehe Max Koecher (1985), S.119.     ☐

3. Siehe Wolfgang Gröbner (1966). Es seien $A=(a_{ik})$ und $B=(b_{ik})$ zwei $n$-zeilige quadratische Matrizen, $C=(c_{ik})=AB$ sei die Produktmatrix. Es ist $c_{ik}=\sum a_{ij}b_{jk}$. Die Zeilenvektoren von $C$ sind

wobei $b_j=\sum b_{jk}\varepsilon_k$ die Zeilenvektoren von $B$ bedeuten. Nun ist

Durch Vergleich der ersten und letzten Zeile sieht man $\left|C\right| = \left|A\right| \left|B\right|$, also $\left|AB\right| = \left|A\right| \left|B\right|$.

4. Der oben abgeleitete Determinantenproduktsatz, wie auch letztlich das kanonische Skalarprodukt, ist ein Spezialfall der Formel von Cauchy/Binet, auch Determinantenproduktsatz für rechteckige Matrizen genannt. Cauchy, Augustin Louis (1789--1857) Binet, Jacques Philipe Marie (1786--1856)

Es sei $A=(a_{ik})$ eine $m\times n$-Matrix, $B=(b_{k\ell})$ eine $n\times s$-Matrix. Ihr Produkt $AB=C=(c_{i\ell})$ ist eine $m\times s$-Matrix mit den Elementen

Jeder Zeilenvektor $c_i$ von $C$ ist

mit den Zeilenvektoren $b_k$ der Matrix $B$ zu $b_k = \sum_\ell b_{k\ell}\varepsilon_\ell$. Nun ist

Durch Vergleich der Koeffizienten vor $\varepsilon_{\ell_1\ell_2\ldots\ell_m}$ findet man

5. Diese Formel kann man noch etwas verallgemeinern, wenn man statt $c_1\land c_2\land\cdots\land c_m$ das äußere Produkt von irgend welchen $r$ Zeilenvektoren $c_{i_1}\land c_{i_2}\land\cdots\land c_{i_r}$ auf genau die gleiche Weise auswertet:

In Worten: Jede $r$-zeilige Unterdeterminante der Produktmatrix ist darstellbar als Summe von Produkten $r$-reihiger Unterdeterminanten aus $A$ und $B$, die so kombiniert sind, daß jeweils die Spaltenindizes der ersten mit den Spaltenindizes der zweiten übereinstimmen, während die Zeilenindizes der ersten und die Spaltenindizes der zweiten mit den entsprechenden Indizes in der Produktmatrix übereinstimmen.

6. Man untersucht nun Spezialfälle der obigen Formel. Ist $r=m=s$ (also $C$ quadratisch), so hat man

Ist $n<m$, so gilt $\left|C\right|=0$. Setzt man $B=A^\top$, dann ist einerseits

mit den Zeilenvektoren $a_i$ der Matrix $A$. Andererseits ist $B\multisub mk = A\multisup mk$ und zusammen mit

ergibt sich

Eine Anwendung dieser Formel liefert mit $m=2$ und $A={a_1a_2\ldots a_m\choose b_1b_2\ldots b_m}$ die Formel von Lagrange, Lagrange, Joseph Louis (1736--1813)

Sind die $a_1,a_2,\ldots,a_m$ paarweise othogonal, also

so folgt unmittelbar aus $(*)$

In Worten: Der Betrag des äußeren Produktes von paarweise othogonalen Vektoren ist gleich dem Produkt ihrer Beträge. Dies ist die anschauliche Bedeutung des Spatproduktes. Das Volumen, welches von paarweise othogonalen Vektoren aufgespannt wird, ist gleich dem Produkt der Seitenlängen.

4. Der Laplacesche Entwicklungssatz

1. Siehe Wolfgang Gröbner (1966). Es seien $(i_1,\ldots,i_r)$ und $(i'_1,\ldots,i'_s)$ zueinander komplementäre Anordnungen, also $r+s=n$,

und $(i_1,\ldots,i_r, i'_1,\ldots,i'_s)$ ist eine Permutation von $(1,2,\ldots,n)$, also $s=n-r$. Komplementär geordnete Anordnungen $(i_1,\ldots,i_r, i'_1,\ldots,i'_s)$ brauchen

Transpositionen um die natürliche Anordnung $(1,\ldots,n)$ zu erreichen.

Durch Zusammenfassen von Zeilenvektoren von $A$ rechnet man

mit

und es wurde benutzt

oder allgemeiner

Zur Schreibvereinfachung definiert man das algebraische Komplement $\alpha\multisubsup rik$ zu

Statt algebraisches Komplement sagt man auch Adjunkte der Unterdeterminante $A\multisubsup rik$. Mit dieser Notation erhält man den

2. Satz: Allgemeiner Laplacescher Entwicklungssatz. Laplace, Pierre Simon (1749--1827). Man erhält den Wert der $n$-reihigen Determinante $\left|A\right|$, entwickelt nach nach den Zeilen $i_1,i_2,\ldots,i_r$, ($1\le i_1<i_2<\ldots<i_r\le n$), indem man alle $r$-reihigen Unterdeterminanten dieser $r$ Zeilen bildet, sie mit ihren algebraischen Komplementen multipliziert und dann addiert:

Die Summen sind über alle $n\choose r$ Kombination $(k_1,\ldots,k_r)$, bzw. $(i_1,\ldots,i_r)$ zu erstrecken.

3. Nach dem oben hergeleiteten gilt offensichtlich leicht allgemeiner

Für $r=1$ erhält man das übliche Entwickeln nach einer Zeile oder Spalte insbesondere

Wie üblich für $\xi_i^j$: $i$ Zeilenindex, $j$ Spaltenindex, für $(\alpha)$ also transponierte Matrix. Damit liegt eine explizite Beschreibung der inversen Matrix vor, also $\alpha_i^j / \left|A\right|$ für das $(j,i)$-Element der Inversen.

4. Satz: (Minor Inverser) Es sei $B=A^{-1}$, wobei $A$ invertierbar sei. Jeden Minor der Inversen kann man ausdrücken durch die Adjunkte der Ursprungsmatrix:

Beweis: Nach Cauchy/Binet ist

Nach dem Laplaceschen Entwicklungssatz ist

Es sind $(A\multisubsup rik)_k$ und $(\alpha\multisubsup rk\ell)_k$ beides Matrizen mit ${n\choose r}={n\choose n-r}$ Zeilen und Spalten. Nach $(*)$ ist $(B\multisubsup rk\ell)_k$ offensichtlich Inverse, genauso aber auch $\alpha\multisubsup rik / \left|A\right|$. Da Inversen eindeutig bestimmt sind, folgt Gleichheit.     ☐

5. Beispiel: Sowohl für Cauchy/Binet, Laplaceschen Entwicklungssatz als auch Minoren Inverser. Es seien

(1) Die Determinante von $A$ berechnet man z.B. so:

Hierbei muß nicht wie bei dem Laplaceschen Entwicklungssatz nach einer Zeile (oder Spalte) immer ein Vorzeichenwechsel von einem Term zum nächsten stattfinden.

(2) Es ist $AB=:C=I$. Also nach Cauchy/Binet wie oben $4\choose2$ Summanden

(3) Für den Minor $B_{12}^{34}$ der Inversen $B$ rechnet man

genauso

5. Weitere Folgerungen aus dem Satz von Cauchy/Binet

Aufgrund seiner großen Bedeutung sei für den Determinantenmultiplikationssatz von Cauchy/Binet ein weiterer Beweis angegeben, der nicht Bezug nimmt auf das äußere Produkt.

1. Satz: (Satz von Cauchy/Binet) Es sei $C=AB$. Dann gilt $C_{1\ldots r}^{1\ldots r} = \sum_i A\multisup ri B\multisub ri$.

Beweis: (für Cauchy/Binet) siehe Gantmacher, Felix R. (1908--1964), Gantmacher (1986). Man rechnet

Unter allen $n^r$ Summanden sind nur $n(n-1)\ldots(n-r+1)={n\choose r}r!$ Summanden von Interesse, bei denen die Minoren $A\multisup ri$ nicht zwei, drei, $\ldots$, $r$ gleiche Spalten enthalten. Von den ${n\choose r}r!$ sind aber wiederum nur $n\choose r$ echt verschieden, die restlichen sind nichts anderes als Vertauschungen zweier Spalten. Also rechnet man weiter

    ☐

2. Der Satz von Cauchy/Binet liest sich für mehr als zwei Matrizen wie folgt

3. Es sei

die ^{$p$-te assoziierte Matrix} zu $A$.

Die Anordnungen seien in lexikographischer Reihenfolge durchlaufen. Beispielsweise erhält man für eine $4\times4$ Matrix $A$ die $6\times6$ Matrix

Eine Umformulierung des Satzes von Cauchy/Binet ist: Aus $C=AB$ folgt ${\cal C}_p = {\cal A}_p {\cal B}_p$, $p=1,2,\ldots,n$. Insbesondere: Aus $B=A^{-1}$ folgt ${\cal B}_p = {\cal A}_p^{-1}$, $p=1,2,\ldots,n$.

4. Satz: Es sei $A=(a_{ij})_{i,j=1}^n$ und

Dann gilt

Beweis: Siehe Felix Ruvimovich Gantmacher (1908--1964), Gantmacher, 1986, "Matrizentheorie", §3.7. Die Potenz $(-\lambda)^{n-p}$ tritt in denjenigen Termen von $\left|A-\lambda I\right|$ auf, die

enthalten. Anwendung des allgemeinen Laplaceschen Entwicklungssatzes entwickelt nach $(k_1,\ldots,k_{n-p})$ liefert

wobei $(i_1,\ldots,i_p)$ die zu $(k_1,\ldots,k_{n-p})$ komplementäre Anordnung ist, also $\{k_1,\ldots,k_{n-p},{\mskip 3mu}i_1,\ldots,i_p\} = \{1,\ldots,n\}$. Bildet man alle möglichen ${n\choose n-p}={n\choose p}$ Kombinationen von $n-p$ Elementen $k_1<\cdots<k_{n-p}$, die besagte Diagonalelemente enthalten, so erhält man genau $n\choose p$ Minoren als Summe, die die Koeffizienten vor $(-\lambda)^{n-p}$ ausmachen.     ☐

5. Beispiel: zu $c_{n-k}=\sum_i A\multisubsup kii$ im Falle $n=3$. Für

erhält man

6. Eine direkte Folge ist der Vietascher Wurzelsatz. Vieta (siehe Viète), Fran\c cois Viéte (1540--1603). Entweder benutzt man eine Jordansche Normalform ($A=XJX^{-1}$) oder eine Schursche Normalform ($A=UT\adj U$). Das charakteristische Polynom bleibt bei einer Ähnlichkeitstransformation invariant, daher

Es wird nicht behauptet, daß i.a. $\lambda_{i_1}\ldots\lambda_{i_k}=A\multisubsup kii$. Beispielsweise für eine invertierbare Begleitmatrix $C_1\in\mathbb{C}^{n\times n}$ gilt $\lambda_1\ldots\lambda_k\ne (C_1)_{1\ldots k}^{1\ldots k}=0$, für $k<n$.

7. Satz: Bei zwei diagonalähnlichen Matrizen $A,B\in\mathbb{C}^{n\times n}$ mögen sämtliche Eigenvektoren gleich sein. Dann gilt: $AB=BA$, d.h. $A$ und $B$ kommutieren.

Beweis: $X$ enthalte sämtliche Eigenvektoren, $D_1=\mathop{\rm diag}\lambda_i$, $D_2=\mathop{\rm diag}\mu_i$, $A=XD_1X^{-1}$, $B=XD_2X^{-1}$. Also $AB=XD_1X^{-1}XD_2X^{-1}=XD_1D_2X^{-1}=XD_2X^{-1}XD_1X^{-1}=BA$.     ☐

8. Satz: Es gelte $AB=BA$. Dann gilt: $A$ und $B$ haben gemeinsame Eigenvektoren.

Beweis: Siehe James H. Wilkinson (1919--1986), Wilkinson (1965) "The Algebraic Eigenvalue Problem", siehe Gantmacher, Felix R. (1908--1964), Gantmacher (1986) "Matrizentheorie", §9.10. Für ein beliebiges Eigenelement $(\lambda,x)$ von $A$ gilt $AB^kx=\lambda B^kx$, $k=0,1,2,\ldots$ In der Vektorfolge $x$, $Bx$, $B^2x$, $\ldots$ seien die ersten $p$ Vektoren linear unabhängig, also der $(p+1)$-te Vektor $B^px$ ist eine Linearkombination der $p$ vorhergehenden. Der Unterraum ${\cal S}:=\left<x,Bx,\ldots,B^{p-1}x\right>$ ist bzgl. $B$ invariant, also $B{\cal S}\subseteq\cal S$, daher existiert ein Eigenvektor $y\in\cal S$ für $B|\cal S$, damit auch für $B$. $AB^kx=\lambda B^kx$ zeigt, daß $x$, $Bx$, $B^2x$, $\ldots$ Eigenvektoren zum selben Eigenwert $\lambda$ sind. Insbesondere jede Linearkombination dieser Vektoren ist Eigenvektor von $A$, also auch $y\in\cal S$.     ☐

9. Bemerkung: Beim Beweis war wesentlich, daß $B$ einen Eigenvektor besitzt. Bei komplexen Matrizen ist dies aufgrund des Fundamentalsatzes der Algebra klar. Bei reellen Matrizen (über $\mathbb{R}$) braucht kein reeller Eigenwert zu existieren und somit auch kein Eigenvektor. Die Drehungsmatrix $T={\cos\alpha{\mskip 3mu}-\sin\alpha\choose\sin\alpha{\mskip 3mu}\cos\alpha}$ hat für geeignetes $\alpha$ keinen reellen Eigenwert. Anschaulich ist dies ersichtlich, weil nicht jede Drehung streckt, staucht oder Fixpunkte hat. Algebraisch ist dies ersichtlich, weil $\det(A-\lambda I)= \lambda^2-2\lambda\cos\alpha+1=(\lambda-\cos\alpha)^2+(1-\cos^2\alpha)$ nicht für jedes $\alpha$ über $\mathbb{R}$ zerfällt. Sehr wohl hat $T$ jedoch in $\mathbb{C}$ die beiden Eigenwerte $\lambda=\pm i\sin\alpha$. Der Satz bleibt richtig, wenn man im Reellen zusätzlich fordert, daß $B$ nur reelle Eigenwerte hat, z.B. falls $B$ hermitesch ist. Der Satz bleibt auch richtig, wenn man voraussetzt: $A$ und $B$ enthalten $1\times1$ Jordanblöcke (lineare Elementarteiler).

10. Satz: Es sei $A\in\mathbb{C}^{m\times n}$ und $B\in\mathbb{C}^{n\times m}$. Sind beide Matrizen quadratisch ($m=n$) so haben $AB$ und $BA$ dasselbe charakteristische Polynom und damit die gleichen Eigenwerte samt Multiplizitäten. Im Falle $m\ne n$ haben $AB$ und $BA$ die gleichen Eigenwerte samt Multiplizitäten außer, daß das Produkt der höheren Ordnung $\left|m-n\right|$ zusätzliche Nullen im Spektrum hat.

Beweis: siehe Wilkinson, J.H., Wilkinson (1965). Es ist

und

Also

Für $\mu=0$ beachte man $\left|AB\right|=\left|BA\right|$. Der Fall $m\ne n$ wird genauso bewiesen.     ☐

Den Beweis hätte man auch direkt über die Koeffizienten des charakteristischen Polynomes führen können. Nämlich mit

berechnet man die $c_i$ und $d_i$ zu

Vertauschung von $i$ und $\ell$ in einer der beiden Summen zeigt Gleichheit, einmal abgesehen von möglichen “Stellenverschiebungen”. Also $c_{n-k+\ell}=d_{n-k}$, was aber gerade Multiplikation des charakteristischen Polynomes mit $\lambda^\ell$ bedeutet.

Für den Fall $\ell=1$ gilt häufig $L(\lambda)=I\lambda-A$.

1. Vektorräume und lineare Abbildungen

1. Definition: (1) Ein Vektor $a_1$ heißt linear-abhängig von den Vektoren $a_2,\ldots,a_n$ genau dann, wenn $a_1$ lineares Komposituum dieser $(n-1)$ Vektoren ist, also

In Zeichen: $a_1{\mathrel{\underline\perp}}(a_2,\ldots,a_n)$. Die $n$ Vektoren $a_1,\ldots,a_n$ heißen dann ebenfalls linear-abhängig, in Zeichen ${\mathrel{\underline\perp}}(a_1,\ldots,a_n)$.

(2) $a_1$ ist von $a_2,\ldots,a_n$ linear-unabhängig genau dann, wenn $a_1$ von $a_2,\ldots,a_n$ nicht linear-abhängig ist, also $a_1$ nicht als lineares Komposituum der anderen $(n-1)$ Vektoren darstellbar ist. In Zeichen $a_1{\mathrel{\underline{\not\perp}}}(a_2,\ldots,a_n)$.

(3) Ist $a_1{\mathrel{\underline{\not\perp}}} a_2,\ldots,a_n$ linear-unabhängig, $a_2{\mathrel{\underline{\not\perp}}} a_1,a_3,\ldots,a_n$, $\ldots$, $a_n{\mathrel{\underline{\not\perp}}} a_1,\ldots,a_{n-1}$, so heißt die Vektorfamilie $(a_1,\ldots,a_n)$ linear-unabhängig (schlechthin), in Zeichen ${\mathrel{\underline{\not\perp}}}(a_1,\ldots,a_n)$.

Ist $a_1$ von $a_2,\ldots,a_n$ linear-abhängig, so ist $a_1$ in gewisser Hinsicht überflüssig, da $a_1$ ja aus den anderen Vektoren zusammengesetzt werden kann. Liegen $a_2,\ldots,a_n$ in einer Ebene, so liegt damit natürlich auch $a_1$ in der gleichen Ebene. Man beachte, daß eine (zweistellige) Relation zwischen einem Vektor und $(n-1)$ anderen Vektoren definiert wurde und eine Eigenschaft zwischen $n$ Vektoren, also eine $n$-stellige Relation.

2. Definition und Eigenschaften von Standard-Tripeln

Gegeben sei das monische Matrixpolynom

Die Vektorfamilie $x_0,\ldots,x_k$, mit $x_0\ne\bf0$, $x_i\in\mathbb{C}^{n\times1}$, heißt rechte Jordan-Kette (oder auch rechte Keldysh-Kette), Keldysh, M.V., der Länge $(k+1)$ für das Matrixpolynom $L(\lambda)$ zum Eigenwert $\lambda_0$ genau dann, wenn

Die hierbei links auftretende Matrix $\mathbb{P}$ ist natürlich nicht invertierbar, weil $L(\lambda_0)$ nicht invertierbar ist.

Die Vektorfamilie $y_0,\ldots,y_k$, mit $y_0\ne\bf0^\top$, $y_i\in\mathbb{C}^{1\times n}$, heißt linke Jordan-Kette der Länge $(k+1)$ für das Matrixpolynom $L(\lambda)$ zum Eigenwert $\lambda_0$ genau dann, wenn

d.h. also, wenn $y_0^\top,\ldots,y_k^\top$ eine rechte Jordan-Kette ist.

Das Paar von Matrizen von Matrizen $(X,T)$, mit $X$ von der Größe $n\times n\ell$ und $T$ der Größe $n\ell\times n\ell$, heißt Standard-Paar genau dann, wenn gilt:

1. $\mathop{\rm col}(XT^i)_{i=0}^{\ell-1}$ ist invertierbar,
2. $\sum_{i=0}^\ell A_iXT^i=\bf 0$.

Ist $T$ eine Jordan-Matrix, so heißt das Paar $(X,T)$ auch Jordan-Paar.

Das Matrizentripel $(X,T,Y)$, mit $X$ der Größe $n\times n\ell$, $T$ der Größe $n\ell\times n\ell$ und $Y$ der Größe $n\ell\times n$, heißt Standard-Tripel des Matrixpolynoms $L(\lambda)$ genau dann, wenn gilt:

1. $(X,T)$ ist Standard-Paar,

Ist $T$ wiederum eine Jordan-Matrix, so heißt $(X,T,Y)$ auch Jordan-Tripel.

Ist $(X,T,Y)$ Jordan-Tripel, dann sind die Spalten von $X$ rechte Jordanketten (Keldysh-Ketten), Keldysh, M.V., von $L(\lambda)$, falls $X$ derart in Blöcke aufgespalten wird, sodaß diese konsistent mit der Unterteilung der Jordan-Matrix $J$ sind. Hierzu dual sind die Zeilen von $Y$ Links-Jordan-Ketten zu $L(\lambda)$. Zusammenfassend entnimmt man die nötigen Dimensionen der Matrizen $X$, $T$ und $Y$ dem Schema

Ist $(X,T,Y)$ Standard-Tripel, so gilt

1. Äquivalente Charakterisierungen für Standard-Tripel. Es gelten die folgenden Eigenschaften. Das Matrizentripel $(X,T,Y)$ ist genau dann Standard-Tripel, wenn für die Inverse des Matrixpolynomes $L(\lambda)$ die Darstellung gilt

$L^{-1}(\lambda)$ kann man auffassen als Übertragungsfunktion des linearen Systems

Weiterhin gilt

wobei $\Gamma$ eine rektifizierbare Kurve ist, sodaß $\sigma(L)$ innerhalb von $\Gamma$ liegt, und $f$ ist eine holomorphe Funktion innerhalb von $\Gamma$ und innerhalb einer Umgebung von $\Gamma$.

2. Linearisierungen. Das Matrixpolynom $I\mu-A$ der Größe $(n+p)\times(n+p)$ ist eine Linearisierung des Matrixpolynomes $L(\mu)$ der Größe $\ell\times\ell$ und des Grades $n$ genau dann, wenn

Zwei Matrixpolynome $M_1(\mu)$ und $M_2(\mu)$ sind äquivalent, also $M_1(\mu)\sim M_2(\mu)$, genau dann, wenn

mit Matrixpolynomen $E(\mu)$ und $F(\mu)$, mit nicht verschwindender konstanter Determinante. Offensichtlich muß $n+p=n\ell$ sein. Zwei Linearisierungen sind stets zueinander ähnlich. Jede zu einer Linearisierung ähnliche Matrix, ist ebenfalls eine Linearisierung. Nebenläufig sei darauf hingewiesen, daß bei quadratischen Matrizen, jede Matrix zu ihrer Transponierten ähnlich ist. Weiter gilt nun der

3. Satz: Ist eine Matrix $T\in\mathbb{C}^{m\times m}$ gegeben, so ist $T$ genau dann eine Linearisierung eines monisches Matrixpolynoms vom Grade $\ell$ und der Größe $n\times n$, wenn die beiden folgenden Bedingungen erfüllt sind:

1. $m=n\ell$ und
2. $\displaystyle\max_{\lambda\in\mathbb{C}}\dim\ker(I\lambda-T)\le n$.

Den Beweis führt man auf den Smith'schen Normalformensatz zurück. Zum Beweise dieser und anderer hier relevanter Tatsachen, sei auf das Buch von Gohberg/Lancaster/Rodman (1982) hingewiesen, wo auch weiterführende Literaturstellen zu diesem Thema angegeben werden. Autoren sind Gohberg, Izrael' TSudikovich, Lancaster, Peter und Rodman, Leiba.

4. Matrixdifferenzengleichungen und Standard-Tripel. Bei linearen Mehrschrittverfahren der Form

tauchen in natürlicher Form skalare Differenzengleichungen auf. Bei zyklischen, linearen Verfahren, wie z.B. der Form

tauchen Matrixdifferenzengleichungen der Form

in ebenso natürlicher Weise auf. Gelegentlich ist es von Vorteil, eine Darstellung für die Lösung der Differenzengleichung zu haben, welche deutlich macht, wie sämtlich bisher berechneten Werte für nachfolgende Werte eingehen.

5. Satz: Es gilt für die Lösung der Matrixdifferenzengleichung

die Darstellung der Lösung zu

wobei $(X,T,Y)$ Standard-Tripel ist zum Matrixpolynom

Der Vektor $c\in\mathbb{C}^{n\ell}$ ist durch Vorgabe der Startwerte

eindeutig bestimmt und gegeben durch

Setzt man $R=\mathop{\rm row}_{i=0}^{\ell-1}T^iY$, $Q=\mathop{\rm col}_{i=0}^{\ell-1}XT^i$, so ist $RBQ=I$ und $c=RBa=Q^{-1}a$.

eklausmeier.goip.de

Proud member of the exclusive 250KB Club!

Added: 2024-01-19 | Last updated: 2024-01-19

eklausmeier.goip.de is a member of the exclusive 250KB Club. The page weighs only 78kb and has a content-to-bloat ratio of 13%.

They are now entitled to add one of those shiny badges to your page. But don't forget, even though I tried to make them as small as possibe, a badge will add some kilobytes to your page weight. A code snipped can be found by clicking on the respective badge.

While the overall size of 78kb, compressed size, is OK, the bloat ratio of 13% is not so good. I.e., 87% is effectively bloat. In my case the major contributing factors are:

1. Google fonts, no fault on Google
2. JavaScript for Pagefind for having instant search

For example, the post Moved Blog To eklausmeier.goip.de measured with tools.pingdom.com loads in 244ms from Frankfurt and needs 8 requests.

The distribution among content type is as below.

Again, 90% is fonts, script, and CSS, i.e., bloat. Without losing any information, but with losing appearance and slickness I could spare 80%!

Looking at the waterfall diagram one can see that dropping fonts would not lead to any significant faster website. This is because Google is pretty fast serving all those fonts. Similarly, Pagefind's processing can be seen overlapping the other processing, so not adding much waiting.

Though I am also a little guilty in the overall website obesity crisis.

Most of the talk about web performance is similarly technical, involving compression, asynchronous loading, sequencing assets, batching HTTP requests, pipelining, and minification.

All of it obscures a simpler solution.

If you're only going to the corner store, ride a bicycle.

If you're only displaying five sentences of text, use vanilla HTML. Hell, serve a textfile! Then you won't need compression hacks, integral signs, or elaborate Gantt charts of what assets load in what order.

Browsers are really, really good at rendering vanilla HTML.

We have the technology.

Being a member of the 250KB club is not very surprising as I am already a member of the 512KB club, in particular their "green team", i.e., the team with websites smaller than 100kB uncompressed.

1. Original: WordPress version, lemire.me
2. Modified: Simplified Saaze version of Lemire

The original website is hosted by SiteGround and Cloudflare. It uses WordPress.

1. Comparison. For the comparison I use the website tools.pingdom.com, which provides various metrics to evaluate the performance of a website:

1. Page size
2. Number of requests
3. Load time
4. Concrete tips to improve performance
5. Waterfall diagram of requests
6. Breakdown of content types

All tests in Pingdom were conducted for Europe/Frankfurt, as I host all stuff on below machine in my living room not far from Frankfurt.

The post in question is Fast integer compression with Stream VByte on ARM Neon processors. The version using Simplified Saaze is here. This post has no comments, therefore the WordPress site has no disadvantage against the Simplified Saaze powered site. This post contains C code shown in syntax-highlighted form.

The results are thus:

Original (WordPress)	Modified (Simplified Saaze)

The results for the original website, based on WordPress, are indeed worse on every dimension: page size, load time, number of requests. In comparison to the modified version using Simplified Saaze the ratio is roughly:

1. Page size is more than 4:1
2. Load time is almost 3:1
3. Number of requests is 4:1

So Simplified Saaze is better in all dimensions by a factor. This is particularly striking as the Simplified Saaze version is entirely self-hosted, i.e., upload to the internet is limited to 50 MBit/s!

The recommendations for the original website are therefore not overly surprising:

The missing compression is clearly an oversight on the web-server part.

The breakdown of the content type for the original website is:

I uploaded the Simplified Saaze version to Netlify, which provides CDN functionality. I measured again the WordPress post requested from San Francisco, and the Simplified Saaze version from San Francisco. The measurements are pretty similar to the Frankfurt results.

Original (WordPress) San Francisco	Modified (Simplified Saaze) San Francisco

2. Modified website. The breakdown of the modified site, based on Simplified Saaze, is as below.

Actual loading of the modified site will roughly follow below waterfall diagram. This waterfall diagram shows that a major part of the loading time is spent in syntax highlighting (prism.js) and searching (pagefind). The fonts from Google load in record time.

3. Security considerations. Prof. Lemire's blog had been the target of a hack in 2008: My blog got hacked. Using a static site this attack could probably have been prevented, assuming HashOver is not affected.

End of 2008 problems still persisted: Need help protecting my blog.

A site using Markdown files as input is easy to backup. This is way easier to backup than a database. Just think about any schema changes in the databases during version upgrades. See Simplified Saaze:

Simplified Saaze works with ordinay files in your filesystem. No database required. This means less setup and maintenance, better security and more speed.

Storing your Markdown files in Git is one option.

4. Caching content. Prof. Lemire reported caching problems:

I estimate that I get somewhere between 30,000 and 50,000 unique visitors a month. Despite my efforts, my blog keeps on failing under the load. It becomes unavailable for hours.

These caching problems would go away with a static site. Obviously. The static site would handle the "Slashdot effect" quite effectively.

In 2022 the internet router was defective. This time Vodafone confirmed that the router is fine. The fault is on their end.

BetterUptime noticed the error in a timely fashion via e-mail, which, of course, I could not read, as I had no internet:

Monitor: eklausmeier.goip.de/…txt
Checked URL: GET https://eklausmeier.goip.de/betterUptime.txt
Cause: Failure when receiving data from the peer

Started at: 8 Jan 2024 at 06:53pm CET

Since around 22:00 (CET) internet is available again. BetterUptime reported a resolved incident at 22:55 (CET):

Monitor: eklausmeier.goip.de/…txt
Checked URL: GET https://eklausmeier.goip.de/betterUptime.txt
Cause: Failure when receiving data from the peer

Started at: 8 Jan 2024 at 06:53pm CET
Resolved at: 8 Jan 2024 at 10:55pm CET (automatically)
Length: 3 hours and 58 seconds

So overall, betterUptime did a good job here.

Cloud-Computing ist in aller Munde. Man hört:

1. Cloud ist modern.
2. Cloud ist grün.
3. Cloud spare Kosten.
4. Cloud ist verkaufsfördernd.

Jedoch, die Euphorie bekommt Risse. Die vorhergesagten Kostenersparnisse treten nicht ein. Der Aufbau einer Cloud-Infrastruktur benötigt Zeit und erfordert spezielle Kenntnisse und Erfahrungen, die nicht überall anzutreffen sind. Nun mehren sich die Berichte, daß eine Reihe von hochkarätigen Firmen die Cloud wieder verlassen und zu selbst-administrierten Rechenzentren oder Colocationen zurückkehren. Was ist passiert?

1. Der Cloud Markt. Bevor wir das beantworten, ein Blick auf die Entwicklung des Cloud-Marktes. Die bekannten Spieler im Markt sind:

1. AWS - Amazon Web Services
2. Google Cloud Platform
3. Microsoft Azure
4. Oracle Cloud

Daneben gibt es eine Vielzahl von kleineren und regionalen Anbietern. Beispielsweise erzeugte Flexential im November 2023 Negativschlagzeilen.

In nachfolgender Tabelle sind die Umsätze in Milliarden (10⁹) USD angegeben, gerundet auf volle Milliarden.

Zahlen aus den Geschäftsberichten, siehe Literatur.

Es herrschte und es herrscht weiterhin Goldgräberstimmung in Anbetracht obiger Umsätze.

2. Klischees. Der ehemalige Google Ingenieur Nima Badizadegan beschreibt in Use One Big Server, daß parallel zum Wachstum von Cloud-Infrastruktur auch die Rechenleistung von Hardware enorm gestiegen ist. Ein kleiner, moderner Server deckt den Bedarf zahlreicher Anwendungen problemlos ab. Mehr noch, die üblichen Klischees von den Vorteilen einer Cloud-Lösung stimmen nicht, oder nur zum Teil. Im Nachfolgenden zugespitzt zusammengefaßt.

1. Benutze ich Cloud-Infrastruktur, dann benötige ich keine Systemadministratoren — nein, das stimmt nicht.
2. Benutze ich Cloud-Infrastruktur, dann muß ich mich nicht um Sicherheits-Patches kümmern — nein, das stimmt nicht.
3. Benutze ich Cloud-Infrastruktur, dann muß ich mich nicht darum kümmern, ob der Rechner gerade nicht verfügbar ist — nein, das stimmt nicht.
4. Ich kann in einer Cloud-Infrastruktur schneller Software entwickeln &mdash nein, das stimmt nicht.
5. Meine Arbeitslast für die Hardware ist sehr stark schwankend, mal viel, mal wenig — hier haben wir in der Tat einen echten Kandidaten.

Stark schwankende Rechenleistung ist der idealtypische Anwendungsfall für Cloud-Infrastruktur. Paart man dies mit der schnellen Verfügbarkeit der Rechenleistung, so begünstigt dies agiles Arbeiten und hohe Flexibilität. Hier ist auch der Fall der Miete von Spezialhardware zu nennen, wie beispielsweise GPUs oder FPGA. Ein Beispiel hierfür ist Modal Labs.

Ein anderes Klischee ist, daß Cloud-Infrastruktur besonders grün sei. Dazu muß man wissen, daß die großen Cloud-Anbieter, wie Amazon, Google, Microsoft, u.s.w., permanent Rechenkapazität vorhalten müssen. Andernfalls wäre es ja nicht möglich, daß kurzfristig zusätzliche Leistung zur Verfügung steht. Dieses Vorhalten von Leistung kostet Hardware und natürlich auch Strom und Abwärme, die abgeführt werden muß. Es ist das gleiche Dilemma, in dem auch Fondsgesellschaften stecken: Diese müssen permanent eine Barreserve vorhalten, sodaß sie Verkäufern des Fonds aus der Barreserve bedienen können, ohne direkt ihre Kerninvestments verkaufen zu müssen, um an Bargeld zu kommen. Ähnlich auch bei den Autoverleihern, wie Avis, Sixt, Europcar u.s.w.: Es müssen permanent Autos vorgehalten werden, um den Verleihdienst jederzeit anbieten zu können. Im Unterschied dazu hat derjenige, der sich die Hardware kauft, die er für seinen Zweck benötigt, nicht unnütz Kapazität in seinem Rechenzentrum.

Die Cloud ist insoweit grün, als daß der Cloud-Anbieter aus Eigeninteresse sein Rechenzentrum sehr effizient baut. D.h. Rechenzentren dieser Anbieter erreichen eine "Power Usage Effectiveness" von ca. 1,1 oder sogar darunter. Power Usage Effectiveness ist das Verhältnis von Gesamtleistung (Kühlung+Hardware) zu aufgenommer Leistung der IT-Hardware. Spätestens ab 2030 müssen alle Rechenzentren eine Power Usage Effectiveness von kleiner oder gleich 1,3 erreichen (§11 Abs. 1 & 2), siehe Das Energieeffizienzgesetz.

3. Kosten. Vergleicht man die Miete von Rechnern bei einem großen Cloud-Anbieter mit der Selbstbeschaffung entsprechender Rechner, so ist die Beschaffung und der Betrieb des eigenen Rechners fast immer günstiger. Und zwar deutlich günstiger. Hierzu Nima Badizadegan:

Being cloudy is expensive. Generally, I would anticipate a 5-30x price premium depending on what you buy from a cloud company, and depending on the baseline. Not 5-30%, a factor of between 5 and 30.

Dies sind keine Einzelbeobachtungen. Kristian Köhntopp, Cloud Architekt Syseleven und ehemals Firma Booking, nun HERE Technologies, kommt zu ganz ähnlichen Ergebnissen:

Weil Cloud unglaublich teuer ist. In meinen Kostenrechnungen liegen die Kosten für Cloud-Deployments pro Monat in etwa gleichauf mit (oder höher als!) Kostenrechnungen für Bare Metal in eigenen Rechenzentren pro Jahr (Strom, Netz, anteilige Netzwerk-Hardware, Rechenzentrumsplatz und alles andere inbegriffen).

Und weiter:

Andersherum bedeutet das, daß man das eigene Bare Metal hemmungslos überdimensionieren kann und immer noch unter AWS-Preisen herauskommt.

4. Repatriierung. Bei diesen immensen Kostenunterschieden ist es nur eine Frage der Zeit bis die ersten bekannten Firmen umsatteln.

1. Dropbox wechselte von AWS zu einem eigenem Rechenzentrum und spart 75 Millionen USD.
2. 37signals.com (Handelsmarken Basecamp und HEY) spart monatlich 60% der Kosten, das sind ca. 10 Millionen USD.
3. Twitter berichtet Oktober 2023 ebenfalls von einer 60% Kostenreduktion, das sind 100 Millionen USD.

Optimized our usage of cloud service providers and began doing much more on-prem. This shift has reduced our monthly cloud costs by 60%.

4. Sofascore reduzierte seine Kosten um den Faktor 10.

Es zeichnet sich sogar das Berufsbild eines Cloud-Repatriierungsexperten ab. Dies letztlich verursacht durch eine zu hastige und einseitige Migration zahlreicher Rechenzentren in die Cloud in den letzten Jahren. Nun, wo der Kostenschuh drückt, besinnt man sich auf kosteneffiziente und angepaßte RZ-Lösungen.

5. Fazit. Cloud-Infrastruktur erweitert die Möglichkeiten des Software-Architekten und bietet neue Möglichkeiten für Web-Anwendungen, Datenanalysen und Speicherung. Insbesondere ihre zügige Verfügbarkeit erleichtert agiles Arbeiten — Hardware auf Knopfdruck.

Hingegen, eine schemenhafte Anwendung neuer Dienste führt nicht zwangsläufig zu kosteneffizienten Lösungen. Die in manchen Unternehmen anzutreffende "Cloud first" Strategie (Vorrang der Cloud vor eigenem RZ) schränkt die Vielfalt der Möglichkeiten ein. Sie ersetzt die Tugenden Sparsamkeit und Handwerklichkeit durch Zeitgeist und Gehorsam.

Literatur

1. Nima Badizadegan on Cloud Computing
2. Cloudflare Dashboard Down
3. Amazon Annual reports, proxies and shareholder letters
4. Amazon 2018 Annual Report
5. Amazon 2020 Annual Report
6. Amazon 2021 Annual Report
7. Alphabet 2022 Annual Report
8. Microsoft Annual Report 2017
9. Microsoft Annual Report 2020
10. Microsoft Annual Report 2021
11. Microsoft Annual Report 2022
12. Various Quotes from Kristian Köhntopp
13. David Heinemeier Hansson on Cloud Computing
14. Josep Stuhli On Scaling to 20 Million Users
15. Why companies are leaving the cloud

These numbers are given by:

for i in `seq 2004 2023`; do grep 'h2 class="entry-title"' b*.html | grep -c me/blog/$i/; done

In total there are 2,224 blog posts over 20 years of permanent blogging. It can clearly be seen that the blog is updated on a regular basis, and many readers interact with the content.

Prof. Lemire values to have control over his blog, therefore doesn't use Medium or similar offers. Some key functionalities:

1. Allows WordPress comments
2. Informs e-mail subscribers about new posts, he has over 12,500 mail subscribers
3. Provides search-functionaly on his blog
4. Doesn't show any advertisements
5. Provides an Atom RSS feed
6. Blog posts are all in English
7. Doesn't use categories or tags
8. Doesn't use the <!--more--> tag
9. WordPress theme is based on "Twenty-Fifteen"
10. There is no regular sitemap.xml for the blog posts

1. Converting WordPress blog. Download all blog posts via Perl script bloglemirecurl. This script downloads the so called "pages", which in turn contains 20 blog posts. This HTML file, which contains 20 blog posts, is then converted to Markdown via Perl script bloglemiremd.

bloglemiremd b*.html

The Markdown files are placed in /tmp/lemire. As usual you might need a few rounds to eliminate obvious conversion errors. Finally you copy the Markdown files from /tmp/lemire to your final destination.

There are 14 blog posts, which reside at the top of the directory, which are not part of the timeline. These posts are accessed via the left navigation bar (in blue). To convert these posts use

bloglemiremd -t *-*.html pred*.html

Again, the converted HTML files are stored under /tmp/lemire for inspection. Once you are fine with them, copy them to the final destination.

Go to .../content/blog and run below loop using blogdate to create an index.md for each year:

for i in `seq 2004 2023`; do blogdate -p/lemire/blog/ -y$i $i/*.md > $i/index.md; done

Embedding icon in head-template file:

1. Download icon: curl https://lemire.me/blog/wp-content/uploads/2015/10/profile2011_152-150x150.jpg -o pr.jpg
2. Converting to 32x32 size: convert -resize 32x32 pr.jpg pr32x32.jpg
3. Base64-encoding file: base64 -w0 pr32x32.jpg

Size comparison for this icon: original JPG is 6,699 bytes, converted image is 934 bytes, base64-encoded is finally 1,248 bytes.

2. Installation. The entire theme including content and Simplified Saaze is installed via composer.

$ time composer create-project eklausme/saaze-lemire
Creating a "eklausme/saaze-lemire" project at "./saaze-lemire"
Installing eklausme/saaze-lemire (v1.0)
  - Downloading eklausme/saaze-lemire (v1.0)
  - Installing eklausme/saaze-lemire (v1.0): Extracting archive
Created project in /tmp/saaze-lemire
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking eklausme/saaze (v1.34)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Downloading eklausme/saaze (v1.34)
  - Installing eklausme/saaze (v1.34): Extracting archive
Generating optimized autoload files
No security vulnerability advisories found.
        real 1.85s
        user 0.27s
        sys 0
        swapped 0
        total space 0

You need to compile a single C file once:

cd vendor/eklausme/saaze
cc -fPIC -Wall -O2 -shared php_md4c_toHtml.c -o php_md4c_toHtml.so -lmd4c-html

Now you can run php saaze.

As mentioned Simplified Saaze is already installed via above composer command. In case you want to take a separate view at the Simplified Saaze source code see saaze.

3. Building static site. Running Simplified Saaze on all 2,224 blog posts:

saaze-lemire: time php saaze -rb /tmp/build
Building static site in /tmp/build...
        execute(): filePath=/home/klm/php/saaze-lemire/content/blog.yml, nentries=2224, totalPages=112, entries_per_page=20
Finished creating 1 collections, 1 with index, and 2259 entries (0.39 secs / 22.55MB)
#collections=1, YamlParser=0.0314/2260-1, md2html=0.0362, MathParser=0.0167/2259, renderEntry=2259, content=2259/0, excerpt=0/0
        real 0.41s
        user 0.26s
        sys 0
        swapped 0
        total space 0

In less than half a second the generation of all static files is completed. Machine in question: CPU is Ryzen 7 5700G, max clock 4.6 GHz, running on Arch Linux with kernel 6.6.8.

A screenshot of the theme is here:

The screenshot shows the results of a search, here for "WordPress".

The theme also features Pagefind. I have written on Pagefind: Searching in Static Sites. Creating the Pagefind index goes like this:

/tmp/build: time pagefind -s . --exclude-selectors aside --exclude-selectors footer

Running Pagefind v1.0.4
Running from: "/tmp/build"
Source:       ""
Output:       "pagefind"

[Walking source directory]
Found 2372 files matching **/*.{html}

[Parsing files]
Did not find a data-pagefind-body element on the site.
↳ Indexing all <body> elements on the site.

[Reading languages]
Discovered 1 language: en

[Building search indexes]
Total:
  Indexed 1 language
  Indexed 2372 pages
  Indexed 29164 words
  Indexed 0 filters
  Indexed 0 sorts

Finished in 5.325 seconds
        real 5.43s
        user 4.50s
        sys 0
        swapped 0
        total space 0

The index creation is way slower than creating all static pages.

4. Webserver rewrite rules. The conversion from WordPress to Markdown placed all blog posts from one year into a single directory at the same level. For example, the posts

https://lemire.me/blog/2006/01/03/are-debuggers-obselete/

is in directory .../content/blog/2006 and in file

01-03-are-debuggers-obselete.md

On my webserver the URL can be both, watch out for dash vs. slash:

1. https://eklausmeier.goip.de/lemire/blog/2006/01-03-are-debuggers-obselete
2. https://eklausmeier.goip.de/lemire/blog/2006/01/03/are-debuggers-obselete

Watch out for the / slashes. This is accomplished by below rewriting rule in the NGINX configuration file:

rewrite "^/lemire/blog/(\d\d\d\d)/(\d\d)/(\d\d)/(.*)"  "/lemire/blog/$1/$2-$3-$4";

Instead of above rewriting rule once could place above Markdown file in the following directory

.../content/blog/2006/01/03

But this would create a lot of directories, which essentially all contain only a single file.

5. Fetching comments from WordPress. Perl script bloglemirecurlcomment scans through above "pages", i.e., collection of 20 blog posts. These pages contain 20 URLs. These URLs are fetched via curl. Essentially, this duplicates the blog posts, but at least we now have the comments for each post as well.

for i in `seq 1 112`; do bloglemirecurlcomment ../b$i.html; done

These HTML files are then processed by bloglemirecomment, which scans for <h2 class="comments-title"> and writes out the comment file. Each comment file is generated from the original blog post file by adding the word -comment- to the file name after the day.

Each comment file has index: false, i.e., it will not show up in the index. Though, all content is fully searchable.

In addition the Perl script blogdate adds a link to each comment file. Calling is like:

for i in `seq 2004 2023`; do ( cd $i; ~/php/saaze-lemire/bin/blogdate -y$i *.md > index.md ) done

Counting the number of comments per year is like:

#!/bin/perl -W
# Count comments per year

use strict;

my ($year,%H) = (0,());

while (<>) {
    $year = $1 if (/<link rel="canonical" href="https:\/\/lemire.me\/blog\/(\d\d\d\d)\/(\d\d)\/(\d\d)\//);
    if (/(\w+) thought(|s) on &ldquo;/) {
        my $cnt = $1;
        $cnt = 1 if ($cnt eq 'One');
        $H{$year} += $cnt;
    }
}

for (sort keys %H) {
    printf("%04d\t%d\n",$_,$H{$_});
}

6. Building static site with separate comment pages. Generating all static pages for the entire blog including comments is:

saaze-lemire: time php saaze -rb /tmp/build
Building static site in /tmp/build...
        execute(): filePath=/home/klm/php/saaze-lemire/content/blog.yml, nentries=2224, totalPages=112, entries_per_page=20
Finished creating 1 collections, 1 with index, and 3935 entries (0.89 secs / 66.49MB)
#collections=1, YamlParser=0.0630/3936-1, md2html=0.0895, MathParser=0.0575/3935, renderEntry=3935, content=3935/0, excerpt=0/0
        real 0.91s
        user 0.56s
        sys 0
        swapped 0
        total space 0

This time can be reduced to 0.46 seconds, see Parallelizing the Output of Simplified Saaze.

Generating the pagefind index for 4048 files takes roughly 12 seconds:

/tmp/build: time pagefind -s . --exclude-selectors aside --exclude-selectors footer

Running Pagefind v1.0.4
Running from: "/tmp/build"
Source:       ""
Output:       "pagefind"

[Walking source directory]
Found 4048 files matching **/*.{html}

[Parsing files]
Did not find a data-pagefind-body element on the site.
↳ Indexing all <body> elements on the site.

[Reading languages]
Discovered 1 language: en

[Building search indexes]
Total:
  Indexed 1 language
  Indexed 4048 pages
  Indexed 60783 words
  Indexed 0 filters
  Indexed 0 sorts

Finished in 11.412 seconds
        real 11.59s
        user 10.22s
        sys 0
        swapped 0
        total space 0

Simplified Saaze allows to generate single files, i.e., only a single blog post can be processed by Simplified Saaze, see Single file generation. This can be used to significantly reduce the generation time.

7. HTML validation. The original site lemire.me contains more than 90 warnings and errors. See W3 Nu Html Checker.

The new site contains no errors or warnings.

8. Recap. Prof. Lemire is quite hesitant to move all static:

Several commenters pointed out that I could just drop WordPress and use something else. I fear that they greatly underestimate how hard this would be. Yes, I know about things like Hugo. My relatively simple home page is built using Hugo… and it took me nearly took weeks of hacking to get it to be how I want. Porting my blog to something like Hugo would be a major disruption, might imply moving to disqus (see point above) and so forth.

Porting Prof. Lemire's blog started in 12-Dec-2023 and was "finished" 14-Jan-2024 including porting all comments to HashOver. Of course, I did not work on this full-time.

There are still some open issues pending regarding conversion and functionality:

1. Some pages have wrong formatting, e.g., there is bold printing in the converted site not present in the original.
2. Left and right double quotes have been converted to HTML codes. Entering those is not very convenient. We clearly want SmartyPants.
3. Five URLs were not correctly mapped as they contain special characters.
4. E-mail subscriptions is absent. Although I doubt that there really 12,500 active subscribers. Though, there are probably a lot, which want to get noticed when something new arrives. One possible approach is to use Buttondown. For example, Buttondown can send e-mails based on RSS, see below screenshot from the "Settings" dialog in Buttondown.

2. Solution. I already knew that a Pandoc approach does not work. For the conversion I modified the two Perl scripts used for the journal article conversion:

1. blogparsec
2. blogbibtex

The result is in:

1. blogOnlineDialARide
2. blogbibtex

Using those two script, creating the Markdown file goes like this:

blogOnlineDialARide einleitung.tex chapter1.tex chapter2.tex chapter3.tex > .../2020/10-15-online-dial-a-ride.md
blogbibtex thesis.bib >> .../2020/10-15-online-dial-a-ride.md

The file 10-15-online-dial-a-ride.md still needs some manual editing:

1. move the table-of-content to the top, as this is appended at the end with blogbibtex
2. insert an image for the algorithm

3. blogOnlineDialARide script. Some notes on this Perl script. The input to this script is the concatenation of all relevant LaTeX files.

First define some variables and use strict mode.

use strict;
my ($ignore,$inTable,$inAlgo,
    $chapterCnt,$sectionCnt,$subSectionCnt,$theoremCnt,$itemCnt,
    $claimCnt,$eqnCnt,$eqnFlag,$tableCnt,$tabInsert,$caseCnt,
    $enumerate,$prefix) = (0,0,0, 0,0,0,0,0, 0,0,0,0,0,0, 0,"");
my (@sections) = ();
my (%H,%Hphp) = ( (), () );  # hash for key=\label, value=\ref, in our case for lemmatas and theorems

The frontmatter header is a simple here-document. Also, it defines some PHP variables, which are needed as the thesis makes some forward references to tables, and lemmas, and we want a single pass over the document file only.

print <<'EOF';
---
date: "2020-10-15 14:00:00"
title: "Online Dial-A-Ride"
description: "We consider the online Dial-a-Ride Problem where objects are to be transported between points in a metric space in the shortest possible completion time."
MathJax: true
categories: ["mathematics"]
tags: ["ABORT-OR-REPLAN", "Dial-A-Ride", "online optimization"]
author: "Roman Edenhofer"
---


<!-- https://docs.mathjax.org/en/latest/input/tex/eqnumbers.html -->
<script type="text/javascript">
    window.MathJax = { tex: { tags: 'ams' } };
</script>

<?php	// forward references in text
    $tab__ABORT = "1";
    $tab__AAW = "2";
    $tab__state_of_the_art = "3";
    $lemma__new_extreme = "3.11";
    $lemma__waiting = "3.12";
    $lemma__aborting = "3.13";
    $lemma__abc = "3.14";
    $lemma__unique_tour = "3.15";
    $lemma__upwards = "!unknown!";
?>

EOF

The main loop looks at each line in main.tex. After the loop the literature section is added, then all sections collected so far are printed.

while (<>) {
    chomp;
    if (/\\end\{tabular\}/) { $ignore = 0; next; }
    next if ($ignore);

    next if (/\\addcontentsline\{toc\}/);
    (...)
    print $prefix . $_ . "\n";
}


print "## Literature<a id=Literature></a>\n";
for (@sections) {
    print $_ . "\n";
}
++$sectionCnt;
print "- [$sectionCnt. Literature](#Literature)\n";

What follows is the part which is marked as (...) in above code. First of all, just drop irrelevant space.

    # Space handling
    s/\s+$//g;	# rtrim
    s/^\s+//g;	# ltrim, i.e., erase leading space
    s/\~/ /g;

    s/\s+%\s+[^%].+$//;	# Drop LaTeX comments
    s/^%.*//g;

The tables are replaces by manually entered Markdown tables:

    s/\\normalsize//;
    if (/\\end\{table\}/) {
        print $table[$tabInsert++];
        $inTable = 0;
        next;
    }
    if ($inTable) {
        s/\\caption\{([^\}]+)\}/\n\n__Table $tableCnt:__ $1\n/;
    }
    if (/\\begin\{table\}/) {
        ($ignore,$inTable) = (1,1);
        next;
    }

The @table array is initialized at the top of the Perl file like this:

my @table = (
'
Case    | ABORT                      | open | closed
--------|----------------------------|------|---------
general | uncapacitated ($c=\infty$) | 3    | 2.5
general | preemptive                 | 3    | 2.5

',
'
Case    | ABORT-AND-WAIT             | open   | closed
--------|----------------------------|--------|---------
general | uncapacitated ($c=\infty$) | 2.4142 | 2.5
general | preemptive                 | 2.4142 | 2.5

',
'
Case    | General Bounds                | open<br>lower bound | open<br>upper bound | closed<br>lower bound | closed<br>upper bound
--------|-------------------------------|---------------------|---------------------|-----------------------|----------------------
general | non-preemptive $(c < \infty)$ | 2.0585 | 2.6180 ([MLipmann][]) | 2 | 2 ([Ascheuer][])
general | uncapacitated $(c=\infty)$    | 2.0346 | 2.4142 ([BjeldeDisser17][]) | 2 | 2
general | preemptive                    | 2.0346 | __2.4142__ | 2 (Thm 3.2 in [Ausiello][]) | 2
general | TSP                           | 2.0346 | 2.4142 | 2 | 2
---     |                               |        |        |   |
line    | non-preemptive $(c < \infty)$ | 2.0585 (Thm 1 in [Birx19][]) | 2.6180 | 1.75 ([BjeldeDisser17][]) | 2
line    | uncapacitated $(c=\infty)$    | 2.0346 | 2.4142 | 1.6404 | 2
line    | preemptive                    | 2.0346 | 2.4142 ([BjeldeDisser17][]) | 1.6404 | 2
line    | TSP                           | 2.0346 ([BjeldeDisser17][]) | 2.4142 ([BjeldeDisser17][]) | 1.6404 (Thm 3.3 in [Ausiello][]) | 1.6404 ([BjeldeDisser17][])
---     |                               |        |        |   |
halfline| non-preemptive $(c < \infty)$ | 1.8968 ([MLipmann][]) | 2.6180 | 1.7071 ([Ascheuer][]) | 2
halfline| uncapacitated $(c=\infty)$    | 1.6272 | 2.4142 | 1.5 | __1.8536__
halfline| preemptive                    | 1.6272 | 2.4142 | 2 | 2
halfline| TSP                           | 1.6272 | 2.4142 ([MLipmann][]) | 1.5 ([MRIN][]) | 1.5 ([MRIN][])


'
);

Then these predefined elements are inserted one by one: $table[0], $table[1], etc.

Many special cases are handled, which are specific to this document.

    # Special cases:
    s/ \\AOR-server / AOR-server /g;	# \AOR outside of math-mode
    s/ while \\\\ / while /;
    # MathJax bug prevention
    s/^Suppose that \$\(L\^\*/Suppose that\n\$\$\n\(L\^\*/;
    s/ p_R\$\./ p_R\.\n\$\$/;
    s/L\^\*/L\^\{\\ast\}/g;
    s/\{t\^start\}/\{eqn: t\^start\}/g;
    # forward reference resolution
    s/\\ref\{lemma: waiting\}/[\<\?=\$lemma__waiting\?>\](#lemma__waiting)/g;
    # MathJax shortcoming
    s/\\makebox\[0pt\]\{\\text\{(|\\scriptsize)/\{\{/;
    # double } resolution in eqn:
    s/eqn: OPT\(t_\{i-j\}\)/eqn: OPT\(t_Ci-jD\)/g;
    s/eqn: p\^\{AOR\}/eqn: p^cAORd/g;
    s/eqn: T\^\{return\}/eqn: T\^CreturnD/g;
    s/eqn: L\^\{\\ast\}/eqn: L\^C\\astD/g;
    # Some simple conversions to Markdown
    s/\\textit\{([^\}]+)\}/_$1_/g;

Display math is enclosed in double dollars keeping the \begin{align} and \end{align} stuff:

    if (/\\begin\{align(|\*)\}/) {
        print "\$\$\n\\begin{align$1}\n";
        next;
    } elsif (/\\end\{align(|\*)\}/) {
        if ($eqnFlag) { print "\\end{align$1}\n\t\\tag{$eqnCnt}\n\$\$\n"; $eqnFlag = 0; }
        else { print "\\end{align$1}\n\$\$\n"; }
        next;
    }

Most algorithms are replaces by HTML quotations. One algorithm, which are particular "complex" is just replaced by an image (screenshot).

    if (/\\begin\{algorithm\}/) {
        ($inAlgo,$prefix) = (1,'> ');
        next;
    } elsif (/\\end\{algorithm\}/) {
        ($inAlgo,$prefix) = (0,'');
        next;
    }
    if ($inAlgo == 1) {
        next if (/\\SetKwData|\\SetKwFunction|\\SetKwInOut/);
        s/\\;$/<br>/;
        s/\\caption\{(.+)\}$/__$1__<br>/;
        s/\\Input\{(.+)\}$/__input:__ $1/;
        s/\\Output\{(.+)\}$/__output:__ $1/;
    }

The most difficult part was to replace numbered theorems, lemmas, definitions, claims with something automatic. I use a combination of Perl numbering, and PHP variables. So forward or back references look like this: [look here: $Perl variable](#$PHPvariable).

    ++$theoremCnt if (/\\begin\{(theorem|lemma|remark)\}/);
    ++$claimCnt if (/\\begin\{claim\}/);
    ++$caseCnt if (/\\begin\{case\}/);
    s/\\begin\{definition\}/<p><\/p>\n\n---\n\n__Definition.__/;
    s/\\begin\{theorem\}/<p><\/p>\n\n---\n\n__Theorem ${chapterCnt}.${theoremCnt}.__/;
    s/\\begin\{lemma\}/<p><\/p>\n\n---\n\n__Lemma ${chapterCnt}.${theoremCnt}.__/;
    s/\\begin\{remark\}/<p><\/p>\n\n---\n\n__Remark ${chapterCnt}.${theoremCnt}.__/;
    s/\\begin\{claim\}/<p><\/p>\n\n__Claim ${claimCnt}.__/;
    s/\\begin\{case\}/<p><\/p>\n\n_Case ${caseCnt}._/;
    s/\\end\{(theorem|lemma|remark|claim|case)\}//;
    s/\\end\{definition\}/\n---\n<p><\/p>\n/;
    s/\\begin\{proof\}/<p><\/p>\n\n_Proof._/;
    s/\\end\{proof\}/&nbsp; &nbsp; &#9744;\n\n/;

    if (/^\\label\{(.+)\}$/) {
        my ($phpvar,$key) = ($1,$1);
        $phpvar =~ s/( |:|"|\^|\{|\}|<|>|\\|\/|\*)/_/g;	# create valid PHP variable out of \label
        $Hphp{$key} = $phpvar;
        if ($key =~ /^(th|lemma)/) {
            $H{$key} = "${chapterCnt}.${theoremCnt}";
        } elsif ($key =~ /^eqn/) {
            ++$eqnCnt;
            $eqnFlag = 1;
            $H{$key} = "${eqnCnt}";
            next;
        } elsif ($key =~ /^claim/) {
            $H{$key} = "${claimCnt}";
        } elsif ($key =~ /^chapter/) {
            $H{$key} = "s${chapterCnt}";
        } elsif ($key =~ /^tab/) {
            ++$tableCnt if (!defined($H{$key}));
            $H{$key} = "${tableCnt}";
        } else {
            $H{$key} = "unknown hash H: key=$key";
        }
        #$_ = '<a id="'.$phpvar.'"></a>';
        $_ = '<a id="'.$phpvar.'"></a><?php $'.$phpvar.'="'.$H{$key}.'"; ?>';
    }
    #s/\\ref\{(.+?)\}(\)|\.| )/\[$H{$1}\](#s$H{$1})$2/g;
    #s/\\ref\{(.+?)\}(\.| )/\[$H{$1}\](#"s$1")$2/g;
    #s/\\ref\{(.+?)\}(\.| )/\[$H{$1}\](#\*<\?=\$$Hphp{$1}\?>\*)$2/g;
    #s/\\ref\{(.+?)\}(\.| )/\[$H{$1}\](#$Hphp{$1})$2/g;
    #good (almost): s/\\ref\{(.+?)\}(\.| )/\[<\?=\$$Hphp{$1}\?>\](#$Hphp{$1})$2/g;
    #while (/\\ref\{(.+?)\}(\.|\)| )/g) {
    while (/\\ref\{([^\}]+?)\}/g) {
        my $key = $1;
        if (!defined($H{$key})) {
            print STDERR "key=|$key| undefined in H\n";
            my $phpvar = $1;
            $phpvar =~ s/( |:|"|\^|\{|\}|<|>|\\|\/|\*)/_/g;	# create valid PHP variable out of \label
            $Hphp{$key} = $phpvar;
            if ($key =~ /^tab/) {	# unfortunately, tables are forward referenced
                ++$tableCnt;
                $H{$key} = "tab${tableCnt}";
            }
        }
        if ($key =~ /^eqn/) {
            s/\\ref\{(.+?)\}/$H{$1}/g;
        } else {
            s/\\ref\{(.+?)\}(\.|\)| )/\[<\?=\$$Hphp{$1}\?>\](#$Hphp{$1})$2/g;
        }
    }

Again, many thesis specific changes.

    # Substitute own TeX macros
    s/\\N([^\w])/\\mathbb\{N\}$1/g;
    s/\\R([^\w])/\\mathbb\{R\}$1/g;
    #s/\\Q([^\w])/\\mathbb\{Q\}$1/g;
    #s/\\M([^\w])/\\mathcal\{M\}$1/g;
    s/\\ABORT/\\hbox\{ABORT\}/g;
    s/\\OPT/\\hbox\{OPT\}/g;
    s/\\ALG/\\hbox\{ALG\}/g;
    s/\\AAW/\\hbox\{AAW\}/g;
    s/\\AOR/\\hbox\{AOR\}/g;
    s/\\DOWN/\\hbox\{DOWN\}/g;
    s/\\abort/\\hbox\{abort\}/g;
    s/\\replan/\\hbox\{replan\}/g;
    s/\\diff/\\hbox\{diff\}/g;
    s/\\prepared/\\hbox\{prepared\}/g;
    s/\\start/\\hbox\{start\}/g;
    s/\\ente/\\hbox\{ente\}/g;
    s/\\move/\\hbox\{move\}/g;
    s/\\waituntil/\\hbox\{waituntil\}/g;
    s/\\return/\\hbox\{return\}/g;
    s/\\new/\\hbox\{new\}/g;
    s/\\Return/__return:__/g;

    s/\\Tilde/\\tilde/g;

    # Lines to drop, not relevant
    next if (/\\DontPrintSemicolon/);

Handling of items in LaTeX.

    if (/\\begin\{itemize\}/) {
        ($enumerate,$itemCnt,$_) = (0,1,'');
    } elsif (/\\begin\{enumerate\}/) {
        ($enumerate,$itemCnt,$_) = (1,1,'');
    } elsif (/\\end\{(itemize|enumerate)\}/) {
        ($enumerate,$itemCnt,$_) = (0,0,'');
    }
    if (/^\\item /) {
        if ($enumerate) {
            s/\\item /${itemCnt}. /;
            ++$itemCnt;
        } else {
            s/\\item /\* /;
        }
    }
    if (/\\item\[([^\]]+)\]/) {
        s/\\item\[([^\]]+)\]/${itemCnt}. /;
        ++$itemCnt;
    }

Handling of chapters, sections, and subsections. For all three I uses different Perl counters. All chapters, sections, etc. can be jumped to. They are referenced by #s, followed by chapter number, section number, etc.

    # sections + subsections
    if (/\\chapter\*\{(\w+)\}/) {	# unnumbered section, line "Introduction"
        my $s = $1;
        push @sections, "- [$s](#s$s)";
        $_ = "\n## $s<a id=s$s></a>\n";
    } elsif (/\\chapter\{(.+?)\}\s*$/) {
        my $s = $1;
        ++$chapterCnt; $sectionCnt = 0; $subSectionCnt = 0; $theoremCnt = 0;
        push @sections, "- [$chapterCnt. $s](#s$chapterCnt)";
        $_ = "\n## $chapterCnt. $s<a id=s$chapterCnt></a>\n";
    } elsif (/\\section\{(.+?)\}\s*$/) {
        my $s = $1;
        ++$sectionCnt; $subSectionCnt = 0;
        push @sections, "- [$chapterCnt.$sectionCnt $s](#s${chapterCnt}_${sectionCnt})";
        $_ = "\n### $chapterCnt.$sectionCnt $s<a id=s${chapterCnt}_$sectionCnt></a>\n";
    } elsif (/\\subsection\{(.+?)\}\s*$/) {
        my $s = $1;
        ++$subSectionCnt;
        push @sections, "\t- [$chapterCnt.$sectionCnt.$subSectionCnt $s](#s${chapterCnt}_${sectionCnt}_$subSectionCnt)";
        $_ = "\n#### $chapterCnt.$sectionCnt.$subSectionCnt $s<a id=s${chapterCnt}_${sectionCnt}_$subSectionCnt></a>\n";
    }

Citations are easy. I use a feature in Markdown/Commonmark called Link references. The link in the text is like this [literature123][], the actual definition can be anywhere, for example, at the end of the document. It looks like this: [literature123]: ....

    # Citations
    s/\\citeauthor\{([\-\w]+)\} \\cite\{([\-\w]+)\}/\[$1\]\[\]/g;

During development of this Perl script I used Beyond Compare again quite intensively, to compare the original against the changed file.

4. blogbibtex script. The input to this script is the Bibtex file with all literature references. The Bibtex file looks something like this:

@inproceedings{Ascheuer,
author = {Ascheuer, Norbert and Krumke, Sven Oliver and Rambau, J\"{o}rg},
title = {Online Dial-a-Ride Problems: Minimizing the Completion Time},
year = {2000},
isbn = {3540671412},
publisher = {Springer-Verlag},
address = {Berlin, Heidelberg},
booktitle = {Proceedings of the 17th Annual Symposium on Theoretical Aspects of Computer Science},
pages = {639–650},
numpages = {12},
series = {STACS '00}
}

@article{Ausiello,
author = {Ausiello, Giorgio and Feuerstein, Esteban and Leonardi, S. and Stougie, L. and Talamo, Maurizio},
year = {2001},
month = {04},
pages = {560-581},
title = {Algorithms for the On-Line Travelling Salesman},
volume = {29},
journal = {Algorithmica},
doi = {10.1007/s004530010071}
}

The Perl script is just a slightly modified version of the script used in Converting Journal Article from LaTeX to Markdown.

High-resolution three-dimensional maps of the Milky Way have previously been limited to the immediate vicinity of the Sun. In a collaboration led by the Max Planck Institute for Astrophysics with researchers from Harvard, the Space Telescope Science Institute, and the University of Toronto, we were now able to build a high-resolution map of the Milky Way in 3D out to more than 4,000 light-years. The produced 3D map will be highly useful for a wide range of applications from star formation to cosmological foreground correction.

When we think about the Milky Way, we often think about 2D images of the night sky or artist's impressions of how the Milky Way might look from outside our Galaxy. With the advent of Gaia, we are entering a new era of Milky Way science, in which we begin to unfold our previous 2D view of the Milky Way into a rich 3D picture. In recent years, we started to build 3D maps of the distribution of matter in the immediate vicinity of the Sun out to approximately 1,000 light-years. Thanks to these maps, we were able to study the star formation around the Sun in 3D, made numerous discoveries about the shape, mass, and density of nearby molecular clouds, and learned how supernova feedback shaped the space around the Sun.

At the core of maps of the 3D distribution of matter in the Milky Way lies interstellar dust. Interstellar dust closely traces the distribution of matter, cools gas such that stars can form, agglomerates to form planets, and obscures astrophysical observations. Incidentally, this obscuration allows us to quantify the amount of dust between us, on Earth, and the astrophysical object we want to observe in the background, often stars. We can infer the 3D distribution of dust and thus indirectly trace the distribution of matter in the Galaxy using this information. To do so, we combine millions of measurements of the amount of dust to background objects with distance estimates to said objects from Gaia.

Inferring the distribution of dust in the Milky Way from distances and dust measurements is a computationally intensive, statistical inverse problem. The problem is ill posed: from our limited data and prior knowledge about dust, it is not possible to retrieve a definite answer about the true distribution of dust. Still, the language of statistics allows us to translate our noisy data with a physics-informed model of dust into a 3D dust map with rigorously quantified uncertainties. Until now, however, the computational costs of 3D dust models have limited the size of the probed volume.

Bird's-eye view of the distribution of dust within 4,077 light-years around the Sun. The Sun is at the center and the galactic center is to the right.

Recent progress in our physics-informed model of dust enabled us to probe much larger distances. We put forward a new statistical method to model spatially smooth structures in large volumes – a required component of dust maps. At the heart of the new method is an algorithm to iteratively add ever-finer details to a coarse representation of 3D dust. By adding details iteratively instead of modelling everything at once, the modelling problem drastically simplifies and becomes faster by orders of magnitude.

We combined the new methodological developments with the latest processed Gaia data to create the largest high-resolution map of interstellar dust to date. The new 3D dust map extends 4,077 light-years in all directions from the Sun with a resolution of a few light-years. The produced 3D map will be highly useful for studying the medium between stars in the Milky Way. Understanding the structure of the interstellar medium will help us constrain key relations for star formation. In addition, the 3D dust map will be important for correcting astrophysical observations. For many observations, the interstellar medium in front of the object of interest is a nuisance. The new 3D dust map will allow correcting these measurements for the foreground material in a much larger volume than previous maps.

As of today there are some serious efforts to replace Android with a "real" Linux.

It is speculated that the postmarketOS approach is most promising. In below, I copy various citations from their website.

We are sick of not receiving updates shortly after buying new phones. Sick of the walled gardens deeply integrated into Android and iOS.

The heritage:

postmarketOS is based on Alpine Linux, which is so tiny (less than 10 MB in size) that development of pmOS can be done quickly on any Linux distribution.

The consequence of this:

The above design decisions make it feasible to keep the system up-to-date, for all devices at once! Compared to Android, it makes development more efficient ...

The rather cumbersome Android build system is not used:

We avoid Android's build system entirely. Instead of building a monolithic system image for each and every device, the whole OS is divided into small packages. These same package binaries can be installed on all devices that share the same CPU architecture. Device specific parts are kept as minimal as possible, ideally there is only one device package.

postmarketOS uses the ext4 filesystem!

From the FAQ: Will Android apps be supported?

We support Android apps through Waydroid!

The list of supported devices is quite impressive.

postmarketOS can use below user interfaces:

1. Phosh based on GNOME
2. Plasma Mobile based on KDE
3. Sxmo, a tiling window manager
4. Xfce
5. MATE based on GNOME
6. and others

2. Installation. The accompanying Makefile is ready to use, i.e., no configure is required. Just compile (=make) and install (=make install). On Arch Linux use AUR package mle. One particular good AUR helper is trizen.

3. Size comparison. Comparing the library dependencies for mle, vim and nvim:

$ ldd /bin/mle /bin/vim /bin/nvim
/bin/mle:
        linux-vdso.so.1 (0x00007fff4e28d000)
        libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x00007fb7a6246000)
        liblua.so.5.4 => /usr/lib/liblua.so.5.4 (0x00007fb7a61ff000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007fb7a6112000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fb7a5f30000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7a6362000)
/bin/vim:
        linux-vdso.so.1 (0x00007ffefa6e9000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007fc0cd313000)
        libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x00007fc0cd8f4000)
        libacl.so.1 => /usr/lib/libacl.so.1 (0x00007fc0cd8eb000)
        libgpm.so.2 => /usr/lib/libgpm.so.2 (0x00007fc0cd8e3000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fc0cd131000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fc0cd9a1000)
/bin/nvim:
        linux-vdso.so.1 (0x00007ffdefdbd000)
        libluv.so.1 => /usr/lib/libluv.so.1 (0x00007f190e1bc000)
        libtermkey.so.1 => /usr/lib/libtermkey.so.1 (0x00007f190e1b0000)
        libvterm.so.0 => /usr/lib/libvterm.so.0 (0x00007f190e19d000)
        libmsgpackc.so.2 => /usr/lib/libmsgpackc.so.2 (0x00007f190e194000)
        libtree-sitter.so.0 => /usr/lib/libtree-sitter.so.0 (0x00007f190e166000)
        libunibilium.so.4 => /usr/lib/libunibilium.so.4 (0x00007f190e151000)
        libluajit-5.1.so.2 => /usr/lib/libluajit-5.1.so.2 (0x00007f190e0be000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f190db13000)
        libuv.so.1 => /usr/lib/libuv.so.1 (0x00007f190dadf000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f190daba000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f190d8d8000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f190e223000)

mle uses:

1. uthash for hash maps and linked lists
2. termbox2 for text-based UI
3. PCRE2 for syntax highlighting and search
4. Lua as a macro language

Comparing file sizes of the executables for mle, nano, neovim, and vim:

$ ls -l /bin/mle /bin/nano /bin/vim /bin/nvim
-rwxr-xr-x 1 root root  298752 Oct 29 22:22 /bin/mle*
-rwxr-xr-x 1 root root  278856 Jan 18  2023 /bin/nano*
-rwxr-xr-x 1 root root 4795872 Oct 10 13:39 /bin/nvim*
-rwxr-xr-x 1 root root 4848056 Oct 26 22:39 /bin/vim*

4. Speed comparison. Below is a comparison of the starting times for a 187 MB sized file conducted in 2016 by Adam Saponara:

I tried two files, all stored in /tmp, which is in RAM on Arch Linux kernel 6.6.1:

1. seq 999000 > x9, size is 6.6 MB, time wc x9 is 0.02s
2. seq 9999000 > x9b, size is 76 MB, time wc x9b is 0.17s

Starting times for mle, vim, and neovim are as below:

Apparently, the speed advantage cannot be reproduced with these two particular files. vim and neovim need roughly half of their time actually processing the file, as half of the time is needed for just reading the content, as can be seen by the wc times.

More technical details on benchmarking can be found here: Full soft-wrap implementation #77.

5. Basic usage. In the following we use below abbrevations for keys. As usual, you have to press them all at once.

Some basic file operations within the editor:

mle supports editing multiple files at once. Switching between buffers is by using M-1, M-2, M-3, etc.

Once you press F2 (the help key) then automatically a new buffer is opened. To switch back to your original file you would use M-1.

6. Moving the cursor around. Below commands just move the cursor around and do not change the file content in any way.

7. Copying, deleting, or moving text. Below commands change the content of the file.

When you start mle then whenever you enter a TAB, this will be changed to spaces. To change this behaviour you enter M-o a, and then enter y at the prompt. If you want to go back to the automatic tab to space conversion, then enter a number at the prompt.

8. Useful startup file. Below startup file for mle, also called rc file, named .mlerc, is located in the user's home directory:

-Kklm,,1
-kcmd_move_beginning,C-home,
-kcmd_move_end,C-end,
-nklm
-w1
-t8
-e1
-a0
     <empty line>

Ignore the first four lines for the moment. The meaning is as follows: enable word wrap (-w), set tabsize to 8 characters (-t), enable mouse support (-e), make tabs as tabs (-a). This is equivalent to start mle with below command line arguments:

mle -w1 -t8 -e1 -a0

If the startup file is executable then the output of the file is taken as actual rc file. So the startup can be changed conditionally.

9. Setting or redefining keys. mle allows you to set or redefine key bindings. This is a 3-step process.

1. You define a so called kmap using command line option -K
2. Within this kmap you specify pairs of commands and keys using option -k
3. You instruct mle to use this new kmap with option -n

For example the standard mle key binding for jumping to the end of the file is M-/. In Google Chrome or many editors this is C-end. Specifying this is thus:

mle -K 'klm,,1' -k 'cmd_move_end,C-end,' -n klm <file>

10. Lua macros. Below table information is extracted from uscript.lua.

11. Limitation. Unfortunately mle has some shortcomings.

1. mle does not have a line-wrap functionality. So long lines do not wrap at the end of the screen. For source code files this is fine. But for Markdown files this is a severe restriction.
2. This only happens on st: mle -Qk does not handle Shift-home and home differently, therefore you cannot define the combination of Shift-home to mean "go to top of page". This works perfectly fine on xterm.
3. When cutting text out of file1, then this text is not available, when in file2 there is also text, which has been cut.

Today Cloudflare has its snafu-moments:

Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services.

So even the biggest players in the pond suffer from power outage from time to time.

The following products are currently impacted at the data plane / edge level, meaning that the full product functionality is either partially or fully affected: Logpush, WARP / Zero Trust device posture, Cloudflare dashboard, Cloudflare API, Stream API, Workers API, Alert Notification System.

As I have a copy of this blog on Cloudflare Workers, I cannot update my blog, at least not today.

Uploading a zip-file also does not work.

Added 03-Nov-2023: Cloudflare was hit really hard. Now their dashboard is entirely unavailable.

Text:

The Cloudflare Dashboard is temporarily unavailable.

Please reload this page to try again. If the issue persists, please visit the Cloudflare Status page for up-to-date information regarding any ongoing issues.

It looks that they didn't fully realize how severe their problem really is.

Added 05-Nov-2023: On 04-Nov-2023 Matthew Prince, CEO of Cloudflare, gave a detailed post mortem of the incident. The text Post Mortem on Cloudflare Control Plane and Analytics Outage is worth reading multiple times. It teaches a number of important lessons in resilience. Here I will quote some snippets, party out of context, but highlighting some of the many problems.

The largest of the three facilities in Oregon is run by Flexential.

The mishap started as follows:

On November 2 at 08:50 UTC Portland General Electric (PGE), the utility company that services PDX-04, had an unplanned maintenance event affecting one of their independent power feeds into the building.

This happened without Cloudflare being aware of this. I had already speculated that Cloudflare was not really aware of their mess they were in:

Flexential did not inform Cloudflare that they had failed over to generator power. None of our observability tools were able to detect that the source of power had changed.

Things get worse quite quickly:

At approximately 11:40 UTC, there was a ground fault on a PGE transformer at PDX-04. ... Ground faults with high voltage (12,470 volt) power lines are very bad. Electrical systems are designed to quickly shut down to prevent damage when one occurs. Unfortunately, in this case, the protective measure also shut down all of PDX-04’s generators. This meant that the two sources of power generation for the facility — both the redundant utility lines as well as the 10 generators — were offline.

One mishap doesn't come alone:

PDX-04 also contains a bank of UPS batteries. ... the batteries started to fail after only 4 minutes.

As if it was written by a film author:

the overnight shift (from Flexential) consisted of security and an unaccompanied technician who had only been on the job for a week.

Data center went dark without Cloudflare knowing it:

Between 11:44 and 12:01 UTC, with the generators not fully restarted, the UPS batteries ran out of power and all customers of the data center lost power. Throughout this, Flexential never informed Cloudflare that there was any issue at the facility.

The rest of the text discusses why Cloudflare put some of their production products into a single data center, in this case the faulty one.

We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster.

But the nightmare was not over yet:

At 12:48 UTC, Flexential was able to get the generators restarted. ... When Flexential attempted to power back up Cloudflare's circuits, the circuit breakers were discovered to be faulty.

The next sentence tells you why you should always have enough spare replacement kits:

Flexential began the process of replacing the failed breakers. That required them to source new breakers because more were bad than they had on hand in the facility.

Thundering herd problem, also see Josep Stuhli On Scaling to 20 Million Users:

When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services.

Always test your disaster recovery procedures, if not, then:

A handful of products did not properly get stood up on our disaster recovery sites. These tended to be newer products where we had not fully implemented and tested a disaster recovery procedure.

Cloudflare's data center reported normal operations:

Flexential replaced our failed circuit breakers, restored both utility feeds, and confirmed clean power at 22:48 UTC.

But for Cloudflare the ordeal was not over yet. The had to restart everything:

Beginning first thing on November 3, our team began restoring service in PDX-04. That began with physically booting our network gear then powering up thousands of servers and restoring their services. The state of our services in the data center was unknown as we believed multiple power cycles were likely to have occurred during the incident. Our only safe process to recover was to follow a complete bootstrap of the entire facility.

This was no small feat:

Rebuilding these took 3 hours.

Matthew Prince acknowledges that much is to be learnt from this event:

But we also must expect that entire data centers may fail. Google has a process, where when there’s a significant event or crisis, they can call a Code Yellow or Code Red. In these cases, most or all engineering resources are shifted to addressing the issue at hand.

We have not had such a process in the past, but it’s clear today we need to implement a version of it ourselves

Little gold nugget from the lessons learnt:

Test the blast radius of system failures

Murphy's law applies universally, in particular for software and data centers.

David Heinemeier Hansson from 37signals.com wrote We stand to save $7m over five years from our cloud exit. Also see Dropbox slips 500PB into its Magic Pocket, not spread over AWS: "Shifts 90% of your files from Amazon to in-house systems".

Heinemeier adds on 23-Jun-2023:

The back of the napkin math is that we'll save at least $1.5 million per year by owning our own hardware rather than renting it from Amazon. And crucially, we've been able to do this without changing the size of the operations team at all. Running our applications in the cloud just never provided the promised productivity gains to do with any smaller of a team anyway.

The main difference here is the lag time between needing new servers and seeing them online. It truly is incredible that you can spin up 100 powerful machines in the cloud in just a few minutes, but you also pay dearly for the privilege. And we just don't have such an unpredictable business as to warrant this premium. Given how much money we're saving owning our own hardware, we can afford to dramatically over-provision our server needs, and then when we need more, it still only takes a couple of weeks to show up.

Look at it this way. We spent about half a million dollars buying two pallets of servers from Dell, which added a combined 4,000 vCPUs with 7,680 GB of RAM and 384TB of NVMe storage to our server capacity. This hardware was more than adequate to run all the heritage services we brought home, together with HEY, and give our other Basecamp operations a hardware refresh. And it was less than a third the cost of what we predict we'll be saving EVERY YEAR! This is hardware we'll be amortizing over five years.

David Heinemeier Hansson shows that even after one year operation costs went down by $1m.

Our cloud spend (sans-S3) is down by 60% already. From around $180,000/month to less than $80,000. That's a cool million dollars in savings at the yearly run rate

In his post X celebrates 60% savings from cloud exit on 27-Oct-2023 he cites the Twitter/X engineering team:

Optimized our usage of cloud service providers and began doing much more on-prem. This shift has reduced our monthly cloud costs by 60%. Among the changes we made was a shift of all media/blob artifacts out of the cloud, which reduced our overall cloud data storage size by 60%, and separately, we succeeded in reducing cloud data processing costs by 75%.

Further:

According to earlier reports, X was spending $100 million per year with AWS, so if we take that as a base, they're on track to save $60m/year from the cloud exit achievements so far. Wild!

Added 04-Jan-2024: David Heinemeier Hansson added a FAQ here: The Big Cloud Exit FAQ.

Added 07-Jan-2024: David Heinemeier Hannson added this post Keeping the lights on while leaving the cloud. One single quote:

You don’t need the cloud to get good uptimes. You need mature technologies run on redundant hardware with good backups. Same as it ever was.

Simple, but true.

1. Profiling PHP Programs
2. Profiling PHP Programs #2

Still I am interested whether I missed anything.

In multiple talks Rasmus Lerdorf, the creator of PHP, advertises PHPSPY.

PHPSPY was written by Adam Saponara. The source code is in GitHub: https://github.com/adsr/phpspy.

I ran PHPSPY in top mode for some days using the dynamic mode of Simplified Saaze: phpspy -p 940 -p 17132 -p 61898 -p 61899 -t. The output is below. Some remarks on inclusive and exclusive times or counts:

1. Inclusive counts everything for the function and all its function it calls.
2. Exclusive only counts a particular function.

phpspy -p 940 -p 17132 -p 61898 -p 61899 -@
samp_count=666  err_count=10  func_count=67

tincl      texcl      incl       excl       excl%   func
313        151        0          0          0.00    ComposerAutoloaderInit50920a90746408ba7a500bacdb4908c1::getLoader /home/klm/php/sndsaaze/vendor/composer/autoload_real.php:19
132        103        0          0          0.00    composerRequire50920a90746408ba7a500bacdb4908c1 /home/klm/php/sndsaaze/vendor/composer/autoload_real.php:50
99         99         0          0          0.00    Composer\Autoload\includeFile /home/klm/php/sndsaaze/vendor/composer/ClassLoader.php:569
76         76         0          0          0.00    json_decode <internal>:-1
298        34         0          0          0.00    Saaze\Saaze::run /home/klm/php/sndsaaze/vendor/eklausme/saaze/Saaze.php:32
30         30         0          0          0.00    ComposerAutoloaderInit50920a90746408ba7a500bacdb4908c1::loadClassLoader /home/klm/php/sndsaaze/vendor/composer/autoload_real.php:9
23         23         0          0          0.00    FFI::cdef <internal>:-1
19         19         0          0          0.00    file_get_contents <internal>:-1
15         15         0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-mbstring/bootstrap.php:1
13         13         0          0          0.00    md4c_toHtml <internal>:-1
14         11         0          0          0.00    str_word_count <internal>:-1
10         10         0          0          0.00    yaml_parse <internal>:-1
322        9          0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/autoload.php:1
90         9          0          0          0.00    Saaze\TemplateManager::<main> /home/klm/php/sndsaaze/templates/blog/entry.php:1
8          8          0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-intl-grapheme/bootstrap.php:1
5          5          0          0          0.00    FFI::string <internal>:-1
653        4          0          0          0.00    <main> /home/klm/php/sndsaaze/public/index.php:1
5          4          0          0          0.00    Saaze\TemplateManager::<main> /home/klm/php/sndsaaze/templates/error.php:1
5          3          0          0          0.00    microtime <internal>:-1
4          3          0          0          0.00    strpos <internal>:-1
3          3          0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-ctype/bootstrap.php:1
3          3          0          0          0.00    shell_exec <internal>:-1
27         2          0          0          0.00    Saaze\CollectionArray::loadCollections /home/klm/php/sndsaaze/vendor/eklausme/saaze/CollectionArray.php:27
21         2          0          0          0.00    <main> <internal>:-1
10         2          0          0          0.00    is_dir <internal>:-1
9          2          0          0          0.00    Saaze\Collection::__construct /home/klm/php/sndsaaze/vendor/eklausme/saaze/Collection.php:15
9          2          0          0          0.00    Saaze\TemplateManager::renderError /home/klm/php/sndsaaze/vendor/eklausme/saaze/TemplateManager.php:62
4          2          0          0          0.00    scandir <internal>:-1
3          2          0          0          0.00    strlen <internal>:-1
2          2          0          0          0.00    Saaze\TemplateManager::<main> /home/klm/php/sndsaaze/templates/top-layout.php:1
2          2          0          0          0.00    Saaze\MarkdownContentParser::inlineMath /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:172
2          2          0          0          0.00    strip_tags <internal>:-1
23         1          0          0          0.00    Saaze\MarkdownContentParser::toHtml /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:562
9          1          0          0          0.00    substr <internal>:-1
1          1          0          0          0.00    substr_replace <internal>:-1
1          1          0          0          0.00    usort <internal>:-1
1          1          0          0          0.00    printf <internal>:-1
1          1          0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-intl-normalizer/bootstrap.php:1
1          1          0          0          0.00    ob_end_clean <internal>:-1
1          1          0          0          0.00    str_replace <internal>:-1
1          1          0          0          0.00    file_put_contents <internal>:-1
1          1          0          0          0.00    max <internal>:-1
1          1          0          0          0.00    is_readable <internal>:-1
111        0          0          0          0.00    Saaze\Collection::loadMkdwnRecursive /home/klm/php/sndsaaze/vendor/eklausme/saaze/Collection.php:70
91         0          0          0          0.00    Saaze\TemplateManager::renderEntry /home/klm/php/sndsaaze/vendor/eklausme/saaze/TemplateManager.php:37

Interestingly, the time spent by Composer-classes is greater than the actual runtime of Simplified Saaze!

Added 11-Dec-2023: Measured once again. Results are below.

phpspy -p 879 -p 1015 -p 1016 -p 20333 -@
samp_count=2422  err_count=55  func_count=97

tincl      texcl      incl       excl       excl%   func
1077       491        0          0          0.00    ComposerAutoloaderInit50920a90746408ba7a500bacdb4908c1::getLoader /home/klm/php/sndsaaze/vendor/composer/autoload_real.php:19
506        369        0          0          0.00    composerRequire50920a90746408ba7a500bacdb4908c1 /home/klm/php/sndsaaze/vendor/composer/autoload_real.php:50
353        353        0          0          0.00    json_decode <internal>:-1
335        335        0          0          0.00    Composer\Autoload\includeFile /home/klm/php/sndsaaze/vendor/composer/ClassLoader.php:569
81         81         0          0          0.00    md4c_toHtml <internal>:-1
76         76         0          0          0.00    ComposerAutoloaderInit50920a90746408ba7a500bacdb4908c1::loadClassLoader /home/klm/php/sndsaaze/vendor/composer/autoload_real.php:9
75         75         0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-mbstring/bootstrap.php:1
459        69         0          0          0.00    Saaze\TemplateManager::<main> /home/klm/php/sndsaaze/templates/blog/entry.php:1
67         67         0          0          0.00    FFI::cdef <internal>:-1
58         58         0          0          0.00    file_get_contents <internal>:-1
48         35         0          0          0.00    str_word_count <internal>:-1
1162       29         0          0          0.00    Saaze\Saaze::run /home/klm/php/sndsaaze/vendor/eklausme/saaze/Saaze.php:32
28         28         0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-intl-grapheme/bootstrap.php:1
26         26         0          0          0.00    yaml_parse <internal>:-1
1096       23         0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/autoload.php:1
22         20         0          0          0.00    scandir <internal>:-1
21         20         0          0          0.00    strpos <internal>:-1
491        19         0          0          0.00    Saaze\TemplateManager::renderEntry /home/klm/php/sndsaaze/vendor/eklausme/saaze/TemplateManager.php:37
21         16         0          0          0.00    microtime <internal>:-1
16         16         0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-ctype/bootstrap.php:1
32         15         0          0          0.00    substr <internal>:-1
74         14         0          0          0.00    <main> <internal>:-1
2384       12         0          0          0.00    <main> /home/klm/php/sndsaaze/public/index.php:1
15         12         0          0          0.00    Saaze\MarkdownContentParser::inlineMath /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:172
12         12         0          0          0.00    strip_tags <internal>:-1
20         11         0          0          0.00    is_dir <internal>:-1
12         10         0          0          0.00    Saaze\TemplateManager::<main> /home/klm/php/sndsaaze/templates/top-layout.php:1
7          7          0          0          0.00    str_replace <internal>:-1
260        6          0          0          0.00    Saaze\Collection::loadEntry /home/klm/php/sndsaaze/vendor/eklausme/saaze/Collection.php:82
159        6          0          0          0.00    Saaze\MarkdownContentParser::toHtml /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:562
8          6          0          0          0.00    FFI::string <internal>:-1
6          6          0          0          0.00    shell_exec <internal>:-1
233        5          0          0          0.00    Saaze\Entry::getContentAndExcerpt /home/klm/php/sndsaaze/vendor/eklausme/saaze/Entry.php:86
6          5          0          0          0.00    <main> /home/klm/php/sndsaaze/vendor/symfony/polyfill-intl-normalizer/bootstrap.php:1
5          5          0          0          0.00    function_exists <internal>:-1
71         4          0          0          0.00    Saaze\CollectionArray::loadCollections /home/klm/php/sndsaaze/vendor/eklausme/saaze/CollectionArray.php:27
50         4          0          0          0.00    Saaze\Entry::parseEntry /home/klm/php/sndsaaze/vendor/eklausme/saaze/Entry.php:21
6          4          0          0          0.00    Saaze\TemplateManager::<main> /home/klm/php/sndsaaze/templates/error.php:1
6          4          0          0          0.00    is_readable <internal>:-1
21         3          0          0          0.00    Saaze\Collection::__construct /home/klm/php/sndsaaze/vendor/eklausme/saaze/Collection.php:15
5          3          0          0          0.00    Saaze\TemplateManager::<main> <internal>:-1
4          3          0          0          0.00    Saaze\MarkdownContentParser::myTag /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:212
3          3          0          0          0.00    substr_replace <internal>:-1
3          3          0          0          0.00    max <internal>:-1
3          3          0          0          0.00    Saaze\MarkdownContentParser::displayMath /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:145
3          3          0          0          0.00    dirname <internal>:-1
3          3          0          0          0.00    getenv <internal>:-1
3          3          0          0          0.00    rtrim <internal>:-1
549        2          0          0          0.00    Saaze\Collection::loadMkdwnRecursive /home/klm/php/sndsaaze/vendor/eklausme/saaze/Collection.php:70
72         2          0          0          0.00    Saaze\Config::init /home/klm/php/sndsaaze/vendor/eklausme/saaze/Config.php:14
20         2          0          0          0.00    Saaze\Collection::parseCollection /home/klm/php/sndsaaze/vendor/eklausme/saaze/Collection.php:22
16         2          0          0          0.00    Saaze\MarkdownContentParser::gallery /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:397
5          2          0          0          0.00    date <internal>:-1
3          2          0          0          0.00    strlen <internal>:-1
2          2          0          0          0.00    ltrim <internal>:-1
2          2          0          0          0.00    Saaze\Collection::Saaze\{closure} /home/klm/php/sndsaaze/vendor/eklausme/saaze/Collection.php:54
2          2          0          0          0.00    Saaze\TemplateManager::templateExists /home/klm/php/sndsaaze/vendor/eklausme/saaze/TemplateManager.php:7
2          2          0          0          0.00    Saaze\MarkdownContentParser::twitter /home/klm/php/sndsaaze/vendor/eklausme/saaze/MarkdownContentParser.php:333
2          2          0          0          0.00    fopen <internal>:-1
2          2          0          0          0.00    Composer\Autoload\ClassLoader::register /home/klm/php/sndsaaze/vendor/composer/ClassLoader.php:389
2          2          0          0          0.00    strtotime <internal>:-1
192        1          0          0          0.00    Saaze\Entry::__construct /home/klm/php/sndsaaze/vendor/eklausme/saaze/Entry.php:13
10         1          0          0          0.00    Saaze\Entry::getUrl /home/klm/php/sndsaaze/vendor/eklausme/saaze/Entry.php:74
4          1          0          0          0.00    ob_end_clean <internal>:-1

Results are similar. Clearly even more accentuating the importance of Composer.

Added 08-Feb-2024: I had some trouble getting PHPSPY to work again, see phpspy no longer works #136. I ran PHPSPY again but this time I dropped composer, which was marked as dominant in above PHPSPY sessions. Results for three days are as below.

phpspy -H 9999 --pgrep=php-fpm -@
samp_count=2651  err_count=575856  func_count=58

tincl      texcl      incl       excl       excl%   func
1047       850        0          0          0.00    FFI::string <internal>:-1
230        207        0          0          0.00    substr <internal>:-1
209        206        0          0          0.00    str_replace <internal>:-1
212        194        0          0          0.00    <main> <internal>:-1
269        193        0          0          0.00    max <internal>:-1
200        185        0          0          0.00    strpos <internal>:-1
147        147        0          0          0.00    ctype_space <internal>:-1
102        99         0          0          0.00    printf <internal>:-1
82         82         0          0          0.00    rtrim <internal>:-1
68         68         0          0          0.00    json_decode <internal>:-1
123        43         0          0          0.00    date <internal>:-1
42         41         0          0          0.00    strlen <internal>:-1
30         30         0          0          0.00    strip_tags <internal>:-1
28         28         0          0          0.00    md4c_toHtml <internal>:-1
47         23         0          0          0.00    str_word_count <internal>:-1
21         21         0          0          0.00    preg_match <internal>:-1
18         18         0          0          0.00    str_contains <internal>:-1
17         16         0          0          0.00    implode <internal>:-1
20         15         0          0          0.00    Saaze\TemplateManager::<main> <internal>:-1
15         15         0          0          0.00    strrpos <internal>:-1
14         14         0          0          0.00    is_dir <internal>:-1
13         13         0          0          0.00    define <internal>:-1
58         12         0          0          0.00    microtime <internal>:-1
12         12         0          0          0.00    getenv <internal>:-1
11         11         0          0          0.00    sprintf <internal>:-1
11         11         0          0          0.00    substr_replace <internal>:-1
11         10         0          0          0.00    FFI::cdef <internal>:-1
9          9          0          0          0.00    str_split <internal>:-1
8          8          0          0          0.00    function_exists <internal>:-1
14         7          0          0          0.00    urlencode <internal>:-1
7          7          0          0          0.00    DateTime::__construct <internal>:-1
6          6          0          0          0.00    yaml_parse <internal>:-1
13         5          0          0          0.00    is_array <internal>:-1
5          5          0          0          0.00    ltrim <internal>:-1
4          3          0          0          0.00    explode <internal>:-1
3          3          0          0          0.00    is_readable <internal>:-1
3          3          0          0          0.00    preg_replace <internal>:-1
3          3          0          0          0.00    extension_loaded <internal>:-1
3          3          0          0          0.00    str_starts_with <internal>:-1
3          3          0          0          0.00    spl_autoload_register <internal>:-1
3          3          0          0          0.00    trim <internal>:-1
3          2          0          0          0.00    count <internal>:-1
2          2          0          0          0.00    is_string <internal>:-1
2          2          0          0          0.00    is_bool <internal>:-1
1          1          0          0          0.00    round <internal>:-1
1          1          0          0          0.00    error_reporting <internal>:-1
1          1          0          0          0.00    print_r <internal>:-1
1          1          0          0          0.00    header <internal>:-1
1          1          0          0          0.00    mb_strtolower <internal>:-1
1          1          0          0          0.00    array_key_exists <internal>:-1
1          1          0          0          0.00    spl_autoload_unregister <internal>:-1
1          1          0          0          0.00    gettype <internal>:-1
1          1          0          0          0.00    mb_substr <internal>:-1
1          1          0          0          0.00    ucwords <internal>:-1
1          1          0          0          0.00    openssl_cipher_iv_length <internal>:-1
1          1          0          0          0.00    file_exists <internal>:-1
1          1          0          0          0.00    defined <internal>:-1
1          0          0          0          0.00    is_object <internal>:-1

At a later time:

phpspy -H 9999 --pgrep=php-fpm -@
samp_count=5475  err_count=1004659  func_count=64

tincl      texcl      incl       excl       excl%   func
2301       1842       0          0          0.00    FFI::string <internal>:-1
462        458        0          0          0.00    str_replace <internal>:-1
491        447        0          0          0.00    substr <internal>:-1
467        431        0          0          0.00    strpos <internal>:-1
556        414        0          0          0.00    max <internal>:-1
353        310        0          0          0.00    <main> <internal>:-1
299        299        0          0          0.00    ctype_space <internal>:-1
207        203        0          0          0.00    printf <internal>:-1
145        142        0          0          0.00    rtrim <internal>:-1
122        122        0          0          0.00    json_decode <internal>:-1
88         85         0          0          0.00    strlen <internal>:-1
196        57         0          0          0.00    date <internal>:-1
56         56         0          0          0.00    md4c_toHtml <internal>:-1
55         55         0          0          0.00    strip_tags <internal>:-1
88         47         0          0          0.00    str_word_count <internal>:-1
46         46         0          0          0.00    preg_match <internal>:-1
44         44         0          0          0.00    str_contains <internal>:-1
118        39         0          0          0.00    microtime <internal>:-1
30         30         0          0          0.00    strrpos <internal>:-1
29         29         0          0          0.00    is_dir <internal>:-1
25         25         0          0          0.00    substr_replace <internal>:-1
27         24         0          0          0.00    implode <internal>:-1
29         20         0          0          0.00    Saaze\TemplateManager::<main> <internal>:-1
19         19         0          0          0.00    sprintf <internal>:-1
19         19         0          0          0.00    getenv <internal>:-1
27         16         0          0          0.00    urlencode <internal>:-1
16         16         0          0          0.00    str_split <internal>:-1
18         14         0          0          0.00    FFI::cdef <internal>:-1
14         14         0          0          0.00    define <internal>:-1
31         13         0          0          0.00    is_array <internal>:-1
13         13         0          0          0.00    ltrim <internal>:-1

Eliminating composer indeed cut away all composer related CPU usage. It looks that above run cannot be fully compared to the two previous runs. This time FFI::string became dominant, which was not dominant in previous runs. No PHP source code from Simplified Saaze is visible except Saaze\TemplateManager::<main>. Instead string-processing, like substr(), str_replace(), and strpos() seem to be important.

I am still hesitant how much I can trust above run as the err_count is so alarmingly high.

Added 12-May-2024: Another measurement with phpspy, this time with MD4C used as an extension, i.e., no FFI. Watch out for the very high error count. It is remarkable that the max() function is so dominant.

phpspy -H 9999 --pgrep=php-fpm -@
samp_count=1004  err_count=216898  func_count=39

tincl      texcl      incl       excl       excl%   func
277        200        0          0          0.00    max <internal>:-1
176        176        0          0          0.00    json_decode <internal>:-1
121        121        0          0          0.00    <main> <internal>:-1
106        106        0          0          0.00    printf <internal>:-1
75         75         0          0          0.00    ctype_space <internal>:-1
44         43         0          0          0.00    md4c_toHtml <internal>:-1
45         40         0          0          0.00    strpos <internal>:-1
41         39         0          0          0.00    substr <internal>:-1
29         29         0          0          0.00    strip_tags <internal>:-1
26         21         0          0          0.00    rtrim <internal>:-1
196        20         0          0          0.00    date <internal>:-1
14         14         0          0          0.00    ltrim <internal>:-1
13         13         0          0          0.00    yaml_parse <internal>:-1
11         11         0          0          0.00    implode <internal>:-1
23         10         0          0          0.00    urlencode <internal>:-1
10         10         0          0          0.00    str_replace <internal>:-1
10         10         0          0          0.00    substr_replace <internal>:-1
13         9          0          0          0.00    Saaze\TemplateManager::<main> <internal>:-1
9          9          0          0          0.00    trim <internal>:-1
7          7          0          0          0.00    is_dir <internal>:-1
43         4          0          0          0.00    microtime <internal>:-1
8          4          0          0          0.00    define <internal>:-1
4          4          0          0          0.00    preg_match <internal>:-1
4          4          0          0          0.00    strlen <internal>:-1
5          3          0          0          0.00    str_word_count <internal>:-1
3          3          0          0          0.00    getenv <internal>:-1
3          3          0          0          0.00    spl_autoload_register <internal>:-1
3          3          0          0          0.00    function_exists <internal>:-1
2          2          0          0          0.00    array_key_exists <internal>:-1
2          2          0          0          0.00    str_starts_with <internal>:-1
2          2          0          0          0.00    file_exists <internal>:-1
1          1          0          0          0.00    file_get_contents <internal>:-1
1          1          0          0          0.00    extension_loaded <internal>:-1
1          1          0          0          0.00    is_readable <internal>:-1
1          1          0          0          0.00    base64_encode <internal>:-1
1          1          0          0          0.00    DateTime::__construct <internal>:-1
1          1          0          0          0.00    sprintf <internal>:-1
1          1          0          0          0.00    html_entity_decode <internal>:-1
6          0          0          0          0.00    basename <internal>:-1

1. Easy to install, no JavaScript dependency hell.
2. Easy to add the CSS and the two lines with <script> tag.
3. Creating the index is easy and reasonable quick.

Pagefind was mainly written by Liam Bigelow from New Zealand and is promoted by CloudCannon. It is open source. It is written in Rust and JavaScript.

1. One-time installation. Installing Pagefind is just downloading a single binary from GitHub: select the proper binary for Apple, Linux, or Windows. In my case I used pagefind-v1.0.3-x86_64-unknown-linux-musl.tar.gz for Arch Linux. Unpack with

tar zxf pagefind-v1.0.3-x86_64-unknown-linux-musl.tar.gz

Unpacking the 10 MB archive will create a 22 MB exectuable, which is statically linked and therefore has no dependencies. That's it.

2. Add CSS and JavaScript to template. Add below CSS and JavaScript reference to your template file outside of <body>:

<link href="/pagefind/pagefind-ui.css" rel="stylesheet">
<script src="/pagefind/pagefind-ui.js"></script>
<script>
    window.addEventListener('DOMContentLoaded', (event) => {
        new PagefindUI({ element: "#search", showSubResults: true });
    });
</script>

Then add the actual search dialog in your template inside <body>, in my case to top-layout.php:

<div id="search"></div>

3. Creating index files. This step must repeated whenever you have new content, or rename files. It does not need to be repeated whenever you regenerate your static HTML files. Altough if you want to play safe, you can do just that. Index creation is using the above mentioned executable pagefind. Running this command shows all the options:

$ pagefind -h
Implement search on any static website.

Usage: pagefind [OPTIONS]

Options:
  -s, --site <SITE>
          The location of your built static website
      --output-subdir <OUTPUT_SUBDIR>
          Where to output the search bundle, relative to the processed site
      --output-path <OUTPUT_PATH>
          Where to output the search bundle, relative to the working directory of the command
      --root-selector <ROOT_SELECTOR>
          The element Pagefind should treat as the root of the document. Usually you will want to use the data-pagefind-body attribute instead.
      --exclude-selectors <EXCLUDE_SELECTORS>
          Custom selectors that Pagefind should ignore when indexing. Usually you will want to use the data-pagefind-ignore attribute instead.
      --glob <GLOB>
          The file glob Pagefind uses to find HTML files. Defaults to "**/*.{html}"
      --force-language <FORCE_LANGUAGE>
          Ignore any detected languages and index the whole site as a single language. Expects an ISO 639-1 code.
      --serve
          Serve the source directory after creating the search index
  -v, --verbose
          Print verbose logging while indexing the site. Does not impact the web-facing search.
  -l, --logfile <LOGFILE>
          Path to a logfile to write to. Will replace the file on each run
  -k, --keep-index-url
          Keep "index.html" at the end of search result paths. Defaults to false, stripping "index.html".
  -h, --help
          Print help
  -V, --version
          Print version

This blog uses Simplified Saaze. In the case of Simplified Saaze I generate static files like this:

php saaze -mortb /tmp/build

This builds all static files in /tmp/build, which happens to be in a RAM disk on Arch Linux. Then change to this directory and issue

$ time pagefind -s . --exclude-selectors aside --exclude-selectors footer --force-language=en

Running Pagefind v1.0.3
Running from: "/tmp/build"
Source:       ""
Output:       "pagefind"

[Walking source directory]
Found 555 files matching **/*.{html}

[Parsing files]
Did not find a data-pagefind-body element on the site.
↳ Indexing all <body> elements on the site.

[Reading languages]
Discovered 1 language: en

[Building search indexes]
Total:
  Indexed 1 language
  Indexed 555 pages
  Indexed 33129 words
  Indexed 0 filters
  Indexed 0 sorts

Finished in 1.618 seconds
        real 1.65s
        user 1.49s
        sys 0
        swapped 0
        total space 0

The command

pagefind -s . --force-language=en

would habe been enough in many cases. In my special case I want to exclude content, which resides between <aside> and </aside>, and similarly between <footer> and </footer>.

The option --force-language=en is required in my case as I have English and German posts. Without this option pagefind would create two distinct indexes: You can then either only search in one language but not in the other. By forcing the language pagefind puts everything into a single index. See Multilingual search.

Indexing creates a directory called pagefind. Just copy this directory to your web-server during deployment. This directory looks something like this:

pagefind
├── fragment
│   ├── en_0933ef4.pf_fragment
│   ├── en_100be25.pf_fragment
│   ├── en_10b07a1.pf_fragment
│   ├── . . .
│   └── en_fef8cdb.pf_fragment
├── index
│   ├── en_22c87b9.pf_index
│   ├── en_26afa46.pf_index
│   ├── en_2a80efb.pf_index
│   ├── . . .
│   └── en_fde0a3b.pf_index
├── pagefind.en_d6828bd6ef.pf_meta
├── pagefind-entry.json
├── pagefind.js
├── pagefind-modular-ui.css
├── pagefind-modular-ui.js
├── pagefind-ui.css
├── pagefind-ui.js
├── wasm.en.pagefind
└── wasm.unknown.pagefind

3 directories, 596 files

These files in index are usually around 40KB each, those in fragment are usually around 1-10 KB each. The JavaScript totals 100KB, CSS is less than 20KB.

4. Network traffic. Pagefind was particularly designed to only load small amounts of data over the network. This can be seen from below diagram.

This makes Pagefind particularly attractive performancewise.

5. Using Pagefind as user. Using Pagefind as user is intuitive and needs no further explanation. This blog has Pagefind integrated into every page as of now. Just type a word you want to search, then results will pop-up almost instantly. This instant reaction is no surprise as the actual searching is done in the browser.

There is one slight limitation of Pagefind: currently you cannot search for word groups. I.e., consider Shakespeare's Hamlet:

To be, or not to be, that is the question

Searching for to or be would likely give you lots of results, but probably not the ones you are looking for. Clearly not a problem for this blog, as I do not have lyrics here.

If the article is in Markdown format, it can then be easily transformed into HTML. Having an article in Markdown format has a number of advantages over having the article in LaTeX format:

1. It is much easier to write Markdown than LaTeX
2. Reading HTML is easier than reading a PDF
3. The notion of page, i.e., paper sized page, does not have a good meaning in the world of smartphones, tablet, etc.

Of course, the math in the LaTeX document will be converted to MathJax.

2. Overview of the content of the scientific article. The article briefly describes the importance of dust:

Interstellar dust comprises only 1% of the interstellar medium by mass, but absorbs and re-radiates more than 30 of starlight at infrared wavelengths. As such, dust plays an outsized role in the evolution of galaxies, catalyzing the formation of molecular hydrogen, shielding complex molecules from the UV radiation field, coupling the magnetic field to interstellar gas, and regulating the overall heating and cooling of the interstellar medium.

Dust's ability to scatter and absorb starlight is precisely the reason why we can probe it in three spatial dimensions.

A novel $\cal O(n)$ method called Iterative Charted Refinement (ICR) was used to analyze the more than 122 billion of data from the Gaia mission.

The algorithm ran for 4 weeks using the SLURM workload manager.

We employ a new Python framework called NIFTy.re for deploying NIFTy models to GPUs. NIFTy.re is part of the NIFTy Python package and internally uses JAX to run models on the GPU. We are able to speed up the evaluation of the value and gradient of ... by two orders of magnitude by transitioning from CPUs to GPUs. Our reconstruction ran on a single NVIDIA A100 GPU with 80 GB of memory for about four weeks.

Needless to say, this 4 week run was only one of the very many runs to actually produce the final result.

The result is a 3D dust map

achieving an angular resolution of ${14'}$ ($N_\text{side}=256$). We sample the dust extinction in 516 distance bins spanning 69 pc to 1250 pc. We obtain a maximum distance resolution of 0.4pc at 69pc and a minimum distance resolution of 7pc at 1.25 kpc.

3. Solution. Initially a Pandoc approach was tried. Pandoc and all its dependencies on Arch Linux needs more than half GB (Gigabyte!) of space, just for the installation. After installation the Pandoc approach even failed.

Perl, the workhorse, had to do the job again. For the conversion I created two Perl scripts:

1. blogparsec: converts main.tex, i.e., the actual paper
2. blogbibtex: converts the Bibtex-formatted file literature.bib

Using those two script, creating the Markdown file goes like this:

blogparsec main.tex > 08-03-a-parsec-scale-galactic-3d-dust-map-out-to-1-25-kpc-from-the-sun.md
blogbibtex literature.bib >> 08-03-a-parsec-scale-galactic-3d-dust-map-out-to-1-25-kpc-from-the-sun.md

This file still needs some manual editing. One prominent case is moving the table-of-content to the top, as this is appended at the end.

4. blogparsec script. Some notes on this Perl script. The input to this script is the actual LaTeX text with all the formulas etc.

First define some variables and use strict mode.

#!/bin/perl -W
# Convert paper in "Astronomy & Astrophysics" LaTeX format to something resembling Markdown
# Manual post-processing is still necessary but a lot easier

use strict;
my ($ignore,$sectionCnt,$subSectionCnt,$replaceAlgo,$replaceTable) = (1,0,0,0,0);
my (@sections) = ();

The frontmatter header is a simple here-document:

print <<'EOF';
---
date: "2023-08-03 14:00:00"
title: "A Parsec-Scale Galactic 3D Dust Map out to 1.25 kpc from the Sun"
description: "A 3D map of the spatial distribution of interstellar dust extinction out to a distance of 1.25 kpc from the Sun"
MathJax: true
categories: ["mathematics", "astronomy"]
tags: ["interstellar dust", "interstellar medium", "Milky Way", "Gaia", "Gaussian processes", "Bayesian inference"]
---

EOF

The main loop looks at each line in main.tex. After the loop the literature section is added, then all sections collected so far are printed.

while (<>) {
    $ignore = 0 if (/\\author\{Gordian~Edenhofer/);
    next if ($ignore);

    (...)

    print;

    print "\$\$\n" if (/(\\end\{equation\}|\\end\{align\})/);	# enclose with $$ #2
}


print "## Literature<a id=Literature></a>\n";
for (@sections) {
    print $_ . "\n";
}
++$sectionCnt;
print "- [$sectionCnt. Literature](#Literature)\n";

What follows is the part which is marked as (...) in above code.

Here is the special case for processing algorithm and tables in the paper: the algorithm is simply a screenshot of the original PDF, the table is a here-document:

    # In this particular case we replace the two algorithms with a corresponding screenshot
    if (/^\\begin\{algorithm/) {
        $replaceAlgo = 1;
        next;
    } elsif (/^\s+Pseudocode for ICR creating a GP/) {
        s/^(\s+)//;
        s/(\\left|right)\\/$1\\\\/g;	# probably MathJax bug
        $replaceAlgo = 0;
        print "![](*<?=\$rbase?>*/img/parsec_res/Algorithm1.webp)\n\n";
    } elsif (/^\s+Pseudocode for our expansion point variational/) {
        s/^(\s+)//;
        $replaceAlgo = 0;
        print "![](*<?=\$rbase?>*/img/parsec_res/Algorithm2.webp)\n\n";
    } elsif ($replaceAlgo == 1) { next; }

    if (/^\\begin\{table/) {
        $replaceTable = 1;
        next;
    } elsif (/^\\end\{table/) {
        $replaceTable = 0;
        print <<'EOF';

Parameters of the prior distributions.
The parameters $s$, $\mathrm{scl}$, and $\mathrm{off}$ fully determine $\rho$.
They are jointly chosen to a prior yield the kernel reconstructed in [Leike2020][].



 Name | Distribution | Mean | Standard Deviation | Degrees of Freedom
 -----|--------------|------|--------------------|--------------------
_s_   | Normal       | 0.0  | Kernel from [Leike2020][] | 786,432 &times; 772
scl   | Log-Normal   | 1.0  | 0.5                |  1
off   |  Normal      | $-6.91\left(\approx\ln10^{-3}\right)$ <br>prior median extinction <br>from [Leike2020][] | 1.0 | 1
      |              |      | Shape Parameter    | Scale Parameter  
$n_\sigma$ | Inverse Gamma | 3.0 | 4.0 | #Stars = 53,880,655

EOF
        next;
    } elsif ($replaceTable == 1) { next; }

The header with its authors and institutions needs some extra handling:

s/^\\(author|institute)\{/\n<p>\u$1s:<\/p>\n\n1. /;

s/\~/ /g;

# Authors, institutions, abstract, etc.
s/\(\\begin\{CJK\*.+?CJK\*\}\)//;
s/\\inst\{(.+?)\}/ \($1\)/g;
if (/^\s+\\and/) { print "1. "; next; }
s/^\{% (\w+) heading \(.+$/\n\n_\u$1._ /;
s/^\\abstract/## Abstract/;
s/^\\keywords\{/__Key words.__ /;

Many lines simply are no longer needed in Markdown and therefore dopped:

# Lines to drop, not relevant
next if (/(^\\maketitle|^%\s+|^%In general|^\\date|^\\begin\{figure|^\\end\{figure|\s+\\centering|\s+\\begin\{split\}|\s+\\end\{split\}|^\s*\\label|^\\end\{acknowledgements\}|^\\FloatBarrier|^\\bibliograph|^\\end\{algorithm\}|^\\begin\{appendix|^\\end\{appendix\}|^\\end\{document\})/);

s/\s+%\s+[^%].+$//;	# Drop LaTeX comments
s/\\fnmsep.+$//;	# drop e-mail

Display math is enclosed in double dollars:

print "\$\$\n" if (/(\\begin\{equation\}|\\begin\{align\})/);	# enclose with $$a #1

Images are replaced with the usual Markdown code ![]():

# images
s/\s+\\includegraphics.+res\/(\w+)\}/!\[Photo\]\(\*<\?=\$rbase\?>\*\/img\/parsec_res\/$1\.png)/;
s/\s+\\subcaptionbox\{(.+?)\}\{\%/\n__$1__\n/g;

Some LaTeX macros are not present in MathJax and therefore need to be replaced.

# MathJax doesn't know \nicefrac
s/\\nicefrac\{(.+?)\}\{(.+?)\}/\{$1\}\/\{$2\}/g;
s/\\coloneqq/:=/g;	# MathJax doesn't know \coloneqq + \argmin + \SI
s/\\argmin/\\mathop\{\\hbox\{arg min\}\}/g;
s/\\SI(|\[parse\-numbers=false\])\{(.+?)\}/$2/g;
s/\\SIrange\{(.+?)\}\{(.+?)\}\{(|\\)([^\\]+?)\}/$1 $4 to $2 $4/g;
s/\\nano\\meter/nm/g;
s/\{\\pc\}/pc/g;
s/\{\\kpc\}/kpc/g;
s/(kpc|pc)\$/\\\\,\\hbox\{$1\}\$/g;
s/\{\\cubic\\pc\}/\\\\,\\hbox\{pc\}^3/g;

What looks good in LaTeX does not necessarily look good in Markdown:

s/i\.e\.\\ /i.e., /g;

# Special cases
s/``([A-Za-z])/"$1/g;	# double backquotes in LaTeX have an entirely different meaning than in Markdown

More MathJax specialities:

# These are probably MathJax bugs, which we correct here
s/\$\\tilde\{Q\}_\{\\bar\{\\xi\}\}\$/\$\\tilde\{Q\}\\_\{\\bar\{\\xi\}\}\$/g;
s/\$\\mathcal\{D\}_/\$\\mathcal\{D\}\\_/g;
s/\$P\(d\|\\mathcal\{D\}_/\$P\(d\|\\mathcal\{D\}\\_/g;
s/\$\\mathrm\{sf\}_/\$\\mathrm\{sf\}\\_/g;

Various LaTeX text-macros:

s/\\url\{(.+?)\}/$1/g;	# Markdown automatically URL-ifies URLs, so we can dispense \url{}

# Thousands separator, see https://stackoverflow.com/questions/33442240/perl-printf-to-use-commas-as-thousands-separator
s/\\num\[group-separator=\{,\}\]\{(\d+)\}/scalar reverse(join(",",unpack("(A3)*", reverse int($1))))/eg;

# Code
s/\\lstinline\|(.+?)\|/`$1`/g;
s/\\texttt\{(.+?)\}/`$1`/g;
s/quality\\_flags\$<\$8/quality_flags<8/g;	# special case

# Special cases for preventing code blocks because of indentation
s/   (The angular resolution)/$1/;
s/   (The stated highest r)/$1/;

Section and subsection headers become ## and ### in Markdown:

# sections + subsections
if (/\\section\{(.+?)\}\s*$/) {
    my $s = $1;
    ++$sectionCnt; $subSectionCnt = 0;
    push @sections, "- [$sectionCnt. $s](#s$sectionCnt)";
    $_ = "\n## $sectionCnt. $s<a id=s$sectionCnt></a>\n";
} elsif (/\\subsection\{(.+?)\}\s*$/) {
    my $s = $1;
    ++$subSectionCnt;
    push @sections, "\t- [$sectionCnt.$subSectionCnt $s](#s${sectionCnt}_$subSectionCnt)";
    $_ = "\n### $sectionCnt.$subSectionCnt $s<a id=s${sectionCnt}_$subSectionCnt></a>\n";
}

For footnotes I used block quotes in Markdown.

if (/(\\footnotetext\{%|^\\begin\{acknowledgements\})/) { print "> "; next; }

I fought a little bit with citations and initially had something like:

# Citations
#s/\\citep(|\[.*?\]\[\])\{(\w+)\}/'('.(length($1)>4?substr($1,1,-3).' ':'').'['.join('], [',split(',',$2)).'][])'/eg;
# First approach, now obsolete through eval()-approach
#s/\\citep\{(\w+)\}/([$1][])/g;
#s/\\citep\{(\w+),(\w+)\}/([$1][], [$2][])/g;
#s/\\citep\{(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][])/g;
#s/\\citep\{(\w+),(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][], [$4][])/g;
#s/\\citep\{(\w+),(\w+),(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][], [$4][], [$5][])/g;
#s/\\citep\{(\w+),(\w+),(\w+),(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][], [$4][], [$5][], [$6][])/g;
#s/\\citep\{(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][], [$4][], [$5][], [$6][], [$7][])/g;
#s/\\citep\{(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][], [$4][], [$5][], [$6][], [$7][], [$8][])/g;
#s/\\citep\{(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][], [$4][], [$5][], [$6][], [$7][], [$8][], [$9][])/g;
#s/\\citep\{(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+),(\w+)\}/([$1][], [$2][], [$3][], [$4][], [$5][], [$6][], [$7][], [$8][], [$9][], [$10][])/g;
#s/\\citet\{(\w+)\}/[$1][]/g;

Luckily this can be handled by eval in regex, i.e., watch out for the s///eg, the e is important:

s!\\citep\{([,\w]+)\}!'(['.join('][], [',split(/,/,$1)).'][])'!eg;	# cite-paranthesis without any prefix text
s!\\citep\[(.+?)\]\[\]\{(\w+)\}!'('.$1.' ['.join('][], [',split(/,/,$2)).'][])'!eg;	# citep with prefix text
s!\\(citet|citeauthor)\{([,\w]+)\}!'['.join('][], [',split(/,/,$2)).'][]'!eg;	# we handle citet+citeauthor the same

During development of this Perl script I used Beyond Compare quite intensively, to compare the original against the changed file.

5. blogbibtex script. The input to this script is the Bibtex file with all literature references. The Bibtex file looks something like this:

@book{Draine2011,
  author  = {{Draine}, Bruce T.},
  title   = {{Physics of the Interstellar and Intergalactic Medium}},
  year    = 2011,
  adsurl  = {https://ui.adsabs.harvard.edu/abs/2011piim.book.....D},
  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
@article{Popescu2002,
  author        = {{Popescu}, Cristina C. and {Tuffs}, Richard J.},
  title         = {{The percentage of stellar light re-radiated by dust in late-type Virgo Cluster galaxies}},
  journal       = {\mnras},
  keywords      = {galaxies: clusters: individual: Virgo Cluster, galaxies: fundamental parameters, galaxies: photometry, galaxies: spiral, galaxies: statistics, infrared: galaxies, Astrophysics},
  year          = 2002,
  month         = sep,
  volume        = {335},
  number        = {2},
  pages         = {L41-L44},
  doi           = {10.1046/j.1365-8711.2002.05881.x},
  archiveprefix = {arXiv},
  eprint        = {astro-ph/0208285},
  primaryclass  = {astro-ph},
  adsurl        = {https://ui.adsabs.harvard.edu/abs/2002MNRAS.335L..41P},
  adsnote       = {Provided by the SAO/NASA Astrophysics Data System}
}

The Perl script has some journal names preloaded:

#!/bin/perl -W
# Convert BibTeX to Markdown. Produce the following:
#    1. List of URL targets
#    2. Sorted list of literature entries

use strict;
my ($inArticle,$entry,$entryOrig,$type) = (0,"","");
my %H;	# hash of hash (each element in hash is a yet another hash)
my %Journals = (	# see http://cdsads.u-strasbg.fr/abs_doc/aas_macros.html
    '\aap'   => 'Astronomy & Astrophysics',
    '\aj'    => 'Astronomical Journal',
    '\apj'   => 'The Astrophysical Journal',
    '\apjl'  => 'Astrophysical Journal, Letters',
    '\apjs'  => 'Astrophysical Journal, Supplement',
    '\mnras' => 'Monthly Notices of the RAS',
    '\nat'   => 'Nature'
);

The actual loop populates the hash %H:

while (<>) {
    if (/^@(article|book|inproceedings|misc|software)\{(\w+),$/) {
        ($type,$entry,$entryOrig,$inArticle) = ($1,uc $2,$2,1);
        $H{$entry}{'entry'} = $entryOrig;
        $H{$entry}{'type'} = $type;
        #printf("\t\tentry = |%s|, type = |%s|\n",$entry,$type);
    } elsif ($inArticle) {
        if (/^}\s*$/) { $inArticle = 0; next; }
        if (/^\s+(\w+)\s*=\s*(.+)(|,)$/) {
            my ($key,$value) = ($1,$2);

            # LaTeX foreign language character handling
            $value =~ s/\{\\ss\}/ß/g;
            $value =~ s/\{\\"A\}/Ä/g;
            $value =~ s/\{\\"U\}/Ü/g;
            $value =~ s/\{\\"O\}/Ö/g;
            $value =~ s/\{\\"a\}/ä/g;
            $value =~ s/\{\\"u\}/ü/g;
            $value =~ s/\{\\"i\}/ï/g;
            $value =~ s/\{\\H\{o\}\}/ő/g;
            $value =~ s/\{\\"\\i\}/ï/g;
            $value =~ s/\{\\"o\}/ö/g;
            $value =~ s/\{\\'A\}/Á/g;	# accent aigu
            $value =~ s/\{\\'E\}/É/g;	# accent aigu
            $value =~ s/\{\\'O\}/Ó/g;	# accent aigu
            $value =~ s/\{\\'U\}/Ú/g;	# accent aigu
            $value =~ s/\{\\'a\}/á/g;	# accent aigu
            $value =~ s/\{\\'e\}/é/g;	# accent aigu
            $value =~ s/\{\\'o\}/ó/g;	# accent aigu
            $value =~ s/\{\\'u\}/ú/g;	# accent aigu
            $value =~ s/\{\\`a\}/à/g;	# accent grave
            $value =~ s/\{\\`e\}/è/g;	# accent grave
            $value =~ s/\{\\`u\}/ù/g;	# accent grave
            $value =~ s/\{\\^a\}/â/g;	# accent circonflexe
            $value =~ s/\{\\^e\}/ê/g;	# accent circonflexe
            $value =~ s/\{\\^i\}/î/g;	# accent circonflexe
            $value =~ s/\{\\^\\i\}/î/g;	# accent circonflexe
            $value =~ s/\{\\^o\}/ô/g;	# accent circonflexe
            $value =~ s/\{\\^u\}/û/g;	# accent circonflexe
            $value =~ s/\{\\~A\}/Ã/g;	# minuscule a
            $value =~ s/\{\\~a\}/ã/g;	# minuscule a
            $value =~ s/\{\\~O\}/Õ/g;	# minuscule o
            $value =~ s/\{\\~o\}/õ/g;	# minuscule o
            $value =~ s/\{\\~n\}/ñ/g;	# palatal n
            $value =~ s/\{\\v\{C\}/Č/g;	# grapheme C
            $value =~ s/\{\\v\{c\}/č/g;	# grapheme c
            $value =~ s/\{\\v\{S\}/Š/g;	# grapheme S
            $value =~ s/\{\\v\{s\}/š/g;	# grapheme s
            $value =~ s/\{\\v\{Z\}/Ž/g;	# grapheme Z
            $value =~ s/\{\\v\{z\}/ž/g;	# grapheme z
    
            $value =~ s/\{|\}|\~//g;	# drop {}~
            $value =~ s/,$//;	# drop last comma
            $H{$entry}{$key} = $value;
            #printf("\t\t\tentry = |%s|, key = |%s|\n", $entry, $key);
        }
    }
}

Once everything is loaded into the hash, the hash is printed out in formatted form.

print("\n");
for my $e (sort keys %H) {
    my $He = \%H{$e};
    my $url = 
    printf("[%s]: %s\n", $H{$e}{'entry'},
        exists($H{$e}{'doi'}) ? 'https://doi.org/'.$H{$e}{'doi'}
        : exists($H{$e}{'url'}) ? $H{$e}{'url'} : '#Literature');
}
print("\n");

for my $e (sort keys %H) {
    my ($He,$date,$journal) = (\$H{$e},"","");
    if (exists($$He->{'year'}) && exists($$He->{'month'}) && exists($$He->{'day'})) {
        $date = sprintf("%02d-%s-%d", $$He->{'year'}, $$He->{'month'}, $$He->{'day'});
    } elsif (exists($$He->{'year'}) && exists($$He->{'month'})) {
        my $m = $$He->{'month'};
        $date = "\u$m" . "-" . 	$$He->{'year'};
    } elsif (exists($$He->{'year'})) {
        $date = $$He->{'year'};
    }
    if (exists($$He->{'journal'})) {
        my $t = $$He->{'journal'};
        $journal = ", " . ((substr($t,0,1) eq '\\') ? $Journals{$t} : $t);
        $journal .= ", Vol. " . $$He->{'volume'} if (exists($$He->{'volume'}));
        $journal .= ", Nr. " . $$He->{'number'} if (exists($$He->{'number'}));
        $journal .= ", pp. " . $$He->{'pages'} if (exists($$He->{'pages'}));
    }

    printf("1. \\[%s\\] %s: _%s_, %s%s%s\n", $H{$e}{'entry'}, $H{$e}{'author'},
        defined($H{$e}{'title'}) ? $H{$e}{'title'} : $H{$e}{'howpublished'},
        $date, $journal,
        exists($H{$e}{'doi'}) ? ', https://doi.org/'.$H{$e}{'doi'}
        : exists($H{$e}{'url'}) ? ', ' . $H{$e}{'url'} : ''
    );
}

The output of this blogbibtex script is then appended to the output of the previous script blogparsec.

6. Open issues. I had already worked for two days on these two Perl scripts and wanted to finish it. Therefore the following topics are not adressed but can be solved quite easily.

1. There are still some stray curly braces, which should be removed.
2. Back and forward references, i.e., all these still visible \Cref tags should be converted using link references in Markdown.
3. LaTeX table were converted manually, should be fully automatic.
4. Converting the \begin{algorithm} and \end{algorithm} probably is a lot trickier, as it needs extra CSS to work properly.

1. Original: WordPress version
2. Modified: Simplified Saaze version of Ristorante Panorama

The original website is hosted by Strato. It uses WordPress and Elementor.

1. Comparison. For the comparison I use the website tools.pingdom.com which provides various metrics to evaluate the performance of a website:

1. Page size
2. Number of requests
3. Load time
4. Concrete tips to improve performance
5. Waterfall diagram of requests
6. Breakdown of content types

All tests in Pingdom were conducted for Europe/Frankfurt.

The results are thus:

Original (WordPress)	Modified (Simplified Saaze)

The results for the original website are indeed very bad on every dimension: page size, load time, number of requests. In comparison to the modified version using Simplified Saaze the ratio is roughly:

1. Page size is more than 10:1
2. Load time is almost 8:1
3. Number of requests is 8:1

So Simplified Saaze is better in all dimensions by a large factor. This is particular striking as the Simplified Saaze version is entirely self-hosted, i.e., upload to the internet is 50 MBit/s!

The recommendations for the original website are therefore not overly surprising:

The missing compression is clearly an oversight on the web-server part.

The breakdown of the content type for the original website is:

2. Modified website. The website powered by Simplified Saaze is still very image heavy, but there is no JavaScript, there are no fonts, or megabytes of CSS. The breakdown of the modified site is as below.

Actual loading of the modified site will roughly follow below waterfall diagram. This waterfall diagram shows that the images can all be loaded in parallel, while the actual HTML is one of the prime factors for the overall request time. Also, all images are way smaller, as they all have been converted to WebP format.

3. Competitor. There is another Italian restaurant in town, Ristorante Bella Vista. Pingdom values are as below.

Content type breakdown is:

The good request times can be attributed to:

1. Jpeg images have been scaled down to make them small
2. Less than 100 KB of JavaScript
3. No webfonts

The Bella Vista website is hosted by Hetzner. It uses Weblication CMS, which is also based on PHP like WordPress.

1. It is geared towards restaurants with menus
2. Responsive with media-breaks for 1-column, 2-column, 3-column, and printer output
3. RSS and sitemap
4. Showcase for post-processing, if needed
5. Hero image
6. Background SVG image
7. Animated images and galleries
8. Lightweight and easy to use

Its source code is in GitHub: saaze-panorama.

Here is a screenshot:

The original website uses WordPress, Elementor, and Google Site Kit. The original website has a number of major shortcomings:

1. Terribly slow
2. Loading web-fonts, which are not used
3. Loading images, which are not used
4. Duplicated text on a single webpage
5. RSS feed empty
6. Google indexing disabled

In addition there are various minor glitches:

1. Misspellings
2. Navigation mishaps: redirecting to same page
3. Color contrast sometimes bad: green text on black background
4. favicon icon too small to be human-readable

This theme is the eighth example theme. We had themes migrated from WordPress, from Hugo, from Jekyll. This time again a migration from WordPress with Elementor.

2. Creating restaurant menus with post-processing. A restaurant obviously wants to show its menu. This is done as follows:

## 2. Kalte und warme Vorspeisen<a id=vorpeisen></a>

- Antipasti dela Casa ...... klein €9,50 - groß €12,50
    - mit gegrilltem Gemüse und Fisch
- Shrimps Cocktail `1,b,c,d,n` ...... € 9,50
    - mit Shrimps, Ananas
- Shrimps mit Olivenöl, Cocktailtomaten `1,b,c,n` ...... €12,50
    - mit Knoblauch & Schalotten
- Gebackener Schafskäse `1,b,c,d,n` ...... €10,50
    - mit Tomaten, Oliven und Peperoni

So data entry closely mirrors the output, which looks like this:

The CSS for this "dot-trick" can be found here: Dot Leaders by Bert Bos.

The frontmatter of the Markdown looks like this, indicating that it wants its output to be processed further:

---
title: "Speisekarte"
date: "2023-07-11 21:00:00"
excerpt: "Italienisch-mediterrane Köstlichkeiten, ausgewählte deutsche Spezialitäten, Steaks und Fisch."
heroimg: "Schweinefleisch.webp"
postproc: true
---

Above frontmatter also shows how the hero-image is defined.

The actual post-processing is done in the template-file entry.php:

<?php require SAAZE_PATH . "/templates/top-layout.php"; ?>

    <main>
    <article class=aentry>
<?php
    if (!function_exists('postproc')) {
        // Post-processing of MD4C processed Markdown, not really clean,
        //because probably specific to MD4C, but does the job
        function postproc(string $s) : string {
            //return $s;
            $s = str_replace(
                array(PHP_EOL.'<ul>',
                    PHP_EOL.'<li><p>',
                    '</p>'.PHP_EOL.'</li>',
                    '<ul>'.PHP_EOL,
                    'class=leaders>'.PHP_EOL.'<li><code>'),
                array(PHP_EOL.'<ul class=leaders>',	// add class=leaders to ul
                    PHP_EOL.'<li>',	// strip <p> after <li>
                    '</li>',	// strip </p> before </li>
                    '<ul class=noleaders>'.PHP_EOL,	// 2nd ul must not have leaders but noleaders
                    'class=noleaders>'.PHP_EOL.'<li><code>'),	// Allergene Sonderfall
                $s);
            // replace ABC ...... UVW with ABC+UVW each enclosed in span's
            // catchword is six dots
            return preg_replace(
                '/(' . PHP_EOL . '<li>)(.+)\s+\.\.\.\.\.\.\s+(.+)(<ul|<\/li>)/',
                '$1<span>$2</span><span>$3</span>$4',
                $s
            );
        }
    }
    echo '<h1>' . $entry['title'] . "</h1>\n";
    if (isset($entry['heroimg']))
        printf("<p><img class=heroimg src=\"%s/img/%s\" alt=\"Hero image\"></p>\n",$rbase,$entry['heroimg']);
    $s = ($entry['postproc'] ?? false) ? postproc($entry['content']) : $entry['content'];
    echo $s;


?>
    </article>
    </main>

The post-processing effectively just search-and-replaces certain strings, in our case six dots.

If this theme also wants to mix PHP into Markdown then replace above echo $s with below three PHP lines.

    $s = str_replace('*%3c?','<?',$entry['content']);
    $s = str_replace('?%3e*','?>',$s);
    require 'data:text/plain;base64,'.base64_encode($s);

If you omit above post-processing PHP function postproc() from above template, the template would be pretty simple.

3. Installation. The theme including Simplified Saaze is installed by using composer:

composer create-project eklausme/saaze-panorama

This installs below directory tree:

saaze-panorama
|-- LICENSE
|-- README.md
|-- composer.json
|-- composer.lock
|-- content
|   |-- auxil
|   |   |-- datenschutzerklaerung.md
|   |   `-- impressum.md
|   |-- auxil.yml
|   |-- blog
|   |   |-- aktuell.md
|   |   |-- biergarten.md
|   |   |-- catering.md
|   |   |-- feiern.md
|   |   |-- mittagstisch.md
|   |   |-- pfifferlinge.md
|   |   |-- ristorante.md
|   |   `-- speisekarte.md
|   `-- blog.yml
|-- public
|   |-- img
|   |   |-- Aussenbereich1.jpg
|   |   |-- Aussenbereich1.webp
|   |   |-- Aussenbereich2.jpg
|   |   |-- . . .
|   |   `-- green-orange-and-yellow-pasta-165844-2000x1200-1.webp
|   `-- index.php
|-- saaze
|-- templates
|   |-- bottom-layout.php
|   |-- entry.php
|   |-- error.php
|   |-- head.php
|   |-- index.php
|   |-- overview.php
|   |-- rss.php
|   |-- sitemap.php
|   `-- top-layout.php
`-- vendor
    |-- autoload.php
    |-- composer
    |   |-- ClassLoader.php
    |   |-- InstalledVersions.php
    |   |-- LICENSE
    |   |-- autoload_classmap.php
    |   |-- autoload_namespaces.php
    |   |-- autoload_psr4.php
    |   |-- autoload_real.php
    |   |-- autoload_static.php
    |   |-- installed.json
    |   |-- installed.php
    |   `-- platform_check.php
    `-- eklausme
        `-- saaze
            |-- BuildCommand.php
            |-- Collection.php
            |-- CollectionArray.php
            |-- Config.php
            |-- Entry.php
            |-- LICENSE
            |-- MarkdownContentParser.php
            |-- README.md
            |-- Saaze.php
            |-- SaazeCli.php
            |-- TemplateManager.php
            |-- composer.json
            |-- php_md4c_toHtml.c
            `-- saaze

11 directories, 137 files

Here are two articles if you want to install Simplified Saaze on Windows:

1. Installing Simplified Saaze on Windows 10
2. Installing Simplified Saaze on Windows 10 #2

4. Building and deploying. Change to the directory saaze-panorama. The following commmand builds a static site.

$ time php saaze -morb /tmp/build
Building static site in /tmp/build...
        execute(): filePath=/home/klm/php/saaze-panorama/content/auxil.yml, nentries=2, totalPages=1, entries_per_page=20
        execute(): filePath=/home/klm/php/saaze-panorama/content/blog.yml, nentries=8, totalPages=1, entries_per_page=20
Finished creating 2 collections, 2 with index, and 10 entries (0.02 secs / 1.68MB)
#collections=2, YamlParser=0.0002/12-2, md2html=0.0004, MathParser=0.0003/10, renderEntry=10, content=10/0, excerpt=0/0
        real 0.04s
        user 0.01s
        sys 0
        swapped 0
        total space 0

As can be seen, build time is way below a tenth of a second on a Ryzen 7 5700G. In above scenario we use options -m for generating a sitemap, -o for generating an overview page, -r for generating RSS. Option -b is used to build in /tmp, which on Arch Linux is a RAM disk. Options m, o, and r are entirely optional. I.e., below command would do just as well.

php saaze

The resulting HTML files need to be uploaded to your web-server. Below are the steps to upload to a local web-server assuming you built into /tmp/build. A local web-server is a web-server running on the same machine where you generated the HTML files.

[ -d $DOCROOT ] && rm -rf $DOCROOT
[ -d /tmp/build ] || errorExit "No build directory in /tmp"
mv /tmp/build $DOCROOT

cd $DOCROOT
ln -s $SAAZEROOT/public/img

For local development of your website, you use:

php -S 0:8000 -t public/

This starts a web-server and you can immediately see any changes you make. Above command shows Simplified Saaze in dynamic mode. You can also use this dynamic mode with NGINX by using something like below:

server {
    rewrite "^/(aux|blog)($|/.*)"  "...your-directory.../index.php?/$1$2" last;
}

The dynamic mode of Simplified Saaze has the advantage that you don't need to build any static HTML files. All HTML files are generated on the fly. The disadvantage is that every request will rebuild the requested HTML page, unless you use intensive caching in your web-server.

5. CSS and favicon. The panorama-theme uses a SVG based background image. This was inspired by Matt Visiwig's page on SVG backgrounds. In our case we used "Subtle Prism". We already mentioned the dot-leader CSS for aligned lines of dots.

Generating the favicon was done using the web-page favicon-generator using the two letters 'R' and 'P' with circled background. The favicon is directly embedded into the head.php template. This helps to reduce to number of requests required for the browser to show the web-page.

<link href="data:image/png;base64,iVBORw....ABJRU5ErkJggg==" rel="icon" type="image/png">

I had written on this here: Accelerating Page Load Times by Reducing Requests, Part #2.

The three-column output is realized using CSS grids.

@media screen and (min-width:99rem) {	/* 3 column output */
    .aentry, header, aside, footer { width:var(--klmWidth) }
    .aindex { margin-left:0rem; width:20rem }
    .allcontent { max-width:var(--klmWidth); margin:auto; padding:0rem }
    .agrid-container {
        display:grid;
        justify-content:center;
        column-gap:2rem;
        grid-template-columns: auto auto auto;
        grid-template-areas: 'article article article';
    }
    /* https://www.w3docs.com/snippets/css/how-to-vertically-align-text-next-to-an-image.html */
    .imgcontainer { display:flex; align-items:center }
    .textimg { padding-left:2.5rem }
}

Printing to an old-fashioned printer is handled by a special media break:

@media print {
    h2 { page-break-before: always }
    h1, h2, h3, h4, h5, h6, ul, li, p { color:black }
}

Most notably this is for printing out the menu card. This is considered to be of some importance as you can now have a single source of truth: the menu on the web-page, and the printed menu from the web-page.

6. Home page / index. The index-page or landing page of this theme is somewhat special as it shows all blog posts, but singles out the newest one. This newest post is interesting as it might contain offers of the day, special announcements on opening hours or holidays, etc.

<?php
    if (count($pagination['entries']) > 0) {
        $entry = array_shift($pagination['entries']);	// 1st element, i.e., newest
        echo "<aside>\n" . $entry['content'] . "</aside>\n";
    }
?>

All other posts are handled as usual:

<?php foreach ($pagination['entries'] as $entry) { ?>
    <article class=aindex>
    <h2><a href="<?= $rbase . $entry['url'] ?>"><?= $entry['title'] ?? 'Unknown title' ?></a></h2>
<?php if (isset($entry['heroimg'])) { ?>
    <div class=ixImgContainer><a href="<?=$rbase.$entry['url']?>"><img class=ixImgZoomIn width=300 src="<?=$rbase?>/img/<?=$entry['heroimg']?>" alt=HeroImg></a></div>
<?php } ?>
    <p><?= $entry['excerpt'] ?? '---' ?></p>
    </article>
<?php } ?>

The talk is here:

Meritocracy, a ruling system based on merit:

the notion of a political system in which economic goods or political power are vested in individual people based on ability and talent, rather than wealth or social class. Advancement in such a system is based on performance, as measured through examination or demonstrated achievement.

At first sight this looks like that it will favor good outcome. Malcolm Gladwell analyzes a number of pitfalls.

Below is the transcript of the talk directly from YouTube:

It's a real pleasure to be invited back to Google's Zeitgeist. I think the last time I spoke at this was many, many years ago in Phoenix, and if memory serves, my talk was a critical examination of my decision to agree to talk at Google Zeitgeist. Incredibly, I got invited back, and I so I thought as an encore what I would do is do a critical examination of why all of you were invited to Google Zeitgeist. Now, there is a standard answer to that, which is that this is a gathering of the best and the brightest and all of you have reason to believe that you are the best and the brightest. But my question is: How do you know you're the best and the brightest? And what I want to suggest this morning is that there is a great deal of more uncertainty over that question than you may care to admit, and that paradoxically, this is a very good thing. So, I want to focus in the brief time that I have on the role of gatekeepers, because meritocracies of the sort that we've erected in our world are run by gatekeepers, and I would like to advance a series of propositions to suggest that gatekeepers are really, really bad at what they do. So, there is going to be four of these propositions.

Proposition 1. Gatekeepers very often do not understand the meritocracy that they are supposed to be keeping.

Proposition 1 is that gatekeepers very often do not understand the meritocracy that they're supposed to be keeping the watching the gates for. So, tons of examples, but the one I will focus on is the NIH, National Institute of Health. This is one of the most consequential meritocracies in the world, probably. NIH has a budget of 40 billion dollars a year. They get 80,000 grant applications a year, which represents an extraordinary percentage of the most crucial research we do in the world, and they put together groups of experts who grade each one of those grant applications on a scale of 10 to 90 where 10 is fantastic and 90 is terrible, right. So, this is a classic meritocracy guarded by a group of expert gatekeepers.

So, a couple years ago the Guy, the Deputy Director of extra research at NIH, the Guy running this process decades to try and verify how good the process is, right. So, when you do a score on a grant application, you're making a prediction of how good you think that research is going to turn out to be. So, his question was, well, how good are these predictions? He does a really simple analysis where medicine, the way we judge, the quality of research is how many citations are made to that research fund when finished. He says let's simply correlate the grant score on an application with a number of citations it gets once the research is finally finished. So, what does he discover? He discovers that the correlation between your score and how good your research ends up being is modest to nonexistent, right. Now, we're talking about one of the most crucial meritocracies in modern society. We're talking about $40 billion of intellectual activity, and the guy running the whole show takes a look and discovers the experts who are manning the gates to this particular meritocracy don't know what they're doing. So, why doesn't gate keeping work in this example? Well, one is maybe it's impossible to predict who is a good researcher and who isn't. That is impossible. Maybe it is the groups of experts by virtue of being experts belong to a particular generation of medical research and are hopelessly out of touch with what the next generation of research is supposed to be all about. It doesn't really matter. The point is that this is a meritocracy that is not a meritocracy, right. My favorite response to this guy, Dr. Lower's paper, a bunch of microbiologist published this paper where they said the only rational thing now is to tell all the grant reviewers to go home and shutdown that entire cumbersome process of trying to evaluate all these 80,000 grant applications, just have a big round cylindrical container, but all the applications in the container, and pick them out at random and that should be how we govern the grant process in this country. That strikes me as a system that makes a great deal of sense. Okay.

Proposition No. 2. Meritocracies don't work sometimes because they are run for the benefit of gatekeepers.

Proposition No. 2. Meritocracies don't work sometimes because they are run for the benefit of gatekeepers. Again, any examples.

The LSAT. I got so obsessed with the LSAT a couple years ago I took it. I challenged my assistant to an LSAT contest. So, we all mow about the LSAT. It's six sections, you know, reading, problem solving, logic problems, I've forgotten the others, writing. You get 35 minutes for each section, and your score determines whether you get into an elite law school, and whether you get into an elite law school determines whether you get an elite job once you graduate and whether you get an elite job is a job on the Supreme Court and an invitation to Zeitgeist. You make a distinction between power tests and speed tests. So, a speed test is where I give you a whole lot of relatively easy questions, and I'm interested to see how many you can answer in a given amount of time, right. So, video games are really very often speed tests, right. We play for constraint and see how well you do under that constraint. Power test is where I give you really hard questions, and all I'm really interested in is how many of those questions you can get right. So, scrabble tends to be, is really a power game. Untimed chess is the power game. So, what is the LSAT? Well, the LSAT is a series of very, very hard questions, but if we require that the test taker complete them in a limited period of time. And the time constraint is so strict that it's deliberately strict because we want to make it so hard. We want to make it hard enough that the average test taker cannot answer all the questions in the allotted time. So, what we have here is what a psycho matrician would call a speeded power test. We're collecting power data with a speed constraint. Here is the question: Why do we collect it with a speed constraint? Why is there a 35 minute limit on the six sections of the LSAT? So, take a look at this. This is a† first slide. Here we go. We have two test takers here. We have tortoise and hare. Hare, we all know hare is super speedy, very confident. He answers every one of the 101 questions on the LSAT in the time allotted. He gets 82 of them correct for an accuracy rate of 81.2, and he has an LSTAT score of 165 which puts him in the 94th-percentile, gets a job at Annie law firm, works 80 hours a week, he never sees his kids, his marriage falls apart.

Tortoise, by contrast, allow me to say tortoise is a woman, for no particular woman. But she is super analytic. She doesn't do things quickly. She, whenever she has a hard question, she goes over it 17 times. There is no way tortoise can finish all of the answers to all the questions in the time allotted, so she only gets to 80 questions of which she answers 78 correctly. She has an accuracy rate of 97.5, but she makes so many guesses that she gets penalized. She ends up with an LSAT score of 165, which is the 94th-percentile. She gets a job in Annie law firm, works 80 ours a week, never sees her children, her marriage falls apart and she quits law.

Okay. The LSAT will tell you that these two people hare and tortoise are identical. They both got a score of 165, but who would you rather have as your lawyer? Would you like to call up hare and say have you gotten to my contract yet and hare tell you yeah, I looked at it over lunch, it's fine, right. No.

You want tortoise as your lawyer. You want the person who is an el and doesn't skip ahead, right. But the purpose† the result of creating a speeded power test for the LSAT, the result of having that time constraint is to make hare look better than he is and tortoise look worse than she is actually is, right. Why would we do this in a profession that is based on tortoise thinking? I was so baffled by this I went to see the organization that runs the LSAT, but fancy office build anything New Jersey, huge conference room. They all gathered, and I just said to them, can you explain to me why you have a time limit on the LSAT? Makes no sense. Why not just let them spend all day doing it, right. Ask really hard questions. That is what the law is, hard questions, we take a long time. They charge by the hour for goodness sake. There is no institutional reason why you would want people to move quickly, right. And I had my taperecorder all ready, because I was expecting them to give this long-nuanced answer and their answer was, nah, just easier. How you going to rent the hall for the whole day? All right.

Proposition 3. Meritocratic systems often do not recognize that being real good is not an individual effort, but a team accomplishment.

Proposition 3. This is a crucial one. Particularly for the kind of intellectual work that we do in the modern economy. This is about surgeons. Now, we're all familiar with the observation that the more operations a surgeon does, the more procedures they do, the better they get. There is a learning curve with surgery. That is why we tend to have rare surgeries clustered at major teaching hospitals so we can keep the volume of the surgeon up really high, right. You don't go and get some kind of very particular brain surgery at some rural hospital, you go to the major medical center for this very reason. So, this is a chart demonstrating this. Nor wood operations are very difficult pediatric card dean surgery, and you can see the learning curve with your† if you are under like 150 cases a year, your mortality rate is really high. It's terrible. But once you get to about 400 a year, the mortality rate comes down dramatically by, you know, it's a quarter of what it would be otherwise. This is a pattern that we see throughout all of surgery, and the people who are on this end, right, are the ones who get rewarded, theyíre the ones who make the most money, the ones that have the fan east title, those are the ones that are the winners of the bureaucratic game that is academic medicine. Okay.

But there is a complication with this, and that is what happens to the people, the surgeons who do their Norwood operations at a different hospital? So, lots of surgeons do this, right. You have privileges at more than one hospital. Maybe you do 90% of your procedures at one place, but then you go down the treat or across town or the weekends or whatever and do some at another place. And the answer is that when you leave your regular hospital and moonlight somewhere else, you move from being at this end of the curve and you go all the way back to the other end of the curve. This is a result beautifully demonstrated in this paper. I am going to read to you the conclusion. "Higher volume in a prior period for a given surgeon at a particular hospital is correlated with significantly lower risk adjusted mortality for that surgeon hospital pair." That is what they're talking about. That volume, however, does not significantly improve the surgeon's performance at other hospitals. What does that mean? Well, what that means is that card deat surgery, or any kind of complex surgery is a team activity, right. So, when you are with your team at your regular hospital, you all get better together, but then when you leave on the weekend to moonlight somewhere else, you leave your team behind, and without your team, you're hopeless, right. Now, does the meritocratic system recognize that being a real good surgeon is not an individual accomplishment, but a team accomplishment? No, it doesn't, right. The whole meta contract particular system is based on the assumption that what we're observing here is the greatness of this particular individual surgeon. Now, I would suggest to you that is a pretty big problem, particularly if you are someone who picks your elite surgeon and just happens to be seeing that surgeon at the hospital they're moonlighting at and not their regular place. And I think that this applies to an extraordinary number of complex allegedly marrow contract particular systems. I mean, think about me up here right now. How much of this talk is me? Do you know whether I wrote or whether the team wrote it, right? You don't know. You have no idea how good I am based on this particular talk that I'm giving you without knowing the actual process that I use to come up with these observations. Okay.

Proposition 4. Meritocracies are bad because gatekeepers don't fix them.

Proposition No. 4. Meritocracies are bad because gatekeepers don't fix them. Once you realize there is Anna accumulating body of knowledge that suggests we're not very good at managing meritocracies, then you would assume then that there should be an ongoing process by which we try and improve the quality of the gate keeping function, and it turns out that there isn't. So, again, a million examples, but this is one I've been obsessed with for a while. I wrote about it in my book Outliers in 2008. This is the roster of the 2007 Medicine Hat Tigers. This is the actual chart I used in my book Outliers, and this is a major junior hockey team, so this is one rung below the NHL. The point of this, those who read Outliers will know this, the point of this chart is this is a group of elite hockey players in a country that takes hockey very seriously, and what is most striking about this group are how many of them are born in the first four months of the year. January, January, March, April, September, October. April, and January, January, August, March, May, January, right. Now, this is a very, very well known phenomenon, it's called a relative age effect, and it's a function of the way in which we select† the way in which we structure the particular meritocracy that is elite youth sports. We, in Canada, they're crazy about hockey, so they start forming Al star teams at the age of nine, and at the age of nine, the kids who look like the best hockey players are the ones who are relatively the oldest. If you are born in January, you're going to look better than a kid born in December. So, we take that kid out and put them on an all-star team and give them way more practice time, way better coaches, way more access to good competition, way more encouragement and lo and behold ten years later they are the best. An arbitrary advantage has been elevated to a real advantage. You can see this everywhere. It's true in soccer, basketball, swimming. Any competitive sport that looks to develop and identify talent at an early level, early age, has the problem of creating these relative age effect arbitrary damages. For example, look at this. Schools. This is a study of gifted and talented programs in England, and they have broken down the composition of gifted and talented programs by birth cohort. In Inc. grand is September 1st. If you are in a relatively cohort in your class, your chance of being in a gifted science and math is roughly half of those who are born in the relatively oldest cohort. Basically, if your kid is among the youngest in your class, kiss good bye getting into a gifted and talented program. And of course, we use those to decide who gets into quality schools and we use quality schools to determine who gets advances, on and so. It's the same old system. This is not a meritocracy, something pretending to be a meritocracy. I wrote my book in 2008, and as a result of the long stuff about the effect of my book, there was a public attention to this particular relative age effect. I thought when I was coming here to talk about meritocracies, what I would do was revisit the hockey example and give you† show you about how this particular Canadian institution, Canadian is important to me, has learned its lesson and fixed its ways and no longer pursues a policy that has the misfortune of leading half the talent on the table. So, I decided I would look at the 2022 Canadian junior hockey roster, and let's just go through the birth months, shall we. September, November, June, March, January, January, April, September, January, August, September, July, October, January, February, January, August, April, October, January, March, January, March, February, January. They have learned nothing, right.

They haven't done a single thing to fix the problem. 15 years ago, it was brought to their attention that a country that was more passionate about hockey than almost anything else had created this system that was arbitrarily leaving half the talent on the table, right. This is† no one could possibly be more powerfully motivated to fix this system than Canadians, right. Hockey is the national everything. Have they fixed it? No, they haven't. By the way, has anyone fixed the system? No. Think about your child's elementary school. If you in first and Second Grade, do they divide the kids up and put the January to March kids in one class and the April to June kids in another and the July to September kids in another? No, they don't do that. Right. Even though we've had years and years of evidence that it's completely unfair to ask a January kid to compete with a December kid. When your child does† takes standardize tests, does the child take† do the kids born in December take the standardize tests on the same day as the kids born many January? Yes, they do. Does that make any sense whatsoever? No, it doesn't, right. For some reason we are powerfully incurious about the problems that we created with our meritocracies. We think we know a good research proposal from a bad one, and we don't. We think we know that we think we're selecting the right people for law school, and we aren't. We, you know, we think we know that an individual is responsible for their surgical success, and they aren't. And when we're presented with evidence of the falsity of our systems, what do we do? We do nothing. Now, I said at the beginning that this observation about our failed meritocracies is a very good thing. How can it be? If we fixed meritocracies, then most of us wouldn't be here, right.

But think about it. If we fix the system, the people who would replace us at a conference like this would be so much smarter than we are. This conference would have been so much more fun. Google would make so much more money, and I wouldn't be here. Someone far more gifted than I would be giving this talk, and it would have been infinitely more interesting. Thank you.

Also see The rise and fall of peer review and the followup The dance of the naked emperors by Adam Mastroianni.

brotli_comp_level 10;

It looks that indeed the default Brotli compression level 6 is a sweet spot for Brotli.

1. Measurement. I used below software versions:

1. Arch Linux kernel 6.4.12-arch1-1
2. Brotli 1.0.9-12
3. gzip 1.12-3

Machine is using an AMD Ryzen 7 5700G CPU with 64GB DDR4-3600 RAM.

All files were stored in /tmp, i.e., there were in a RAM disk. All compressed files were also written to /tmp. In total there were 544 HTML files, with roughly 19 MB. The individual HTML files, of course, were smaller, otherwise I could not be member of the 512KB club.

Testing Brotli compression:

/tmp/build: time brotli -kf -q 6 `find . -name \*.html`
        real 0.27s
        user 0.23s
        sys 0
        swapped 0
        total space 0

Testing gzip compression:

/tmp/build: time gzip -kf -9 `find . -name \*.html`
        real 0.34s
        user 0.32s
        sys 0
        swapped 0
        total space 0

Checking total file size:

wc `find . -name \*.br` | tail -3

2. Results. Real- and user-times are given in seconds.

3. Discussion. Even very low compression levels of Brotli lead to a significant reduction in compressed file size, way better than gzip. Starting at compression level 9 Brotli becomes slower than gzip but still compresses way better than gzip. My decision to use compression level 10 was motivated by the fact that many readers of this blog are not from Germany, .e.g., there are either from the US or from India. In this case I hope I can trade CPU time for smaller transferred data across the wire.

1. Motivation. The author and maintainer of Hiawatha, Hugo Leisink, on 18-Feb-2019 stated on his weblog:

Many times, I wondered whether I should keep going on with the project or not, but somehow I always found a reason to continue. But not this time. Recently, a serious issue was found in the Hiawatha webserver and the fact that I didn't care much, made me realize that it's time to stop.

Clearly, Hiawatha will never support HTTP/2 or HTTP/3 ... new features will be based on what I need, not on what is needed for a webserver in general.

Over the years he did not change his opinion on that. So it clearly was time to find a web-server which is fully maintained and offers below functionality:

1. Brotli compression
2. HTTP/3 and QUIC
3. URL rewriting
4. Built-in caching like Varnish

In Set-Up Hiawatha Web-Server I compared the size of various web-servers.

2. NGINX installation. Installing NGINX is pretty simple as it is contained in the Extra-repository of Arch Linux. For installing the Brotli extension you need to install the NGINX source code, then download the Brotli module, and compile the module. Below comment from the original GitHub repository is important:

You will need to use exactly the same ./configure arguments as your Nginx configuration and append --with-compat --add-dynamic-module=/path/to/ngx_brotli to the end, otherwise you will get a "module is not binary compatible" error on startup. You can run nginx -V to get the configuration arguments for your Nginx installation. Then

$ cd nginx-1.25
$ ./configure --with-compat --add-dynamic-module=/path/to/ngx_brotli
$ make modules

A concrete example for installing brotli-1.0rc. Switch to root user and go to directory /usr/src/nginx. In below configure the majority of the command line is the from nginx -V.

./configure --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=stderr --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-cc-opt='-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/nginx-mainline/src=/usr/src/debug/nginx-mainline -flto=auto' --with-ld-opt='-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto' --with-compat --with-debug --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-pcre-jit --with-stream --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-compat --add-dynamic-module=/tmp/ngx_brotli-1.0.0rc

Now make modules and copy the two files

cd objs
cp -p ngx_http_brotli_filter_module.so /usr/lib/nginx/
cp -p ngx_http_brotli_static_module.so /usr/lib/nginx/

3. Brotli compilation for Arch Linux. Go to /tmp directory.

git clone https://github.com/google/ngx_brotli.git
git submodule update --init

Go to NGINX source code: cd /usr/src/nginx, switch to root user.

./configure <output from nginx -V> --with-compat --add-dynamic-module=/tmp/ngx_brotli
make
cd objs
cp -p ngx_http_brotli_filter_module.so /usr/lib/nginx/
cp -p ngx_http_brotli_static_module.so /usr/lib/nginx/
systemctl start nginx

4. NGINX configuration. For the special case w.r.t. body-size I had already written on this here: nginx: 413 Request Entity Too Large - File Upload Issue. The general structure of a NGINX configuration is like below:

some global configuration;
http {
    server A {
        list 80;
    }
    server B {
        listen 443;
    }
}

All the rewriting rules for port 80 and 443 are the same, just copied from the top server to the bottom server config.

#user http;
worker_processes  1;

error_log  /var/log/nginx/error.log;

load_module /usr/lib/nginx/ngx_http_brotli_filter_module.so;
load_module /usr/lib/nginx/ngx_http_brotli_static_module.so;


events {
    worker_connections  1024;
}


http {
    root   /srv/http;
    index  index.html;
    client_max_body_size 15900M;

    http2 on;
    gzip  on;
    brotli on;
    brotli_comp_level 10;
    brotli_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
    gzip_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;

    fastcgi_cache_path /var/cache/nginx/ keys_zone=nginxpc:720m inactive=720m;
    fastcgi_cache_key "$request_method$request_uri";
    fastcgi_cache nginxpc;
    fastcgi_cache_valid 720m;


    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';
    log_format hiawatha_format '$remote_addr|$time_local|$status|$bytes_sent|$request|$http_referer|$http_user_agent|$host:$server_port|$https';
    access_log  /var/log/nginx/access.log hiawatha_format;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    http3 on;
    http3_hq on;
    types_hash_max_size 4096;

    server {
        listen       80;
        server_name  localhost;

        rewrite "^(/*)$" "/blog" redirect;
        rewrite "^/aux/search.php$" "/rewrite/sndsaaze/public/aux/search.php" last;
        rewrite "^/(404\.html|feed\.xml|sitemap\.html|sitemap\.xml)$" "/rewrite/sndsaaze/public/index.php?/$1" last;
        rewrite "^/(aux|blog|music|gallery)($|/.*)"  "/rewrite/sndsaaze/public/index.php?/$1$2" last;

        #charset koi8-r;

        error_page  404              /rewrite/sndsaaze/public/index.php?/404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

        location ~ \.php$ {
            try_files $fastcgi_script_name =404;

            # default fastcgi_params
            include fastcgi_params;

            # fastcgi settings
            fastcgi_pass			unix:/run/php-fpm/php-fpm.sock;
            fastcgi_buffers			8 16k;
            fastcgi_buffer_size		32k;

            # fastcgi params
            fastcgi_param DOCUMENT_ROOT	$realpath_root;
            fastcgi_param SCRIPT_FILENAME	$realpath_root$fastcgi_script_name;
        }

        location ~ ^/ttyd(.*)$ {
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_pass http://eklausmeier.goip.de:7681/$1;
        }
    }


    server {
        listen       443 quic;
        listen       443 ssl;
        server_name  localhost;

        rewrite "^(/*)$" "/blog" redirect;
        rewrite "^/aux/search.php$" "/rewrite/sndsaaze/public/aux/search.php" last;
        rewrite "^/(404\.html|feed\.xml|sitemap\.html|sitemap\.xml)$" "/rewrite/sndsaaze/public/index.php?/$1" last;
        rewrite "^/(aux|blog|music|gallery)($|/.*)"  "/rewrite/sndsaaze/public/index.php?/$1$2" last;

        ssl_certificate      /etc/hiawatha/eklausmeier.goip.de.pem;
        ssl_certificate_key  /etc/hiawatha/eklausmeier.goip.de.pem;

        # From https://blog.qualys.com/product-tech/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";

        location / {
            # used to advertise the availability of HTTP/3
            add_header Alt-Svc 'h3=":443"; ma=86400';
        }

        location ~ \.php$ {
            try_files $fastcgi_script_name =404;

            # default fastcgi_params
            include fastcgi_params;

            # fastcgi settings
            fastcgi_pass			unix:/run/php-fpm/php-fpm.sock;
            fastcgi_buffers			8 16k;
            fastcgi_buffer_size		32k;

            # fastcgi params
            fastcgi_param DOCUMENT_ROOT	$realpath_root;
            fastcgi_param SCRIPT_FILENAME	$realpath_root$fastcgi_script_name;
        }

        location ~ ^/ttyd(.*)$ {
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_pass http://eklausmeier.goip.de:7681/$1;
        }
    }

}

It is a common error to forget:

brotli_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
gzip_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;

See the examples below, where this configuration has been forgotten:

1. Analysis of Performance of Demo Open E-Mobility Site
2. Performance Remarks on PublicoMag Website

Method to check for compression:

curl -D - -H "Accept-Encoding: gzip,deflate,br" --write-out "%{size_download}\n" -o /tmp/prism-css.br http://localhost/jscss/prism.css

5. Caching. By using fastcgi_cache with a quite large retention interval of 720 minutes (=12 hours), I keep already generated pages in the cache for a long time.

If you want to delete specific entries in the cache, the critical line is:

fastcgi_cache_key "$request_method$request_uri";

You can compute the file-name of the cache file by specifying request-method and URL like so:

printf "GET/blog" | md5sum

or

printf "GET/blog/2023/08-29-from-hiawatha-to-nginx" | md5sum

This will print the file-name which you can delete with rm. In our case in the directory /var/cache/nginx. See How to Setup FastCGI Caching with Nginx on your VPS.

6. Deployment. With this very aggressive caching in place, the deployment of my blog changed. Previously, I generated all static files with

php saaze -mortb /tmp/build

and then deployed via

blogdeploy -p

Above deployment script essentially just removes the previous directories and replaces them with the newly generated ones. I did this for the "staging environment" and "production", i.e., on my work-PC and on the self-hosting PC.

Now I rarely generate all static files and use the dynamic mode of Simplified Saaze, i.e., Simplified Saaze generates the HTML file whenever it is actually accessed. Once it is generated then NGINX caches it. So, essentially the generation of the static files is deferred to the actual access time:

$ ls -l /var/cache/nginx | wc
    350    3143   26826

Solution: Buy a new 2 TB SSD, or use an existing 2 TB SSD, for example a Samsung.

Bad idea: Buy a new 4 TB SSD from Crucial and insert it into the A300M.

1. Problem statement. Since May 2020 I own an Asrock A300M mini PC with a Ryzen 3400G CPU. It's a nice and reliable computer, which is used for hosting this blog. There were two problems with the disk:

1. I had a major pacman-upgrade issue, with hundreds of zero-sized packages left on disk.
2. The 2 TB Viper disk was too small, disk was more than 90% full.

I searched for SSD's and found a Crucial 4 TB SSD, which was advertised to work within the A300M-STX. This was quite remarkable as the the Asrock website for the A300M offers no 4 TB SSD. See Storage QVL. Below is the screenshot of the obviously false claim that the Crucial 4 TB SSD works in the A300M.

So I ordered this 4 TB Crucial SSD for 182 EUR. For comparison, here are the SSD prices for my older SSD.

One can clearly see that prices for SSDs have gone down significantly.

2. Problems with the Crucial SSD.

The new Crucial 4TB SSD:

Opening the Asrock A300M after 3 years of uninterrupted service. The fan has accumulated quite some dust.

Mounting the Crucial 4TB SSD in the A300M:

The "old" 2TB SSD from Viper.

The new 4 TB Crucial SSD makes the A300M completely unresponsive, i.e., the A300M does not boot at all.

I tried a number of countermeasures:

1. Update BIOS in A300M: from p3.50 to p3.70 to p.370b
2. Mount Crucial on different M.2 interface
3. I contacted the German customer support

All to no avail.

I checked, whether the Crucial SSD is working properly by putting it into my work PC: It worked flawlessly. So, the aformentioned advertisement is wrong. The Crucial does not work in the Asrock A300M.

Crucial customer support, part of Micron Commercial Products Group, obiously does not understand the problem and responds with generic text fragments.

3. Remedy. I put the new Crucial 4 TB into my work PC. From the work PC I used the "old" Samsung 2 TB:

After so many tries with negative results I am testing the A300M without case for some time:

Reassembling.

After intensive testing the A300M has now two SSD and also 4 TB of SSD storage. Output of lsblk -f is as below:

NAME           FSTYPE      FSVER LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1    vfat        FAT32            A743-0700
└─nvme0n1p2    crypto_LUKS 2                70260d27-bc13-44dd-9b30-168c2be7c72f
  └─viper      ext4        1.0              dac919c6-2f0f-466b-ada8-692ce6d16d91  983.9G    42% /mnt/viper
nvme1n1
├─nvme1n1p1    vfat        FAT32 BOOT_FAT32 EB01-74DD                             116.1M    54% /boot
└─nvme1n1p2    crypto_LUKS 2                9b0766ca-06ce-41d6-9b46-04c66573f3aa
  └─Samsung2TB ext4        1.0              63669b64-5753-44a6-8626-561a6c98ab5b  466.6G    70% /

The CommonMark specification says:

An HTML block is a group of lines that is treated as raw HTML (and will not be escaped in HTML output).

Start condition: line begins with the string <?.

End condition: line contains the string ?>.

So it is possible to embed PHP in Markdown. Unfortunately, not every construct in Markdown passes PHP through undisturbed. For example, links and images, i.e., [reftext](ref) and ![](/imgref) destroy the PHP start- and endtags <? and ?>. Luckily, these small glitches can be cured with some string-replacements.

1. Examples. Embedding PHP code in Markdown allows us to write something like this, below is Markdown:

<?php
    $pkgList = explode("\n",`pacman -Q`);
    $pkg = array();
    foreach ($pkgList as $e) { $f=explode(' ',$e); $pkg[$f[0]??'x'] = $f[1]??''; }
?>

Using neovim version <?=$pkg['neovim']?>.

That's exactly what is done in /uses.

Another example is adding information based on time:

On <?=date('d-M-Y')?> this blog has <?= `find ~klm/php/sndsaaze/content -name \*.md | wc -l` ?> entries.

Above code is used in Blog Anniversary: 500 posts.

Below code also shows the use of PHP within Markdown:

<?php $chap=0; $subchap=0; ?>

# <?=++$chap?>. First chapter
# <?=++$chap?>. Second chapter
## <?=$chap.'.'.(++$subchap)?> Subchapter

This produces:

<h1>1. First chapter</h1>
<h1>2. Second chapter</h1>
<h2>2.1 Subchapter</h2>

When PHP code is included in HTML code, which is legal in Markdown, then no "escaping" with a star (*) is required.

PHP code in HTML code can be included verbatim. For example:

<p>2021-04-02 <a href="<?=$rbase?>/pkg/jpilot_2.0.1-1_amd64.deb">jpilot_2.0.1-1_amd64.deb</a>

Here is an example of PHP code in the URL part in Markdown:

The Markdown for _Simplified Saaze_ can also [contain PHP code](*<?=$rbase?>*/blog/2023/08-27-mixing-php-into-markdown)!

Embedding PHP in ordinary text is also no problem. No escaping with a star (*) is required.

2. Implementation. Within ordinary paragraph text it is easy to just embed PHP, which is passed through to HTML unchanged. In references I now use this character combination to later string-replace any glitches introduced by the Markdown-to-HTML conversion.

[reference text](*<?=$rbase?>*/htmlRef)

With these added asterisks I then later replace any conversion errors. Previously I just used below code to include HTML in Simplified Saaze's templates:

<?= $entry['content'] ?>

Now I use (please mentally uppercase 3c and 3e):

$s = str_replace('*%3c?','<?',$entry['content']);
$s = str_replace('?%3e*','?>',$s);
require 'data:text/plain;base64,'.base64_encode($s);

When using require with data:text then you have to activate this in php.ini:

allow_url_include = On

See allow_url_include.

More information on data-wrappers is here: data:// and the comment by brainbox. Furthermore see PHP include and the comment by sPlayer. RFC 2397 details data:[<mediatype>][;base64],<data>.

1. Downloading rpm. Here we describe using the rpm repository of /edge/centos/8/x86_64/RPMS/. Essentially, this is a precompiled binary in rpm-form. Our version is openlitespeed-1.7.16-1.el8.x86_64.rpm. Another good candidate is openlitespeed-1.7.17-1.el9.x86_64.rpm in /centos/9/x86_64/RPMS/.

2. Unpacking rpm. In the following ols stands for the downloaded rpm-file without the rpm-suffix.

1. Convert rpm to cpio: rpm2cpio ols.rpm > ols.cpio.zstd
2. Uncompress file with unzstd old.cpio.zstd
3. Unpack resulting cpio-archive with cpio -idmv < ols.cpio
4. Move directory ./usr/local/lsws to /usr/local/lsws, and chown -R root:root if you haven't created the files as root user

3. Start webserver.

1. Create Unix group lsadm, and user lsadm with /bin/nologin shell
2. conf-directory: chown -R lsadm:lsadm /usr/local/lsws/conf
3. Check missing libraries: ldd bin/openlitespeed, I had to install missing libcrypt.so.1 library via pacman -S libxcrypt-compat
4. Copy systemd-service for start- and stop, enable service:

cp -p /usr/local/lsws/admin/misc/lshttpd.service /usr/lib/systemd/system/lshttpd.service
systemctl enable lshttpd

The litespeed process changes chmod of various config files to be executable. This is just silly, but you have no chance to correct it, as with every restart of litespeed the chmod is changed again. In total OpenLiteSpeed needs 67 MB of disk space under /usr/local/lsws.

4. Admin console. The original installation contains some glitches, which need to be corrected. Though, overall the admin console is not very useful.

1. Admin console needs below symbolic links: Go to /usr/local/lsws. Then mkdir -p lsphp73/bin lsphp74/bin, change to these two directories and create symlink ln -s ../../fcgi-bin/lsphp
2. Copy a pem-file to admin/conf/, symlink webadmin.crt and webadmin.key to this file, as the admin console enforces https. Alternatively you can change admin/conf/admin_config.conf and edit keyfile and certfile
3. Login to admin console does not work out-of-the-box, therefore edit file admin/html.open/lib/CAuthorizer.php in line 261 and change return $auth; to return true; in PHP function authUser() to fix the authorization issue
4. Admin console logs you out all the time: in PHP function __construct() in lib/CAuthorizer.php comment out the entire if clause for if (isset($_SESSION['timeout']) ...
5. Admin console floods the log-file with various INFO and NOTICE messages
6. Start web-server with /usr/local/lsws/bin/lswsctrl start, webserver listens on port 8088, admin console listens on port 7080

As the admin console is basically useless, I recommend to simply disable it with disableWebadmin 1 in conf/httpd_config.conf. Put it right after the servername line.

Restarting the webserver:

1. stop webserver with lswsctrl stop
2. remove cache with rm -rf cachedata and any sockets with rm admin/tmp/admin.*
3. possibly remove old log file: rm logs/*.log
4. start with lswsctrl start.

5. PHP via FastCGI. OpenLiteSpeed comes with PHP 5.6 installed as special LS-API compiled binary. This version is way too old. Below steps configure PHP as FastCGI.

1. Edit conf/httpd_config.conf and change user and group to http from nobody. The Linux user http is also employed by php-fpm. Change directory ownership chown -R lsadm:http conf, and chown -R http:http tmp`.
2. Enable PHP-FPM (PHP FastCGI Process Manager): systemctl enable php-fpm
3. Start PHP-FPM: systemctl start php-fpm
4. Configure extprocessor for OpenLiteSpeed in conf/httpd_config.conf:

user                      http
group                     http

extprocessor fcgiphp {
   type                    fcgi
   address                 uds://run/php-fpm/php-fpm.sock
   note                    PHP FPM
   maxConns                10
   initTimeout             20
   retryTimeout            10
   respBuffer              0
   autoStart               0
}

Domain socket in extprocessor must match the value in /etc/php/php-fpm.d/www.conf. OpenLiteSpeed should not start php-fpm, therefore autostart 0. The virtual host configuration is as below. The important part is the "context" which relates to fcgi.

# Virtual host config for klmblog
# 13-Aug-2023

docRoot                   /srv/http/
enableGzip                1
enableBr                  1

errorlog $VH_ROOT/logs/error.log {
  useServer               1
  logLevel                DEBUG
  rollingSize             10M
}

accesslog $VH_ROOT/logs/access.log {
  useServer               0
  rollingSize             10M
  keepDays                30
  compressArchive         0
}

context /p/ {
  type                    fcgi
  handler                 fcgiphp
  addDefaultCharset       off
}

context / {
  location                /srv/http/
  allowBrowse             1

  rewrite  {

  }
  addDefaultCharset       off

  phpIniOverride  {

  }
}

6. Tips and tricks.

1. The process-id's of the webserver are pidof litespeed
2. Before accessing the webserver with any queries, check logs/error.log for any entries with [ERROR]

1. Data. Generating static HTML files with Simplified Saaze shows number of posts:

$ time php saaze -mortb /tmp/build
Building static site in /tmp/build...
        execute(): filePath=/home/klm/php/sndsaaze/content/aux.yml, nentries=6, totalPages=1, entries_per_page=20
        execute(): filePath=/home/klm/php/sndsaaze/content/blog.yml, nentries=412, totalPages=21, entries_per_page=20
        execute(): filePath=/home/klm/php/sndsaaze/content/gallery.yml, nentries=6, totalPages=1, entries_per_page=20
        execute(): filePath=/home/klm/php/sndsaaze/content/music.yml, nentries=61, totalPages=4, entries_per_page=20
        execute(): filePath=/home/klm/php/sndsaaze/content/error.yml, nentries=1, totalPages=1, entries_per_page=20
Finished creating 5 collections, 4 with index, and 504 entries (0.17 secs / 11.15MB)
#collections=5, YamlParser=0.0095/511-5, md2html=0.0153, MathParser=0.0082/504, renderEntry=504, content=504/0, excerpt=0/0
        real 0.19s
        user 0.12s
        sys 0
        swapped 0
        total space 0

The number of posts, excluding index.md.

Over the years the posts are distributed as follows, not counting the 14 index.md posts. Two posts have index: false in their frontmatter, and one post has draft: true, see Simplified Saaze. Therefore these posts do not show in the count from php saaze.

2. Counting content. Below Perl script blogyrcnt counts posts per year.

#!/bin/perl -W
# Count number of posts per year

use strict;
my %H;

while (<>) {
    $H{$1} += 1 if (/\.\/(\d\d\d\d)\/\d\d\-\d\d\-/);
}

for (sort keys %H) {
    printf("%4d: %d\n",$_,$H{$_});
}

Run script as follows: Go to content/blog directory, then

$ find . | sort | blogyrcnt`

Similarly for content/music.

As of this blog has entries.