Go: Functional options are slow

from Evan Jones - Software Engineer | Computer Scientist [alt+shift+b] in programming

The Go "functional options" pattern is a way of passing options to a function. The function takes a variable number of arguments, which are themselves functions (a type like ...func(*config). I think it was first introduced by Rob Pike in a 2014 blog post. It is now used by many APIs. For example, gRPC's DialContext(), AWS's LoadDefaultConfig(), and OpenTelemetry's Tracer.Start(). This style should be avoided when performance is critical. This article describes a microbenchmark that shows functional options are slower, require more instructions, are less likely to be inlined, and may require memory allocations. In the worst case, Go must allocate a slice for the ... argument, and an additional object for each option in the slice. Often, the compiler's escape analysis can optimize these away, but not always. In particular, the compiler can never optimize an interface method call. The alternative is to pass a struct with the configuration values. This does not rely on the compiler to be efficient. In my opinion, performance-critical code should be efficient "by design," and not rely on compiler optimizations that may or may not apply. This article presents a brief experiment to try and demonstrate the performance differences. Functional options versus configuration structs The alternative to functional options is passing a configuration struct, like the standard library's http.Server or tls.Config. To make this concrete, let's consider a constructor function that creates a type *Foo with two options: a boolean (enabling a feature that is disabled by default), and an integer (taking a numeric value). With functional options, the code would look like the following: Functional options creating *Foo func NewFoo(options ...FooOption) *Foo { // ... implementation } func CallNewFoo() *Foo { return NewFoo(WithBoolOption(), WithIntOption(42)) } The configuration struct version would look something like the following: Configuration struct creating *Foo func...

23rd May 2022

Remove from reading list Add to reading list [alt+a] Read now [→]

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Evan Jones - Software Engineer | Computer Scientist

Setenv is not Thread Safe and C Doesn't Want to Fix It

You can't safely use the C setenv() or unsetenv() functions in a program that uses threads. Those functions modify global state, and can cause other threads calling getenv() to crash. This also causes crashes in other languages that use those C standard library functions, such as Go's os.Setenv (Go issue) and Rust's std::env::set_var() (Rust issue). I ran into this in a Go program, because Go's built-in DNS resolver can call C's getaddrinfo(), which uses environment variables. This cost me 2 days to track down and file the Go bug. Sadly, this problem has been known for decades. For example, an article from January 2017 said: "None of this is new, but we do re-discover it roughly every five years. See you in 2022." This was only one year off! (She wrote an update in October 2023 after I emailed her about my Go bug.) This is a flaw in the POSIX standard, which extends the C Standard to allow modifying environment varibles. The most infuriating part is that many people who could influence the standard or maintain the C libraries don't see this as a problem. The argument is that the specification clearly documents that setenv() cannot be used with threads. Therefore, if someone does this, the crashes are their fault. We should apparently read every function's specification carefully, not use software written by others, and not use threads. These are unrealistic assumptions in modern software. I think we should instead strive to create APIs that are hard to screw up, and evolve as the ecosystem changes. The C language and standard library continue to play an important role at the base of most software. We either need to figure out how to improve it, or we need to figure out how to abandon it. Why is setenv() not thread-safe? The biggest problem is that getenv() returns a char*, with no need for applications to free it later. One thread could be using this pointer when another thread changes the same environment variable using setenv() or unsetenv(). The getenv() function is perfect if environment variables never change. For example, for accessing a process's initial table of environment variables (see the System V ABI: AMD64 Section 3.4.1). It turns out the C Standard only includes getenv(), so according to C, that is exactly how this should work. However, most implementations also follow the POSIX standard (e.g. POSIX.1-2017), which extends C to include functions that modify the environment. This means the current getenv() API is problematic. Even worse, putenv() adds a char* to the set of environment variables. It is explicitly required that if the application modifies the memory after putenv() returns, it modifies the environment variables. This means applications can modify the value passed to putenv() at any time, without any synchronization. FreeBSD used to implement putenv() by copying the value, but it changed it with FreeBSD 7 in 2008, which suggests some programs really do depend on modifying the environment in this fashion (see FreeBSD putenv man page). As a final problem, environ is a NULL-terminated array of pointers (char**) that an application can read and assign to (see definition in POSIX.1-2017). This is how applications can iterate over all environment variables. Accesses to this array are not thread-safe. However, in my experience many fewer applications use this than getenv() and setenv(). However, this does cause some libraries to not maintain the set of environment variables in a thread-safe way, since they directly update this table. Environment variable implementations Implementations need to choose what do do when an application overwrites an existing variable. I looked at glibc, musl, Solaris/Illumos, and FreeBSD/Apple's C standard libraries, and they make the following choices: Never free environment variables (glibc, Solaris/Illumos): Calling setenv() repeatedly is effectively a memory leak. However, once a value is returned from getenv(), it is immutable and can be used by threads safely. Free the environment variables (musl, FreeBSD/Apple): Using the pointer returned by getenv() after another thread calls setenv() can crash. A second problem is ensuring the set of environment variables is updated in a thread-safe fashion. This is what causes crashes in glibc. glibc uses an array to hold pointers to the "NAME=value" strings. It holds a lock in setenv() when changing this array, but not in getenv(). If a thread calling setenv() needs to resize the array of pointers, it copies the values to a new array and frees the previous one. This can cause other threads executing getenv() to crash, since they are now iterating deallocated memory. This is particularly annoying since glibc already leaks environment variables, and holds a lock in setenv(). All it needs to do is hold the lock inside getenv(), and it would no longer crash. This would make getenv() slightly slower. However, getenv() already uses a linear search of the array, so performance does not appear to be a concern. More sophisticated implementations are possible if this is a problem, such as Solaris/Illumos's lock-free implementation. Why do programs use environment variables? Environment variables useful for configuring shared libraries or language runtimes that are included in other programs. This allows users to change the configuration, without program authors needing to explicitly pass the configuration in. One alternative is command line flags, which requires programs to parse them and pass them in to the libraries. Another alternative are configuration files, which then need some other way to disable or configure, to be able to test new configurations. Environment variables are a simple solution. AS a result, many libraries call getenv() (see a partial list below). Since many libraries are configured through environment variables, a program may need to change these variables to configure the libraries it uses. This is common at application startup. This causes programs to need to call setenv(). Given this issue, it seems like libraries should also provide a way to explicitly configure any settings, and avoid using environment variables. We should fix this problem, and we can In my opinion, it is rediculous that this has been a known problem for so long. It has wasted thousands of hours of people's time, either debugging the problems, or debating what to do about it. We know how to fix the problem. First, we can make a thread-safe implementation, like Illumos/Solaris. This has some limitations: it leaks memory in setenv(), and is still unsafe if a program uses putenv() or the environ variable. However, this is an improvement over the current Linux and Apple implementations. The second solution is to add new APIs to get one and get all environment variables that are thread-safe by design, like Microsoft's getenv_s() (see below for the controversy around C11's "Annex K"). My preferred solution would be to do both. This would reduce the chances of hitting this problem for existing programs and libraries, and also provide a path to avoid the problems entirely for new code or languages like Go and Rust. My rough idea would be the following: Add a function to copy one single environment variable to a user-specified buffer, similar to getenv_s(). Add a thread-safe API to iterate over all environment variables, or to copy all variables out. Mark getenv() as deprecated, recommending the new thread-safe getenv() function instead. Mark putenv() as deprecated, recommending setenv() instead. Mark environ as deprecated, recommending environment variable functions instead. Update the implementation of environment varibles to be thread-safe. This requires leaking memory if getenv() is used on a variable, but we can detect if the old functions are used, and only leak memory in that case. This means programs written in other languages will avoid these problems as soon as their runtimes are updated. Update the C and POSIX standards to require the above changes. This would be progress. The getenv_s / C Standard Annex K controversy Microsoft provides getenv_s(), which copies the environment variable into a caller-provided buffer. This is easy to make thread-safe by holding a read lock while copying the variable. After the function returns, future changes to the environment have no effect. This is included in the C11 Standard as Annex K "Bounds Checking Interfaces". The C standard Annexes are optional features. This Annex includes new functions intended to make it harder to make mistakes with buffers that are the wrong size. The first draft of this extension was published in 2003. This is when Microsoft was focusing on "Trustworthy Computing" after a January 2002 memo from Bill Gates. Basically, Windows wasn't designed to be connected to the Internet, and now that it was, people were finding many security problems. Lots of them were caused by buffer handling mistakes. Microsoft developed new versions of a number of problematic functions, and added checks to the Visual C++ compiler to warn about using the old ones. They then attempted to standardize these functions. My understanding is the people responsible for the Unix POSIX standards did not like the design of these functions, so they refused to implement them. For more details, see Field Experience With Annex K published in September 2015, Stack Overflow: Why didn't glibc implement _s functions? updated March 2023, and Rich Felker of musl on both technical and social reasons for not implementing Annex K from February 2019. I haven't looked at the rest of the functions, but having spent way too long looking at getenv(), the general idea of getenv_s() seems like a good idea to me. Standardizing this would help avoid this problem. Incomplete list of common environment variables This is a list of some uses of environment variables from fairly widely used libraries and services. This shows that environment variables are pretty widely used. Cloud Provider Credentials and Services AWS's SDKs for credentials (e.g. AWS_ACCESS_KEY_ID) Google Cloud Application Default Credentials (e.g. GOOGLE_APPLICATION_CREDENTIALS) Microsoft Azure Default Azure Credential (e.g. AZURE_CLIENT_ID) AWS's Lambda serverless product: sets a large number of variables like AWS_REGION, AWS_LAMBDA_FUNCTION_NAME, and credentials like AWS_SECRET_ACCESS_KEY Google Cloud Run serverless product: configuration like PORT, K_SERVICE, K_REVISION Kubernetes service discovery: Defines variables SERVICE_NAME_HOST and SERVICE_NAME_PORT. Third-party C/C++ Libraries OpenTelemetry: Metrics and tracing. Many environment variables like OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES. OpenSSL: many configurable variables like HTTPS_PROXY, OPENSSL_CONF, OPENSSL_ENGINES. BoringSSL: Google's fork of OpenSSL used in Chrome and others. It reads SSLKEYLOGFILE just like OpenSSL for logging TLS keys for debugging. Libcurl: proxies, SSL/TLS configuration and debugging like HTTPS_PROXY, CURL_SSL_BACKEND, CURL_DEBUG. Libpq Postgres client library: connection parameters including credentials like PGHOSTADDR, PGDATABASE, and PGPASSWORD. Rust Standard Library std::thread RUST_MIN_STACK: Calls std::env::var() on the first call to spawn() a new thread. It is cached in a static atomic variable and never read again. See implementation in thread::min_stack(). std::backtrace RUST_LIB_BACKTRACE: Calls std::env::var() on the first call to capture a backtrace. It is cached in a static atomic variable and never read again. See implementation in Backtrace::enabled().

19th Nov 2023 • 69 votes

Random Load Balancing is Unevenly Distributed

This is a reminder that random load balancing is unevenly distributed. If we distribute a set of items randomly across a set of servers (e.g. by hashing, or by randomly selecting a server), the average number of items on each server is num_items / num_servers. It is easy to assume this means each server has close to the same number of items. However, since we are selecting servers at random, they will have different numbers of items, and the imbalance can be important. For load balancing, a reasonable model is that each server has fixed capacity (e.g. it can serve 3000 requests/second, or store 100 items, etc.). We need to divide the total workload over the servers, so that each server stays below its capacity. This means the number of servers is determined by the most loaded server, not the average. This is a classic balls in bins problem that has been well studied, and there are some interesting theoretical results. However, I wanted some specific numbers, so I wrote a small simulation. The summary is that the imbalance depends on the expected number of items per server (that is, num_items / num_servers). This means workload is more balanced with fewer servers, or with more items. This means that dividing a set of items over more servers makes the distribution more unfair, which is a reason we can get worse than linear scaling of a distributed system. Let's make this more concrete with an example. Let's assume we have a workload of 1000 items, and each server can hold a maximum of 100 items. If we place the exact same number of items on each server, we only need 10 servers, and each of them is completely busy. However, if we place the items randomly, then the median (p50) number of items is 100 items. This means half the servers will have more than 100 items, and will be overloaded. If we want less than a 1% chance of an overloaded server, we need to look at the 99th percentile (p99) server load. We need to use at least 13 servers, which has a p99 load of 97 items. For 14 servers, the average is 77 items, so our servers are on average 23% idle. This shows how the imbalance leads to wasted capacity. This is a bit of an extreme example, because the number of items is small. Let's assume we can make the items 10× smaller, say by dividing them into pieces. Our workload now consists of 10k items, and each server has the capacity to hold 1000 (1k) items. Our perfectly balanced workload still needs 10 servers. With random load balancing, to have a less than 1 in 1000 chance of exceeding our capacity, we only need 11 servers, which has a p99 load of 98 items and a p999 of 100 items. With 11 servers, the average number of items is 910 or 91%, so our servers are only 9% idle. This shows how splitting work into smaller pieces improves load balancing. Another way to look at this is to think about a scaling scenario. Let's go back to our workload of 1000 items, where each server can handle 100 items, and we have 13 servers to ensure we have less than a 1% chance of an overloaded server. Now let's assume the amount of work per item doubles, for example because the service has become more popular, so each item has become larger. Now, each server can hold a maximum of 50 items. If we have perfectly linear scaling, we can double the number of servers from 13 to 26 to handle this workload. However, 26 servers has a p99 of 53 items, so we again have a more than 1% chance of overload. We need to use 28 servers which has a p99 of 50 items. This means we doubled the workload, but had to increase the number of servers from 13 to 28, which is 2.15×. This is sub-linear scaling. As a way to visualize the imbalance, the chart below shows the p99 to average ratio, which is a measure of how imbalanced the system is. If everything is perfectly balanced, the value is 1.0. A value of 2.0 means 1% of servers will have double the number of items of the average server. This shows that the imbalance increases with the number of servers, and increases with fewer items. Power of Two Random Choices Another way to improve load balancing is to have smarter placement. Perfect placement can be hard, but it is often possible to use the "power of two random choices" technique: select two servers at random, and place the item on the least loaded of the two. This makes the distribution much more balanced. For 1000 items and 100 items/server, 11 servers has a p999 of 93 items, so much less than 0.1% chance of overload, compared to needing 14 servers with random load balancing. For the scaling scenario where each server can only handle 50 items, we only need 21 servers to have a p999 of 50 items, compared to 28 servers with random load balancing. The downside of the two choices technique is that each request is now more expensive, since it must query two servers instead of one. However, in many cases where the "item not found" requests are much less expensive than the "item found" requests, this can still be a substantial improvement. For another look at how this improves load balancing, with a nice simulation that includes information delays, see Marc Brooker's blog post. Raw simulation output I will share the code for this simulation later. simulating placing items on servers with random selection iterations=10000 (number of times num_items are placed on num_servers) measures the fraction of items on each server (server_items/num_items) and reports the percentile of all servers in the run P99_AVG_RATIO = p99 / average; approximately the worst server compared to average num_items=1000: num_servers=3 p50=0.33300 p95=0.35800 p99=0.36800 p999=0.37900 AVG=0.33333; P99_AVG_RATIO=1.10400; ITEMS_PER_NODE=333.3 num_servers=5 p50=0.20000 p95=0.22100 p99=0.23000 p999=0.24000 AVG=0.20000; P99_AVG_RATIO=1.15000; ITEMS_PER_NODE=200.0 num_servers=10 p50=0.10000 p95=0.11600 p99=0.12300 p999=0.13100 AVG=0.10000; P99_AVG_RATIO=1.23000; ITEMS_PER_NODE=100.0 num_servers=11 p50=0.09100 p95=0.10600 p99=0.11300 p999=0.12000 AVG=0.09091; P99_AVG_RATIO=1.24300; ITEMS_PER_NODE=90.9 num_servers=12 p50=0.08300 p95=0.09800 p99=0.10400 p999=0.11200 AVG=0.08333; P99_AVG_RATIO=1.24800; ITEMS_PER_NODE=83.3 num_servers=13 p50=0.07700 p95=0.09100 p99=0.09700 p999=0.10400 AVG=0.07692; P99_AVG_RATIO=1.26100; ITEMS_PER_NODE=76.9 num_servers=14 p50=0.07100 p95=0.08500 p99=0.09100 p999=0.09800 AVG=0.07143; P99_AVG_RATIO=1.27400; ITEMS_PER_NODE=71.4 num_servers=25 p50=0.04000 p95=0.05000 p99=0.05500 p999=0.06000 AVG=0.04000; P99_AVG_RATIO=1.37500; ITEMS_PER_NODE=40.0 num_servers=50 p50=0.02000 p95=0.02800 p99=0.03100 p999=0.03500 AVG=0.02000; P99_AVG_RATIO=1.55000; ITEMS_PER_NODE=20.0 num_servers=100 p50=0.01000 p95=0.01500 p99=0.01800 p999=0.02100 AVG=0.01000; P99_AVG_RATIO=1.80000; ITEMS_PER_NODE=10.0 num_servers=1000 p50=0.00100 p95=0.00300 p99=0.00400 p999=0.00500 AVG=0.00100; P99_AVG_RATIO=4.00000; ITEMS_PER_NODE=1.0 num_items=2000: num_servers=3 p50=0.33350 p95=0.35050 p99=0.35850 p999=0.36550 AVG=0.33333; P99_AVG_RATIO=1.07550; ITEMS_PER_NODE=666.7 num_servers=5 p50=0.20000 p95=0.21500 p99=0.22150 p999=0.22850 AVG=0.20000; P99_AVG_RATIO=1.10750; ITEMS_PER_NODE=400.0 num_servers=10 p50=0.10000 p95=0.11100 p99=0.11600 p999=0.12150 AVG=0.10000; P99_AVG_RATIO=1.16000; ITEMS_PER_NODE=200.0 num_servers=11 p50=0.09100 p95=0.10150 p99=0.10650 p999=0.11150 AVG=0.09091; P99_AVG_RATIO=1.17150; ITEMS_PER_NODE=181.8 num_servers=12 p50=0.08350 p95=0.09350 p99=0.09800 p999=0.10300 AVG=0.08333; P99_AVG_RATIO=1.17600; ITEMS_PER_NODE=166.7 num_servers=13 p50=0.07700 p95=0.08700 p99=0.09100 p999=0.09600 AVG=0.07692; P99_AVG_RATIO=1.18300; ITEMS_PER_NODE=153.8 num_servers=14 p50=0.07150 p95=0.08100 p99=0.08500 p999=0.09000 AVG=0.07143; P99_AVG_RATIO=1.19000; ITEMS_PER_NODE=142.9 num_servers=25 p50=0.04000 p95=0.04750 p99=0.05050 p999=0.05450 AVG=0.04000; P99_AVG_RATIO=1.26250; ITEMS_PER_NODE=80.0 num_servers=50 p50=0.02000 p95=0.02550 p99=0.02750 p999=0.03050 AVG=0.02000; P99_AVG_RATIO=1.37500; ITEMS_PER_NODE=40.0 num_servers=100 p50=0.01000 p95=0.01400 p99=0.01550 p999=0.01750 AVG=0.01000; P99_AVG_RATIO=1.55000; ITEMS_PER_NODE=20.0 num_servers=1000 p50=0.00100 p95=0.00250 p99=0.00300 p999=0.00400 AVG=0.00100; P99_AVG_RATIO=3.00000; ITEMS_PER_NODE=2.0 num_items=5000: num_servers=3 p50=0.33340 p95=0.34440 p99=0.34920 p999=0.35400 AVG=0.33333; P99_AVG_RATIO=1.04760; ITEMS_PER_NODE=1666.7 num_servers=5 p50=0.20000 p95=0.20920 p99=0.21320 p999=0.21740 AVG=0.20000; P99_AVG_RATIO=1.06600; ITEMS_PER_NODE=1000.0 num_servers=10 p50=0.10000 p95=0.10700 p99=0.11000 p999=0.11320 AVG=0.10000; P99_AVG_RATIO=1.10000; ITEMS_PER_NODE=500.0 num_servers=11 p50=0.09080 p95=0.09760 p99=0.10040 p999=0.10380 AVG=0.09091; P99_AVG_RATIO=1.10440; ITEMS_PER_NODE=454.5 num_servers=12 p50=0.08340 p95=0.08980 p99=0.09260 p999=0.09580 AVG=0.08333; P99_AVG_RATIO=1.11120; ITEMS_PER_NODE=416.7 num_servers=13 p50=0.07680 p95=0.08320 p99=0.08580 p999=0.08900 AVG=0.07692; P99_AVG_RATIO=1.11540; ITEMS_PER_NODE=384.6 num_servers=14 p50=0.07140 p95=0.07740 p99=0.08000 p999=0.08300 AVG=0.07143; P99_AVG_RATIO=1.12000; ITEMS_PER_NODE=357.1 num_servers=25 p50=0.04000 p95=0.04460 p99=0.04660 p999=0.04880 AVG=0.04000; P99_AVG_RATIO=1.16500; ITEMS_PER_NODE=200.0 num_servers=50 p50=0.02000 p95=0.02340 p99=0.02480 p999=0.02640 AVG=0.02000; P99_AVG_RATIO=1.24000; ITEMS_PER_NODE=100.0 num_servers=100 p50=0.01000 p95=0.01240 p99=0.01340 p999=0.01460 AVG=0.01000; P99_AVG_RATIO=1.34000; ITEMS_PER_NODE=50.0 num_servers=1000 p50=0.00100 p95=0.00180 p99=0.00220 p999=0.00260 AVG=0.00100; P99_AVG_RATIO=2.20000; ITEMS_PER_NODE=5.0 num_items=10000: num_servers=3 p50=0.33330 p95=0.34110 p99=0.34430 p999=0.34820 AVG=0.33333; P99_AVG_RATIO=1.03290; ITEMS_PER_NODE=3333.3 num_servers=5 p50=0.20000 p95=0.20670 p99=0.20950 p999=0.21260 AVG=0.20000; P99_AVG_RATIO=1.04750; ITEMS_PER_NODE=2000.0 num_servers=10 p50=0.10000 p95=0.10500 p99=0.10700 p999=0.10940 AVG=0.10000; P99_AVG_RATIO=1.07000; ITEMS_PER_NODE=1000.0 num_servers=11 p50=0.09090 p95=0.09570 p99=0.09770 p999=0.09990 AVG=0.09091; P99_AVG_RATIO=1.07470; ITEMS_PER_NODE=909.1 num_servers=12 p50=0.08330 p95=0.08790 p99=0.08980 p999=0.09210 AVG=0.08333; P99_AVG_RATIO=1.07760; ITEMS_PER_NODE=833.3 num_servers=13 p50=0.07690 p95=0.08130 p99=0.08320 p999=0.08530 AVG=0.07692; P99_AVG_RATIO=1.08160; ITEMS_PER_NODE=769.2 num_servers=14 p50=0.07140 p95=0.07570 p99=0.07740 p999=0.07950 AVG=0.07143; P99_AVG_RATIO=1.08360; ITEMS_PER_NODE=714.3 num_servers=25 p50=0.04000 p95=0.04330 p99=0.04460 p999=0.04620 AVG=0.04000; P99_AVG_RATIO=1.11500; ITEMS_PER_NODE=400.0 num_servers=50 p50=0.02000 p95=0.02230 p99=0.02330 p999=0.02440 AVG=0.02000; P99_AVG_RATIO=1.16500; ITEMS_PER_NODE=200.0 num_servers=100 p50=0.01000 p95=0.01170 p99=0.01240 p999=0.01320 AVG=0.01000; P99_AVG_RATIO=1.24000; ITEMS_PER_NODE=100.0 num_servers=1000 p50=0.00100 p95=0.00150 p99=0.00180 p999=0.00210 AVG=0.00100; P99_AVG_RATIO=1.80000; ITEMS_PER_NODE=10.0 num_items=100000: num_servers=3 p50=0.33333 p95=0.33579 p99=0.33681 p999=0.33797 AVG=0.33333; P99_AVG_RATIO=1.01043; ITEMS_PER_NODE=33333.3 num_servers=5 p50=0.20000 p95=0.20207 p99=0.20294 p999=0.20393 AVG=0.20000; P99_AVG_RATIO=1.01470; ITEMS_PER_NODE=20000.0 num_servers=10 p50=0.10000 p95=0.10157 p99=0.10222 p999=0.10298 AVG=0.10000; P99_AVG_RATIO=1.02220; ITEMS_PER_NODE=10000.0 num_servers=11 p50=0.09091 p95=0.09241 p99=0.09304 p999=0.09379 AVG=0.09091; P99_AVG_RATIO=1.02344; ITEMS_PER_NODE=9090.9 num_servers=12 p50=0.08334 p95=0.08477 p99=0.08537 p999=0.08602 AVG=0.08333; P99_AVG_RATIO=1.02444; ITEMS_PER_NODE=8333.3 num_servers=13 p50=0.07692 p95=0.07831 p99=0.07888 p999=0.07954 AVG=0.07692; P99_AVG_RATIO=1.02544; ITEMS_PER_NODE=7692.3 num_servers=14 p50=0.07143 p95=0.07277 p99=0.07332 p999=0.07396 AVG=0.07143; P99_AVG_RATIO=1.02648; ITEMS_PER_NODE=7142.9 num_servers=25 p50=0.04000 p95=0.04102 p99=0.04145 p999=0.04193 AVG=0.04000; P99_AVG_RATIO=1.03625; ITEMS_PER_NODE=4000.0 num_servers=50 p50=0.02000 p95=0.02073 p99=0.02103 p999=0.02138 AVG=0.02000; P99_AVG_RATIO=1.05150; ITEMS_PER_NODE=2000.0 num_servers=100 p50=0.01000 p95=0.01052 p99=0.01074 p999=0.01099 AVG=0.01000; P99_AVG_RATIO=1.07400; ITEMS_PER_NODE=1000.0 num_servers=1000 p50=0.00100 p95=0.00117 p99=0.00124 p999=0.00132 AVG=0.00100; P99_AVG_RATIO=1.24000; ITEMS_PER_NODE=100.0 power of two choices num_items=1000: num_servers=3 p50=0.33300 p95=0.33400 p99=0.33500 p999=0.33600 AVG=0.33333; P99_AVG_RATIO=1.00500; ITEMS_PER_NODE=333.3 num_servers=5 p50=0.20000 p95=0.20100 p99=0.20200 p999=0.20300 AVG=0.20000; P99_AVG_RATIO=1.01000; ITEMS_PER_NODE=200.0 num_servers=10 p50=0.10000 p95=0.10100 p99=0.10200 p999=0.10200 AVG=0.10000; P99_AVG_RATIO=1.02000; ITEMS_PER_NODE=100.0 num_servers=11 p50=0.09100 p95=0.09200 p99=0.09300 p999=0.09300 AVG=0.09091; P99_AVG_RATIO=1.02300; ITEMS_PER_NODE=90.9 num_servers=12 p50=0.08300 p95=0.08500 p99=0.08500 p999=0.08600 AVG=0.08333; P99_AVG_RATIO=1.02000; ITEMS_PER_NODE=83.3 num_servers=13 p50=0.07700 p95=0.07800 p99=0.07900 p999=0.07900 AVG=0.07692; P99_AVG_RATIO=1.02700; ITEMS_PER_NODE=76.9 num_servers=14 p50=0.07200 p95=0.07300 p99=0.07300 p999=0.07400 AVG=0.07143; P99_AVG_RATIO=1.02200; ITEMS_PER_NODE=71.4 num_servers=25 p50=0.04000 p95=0.04100 p99=0.04200 p999=0.04200 AVG=0.04000; P99_AVG_RATIO=1.05000; ITEMS_PER_NODE=40.0 num_servers=50 p50=0.02000 p95=0.02100 p99=0.02200 p999=0.02200 AVG=0.02000; P99_AVG_RATIO=1.10000; ITEMS_PER_NODE=20.0 num_servers=100 p50=0.01000 p95=0.01100 p99=0.01200 p999=0.01200 AVG=0.01000; P99_AVG_RATIO=1.20000; ITEMS_PER_NODE=10.0 num_servers=1000 p50=0.00100 p95=0.00200 p99=0.00200 p999=0.00300 AVG=0.00100; P99_AVG_RATIO=2.00000; ITEMS_PER_NODE=1.0 power of two choices num_items=2000: num_servers=3 p50=0.33350 p95=0.33400 p99=0.33400 p999=0.33450 AVG=0.33333; P99_AVG_RATIO=1.00200; ITEMS_PER_NODE=666.7 num_servers=5 p50=0.20000 p95=0.20050 p99=0.20100 p999=0.20150 AVG=0.20000; P99_AVG_RATIO=1.00500; ITEMS_PER_NODE=400.0 num_servers=10 p50=0.10000 p95=0.10050 p99=0.10100 p999=0.10100 AVG=0.10000; P99_AVG_RATIO=1.01000; ITEMS_PER_NODE=200.0 num_servers=11 p50=0.09100 p95=0.09150 p99=0.09200 p999=0.09200 AVG=0.09091; P99_AVG_RATIO=1.01200; ITEMS_PER_NODE=181.8 num_servers=12 p50=0.08350 p95=0.08400 p99=0.08400 p999=0.08450 AVG=0.08333; P99_AVG_RATIO=1.00800; ITEMS_PER_NODE=166.7 num_servers=13 p50=0.07700 p95=0.07750 p99=0.07800 p999=0.07800 AVG=0.07692; P99_AVG_RATIO=1.01400; ITEMS_PER_NODE=153.8 num_servers=14 p50=0.07150 p95=0.07200 p99=0.07250 p999=0.07250 AVG=0.07143; P99_AVG_RATIO=1.01500; ITEMS_PER_NODE=142.9 num_servers=25 p50=0.04000 p95=0.04050 p99=0.04100 p999=0.04100 AVG=0.04000; P99_AVG_RATIO=1.02500; ITEMS_PER_NODE=80.0 num_servers=50 p50=0.02000 p95=0.02050 p99=0.02100 p999=0.02100 AVG=0.02000; P99_AVG_RATIO=1.05000; ITEMS_PER_NODE=40.0 num_servers=100 p50=0.01000 p95=0.01050 p99=0.01100 p999=0.01100 AVG=0.01000; P99_AVG_RATIO=1.10000; ITEMS_PER_NODE=20.0 num_servers=1000 p50=0.00100 p95=0.00150 p99=0.00200 p999=0.00200 AVG=0.00100; P99_AVG_RATIO=2.00000; ITEMS_PER_NODE=2.0 power of two choices num_items=5000: num_servers=3 p50=0.33340 p95=0.33360 p99=0.33360 p999=0.33380 AVG=0.33333; P99_AVG_RATIO=1.00080; ITEMS_PER_NODE=1666.7 num_servers=5 p50=0.20000 p95=0.20020 p99=0.20040 p999=0.20060 AVG=0.20000; P99_AVG_RATIO=1.00200; ITEMS_PER_NODE=1000.0 num_servers=10 p50=0.10000 p95=0.10020 p99=0.10040 p999=0.10040 AVG=0.10000; P99_AVG_RATIO=1.00400; ITEMS_PER_NODE=500.0 num_servers=11 p50=0.09100 p95=0.09120 p99=0.09120 p999=0.09140 AVG=0.09091; P99_AVG_RATIO=1.00320; ITEMS_PER_NODE=454.5 num_servers=12 p50=0.08340 p95=0.08360 p99=0.08360 p999=0.08380 AVG=0.08333; P99_AVG_RATIO=1.00320; ITEMS_PER_NODE=416.7 num_servers=13 p50=0.07700 p95=0.07720 p99=0.07720 p999=0.07740 AVG=0.07692; P99_AVG_RATIO=1.00360; ITEMS_PER_NODE=384.6 num_servers=14 p50=0.07140 p95=0.07160 p99=0.07180 p999=0.07180 AVG=0.07143; P99_AVG_RATIO=1.00520; ITEMS_PER_NODE=357.1 num_servers=25 p50=0.04000 p95=0.04020 p99=0.04040 p999=0.04040 AVG=0.04000; P99_AVG_RATIO=1.01000; ITEMS_PER_NODE=200.0 num_servers=50 p50=0.02000 p95=0.02020 p99=0.02040 p999=0.02040 AVG=0.02000; P99_AVG_RATIO=1.02000; ITEMS_PER_NODE=100.0 num_servers=100 p50=0.01000 p95=0.01020 p99=0.01040 p999=0.01040 AVG=0.01000; P99_AVG_RATIO=1.04000; ITEMS_PER_NODE=50.0 num_servers=1000 p50=0.00100 p95=0.00120 p99=0.00140 p999=0.00140 AVG=0.00100; P99_AVG_RATIO=1.40000; ITEMS_PER_NODE=5.0 power of two choices num_items=10000: num_servers=3 p50=0.33330 p95=0.33340 p99=0.33350 p999=0.33360 AVG=0.33333; P99_AVG_RATIO=1.00050; ITEMS_PER_NODE=3333.3 num_servers=5 p50=0.20000 p95=0.20010 p99=0.20020 p999=0.20030 AVG=0.20000; P99_AVG_RATIO=1.00100; ITEMS_PER_NODE=2000.0 num_servers=10 p50=0.10000 p95=0.10010 p99=0.10020 p999=0.10020 AVG=0.10000; P99_AVG_RATIO=1.00200; ITEMS_PER_NODE=1000.0 num_servers=11 p50=0.09090 p95=0.09100 p99=0.09110 p999=0.09110 AVG=0.09091; P99_AVG_RATIO=1.00210; ITEMS_PER_NODE=909.1 num_servers=12 p50=0.08330 p95=0.08350 p99=0.08350 p999=0.08360 AVG=0.08333; P99_AVG_RATIO=1.00200; ITEMS_PER_NODE=833.3 num_servers=13 p50=0.07690 p95=0.07700 p99=0.07710 p999=0.07720 AVG=0.07692; P99_AVG_RATIO=1.00230; ITEMS_PER_NODE=769.2 num_servers=14 p50=0.07140 p95=0.07160 p99=0.07160 p999=0.07170 AVG=0.07143; P99_AVG_RATIO=1.00240; ITEMS_PER_NODE=714.3 num_servers=25 p50=0.04000 p95=0.04010 p99=0.04020 p999=0.04020 AVG=0.04000; P99_AVG_RATIO=1.00500; ITEMS_PER_NODE=400.0 num_servers=50 p50=0.02000 p95=0.02010 p99=0.02020 p999=0.02020 AVG=0.02000; P99_AVG_RATIO=1.01000; ITEMS_PER_NODE=200.0 num_servers=100 p50=0.01000 p95=0.01010 p99=0.01020 p999=0.01020 AVG=0.01000; P99_AVG_RATIO=1.02000; ITEMS_PER_NODE=100.0 num_servers=1000 p50=0.00100 p95=0.00110 p99=0.00120 p999=0.00120 AVG=0.00100; P99_AVG_RATIO=1.20000; ITEMS_PER_NODE=10.0 power of two choices num_items=100000: num_servers=3 p50=0.33333 p95=0.33334 p99=0.33335 p999=0.33336 AVG=0.33333; P99_AVG_RATIO=1.00005; ITEMS_PER_NODE=33333.3 num_servers=5 p50=0.20000 p95=0.20001 p99=0.20002 p999=0.20003 AVG=0.20000; P99_AVG_RATIO=1.00010; ITEMS_PER_NODE=20000.0 num_servers=10 p50=0.10000 p95=0.10001 p99=0.10002 p999=0.10002 AVG=0.10000; P99_AVG_RATIO=1.00020; ITEMS_PER_NODE=10000.0 num_servers=11 p50=0.09091 p95=0.09092 p99=0.09093 p999=0.09093 AVG=0.09091; P99_AVG_RATIO=1.00023; ITEMS_PER_NODE=9090.9 num_servers=12 p50=0.08333 p95=0.08335 p99=0.08335 p999=0.08336 AVG=0.08333; P99_AVG_RATIO=1.00020; ITEMS_PER_NODE=8333.3 num_servers=13 p50=0.07692 p95=0.07694 p99=0.07694 p999=0.07695 AVG=0.07692; P99_AVG_RATIO=1.00022; ITEMS_PER_NODE=7692.3 num_servers=14 p50=0.07143 p95=0.07144 p99=0.07145 p999=0.07145 AVG=0.07143; P99_AVG_RATIO=1.00030; ITEMS_PER_NODE=7142.9 num_servers=25 p50=0.04000 p95=0.04001 p99=0.04002 p999=0.04002 AVG=0.04000; P99_AVG_RATIO=1.00050; ITEMS_PER_NODE=4000.0 num_servers=50 p50=0.02000 p95=0.02001 p99=0.02002 p999=0.02002 AVG=0.02000; P99_AVG_RATIO=1.00100; ITEMS_PER_NODE=2000.0 num_servers=100 p50=0.01000 p95=0.01001 p99=0.01002 p999=0.01002 AVG=0.01000; P99_AVG_RATIO=1.00200; ITEMS_PER_NODE=1000.0 num_servers=1000 p50=0.00100 p95=0.00101 p99=0.00102 p999=0.00102 AVG=0.00100; P99_AVG_RATIO=1.02000; ITEMS_PER_NODE=100.0

29th Aug 2023 • 57 votes

How much does the read/write buffer size matter for socket throughput?

The read() and write() system calls take a variable-length byte array as an argument. As a simplified model, the time for the system call should be some constant "per-call" time, plus time directly proportional to the number of bytes in the array. That is, the time for each call should be time = (per_call_minimum_time) + (array_len) × (per_byte_time). With this model, using a larger buffer should increase throughput, asymptotically approaching 1/per_byte_time. I was curious: do real system calls behave this way? What are the ideal buffer sizes for read() and write() if we want to maximize throughput? I decided to do some experiments with blocking I/O. These are not rigorous, and I suspect the results will vary significantly if the hardware and software are different than one the system I tested. The really short answer is that a buffer of 32 KiB is a good starting point on today's systems, and I would want to measure the performance to go beyond that. However, for large writes, performance can increase. On Linux, the simple model holds for small buffers (≤ 4 KiB), but once the program approaches the maximum throughput, the throughput becomes highly variable and in many cases decreases as the buffers get larger. For blocking I/O, approximately 32 KiB is large enough to hit the maximum throughput for read(), but write() throughput improves with buffers up to around 256 KiB - 1 MiB. The reason for the asymmetry is that the Linux kernel will only write less than the entire buffer (a "short write") if there is an error (e.g. a signal causing EINTR). Thus, larger write buffers means the operating system needs to switch to the process less often. On the other head, "short reads", where a read() returns less than the maximum length, become increasingly common as the buffer size increases, which diminishes the benefit. There is a SO_RCVLOWAT socket option to change this that I did not test. The experiments were run on two 16 CPU Google Cloud T2D instances, which use AMD EPYC Milan processors (3rd generation, released in 2021). Each core is a real physical core. I used Ubuntu 23.04 running kernel 6.2.0-1005-gcp. My benchmark program is written in Rust and is available on Github. On localhost, Unix sockets were able to transfer data at approximately 9000 MiB/s. Localhost TCP sockets were a bit slower, around 7000 MiB/s. When using two separate cloud VMs with a networking throughput limit of 32 Gbps = 3800 MiB/s, I needed to use 6 TCP sockets to reliably reach that maximum throughput. A single TCP socket gets around 1400 MiB/s with 256 KiB buffers, with peaks as high as 2200 MiB/s. Experiment 1: /dev/zero and /dev/urandom My first experiment is reading from the /dev/zero and /dev/urandom devices. These are software devices implemented by the kernel, so they should have low overhead and low variability, since other tasks are not involved. Reading from /dev/urandom should be much slower than /dev/zero since the kernel must generate random bytes, rather than just zeros. The chart below shows the throughput for reading from /dev/zero as the buffer size is increased. The results show that the basic linear time per system call model holds until the system reaches maximum throughput (256 kiB buffer = 39000 MiB/s for /dev/zero, or 16 kiB = 410 MiB/s for /dev/urandom). As the buffer size increases further, the throughput decreases as the buffers get too big. This suggests that some other cost for larger buffers starts to outweigh the reduction in number of system calls. Perhaps CPU caches become less effective? The AMD EPYC Milan (3rd gen) CPU I tested on has 32 KiB of L1 data cache and 512 KiB of L2 data cache per core. The performance decreases don't exactly line up with these numbers, but it still seems plausible. The numbers for /dev/urandom are substantially lower, but otherwise similar. I did a linear least-squares fit on the average time per system call, shown in the following chart. If I use all the data, the fit is not good, because the trend changes for larger buffers. However, if I use the data up to the maximum throughput at 256 KiB, the fit is very good, as shown on the chart below. The linear fit models the minimum time per system call as 167 ns, with 0.0235 ns/byte additional time. If we want to use smaller buffers, using a 64 KiB buffer for reading from /dev/zero gets within 95% of the maximum throughput. Experiment 2: Unix and localhost TCP sockets Exchanging data with other processes is the thing I am actually interested in, so I tested Unix and TCP sockets on a single machine. In this case, I varied both the write buffer size and the read buffer size. Unfortunately, these results vary a lot. A more robust comparison would require running each experiment many times, and using some sort of statistical comparison. However, this "quick and dirty" experiment satisfied my curiousity, so I didn't do that. As a result, my conclusions here are vague. The simple model that increasing buffer size should decrease overhead is true, but only until the buffers are about 4 KiB. Above that point, the results start to be highly variable, and it is much harder to draw general conclusion. However, appears that increasing the write buffer size generally is quite helpful up to at least 256 KiB, and often needed as much as 1 MiB to get the highest localhost throughput. I suspect this is because on Linux with blocking sockets, write() will not return until it has written all the data in the buffer, unless there is an error (e.g. EINTR). As a result, passing a large buffer means the kernel can do a lot of the work without needing to switch back to user space. Unfortunately, the same is not true for read(), which often returns "short reads" with any data that is available in the buffer. This starts with buffer sizes around 2 KiB, with the percentage of short reads increasing as the buffer size gets larger. This means the simple model does not hold, because we aren't actually increasing the bytes per read call. I suspect this is a factor which means this microbenchmark is likely not representative of real programs. A real program will do something with the buffer, which will provide time for more data to be buffered in the kernel, and would probably decrease the number of short reads. This likely means larger buffers are in practice more useful than this microbenchmark suggests. As a result of this, the highest throughput often was achievable with small read buffers. I'm somewhat arbitrarily selecting 16 KiB at the best read buffer, and 256 KiB as the best write buffer, although a 1 MiB write buffer seems to be To give a sense of how variable the results are, the plot below shows the local Unix socket throughput for each read and write buffer throughput size. I apologize for the ugly plot. I did not want to spend the time to make it more beautiful. This plot is interactive so you can slice the data to the area of interest. I recommend zooming in to the left hand size with read buffers up to about 300 KiB. The first thing to note is at least on Linux with blocking sockets, the writer will almost never have a "short write", where the write system call returns before writing all the data in the buffer. Unless there is a signal (EINTR) or some other "error" condition, write() will not return until all the bytes are written. The same is not true for reads. The read() system call will often return a "short" read, starting around buffer sizes of 2 KiB. The percentage of short reads generally increases as buffer sizes get bigger, which is logical. Another note is that sockets have in-kernel send and receive buffers. I did not tune these at all. It is possible that better performance is possible by tuning these settings, but that was not my goal. I wanted to know what happens "out of the box" for general-purpose programs without any special tuning. Experiment 3: TCP between two hosts In this experiment, I used two separate hosts connected with 32 Gbps networking in Google Cloud. I first tested the TCP throughput using iperf, to independently verify the network performance. A single TCP connection with iperf is not enough to fully utilize the network. I tried fiddling with some command line options and with Kernel settings like net.ipv4.tcp_rmem and wasn't able to get much better than about 12 Gb/s = 1400 MiB/s. The throughput is also highly varible. Here is some example output with iperf reporting at 2 second intervals, where you can see the throughput ranging from 10 to 19 Gb/s, with an average over the entire interval of 12 Gb/s. To hit the maximum network throughput, I need to use 6 or more parallel TCP connections (iperf -c IP_ADDRESS --time 60 --interval 2 -l 262144 -P 6). Using 3 connections gets around 26 Gb/s, and using 4 or 5 will occasionally hit the maximum, but will also occasionally drop down. Using at least 6 seems to reliably stay at the maximum. Due to this variability, it is hard to draw any conclusions about buffer size. In particular: a single TCP connection is not limited by CPU. The system uses about 40% of a single CPU core, basically all in the kernel. This is more about how the buffer sizes may impact scheduling choices. That said, it is clear that you cannot hit the maximum throughput with a small write buffer. The experiments with 4 KiB write buffers reached approximately 300 MiB/s, while an 8 KiB write buffer was much faster, around 1400 MiB/s. Larger still generally seems better, up to around 256 KiB, which occasionally reached 2200 MiB/s = 17.6 Gb/s. The plot below shows the TCP socket throughput for each read and write buffer size. Again, I apologize for the ugly plot.

16th Jul 2023 • 103 votes

The C Standard Library Function isspace() Depends on Locale

This is a post for myself, because I wasted a lot of time understanding this bug, and I want to be able to remember it in the future. I expect close to zero others to be interested. The C standard library function isspace() returns a non-zero value (true) for the six "standard" ASCII white-space characters ('\t', '\n', '\v', '\f', '\r', ' '), and any locale-specific characters. By default, a program starts in the "C" locale, which will only return true for the six ASCII white-space characters. However, if the program changes locales, it can return true for other values. As a result, unless you really understand locales, you should use your own version of this function, or ICU4C's u_isspace() function. An implementation of isspace() for ASCII is one line: /* Returns true for the 6 ASCII white-space characters: \t \n \v \f \r ' '. */ int isspace_ascii(int c) { return c == '\t' || c == '\n' || c == '\v' || c == '\f' || c == '\r' || c == ' '; } I ran into this because On Mac OS X, Postgres switches to the system's default locale, which is something that uses UTF-8 (e.g. en_US.UTF-8, fr_CA.UTF-8, etc). In this case, isspace() returns true for Unicode white-space values, which includes 0x85 = NEL = Next Line, and 0xA0 = NBSP = No-Break Space. This caused a bug in parsing Postgres Hstore values that use Unicode. I have attempted to submit a patch to fix this (mailing list post, commitfest entry). For a program to demonstrate the behaviour on different systems, see isspace_locale on Github.

6th Jun 2023 • 110 votes

Huge Pages are a Good Idea

Nearly all programs are written to access virtual memory addresses, which the CPU must translate to physical addresses. These translations are usually fast because the mappings are cached in the CPU's Translation Lookaside Buffer (TLB). Unfortunately, virtual memory on x86 has used a 4 kiB page size since the 386 was released in 1985, when computers had a bit less memory than they do today. Also unfortunately, TLBs are pretty small because they need to be fast. For example, AMD's Zen 4 Microarchitecture, which first shipped in September 2022, has a first level data TLB with 72 entries, and a second level TLB with 3072 entries. This means when an application's working set is larger than approximately 4 kiB × 3072 = 12 MiB, some memory accesses will require page table lookups, multiplying the number of memory accesses required. This is a brand-new CPU, with one of the biggest TLBs on the market, so most systems will be worse. Using larger virtual memory page sizes (aka huge pages) can reduce page mapping overhead substantially. Since RAM is so much larger than it was in 1985, a larger page size seems like obviously a good idea to me. In 2021, Google published a paper about making their malloc implementation (TCMalloc) huge page aware (called Temeraire). They report this improved average requests-per-second throughput across their fleet by 7%, by increasing the amount of memory that is backed by huge pages. This made me curious about the "best case" performance benefits. I wrote a small program that allocates 4 GiB, then randomly reads uint64 values from it. On my Intel 11th generation Core i5-1135G7 (Tiger Lake) from 2020, using 2 MiB huge pages is 2.9× faster. I also tried 1 GiB pages, which is 3.1× faster than 4 kiB pages, but only 8% faster than 2 MiB pages. My conclusion: Using madvise() to get the kernel to use huge pages seems like a relatively easy performance win for applications that use a large amount of RAM. Unfortunately, using larger pages is not without its disadvantages. Notably, when the Linux kernel's transparent huge page implementation was first introduced, it was enabled by default, which caused many performance problems. See the section below for more details. Today's default to use huge pages only for applications that opt-in (aka madvise) should improve this. The kernel's policies for managing huge pages have also changed since then, and are hopefully better now. At the very least, the fact that Google uses transparent huge pages for all their applications is some evidence that this can work for a wide variety of workloads. The second problem with larger page sizes is software incompatibility, since so much software is only tested on x86 with 4 kiB pages. Linux on ARM64 used to default to 64 kiB pages. However, this caused many problems (e.g. dotnet, Go, Chrome, jemalloc, Asahi Linux list of broken software). It appears that around 2020 most distributions switched to 4 kiB pages to avoid these problems (e.g. RedHat RHEL9 change in 2021, Ubuntu note about the page size change). Page size historical details Other CPU architectures have made different page size choices. Notably, iOS and Mac OS X on ARM64 uses 16 kiB pages (ARM64 aka aarch64 supports 4, 16, and 64 kiB pages, although specific CPUs will only support some of them). Alpha and Sparc used 8 kiB pages. PowerPC on Linux uses 64 kiB pages, although Redhat/Fedora are considering switching to 4 kiB due to the same compatibility issues. See page sizes used by Windows on various processors. Latency and throughput problems with transparent huge pages The Linux kernel's implementation of transparent huge pages has been the source of performance problems. When introduced, it was initially enabled for all processes and memory regions by default. This caused a large number of problems, which eventually caused the kernel's default to change to madvise, where programs have to opt-in to use huge pages (see Nelson Elhage's summary (2017), and Ubuntu bug that changed the default (2017/released 2019). The performance problems are rare high latency (e.g. operations being substantially slower than normal), throughput issues due to excess CPU consumption of the kernel background tasks, or substantial increases in memory usage. Some examples are Hadoop (2012), TokuDB/MySQL (2014), Redis/jemalloc (2015), TiKV/TiDB (2020). The problems seem to fall into the following categories: Increasing memory usage by making fragmentation worse: using transparent huge pages rounds allocations up to 2 MiB. If an application allocates many separate memory regions, this can cause lots of memory to be wasted. Most of the problems have been where an application uses a large amount of memory, then frees a lot of it, leaving "holes" in the large pages. Sometimes the kernel's transparent page policy can decide to turn these back into huge pages, which causes the memory usage to increase. For example, see a Go bug (2015) and the corresponding kernel bug report (2015). The fix for Go was to only return memory on huge page granularity. This also happened to Redis with jemalloc (2015) malloc implementations that are not huge page aware may add more kernel CPU overhead: When returning memory to the operating system, if the memory allocator is not aware of huge pages, it may return part of a huge page. This causes the kernel to split the huge page back into separate 4 kib pages. This adds overhead, and also fragments memory, making fewer huge pages available, causing the kernel to do more work the next time it tries to allocate a huge page. This article about TokuDB from 2014 suggests that it ran into this problem with jemalloc. The good news is that it now seems like all major malloc implementations (jemalloc, tcmalloc, mimalloc, and glibc malloc) all have some huge page support, which should make this less bad. slow memory allocations due to fragmentation (latency): When trying to allocate a huge page, the kernel may spend time moving memory around to free up a page. See a detailed thread about impacts on the JVM (2017). The kernel's current default is to only do this for regions that have opted in with madvise. This should mean that other processes won't be penalized too much by this, but it does mean the process that called madvise could be stalled briefly when allocating new pages. One way to avoid this is to immediately touch every huge page in an allocation, to cause the cost to happen up front. This would work well for allocations that are made at program startup, such as caches. fork() e.g. Redis: Calling fork marks all of the process's pages as copy-on-write. Then when a single byte on a page is modified, the page must be copied. Redis uses fork to create a read-only "snapshot" of memory, when writing a checkpoint to disk. Since huge pages are 512X larger than "normal" pages, the time to copy a page increases by 512X. It also means the memory usage is higher, since modifying a single byte causes 2 MiB to be copied, instead of only 4 kiB. Using fork() in this way with huge pages seems like a bad idea. See details about a workload that causes this behavior (2014). References Huge Page Demo Evan Jones 2022-01-18: My huge page demonstration program. Larger Pages: Richard Sites 2022-05-06: Argues we should increase the minimum page size to 64 kiB, and maintain compatibility by using access flags on 4 kiB sub-pages. Stack Overflow: Why is the page size 4 KB? Answer by Hadi Brais 2018-04-26: a great look at the history of why 4 kiB pages were chosen. Using huge pages on Linux: Erik Rigtorp 2020-10-08: A hash table benchmark in C++ with results for both transparent and explicit huge pages. Reliably allocating huge pages in Linux: Francesco Mazzoli 2021-11-22: Includes C code describing how to verify if an address is a huge page. Intel Coffee Lake Microarchitecture (2017 aka Core 9th gen): L1 Data TLB: 64 entries for 4 kiB pages / 32 entries for 2 MiB pages / 4 entries for 1 GiB pages ; L2 unified TLB: 1536 for 4 kiB/2 MiB pages; 16 entries for 1 GiB pages.

16th Jan 2023 • 62 votes

More in programming

TokyoDev Welcomes Its Head of Operations

TokyoDev has grown substantially over the past few years. We’ve expanded our community, dramatically increased the scope and frequency of our events, and have curated a library of articles for international developers. Despite all that, the way we run the business hasn’t changed much. I’ve continued to coordinate most of these activities myself, making the final decisions and keeping everything connected. In short, TokyoDev cannot run without my daily input. That approach worked well when TokyoDev was smaller, but it’s increasingly limiting how much we can accomplish. To help the business grow further, I wanted to find someone to take over some of my responsibilities and bring a fresh perspective. This is a significant step for me personally, as while I have worked with many contractors over the years, I have never actually hired a regular employee. Most importantly, I wanted to ensure that this person was someone I could trust to understand what TokyoDev stands for, and to make decisions in line with our philosophy. So I’m happy to announce that Scott Rothrock is joining TokyoDev full time as our Head of Operations. About Scott Scott has lived in Japan for nearly 20 years and has worked as a software engineer in multiple industries and environments, including small Japanese-owned businesses, overseas employers, and international companies based in Japan. Until the end of October 2026, he was the head of the Japan Markets team at Kraken Technologies, where he led an international team of software engineers in delivering complicated, regulated products on strict timelines for Japanese electricity and gas retailers, while also ensuring compliance with evolving organizational and global standards. He’s made a name for himself both inside and outside Japan as a kind, effective mentor who can cultivate great engineers and leaders. Scott’s previous role in TokyoDev Some of you may already know Scott in another capacity, as he has been involved with TokyoDev since 2022. He began as a community volunteer and later became our community manager, which also included responsibilities such as managing our article pipeline from submission to publishing. When I created TokyoDev’s Discord, I had no previous experience running an online community. Scott joined soon after I started and began helping other members, stepping in when people behaved inappropriately, and configuring our roles and permissions and safety features. I was impressed by his initiative and asked him to become an official moderator in January 2023. Outside of his role as a moderator, Scott has always pushed TokyoDev to grow; for example, he arranged internet events such as livestreamed AMAs and interviews. He organized TokyoDev’s first meetup as a volunteer in February 2023, which taught me that there was interest in TokyoDev events outside of the internet. By the end of 2023, I had brought him on as our part-time community manager. What I saw throughout his tenure was that Scott continued to take initiative in expanding TokyoDev. Scott was one of our first external contributors, writing about his experience living with a disability in Japan. He encouraged me to expand and refine both our editing and submission processes so that we could reach out to even more contributors and platform their perspectives. The editorial team now consists of him and Rebecca Callahan, and they’ve had a hand in every article that TokyoDev has published. In other words, since 2022, Scott has shown that he cares about TokyoDev as much as I do. He is invested in what we do and why we do it, and shares my vision for what TokyoDev can become. What changes now Scott’s experiences as an engineer, a mentor, and a leader make me confident that, as our Head of Operations, he will continue to help TokyoDev mature. A big part of what has made TokyoDev successful is that I see the software developers who use our site as my peers and genuinely want to help them. Scott shares that perspective, and will join me in ensuring that TokyoDev remains true to its principles while we grow. So what changes in the short term? Nothing dramatic. I will teach Scott about managing TokyoDev as a business, and he will document our processes and products as a part of that learning. He will also look for areas of improvement while ensuring that we remain compliant with regulations and labor law. Over time, I will hand over some other responsibilities. I will continue to run TokyoDev, but TokyoDev will become less dependent on me for its day-to-day operations. Long term, I’m excited about what this will enable us to do. TokyoDev has always been about helping international software developers build successful careers in Japan, and bringing Scott on full time puts us in a much better position to continue doing that. We look forward to writing the next chapter of TokyoDev together.

2 hours ago • 1 votes

A proposal: how to hire junior engineers in the age of AI

3 hours ago • 1 votes

Speeding up Kidz Fun Art & making Google love it

Kidz Fun Art is a drawing, colouring and animation app for kids that I’ve been building since 2021. It runs in the browser also as a simple wrapper app on iOS, iPadOS and Windows. It wasn’t getting a great click through rate on Google, even for people who directly searched for it. So, for two days I took a hard look at how fast it loads and how search engines see it. It wasn’t flattering. This post covers what I found, what I changed and what the graphs say now. Short version: on a simulated slow phone, the homepage’s Largest Contentful Paint got about 2.6× faster (15.9 s to 6.1 s) and its first load is about 82% smaller (4.2 MB to 0.75 MB). I also fixed a set of SEO mistakes that are, frankly, embarrassing to write down. I’ll write them down anyway. Part 1: Google had no idea what the site was for I started by reading 16 months of Google Search Console data. 9,380 clicks in total. About 9,200 came from people who had typed “kidz fun art” or a close spelling. People searching for what the app actually does (“coloring app for kids”, “ai art for kids”, “make a comic book”) sent the site 164 clicks. In 16 months. Total. When there was a relevant page, Google ranked it well. The site was #1 for things like “drawing app for kids ipad” and a German query that means “paint by numbers app without ads”. There just weren’t many relevant pages. Then I looked at the code, and the cause was obvious. The homepage <title> was “Kidz Fun Art” in all 12 languages. That’s a fine name, and a terrible title. It tells a search engine nothing. The homepage’s first <h1> was the pricing table’s “Choose your experience”. I was, in effect, telling Google that the most important thing on my homepage was the checkout. Every blog post had the same meta description: a generic line about “news, updates, tips”. Each post’s front matter already had a perfectly good excerpt field that nobody read. Blog posts had two <h1>s: the site header and the post title. There was no structured data at all. The sitemap that redirected every single URL This one stings. The sitemap is the one file whose whole job is to say “here are my URLs, please index them”. Mine listed 17 URLs, and every one of them redirected. The sitemap build used an environment variable that resolved to kidzfun.art. The site lives on www.kidzfun.art, so every URL got a 308. Only the English blog posts were listed. The other 11 languages appeared only as hreflang alternates, which pointed at redirecting URLs too. There were no <link rel="canonical"> tags on any page and no hreflang in the page <head>. Google was left to guess which URL was the real one, and it guessed wrong a lot. What I changed: The sitemap is now built on https://www.kidzfun.art, with 204 URLs: every home page, blog index, blog post, About page and gallery page, in all 12 languages. Each one lists all of its language alternates plus an x-default. A shared SeoLinks component adds an absolute canonical (www, no query string) and the full hreflang set to every public page. The ?v= cache-busting URLs that had been showing up as duplicates now canonicalise to the clean URL. /xx/blog/yy/... URLs, which returned a 500 in production (oops), now 308 to /blog/yy/.... Login, account, debug, export and embed pages send X-Robots-Tag: noindex, follow from next.config.js, so they stop using up crawl budget. robots.txt now mentions the sitemap. Novel idea. Why Google skipped half the site Search Console’s Pages report had 93 of 174 known pages not indexed. Every reason on that list traces back to the mistakes above: “Page with redirect” (22): that sitemap, plus internal links to the non-www host. “Duplicate without user-selected canonical” (18): I hadn’t selected any canonicals. Fair cop. “Crawled, currently not indexed” (39): mostly translated posts that Google reached through redirecting URLs, plus thin pages like /login. “Discovered, currently not indexed” (12): Google found these and decided they weren’t worth a visit. With no sitemap entry and no hreflang, I hadn’t given it a reason to. On-page fixes Titles and descriptions: Each blog post got its own seoTitle and description, in all 12 languages. The meta description now comes from the post’s excerpt, which is what that field was for all along. The About pages take their title and description from their own localized Markdown. They had been sharing an English title and a generic description. One <h1> per page: A # heading inside a post’s Markdown now renders as an <h2>, so the post title is the only <h1>. The About page opts back in with a keepH1 prop. Its Markdown heading is the page title. This fix arrived in two parts. The first part left the About page with zero <h1>s. I noticed, eventually. Structured data and social tags: JSON-LD for the home page (the app), blog posts (articles) and the embedded videos. Open Graph and X/Twitter tags on every public page, so shared links get a proper card. A real 404 for deleted gallery images: The AI gallery shows a friendly “AI Artwork Not Found” page when an image has been deleted. It used to return 200 OK, which Google calls a “soft 404” and dislikes. It now returns a 404 with noindex, and a 503 if the Firestore lookup itself fails, so a database hiccup tells crawlers “come back later” rather than “this is gone”. Part 2: Making the homepage fast The homepage is the app. It works offline, so every dialog (save, stickers, colouring pages, “get an adult”, “no wifi”) is server-rendered into the HTML up front. That’s a deliberate trade-off, and I kept it because I want new users to be able to immediately be able to start drawing as soon as they launch the app. What I hadn’t thought about was what it meant for the network. What the homepage downloaded before the app could start I measured the live site with a Lighthouse-style mobile profile: slow 4G (1.6 Mbps, 150 ms latency), 4× CPU slowdown, empty cache. First load: 4.5 MB over 198 requests. The homepage HTML alone was 358 KB, with about 77 <img> tags in hidden dialogs. The browser dutifully fetched every one of those images. The script that starts the app (main.min.js, 191 KB) sat at the very end of the document, so the browser found it last and gave it whatever bandwidth was left. The app script finished arriving at about 14 seconds. The CPU was not the problem. Every long task was under 300 ms. This was purely about bandwidth and request order. I had built a very polite queue and put my own app at the back of it. Watch the queue The waterfall is a simplified illustration. The milestone times (first paint, LCP, load) are measured: slow 4G + 4× CPU, local production build, median of 3 runs. My Largest Contentful Paint was the “Contact support” link The canvas doesn’t count toward LCP, and the toolbar buttons are small. So on a slow load the largest thing painted was… the loading dialog. The loading dialog fades in its “Taking a while? Contact support” text after 5 to 7.5 seconds. On a slow connection, that text became the page’s LCP element, at 15.9 s. I was measuring my page speed by how long it took to suggest the user complain about my page speed. There’s a lesson in there somewhere. The fix wasn’t to change the dialog. It was to make the app start sooner, so it paints before that text ever appears. What I changed Preload the app script. I added <link rel="preload" as="script"> in <head> for main.min.js and errorHandler.min.js. The <script> tags stay where they were, so execution order doesn’t change. The browser just starts fetching them before any image. This one line did more than anything else on this list. It’s a little humbling. Turn the three dialog GIFs into MP4s. Three animated GIFs: an arrow pushing pictures into a tablet (927 KB), a dancing rainbow pencil (650 KB) and a paper plane (305 KB). That’s 1.9 MB of decoration in dialogs most visitors never open on their first visit. I re-encoded them with ffmpeg: H.264 High profile, -preset veryslow -crf 20, yuv420p, +faststart, constant frame rate taken from each GIF’s frame delay. The commands are committed as scripts/gif-to-mp4.sh. The result is 130 KB in total, with SSIM between 0.988 and 0.993 against the originals. The pencil needed tpad=stop_mode=clone:stop_duration=0.2, because ffmpeg’s GIF reader drops the final 250 ms hold and the dance lost its big finish. The new <video autoplay loop muted playsinline> tags render at exactly the same size and class as the old <img> tags. Alt text moved to aria-label. Locked colouring thumbnails were full-size PNGs. In the colouring page picker, locked pages used the full-size colouring PNG as their background image, inlined into the HTML. That came to about 1.3 MB on every first visit, to draw thumbnails in a dialog that was closed. They now use the small image, set when the picker opens, like the unlocked ones always did. Sticker thumbnails wait their turn. I tried loading="lazy" first. It didn’t help: Chrome fetched the images before the stylesheet that hides the dialogs had arrived, so as far as it knew they were visible. Sticker thumbnails and the media library’s YouTube thumbnail now use data-deferred-src. They load 3 seconds after the load event, or as soon as their dialog opens, whichever comes first. The service worker preload starts after load. The offline preload queue (about 2 MB of colouring pages and stickers) used to start while the page was still loading and compete with it. Now it starts after the load event. Same files, same offline result, no fighting for bandwidth. Analytics is async. Nothing waited for it, so nothing needed to block on it. Cache headers. /images/*: max-age=86400, stale-while-revalidate=604800. It had been max-age=0, so every image was re-checked on every visit. Versioned /js, /css and /addToHomescreen files are immutable for a year in production. Offline still has to work Offline support is a core feature, so “just don’t download it” was never an option. Some of the changes above put it at risk, and a couple of things turned up along the way: Safari plays video from byte ranges. Safari requests video with Range: bytes=.... Answer that with a full cached response and the video won’t play. The service worker now answers Range requests from its cache with a proper 206 Partial Content slice (lib/util/rangeResponse.ts, with unit tests). First-visit gap: on a first visit the service worker isn’t controlling the page yet, so dialog images only reached the offline cache on the second online visit. The page’s own dialog images and videos now go to the front of the offline preload queue on the first visit. I nearly downloaded my whole site, 12 times over. The service worker cached every <link href> it found in the page. The new SEO work had just added canonical and hreflang <link>s pointing at all 12 localized homepages. So every new visitor would have downloaded about 12 extra homepages, in languages they don’t speak, “for offline use”. I caught it before production. The service worker now skips rel=canonical and rel=alternate. Safari stopped the service worker mid-download. The old preload queue ran on timers after the request had been answered. Safari considers an idle service worker fair game, so it stopped it after the first few images. The page now sends images in batches of 10, and the service worker caches each batch before it replies. A page lock-up I’d never noticed. resizeDialog added a new “image loaded” listener every time the window resized. With lots of images loading at once, the listeners multiplied until the page locked up. It now attaches one listener per image, which, in hindsight, is how many it needed. Part 3: The blog Blog index cards used the full 1920 px screenshots as covers, up to 500 KB each, in a box about 200 px wide. Each card was a little billboard, shrunk down by the browser on every visit. Covers now go through next/image with sizes that match the actual card widths, so the browser gets a resized WebP. The first few cards get priority. The rest load lazily. The first sizes hint was too small at some widths and made covers blurry. It now matches the real card widths, up to about 630 px between 769 px and 1028 px wide. Post pages: The cover image gets fetchpriority="high", and images further down the post load lazily. Five images were recompressed in place, keeping the same filenames and dimensions. The PNGs came out about 4× smaller and the JPEGs about 40% smaller. I deleted a 1.1 MB cover.png that nothing referenced. All 12 translations of that post used cover.jpg. The PNG had just been sitting in public/ taking up space. Results on the slow 4G profile: /blog LCP went from 13.3 s to 1.8 s. A typical post went from 3.9 s to 1.7 s. The gallery didn’t get faster, on purpose. It got a 5-minute edge cache and a cached Firestore image count, which fixed its slow server responses. I decided not to change its 20 full-size PNG thumbnails, so its lab LCP barely moved (8.4 s to 8.6 s). That’s a choice, and the chart owns it. Part 4: Lighthouse homework After the speed work went live I ran Lighthouse on the production homepage (mobile) and cleaned up what it flagged. Performance stayed in the low 70s, so I have some more work to do, but it’s moved nicely in the right direction. Part 5: Telling the search engines Fixing the site is half the job. The other half is getting crawlers to come back and notice. Google Search Console: Resubmitted the www sitemap. Clicked “Validate fix” on “Duplicate without user-selected canonical” and on the redirect errors. Requested indexing for the main language homepages and the top blog posts. The quota is about 10 a day, so it’ll take a few days. Wrote down a baseline from the Performance report, so in 4 to 8 weeks I can tell whether any of this worked. Bing Webmaster Tools: Submitted all 204 sitemap URLs through URL Submission. The daily quota is 10,000, so no trickle-feeding needed. Bing’s index also feeds DuckDuckGo and several AI search tools, so it’s cheap extra reach. Most of Bing’s “SEO recommendations” pointed at old ?v= URLs and old titles. They’ll clear as Bing recrawls. What I’m not watching: the Lighthouse score. Google ranks on real-user Chrome data (the Chrome UX Report): a rolling 28-day window, judged at the 75th percentile. So the honest answer to “did it work?” is “ask me in a month”. How I kept from breaking things The app has hundreds of unit and integration tests, all of which still pass (even though they do take over 30 minutes to run). I also manually regression tested all the riskiest changes. Hopefully I don’t make things worse before they get better… Lessons, mostly at my own expense Read your own sitemap. Actually open it and click a URL. Mine had been quietly redirecting for who knows how long. <link rel="preload"> for the script that boots your app is the cheapest big win there is. Browsers fetch resources in the order they find them. If your app script is at the bottom of a 358 KB document full of images, it really is last in line. Check what your LCP element actually is before you try to make it faster. Mine was a “Contact support” link. GIF is an image format from 1987 that I was using as a video format. A real video codec beats it by an order of magnitude at the same quality. try/catch doesn’t catch promise rejections. I knew that. My code didn’t. Every <link> you add to <head> is something your service worker might decide to download. Speed scores are for diagnosis. Rankings come from real users. Fix what the lab finds, then wait for the field data. Epilogue You’re still here? Wow. Cool. After all this, I’m hoping that users notice nothing other than quicker app start times. I’m looking forward to keeping an eye on traffic and retention in the coming weeks. Wish me luck!

4 hours ago • 1 votes

My new digital home

It's time for a dev blogger's favorite topic: blogging about my new blogging setup. After migrating my blog from Hexo to Eleventy, building a photography website and creating a personal one-pager, it was time to find a new digital home for all my projects. If you notice any issue after the migration, please let me know. I redirect the RSS feed to the new domain, but you might want to update the feed URL in your reader. Motivation I've been maintaining three different websites under two different names: a mostly developer-oriented blog, a photography portfolio and a one-pager gluing everything together. When I started blogging about my sabbatical journey, I was thinking about making yet another page to create a clear separation. I've decided to do the opposite and merge all my websites into one instead. As I further reduce my dependency on social media and third-party services, it has become even more important to have my own space and practice POSSE. Every decision I've made was driven by two main goals: simplicity and personality. Setup I've stopped chasing the latest and greatest. I want to keep the setup as clean as possible, which means pulling fewer dependencies. Every package adds complexity and maintainability costs, so I've only added what was necessary — no more Gulp, Sass or Tailwind. Instead of taking one of my existing websites as a starting point, I created this project from scratch. I reused some of my previous code, but I didn't want to take over any technical debt (especially with the advancements in CSS). The foundation of this website hasn't changed: Eleventy (Build Awesome). It covers the fundamentals of a static site and offers a great level of customization. I've made my project more and less dependent on Eleventy at the same time: I replaced Gulp as my build pipeline with pure Eleventy processing. I replaced most shortcodes with Markdown alternatives (and markdown-it plugins), which are better supported by editors like Obsidian. I've also replaced the Nunjucks template engine with Liquid, which comes with a few trade-offs. The main reason was to avoid creating a new collection for every set of posts I render. With Liquid, you can apply array functions like filtering or sorting in the template itself: {% assign drafts = collections.all | where: "data.isDraft" | reverse %} {% for draft in drafts %} … {% endfor %} Another advantage is having date formatting included, so I don't need another library or custom code. You can even set a default date format, so you don't have to provide it each time. What I don't like is the verbosity of the template language, although I've gotten used to it by now. Nunjucks: {{ "/tags/" + tag + "/" }} {{ modified or created or date }} {% set perspectives = ["dev", "photography", "life"] %} Liquid: {{ "/tags/" | append: tag | append: "/" }} {{ modified | default: created | default: date }} {% assign perspectives = "dev, photography, life" | split: ", " %} After migrating my existing content, I've noticed the build taking way too long, both in the dev server and the production build. I've found out that the Eleventy template cache is turned off for Liquid by default. This was useful during development, but when updating actual content, it caused every template and partial to be constantly recalculated. After turning the cache on, this issue was resolved: eleventyConfig.setLiquidOptions({ cache: true }); Check the colophon for the full tech stack overview. Perspectives The main challenge was finding a way to separate and present my main personas. The Venn diagram intersection of people interested in software development, photography and my personal life is probably small. I could use tags, but they don't offer the level of personalization I was looking for. Most personal blogs separate the content by post type (blog post, note, photo), but this doesn't fit my use case. Instead, I've decided to structure my website around three perspectives, a term I have borrowed from the Eclipse IDE. Each perspective comes with its own avatar, tagline and bio. If someone is interested in everything I share (hi Mom!), they can still view all my posts. Design Since building my photography website, I've made many changes, but I never got around to share an update. The biggest change was the design, which I have used as a blueprint for the new website. I've decided to mimic a social timeline/feed, which most people are familiar with. At the same time, it should be clear that it's a personal website and not yet another walled garden. Typography is an important part of every design. The first step is to choose between a system font stack and a custom font, which comes with the usual performance vs. personality trade-offs. I've decided to go with a uniform look across all devices. I've settled on Source Sans 3 for copy text and Hack for code. Another big change was introducing a sidebar. I always felt like the empty horizontal space was "wasted" on a desktop device, but getting the sidebar UX right is challenging. I'm happy with the result, especially with the last two panels adding a personal touch to the design. Images I've made several changes to the handling of image files. To make them compatible with Markdown editors, I had to replace my custom picture shortcode. The official Eleventy image plugin is a popular choice, but it violates my simplicity goal and doesn't support everything I needed. I've decided to go with a plain Markdown image markup and a single image file (no format fallback or 2x version for retina displays). Additionally, I wrote a markdown-it plugin to turn the optional title parameter into a caption. As for the file format, both WebP and AVIF have excellent browser support. For each image, I select the format based on the quality and file size. For my photos, I'm using WebP for the timeline thumbnails and JPG for the full-size version. Theme picker There have been some recent discussions about dark mode toggles, and I considered going for a light/dark solution based on the user's OS preference. Driven by the personality goal, I knew I had to add at least one more theme, so a proper theme picker was necessary. Similar to navigation, people like to hide the theme selection on mobile devices behind a burger menu or a dropdown. I prefer the solution by Sarah L. Fossheim, where all available themes are displayed above the page footer. A friend was scrolling through my new website, and their first question was "Why is the theme picker in the footer?". This tells me the solution was a good idea — I don't think he would have noticed a typical theme button in the header. The implementation follows standard practices: Use the light or dark theme as default based on the user's OS or browser preference. Store a selected theme in the local storage and use it as the single source of truth. Prevent a flash of inaccurate color theme. Make the theme play nicely with bfcache. I might have slightly gone overboard with my third theme, but the nostalgia kept me adding more and more features. Enjoy going back to the 90s: Feeds As browser vendors decided to remove XSLT support, I didn't bother to style my feed. Instead, I've created a new web feeds page, listing all available feeds and a short explanation. I provide one feed for every perspective and one with all my posts combined. I once offered tag-based RSS feeds. This was useful for my sabbatical, but even there I had only a single subscriber. Now, all tag feeds redirect to their main perspective feed. I've also dropped the JSON feed, despite having at least 16 subscribers. Given that every web feed reader supports Atom feeds, I don't see the value of providing a JSON alternative. A JSON feed doesn't add much complexity, but handling different media types (post, photo) and properly escaping the content is one more thing to implement and maintain. Quality assurance I've put a lot of effort into my new digital home, and I want to make sure it stays in good shape. I've included several tools, most running automatically as part of the CI pipeline: eleventyDataSchema — lint Eleventy data values like titles or dates markdownlint — format/lint Markdown files Vale — lint content (grammar, writing style, formatting) Lychee — find broken links (internal and external) Evaluatory — check accessibility and validate HTML Oxlint — lint JavaScript files Eleventy linters are another option to analyze a template's output, but I haven't found a use case for it yet. Not everything can be automated, so I also did the following manually: Test with Chrome, Firefox and Safari Test with Polypane Test with Google Lighthouse Test with a screen reader Test with JavaScript disabled Test with CSS disabled Test at 200% zoom Test with a bigger browser font size Test with custom fonts disabled Test with delayed assets Performance It's difficult to compare the performance of two separate websites with that of a combined one. Some parts got worse, e.g. bigger font files and more CSS. At the same time, I've noticed that gzip compression was turned off for anything but HTML for a few years now (I must have missed it when upgrading nginx) 🙈. One improvement was to implement on-demand assets. For example, the lite-youtube-embed CSS and JavaScript is only loaded if a <lite-youtube> element is present on the page. As before, JavaScript is fully optional (except for interactive demos in individual posts). I've been using a mix of <noscript> and CSS to hide elements that require JS (like the theme picker). I don't like the user experience of pagination, so I avoided it for years. But now that I have almost 300 posts, I've decided to finally add it. I display 30 posts per page, with images further down still being lazy-loaded for better performance. I haven't applied pagination to tag pages, as that's more complex with Eleventy. Bits and pieces Here are some other noteworthy changes: I've removed post-specific social share images and use a single WebP Open Graph image instead. I've dropped JSON-LD. Open Graph and microformats are sufficient. "Naming is hard" is a common expression in software development, but it also applies to photograph titles. I've decided to use a counter as the title, e.g. #0100. Sometimes I do come up with a title that I love, like the Castle in the Sky, so I might mention it in the description. On a photo page, you can now open the previous/next photo using the left/right arrow keys (in addition to the existing navigation buttons). It's now possible to filter photos by date, gear, or location. Check the explore page for an overview. I've learned about and used some experimental CSS features (with progressive enhancement), like scroll-triggered animations or squircle corners. I've added an AI statement on my LLM usage. I finally have a proper about page. Cool URIs don't change, but it's inevitable when moving to a new domain. All my previous URLs are redirected, except for some assets that are no longer used. I will keep the old domain, so the redirects will continue to work for a foreseeable future. I've kept my previous Easter egg and added a few more. Have a happy hunt! 🕵️‍♂️ Inspiration I visited hundreds of personal websites looking for inspiration. Here are a few that I kept coming back to: Emma Goto. I love Emma's approach to presenting the variety of her interests, even though I went a different route. Brennan Kenneth Brown. This website has so much to offer. It's a great example of what the open web looks like. Rex Barrett. The sentence "Collecting hobbies is my hobby." hit home — so much that I had to steal borrow it for my short life bio. Conclusion This project was mostly about merging the content of two websites into one. I knew this would take a considerable amount of time, but I still greatly underestimated the effort. In the end, it was all about the journey, not the destination. I loved fine-tuning the design and tweaking little details that probably no one will ever notice. I still have many more ideas left in my backlog. How do you like my new digital home? If you have an opinion, let me know! Reply via email

12 hours ago • 1 votes

Industrial Growth

New electronica album by Warrington-Runcorn New Town Development Plan

12 hours ago • 1 votes

BoredReading is a way to read what real humans are writing. Every hour, we find freshly written articles from hundreds of blogs. We've got some junk articles that sneak in, but mostly it's good long form writing, and sometimes hidden gems worth sharing!