More from Krzysztof Kowalczyk blog
New in the SumatraPDF pre-release builds: DDE commands accept arguments Commands sent via DDE can take arguments, the same as in custom shortcuts (#5383). Loading message in tab While a document loads, its tab shows a “loading” message instead of the home page (#5385). Install 32-bit on 64-bit Windows The installer lets you install the 32-bit version on 64-bit Windows (#5379). Changes for this day · Full changelog
I’m porting gpui-component (a Rust UI component library built on GPUI) to C++ as gpui-cpp. By which I mean: my friend Claude does the porting, I’m just directing. It uses markdown-rs (a CommonMark + GFM parser) markdown parser so I ported it too. Then I optimized it. This post describes what I did with the intention of teaching other how to optimize C++ code. The starting point There are 2 kinds of markdown parser: those that stream nodes as they parse those that build an AST in memory markdown-rs builds an AST. The game is about minimizing the size of AST node. In Rust there are various kinds of nodes, the largest being 152 bytes. Claude generated a single Node struct of 232 bytes. I got it down to 16 bytes. Here’s the initial Node struct, before optimizations: Node, 232 bytes k children 24 position 24 8 string fields — 128 bytes align 24 nums 16 grey = padding and small fields · blue = growable vector · yellow = pointer+length strings Node, 16 bytes (same scale) lastKid · sibling · firstStr · kind+flags Where the 232 went: 8 string fields at 16 bytes each (a char* plus a length), two growable vectors at 24 bytes each (children and table alignments), a 24-byte unist Position (line, column and offset at each end), six bools one to a byte, and the padding all of that dragged in. Every node in the tree pays for every field, whichever kind it is. A Text node uses one string field and nothing else. Arena allocator It’s important that all allocations are done in an arena. Nodes in a parse tree all have the same lifetime which makes it a perfect use for an arena: a bump allocator that can only grow. The only way to free memory is to reset the arena. This is different than calling malloc() to allocate each node individually and then having to call free(). It makes it easy to measure memory usage: check the arena size after parsing. It also allows optimization tricks like compressing pointers. How I measured bun cmd/bench.ts markdown parses 64 KB of markdown in four shapes and reports the arena bytes the parse allocated: prose — paragraphs, emphasis, links nested lists — deep blockquotes and lists gfm tables — tables all the way down entities — text that is mostly &-style character references The number is the whole arena: nodes, the tokenizer’s event list, and the strings. Not just sizeof(Node) × node count. We also measure parsing time to make sure we don’t trade size for speed. Baseline, 64 KB of source: prose 1646.1 KB (25.7× the source) nested lists 1067.9 KB gfm tables 2926.0 KB entities 660.2 KB 1. Pointer compression for strings (bed71ee) On 64-bit platforms, pointers are 8 bytes. Pointer compression reduces this to 4 bytes by calculating a 32-bit offset against a base pointer. Google used compressed pointers in v8 with great result. Reduced memory usage and increased speed. Our string type is the simplest possible string: struct Str { char* data; size_t len; }; That’s at least 12 bytes per string, if len is 4 bytes. Due to alignment, the size is 16 bytes. Strings are allocated in Arena so we can use the beginning of an arena as a base pointer and optimize the pointer from 8 bytes to 4 bytes. We typedef ArenaStr as uint64_t. The lower 4 bytes is uint32_t compressed pointer and upper uint32_t is size. We reduced the overhead of strings from 16 bytes to 8 bytes. Times 8 strings that’s 64 bytes saved per node. Added helper functions for allocating ArenaStr in arena and converting ArenaStr to Str. Savings: 8 strings * 8 bytes, 64 bytes per node: 232 → 168 bytes. shape start before after vs before vs start prose 1646.1 KB 1646.1 KB 1285.9 KB -21.9% -21.9% nested lists 1067.9 KB 1067.9 KB 867.5 KB -18.8% -18.8% gfm tables 2926.0 KB 2926.0 KB 2269.7 KB -22.4% -22.4% entities 660.2 KB 660.2 KB 626.2 KB -5.1% -5.1% 2. Growing arena strings in place (a9d4f3a) Some strings had to grow. Arena allocator doesn’t provide freeing or reallocation. You can only allocate new strings, which wastes memory by leaving dead copies of the string we were appending to. We can grow the last allocated string and that’s what this change does. Luckily, most appends were done to the last string. ArenaStrAppend checks whether the string ends exactly where the arena’s next allocation would begin. If it does, the new bytes are pushed straight onto it and nothing is copied. Decoding HTML entities (e.g. &) broke that optimization by doing an allocation before appending to the string. We switched to decoding entities into a 4-byte stack buffer which enabled optimized append. shape start before after vs before vs start prose 1646.1 KB 1285.9 KB 1285.9 KB +0.0% -21.9% nested lists 1067.9 KB 867.5 KB 729.2 KB -15.9% -31.7% gfm tables 2926.0 KB 2269.7 KB 2269.7 KB +0.0% -22.4% entities 660.2 KB 626.2 KB 163.8 KB -73.8% -75.2% 3. Re-order struct fields, pack the bools (5c0ce6e) Unless told to pack the layout of the struct, C++ compilers align struct fields to the size of the largest primitive type. If you sandwich a bool between 2 uint64_t values, the bool will occupy 8 bytes (sizeof(uint64_t)) instead of 1 byte as it should. Our Node had such wasted space due to padding. My friend Claude was careless. A simple fix is to re-arrange fields, putting the largest first. We also had six bool field which we packed into a uint8_t flags field. Result: 168 → 144 bytes, with no padding at all. We’re beating Rust version now. declaration order: bool after vector = 7 bytes of padding, six times over vector b padding strings b padding largest first, bools in one byte: no padding vectors strings nums f shape start before after vs before vs start prose 1646.1 KB 1285.9 KB 1150.9 KB -10.5% -30.1% nested lists 1067.9 KB 729.2 KB 654.0 KB -10.3% -38.8% gfm tables 2926.0 KB 2269.7 KB 2023.6 KB -10.8% -30.8% entities 660.2 KB 163.8 KB 151.0 KB -7.8% -77.1% Free bytes: same fields, same code, different order. 4. Pointer compression for everything (07ec80f) We compress pointer for all objects allocated in the arena, like we compressed a pointer to the string. ArenaVec<Node*> children held 8-byte addresses; ArenaPtr<T> is a 4-byte offset into the arena’s position space, resolved by ArenaAtOffset. Zero is null, which costs nothing because no allocation ever lands at offset zero. The Node itself doesn’t change size — a vector handle is the same three words whatever it holds — so all of the saving is in the child arrays. shape start before after vs before vs start prose 1646.1 KB 1150.9 KB 1091.9 KB -5.1% -33.7% nested lists 1067.9 KB 654.0 KB 611.6 KB -6.5% -42.7% gfm tables 2926.0 KB 2023.6 KB 1866.9 KB -7.7% -36.2% entities 660.2 KB 151.0 KB 144.9 KB -4.0% -78.1% These shapes rank by children-per-node rather than by node count, which is why tables moved most. 5. Varint encoding string length (f9ebc34) ArenaStr was an offset and a length in 8 bytes. Now it’s the offset alone — 4 bytes — and the length is varint-encoded at the beginning of the string data: [varint len][string bytes][NUL] There are many varint encoding schemes. This one is for unsigned number and codes number < 128 as a single byte. Most strings are below that threshold, so they use a single byte for the varint length, saving roughly 3 bytes per string. Node shrinks from 144 → 112 bytes. Str — pointer + length, 16 bytes per field char* s int64 len ArenaStr — offset + length, 8 bytes u32 off u32 len ArenaStr — offset alone, 4 bytes; the length lives in the arena u32 off len characters 0 Caveat: An offset-and-length string can point at a slice of another string, and a length-prefixed one can’t. We weren’t doing it so it doesn’t apply here. shape start before after vs before vs start prose 1646.1 KB 1091.9 KB 918.0 KB -15.9% -44.2% nested lists 1067.9 KB 611.6 KB 512.3 KB -16.2% -52.0% gfm tables 2926.0 KB 1866.9 KB 1543.6 KB -17.3% -47.2% entities 660.2 KB 144.9 KB 128.5 KB -11.3% -80.5% 6. Fusing exclusive fields (5467a18) A List has a start number. A Heading has a depth. No node is ever both, so they became one uint32_t startOrDepth and kind says which it means. It didn’t shrink the size of Node due to the alignment padding but we did it anyway hoping that future optimization would shrink below padding. shape start before after vs before vs start prose 1646.1 KB 918.0 KB 918.0 KB +0.0% -44.2% nested lists 1067.9 KB 512.3 KB 512.3 KB +0.0% -52.0% gfm tables 2926.0 KB 1543.6 KB 1543.6 KB +0.0% -47.2% entities 660.2 KB 128.5 KB 128.5 KB +0.0% -80.5% 7. Compressing text position (ed5e807) Each Node carried the info about its position in parsed text. It was expensive because it was stored as start and end fields and each of them was: a uint32_t line a uint32_t column a uint32_t offset That’s 4*3*2 = 24 bytes. I assume this info is for debugging so not important for me. I replaced it with 2 uint32_t offsets into a source markdown string, srcStart and srcEnd. We can reconstruct the line/column position from that and the source string. shape start before after vs before vs start prose 1646.1 KB 918.0 KB 828.0 KB -9.8% -49.7% nested lists 1067.9 KB 512.3 KB 462.2 KB -9.8% -56.7% gfm tables 2926.0 KB 1543.6 KB 1379.6 KB -10.6% -52.9% entities 660.2 KB 128.5 KB 120.0 KB -6.6% -81.8% 8. Further compression text position (6a558c4) srcEnd is always after srcStart so we can delta-encode it and shrink to uint16_t. What if it’s bigger than 64 KB? I don’t care, we store it as 65535. This is another case where due to padding we didn’t shrink the struct size. But wait for it. shape start before after vs before vs start prose 1646.1 KB 828.0 KB 828.0 KB +0.0% -49.7% nested lists 1067.9 KB 462.2 KB 462.2 KB +0.0% -56.7% gfm tables 2926.0 KB 1379.6 KB 1379.6 KB +0.0% -52.9% entities 660.2 KB 120.0 KB 120.0 KB +0.0% -81.8% 9. Optimizing storing children (d6c4abc) Some nodes have children that were stored as a growable vector. Empty vector was 24 bytes in the node. We replaced it with a ring of compressed pointers: the parent names its last child, each child names the next one, and the last child wraps back to the first. vector: 24 bytes in the node + a separate array of links ptr · len · cap kid0 kid1 kid2 spare spare ring: 4 bytes in the parent, 4 in each child, nothing else allocated parent kid0 kid1 kid2 lastKid We use a ring and not just a linked list because appending is the only thing the parser does to a child list. A single linked list requires walking the list to find the end, while a ring does not. Saving: 96 → 80 bytes. shape start before after vs before vs start prose 1646.1 KB 828.0 KB 619.0 KB -25.2% -62.4% nested lists 1067.9 KB 462.2 KB 308.8 KB -33.2% -71.1% gfm tables 2926.0 KB 1379.6 KB 898.7 KB -34.9% -69.3% entities 660.2 KB 120.0 KB 98.9 KB -17.6% -85.0% Caveat: accessing a child by index would require a walk through the ring, so indexing in a loop would be quadratic. In our code we only ask for the first or the last. 10. Compressing table alignments info (ca0818c) For tables we were storing column alignments in a separate vector on every node, even though only Table nodes have them. Another 24 bytes per node. We switched to a compressed pointer which points to an optimized representation of the column alignments. There are four alignments (left, right, center, none), so a column needs 2 bits: [varint count][2 bits a column, four to a byte] The whole list is known when the table is entered, so it’s counted, allocated once and filled. For an 8-column table that’s 3 bytes in the arena and a 4-byte offset in the node. Saving: 80 → 60 bytes. We saved more than the 20 bytes because with the last pointer-holding member gone alignof(Node) fell from 8 to 4. shape start before after vs before vs start prose 1646.1 KB 619.0 KB 519.3 KB -16.1% -68.5% nested lists 1067.9 KB 308.8 KB 256.3 KB -17.0% -76.0% gfm tables 2926.0 KB 898.7 KB 710.3 KB -21.0% -75.7% entities 660.2 KB 98.9 KB 89.2 KB -9.8% -86.5% The block is pushed byte-aligned rather than through the general allocator, which rounds to 8 and would have handed back exactly what the varint saved. 11. Fusing exclusive fields (07444d6) Previously we fused exclusive fields start of a List node and depth of a Heading node into a single uint32_t. We fused Table node column alignments info from previous optimization into the same field. We called it uint32_t perKind, and kind says what kind of value it is. Saving: 60 → 56 bytes. shape start before after vs before vs start prose 1646.1 KB 519.3 KB 483.9 KB -6.8% -70.6% nested lists 1067.9 KB 256.3 KB 233.7 KB -8.8% -78.1% gfm tables 2926.0 KB 710.3 KB 646.1 KB -9.0% -77.9% entities 660.2 KB 89.2 KB 86.1 KB -3.5% -87.0% 12. Optimizing eight strings (521e32e) We had 8 strings that were not all used by all nodes. Instead of figuring out how many strings we need at most, I created a linked list of strings in the arena. They are different than regular strings in that they carry a 4 byte compressed pointer to the next string within the arena and the kind of the strings. [u32 next][u8 kind][varint len][len bytes][NUL] We can add as many kinds of strings as we need but we only pay for used strings + 5 byte per-string overhead. Some nodes don’t have any strings. 8 fields: 32 bytes on every node, 7 of them empty on almost all of them value url title alt ident label lang meta 1 field: 4 bytes, and a record only for what the node actually carries first next kind len characters 0 a stored string costs 5 bytes more · a node storing none saves 28 New records go on the head, so storing is O(1), and the walk that finds a kind is at most 8 long and is almost always 1 or 0. In-place growth still works, because a record being the newest thing in the arena is the same condition it always was. Saving: 56 → 28 bytes. shape start before after vs before vs start prose 1646.1 KB 483.9 KB 358.3 KB -26.0% -78.2% nested lists 1067.9 KB 233.7 KB 159.1 KB -31.9% -85.1% gfm tables 2926.0 KB 646.1 KB 402.9 KB -37.6% -86.2% entities 660.2 KB 86.1 KB 73.7 KB -14.4% -88.8% 13. Fusing two enums into one (f3c14b9) As it happened we had two enums: one needed 6 bits another needed 2 bits We fused them from 2 bytes to 1 byte. Because this 1 byte saving dropped below padding we saved 4 bytes and went from 28 → 24 bytes. shape start before after vs before vs start prose 1646.1 KB 358.3 KB 321.2 KB -10.4% -80.5% nested lists 1067.9 KB 159.1 KB 136.5 KB -14.2% -87.2% gfm tables 2926.0 KB 402.9 KB 341.9 KB -15.1% -88.3% entities 660.2 KB 73.7 KB 70.4 KB -4.5% -89.3% 14. Remove position, reduce allocator’s alignment (861c803) At this point I decided that I didn’t need the position so I removed it. Other markdown parsers don’t carry it around so it doesn’t seem very useful. I reduced overhead of perKind by converting it to a record in the string list from step 12 — varint-encoded, under its own kind byte. A List, Heading or Table pays ~8 bytes for it; every other node pays nothing, where a field cost 4 bytes on all of them. Savings: 24 → 16 bytes. For safety arena allocator aligns allocations to 8 bytes but a 16 bytes Node can be allocated at 4 bytes, which we did. This reduces wasted space between allocations. shape start before after vs before vs start prose 1646.1 KB 321.2 KB 272.0 KB -15.3% -83.5% nested lists 1067.9 KB 136.5 KB 110.2 KB -19.3% -89.7% gfm tables 2926.0 KB 341.9 KB 250.5 KB -26.7% -91.4% entities 660.2 KB 70.4 KB 65.6 KB -6.8% -90.1% End results The results are pretty dramatic: sizeof(Node) prose nested tables entities start 232 1646.1 KB 1067.9 KB 2926.0 KB 660.2 KB end 16 272.0 KB 110.2 KB 250.5 KB 65.6 KB -93% -83.5% -89.7% -91.4% -90.1% A parse of 64 KB of prose cost 25.7× the source in arena bytes. It costs 4.2× now. The entities shape went from 10.3× to 1.02×. The speed was unchanged. Fastest of 3 runs: prose 8.47 → 8.22 ms nested 9.45 → 9.26 ms tables 12.88 → 12.92 ms entities 5.90 → 5.85 ms Those are within margin of error. The phase of building the tree got a measurable speed up: 0.397 → 0.302 ms, about 24% faster. This is from allocating less and touching fewer cache lines. This is not visible on micro benchmarks, but using less memory will slightly speed up the rest of the application. Lessons learned Arranging struct fields by size is good. It costs literally nothing. Pointer compression is good. 8 bytes become 4 bytes and the cost of converting back and forth is negligible, as Google shown in their v8 blog post and is re-inforced by our benchmarks Varint-encoding is good. Most strings are short so varint encoding can save 3 bytes per string on average. Moving rare fields out of line is good. The way we reduced 8 strings into an out-of-line list. Only pays off if savings is bigger than the cost of additional metadata. sizeof only drops when the saving crosses an alignment boundary. Two of our changes didn’t reduce size of Node struct but it paid off in later optimizations. The allocator’s alignment is part of sizeof. A 28-byte struct from an 8-aligned bump allocator is 32 bytes. We need benchmarks. You can’t improve what you can’t measure. Our benchmarks measured both memory usage and speed, to ensure we didn’t regress speed to save memory.
I maintain SumatraPDF on GitHub. People fork it and make their own changes. I want to know which forks are active and what they’re working on. GitHub has a Network tab for this. I find it lousy. It’s hard to see active forks at a glance. Panning and zooming the UI is slow and fiddly. I just want a sorted list of forks with actual changes. So I wrote a small script: github-active-forks.ts. What it does It uses the GitHub API to find forks that have ahead commits (changes not in upstream) from the last year. For each active branch it prints: a link to the branch on GitHub short sha, author, relative date, first line of commit message Forks are sorted by most recent activity, oldest first. Commits within a branch are oldest first. It compares each fork branch against the matching upstream branch (e.g. rel3.6working vs rel3.6working), not always master. That way you only see commits unique to the fork. How to run it Download the script: curl -O https://gist.githubusercontent.com/kjk/71a7679dd408d52bc612cdf5eecace58/raw/7d7f605dde6b739424e5190a79e3988477312e74/github-active-forks.ts You need Bun and the GitHub CLI logged in (gh auth login). Or set GITHUB_TOKEN. Run it on any repo as owner/repo: bun github-active-forks.ts sumatrapdfreader/sumatrapdf Progress goes to stderr; results go to stdout, so you can redirect to a file: bun github-active-forks.ts sumatrapdfreader/sumatrapdf > forks.txt Example output Excerpt from running on SumatraPDF (74 forks with ahead commits): https://github.com/wackget/sumatrapdf-Feldherren-version/tree/single-page-fit-scrollbar (289d ago) c53a51d wackget 294d ago Added scrollbar usable in Fit a Single Page display mode. f927640 wackget 294d ago scrollbar in single page mode now obeys the hideScrollbars setting. 7f57861 wackget 288d ago improved scrolling speed when zoomed in, in non-continuous single-page view. https://github.com/xyzzyx99/sumatrapdf/tree/rel3.6working (5d ago) c8d1f7b xyzzyx99 13d ago Add GitHub Actions workflow for building project 8a92547 xyzzyx99 13d ago Fix bugs, revert using version 3.6 1f27324 xyzzyx99 13d ago Save and restore CHM scroll position 24e7974 xyzzyx99 6d ago Preserve current CHM URL across tab restore 30b1cb4 xyzzyx99 5d ago Update README with new bug fixes and formatting https://github.com/lsq/sumatrapdf/tree/dev (today) 72d7069 lsq 11d ago feat: support share file via socket a1fc427 lsq 10d ago feat: open home page 938bc07 lsq today add localsend v2.1 api support Much easier to scan than the Network graph.
A new JavaScript library pretext for fast text measuring / layout popped up on social media. Potentially interesting given its focus on speeding up text rendering in web apps and me writing web apps and liking them being fast. I looked at the code and saw a function isCJK(). Given my 3 decades of programming and performance optimization, it looked like it could be sped up. This is a story about ideas on making JavaScript faster and the process of quickly implementing and benchmarking them. The code export function isCJK(s: string): boolean { for (const ch of s) { const c = ch.codePointAt(0)! if ((c >= 0x4E00 && c <= 0x9FFF) || (c >= 0x3400 && c <= 0x4DBF) || (c >= 0x20000 && c <= 0x2A6DF) || (c >= 0x2A700 && c <= 0x2B73F) || (c >= 0x2B740 && c <= 0x2B81F) || (c >= 0x2B820 && c <= 0x2CEAF) || (c >= 0x2CEB0 && c <= 0x2EBEF) || (c >= 0x30000 && c <= 0x3134F) || (c >= 0xF900 && c <= 0xFAFF) || (c >= 0x2F800 && c <= 0x2FA1F) || (c >= 0x3000 && c <= 0x303F) || (c >= 0x3040 && c <= 0x309F) || (c >= 0x30A0 && c <= 0x30FF) || (c >= 0xAC00 && c <= 0xD7AF) || (c >= 0xFF00 && c <= 0xFFEF)) { return true } } return false } My spider sense tingling To make code run fast you have to have mechanical sympathy. You need a good mental model of how CPUs and programming languages work, at the low level. Because I have mechanical sympathy, I know that to evaluate multiple || statements, the program has to check every statement until it finds one that is true. For the case of not matching any range, it has to do all 15 comparisons. My immediate thought was that most characters are ascii (non-cjk) and therefore require 15 comparisons. An early exit should speed things up: if (c < 0x300) continue;. It adds 1 comparison for matching cases but removes 14 (15 - 1) for the most common case of ascii characters. Hunch is good, benchmark is better Micro-benchmarking is subtle. In the past I would google for a JavaScript benchmarking library and trust that it’s sound. Claude wrote a sound, simple benchmark function: function bench(name, fn, iterations = 1_000_000) { for (let i = 0; i < 10_000; i++) fn(); // warmup const start = performance.now(); for (let i = 0; i < iterations; i++) fn(); const elapsed = performance.now() - start; const opsPerSec = ((iterations / elapsed) * 1000).toFixed(0); console.log(`${name}: ${elapsed.toFixed(2)} ms (${opsPerSec} ops/sec)`); } // to call: bench("isCJKRegex ", () => isCJKRegex(input)); The subtle part is warmup phase. JavaScript JIT engines that you’re likely to use (v8 in Chrome, JavaScriptCore in Safari, SpiderMonkey in FireFox) have several optimization tiers and only do most aggressive optimization for frequently called functions. The warmup phase forces JIT to compile fn with most aggressive settings. This is only fair - I assume isCJK is called frequently. If it wasn’t, there’s no point sweating to optimize it. The new AI normal In the past I would have to manually implement each optimization idea. With AI I can delegate the coding and therefore test more ideas. The following was done with Claude 4.6. 100% vibe coding There are levels of delegating things to AI. On the extreme side of giving up control is 100% vibe coding. I had 2 ideas for performance optimization: early exit replace 15 linear checks with binary search But what if I didn’t? What if I just ask AI to come up with ideas? The following is my conversation with Claude 4.6 and what I’ve learned. I created is_cjk.js file and pasted the original function. look at is_cjk.js. come up with 2 ideas for how to speed up isCJK(). Implement those as separate functions and write a javascript micro-benchmark that compares those 2 new implementations against isCJK Two ideas: Pre-compiled regex — push the entire check into the native regex engine, avoiding JS-level for…of iteration and codePointAt() overhead. Typed array bitmap for BMP — for BMP characters (the vast majority of real text), a Uint8Array bitmap gives O(1) lookup with a single array access. Fall back to range checks only for supplementary plane characters. I have to admit: I recognize those as valid optimization ideas but they didn’t immediately occur to me. Regex works because regex implementations in JavaScript engines are insanely optimized. Bitmap is a known optimization for doing true/false lookup for numbers. Let’s say you have total range 0-255 and you want a function that checks if a number is in range 1-10, 20-30, 40-50, 60-70, 80-90. That’s 5 if statements, each doing 2 comparisons. You can instead use 1 bit for each of the 256 numbers (i.e. 256⁄8 = 32 bytes), pre-compute the array of bytes and set the bit for the ranges. Then lookup is just math to get byte offset and bit, load the byte, check the bit. Super fast. Claude not only wrote the benchmark function, 2 new implementations but also ran the benchmarks using node and showed me results in a readable way. The bitmap approach was significantly faster. Human assisted coding Bitmap was the fastest and yet I discarded it using my human judgement. The bitmap needed 65kB of memory and that’s just too much for this function. That’s where human judgement comes in to improve collaboration with AI. I didn’t specify that I also care about memory usage so AI wrote fast code that was using too much memory. You’re benchmarking what? Here’s another tricky part of benchmarking: knowing what you benchmark. There are 3 popular JavaScript engines. Claude ran benchmarks using node which uses Chrome’s v8. That’s good because that’s the most popular browser and therefore most popular JavaScript JIT engine. It’s also good to sanity-check with at least one other JavaScript engine. bun uses Safari’s JavaScriptCore so I asked Claude: run the tests using bun The results were similar, which is good. We don’t want code that is fast in one engine but slow in another. We don’t control which browser the user of our code is running. Continuing collaboration with AI I had some more ideas so here’s the rest of my conversation with Claude. don’t benchmark bitmap, it uses too much memory; write a variant of isCJK that does an early false exit if char is less than smallest Early exit is my first insight I wanted to test. As expected, early exit is good 2x speedup for latin strings, although regex beats it on cjk strings. That is the curse of engineering trade-offs: you have to decide if you want to optimize for cjk strings or non-cjk strings. now implement a variant that does inline bisect / binary search That was my second idea: do a binary search instead of sequential if comparisons. It was faster than original but slower than regex / early exit. Also, Claude didn’t do what I meant. It stored the ranges as array: // Flat array: [lo0, hi0, lo1, hi1, ...] sorted by lo const cjkRanges = new Uint32Array([ 0x3000, 0x303f, 0x3040, 0x309f, ... That’s what I meant: implement a variant isCJBisect that doesn’t use array but unrolls binary search logic into if statements The implementation is gnarly: function inCJKUnrolled(c) { if (c < 0x3000) return false; if (c < 0xff00) { if (c < 0x3400) { if (c < 0x3040) { return c <= 0x303f; } else if (c > 0x309f) { return c >= 0x30a0 && c <= 0x30ff; } return true; // ... more gnarly code But what do I care? Claude wrote it. Arguably it’s hard for me to review for correctness but it’s perfectly testable. We can write a test to compare if it returns the same result as original function for every possible input. As expected, unrolled version was faster than array-based but not always faster than regex or early exit. But Claude wrote something that caught my eye: The for…of + codePointAt iterator overhead remains the bottleneck shared by all non-regex variants on long strings. It’s well known that for ... of loop is slower than for (let i = 0; i < n; i++). It’s one of those things that you need to know. I knew this but I didn’t connect the dots in this particular case. write variant of isCJKEarly which replaces for of with a faster loop Result: isCJKEarlyFast crushes everything across the board on both runtimes The code for isCKJEarlyFast. Looks like we have a winner: Replacing for…of + codePointAt() with charCodeAt() + index loop eliminates the iterator overhead, which was the main bottleneck. The early c < 0x3000 bail then skips all range checks for ASCII/latin characters cheaply. Only regex still wins on Node’s long latin miss case thanks to V8’s native string scanning. There’s a subtlety that makes the rewrite of for .. of into for (let i = 0; i < n; i++) tricky. Logically JavaScript strings are composed from 16-bit numbers (characters). Some Unicode code-points are bigger than 16-bit and are stored as 2 16-bit numbers, called surrogate pairs. So writing a loop over unicode codepoints needs to take this into account. It’s tricky code but again, what do I care? Claude wrote it and it’s perfectly testable. Pressing on The good thing about claude writing most of the code and removing other tedium like running the benchmarks is that we can do more in the same amount of time. So I explored even more. I noticed: export function isCJKUnrolled(s) { for (const ch of s) { if (inCJKUnrolled(ch.codePointAt(0))) return true; } return false; } What happens when inCJKUnrolled is inlined? inline inCJKUnrolled into isCJKUnrolled Result: Done. Inlining didn’t help — it actually got slightly slower in most cases (the engine was already inlining the small function). No change in rankings; EarlyFast remains the winner. Turns out it’s not faster, which was surprising. Inlining is supposed to remove overhead of function calls, which matters in loops. Surprising isn’t bad. We run benchmarks to validate our hunches. For completness, I wanted to compare with original where for .. of was rewritten. also write original with for of replaced by standard for loop and bench all and give results as markdown table Claude was nice enough to show benchmark results in a nicely formatted table, without me even asking. It just knows things. But as you can see above, you can ask it for results in markdown table to e.g. include in a GitHub bug report. Here are final results. The number is operations/second in millions. Higher is better. Bun Scenario Original Regex Early Bisect Unrolled EarlyFast ForLoop Single CJK 11M 22M 12M 12M 16M 79M 57M Single latin 20M 24M 49M 21M 22M 40M 36M CJK string 12M 18M 17M 12M 14M 42M 42M Latin string 1.5M 4.7M 3.6M 3.0M 3.5M 17M 6.5M Mixed string 6.5M 5.9M 9.9M 8.6M 13M 62M 34M Node Scenario Original Regex Early Bisect Unrolled EarlyFast ForLoop Single CJK 85M 54M 51M 44M 56M 112M 110M Single latin 59M 72M 83M 55M 74M 115M 103M CJK string 79M 55M 60M 55M 64M 90M 129M Latin string 4.2M 57M 4.3M 2.8M 4.5M 12M 8.3M Mixed string 14M 15M 16M 10M 15M 41M 38M Conclusions AI is a big unlock. It took me under 30 minutes to test various hypotheses and find out a significant speed up. Without Claude it would take several hours and I would likely not do it at all. It’s just not important enough to spend a working day on it. To get best results we still need to apply human judgement and guide the AI. Programming expert knowledge An expert is simply someone who knows things. We know things because we learn them. If you were paying attention you might have learned the following things: importance of warmup phase when benchmarking JIT compilers for .. of is slower than for (let i = 0; i < n; i++) regex matching in JavaScript engines is fast subtlety of surrogate pairs in JavaScript strings using bitmaps to speed up range lookups Resources All the code is in https://gist.github.com/kjk/bdbea9d90c3bb0454fbe26353c521bfd I like to write fast code. If you want a fast bookmark manager / note taker, try MarkLexis. If you want a fast PDF / ebook / comic book reader for Windows, try SumatraPDF.
So you’ve installed Visual Studio and you want to run the compiler cl.exe from command-line. Microsoft makes it surprisingly hard. They give you a shortcut which opens a terminal window with cmd.exe setup for compilation. But I don’t want a separate window, I want to use the terminal app. You can run cmd.exe /k "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x64 (location for your setup can be different). But I don’t want to run inside cmd.exe. I want to use powershell. What exactly does vcvarsall.bat do? Not much: it just sets some env variables and updates PATH. We can reverse-engineer what it does: cmd.exe set >before.txt cmd.exe /k "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x64 set >after.txt Now compare before.txt and after.txt to see what changed. I asked AI to do it for me and here’s the beginning of what I found: CommandPromptType=Native DevEnvDir=C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\ ExtensionSdkDir=C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs EXTERNAL_INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um Framework40Version=v4.0 FrameworkDir=C:\Windows\Microsoft.NET\Framework64\ FrameworkDir64=C:\Windows\Microsoft.NET\Framework64\ ... more stuff I saved that to diff.txt file. Now that we have that we can ask our friendly AI to convert this to a powershell function: write powershell function that sets variables as seen in @diff.txt I get something like this: function set-msvc { <# .SYNOPSIS Sets Visual Studio Developer Command Prompt environment variables. .DESCRIPTION This function sets all the environment variables needed for Visual Studio 2022 native x64 development, equivalent to running vcvars64.bat. .EXAMPLE Set-VsDevEnv #> [CmdletBinding()] param() $env:CommandPromptType = "Native" $env:DevEnvDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\" ... more stuff We need to change one thing: $env::PATH from absolute to just adding paths to current $env::PATH so that it’ll adapt to future changes in PATH. Now add that to $PROFILE and you can type set-msvc to setup powershell to have access to cl.exe, msbuild and other tools. Below is the whole function but note that this is only for amd64 toolset. For 32-bit or arm setup would be different. You can get by re-running the above logic for vcvarsall.bat with different arguments. function set-msvc { <# .SYNOPSIS Sets Visual Studio Developer Command Prompt environment variables. .DESCRIPTION This function sets all the environment variables needed for Visual Studio 2022 native x64 development, equivalent to running vcvars64.bat. .EXAMPLE Set-VsDevEnv #> [CmdletBinding()] param() $env:CommandPromptType = "Native" $env:DevEnvDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\" $env:ExtensionSdkDir = "C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs" $env:EXTERNAL_INCLUDE = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" $env:Framework40Version = "v4.0" $env:FrameworkDir = "C:\Windows\Microsoft.NET\Framework64\" $env:FrameworkDir64 = "C:\Windows\Microsoft.NET\Framework64\" $env:FrameworkVersion = "v4.0.30319" $env:FrameworkVersion64 = "v4.0.30319" $env:FSHARPINSTALLDIR = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools" $env:HTMLHelpDir = "C:\Program Files (x86)\HTML Help Workshop" $env:INCLUDE = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" $env:is_x64_arch = "true" $env:LIB = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x64" $env:LIBPATH = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\lib\x86\store\references;C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.26100.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.26100.0;C:\Windows\Microsoft.NET\Framework64\v4.0.30319" $env:NETFXSDKDir = "C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\" $env:Path = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin;C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;" + $env::Path + ";C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg" $env:Platform = "x64" $env:UCRTVersion = "10.0.26100.0" $env:UniversalCRTSdkDir = "C:\Program Files (x86)\Windows Kits\10\" $env:VCIDEInstallDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\" $env:VCINSTALLDIR = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\" $env:VCPKG_ROOT = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg" $env:VCToolsInstallDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\" $env:VCToolsRedistDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Redist\MSVC\14.44.35112\" $env:VCToolsVersion = "14.44.35207" $env:VisualStudioVersion = "17.0" $env:VS170COMNTOOLS = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\" $env:VSCMD_ARG_app_plat = "Desktop" $env:VSCMD_ARG_HOST_ARCH = "x64" $env:VSCMD_ARG_TGT_ARCH = "x64" $env:VSCMD_VER = "17.14.24" $env:VSINSTALLDIR = "C:\Program Files\Microsoft Visual Studio\2022\Community\" $env:WindowsLibPath = "C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.26100.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.26100.0" $env:WindowsSdkBinPath = "C:\Program Files (x86)\Windows Kits\10\bin\" $env:WindowsSdkDir = "C:\Program Files (x86)\Windows Kits\10\" $env:WindowsSDKLibVersion = "10.0.26100.0\" $env:WindowsSdkVerBinPath = "C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\" $env:WindowsSDKVersion = "10.0.26100.0\" $env:WindowsSDK_ExecutablePath_x64 = "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\" $env:WindowsSDK_ExecutablePath_x86 = "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\" $env:__DOTNET_ADD_64BIT = "1" $env:__DOTNET_PREFERRED_BITNESS = "64" $env:__VSCMD_PREINIT_PATH = "C:\Program Files\PowerShell\7;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Microsoft VS Code\bin;c:\Users\kjk\AppData\Local\Programs\cursor\resources\app\bin;C:\Program Files\gs\gs10.03.1\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\GitHub CLI\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Rust stable MSVC 1.88\bin;C:\Program Files\PowerShell\7\;C:\Program Files\nodejs\;C:\Program Files\dotnet\;C:\Program Files\CMake\bin;C:\Program Files\Go\bin;C:\Program Files\RedHat\Podman\;C:\Program Files\Tailscale\;C:\Program Files\Git\cmd;C:\Program Files\Docker\Docker\resources\bin;C:\Users\kjk\AppData\Local\Microsoft\WindowsApps;C:\Users\kjk\AppData\Local\Microsoft\WinGet\Links;C:\Users\kjk\.bun\bin;C:\Users\kjk\.dotnet\tools;C:\Users\kjk\go\bin;C:\Users\kjk\AppData\Local\Programs\superfile\;C:\Users\kjk\AppData\Local\Microsoft\WindowsApps;C:\Users\kjk\AppData\Local\GitHubDesktop\bin;C:\Users\kjk\AppData\Local\Programs\cursor\resources\app\bin;C:\Users\kjk\AppData\Roaming\npm;C:\Users\kjk\.dotnet\tools;C:\Users\kjk\AppData\Local\Programs\Antigravity\bin;C:\Users\kjk\AppData\Local\Programs\Zed\bin;C:\Users\kjk\go\bin;C:\Users\kjk\OneDrive\bin;C:\Users\kjk\.bin\jai\bin;C:\Users\kjk\.bin\mupdf-1.27.0;C:\Users\kjk\.local\bin;C:\Users\kjk\AppData\Local\Programs\WinMerge;C:\Users\kjk\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\Scripts;C:\Users\kjk\OneDrive\bin\sublime_text" Write-Host "Visual Studio 2022 Developer Environment configured for x64." -ForegroundColor Green }
More in programming
Download Now View on pouet.net Intro In the late 80s, I spent an embarrassing number of hours hunched over my ZX Spectrum, typing in Sinclair BASIC programs and rudimentary assembler routines copied from magazines, trying to re-create the Amiga crack intros I’d seen round at a friend’s house. Obviously, they never came even slightly close - but now, many decades later, I’ve finally managed it! I’m now the proud author of a hand-rolled retro demoscene intro that runs on a stock 128Kb Spectrum. It features all those late 80s tropes: A smooth scrolltext (with ‘leet custom fonts!), a three-channel AY soundtrack, bouncing VU meters and the essential parallax starfield. It’s currently 10287 bytes of code, or 19552 bytes as a complete tape image - which, given I’ve been working on this since December 13th last year, means my average rate of code-production is a paltry ~ 455 bytes per week! This blog post is a technical overview of how I got there, some cool tricks I picked up, and everything I had to learn (and re-learn) about the Speccy and Z80 assembly along the way. It simultaneously rates as one of my favourite side-projects ever, and also the one that most often nearly triggered a foot put through my monitor in frustration. I’ll reflect on all that at the end, but for now if you just want to see the finished product, the download button is at the top of the article and the included .tap cassette tape image should load into any emulator. There’s also a YouTube capture, links to the production on pouet.net, and you can run it from within a web browser by visiting my TNFS site portal - check option 4 for recent uploads. Be Kind, Rewind But let’s skip back briefly to the late 80s. Coming from the world of an 8-bit, cassette tape loading computer, seeing an Amiga for the first time was like encountering something from outer space. Like everyone who saw one back in the day, I was blown away by the amazing graphics and sound on display, not to mention the library of ground-breaking games. But I found myself even more captivated by the mysterious (and totally badass) “intros” displayed at the beginning of pirated games. Created by the software piracy groups that had cracked the copy protection and distributed the floppy-disk based games, they were mini works of art in their own right - music, graphics and usually some cool-looking effect to show off the group’s prowess. There’s a great collection of these intros (or “cracktros”) archived on YouTube. Watch a few of those videos and you’ll quickly notice the “design” tropes that almost every cracktro shares: a cool, edgy logo at the top of the screen (bonus points if it’s dripping blood or in any way “metal”), some boastful text displayed on top of an effect of some sort - usually a scrolling parallax starfield, or rotating 3d object - and a further scroll-text full of shout outs, greetings to fellow pseudonymous crackers and adverts for illegal BBS systems galore. Needless to say, to a kid in the 80s/early 90s this was possibly the coolest thing ever. I thought this one was the absolute bomb when I first saw it. Sine-scroller, pirate BBS adverts, cool logo with a sword - and of course, a starfield. There’s something for everyone here. The music from this is burned into my brain. I think pretty much everyone of my classmates knew this intro given the popularity of Lemmings in the early 90s. Skid Row as a group are still active to this day and this is probably the quintessential Amiga cracktro of the period. Endless Piracy. Edgy. Another cool logo, sine-wave scroller, starfield and boasting text. The classics never go out of style. Another great combination of all the essential ingredients. Cool logo, even cooler text font, scrolling starfield and a scroll-text running along the bottom. Vintage stuff. Logo. Text. Scroller. But no starfield - otherwise a solid effort from Anthrox here :) The demoscene grew out of these intros. The cracking eventually faded into the background, and coders, “graphicians” and musicians all contributed to feature-length productions that showcased their abilities, competed with other groups and pushed the machine to its limits. In amazing feats of ingenuity and pure “code as an art form”, the scene lives on today, with groups producing breathtaking productions like this and this - bear in mind, both of those are running on a 7Mhz home computer equipped with a floppy disk, 1Mb of memory and a sound chip from 1985. But much as I admire the sheer skill and dedication that takes, it’s the early “single screen” scrolly-text-and-starfield cracktros I’ve always found captivating. They took on a sort of mystical quality for me, hinting at a scene and a culture I could only ever experience from the outside - which is exactly why I spent all those hours in Sinclair BASIC, trying in vain to re-create them, and dreaming of the day I’d eventually have an Amiga of my own. The Jump Back I eventually did trade up to a range of Amiga systems, joined a few “scene” groups myself and threw myself into a variety of activities until exams and the end of my teenage years beckoned. I dabbled with Z80 code back then, and at one point knew enough 68k assembly to bluff my way through - partly from the Amiga’s heyday, partly from a few “embedded systems” modules at University where I had to program a set of traffic lights or lift-controllers. Given there’s not much call for that sort of thing at my $DAYJOB, though, it had all pretty much faded from my memory. I have kept working on a bunch of projects in my spare time, and one of my favourites is my TNFS site - sort of an old-school BBS system or interactive “website” for the Spectrum, which grew out of a fun experiment to see how far I could push Sinclair BASIC. It relies on a modern network card fitted to the Spectrum, which is how I came to be chatting to “Desertkun”, another TNFS-protocol enthusiast. He’s now producing a great new WiFi network card for the Speccy, and was kind enough to send me a few of the prototypes to test out for him. I was massively inspired by his hardware and low-level coding chops, and during one of our video calls where we’d been comparing our development setups, he asked why I didn’t write any Z80 assembly myself. A throw-away comment, but the challenge was set! Now, I couldn’t face re-writing the whole site in asm, but I figured I could produce a self-contained intro to promote it in the style of the early “BBStros”, and finally be one of the cool kids with a scene production of my own :) I mean, how hard could it be ? (ooooh - foreshadowing!) Dev Environment My development environment through this was pretty straightforward, and is the same stack I use for developing my TNFS site. I just used a Makefile-driven build system which ran my code through the Pasmo assembler, my fork of zmakebas to build BASIC files, and the FUSE emulator for testing. For final testing on the real hardware, I uploaded my .tap files to my local TNFS server where I could run them on my +3. A few additional tools I have to call out because they ended up making a big difference - one is the amazing web-based “asm-editor” I discovered while looking up Z80 instructions. Not only does this site have a great reference of all the Z80 instructions, but it also has an editor where you can run example code and see how the memory and/or various registers change state. Take for example, the djnz instruction - you can build it and step through the example loop construct one instruction at a time. Or try my example of an add instruction in the screenshot to see how the carry flag works when a bit gets shifted out of a register. Although it uses a slightly different syntax to Pasmo (so I couldn’t drop my code directly in it), it massively helped when manually stepping through things. And lastly, probably the most significant debugging tools I used on this project - a notepad and pen, and my two dogs (say “Hi!” to Biscuit & Pebble…) I’ll write some more about this in the wrap-up, but this really was a highly unusual project for me, coming from decades of much higher-level languages and constructs. Most of my z80 code I only ever really understood after drawing out the bytes and bitmasks, keeping a table of register flags and drawing chunks of screen memory out by hand. And I found my evening walks with the two dogs provided ideal rubber duck exercises, although I’m not sure what they both made of my protracted mutterings about the hl register pair. Either way, debugging-by-dog is highly recommended! The Starting Point So, let’s get started. I followed a great “Hello, world!” example from futurice.com to validate my build environment. Beyond that, the first thing I wanted to do was load in my logo and background image I’d use as a “stage” for the remainder of the planned intro. I wasn’t really sure what I was going to do at this point, just that I wanted my logo at the top, and a scrolltext along the bottom row of the screen. As my TNFS site header already looked kinda demoscene-y, I made a quick edit putting in place two blue guide-bars inside which I’d put further content. Memory map courtesy of breakintoprogram.co.uk A Spectrum SCREEN$ file (usually displayed as loading screens) is just a dump of screen memory, starting from byte 16384, and running for 6912 bytes. The weird layout of this is very relevant later, but in order to display it all you need to do is load your file into that memory location. After a few hour’s work of reviewing Speccy internals and following various guides, I had a very simple starting point for my demo that served as the basis for everything else that came after. org $8000 screen EQU $4000 ; address of the first byte of screen memory, 16384 in dec scrlen EQU 6912 ; total length of the screen memory inc. attributes ula EQU $FE ; I/O port for the ULA to control the border black EQU 0 ; Spectrum colour number, the exact same ones BASIC uses start: ld a,black ; Set the border colour to black using the ULA out (ula),a ; Copy my logo header picture to screen memory ld hl,header ; HL = location of my screen file in this code ld de,screen ; DE = start of screen memory ld bc,scrlen ; BC = 6912 bytes to copy ldir ; block copy, see https://jnz.dk/z80/ldir.html ; busy loop - rest of intro goes here :P hold: halt ; sleep until the next interrupt (1/50 second) jr hold ; then jump back and sleep again, for ever header: incbin "assets/header.scr" ; my logo header/background end start Not much to look at now, but this simple starting point introduced me to important concepts like interfacing with the Spectrum’s hardware (ULA chip to start with, and AY sound chip later) via the out instruction, how to pull in binary files using Pasmo’s INCBIN directive, using the Z80’s block copy ldir instruction, the use of register pairs for 16-bit addressing, and simple branching to labels. Hello World The next step was to print something on the screen and explore some of the Speccy’s built-in ROM routines, which are somewhat analagous to syscalls on modern operating systems. There are plenty of disassemblies and guides to the Spectrum ROM available on the internet, one of the best commented and linked one is at https://skoolkid.github.io/rom/. Using this as a reference alongside the various “Hello, world!” tutorials showed me how this sort of thing is built up: at EQU 22 ; "at" control character used to position the cursor row EQU 11 ; coordinates where to print the message col EQU 9 prstring EQU $203C ; where the pr_string routine lives in ROM ld de,msg ; DE = address of the first character ld bc,msglen ; BC = how many characters to print call prstring ; print the lot ; busy loop is the same... msg: db at,row,col ; equivalent of "PRINT AT 11,9" in BASIC db "Hello, world!" msglen EQU $-msg ; 16 bytes in all, pasmo shortcut, $=program counter I ended up mostly not using the ROM routines in my code as they dragged in all sorts of things I didn’t need, but it was massively useful to understand how they worked as they introduced me to yet more commonly-used constructs. For example, the PR_STRING routine defined here shows how you can build a loop over each character. If you’re just starting out, I highly recommend going over the 8 lines in that routine with a notepad and pen, working out the contents of the registers and following the loop constructs. I also made a version of it in asm-editor you can try. If you click that link, you’ll see a version of the code loaded into the web editor. Click “Build”, and then step through the code using the “Step” button - although no text will be actually displayed (I skipped over the call to the Spectrum ROM “restart” vector that does this), you’ll see the various states of the registers and for example the BC register counting down as each byte of the string is processed. Adding Some Style Problem is, the Speccy’s built-in font is not exactly dripping with demoscene credibility. I wanted to use some stylish fonts and this turned out to be super easy - a quick search turned up Jim Blimey’s awesome font collection, which included a ready-to-use set of assembler files and some instructions which soon led me to these additions in my code: ; CHARS is the address the ROM's printing routine reads to find the font. ; Picking our own font is simply a matter of over-writing this chars EQU $5C36 ; from Jim's docs - POKE 23607=5C36 in hex ld hl,myfont-256 ; HL = font address - 256 as per docs ld (chars),hl ; from now on the ROM draws with it! ; rest of code is the same myfont: include "textfont.asm" ; "Cartoon" font from jimblimey.com There we go! Now that’s looking a little more elite. Sorry, I mean 31337. Strap in folks, we’re leaving lamer city… So at this point, a month or so in to my dabblings and I’d started to get a reasonable understanding of common idioms and was starting to feel like this was a project that might actually go somewhere. The problem started when I moved on to trying to implement actual demo effects: It became very apparent, very quickly, just how much I didn’t know about how this all worked. Not to mention the fact that at this level, you’re really closely exposed to the physical hardware intricacies, including the design decisions and trade-offs that the hardware designers had to make when working with the extremely limited specs of the early 1980s. Cursed Screen Layout The first such problem - and the cause of probably 90% of my pain - is the Speccy’s extremely weird screen layout. Rather than standard bitmapped display you might expect on more modern hardware, it is instead split into 3 sections, each with a set of interleaved rows (so if you keep adding 1 to a screen address, eventually instead of reaching the next row, you’ll instead end up 8 rows down, and so on until you hit the bottom of the section and then loop back to the row underneath where you started!) followed by a block of “attribute” bytes which determine the colours of each 8x8 block. The attribute blocks are the reason why each 8x8 character cell can only have two colours (a PAPER and INK attribute) which led to the garish “colour-clash” of spectrum games and distinctive style where artists try to keep inside 8x8 blocks as much as possible. But it was the layout of the bitmap portion that is such a mind twister. Download the video You can see this illustrated whenever a game reads in a loading screen from tape. You can see each section built up with the interleaved rows one at a time, and then finally the colour and other attributes (like flashing ). There’s a lot of detail at the fantastic Break Into Program site. In the defense of the designers, they were working with extremely limited hardware and I don’t think the Speccy was ever imagined as a games machine. When you start shuffling bytes around, you realise that this approach is really optimised for displaying text which is neatly arranged into 8x8 character boundaries. For example, using this “scrambled” bit order, if you have a screen address in the hl register pair, you can quickly get to the next pixel line of a given address by simply incrementing the “high” byte. There’s a bunch of other “cheap moves” you can make by simply shifting bits around: Move Instruction Right 8 pixels inc l Left 8 pixels dec l Down 1 pixel line inc h Up 1 pixel line dec h Down 1 character row add 32 to l It was a reasonable choice for a machine that would probably spend a lot of it’s time displaying BASIC program listings - it’s only when you start trying to write demo or sprite routines that it starts to feel really weird. This also means you’ll pretty quickly need a routine to convert X,Y addresses to the relevant memory addresses. Again, L Break Into Program has a useful routine and if you disassemble more or less any spectrum game or demo, you’ll find a version of this procedure somewhere. And yes, this all became horribly relevant later on… The Scroller Download the video Admittedly, not the most interesting demonstration - but this was the first time I had made something move on the Spectrum’s screen with assembly! I watched this with pride more times than I care to admit :) I had a snazzy logo and “Hello World” text, but that doesn’t make for a very interesting intro! The first effect I wanted was the scrolltext - without that, it’s not really an intro. Now, the Amiga had all kinds of hardware sprites, blitter operations, custom chips but the Spectrum had none of that. I was going to have to work out how to do things the hard way - “banging on the metal”, literally by shifting bits around the screen memory by hand and having the CPU do all the work. This is where understanding the Speccy’s weird screen and character cell layout is so important. It turns out there’s a neat trick as long as I stay inside one character row - there’s a RL instruction, which takes a byte and “rotates” it left. It inserts whatever is in the carry flag into the rightmost bit, and whatever gets shifted out ends up in the carry bit again. That means if you string a bunch of them together, you can move a set of pixels across the screen - the moment they “fall off” one character cell, they end up in the carry flag and then the next rl instruction inserts them at the beginning of the next character cell to the left. Repeat 32 times, and you’ve just moved a pixel (or set of pixels) across a line of the screen: linestart EQU $4880 ; leftmost byte of the line 12 rows down lineend EQU linestart+31 ; rightmost byte column ($489F) linelen EQU 32 ; one whole line is 32 bytes (character cells) across dot EQU %00000001 ; the dot "shape", just one pixel lit at the far right end newdot: ld a,dot ; load the dot shape (only rightmost pixel lit) into a ld (lineend),a ; store it in the rightmost byte of the line, pixel is now lit frame: halt ; wait for the next frame, otherwise it whizzes across too fast to see ld hl,lineend ; hl = last (rightmost) byte of the line ld b,linelen ; loop counter, 32 bytes to work through or a ; trick to clear the carry flag, or against itself leaves a register ; unchanged but clears various bits including the carry flag ; see https://jnz.dk/z80/or_r.html shift: rl (hl) ; rotate this byte left (00000001 -> 00000010) dec hl ; point hl at the next byte to the left djnz shift ; loop construct: b=b-1 and then jump back to shift label until b=0 jr c,newdot ; now we've shifted the whole line, so if the carry flag is set now it means ; the dot has just left the screen, so we jump back to the start jr frame ; otherwise it is still somewhere on the screen, so keep shifting! Download the video Hell yeah. It’s almost a proper scrolltext now! Wrapping this in another loop that increments h each time (remember - that gets us moving down a scanline) gets us moving an entire character cell. Here you can see it moving an entire row of my “Hello world” message. There was however a lot more work needed to make a proper scrolltext, with each letter appearing smoothly. I started off by just printing an entire character at a time on the right-hand edge of the screen - it worked, but was pretty ugly as you saw each new character appear at once in the right-hand side of the screen, scrolling 8 pixels over and then the next character flashed up. Instead, I wanted to make each character appear pixel by pixel and that required a bit more bit juggling. The routine I ended up with effectively streams each character in, line by line, pixel by pixel using the font data directly instead of calling the ROM print character routine. It does this by looking up the ASCII code for a character, and then fetching the corresponding font representation into a 8-byte buffer, one byte for each line of data. The font data starts with the space character (character code 32), so a lookup is simply font code base address + (ascii code - 32) * 8 to get to the bits for that letter. Arranged into an 8x8 grid, the bits for the letter H in the font data look like this. For each row of bits, I shift the line left using a neat trick. The byte is loaded into the a register, and then by adding itself to itself - add a,a we shift the value left (in binary, multiplying by 2 moves everything over one bit) so that the pixel I want to draw ends up dropping into the carry flag. This is an idiomatic way of doing it in Z80 code as it’s faster and smaller than the sla instruction. And every cycle quickly adds up when you’re re-drawing 256 bytes every frame! The character buffer then looks like this, with the top line shifted. Now, with the hl register pair pointing at the correct location on the screen, I can then use the rl trick discussed above. This shifts a scanline from the bottom character row of the screen left by one pixel and draws my pixel in the rightmost position by moving it in from the carry flag. Looping 32 times over this moves the entire row by 1 pixel with my new pixel inserted at the right. I then repeat this step again using inc h to move down one scan line until the entire character column has been drawn. At this point, the character buffer looks like this, and all 8 lines of the bottom screen row have all been shifted left by one bit by looping over all 32 character columns. The whole thing then repeats another 7 times until the entire character has been smoothly scrolled onto the screen. So just to re-state this because I think it’s worth the emphasis - just to scroll a message across the bottom of the screen, we’re having to loop over and re-draw 256 pixels every frame! It shows how much faster assembly is than the BASIC interpreter built into the spectrum’s ROM. I found it amazing to see my code finally working and imagining all the bits flying around and registers flapping away. It really makes me appreciate just how much effort went into creating the games I played back then. Music The original Amiga module I converted. A proper 1980s ear-worm! Of course, any good intro needs some music. On the Amiga, the standard was the “tracker” module format, which has a long and involved history of different forks, versions and tweaks. A song is built up from a series of patterns, each made up of rows of instructions. Each instruction usually includes note or pitch information, which sample to play and various parameters such as volume, effects - or in the case of some instructions, which position in the song to jump to. In many ways, it’s almost like a programming language for music. Using ProTracker as an example, a typical note cell is 4 bytes, shown on-screen as something like C-2 01 C20. That breaks down into three fields: the note and octave (C-2), the sample number (01), and the effect and its parameter (C20 - in this case, “set the volume to $20”, which is half of the maximum $40). A close-up of Protracker playing a tune. Note the VUmeters - another classic demo effect I would pinch later on! For an impressive demonstration of what’s possible with a tracker and to see what the note data looks like on-screen, check out the work of h0ffman, and in particular his winning “Professional Tracker” entry from the Revision 2014 competition. He’s an undisputed master of the science (or art!) of tracking, and his “Secrets of Protracker” talk gives a great overview of what’s possible. The Amiga’s sound hardware far eclipsed anything else on the market at the time, and the ZX Spectrum is paltry by comparison. The original 48K models only offered a rudimentary “beeper”, but the later 128K models included a 3-channel AY-3-8912 sound chip. This was very popular in 8-bit micros of the time, and a more-or-less identical variant was also used in the Atari ST. While it only offers basic waveforms and sound processing, it does enable multi-channel “chip music”, and so it became the mainstay of the ZX Spectrum demoscene. It also spawned a variety of SoundTracker clones on the Spectrum itself, although most musicians these days prefer to work in modern tools like Vortex Tracker II, which runs natively under Windows. The Conversion Pex looking back on his demos - this starts at the point where he features my letter and plays a bit of my cover version! For my intro, I wanted to port an Amiga module - and I had the perfect one in mind. One of my favourite Amiga music disks was “Sounds Of Gnome”, released in 1989 by the Swedish duo “Mahoney & Kaktus”. Mahoney also wrote NoiseTracker, one of the first really widely used trackers on the Amiga, and the program that served as my teenage introduction to making computer music of my own. Back in 2018 I made a rock cover of one of the other tunes on that disk, and wrote to Mahoney (real name “Pex”) about it. He ended up showing my “music video” at a demo conference, which I am eternally grateful for! So it seemed only fitting that I mangle another song from that collection with a Spectrum conversion… Sorry, guys! I picked the track “Milkman” which was written by “Glue Master”, an associate of Mahoney’s who had a few of his tracks featured on the compilation. It’s an ideal track for converting: It’s short (only 7 patterns in total), most of those patterns share the same chord sequence, and it looked fairly easy to squeeze the original 4 channels down into the Spectrum’s 3. Plus, it’s an incredibly catchy tune and I knew it well - I’ve been listening to it for 35+ years and often jam along to it on guitar or bass. The conversion started off pretty easily: all I really had to do was copy the patterns note-for-note from the Amiga mod, which I had open in OpenMPT as my Windows tracker. Below is a session with both trackers open on the same pattern - the original Amiga mod on the left, and my 3-channel Spectrum conversion on the right. Note the 11.7 in the 3rd channel of the Spectrum version. This is a “pitch slide down” effect in Vortex Tracker syntax, which drops the pitch by about a quarter-tone. It’s repeated on the next line, so the overall effect is a slide down from the A# that leads in to the next bass note. The original Amiga mod does the same thing using its own syntax - those are the 202s you can see on the same notes, at the bottom of the 4th track. Using just a single “beep” sound for everything was enough to check everything was working, and after a few weeks of hacking on it and re-arranging things in the evenings, I had a basic version which sounded surprisingly good for a first draft: Download the MP3 instead. I have to give a special mention to this forum post which was a massive help getting me started. It’s probably the best introduction to Vortex Tracker I’ve found - many, many thanks to the original author for taking the time to write it all up! The real challenge was making it sound as close to the original as possible. That meant building up a library of better-sounding samples for the different parts, and making extensive use of Vortex Tracker’s “ornaments”. Vortex Tracker comes with a great selection of sounds, and fortunately I’m also a weirdo who likes to listen to chiptunes in the background while I work! So whenever I found something particularly enticing, I filed it away for later pilfering inspiration. A Vortex Tracker sample isn’t really a “sample” in the Amiga sense of a recorded sound - it’s a definition of how a note sounds, one line per frame. Each line can mix in a tone or the white noise generator (useful for snare effects and suchlike), along with various other parameters. Ornaments, on the other hand, give you extra control over what pitch a note plays - and this enables really useful tricks like “arpeggio chords”. For example, an ornament of 0,4 makes the channel switch rapidly (25 times a second) between the note being played and the one 4 semitones above it. It happens so quickly that it sounds much like two notes played at the same time, which means you can build up full-sounding triad chords using only two channels. Take the opening chord in my fully converted version, Cm: It plays C in the first channel and the minor 3rd (D#) in the second, then I apply an ornament to the second channel to bring it up 4 semitones to the 5th (G). That gives all three notes of the chord, while keeping the 3rd channel free for other parts. As well as making much more efficient use of the limited 3 channels, this also gives Spectrum music that characteristic “warbling” sound, which I find really appealing in a retro kind of way! Between tricks like this and samples that mix tones with white noise (to make a combined “bass-snare” sound, for example), I managed to condense the full track down to 3 channels. Many months later, I have to say I’m really pleased with the final result. Playback The original site that was home to Vortex Tracker, documentation and other resources appears to be offline at the moment. I managed to grab a copy of everything I needed from archive.org - you’ll want the PTxTools.7z file for the playback routines discussed here. The playback was simple in theory: As this overview notes, you basically just pull in the player routine, include your module file and call the playback routine once per frame in your main loop. The catch is, none of the source code versions supplied in the archive worked with my Pasmo assembler. Fortunately, there is a pre-compiled binary object shipped alongside the source (VTII10bG in the PTxTools/PT3Play directory). This just has to be loaded in at address $C000 - this cannot be changed, as it has lots of absolute addresses baked in. My code to handle module playback now looks something like this, with the binary module loaded in place: ; Define the start address of the player routine and functions we need player EQU $C000 ; hard-coded org address of binary player plinit EQU player+3 ; init function in docs, module address in hl plplay EQU player+5 ; play and move on one frame ; from the docs - this "analyzes PT3 module header and prepares ; corresponding note and volume tables" ld hl,moduleimg call plinit ; main loop hold: halt call plplay call scroll ; other effects / routines go here... jr hold moduleimg: incbin "assets/music.pt3" Starfield Download the video Well, that didn’t work as hoped. TNFStro - tortoise edition. This brings me to the main event. No self-respecting cracktro is complete without a starfield! My first, naive approach to the classic starfield effect was to scatter random stars across the screen, then move down each scan-line applying the same rl trick I’d used for the scroll-text. I figured I could get parallax by shifting each row a different amount: One pixel for the first row, two for the second, three for the third, then back to one again, all the way down the screen to give the impression of several “fields” of stars moving at different speeds. It turned out there were two major problems with that approach. Firstly, shifting an entire row smeared whatever else was on it along with the stars - which I’d expected, and had filed away as a problem for future me. The bigger issue was speed: by the time I’d finished the first three rows I’d already performed hundreds of shift operations, and it was nowhere near fast enough to fit inside a single frame. Everything crawled across the screen, and the music playback slowed down so much that you could actually hear the arpeggio ornaments warbling away as separate notes. Clearly I was going to have to keep track of the individual stars instead. I found this routine, which was more-or-less what I had in mind and started me off with the idea of a structure to hold the state of each star. It also uses a screen address lookup much like the one from “Break Into Program”. Download the video Dammit. Back with another funeral dirge rendition :( That got things moving, but the co-ordinate lookup and tracking proved expensive to do (it has to run for every star, every frame as well as keep track of the old position for erasing) and I’d been ignoring the bigger question how to display stuff on top of the starfield. My original idea was simply to draw the stars and then paint text over them each frame, which blew the frame budget just as comprehensively as my first attempt. If I wanted something more interesting than an empty starfield, I’d have to keep track of which cells were occupied by text and avoid drawing stars in them in the first place, rather than over-painting them every time. After lots of scribbling in my notepad and visualising rows of bits, I eventually solved both challenges by “cheating” fairly liberally. So now, I don’t generate a random starfield at all. Instead, I built a table of pre-calculated Y and X position pairs: stdata: defb 109, 25, 88,182, 42, 96, 155,226 defb 59,254, 137,169, 160, 84, 125,193 ; ... and so on, for 32 stars ... This saves faffing about with an RNG, and once I have a layout that looks good I know it’ll always look good. I can also “fudge” things and ensure no two stars share the same y co-ordinate which sidesteps a whole class of problems like stars sharing a scanline, and having to deal with them overtaking each other. The “random” layout is generated by a simple Ruby script: #!/usr/bin/env ruby PAIRS = 32 PER_LINE = 4 # pairs per line Y_RANGE = 40..175 # exclude rows taken up by my logo & bars X_RANGE = 0..255 raise "y range too small" if Y_RANGE.size < PAIRS # no duplicate y values ys = Y_RANGE.to_a.sample(PAIRS) pairs = ys.map { |y| [y, rand(X_RANGE)] } pairs.each_slice(PER_LINE) do |row| body = row.map { |y, x| format('%3d,%3d,', y, x) }.join(' ') puts "defb #{body.chomp(',')}" end When the routine initialises, I convert those co-ordinate pairs into a table that tracks the state of each star. So rather than storing a position and translating it to a screen address every frame, each of my star records now looks like this (offsets from the base address): STSCR EQU 0 ; screen address of the byte the star is in, e.g. $4123 STOCC EQU 2 ; a pointer to the "occupancy check" table cell STMASK EQU 4 ; the star's bit pattern, e.g. %00010000 STSTEP EQU 5 ; "speed" - num. of pixels this star travels per frame The key thing is that there are no X/Y co-ordinates at all. For each star I store the screen byte it sits in and the bit mask within that byte, so it’s already in the form the screen writes need. That means no address conversion per frame, just simple bit shifting and rotation. Moving a star is then dead simple. I rotate its mask left with rlca, and when the bit rotates out of bit 7, it lands back in bit 0 with the carry flag set. That’s the signal I need to move the star one character cell to the left, which is one of the “cheap moves” from earlier. Because I’m tracking which cell each star is in, the collision check comes almost for free. That’s what the second pointer is for: STOCC (Star Occupancy) moves in step with STSCR through a table holding one byte per character cell. The “typewriter” routine sets a byte to 1 for every cell containing a text character or the cursor, so checking whether a star is hidden behind text is a single or instruction. If the cell is occupied, I skip drawing the star but carry on rotating and stepping it along until it reaches a cell where it can be displayed: ld l,(ix+STOCC) ld h,(ix+STOCC+1) ld a,(hl) ; A = the occupancy check for this cell or a ; test A against zero jr nz,stmove ; it's occupied, so leave the screen alone Parallax is still handled by moving each star a different number of pixels per frame, but the cost is a lot lower this time as I’m only touching the cells the stars actually occupy, rather than every byte of every row. Download the video That’s looking more like it! The typewriter routine is a simple copy of the font data from memory, and is very cheap as we can use the “cheap moves” to quickly fetch the 8x8 block. I also skip drawing any space character so it leaves the “occupancy cell” unmarked. This means the stars keep drifting through the gaps in the text - you can see here the odd star passing in between the words. I switched to ALL CAPS writing for the final version which helps the effect even more as the font is quite blocky, so you don’t really notice stars skipping the spaces inside a glyph at all. Misc Notes A few odds and ends that didn’t warrant a section of their own… VU Meters Some sort of graphic display bouncing along with the audio playback was another common early demo trope. I did originally consider a real-time graphic equaliser, but that would have been far too complex, probably in terms of spare cycles per frame, and definitely in terms of my limited mental capacity at this point! So I settled for a simple set of bouncing meter bars instead. It’s worth a mention mostly because it gives me an excuse to link to the awesome chibiakumas website! I just love the late 90s web design on display - one can argue that readability and accessibility take a hit, but it’s just so much fun to encounter a site that still looks like this. It’s a clear labour of love and there’s a wealth of content there as well, if your eyeballs are up to it. Their guide on Making Sound with the AY-3-8910 was what showed me how to get the data I needed to draw the meter bars. The examples write a register number to $FFFD to select a register, and I realised that once a register is selected, I could use the in instruction to read its value back out instead of writing a new one. As the page says, R8, R9 and R10 hold the channel amplitudes, so they feed nicely into the height parameter for drawing a bar. Getting the amplitude for channel A is then as simple as: ld a,8 ; Channel A register call ayread ; ... rest of code here ... ayread: ld bc,$FFFD ; Register select port out (c),a in a,(c) ret Twinkle, Twinkle, Little Star This is a simple effect that makes the header logo a little more interesting. I’m not sure where I originally got the idea - I’ve definitely seen something like it on a few late 80s/early 90s demos, though. All I wanted was to pick random star dots in the logo background and animate them. Here’s the header image loaded in ZXPaintbrush with the character cell grid showing. By happy chance, there are four stars which each sit neatly inside a single character cell with enough surrounding space, so they can be used to give an animated “twinkle” effect by drawing a cross out from the centre point and back in again. These dots map to the following co-ordinates, which target the top scanline of each fully-grown star (two pixels up from the centre): twkdata: defb 1,5 ; char row 0, col 5, centre y=3 ink 5 defb 1,20 ; char row 0, col 20, centre y=3 ink 1 bright defb 26,7 ; char row 3, col 7, centre y=4 ink 4 defb 25,22 ; char row 3, col 22, centre y=3 ink 6 bright The “twinkle” routine itself is simple enough: it adds a pixel on either axis to grow the star, waits a few frames, then shrinks it back down again. The interesting bit was how I made the twinkle order pseudo-random. Remember that my starfield was entirely pre-calculated from a data table generated by a Ruby script, which meant I’d never actually needed a random number generator in assembly. From Jonathan Cauldwell’s “How To Write Spectrum Games” e-book, I found another great “hack” which simply walks through the start of the Spectrum’s ROM, picking out random-looking bytes as it goes. As he says, he uses it in most of his games, and it does the job in a really simple way - as long as you stay below the block of structured, repeating data that sits 8Kb into the ROM. twkrand: push hl ld hl,(twkseed) ld a,h and 31 ; keep it within the first 8k of ROM ld h,a ld a,(hl) ; the "random" byte inc hl ld (twkseed),hl pop hl ret Out Of Space About halfway through writing the intro, I hit a fairly predictable problem: I had run out of memory. In an attempt to keep everything self-contained in one block of code, I was pulling all of the assets (the screen background, the player binary, the music file etc.) into the program itself, and then using the ldir block copy instruction to move each one into its correct location. Since that meant effectively storing every block twice, and the player routine had a hard-coded start address of $C000, I quickly ran out of room and my code started crashing into the playback routine. The simplest option was to split things out into separate code blocks, load each one directly to the address it needed to live at, and wrap the lot in a multi-stage BASIC tape loader: 10 CLEAR 32767 20 BORDER 0: PAPER 0: INK 0: CLS 30 LOAD "" CODE 40 LOAD "" CODE 50 LOAD "" CODE 60 RANDOMIZE USR 32768 This loads the main program, then the player routine directly into $C000, and finally the background screen straight into screen memory. It does mean the background starts appearing before the demo has fully loaded, but as I imagine most people will be seeing this running in an emulator or on my TNFS site, it doesn’t really matter. Plus, at this point I was in danger of disappearing down yet more rabbit holes (I still really want to work out how to do multi-colour border effects!) and I’d never have got this thing released. The Zen Of Z80 This whole project is a really weird one to look back at. On the one hand, I’ve fulfilled a childhood ambition and have something I’m very proud of. It’s also pretty unusual these days to end up with a binary where you can say pretty much what every single byte is doing and the story of how it got there. But on the other hand - I now have an answer for why I don’t code in assembly: It’s because I value (what’s left) of my sanity! The thing is, in order to write code like this you have to start thinking like the machine itself. You have to visualise the registers and the status flags flapping away, graphics as lines of bits slotted into strange orders, loops and conditionals as binary logic and so on. Getting this close to the hardware definitely brought about a greater sense of attachment to the hardware of my childhood, and it’s easy to see why assembly coders and sceners get so passionate about it. That said, I’m in no hurry to start re-writing my body of work in assembly! The projects I really enjoy are those that give me an excuse to learn something for the sheer hell of it, and to slow down a little from the ever-increasing velocity and demands of the modern work world. Going back to Z80 code is probably about the furthest I could have taken this, short of going into hardware hacking. I found it at times an intriguing puzzle, often a frustrating exercise that nearly had me throwing in the towel at several stages. This project has certainly taken the longest to go from the “idea” stage of any given feature to any kind of meaningful output. True, I didn’t spend every day of the last year working on it - my actual coding time was probably only a few hours over a couple of evenings a week. But I’ve certainly been thinking about it and pondering it most days (as my dogs will attest). The biggest breakthroughs and “aha!” moments usually didn’t come from behind the keyboard; they came from scribbles in my notebook and thinking about processor states while doing something unrelated. It has however given me massive appreciation and respect for the coders who pull this stuff off and make it look easy. And that goes double for those kids and bedroom coding superstars who managed all of this back in the 80s, and had to work it all out by experiment themselves. I only managed to get this far because of numerous books, online tutorials, GitHub repositories with sample code, YouTube videos… I can’t imagine how much longer this would have all taken me if the only resources I had were my Spectrum, a cassette tape recorder and a notepad. So, as I draw a line under this project - for now! - I raise a glass, and in the words of h0ffman: “Shout out to the hackers, graphicians and protrackers.”
A decade ago, a little bit of history was made. I didn't realize it, but a colleague made a great point, one of those real mind-changing points that seem too obvious to admit same-day. But, the next day, calver.org was born. At the time my team maintained the Python infrastructure for eBay and PayPal, and we were stuck deciding whether we were really ready for a "major" 1.0 release. Semantic Versioning was the only game in town and "major" means "big", right?! Thankfully, a wiser colleague mentioned: Ubuntu and Twisted don't struggle with version number debates. They slap a date on it and keep shipping. In fact, their date-based versions were even better because you always knew where it stood, in terms of updatedness and support. The only problem is that no one really knew about it. Somehow, this problem solving versioning alternative, arguably as old as history itself, had gone nameless for millenia, conspiring to make me feel foolish in an office meeting. Never again! Ten years of adoption Fast forward 10 years, we've seen CalVer adopted by Apple, Nvidia, JetBrains, and countless others. (We have a timeline!) The site may have more inbound links than any other project of mine. Apple made the biggest jump, at WWDC 2025: iOS went from 18 to 26 macOS from 15 to 26 watchOS from 11 to 26 and visionOS from 2 to 26 All landing on one, consistent number like a car's model year. I still remember the texts from the Venn diagram fanbase of my friends who love Apple and reasonable versioning. No such texts from when NVIDIA announced calendar versions across the GPU Operator, RAPIDS, and its monthly NGC containers, but still very cool. Open source, too: Home Assistant, pip, CockroachDB, and yt-dlp all ship on dates, with plenty more on the users page. The conversation even reached the language core; PEP 2026 proposed versioning CPython as 3.YY, and it almost happened, too. And it's never too late, time marches on! Fixing the notation But I don't think I got every detail right from day 1. That's the main motivator for CalVer 26. It's high time to start righting a couple idiosyncratic token design choices, starting with some additions: Meaning Before 26.0 26.0 Full year YYYY YYYY Short year (6, 16) YY YY Zero-padded year (06, 16) 0Y 0Y Short month (1 ... 12) MM M Zero-padded month (01 ... 12) 0M 0M Short week (1 ... 52) WW W Zero-padded week (01 ... 52) 0W 0W Short day (1 ... 31) DD D Zero-padded day (01 ... 31) 0D 0D Seeing double First, the doubled letters. From the first version (16.6), MM and DD meant the unpadded month and day, which reads backwards to anyone who knows date formats (ISO 8601's YYYY-MM-DD, Java, moment.js, day.js), as some community members correctly pointed out. I was ready to flip them, until I checked what people actually use: most projects with a YY.MM.MICRO badge (conda, Twisted, Ansible's tooling) don't pad, and more than a dozen other version management tools (like bumpver and bump-my-version) implement the old meaning. So, it's too late to flip MM's meaning. Instead, 26.0 deprecates it and offers a more explicit and hopefully clearer option: M is the short month, 0M the padded one, and MM is a technically-retired synonym for M. In case you're wondering, the explicit 0M was me being overinspired by Ubuntu's approach, perhaps: 6.06 pads its month but not its year, and YY.0M says exactly that. To pad or not to pad I think it's worth a detour into why padding is even a thing anyways. It's become important now that new ecosystems have emerged that enforced SemVer formatting semantics, and I wanted clear guidance about on the spec site. SemVer forbids leading zeros outright, so Cargo rejects 26.04.0 and Go modules reject v26.04.0. Even Python's packaging spec normalizes leading zeros away, so you can tag 2026.08.19 if you want, but PyPI will still show 2026.8.19. NVIDIA's GPU Operator docs put it this way: "Zero padding is omitted for month to be still compatible with semantic versioning." CalVer was always intended to drop in where SemVer was used. So 26.0 recommends unpadded (YYYY.M.D) as a sane "pure" default for software libraries. But libraries are not the only objects of versioning schemes. The exception is a version that becomes a filename, an image tag, or an object-store key that gets listed lexically. There, padding keeps 26.10 sorted after 26.09, which is why Ubuntu, NixOS, and NVIDIA's own NGC containers pad. More evidence of teams designing their versions. We love to see it. Our FAQ has a longer discussion of the padding issue, as well. Optional segments There was never any rule against them, but 26.0 makes optional trailing segments more explicit with square brackets. Now, yt-dlp's scheme can finally be written down: YYYY.0M.0D[.MICRO]. For the CalVer badges I could find on GitHub, they all stay valid for now. I've got a note on the deprecated spellings and a new copy-paste badge section for new ones. What else is new? It's always a great time to add more citations to the site. A spec changelog; the spec now versions itself: 16.6, 19.7, 26.0. A FAQ: breaking changes, same-day releases, and padding. The users page, rebuilt by category, with past users of note (schemes change; that's fine) and tooling. Case studies: Apple and NVIDIA in, yt-dlp replacing youtube-dl. Much of the thinking behind these changes happened in the GitHub issue tracker over the years, and 19 or so issues close with this release. That's where ideas for CalVer should go, so by all means, open an issue, and we'll get it sorted! In due time, of course. In closing, I can't believe I still love belaboring these numbers so much. Thanks to all (but especially Mark, Glyph, Hugo, issue reporters, translators, and maintainers) for the discussion, and ultimately making the most timely versioning system a timeless classic. See also 2016 announcement Designing a version My Yap on Why CalVer beats Semver
Four sincere attempts at OKRs, four failures, and every time: "you didn't do it right." Maybe the framework doesn't fit you -- so how do you find one that does?
Multiple concurrent writers. Multiple processes. Same SQLite. No modifications.
Brilliant jerks, ZIRP-era managers, and how psychological safety lost the plot. Part 3 of my conversation with Dr. Cat Hicks.