Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Krzysztof Kowalczyk blog
I’m porting gpui-component (a Rust UI component library built on GPUI) to C++ as gpui-cpp. By which I mean: my friend Claude does the porting, I’m just directing. It uses markdown-rs (a CommonMark + GFM parser) markdown parser so I ported it too. Then I optimized it. This post describes what I did with the intention of teaching other how to optimize C++ code. The starting point There are 2 kinds of markdown parser: those that stream nodes as they parse those that build an AST in memory markdown-rs builds an AST. The game is about minimizing the size of AST node. In Rust there are various kinds of nodes, the largest being 152 bytes. Claude generated a single Node struct of 232 bytes. I got it down to 16 bytes. Here’s the initial Node struct, before optimizations: Node, 232 bytes k children 24 position 24 8 string fields — 128 bytes align 24 nums 16 grey = padding and small fields · blue = growable vector · yellow = pointer+length strings Node, 16 bytes (same scale) lastKid · sibling · firstStr · kind+flags Where the 232 went: 8 string fields at 16 bytes each (a char* plus a length), two growable vectors at 24 bytes each (children and table alignments), a 24-byte unist Position (line, column and offset at each end), six bools one to a byte, and the padding all of that dragged in. Every node in the tree pays for every field, whichever kind it is. A Text node uses one string field and nothing else. Arena allocator It’s important that all allocations are done in an arena. Nodes in a parse tree all have the same lifetime which makes it a perfect use for an arena: a bump allocator that can only grow. The only way to free memory is to reset the arena. This is different than calling malloc() to allocate each node individually and then having to call free(). It makes it easy to measure memory usage: check the arena size after parsing. It also allows optimization tricks like compressing pointers. How I measured bun cmd/bench.ts markdown parses 64 KB of markdown in four shapes and reports the arena bytes the parse allocated: prose — paragraphs, emphasis, links nested lists — deep blockquotes and lists gfm tables — tables all the way down entities — text that is mostly &-style character references The number is the whole arena: nodes, the tokenizer’s event list, and the strings. Not just sizeof(Node) × node count. We also measure parsing time to make sure we don’t trade size for speed. Baseline, 64 KB of source: prose 1646.1 KB (25.7× the source) nested lists 1067.9 KB gfm tables 2926.0 KB entities 660.2 KB 1. Pointer compression for strings (bed71ee) On 64-bit platforms, pointers are 8 bytes. Pointer compression reduces this to 4 bytes by calculating a 32-bit offset against a base pointer. Google used compressed pointers in v8 with great result. Reduced memory usage and increased speed. Our string type is the simplest possible string: struct Str { char* data; size_t len; }; That’s at least 12 bytes per string, if len is 4 bytes. Due to alignment, the size is 16 bytes. Strings are allocated in Arena so we can use the beginning of an arena as a base pointer and optimize the pointer from 8 bytes to 4 bytes. We typedef ArenaStr as uint64_t. The lower 4 bytes is uint32_t compressed pointer and upper uint32_t is size. We reduced the overhead of strings from 16 bytes to 8 bytes. Times 8 strings that’s 64 bytes saved per node. Added helper functions for allocating ArenaStr in arena and converting ArenaStr to Str. Savings: 8 strings * 8 bytes, 64 bytes per node: 232 → 168 bytes. shape start before after vs before vs start prose 1646.1 KB 1646.1 KB 1285.9 KB -21.9% -21.9% nested lists 1067.9 KB 1067.9 KB 867.5 KB -18.8% -18.8% gfm tables 2926.0 KB 2926.0 KB 2269.7 KB -22.4% -22.4% entities 660.2 KB 660.2 KB 626.2 KB -5.1% -5.1% 2. Growing arena strings in place (a9d4f3a) Some strings had to grow. Arena allocator doesn’t provide freeing or reallocation. You can only allocate new strings, which wastes memory by leaving dead copies of the string we were appending to. We can grow the last allocated string and that’s what this change does. Luckily, most appends were done to the last string. ArenaStrAppend checks whether the string ends exactly where the arena’s next allocation would begin. If it does, the new bytes are pushed straight onto it and nothing is copied. Decoding HTML entities (e.g. &) broke that optimization by doing an allocation before appending to the string. We switched to decoding entities into a 4-byte stack buffer which enabled optimized append. shape start before after vs before vs start prose 1646.1 KB 1285.9 KB 1285.9 KB +0.0% -21.9% nested lists 1067.9 KB 867.5 KB 729.2 KB -15.9% -31.7% gfm tables 2926.0 KB 2269.7 KB 2269.7 KB +0.0% -22.4% entities 660.2 KB 626.2 KB 163.8 KB -73.8% -75.2% 3. Re-order struct fields, pack the bools (5c0ce6e) Unless told to pack the layout of the struct, C++ compilers align struct fields to the size of the largest primitive type. If you sandwich a bool between 2 uint64_t values, the bool will occupy 8 bytes (sizeof(uint64_t)) instead of 1 byte as it should. Our Node had such wasted space due to padding. My friend Claude was careless. A simple fix is to re-arrange fields, putting the largest first. We also had six bool field which we packed into a uint8_t flags field. Result: 168 → 144 bytes, with no padding at all. We’re beating Rust version now. declaration order: bool after vector = 7 bytes of padding, six times over vector b padding strings b padding largest first, bools in one byte: no padding vectors strings nums f shape start before after vs before vs start prose 1646.1 KB 1285.9 KB 1150.9 KB -10.5% -30.1% nested lists 1067.9 KB 729.2 KB 654.0 KB -10.3% -38.8% gfm tables 2926.0 KB 2269.7 KB 2023.6 KB -10.8% -30.8% entities 660.2 KB 163.8 KB 151.0 KB -7.8% -77.1% Free bytes: same fields, same code, different order. 4. Pointer compression for everything (07ec80f) We compress pointer for all objects allocated in the arena, like we compressed a pointer to the string. ArenaVec<Node*> children held 8-byte addresses; ArenaPtr<T> is a 4-byte offset into the arena’s position space, resolved by ArenaAtOffset. Zero is null, which costs nothing because no allocation ever lands at offset zero. The Node itself doesn’t change size — a vector handle is the same three words whatever it holds — so all of the saving is in the child arrays. shape start before after vs before vs start prose 1646.1 KB 1150.9 KB 1091.9 KB -5.1% -33.7% nested lists 1067.9 KB 654.0 KB 611.6 KB -6.5% -42.7% gfm tables 2926.0 KB 2023.6 KB 1866.9 KB -7.7% -36.2% entities 660.2 KB 151.0 KB 144.9 KB -4.0% -78.1% These shapes rank by children-per-node rather than by node count, which is why tables moved most. 5. Varint encoding string length (f9ebc34) ArenaStr was an offset and a length in 8 bytes. Now it’s the offset alone — 4 bytes — and the length is varint-encoded at the beginning of the string data: [varint len][string bytes][NUL] There are many varint encoding schemes. This one is for unsigned number and codes number < 128 as a single byte. Most strings are below that threshold, so they use a single byte for the varint length, saving roughly 3 bytes per string. Node shrinks from 144 → 112 bytes. Str — pointer + length, 16 bytes per field char* s int64 len ArenaStr — offset + length, 8 bytes u32 off u32 len ArenaStr — offset alone, 4 bytes; the length lives in the arena u32 off len characters 0 Caveat: An offset-and-length string can point at a slice of another string, and a length-prefixed one can’t. We weren’t doing it so it doesn’t apply here. shape start before after vs before vs start prose 1646.1 KB 1091.9 KB 918.0 KB -15.9% -44.2% nested lists 1067.9 KB 611.6 KB 512.3 KB -16.2% -52.0% gfm tables 2926.0 KB 1866.9 KB 1543.6 KB -17.3% -47.2% entities 660.2 KB 144.9 KB 128.5 KB -11.3% -80.5% 6. Fusing exclusive fields (5467a18) A List has a start number. A Heading has a depth. No node is ever both, so they became one uint32_t startOrDepth and kind says which it means. It didn’t shrink the size of Node due to the alignment padding but we did it anyway hoping that future optimization would shrink below padding. shape start before after vs before vs start prose 1646.1 KB 918.0 KB 918.0 KB +0.0% -44.2% nested lists 1067.9 KB 512.3 KB 512.3 KB +0.0% -52.0% gfm tables 2926.0 KB 1543.6 KB 1543.6 KB +0.0% -47.2% entities 660.2 KB 128.5 KB 128.5 KB +0.0% -80.5% 7. Compressing text position (ed5e807) Each Node carried the info about its position in parsed text. It was expensive because it was stored as start and end fields and each of them was: a uint32_t line a uint32_t column a uint32_t offset That’s 4*3*2 = 24 bytes. I assume this info is for debugging so not important for me. I replaced it with 2 uint32_t offsets into a source markdown string, srcStart and srcEnd. We can reconstruct the line/column position from that and the source string. shape start before after vs before vs start prose 1646.1 KB 918.0 KB 828.0 KB -9.8% -49.7% nested lists 1067.9 KB 512.3 KB 462.2 KB -9.8% -56.7% gfm tables 2926.0 KB 1543.6 KB 1379.6 KB -10.6% -52.9% entities 660.2 KB 128.5 KB 120.0 KB -6.6% -81.8% 8. Further compression text position (6a558c4) srcEnd is always after srcStart so we can delta-encode it and shrink to uint16_t. What if it’s bigger than 64 KB? I don’t care, we store it as 65535. This is another case where due to padding we didn’t shrink the struct size. But wait for it. shape start before after vs before vs start prose 1646.1 KB 828.0 KB 828.0 KB +0.0% -49.7% nested lists 1067.9 KB 462.2 KB 462.2 KB +0.0% -56.7% gfm tables 2926.0 KB 1379.6 KB 1379.6 KB +0.0% -52.9% entities 660.2 KB 120.0 KB 120.0 KB +0.0% -81.8% 9. Optimizing storing children (d6c4abc) Some nodes have children that were stored as a growable vector. Empty vector was 24 bytes in the node. We replaced it with a ring of compressed pointers: the parent names its last child, each child names the next one, and the last child wraps back to the first. vector: 24 bytes in the node + a separate array of links ptr · len · cap kid0 kid1 kid2 spare spare ring: 4 bytes in the parent, 4 in each child, nothing else allocated parent kid0 kid1 kid2 lastKid We use a ring and not just a linked list because appending is the only thing the parser does to a child list. A single linked list requires walking the list to find the end, while a ring does not. Saving: 96 → 80 bytes. shape start before after vs before vs start prose 1646.1 KB 828.0 KB 619.0 KB -25.2% -62.4% nested lists 1067.9 KB 462.2 KB 308.8 KB -33.2% -71.1% gfm tables 2926.0 KB 1379.6 KB 898.7 KB -34.9% -69.3% entities 660.2 KB 120.0 KB 98.9 KB -17.6% -85.0% Caveat: accessing a child by index would require a walk through the ring, so indexing in a loop would be quadratic. In our code we only ask for the first or the last. 10. Compressing table alignments info (ca0818c) For tables we were storing column alignments in a separate vector on every node, even though only Table nodes have them. Another 24 bytes per node. We switched to a compressed pointer which points to an optimized representation of the column alignments. There are four alignments (left, right, center, none), so a column needs 2 bits: [varint count][2 bits a column, four to a byte] The whole list is known when the table is entered, so it’s counted, allocated once and filled. For an 8-column table that’s 3 bytes in the arena and a 4-byte offset in the node. Saving: 80 → 60 bytes. We saved more than the 20 bytes because with the last pointer-holding member gone alignof(Node) fell from 8 to 4. shape start before after vs before vs start prose 1646.1 KB 619.0 KB 519.3 KB -16.1% -68.5% nested lists 1067.9 KB 308.8 KB 256.3 KB -17.0% -76.0% gfm tables 2926.0 KB 898.7 KB 710.3 KB -21.0% -75.7% entities 660.2 KB 98.9 KB 89.2 KB -9.8% -86.5% The block is pushed byte-aligned rather than through the general allocator, which rounds to 8 and would have handed back exactly what the varint saved. 11. Fusing exclusive fields (07444d6) Previously we fused exclusive fields start of a List node and depth of a Heading node into a single uint32_t. We fused Table node column alignments info from previous optimization into the same field. We called it uint32_t perKind, and kind says what kind of value it is. Saving: 60 → 56 bytes. shape start before after vs before vs start prose 1646.1 KB 519.3 KB 483.9 KB -6.8% -70.6% nested lists 1067.9 KB 256.3 KB 233.7 KB -8.8% -78.1% gfm tables 2926.0 KB 710.3 KB 646.1 KB -9.0% -77.9% entities 660.2 KB 89.2 KB 86.1 KB -3.5% -87.0% 12. Optimizing eight strings (521e32e) We had 8 strings that were not all used by all nodes. Instead of figuring out how many strings we need at most, I created a linked list of strings in the arena. They are different than regular strings in that they carry a 4 byte compressed pointer to the next string within the arena and the kind of the strings. [u32 next][u8 kind][varint len][len bytes][NUL] We can add as many kinds of strings as we need but we only pay for used strings + 5 byte per-string overhead. Some nodes don’t have any strings. 8 fields: 32 bytes on every node, 7 of them empty on almost all of them value url title alt ident label lang meta 1 field: 4 bytes, and a record only for what the node actually carries first next kind len characters 0 a stored string costs 5 bytes more · a node storing none saves 28 New records go on the head, so storing is O(1), and the walk that finds a kind is at most 8 long and is almost always 1 or 0. In-place growth still works, because a record being the newest thing in the arena is the same condition it always was. Saving: 56 → 28 bytes. shape start before after vs before vs start prose 1646.1 KB 483.9 KB 358.3 KB -26.0% -78.2% nested lists 1067.9 KB 233.7 KB 159.1 KB -31.9% -85.1% gfm tables 2926.0 KB 646.1 KB 402.9 KB -37.6% -86.2% entities 660.2 KB 86.1 KB 73.7 KB -14.4% -88.8% 13. Fusing two enums into one (f3c14b9) As it happened we had two enums: one needed 6 bits another needed 2 bits We fused them from 2 bytes to 1 byte. Because this 1 byte saving dropped below padding we saved 4 bytes and went from 28 → 24 bytes. shape start before after vs before vs start prose 1646.1 KB 358.3 KB 321.2 KB -10.4% -80.5% nested lists 1067.9 KB 159.1 KB 136.5 KB -14.2% -87.2% gfm tables 2926.0 KB 402.9 KB 341.9 KB -15.1% -88.3% entities 660.2 KB 73.7 KB 70.4 KB -4.5% -89.3% 14. Remove position, reduce allocator’s alignment (861c803) At this point I decided that I didn’t need the position so I removed it. Other markdown parsers don’t carry it around so it doesn’t seem very useful. I reduced overhead of perKind by converting it to a record in the string list from step 12 — varint-encoded, under its own kind byte. A List, Heading or Table pays ~8 bytes for it; every other node pays nothing, where a field cost 4 bytes on all of them. Savings: 24 → 16 bytes. For safety arena allocator aligns allocations to 8 bytes but a 16 bytes Node can be allocated at 4 bytes, which we did. This reduces wasted space between allocations. shape start before after vs before vs start prose 1646.1 KB 321.2 KB 272.0 KB -15.3% -83.5% nested lists 1067.9 KB 136.5 KB 110.2 KB -19.3% -89.7% gfm tables 2926.0 KB 341.9 KB 250.5 KB -26.7% -91.4% entities 660.2 KB 70.4 KB 65.6 KB -6.8% -90.1% End results The results are pretty dramatic: sizeof(Node) prose nested tables entities start 232 1646.1 KB 1067.9 KB 2926.0 KB 660.2 KB end 16 272.0 KB 110.2 KB 250.5 KB 65.6 KB -93% -83.5% -89.7% -91.4% -90.1% A parse of 64 KB of prose cost 25.7× the source in arena bytes. It costs 4.2× now. The entities shape went from 10.3× to 1.02×. The speed was unchanged. Fastest of 3 runs: prose 8.47 → 8.22 ms nested 9.45 → 9.26 ms tables 12.88 → 12.92 ms entities 5.90 → 5.85 ms Those are within margin of error. The phase of building the tree got a measurable speed up: 0.397 → 0.302 ms, about 24% faster. This is from allocating less and touching fewer cache lines. This is not visible on micro benchmarks, but using less memory will slightly speed up the rest of the application. Lessons learned Arranging struct fields by size is good. It costs literally nothing. Pointer compression is good. 8 bytes become 4 bytes and the cost of converting back and forth is negligible, as Google shown in their v8 blog post and is re-inforced by our benchmarks Varint-encoding is good. Most strings are short so varint encoding can save 3 bytes per string on average. Moving rare fields out of line is good. The way we reduced 8 strings into an out-of-line list. Only pays off if savings is bigger than the cost of additional metadata. sizeof only drops when the saving crosses an alignment boundary. Two of our changes didn’t reduce size of Node struct but it paid off in later optimizations. The allocator’s alignment is part of sizeof. A 28-byte struct from an 8-aligned bump allocator is 32 bytes. We need benchmarks. You can’t improve what you can’t measure. Our benchmarks measured both memory usage and speed, to ensure we didn’t regress speed to save memory.
I maintain SumatraPDF on GitHub. People fork it and make their own changes. I want to know which forks are active and what they’re working on. GitHub has a Network tab for this. I find it lousy. It’s hard to see active forks at a glance. Panning and zooming the UI is slow and fiddly. I just want a sorted list of forks with actual changes. So I wrote a small script: github-active-forks.ts. What it does It uses the GitHub API to find forks that have ahead commits (changes not in upstream) from the last year. For each active branch it prints: a link to the branch on GitHub short sha, author, relative date, first line of commit message Forks are sorted by most recent activity, oldest first. Commits within a branch are oldest first. It compares each fork branch against the matching upstream branch (e.g. rel3.6working vs rel3.6working), not always master. That way you only see commits unique to the fork. How to run it Download the script: curl -O https://gist.githubusercontent.com/kjk/71a7679dd408d52bc612cdf5eecace58/raw/7d7f605dde6b739424e5190a79e3988477312e74/github-active-forks.ts You need Bun and the GitHub CLI logged in (gh auth login). Or set GITHUB_TOKEN. Run it on any repo as owner/repo: bun github-active-forks.ts sumatrapdfreader/sumatrapdf Progress goes to stderr; results go to stdout, so you can redirect to a file: bun github-active-forks.ts sumatrapdfreader/sumatrapdf > forks.txt Example output Excerpt from running on SumatraPDF (74 forks with ahead commits): https://github.com/wackget/sumatrapdf-Feldherren-version/tree/single-page-fit-scrollbar (289d ago) c53a51d wackget 294d ago Added scrollbar usable in Fit a Single Page display mode. f927640 wackget 294d ago scrollbar in single page mode now obeys the hideScrollbars setting. 7f57861 wackget 288d ago improved scrolling speed when zoomed in, in non-continuous single-page view. https://github.com/xyzzyx99/sumatrapdf/tree/rel3.6working (5d ago) c8d1f7b xyzzyx99 13d ago Add GitHub Actions workflow for building project 8a92547 xyzzyx99 13d ago Fix bugs, revert using version 3.6 1f27324 xyzzyx99 13d ago Save and restore CHM scroll position 24e7974 xyzzyx99 6d ago Preserve current CHM URL across tab restore 30b1cb4 xyzzyx99 5d ago Update README with new bug fixes and formatting https://github.com/lsq/sumatrapdf/tree/dev (today) 72d7069 lsq 11d ago feat: support share file via socket a1fc427 lsq 10d ago feat: open home page 938bc07 lsq today add localsend v2.1 api support Much easier to scan than the Network graph.
A new JavaScript library pretext for fast text measuring / layout popped up on social media. Potentially interesting given its focus on speeding up text rendering in web apps and me writing web apps and liking them being fast. I looked at the code and saw a function isCJK(). Given my 3 decades of programming and performance optimization, it looked like it could be sped up. This is a story about ideas on making JavaScript faster and the process of quickly implementing and benchmarking them. The code export function isCJK(s: string): boolean { for (const ch of s) { const c = ch.codePointAt(0)! if ((c >= 0x4E00 && c <= 0x9FFF) || (c >= 0x3400 && c <= 0x4DBF) || (c >= 0x20000 && c <= 0x2A6DF) || (c >= 0x2A700 && c <= 0x2B73F) || (c >= 0x2B740 && c <= 0x2B81F) || (c >= 0x2B820 && c <= 0x2CEAF) || (c >= 0x2CEB0 && c <= 0x2EBEF) || (c >= 0x30000 && c <= 0x3134F) || (c >= 0xF900 && c <= 0xFAFF) || (c >= 0x2F800 && c <= 0x2FA1F) || (c >= 0x3000 && c <= 0x303F) || (c >= 0x3040 && c <= 0x309F) || (c >= 0x30A0 && c <= 0x30FF) || (c >= 0xAC00 && c <= 0xD7AF) || (c >= 0xFF00 && c <= 0xFFEF)) { return true } } return false } My spider sense tingling To make code run fast you have to have mechanical sympathy. You need a good mental model of how CPUs and programming languages work, at the low level. Because I have mechanical sympathy, I know that to evaluate multiple || statements, the program has to check every statement until it finds one that is true. For the case of not matching any range, it has to do all 15 comparisons. My immediate thought was that most characters are ascii (non-cjk) and therefore require 15 comparisons. An early exit should speed things up: if (c < 0x300) continue;. It adds 1 comparison for matching cases but removes 14 (15 - 1) for the most common case of ascii characters. Hunch is good, benchmark is better Micro-benchmarking is subtle. In the past I would google for a JavaScript benchmarking library and trust that it’s sound. Claude wrote a sound, simple benchmark function: function bench(name, fn, iterations = 1_000_000) { for (let i = 0; i < 10_000; i++) fn(); // warmup const start = performance.now(); for (let i = 0; i < iterations; i++) fn(); const elapsed = performance.now() - start; const opsPerSec = ((iterations / elapsed) * 1000).toFixed(0); console.log(`${name}: ${elapsed.toFixed(2)} ms (${opsPerSec} ops/sec)`); } // to call: bench("isCJKRegex ", () => isCJKRegex(input)); The subtle part is warmup phase. JavaScript JIT engines that you’re likely to use (v8 in Chrome, JavaScriptCore in Safari, SpiderMonkey in FireFox) have several optimization tiers and only do most aggressive optimization for frequently called functions. The warmup phase forces JIT to compile fn with most aggressive settings. This is only fair - I assume isCJK is called frequently. If it wasn’t, there’s no point sweating to optimize it. The new AI normal In the past I would have to manually implement each optimization idea. With AI I can delegate the coding and therefore test more ideas. The following was done with Claude 4.6. 100% vibe coding There are levels of delegating things to AI. On the extreme side of giving up control is 100% vibe coding. I had 2 ideas for performance optimization: early exit replace 15 linear checks with binary search But what if I didn’t? What if I just ask AI to come up with ideas? The following is my conversation with Claude 4.6 and what I’ve learned. I created is_cjk.js file and pasted the original function. look at is_cjk.js. come up with 2 ideas for how to speed up isCJK(). Implement those as separate functions and write a javascript micro-benchmark that compares those 2 new implementations against isCJK Two ideas: Pre-compiled regex — push the entire check into the native regex engine, avoiding JS-level for…of iteration and codePointAt() overhead. Typed array bitmap for BMP — for BMP characters (the vast majority of real text), a Uint8Array bitmap gives O(1) lookup with a single array access. Fall back to range checks only for supplementary plane characters. I have to admit: I recognize those as valid optimization ideas but they didn’t immediately occur to me. Regex works because regex implementations in JavaScript engines are insanely optimized. Bitmap is a known optimization for doing true/false lookup for numbers. Let’s say you have total range 0-255 and you want a function that checks if a number is in range 1-10, 20-30, 40-50, 60-70, 80-90. That’s 5 if statements, each doing 2 comparisons. You can instead use 1 bit for each of the 256 numbers (i.e. 256⁄8 = 32 bytes), pre-compute the array of bytes and set the bit for the ranges. Then lookup is just math to get byte offset and bit, load the byte, check the bit. Super fast. Claude not only wrote the benchmark function, 2 new implementations but also ran the benchmarks using node and showed me results in a readable way. The bitmap approach was significantly faster. Human assisted coding Bitmap was the fastest and yet I discarded it using my human judgement. The bitmap needed 65kB of memory and that’s just too much for this function. That’s where human judgement comes in to improve collaboration with AI. I didn’t specify that I also care about memory usage so AI wrote fast code that was using too much memory. You’re benchmarking what? Here’s another tricky part of benchmarking: knowing what you benchmark. There are 3 popular JavaScript engines. Claude ran benchmarks using node which uses Chrome’s v8. That’s good because that’s the most popular browser and therefore most popular JavaScript JIT engine. It’s also good to sanity-check with at least one other JavaScript engine. bun uses Safari’s JavaScriptCore so I asked Claude: run the tests using bun The results were similar, which is good. We don’t want code that is fast in one engine but slow in another. We don’t control which browser the user of our code is running. Continuing collaboration with AI I had some more ideas so here’s the rest of my conversation with Claude. don’t benchmark bitmap, it uses too much memory; write a variant of isCJK that does an early false exit if char is less than smallest Early exit is my first insight I wanted to test. As expected, early exit is good 2x speedup for latin strings, although regex beats it on cjk strings. That is the curse of engineering trade-offs: you have to decide if you want to optimize for cjk strings or non-cjk strings. now implement a variant that does inline bisect / binary search That was my second idea: do a binary search instead of sequential if comparisons. It was faster than original but slower than regex / early exit. Also, Claude didn’t do what I meant. It stored the ranges as array: // Flat array: [lo0, hi0, lo1, hi1, ...] sorted by lo const cjkRanges = new Uint32Array([ 0x3000, 0x303f, 0x3040, 0x309f, ... That’s what I meant: implement a variant isCJBisect that doesn’t use array but unrolls binary search logic into if statements The implementation is gnarly: function inCJKUnrolled(c) { if (c < 0x3000) return false; if (c < 0xff00) { if (c < 0x3400) { if (c < 0x3040) { return c <= 0x303f; } else if (c > 0x309f) { return c >= 0x30a0 && c <= 0x30ff; } return true; // ... more gnarly code But what do I care? Claude wrote it. Arguably it’s hard for me to review for correctness but it’s perfectly testable. We can write a test to compare if it returns the same result as original function for every possible input. As expected, unrolled version was faster than array-based but not always faster than regex or early exit. But Claude wrote something that caught my eye: The for…of + codePointAt iterator overhead remains the bottleneck shared by all non-regex variants on long strings. It’s well known that for ... of loop is slower than for (let i = 0; i < n; i++). It’s one of those things that you need to know. I knew this but I didn’t connect the dots in this particular case. write variant of isCJKEarly which replaces for of with a faster loop Result: isCJKEarlyFast crushes everything across the board on both runtimes The code for isCKJEarlyFast. Looks like we have a winner: Replacing for…of + codePointAt() with charCodeAt() + index loop eliminates the iterator overhead, which was the main bottleneck. The early c < 0x3000 bail then skips all range checks for ASCII/latin characters cheaply. Only regex still wins on Node’s long latin miss case thanks to V8’s native string scanning. There’s a subtlety that makes the rewrite of for .. of into for (let i = 0; i < n; i++) tricky. Logically JavaScript strings are composed from 16-bit numbers (characters). Some Unicode code-points are bigger than 16-bit and are stored as 2 16-bit numbers, called surrogate pairs. So writing a loop over unicode codepoints needs to take this into account. It’s tricky code but again, what do I care? Claude wrote it and it’s perfectly testable. Pressing on The good thing about claude writing most of the code and removing other tedium like running the benchmarks is that we can do more in the same amount of time. So I explored even more. I noticed: export function isCJKUnrolled(s) { for (const ch of s) { if (inCJKUnrolled(ch.codePointAt(0))) return true; } return false; } What happens when inCJKUnrolled is inlined? inline inCJKUnrolled into isCJKUnrolled Result: Done. Inlining didn’t help — it actually got slightly slower in most cases (the engine was already inlining the small function). No change in rankings; EarlyFast remains the winner. Turns out it’s not faster, which was surprising. Inlining is supposed to remove overhead of function calls, which matters in loops. Surprising isn’t bad. We run benchmarks to validate our hunches. For completness, I wanted to compare with original where for .. of was rewritten. also write original with for of replaced by standard for loop and bench all and give results as markdown table Claude was nice enough to show benchmark results in a nicely formatted table, without me even asking. It just knows things. But as you can see above, you can ask it for results in markdown table to e.g. include in a GitHub bug report. Here are final results. The number is operations/second in millions. Higher is better. Bun Scenario Original Regex Early Bisect Unrolled EarlyFast ForLoop Single CJK 11M 22M 12M 12M 16M 79M 57M Single latin 20M 24M 49M 21M 22M 40M 36M CJK string 12M 18M 17M 12M 14M 42M 42M Latin string 1.5M 4.7M 3.6M 3.0M 3.5M 17M 6.5M Mixed string 6.5M 5.9M 9.9M 8.6M 13M 62M 34M Node Scenario Original Regex Early Bisect Unrolled EarlyFast ForLoop Single CJK 85M 54M 51M 44M 56M 112M 110M Single latin 59M 72M 83M 55M 74M 115M 103M CJK string 79M 55M 60M 55M 64M 90M 129M Latin string 4.2M 57M 4.3M 2.8M 4.5M 12M 8.3M Mixed string 14M 15M 16M 10M 15M 41M 38M Conclusions AI is a big unlock. It took me under 30 minutes to test various hypotheses and find out a significant speed up. Without Claude it would take several hours and I would likely not do it at all. It’s just not important enough to spend a working day on it. To get best results we still need to apply human judgement and guide the AI. Programming expert knowledge An expert is simply someone who knows things. We know things because we learn them. If you were paying attention you might have learned the following things: importance of warmup phase when benchmarking JIT compilers for .. of is slower than for (let i = 0; i < n; i++) regex matching in JavaScript engines is fast subtlety of surrogate pairs in JavaScript strings using bitmaps to speed up range lookups Resources All the code is in https://gist.github.com/kjk/bdbea9d90c3bb0454fbe26353c521bfd I like to write fast code. If you want a fast bookmark manager / note taker, try MarkLexis. If you want a fast PDF / ebook / comic book reader for Windows, try SumatraPDF.
So you’ve installed Visual Studio and you want to run the compiler cl.exe from command-line. Microsoft makes it surprisingly hard. They give you a shortcut which opens a terminal window with cmd.exe setup for compilation. But I don’t want a separate window, I want to use the terminal app. You can run cmd.exe /k "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x64 (location for your setup can be different). But I don’t want to run inside cmd.exe. I want to use powershell. What exactly does vcvarsall.bat do? Not much: it just sets some env variables and updates PATH. We can reverse-engineer what it does: cmd.exe set >before.txt cmd.exe /k "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x64 set >after.txt Now compare before.txt and after.txt to see what changed. I asked AI to do it for me and here’s the beginning of what I found: CommandPromptType=Native DevEnvDir=C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\ ExtensionSdkDir=C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs EXTERNAL_INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um Framework40Version=v4.0 FrameworkDir=C:\Windows\Microsoft.NET\Framework64\ FrameworkDir64=C:\Windows\Microsoft.NET\Framework64\ ... more stuff I saved that to diff.txt file. Now that we have that we can ask our friendly AI to convert this to a powershell function: write powershell function that sets variables as seen in @diff.txt I get something like this: function set-msvc { <# .SYNOPSIS Sets Visual Studio Developer Command Prompt environment variables. .DESCRIPTION This function sets all the environment variables needed for Visual Studio 2022 native x64 development, equivalent to running vcvars64.bat. .EXAMPLE Set-VsDevEnv #> [CmdletBinding()] param() $env:CommandPromptType = "Native" $env:DevEnvDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\" ... more stuff We need to change one thing: $env::PATH from absolute to just adding paths to current $env::PATH so that it’ll adapt to future changes in PATH. Now add that to $PROFILE and you can type set-msvc to setup powershell to have access to cl.exe, msbuild and other tools. Below is the whole function but note that this is only for amd64 toolset. For 32-bit or arm setup would be different. You can get by re-running the above logic for vcvarsall.bat with different arguments. function set-msvc { <# .SYNOPSIS Sets Visual Studio Developer Command Prompt environment variables. .DESCRIPTION This function sets all the environment variables needed for Visual Studio 2022 native x64 development, equivalent to running vcvars64.bat. .EXAMPLE Set-VsDevEnv #> [CmdletBinding()] param() $env:CommandPromptType = "Native" $env:DevEnvDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\" $env:ExtensionSdkDir = "C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs" $env:EXTERNAL_INCLUDE = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" $env:Framework40Version = "v4.0" $env:FrameworkDir = "C:\Windows\Microsoft.NET\Framework64\" $env:FrameworkDir64 = "C:\Windows\Microsoft.NET\Framework64\" $env:FrameworkVersion = "v4.0.30319" $env:FrameworkVersion64 = "v4.0.30319" $env:FSHARPINSTALLDIR = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools" $env:HTMLHelpDir = "C:\Program Files (x86)\HTML Help Workshop" $env:INCLUDE = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" $env:is_x64_arch = "true" $env:LIB = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x64" $env:LIBPATH = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\lib\x86\store\references;C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.26100.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.26100.0;C:\Windows\Microsoft.NET\Framework64\v4.0.30319" $env:NETFXSDKDir = "C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\" $env:Path = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin;C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;" + $env::Path + ";C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg" $env:Platform = "x64" $env:UCRTVersion = "10.0.26100.0" $env:UniversalCRTSdkDir = "C:\Program Files (x86)\Windows Kits\10\" $env:VCIDEInstallDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\" $env:VCINSTALLDIR = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\" $env:VCPKG_ROOT = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg" $env:VCToolsInstallDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\" $env:VCToolsRedistDir = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Redist\MSVC\14.44.35112\" $env:VCToolsVersion = "14.44.35207" $env:VisualStudioVersion = "17.0" $env:VS170COMNTOOLS = "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\" $env:VSCMD_ARG_app_plat = "Desktop" $env:VSCMD_ARG_HOST_ARCH = "x64" $env:VSCMD_ARG_TGT_ARCH = "x64" $env:VSCMD_VER = "17.14.24" $env:VSINSTALLDIR = "C:\Program Files\Microsoft Visual Studio\2022\Community\" $env:WindowsLibPath = "C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.26100.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.26100.0" $env:WindowsSdkBinPath = "C:\Program Files (x86)\Windows Kits\10\bin\" $env:WindowsSdkDir = "C:\Program Files (x86)\Windows Kits\10\" $env:WindowsSDKLibVersion = "10.0.26100.0\" $env:WindowsSdkVerBinPath = "C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\" $env:WindowsSDKVersion = "10.0.26100.0\" $env:WindowsSDK_ExecutablePath_x64 = "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\" $env:WindowsSDK_ExecutablePath_x86 = "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\" $env:__DOTNET_ADD_64BIT = "1" $env:__DOTNET_PREFERRED_BITNESS = "64" $env:__VSCMD_PREINIT_PATH = "C:\Program Files\PowerShell\7;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Microsoft VS Code\bin;c:\Users\kjk\AppData\Local\Programs\cursor\resources\app\bin;C:\Program Files\gs\gs10.03.1\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\GitHub CLI\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Rust stable MSVC 1.88\bin;C:\Program Files\PowerShell\7\;C:\Program Files\nodejs\;C:\Program Files\dotnet\;C:\Program Files\CMake\bin;C:\Program Files\Go\bin;C:\Program Files\RedHat\Podman\;C:\Program Files\Tailscale\;C:\Program Files\Git\cmd;C:\Program Files\Docker\Docker\resources\bin;C:\Users\kjk\AppData\Local\Microsoft\WindowsApps;C:\Users\kjk\AppData\Local\Microsoft\WinGet\Links;C:\Users\kjk\.bun\bin;C:\Users\kjk\.dotnet\tools;C:\Users\kjk\go\bin;C:\Users\kjk\AppData\Local\Programs\superfile\;C:\Users\kjk\AppData\Local\Microsoft\WindowsApps;C:\Users\kjk\AppData\Local\GitHubDesktop\bin;C:\Users\kjk\AppData\Local\Programs\cursor\resources\app\bin;C:\Users\kjk\AppData\Roaming\npm;C:\Users\kjk\.dotnet\tools;C:\Users\kjk\AppData\Local\Programs\Antigravity\bin;C:\Users\kjk\AppData\Local\Programs\Zed\bin;C:\Users\kjk\go\bin;C:\Users\kjk\OneDrive\bin;C:\Users\kjk\.bin\jai\bin;C:\Users\kjk\.bin\mupdf-1.27.0;C:\Users\kjk\.local\bin;C:\Users\kjk\AppData\Local\Programs\WinMerge;C:\Users\kjk\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\Scripts;C:\Users\kjk\OneDrive\bin\sublime_text" Write-Host "Visual Studio 2022 Developer Environment configured for x64." -ForegroundColor Green }
TOON stands for a Token-Oriented Object Notation. It’s a new text format that has the same capability as JSON but uses less space. It was invented to lower costs of sending data (tokens) to LLM AIs but it has 2 advantages over JSON: smaller than JSON more readable than JSON It has implementation in many programming languages, including those I care about: JavaScript and Go. Therefore it’s a good use for non-AI cases e.g.: logging structured data sending data from server to client Here’s an example of JSON and TOON formats: { "table": "A", "currency": "dolaramerykański", "code": "USD", "rates": [ { "no": "001/A/NBP/2024", "effectiveDate": "2024-01-02", "mid": 3.9432 }, { "no": "002/A/NBP/2024", "effectiveDate": "2024-01-03", "mid": 3.9909 }, ] } table: A currency: dolaramerykański code: USD rates[252]{no,effectiveDate,mid}: 001/A/NBP/2024,2024-01-02,3.9432 002/A/NBP/2024,2024-01-03,3.9909 As a result, I’m switching to using TOON whenever possible.
More in programming
After a write-up in the New York Times, Mommy Bloggers had two options. Either lean in, or step back. Given how popular it became after that, it's not hard to guess which option they chose. The post Mommy bloggers react appeared first on The History of the Web.
I'm quite a bit late on this one, but Haunt version 0.4.0 was released released back in July. I haven't had much time for blogging, but I'm catching up now! This release contains a small set of improvements and bug fixes since the 0.3.0 release in 2024. About Haunt Haunt is a static site generator that uses the Guile Scheme as its configuration language. It aims to be simple, functional, and extensible. Features include: Easy blog and Atom/RSS feed generation Markdown post support Simple development server for viewing edits before publishing Purely functional build process User extensibility Notable changes Added support for HTML in Markdown documents. This was a long time coming because guile-markdown did not support it and the library was abandoned by the original maintainer. As part of my work at Spritely, we forked it, implemented the relevant portions of the CommonMark specification, and released it. Spritely's guile-commonmark fork is now considered to be the official upstream by Guix and others. A further consequence of this is that guile-lib is now a required dependency for building Haunt as we need the (htmlprag) module to parse Markdown documents with embedded HTML. html->shtml from guile-lib's (htmlprag) module is now used instead of xml->sxml in the HTML reader. It was silly of me to use xml->sxml for this purpose years ago, but at the time I wanted guile-lib to be an optional dependency. Added haunt new subcommand for creating a new site. Added default directory, template, and prefix arguments to flat-pages procedure. Added support for index metadata flag to flat pages for pretty URLs. Flat pages now receive all page metadata, not just the page title. This is a breaking change from 0.3.0. Added .scm as an additional extension for sxml-reader. make-file-extension-matcher now supports multiple extensions. Fixed emission of <script> and <style> elements. Fixed handling of no available reader in flat pages builder. Fixed unreachable error handling clause when a reader is not found for a post. Fixed default blog theme template missing an <html> tag. Fixed overloaded -h option in haunt serve. Deprecated post in Skribe reader in favor of document. Download Haunt 0.4.0 is already available in Guix: guix pull guix install haunt See the Haunt project page for information on how to build from source. Thank you to Camilo Rodrigues, Noé Lopez, jgart, Jakob L. Kreuze, and Daniel Meißner for their contributions to this release! Happy haunting!
This is a transcript from a talk I gave at the German Perl Workshop earlier this year. If you'd prefer to watch the video recording, you can find it here. I have lots of photographic projects on the go. Lots of these being on film, as some of these I started shooting a long time ago. I don’t have any particular loyalty or attraction to film, it’s just that I started shooting many of these projects before affordable medium format digital was available. Since I mostly shoot medium/large format film I never really jumped to digital until recently, so film has continued to feature heavily in my workflow. That said, it’s a pain in the arse to shoot film now given the spiraling costs, limited availability, and issues around traveling with it: modern airport CT scanners, being rolled out across many airports, are much more convenient but will fog film. Asking for a hand inspection often comes down to arbitrary timing - how busy the security is, how experienced the operator is, or if you’re lucky/unlucky. I’ve had film forced to be scanned (and fogged) and politely argued with security on more than one occasion. I don’t want to deal with that so don’t travel with film anymore, thus I am shooting less of it and have mostly moved to digital. I still have a tonne of film I need to scan and process however. Here’s just some of the binders and files of film. I don’t plan to scan all of this, but I do plan to scan the ones I need to. Probably in the region of a couple of thousand frames. I want to scan to the highest possible quality (within reason) for archiving, book projects, and large prints. If you’re wondering how large I print, it can be up to 160x60cm panoramics for selling. This is restricted by the size of my printer (that’s another story). Three Years Ago Three years ago I almost bought a scanner. I ended up blogging about it and the post got a bit of traction on Hacker News (HN). I’m never quite sure which posts I submit will pique the interest of the users. I’ll spend months chipping away at a draft and when I post it it tanks. Or I’ll cobble something together in twenty minutes, like the linked one above, and it gets 440 points and over 300 comments… The thread had some useful suggestions and some not so useful ones, the not so useful ones being effectively “buy an Epson”: I’ve had one for fifteen years and it’s not good enough for large prints or archiving. It’s passable for web stuff and smaller prints, but for my recent use cases? Not even close. Ten years ago I had negatives scanned with a high resolution scanner for the first time and recently, wanting to scan my archives for various projects, I decided I should invest in one of those scanners. The Original Plan The plan, back in 2023, was simple: Buy scanner (at significantly reduced rate) Scan all my film Sell scanner Profit! And I mean profit - the scanner that I almost bought was being offered to me at about 2/3rd of the price they usually sell. And they’re becoming harder to find in working order so the prices are going up. Or profit in not having to pay > 25.- CHF per frame to have someone else do this. You can see the pricing from The Film Lab. You can read the original blog post to find out more about the scanner in question, so I won’t repeat it here. Other than the parts being relevant to the rest of this post, namely that the scanner was showing hard and soft problems. The software that drives the scanner was last updated in 2012, it’s proprietary and closed source, requiring 32bit architecture and no third party drivers or software exist. So you are stuck using old software/computers to run it. Or maybe you could use emulation / virtualisation? The problem there is that the interface is firewire, or SCSI on the even older models, and firewire is known to be problematic on these scanners as the controllers start to go bad after a decade of continued use. That’s a risk, and the scanner was very much EOL as the firewire controller was dying: both ports were bad that suggests controller, not ports. The scanner would have been €5,000 to purchase and then €3,000 (ish) to repair. Or, as HN suggested - just open it up and use a soldering iron. I’m not going to drop 5k on something and then start poking it with a soldering iron. I’ll pass on that thanks. Camera Scanning In the meantime I’ve been camera scanning, which you can read about in another blog post. But how does that compare cost wise? It’s expensive because you’ll need a high resolution camera, a macro lens, copy stand, negative carrier/holder, and quality light source. You’ll look to spend anything from three to five thousand Euros on everything. Camera scanning does actually work well, in that it’s close to a high resolution dedicated scanner. But you have to setup the entire thing every time you want to use it, including ensuring everything is straight and parallel. It also suffers from the same weakness as most other scanning methods. What do you think that is? Film Flatness Or lack thereof: Film is rarely flat, especially so with 35mm. These are pretty mild examples of curl. It tends to be flatter in the larger formats but then you get into flatness issues due to it sagging. The smallest difference in the film plane can cause major issues in sharpness due to focus fall off (film scanning is essentially macro photography). Any workflow or solution that does not take this into account is significantly compromised. And the workflow is only as good as its weakest part. This is the biggest problem in scanning film - all other considerations are more than adequate these days: resolution, dynamic range, etc. However, most negative carriers don’t keep the film perfectly flat. This has always been a problem - this is from a book called “Edge of Darkness” which is about traditional analog photography and printing, and summarises the problems of negative carriers thusly: “if you use a glassless negative carrier, you might as well just buy the cheapest enlarging lens you can find. You are simply throwing away the money and sharpness you paid for it in your enlarging lens, and also in your fine camera and the expensive lenses you bought for it… No film will lie flat in a glassless carrier. That’s right, none… There is no avoiding this issue. Use glass.” So you have to use (anti-newton ring) glass, which introduces other issues - you’ve now got extra glass in the transmission path, and dust (which isn’t a massive problem, but a pain nonetheless). You could use drum scanning, which is absurdly impractical from a cost and operating point of view. Or you could use a Flextight, the scanner I almost bought three years ago. Interim Solution I stuck with camera scanning, but wasn’t happy though, because of film flatness and the setup faff. So of course I started looking for another scanner. I was idly browsing near the end of 2025 and came across this one. It’s exactly the same spec as the one I tried three years ago, except SCSI not Firewire so less prone to failure. It just predates Hasselblad buying Imacon (so is pre the rebranding, etc). It was in Switzerland so I could inspect and pick it up. It was also significantly cheaper than the previous one I had looked at, so worth a punt even if I needed to take a soldering iron to it. We went to St Gallen for a weekend and I picked it up. Here’s the software interface back in my studio. Look at that marvelous interface! None of that liquid glass bollocks. The first scans were promising, but I had the sense things needed some TLC. The first thing was calibrating the focus, which the software can do in combination with a focus slide. I was lucky that the focus slide was included with the scanner and I’m not sure what I would have done otherwise. Probably paid a fortune for a replacement? Possibly a lot of manual trial and error with the software? After doing that I scanned images of the 1951 USAF resolution test chart (taken on ultra high resolution 35mm film): That’s what the resulting scan looked like. Notice that it’s sharp from edge to edge, corner to corner. At 100% crop we can resolve around 110 to 123 line pairs per mm, which equates to about 5,600 to 6,300 DPI. This is beyond the limit of most 35mm lenses, but importantly - exactly to spec for this scanner. So I was happy the focus was calibrated. If you’re curious this is the same target with the camera scanning setup. It’s close, but we’ve got another variable in the workflow, several even, and that impacts the results. It’s not as sharp, and the extra glass in the transmission path causes aberrations. Another thing that needed attention was the power supply. The seller mentioned that “sometimes it takes five minutes to warm up”. Sometimes it was more than five minutes, and the power supply would click click click away. So that needed fixing and it was easy enough to find a compatible new replacement, however it cost 200 Euros. Expensive! The third problem I noticed was that some of the scans were coming out stretched. Often about 10% too wide/long, sometimes more than that. My panoramics looked panoooooooramic. I did some research and someone suggested this might be a “buffering issue”, which I thought was nonsense. Doing some testing I heard slipping sounds when the scanner was pulling the film into the body. After more research I stumbled on a post that suggested the belts need replacing. I opened the scanner up, and sure enough: A ha! You can’t quite see that the one on the back is even worse. I replaced those with compatible belts: 535 synchroflex t 2.5/245. Problem solved. The fourth problem was that the film holders were old and/or had been mishandled. They were falling apart and held together with electrical tape or glue, which didn’t seem optimal. Replacements cost 350 Euros in total for the four I needed. They’re now available cheaper from China, since the patents have expired. Or, you know, China. They used to cost about 200 Euros each from Hasselblad. The fifth problem, which is a potential one and hasn’t manifested yet, is that the lamps may eventually need replacing. I picked up a couple for 25 Euros. That seemed like a reasonable thing to do while they’re still available. Success? Let’s add up the costs of acquiring this scanner and renovating it: Scanner: 1,750.- CHF Power Supply: 175.- CHF Belts: 25.- CHF Film Holders: 350.- CHF Lamps: 25.- CHF Total: 2,325.- CHF (c. 2,500 EUR) In the last year (since acquiring the scanner) I have scanned: c. 250 panoramics frames (~ 6,000 CHF) c. 2,500 medium format frames (~ 80,000 CHF) c. 200 large format frames (~ 9,000 CHF) The figures in parentheses are what it would have cost me to have that number of frames scanned by a third party. That is, er, quite a saving. Also quite a lucrative business model perhaps? I think I can argue the cost of the scanner was a very good investment, and I haven’t finished using it yet. Even if it were to stop working tomorrow, it has already paid for itself many times over. Could it stop working tomorrow? Yes, because of other issues that will be harder to solve. The Bigger Issue(s)? A Power Mac G4 (discontinued in 2004). This came with the scanner, the necessary hardware and software to drive it, and is almost certainly living on borrowed time. Spinning metal is never good in the long-term. I’ll maybe purchase a backup soon, as these can still be found for a couple of hundred Euros. The key thing though, is that this very expensive, very high quality scanner, will at some point be rendered useless by the upgrade treadmill because the software required to run it will be increasingly difficult to run. A scanner that is still used by businesses, educational institutions, and individuals like me. A scanner that originally cost tens of thousands of Euros less than a decade ago. The upgrade treadmill is constantly whirring away. This is from the top of the Seattle Space Needle. “Do not upgrade anything on computer”. Clearly that notice speaks of someone being bitten by an upgrade at some point. I wonder is anyone else feeling the fatigue? Security updates, sure I can understand. But feature creep and trivialities? No! What tangible benefits have the last ten, fifteen, or even twenty years of OS updates brought? Other than security, and compatibility with newer hardware? New hardware is great, really, but by association forced deprecation of older hardware. No! It feels like the upgrade treadmill gets faster and steeper every year. Add to that subscription lock-in and dead endpoints: “I couldn’t vacuum my house because an SSL cert had expired” is what someone told me earlier this year. Fortunately this person is a software engineer so ended up man-in-the-middling the network traffic to get the vacuum cleaner to work again (no SSL-pinning it seems). “GoPro is announcing the end of life of the GoPro Quik app for macOS, effective at the end of 2024”. They discontinued the former in favour of their mobile app, which requires an account, login, subscription, and so on. I just want to transfer the videos from the hardware, I don’t need any of this crap (I don’t need any of that crap, it turns out GoPro haven’t locked the device down enough to prevent using third party apps to access the files. Yet). And, of course, software has to be in everything. These days the scanner would/could have an embedded Raspberry PI? Just a keyboard and mouse input, monitor and USB output would reduce the surface area, connectivity issues, and software dependency. Or software is never done? Because: externalities. I guess software is “done” when it’s no longer supported? Marciano Planque has a good piece on this: When hardware products reach end-of-life (EOL), companies should be forced to open-source the software. I think that’s a fair thing to say. I suspect Hasselblad/Imacon never open-sourced the software due to licensing issues. Or they just lost the source. Or they just don’t care, I don’t know. Maybe some combination of the three. And, inevitably, discontinued hardware like this scanner. Or, that is to say, discontinued parts? What about regulation changes? The panoramics I shoot are with a camera that was discontinued in 2004 because EU regulation banned lead solder in circuit boards. The company decided redesigning the parts wasn’t worth it. Old hardware has new exciting ways to fail. As time goes on components will fail or loosen - components that were expected to last decades. Then that results in tribal knowledge, or worse link rot and QR code rot. A lot of this stuff is hidden in walled gardens. There’s a Facebook Imacon group, for example. Why in the ever-loving fuck is a group for technical people, by technical people, on Facebook? Then there’s misleading AI. “My flextight scans are coming out stretched, what might the problem be?” LLM’s have gobbled up all the right information, and all the wrong information. Or information that is massively out of date. Nowhere in the suggestions here does it mention the belts might need replacing, which, according to my own research, is the most common reason these days. Legacy Software A decade ago I wrote an essay that also hit the front page of HN: All Software is Legacy. I think it is still relevant today, some parts not so much given we are now in The Age of Prompt, but mostly it’s still true. Nicholas always said “legacy software is the ugly stuff that makes you money”, which I think is true. But now it’s the stuff that surrounds us, like when I want to withdraw cash (guess what software most cash machines are still running?). Or when I want to take a train - when I gave this talk in Germany I had to get from the airport to the city centre. The ticket machines were disabled with a sign saying “no longer in use, download the app”. Then register. Then buy the ticket. I just want to give you money. Or when I wanted to pay for parking while stopping off at some random town in the UK - the same situation as with the ticket machines. “Download the app, register, pay”. Fuck that, I went and parked somewhere else. I just want to park, I don’t want to fight with software. Or if I want to hire a bike (not pictured: the half dozen apps on my phone to hire a bike). And when I want to buy stuff from a shop… One of the self-checkouts crashed recently in the coop, rebooting into a version of SUSE Linux from well over a decade ago. We’re collectively creating more and more of this everyday, letting it out into the world where it becomes a future liability for someone or the death knell for something. A pile of bikes, an unplugged ticket machine, a top of the line but no longer driveable scanner. References Imacon Users Group (the non-Facebook group) The state of Hasselblad Flextight scanners (2019) 1951 USAF resolution test chart Vlads Test Target Printer Story Original Scanner Blog Responses to HN Camera Scanning All Software is Legacy Repair Cafe
The Tetris effect is one of psychology’s most easy to reproduce experiments. Simply spend a bit of time playing the eponymous game every day for a few weeks. After a little while, you’ll start recognizing familiar Tetromino shapes in clouds, buildings, and everyday objects. You might even see them appear before your eyes when you start falling asleep. Tom Tang Attention hijacking There’s one lesson the Tetris effect teaches us: whatever you focus on long enough will end up shaping your thoughts. This can be a good thing since it’s how we learn new skills and discover new ideas. Sadly, less and less of our attention is focused intentionally. Instead of picking what we want to see we let other people decide what is supposed to be good for us. Do you want to watch a video? YouTube knows you like cooking and art streams. But why not also recommend a few clips about the stock market bubble, global warming, and the war in Iran. Doomscrolling will make you stay longer and click on a few more ads. Do you want to listen to music? Just open a Spotify playlist and let the algorithm figure out what you like. Please ignore the AI slop they will insert in between real songs to avoid paying royalties to real artists. Do you want to know how your colleagues are doing? Too bad, LinkedIn will bury any relevant career news between the opinion of complete strangers. It is surely just a coincidence that those strangers happen to be shilling whatever Microsoft is invested in at the moment. Do you want the opinion of strangers on a product? Well those Redditors you wanted to ask are probably just a bunch of LLMs talking to a bunch of Russian trolls now. I hope you didn’t value their opinion too much. If, like me and most people, you spend the major part of your day focused on your device, there’s no doubt it’s affecting you. And when you let someone else dictate what appears on your screen, it’s the same as giving them the key to your brain. New York Said Back to an intentional internet The internet wasn’t always like that. Before recommendation algorithms where a thing, you had to decide what you would be doing on the computer. You didn’t really have one big app that you could open and order it to entertain you. Instead, you had a few dozen of bookmarks to websites, each with a specific idea in mind. A site for video game news, that one website with lots of tutorials, a blog about anime that didn’t update often enough, a wiki about a TV show from the 90s… Of course awful things existed on the web. We had Encyclopedia Dramatica and Rotten.com, but you actually had to put the effort to go there if you wanted. Nobody was going to put pictures of dead kids and far-right propaganda as a suggestion after a pancake recipe or a cat video. The good thing is that this intentional internet is still around. It has just been a bit buried below the corporate web, but it’s not very hard to find. After all you’re on this blog, so you probably already have a good idea about it. The main difference between this time and now is you. When you want to get back to reading blogs, RSS feeds, and finish that tutorial instead of doomscrolling shorts, you have to get used to a slower internet. One where content is not infinite and doesn’t get updated every click. But like every habit, the only thing you have to do is to keep at it. And if you pay enough attention to it, something will click in your brain.
