More from wingolog
Good evening, friends. Tonight I have a few loosely-knit stories. A couple years ago, my house was heated by a . It was awful from both an environmental and a geopolitical perspective: environmental, as I would emit somewhere around 2.5 tons of CO2 equivalent per year to heat my home, which compares poorly to the target total CO2e emissions of 2 tons per year per person; and geopolitical, because although France gets 40% of its gas from Norway, with whom we have no beef, all the rest is a problem in some way. (Algeria, 10%, is the least of my worries; the 20% for Russia and the US respectively are the most, followed by 10% for the Gulf states.)condensing gas boiler Still, natural gas is better than fuel oil, which we had at my former rental house. It is a lamentably visceral experience to call up the fuel provider and say, yes, , can you drive a diesel-powered tanker truck out to my house, unroll the hose, and pour out 1500 liters of toxic fuel oil into a tank under my garden. Yes, I will just burn it all. Sure, see you again next year.s’il vous plaît Some friends of mine recently had their fuel boiler die, which is itself an experience: one of them came over to visit, completely covered in soot, saying that the chimneysweep (whom he also has to call every year) said that his boiler is on its way out, that the chimney is completely clogged, and now because of the cleaning his basement is also covered in soot; awful. What to replace it with? Apparently despite the prohibition on new fuel-oil boiler installs, it might be possible to just install a new one; or they could hook up to natural gas from the street; or they could install a heat pump. Which to do? To all these questions there is a moral answer, which we can phrase in terms in CO2 emissions and localized PM2.5 pollution, and it is always and everywhere to stop burning things. But fortunately we don’t need to rely only on moralism: electrification is just better, in essentially all ways. Owning and operating an electric car is a better experience than a petrol car. Induction stoves are better than gas; I know, I did not believe this for the longest time, but I was wrong. The experience of using a heat pump is pretty much equivalent to gas, so it’s a harder sell, but it is a relief to no longer have a pressurized methane tube connected to my house. In the end, I think my neighbors are going to go for the heat pump, despite the 20k€ price tag, labor included. (Oddly, I think the deciding factor was that my neighbor confessed to having had a long chat with an AI chatbot, after which she felt she had a good understanding of the proposed solution and its tradeoffs; make of that what you will!) In late November I got some brave lads to install nineteen solar panels on my roof. Each of these magic rectangles can make up to 500W of power in optimal conditions, but my house faces south, with the roof inclined east and west, so it’s unlikely that I will ever hit the full 9.5 kW of potential power. December was... very dark. The panels produced a total of 145 kWh over the month, but I used 1250 kWh of electricity, essentially all to run the heat pump. I live in a basin that is mostly covered by low clouds from November to February, and slanty photons couldn’t make much headway through the fog. The house is well-insulated (20-25 cm of wood-fiber exterior insulation on sides, 40 under the roof, though it is an old house with a few less-insulated bits), so it’s not that I am leaking lots of heat, and I have a combination of low-temperature floor heating and low-temperature radiators, so it’s not that I’m running the heat pump inefficiently to generate a too-high output temperature; it’s just, you know, cold in winter. A typical day would be between 1 and 5 degrees C. Cold; cold and dark. Things got a little better in January: 285 kWh produced, though the heating needs are higher than in December, with 1450 kWh total consumed. In February we grew to 419 kWh produced, for 850 kWh consumed. In March we equalized, with about 850 kWh produced and consumed, but although the bulk of my consumption in this month is for heating, the “need” to heat overnight meant that I consume from the grid overnight, but feed in to the grid during the day. I have a small battery (7 kWh), but it’s not enough to store the “excess” electricity generated in a day; I should probably arrange to have the system heat only during the day in these months, to avoid taking from the grid. With practically no heating needs now, as you can imagine, I am just feeding a lot of excess to the grid. We’re halfway through May, just coming through a cold snap (the peasant lore is that we just passed the , the date you need to wait for to plant crops that aren’t frost-hardy), but still we’ve produced more than twice as much as we’ve consumed (550 kWh vs 220 kWh), and essentially all the excess goes to the grid. The 7 kWh battery is quite enough to cover night-time electricity needs.saints de glace I didn’t know before, but often a solar panel installation doesn’t work when the grid is down. This is because the inverters that convert the DC from the panels to AC for the house need to match phase with the grid, and if the grid’s phase signal is down, they stop. It’s also for safety, so that line workers can repair downed lines without worrying that every house is a live wire. I spent a little extra to install a that allows the house to run in “island mode” if the grid is down. We almost never have that situation here, though, but it seemed prudent that if we were going all-in on electricity, that perhaps we should take precautions.cutout When you buy a solar installation, you can either have little DC/AC inverters attached to the back of each panel (), or feed DC from all panels wired in series (they call them ; there may be 2 or 3 of them in a home setup) to a central inverter. I have the latter. The panels happen to be assembled locally by , though surely the cells themselves are from China. My panels are installed on top of the ceramic roof tiles with little clips and an aluminum structure. (It used to be that sometimes panels would replace tiles and become the roof. That’s not done so much any more here.) Installation is, like, 60% of the price of solar. Often you need scaffolding, though my installers just used ladders; perhaps living in the mountains where I am, there are more people used to doing ropes and rock-climbing and such. I don’t think they took as much care of themselves as they should, though.microinvertersstringsMaviWatt My inverter is made by Huawei (SUN2000), as is my battery and the cutout (“backup”) box. Some batteries have their own microinverter, allowing them to consume and produce AC, but this one is DC, hence the need to have the same brand as the inverter. It sends all my electricity usage data to China or something, so that it can send it to the app on my phone. It’s not ideal from an geopolitical perspective but it is good kit. Although we haven’t hit the height of summer yet, I would like to offer a few observations that have precipitated out of solution. Firstly, at least in my house, the baseline load without heating is pretty low: 200 or 300 watts or so. (I didn’t know this before looking at Huawei’s app.) We have a recently renovated, not tiny, but otherwise normal sort of house with, you know, the usual lot of modern conveniences, idle chargers plugged in here and there, and also my work computers and such, and it all runs on less than a handful of the old 60W bulbs. That’s interesting. As far as actual load, there are only a few things that count: heating, when it’s cold; it can easily average 2 kW on a cold day. Plug in the electric car (I don’t have a wall box yet, just with the mains plug), that’s another kilowatt. I hardly drive, though, so it’s not a huge load. Using hot water is perhaps the most surprising thing: it can cause a spike up to 6 kW, over a short time, despite the heat coming from the heat pump; probably there is some tuning to do there. The oven and stove are little tiny blips. There’s the kettle, but it’s also a little blip. : not the dishwasher, not the washing machine, nothing. You can leave the lights on all day and it just doesn’t matter.Nothing else matters Call me naïve, but I had hoped that solar would help my electricity usage in winter. This is simply not the case. Though the heat pump is efficient, there does not appear to be a magical energy solution for December, which is the bulk of my energy usage. My electricity bill is fixed-rate: 20 cents per kWh used. Using 4000 kWh or so from the grid over winter costs me 800€; annoying. I don’t have a natural before-and-after experiment as we added on to the house as we were renovating, but for context, in my previous poorly-insulated rental house that was half the size of this one, we’d pay 2000€ or so per year for heating oil. Perhaps I can lower the 800€ via variable-rate metering, to let the battery do some arbitrage, but there are some fundamental constraints that can’t be finagled away. When I got my solar panels, I was resigned to never getting peak power, as they are on two different sections of the roof. It turns out that doesn’t matter: firstly, because 9.5 kW is a lot of power, as you can appreciate from the numbers above. I could never do anything with 9 kW. But secondly, because power isn’t equally valuable at different times of the day: by having east and west roof pitches, I can start producing earlier and continue producing later than if I had, say, a flat roof with panels tilted to the south. And the morning and the evening are the peak hours both for my house and for the grid, so that lets me consume more of my local production both when I need it and when the grid is under higher stress. I was interested to hear that Alec Watson of had reservations about residential rooftop solar. I found a , which has a delightfully socialist character. His beef is partly due to the scheme in some parts of the US, in which each kWh fed to the grid makes your meter run backwards; Watson finds it unfair, because it lets those wealthy households who have the capital to install solar to opt out of paying for the grid, which is a social good. In some cases, these households actually capture a part of what consumers pay for the grid, unlike industrial producers who are paid wholesale rates that don’t include transmission. Also, he finds it less efficient overall to install solar panels on houses rather than in bigger solar parks; each euro that society allocates to solar would go farther if we pooled them together.Technology Connectionsvideo in which he explains his perspectivenet metering Both points are interesting, but I would offer a couple responses. Firstly, at least in Europe, net metering is not really a thing; we have smart meters and I hear from friends in Portugal that there can even be a charge for grid injection at some times, if the grid is overloaded. France’s case is a bit weirder; I wouldn’t have gotten as large a system as I did, but there was a government program to offer a fixed buyback rate of 7 cents per kWh, stable for 20 years, if you installed more than 9 kW of panels. But given the lack of solar in December, I still pay the grid when I need energy the most. Putting solar panels on roofs is indeed less efficient than putting them on a field. But, we are not in a situation of scarce solar panels: . An incentive like the 7-cent buyback rate encourages capital allocation to solar, effectively calling these panels into existence. The bank loans me 20k€ at 4%, and the elimination of 3000 kWh that I would have bought from the grid in a year plus the 9000 kWh that I sell to the grid covers the cost entirely, and I get a life insurance policy on the remaining principal. It’s not a investment financially but it doesn’t cost me anything either.China could make another 350 GW of panels this year if there were demandgreat As a person with a conscience, I have always experienced questions of energy as questions of sin; to leave a light on is not simply inefficient but a moral failing. Each kilometer a car travels on fossil fuel carries with it a quantum of guilt and must be justified in some way, otherwise a moral stain attaches. 8 or 9 months out of the year, I live in a world of abundance: the electrical generation capacity that I have called into existence is free, clean, and much, much more than I need. Owning and operating a car still has externalities, but the emissions and cost aspects are entirely gone. It’s a funny feeling, and disorienting.Solar panels and electrification changes all this. I grew up in the south of the US, where everyone has air conditioning. I came to see it as sinful, too; burning things and making emissions just so you could be a bit more comfortable. I haven’t lived in air conditioning since then, but it does get hot in summer, and I would be more comfortable if I could pump heat out of my house. I have excess power available right when air conditioning (or, in my case, floor cooling) is needed. On a societal level, solar plus air conditioning is going to be a key part keeping our cities liveable while we ride out higher temperatures.Now I can. It is with a sense of dissonance, then, that I have been experiencing Datacenter Discourse™: there is a lingering language of sin proceeding from an environmentalism born in penury, in a world in which every kilowatt-hour is precious and scarce. If China has unallocated capacity for another 350 GW of panels this year, why stress about a few GW of datacenters? Of course, there are many aspects to these AI datacenters, but today I am just thinking about energy. Given that each GW of datacenter places extra demand on a grid, equivalent to 3 million times my home’s baseline load, or maybe 300 thousand of its winter load, if society wants this kind of datacenter to be a thing, it needs to add that amount of clean energy to the grid, with adequate battery storage to even out supply. We should, as a society, require this via legislation, because the market seems only too happy to use natural gas or even coal if it is marginally cheaper. At least if the datacenter boom busts, we’d be left with more clean energy production. Conversely... and I don’t think I’m going too far here, but causing new fossil generation to come online in 2026, or even prolonging the life of existing generation, should result in the state confiscating all property of those responsible. (I have moderated my previous position, which was hanging.) Such people are not fit to live in society, so society should not allow them to own things. Anyway. I think that those of us that wish “AI” were not a thing are losing the battle, and that we should prepare to fall back to more defensible positions; otherwise we risk a rout. A requirement to bring additional clean capacity online in sufficient amounts should be a baseline ask when it comes to datacenters. We have the productive capacity in the form of solar panels, at an affordable price, more than enough space in terms of the existing cropland that is inefficiently turned into ethanol to burn, batteries are a thing, and we just lack the political will to turn what could be into what is. And as for AI datacenters themselves: there are enough aspects to argue about as it is. We do ourselves a disservice by weighing down the Discourse with outdated ideas of what is and isn’t possible. soot solar sedimentation sin ‘centers
Over on his excellent blog, from having ported a bytecode virtual machine to . He finds that his tail-calling interpreter written in Rust beats his switch-based interpreter, and even beats hand-coded assembly on some platforms.Matt Keeter posts some resultstail-calling style He also compares tail-calling versus switch-based interpreters on WebAssembly, and concludes that performance of tail-calling interpreters in Wasm is terrible: In this article, I would like to argue the opposite: patterns that generate good assembly map just fine to the Wasm stack machine, and the underperformance of V8, SpiderMonkey, and Wasmtime is an accident. I re-ran Matt’s experiment locally on my x86-64 machine (AMD Ryzen Threadripper PRO 5955WX). I tested three toolchains: For each of these toolchains, I tested Raven as implemented in Rust in both “switch-based” and “tail-calling” modes. Additionally, Matt has a Raven implementation written directly in assembly; I test this as well, for the native toolchain. All results use nightly/git toolchains from 7 April 2026. My results confirm Matt’s for the native and wasmtime toolchains, but wastrel puts them in context: We can read this chart from left to right: a switch-based interpreter written in Rust is 1.5× slower than a tail-calling interpreter, and the tail-calling interpreter just about reaches the speed of hand-written assembler. (Testing on AArch64, Matt even sees the tail-calling interpreter beating his hand-written assembler.) Then moving to WebAssembly run using Wasmtime, we see that Wasmtime takes 4.3× as much time to run the switch-based interpreter, compared to the fastest run from the hand-written assembler, and worse, actually shows 6.5× overhead for the tail-calling interpreter. Hence Matt’s conclusions: there must be something wrong with WebAssembly. But if we compare to , we see a different story: Wastrel runs the basic interpreter with 2.4× overhead, and the tail-calling interpreter improves on this marginally with a 2.3x overhead. Now, granted, two-point-whatever-x is not one; Matt’s Raven VM still runs slower in Wasm than when compiled natively. Still, a tail-calling interpreter is inherently a pretty good idea.Wastrel When I think about it, there’s no reason that the switch-based interpreter should be slower when compiled via Wastrel than when compiled via . Memory accesses via Wasm should actually be cheaper due to 32-bit pointers, and all the rest of it should be pretty much the same. I looked at the assembly that Wastrel produces and I see most of the patterns that I would expect.rustc I do see, however, that Wastrel repeatedly reloads a value, containing the address (and size) of main memory. I need to figure out a way to keep this value in registers. I don’t know what’s up with the other Wasm implementations here; for Wastrel, I get 98% of time spent in the single interpreter function, and surely this is bread-and-butter for an optimizing compiler such as Cranelift. I tried pre-compilation in Wasmtime but it didn’t help. It could be that there is a different Wasmtime configuration that allows for higher performance.struct memory Things are more nuanced for the tail-calling VM. When compiling natively, Matt is careful to use a calling convention for the opcode-implementing functions, which allows LLVM to allocate more registers to function parameters; this is just as well, as it seems that his opcodes have around 9 parameters. Wastrel currently uses GCC’s default calling convention, which only has 6 registers for non-floating-point arguments on x86-64, leaving three values to be passed via global variables (described ); this obviously will be slower than the native build. Perhaps Wastrel should add the equivalent annotation to tail-calling functions.preserve_nonehere On the one hand, Cranelift (and V8) are a bit more constrained than Wastrel by their function-at-a-time compilation model that privileges latency over throughput; and as they allow Wasm modules to be instantiated at run-time, functions are effectively closures, in which the “instance” is an additional hidden dynamic parameter. On the other hand, these compilers get to choose an ABI; last I looked into it, SpiderMonkey used the equivalent of , which would allow it to allocate more registers to function parameters. But it doesn’t: you only get 6 register arguments on x86-64, and only 8 on AArch64. Something to fix, perhaps, in the Wasm engines, but also something to keep in mind when making tail-calling virtual machines: there are only so many registers available for VM state.preserve_none Well friends, you know us compiler types: we walk a line between collegial and catty. In that regard, I won’t deny that I was delighted when I saw the Wastrel numbers coming in better than Wasmtime! Of course, most of the credit goes to GCC; Wastrel is a relatively small wrapper on top. But my message is not about the relative worth of different Wasm implementations. Rather, it is that : a fast implementation of a particular algorithm is of use to everyone who uses that algorithm, whether they use that implementation or not.performance oracles are a public good This happens in two ways. Firstly, faster implementations advance the state of the art, and through competition-driven convergence will in time result in better performance for all implementations. Someone in Google will see these benchmarks, turn them into an OKR, and golf their way to a faster web and also hopefully a bonus. Secondly, there is a dialectic between the state of the art and our collective imagination of what is possible, and advancing one will eventually ratchet the other forward. We can forgive the conclusion that “patterns which generate good assembly don’t map well to the WASM stack machine” as long as Wasm implementations fall short; but having showed that good performance is possible, our toolkit of applicable patterns in source languages also expands to new horizons. Well, that is all for today. Until next time, happy hacking! 1.2× slower on Firefox, 3.7× slower on Chrome, and 4.6× slower in wasmtime. I guess patterns which generate good assembly don't map well to the WASM stack machine, and the JITs aren't smart enough to lower it to optimal machine code. some numbers where does the time go the value of time Compiled natively via / cargorustc Compiled to WebAssembly, then run with Wasmtime Compiled to WebAssembly, then run with Wastrel
Good evening. Let’s talk about free trade! Last time, , which looks at how the cause of free trade was taken up by a motley crew of anti-imperialists, internationalists, pacifists, marxists, and classical liberals in the nineteenth century. Protectionism was the prerogative of empire—only available to those with a navy—and it so it makes sense that idealists might support “peace through trade”. So how did free trade go from a cause of the “another world is possible” crowd to the halls of the WTO? Did we leftists catch a case of buyer’s remorse, or did the goods delivered simply not correspond to the order?we discussed Marc-William Palen’s Pax Economica To make an attempt at an answer, we need more history. From the acknowledgements of :Quinn Slobodian’s Globalists Slobodian’s approach is to pull on the thread that centers around the WTO itself. He ends up identifying what he calls the “Geneva School” of neoliberalism: from Mise’s circle in Vienna, to the International Chamber of Commerce in Paris, to the Hayek-inspired Mont Pèlerin Society, to Petersmann of the WTO precursor GATT organization, Röpke of the Geneva Graduate Institute of International Studies, and their lesser successors of the 1970s and 1980s. The thesis that Slobodian ends up drawing is that neoliberalism is not actually a fundamentalism, but rather an ideology that placed the value of free-flowing commerce above everything else: above democracy, above sovereignty, above peace, and that as such it actually requires active instutional design to protect commerce from the dangers of, say, hard-won gains by working people in one country (Austria, 1927), expropriation of foreign-owned plantations in favor of landless peasants (Guatemala, 1952), internal redistribution within countries transitioning out of minority rule (South Africa, 1996), decolonization (1945-1975 or so), or just the election of a moderate socialist at the ballot box (Chile, 1971).laissez-faire Now, dear reader, I admit to the conceit that if you are reading this, probably you are a leftist also, and if not, at least you are interested in understanding how it is that we think, with what baubles do we populate our mental attics, that sort of thing. Well, friend, you know that by the time we get to Chile and Allende we are stomping and clapping our hands and shouting in an extasy of indignant sectarian righteousness. And that therefore should we invoke the spectre of neoliberalism, it is with the deepest of disgust and disdain: this project and all it stands for is against me and mine. I hate it like I hated Henry Kissinger, which is to say, .a lot, viscerally, it hurts now to think of it, rest in piss you bastard And yet, I’m still left wondering what became of the odd alliance of Marx with Manchester liberalism. Palen’s continues to sketch a thin line through the twentieth century, focusing on showing the continued presence of commercial-peace exponents despite it not turning out to be our century. But the rightward turn of the main contingent of free-trade supporters is not explained. I have an idea about how it is that this happened; it is anything but scholarly, but here we go.Pax Economica Let us take out our coarsest brush to paint a crude story: the 19th century begins in the wake of the American and French revolutions, making the third estate and the bourgeoisie together the revolutionary actors of history. It was a time in which “we” could imagine organizing society in different ways, the age of the utopian imaginary, but overlaid with the structures of the old, old money, old land ownership, revanchist monarchs, old power, old empire. In this context, Cobden’s was insurgent, heterodox, asking for a specific political change with the goal of making life on earth better for the masses. Free trade was a means to an end. Not all Cobdenites had the same ends, but Marx and Manchester both did have ends, and they happened to coincide in the means.Anti-Corn Law League Come the close of the Great War in 1918, times have changed. The bourgeoisie have replaced the nobility as the incumbent power, and those erstwhile bourgeois campaigners now have to choose between idealism and their own interest. But how to choose? Some bourgeois campaigners will choose a kind of humanist notion of progress; this is the thread traced by Palen, through the , the Young Women’s Christian Association, the , and others.Carnegie Endowment for International PeaceHaslemere Group Some actors are not part of the hegemonic bourgeoisie at all, and so have other interests. The newly independent nations after decolonization have more motive to upend the system than to preserve it; their approach to free trade has both tactical and ideological components. Tactical, in the sense that they wanted access to first-world markets, but also sometimes some protections for their own industries; ideological, in the sense that they often acted in solidarity with other new nations against the dominant powers. In addition to the new nations, the Soviet bloc had its own semi-imperial project, and its own specific set of external threats; we cannot blame them for being tactical either. And then you have Ludwig von Mises. Slobodian hints at Mises’ youth in the Austro-Hungarian empire, a vast domain of many languages and peoples but united by trade and the order imposed by monarchy. After the war and the breakup of the empire, I can only imagine—and here I am imagining, this is not a well-evidenced conclusion—I imagine he felt a sense of loss. In the inter-war, he holds court as the of the Vienna Chamber of Commerce, trying to put the puzzle pieces back together, to reconstruct the total integration of imperial commerce, but from within . When in 1927, , the city went on general strike, and workers burned down the ministry of justice. Police responded violently, killing 89 people and injuring over 1000. Mises was delighted: order was restored.doyenRed Viennaa court decision acquitted a fascist milicia that fired into a crowd, killing a worker and a child And now, a parenthesis. I grew up Catholic, in a ordinary kind of way. Then in my early teens, I concluded that if faith meant anything, it has to burn with a kind of fervor; I became an evangelical Catholic, if such is a thing. There were special camps you could go to with intense emotional experiences and people singing together and all of that is God, did you know? Did you know? The feelings attenuated over time but I am a finisher, and so I got confirmed towards the end of high school. I went off to university for physics and stuff and eventually, painfully, agonizingly concluded there was no space for God in the equations. Losing God was incredibly traumatic for me. Not that I missed, like, the idea of some guy, but as someone who wants things to make sense, to have meaning, to be based on something, anything at all: losing a core value or morality invalidated so many ideas I had about the world and about myself. What is the good life, a life well led? What is true and right in a way that is not contingent on history? I am embarrassed to say that for a while I took the UN declaration of human rights to be axiomatic. When I think about Mise’s reaction to the 1927 general strike in Vienna, I think about how I scrambled to find something, anything, to replace my faith in God. As the space for God shrank with every advance in science, some chose to identify God with his works, and then to progressively ascribe divine qualities to those works: perhaps commerce is axiomatically Good, and yet ineffable, in the sense that it is Good on its own, and that no mortal act can improve upon it. How else can we interpret Hayek’s relationship with the market except as awe in the presence of the divine? This is how I have come to understand the neoliberal value system: a monotheism with mammon as godhead. There may be different schools within it, but all of the faithful worship the same when they have to choose between, say, commerce and democracy, commerce and worker’s rights, commerce and environmental regulation, commerce and taxation, commerce and opposition to apartheid. It’s a weird choice of deity. Now that God is dead, one could have chosen anything to take His place, and these guys chose the “global economy”. I would pity them if I still had a proper Christian heart. I think that neoliberals made a miscalculation when they concluded that the peace of is not predicated on justice. Sure, in the short run, you can do business with Pinochet’s Chile, privatize the national mining companies, and cut unemployment benefits, but not without incurring moral damage; people will see through it, in time, as they did in Seattle in 1999. Slobodian refers to the ratification of the WTO as a Pyrrhic victory; in their triumph, neoliberals painted a target on their backs.doux commerce Where does this leave us now? And what about Mercosur? I’m starting to feel the shape of an answer, but I’m not there yet. I think we’ll cover the gap between Seattle and the present day in a future dispatch. Until then, let’s take care of one other; as spoke the prophet Pratchett, there’s no justice, just us. This book is a long-simmering product of the Seattle protests against the World Trade Organization in 1999. I was part of a generation that came of age after the Cold War's end. We became adolescents in the midst of talk of globalization and the End of History. In the more hyperactive versions of this talk, we were made to think that nations were over and the one indisputable bond uniting humanity was the global economy. Seattle was a moment when we started to make collective sense of what was going on and take back the story line. I did not make the trip north from Portland but many of my friends and acquaintances did, painting giant fists red to strap to backpacks and coming back with takes of zip ties and pepper spray, nights in jail, and encounters with police—tales they spun into war stories and theses. This book is an apology for not being there and an attempt to rediscover in words what the concept was that they went there to fight. papier-mâché two theologies means without end
Hello friends! Today, a quick note: the ahead-of-time WebAssembly compiler now supports managed memory via garbage collection!Wastrel The quickest demo I have is that you should check out and build wastrel itself: Then run a quick check with :hello, world Now give a check to , a classic GC micro-benchmark:gcbench We set to get those last 4 lines. So, this is a microbenchmark: it runs for only 138 ms, and the heap is tiny (26.7 MB). It does collect 30 times, which is something.WASTREL_PRINT_STATS=1 I know what you are thinking: OK, it’s a microbenchmark, but can it tell us anything about how Wastrel compares to V8? Well, probably so: Which is to say, V8 takes more CPU time (230ms vs 209ms) and more wall-clock time (200ms vs 138ms). Also it uses twice as much managed memory (48 MB vs 26.7 MB), and more than that for the total process (88 MB vs 34 MB, not shown). Let’s try with , which at least has a larger active heap size. This time we’ll compile a binary and then run it:quads Compare to V8 via node: Which is to say, : 2460ms (v8) vs 849ms (wastrel), and 383MB vs 141 MB.wastrel is almost three times as fast, while using almost three times less memory So, yes, the V8 times include the time to compile the wasm module on the fly. No idea what is going on with tiering, either, but I understand that tiering up is a thing these days; this is node v22.14, released about a year ago, for what that’s worth. Also, there is a V8-specific module to do some impedance-matching with regards to strings; in Wastrel they are WTF-8 byte arrays, whereas in Node they are JS strings. But it’s not a string benchmark, so I doubt that’s a significant factor. I think the performance edge comes in having the program ahead-of-time: you can statically allocate type checks, statically allocate object shapes, and the compiler can see through it all. But I don’t really know yet, as I just got everything working this week. Wastrel with GC is demo-quality, thus far. If you’re interested in the back-story and the making-of, see article from October, or the FOSDEM talk from last week:my intro to Wastrel Slides , if that’s your thing.here More to share on this next week, but for now I just wanted to get the word out. Happy hacking and have a nice weekend! hello, world is it good? improving on v8, really? zowee! git clone https://codeberg.org/andywingo/wastrel cd wastrel guix shell # alternately: sudo apt install guile-3.0 guile-3.0-dev \ # pkg-config gcc automake autoconf make autoreconf -vif && ./configure make -j $ ./pre-inst-env wastrel examples/simple-string.wat Hello, world! $ WASTREL_PRINT_STATS=1 ./pre-inst-env wastrel examples/gcbench.wat Garbage Collector Test Creating long-lived binary tree of depth 16 Creating a long-lived array of 500000 doubles Creating 33824 trees of depth 4 Top-down construction: 10.189 msec Bottom-up construction: 8.629 msec Creating 8256 trees of depth 6 Top-down construction: 8.075 msec Bottom-up construction: 8.754 msec Creating 2052 trees of depth 8 Top-down construction: 7.980 msec Bottom-up construction: 8.030 msec Creating 512 trees of depth 10 Top-down construction: 7.719 msec Bottom-up construction: 9.631 msec Creating 128 trees of depth 12 Top-down construction: 11.084 msec Bottom-up construction: 9.315 msec Creating 32 trees of depth 14 Top-down construction: 9.023 msec Bottom-up construction: 20.670 msec Creating 8 trees of depth 16 Top-down construction: 9.212 msec Bottom-up construction: 9.002 msec Completed 32 major collections (0 minor). 138.673 ms total time (12.603 stopped); 209.372 ms CPU time (83.327 stopped). 0.368 ms median pause time, 0.512 p95, 0.800 max. Heap size is 26.739 MB (max 26.739 MB); peak live data 5.548 MB. $ guix shell node time -- \ time node js-runtime/run.js -- \ js-runtime/wtf8.wasm examples/gcbench.wasm Garbage Collector Test [... some output elided ...] total_heap_size: 48082944 [...] 0.23user 0.03system 0:00.20elapsed 128%CPU (0avgtext+0avgdata 87844maxresident)k 0inputs+0outputs (0major+13325minor)pagefaults 0swaps $ ./pre-inst-env wastrel compile -o quads examples/quads.wat $ WASTREL_PRINT_STATS=1 guix shell time -- time ./quads Making quad tree of depth 10 (1398101 nodes). construction: 23.274 msec Allocating garbage tree of depth 9 (349525 nodes), 60 times, validating live tree each time. allocation loop: 826.310 msec quads test: 860.018 msec Completed 26 major collections (0 minor). 848.825 ms total time (85.533 stopped); 1349.199 ms CPU time (585.936 stopped). 3.456 ms median pause time, 3.840 p95, 5.888 max. Heap size is 133.333 MB (max 133.333 MB); peak live data 82.416 MB. 1.35user 0.01system 0:00.86elapsed 157%CPU (0avgtext+0avgdata 141496maxresident)k 0inputs+0outputs (0major+231minor)pagefaults 0swaps $ guix shell node time -- time node js-runtime/run.js -- js-runtime/wtf8.wasm examples/quads.wasm Making quad tree of depth 10 (1398101 nodes). construction: 64.524 msec Allocating garbage tree of depth 9 (349525 nodes), 60 times, validating live tree each time. allocation loop: 2288.092 msec quads test: 2394.361 msec total_heap_size: 156798976 [...] 3.74user 0.24system 0:02.46elapsed 161%CPU (0avgtext+0avgdata 382992maxresident)k 0inputs+0outputs (0major+87866minor)pagefaults 0swaps
Hey hey happy new year, friends! Today I was going over some V8 code that touched : allocating objects directly in the old space instead of the nursery. I knew the theory here but I had never looked into the mechanism. Today’s post is a quick overview of how it’s done.pre-tenuring In a JavaScript program, there are a number of source code locations that allocate. Statistically speaking, any given allocation is likely to be short-lived, so generational garbage collection partitions freshly-allocated objects into their own space. In that way, when the system runs out of memory, it can preferentially reclaim memory from the nursery space instead of groveling over the whole heap. But you know what they say: there are lies, damn lies, and statistics. Some programs are outliers, allocating objects in such a way that they don’t die young, or at least not young enough. In those cases, allocating into the nursery is just overhead, because minor collection won’t reclaim much memory (because too many objects survive), and because of useless copying as the object is scavenged within the nursery or promoted into the old generation. It would have been better to eagerly tenure such allocations into the old generation in the first place. (The more I think about it, the funnier is as a term; what if some PhD programs could pre-allocate their graduates into named chairs? Is going straight to industry the equivalent of dying young? Does collaborating on a paper with a full professor imply a write barrier? But I digress.)pre-tenuring Among the set of allocation sites in a program, a subset should pre-tenure their objects. How can we know which ones? There is a literature of static techniques, but this is JavaScript, so the answer in general is dynamic: we should observe how many objects survive collection, organized by allocation site, then optimize to assume that the future will be like the past, falling back to a general path if the assumptions fail to hold. The high-level overview of how V8 implements pre-tenuring is based on per-program-point objects, and per-allocation objects that point back to their corresponding AllocationSite. Initially, V8 doesn’t know what program points would profit from pre-tenuring, and instead allocates everything in the nursery. Here’s a quick picture:AllocationSiteAllocationMemento Here we show that there are two allocation sites, and . V8 is currently allocating into a linear allocation buffer (LAB) in the nursery, and has allocated three objects. After each of these objects is an ; in this example, and are objects that point to and points to . When V8 allocates an object, it (if available; it’s possible an allocation comes from C++ or something where we don’t have an ).Site1Site2AllocationMementoM1M3AllocationMementoSite1M2Site2AllocationSiteincrements the “created” counter on the corresponding AllocationSite When the free space in the LAB is too small for an allocation, V8 gets another LAB, or collects if there are no more LABs in the nursery. When V8 does a minor collection, as the scavenger visits objects, it will . If so, it dereferences the memento to find the , then increments its “found” counter, and adds the to a set. , it is enqueued for a pre-tenuring decision; get marked for pre-tenuring.look to see if the object is followed by an AllocationMementoOnce an AllocationSite has had 100 allocationssites with 85% survivalAllocationSiteAllocationSite If an allocation site is marked as needing pre-tenuring, the code in which it is embedded it will get de-optimized, and then next time it is optimized, the code generator arranges to allocate into the old generation instead of the default nursery. Finally, if a major collection collects more than 90% of the old generation, V8 , under the assumption that pre-tenuring was actually premature.resets all pre-tenured allocation sites What kinds of allocation sites are eligible for pre-tenuring? Sometimes it depends on object kind; wasm memories, for example, are almost always long-lived, so they are always pre-tenured. Sometimes it depends on who is doing the allocation; allocations from the bootstrapper, literals allocated by the parser, and many allocations from C++ go straight to the old generation. And sometimes the compiler has enough information to determine that pre-tenuring might be a good idea, as when it .generates a store of a fresh object to a field in an known-old object But otherwise I thought that the whole AllocationSite mechanism would apply generally, to any object creation. It turns out, nope: it seems to only apply to object literals, array literals, and . Weird, right? I guess it makes sense in that these are the ways to create objects that also creates the field values at creation-time, allowing the whole block to be allocated to the same space. If instead you make a pre-tenured object and then initialize it via a sequence of stores, this would likely create old-to-new edges, preventing the new objects from dying young while incurring the penalty of copying and write barriers. Still, I think there is probably some juice to squeeze here for pre-tenuring of class-style allocations, at least in the optimizing compiler or in short inline caches.new Array I suspect this state of affairs is somewhat historical, as the AllocationSite mechanism seems to have originated with and V8’s “boilerplate” object literal allocators; both of these predate per-AllocationSite pre-tenuring decisions.typed array storage strategies Well that’s adaptive pre-tenuring in V8! I thought the “just stick a memento after the object” approach is pleasantly simple, and if you are only bumping creation counters from baseline compilation tiers, it likely amortizes out to a win. But does the restricted application to literals point to a fundamental constraint, or is it just accident? If you have any insight, let me know :) Until then, happy hacking! allocation sites my runtime doth object tenure for me but not for thee fin A linear allocation buffer containing objects allocated with allocation mementos
More in programming
I have a lot of digital cross-stitch patterns and until recently they were a mess. My “collection” was a series of several hundred PDFs, disorganised and spread across my disk. Recently, I was looking for a pattern and it took me several minutes to find it. I had a brief moment of panic, thinking that I’d lost it, and the website this pattern came from is long since gone. Fortunately it did turn up, but it was clear I had to organise my pattern collection. Some people would use apps like Pattern Keeper or Flosshaven, but I just created another static website on my media drive. This is what it looks like: The patterns are arranged in a masonry layout with a preview thumbnail for each pattern, which is normally the first page of the PDF. If a pattern has multiple files – for example, different files for colour and black-and-white printing – there’s a list of links below the thumbnail. I can also add a source link to where I bought the pattern, if I want to find more patterns by the same author. Internally, it’s structured like my other static websites. The metadata for the patterns is stored in a JavaScript file, and then I have an HTML page which renders that metadata in this visual design. My pattern collection doesn’t change very often, so I’m adding new files by hand. There are no sorting or filtering options, because my collection is small enough that they’re unnecessary. I can scroll through the entire list in less than a minute and find whatever I need. I’d like to start making my own clothes soon, and I expect to copy this design to store clothing patterns. This collection now sits alongside my saved bookmarks, podcasts, TV shows, and more. I can keep all my patterns organised, and I won’t lose them in the depths of my hard drive. Technical tricks There are a couple of technical pieces I’m quite pleased with in this process: Masonry layout. I love the cascading grid style of a masonry layout, and I think they’re great when you have variable-sized content – like my pattern cards. I’ve been using them for over a decade; screenshots of my early web apps show my experiments with David Desandro’s Masonry.js library. Masonry layouts are now a native feature of CSS called “Grid Lanes”, which is simpler and more performant than the JavaScript approach. I’m excited to use these layouts in even more places. Thumbnails of PDFs. Most digital postage patterns are shipped as PDFs, and I’m creating the thumbnail images with my create_thumbnail tool. I had to extend my tool to support PDFs, which turned out to be straightforward with the pdf_oxide crate. I’d not come across Oxide before, but it’s a cross-language library for working with PDFs, and I can imagine using it in future. A coloured stripe for the palette. In the metadata for each project, I have the complete list of DMC thread colours and the pattern dimensions. I have a lookup table that converts DMC thread colours to hex codes, and then I’m able to create a linear gradient in CSS to render a coloured stripe of the pattern colours. Here’s an example from a pattern with four colours: border-bottom: 12px solid transparent; border-image: linear-gradient( to right, #7a451f 0%, #7a451f 25%, #985e33 25%, #985e33 50%, #e4bb8e 50%, #e4bb8e 75%, #361f0e 75%, #361f0e 100% ) 1; Building the HTML, CSS, and JavaScript was very quick – I have notes on how I build my media sites, and I’ve done it plenty of times before. I threw together the initial prototype in less than 15 minutes. The longer part was finding all the PDFs scattered across my disk – but now that’s done, and I know exactly where to find my patterns. [If the formatting of this post looks odd in your feed reader, visit the original article]
Download Now View on pouet.net Intro In the late 80s, I spent an embarrassing number of hours hunched over my ZX Spectrum, typing in Sinclair BASIC programs and rudimentary assembler routines copied from magazines, trying to re-create the Amiga crack intros I’d seen round at a friend’s house. Obviously, they never came even slightly close - but now, many decades later, I’ve finally managed it! I’m now the proud author of a hand-rolled retro demoscene intro that runs on a stock 128Kb Spectrum. It features all those late 80s tropes: A smooth scrolltext (with ‘leet custom fonts!), a three-channel AY soundtrack, bouncing VU meters and the essential parallax starfield. It’s currently 10287 bytes of code, or 19552 bytes as a complete tape image - which, given I’ve been working on this since December 13th last year, means my average rate of code-production is a paltry ~ 455 bytes per week! This blog post is a technical overview of how I got there, some cool tricks I picked up, and everything I had to learn (and re-learn) about the Speccy and Z80 assembly along the way. It simultaneously rates as one of my favourite side-projects ever, and also the one that most often nearly triggered a foot put through my monitor in frustration. I’ll reflect on all that at the end, but for now if you just want to see the finished product, the download button is at the top of the article and the included .tap cassette tape image should load into any emulator. There’s also a YouTube capture, links to the production on pouet.net, and you can run it from within a web browser by visiting my TNFS site portal - check option 4 for recent uploads. Be Kind, Rewind But let’s skip back briefly to the late 80s. Coming from the world of an 8-bit, cassette tape loading computer, seeing an Amiga for the first time was like encountering something from outer space. Like everyone who saw one back in the day, I was blown away by the amazing graphics and sound on display, not to mention the library of ground-breaking games. But I found myself even more captivated by the mysterious (and totally badass) “intros” displayed at the beginning of pirated games. Created by the software piracy groups that had cracked the copy protection and distributed the floppy-disk based games, they were mini works of art in their own right - music, graphics and usually some cool-looking effect to show off the group’s prowess. There’s a great collection of these intros (or “cracktros”) archived on YouTube. Watch a few of those videos and you’ll quickly notice the “design” tropes that almost every cracktro shares: a cool, edgy logo at the top of the screen (bonus points if it’s dripping blood or in any way “metal”), some boastful text displayed on top of an effect of some sort - usually a scrolling parallax starfield, or rotating 3d object - and a further scroll-text full of shout outs, greetings to fellow pseudonymous crackers and adverts for illegal BBS systems galore. Needless to say, to a kid in the 80s/early 90s this was possibly the coolest thing ever. I thought this one was the absolute bomb when I first saw it. Sine-scroller, pirate BBS adverts, cool logo with a sword - and of course, a starfield. There’s something for everyone here. The music from this is burned into my brain. I think pretty much everyone of my classmates knew this intro given the popularity of Lemmings in the early 90s. Skid Row as a group are still active to this day and this is probably the quintessential Amiga cracktro of the period. Endless Piracy. Edgy. Another cool logo, sine-wave scroller, starfield and boasting text. The classics never go out of style. Another great combination of all the essential ingredients. Cool logo, even cooler text font, scrolling starfield and a scroll-text running along the bottom. Vintage stuff. Logo. Text. Scroller. But no starfield - otherwise a solid effort from Anthrox here :) The demoscene grew out of these intros. The cracking eventually faded into the background, and coders, “graphicians” and musicians all contributed to feature-length productions that showcased their abilities, competed with other groups and pushed the machine to its limits. In amazing feats of ingenuity and pure “code as an art form”, the scene lives on today, with groups producing breathtaking productions like this and this - bear in mind, both of those are running on a 7Mhz home computer equipped with a floppy disk, 1Mb of memory and a sound chip from 1985. But much as I admire the sheer skill and dedication that takes, it’s the early “single screen” scrolly-text-and-starfield cracktros I’ve always found captivating. They took on a sort of mystical quality for me, hinting at a scene and a culture I could only ever experience from the outside - which is exactly why I spent all those hours in Sinclair BASIC, trying in vain to re-create them, and dreaming of the day I’d eventually have an Amiga of my own. The Jump Back I eventually did trade up to a range of Amiga systems, joined a few “scene” groups myself and threw myself into a variety of activities until exams and the end of my teenage years beckoned. I dabbled with Z80 code back then, and at one point knew enough 68k assembly to bluff my way through - partly from the Amiga’s heyday, partly from a few “embedded systems” modules at University where I had to program a set of traffic lights or lift-controllers. Given there’s not much call for that sort of thing at my $DAYJOB, though, it had all pretty much faded from my memory. I have kept working on a bunch of projects in my spare time, and one of my favourites is my TNFS site - sort of an old-school BBS system or interactive “website” for the Spectrum, which grew out of a fun experiment to see how far I could push Sinclair BASIC. It relies on a modern network card fitted to the Spectrum, which is how I came to be chatting to “Desertkun”, another TNFS-protocol enthusiast. He’s now producing a great new WiFi network card for the Speccy, and was kind enough to send me a few of the prototypes to test out for him. I was massively inspired by his hardware and low-level coding chops, and during one of our video calls where we’d been comparing our development setups, he asked why I didn’t write any Z80 assembly myself. A throw-away comment, but the challenge was set! Now, I couldn’t face re-writing the whole site in asm, but I figured I could produce a self-contained intro to promote it in the style of the early “BBStros”, and finally be one of the cool kids with a scene production of my own :) I mean, how hard could it be ? (ooooh - foreshadowing!) Dev Environment My development environment through this was pretty straightforward, and is the same stack I use for developing my TNFS site. I just used a Makefile-driven build system which ran my code through the Pasmo assembler, my fork of zmakebas to build BASIC files, and the FUSE emulator for testing. For final testing on the real hardware, I uploaded my .tap files to my local TNFS server where I could run them on my +3. A few additional tools I have to call out because they ended up making a big difference - one is the amazing web-based “asm-editor” I discovered while looking up Z80 instructions. Not only does this site have a great reference of all the Z80 instructions, but it also has an editor where you can run example code and see how the memory and/or various registers change state. Take for example, the djnz instruction - you can build it and step through the example loop construct one instruction at a time. Or try my example of an add instruction in the screenshot to see how the carry flag works when a bit gets shifted out of a register. Although it uses a slightly different syntax to Pasmo (so I couldn’t drop my code directly in it), it massively helped when manually stepping through things. And lastly, probably the most significant debugging tools I used on this project - a notepad and pen, and my two dogs (say “Hi!” to Biscuit & Pebble…) I’ll write some more about this in the wrap-up, but this really was a highly unusual project for me, coming from decades of much higher-level languages and constructs. Most of my z80 code I only ever really understood after drawing out the bytes and bitmasks, keeping a table of register flags and drawing chunks of screen memory out by hand. And I found my evening walks with the two dogs provided ideal rubber duck exercises, although I’m not sure what they both made of my protracted mutterings about the hl register pair. Either way, debugging-by-dog is highly recommended! The Starting Point So, let’s get started. I followed a great “Hello, world!” example from futurice.com to validate my build environment. Beyond that, the first thing I wanted to do was load in my logo and background image I’d use as a “stage” for the remainder of the planned intro. I wasn’t really sure what I was going to do at this point, just that I wanted my logo at the top, and a scrolltext along the bottom row of the screen. As my TNFS site header already looked kinda demoscene-y, I made a quick edit putting in place two blue guide-bars inside which I’d put further content. Memory map courtesy of breakintoprogram.co.uk A Spectrum SCREEN$ file (usually displayed as loading screens) is just a dump of screen memory, starting from byte 16384, and running for 6912 bytes. The weird layout of this is very relevant later, but in order to display it all you need to do is load your file into that memory location. After a few hour’s work of reviewing Speccy internals and following various guides, I had a very simple starting point for my demo that served as the basis for everything else that came after. org $8000 screen EQU $4000 ; address of the first byte of screen memory, 16384 in dec scrlen EQU 6912 ; total length of the screen memory inc. attributes ula EQU $FE ; I/O port for the ULA to control the border black EQU 0 ; Spectrum colour number, the exact same ones BASIC uses start: ld a,black ; Set the border colour to black using the ULA out (ula),a ; Copy my logo header picture to screen memory ld hl,header ; HL = location of my screen file in this code ld de,screen ; DE = start of screen memory ld bc,scrlen ; BC = 6912 bytes to copy ldir ; block copy, see https://jnz.dk/z80/ldir.html ; busy loop - rest of intro goes here :P hold: halt ; sleep until the next interrupt (1/50 second) jr hold ; then jump back and sleep again, for ever header: incbin "assets/header.scr" ; my logo header/background end start Not much to look at now, but this simple starting point introduced me to important concepts like interfacing with the Spectrum’s hardware (ULA chip to start with, and AY sound chip later) via the out instruction, how to pull in binary files using Pasmo’s INCBIN directive, using the Z80’s block copy ldir instruction, the use of register pairs for 16-bit addressing, and simple branching to labels. Hello World The next step was to print something on the screen and explore some of the Speccy’s built-in ROM routines, which are somewhat analagous to syscalls on modern operating systems. There are plenty of disassemblies and guides to the Spectrum ROM available on the internet, one of the best commented and linked one is at https://skoolkid.github.io/rom/. Using this as a reference alongside the various “Hello, world!” tutorials showed me how this sort of thing is built up: at EQU 22 ; "at" control character used to position the cursor row EQU 11 ; coordinates where to print the message col EQU 9 prstring EQU $203C ; where the pr_string routine lives in ROM ld de,msg ; DE = address of the first character ld bc,msglen ; BC = how many characters to print call prstring ; print the lot ; busy loop is the same... msg: db at,row,col ; equivalent of "PRINT AT 11,9" in BASIC db "Hello, world!" msglen EQU $-msg ; 16 bytes in all, pasmo shortcut, $=program counter I ended up mostly not using the ROM routines in my code as they dragged in all sorts of things I didn’t need, but it was massively useful to understand how they worked as they introduced me to yet more commonly-used constructs. For example, the PR_STRING routine defined here shows how you can build a loop over each character. If you’re just starting out, I highly recommend going over the 8 lines in that routine with a notepad and pen, working out the contents of the registers and following the loop constructs. I also made a version of it in asm-editor you can try. If you click that link, you’ll see a version of the code loaded into the web editor. Click “Build”, and then step through the code using the “Step” button - although no text will be actually displayed (I skipped over the call to the Spectrum ROM “restart” vector that does this), you’ll see the various states of the registers and for example the BC register counting down as each byte of the string is processed. Adding Some Style Problem is, the Speccy’s built-in font is not exactly dripping with demoscene credibility. I wanted to use some stylish fonts and this turned out to be super easy - a quick search turned up Jim Blimey’s awesome font collection, which included a ready-to-use set of assembler files and some instructions which soon led me to these additions in my code: ; CHARS is the address the ROM's printing routine reads to find the font. ; Picking our own font is simply a matter of over-writing this chars EQU $5C36 ; from Jim's docs - POKE 23607=5C36 in hex ld hl,myfont-256 ; HL = font address - 256 as per docs ld (chars),hl ; from now on the ROM draws with it! ; rest of code is the same myfont: include "textfont.asm" ; "Cartoon" font from jimblimey.com There we go! Now that’s looking a little more elite. Sorry, I mean 31337. Strap in folks, we’re leaving lamer city… So at this point, a month or so in to my dabblings and I’d started to get a reasonable understanding of common idioms and was starting to feel like this was a project that might actually go somewhere. The problem started when I moved on to trying to implement actual demo effects: It became very apparent, very quickly, just how much I didn’t know about how this all worked. Not to mention the fact that at this level, you’re really closely exposed to the physical hardware intricacies, including the design decisions and trade-offs that the hardware designers had to make when working with the extremely limited specs of the early 1980s. Cursed Screen Layout The first such problem - and the cause of probably 90% of my pain - is the Speccy’s extremely weird screen layout. Rather than standard bitmapped display you might expect on more modern hardware, it is instead split into 3 sections, each with a set of interleaved rows (so if you keep adding 1 to a screen address, eventually instead of reaching the next row, you’ll instead end up 8 rows down, and so on until you hit the bottom of the section and then loop back to the row underneath where you started!) followed by a block of “attribute” bytes which determine the colours of each 8x8 block. The attribute blocks are the reason why each 8x8 character cell can only have two colours (a PAPER and INK attribute) which led to the garish “colour-clash” of spectrum games and distinctive style where artists try to keep inside 8x8 blocks as much as possible. But it was the layout of the bitmap portion that is such a mind twister. Download the video You can see this illustrated whenever a game reads in a loading screen from tape. You can see each section built up with the interleaved rows one at a time, and then finally the colour and other attributes (like flashing ). There’s a lot of detail at the fantastic Break Into Program site. In the defense of the designers, they were working with extremely limited hardware and I don’t think the Speccy was ever imagined as a games machine. When you start shuffling bytes around, you realise that this approach is really optimised for displaying text which is neatly arranged into 8x8 character boundaries. For example, using this “scrambled” bit order, if you have a screen address in the hl register pair, you can quickly get to the next pixel line of a given address by simply incrementing the “high” byte. There’s a bunch of other “cheap moves” you can make by simply shifting bits around: Move Instruction Right 8 pixels inc l Left 8 pixels dec l Down 1 pixel line inc h Up 1 pixel line dec h Down 1 character row add 32 to l It was a reasonable choice for a machine that would probably spend a lot of it’s time displaying BASIC program listings - it’s only when you start trying to write demo or sprite routines that it starts to feel really weird. This also means you’ll pretty quickly need a routine to convert X,Y addresses to the relevant memory addresses. Again, L Break Into Program has a useful routine and if you disassemble more or less any spectrum game or demo, you’ll find a version of this procedure somewhere. And yes, this all became horribly relevant later on… The Scroller Download the video Admittedly, not the most interesting demonstration - but this was the first time I had made something move on the Spectrum’s screen with assembly! I watched this with pride more times than I care to admit :) I had a snazzy logo and “Hello World” text, but that doesn’t make for a very interesting intro! The first effect I wanted was the scrolltext - without that, it’s not really an intro. Now, the Amiga had all kinds of hardware sprites, blitter operations, custom chips but the Spectrum had none of that. I was going to have to work out how to do things the hard way - “banging on the metal”, literally by shifting bits around the screen memory by hand and having the CPU do all the work. This is where understanding the Speccy’s weird screen and character cell layout is so important. It turns out there’s a neat trick as long as I stay inside one character row - there’s a RL instruction, which takes a byte and “rotates” it left. It inserts whatever is in the carry flag into the rightmost bit, and whatever gets shifted out ends up in the carry bit again. That means if you string a bunch of them together, you can move a set of pixels across the screen - the moment they “fall off” one character cell, they end up in the carry flag and then the next rl instruction inserts them at the beginning of the next character cell to the left. Repeat 32 times, and you’ve just moved a pixel (or set of pixels) across a line of the screen: linestart EQU $4880 ; leftmost byte of the line 12 rows down lineend EQU linestart+31 ; rightmost byte column ($489F) linelen EQU 32 ; one whole line is 32 bytes (character cells) across dot EQU %00000001 ; the dot "shape", just one pixel lit at the far right end newdot: ld a,dot ; load the dot shape (only rightmost pixel lit) into a ld (lineend),a ; store it in the rightmost byte of the line, pixel is now lit frame: halt ; wait for the next frame, otherwise it whizzes across too fast to see ld hl,lineend ; hl = last (rightmost) byte of the line ld b,linelen ; loop counter, 32 bytes to work through or a ; trick to clear the carry flag, or against itself leaves a register ; unchanged but clears various bits including the carry flag ; see https://jnz.dk/z80/or_r.html shift: rl (hl) ; rotate this byte left (00000001 -> 00000010) dec hl ; point hl at the next byte to the left djnz shift ; loop construct: b=b-1 and then jump back to shift label until b=0 jr c,newdot ; now we've shifted the whole line, so if the carry flag is set now it means ; the dot has just left the screen, so we jump back to the start jr frame ; otherwise it is still somewhere on the screen, so keep shifting! Download the video Hell yeah. It’s almost a proper scrolltext now! Wrapping this in another loop that increments h each time (remember - that gets us moving down a scanline) gets us moving an entire character cell. Here you can see it moving an entire row of my “Hello world” message. There was however a lot more work needed to make a proper scrolltext, with each letter appearing smoothly. I started off by just printing an entire character at a time on the right-hand edge of the screen - it worked, but was pretty ugly as you saw each new character appear at once in the right-hand side of the screen, scrolling 8 pixels over and then the next character flashed up. Instead, I wanted to make each character appear pixel by pixel and that required a bit more bit juggling. The routine I ended up with effectively streams each character in, line by line, pixel by pixel using the font data directly instead of calling the ROM print character routine. It does this by looking up the ASCII code for a character, and then fetching the corresponding font representation into a 8-byte buffer, one byte for each line of data. The font data starts with the space character (character code 32), so a lookup is simply font code base address + (ascii code - 32) * 8 to get to the bits for that letter. Arranged into an 8x8 grid, the bits for the letter H in the font data look like this. For each row of bits, I shift the line left using a neat trick. The byte is loaded into the a register, and then by adding itself to itself - add a,a we shift the value left (in binary, multiplying by 2 moves everything over one bit) so that the pixel I want to draw ends up dropping into the carry flag. This is an idiomatic way of doing it in Z80 code as it’s faster and smaller than the sla instruction. And every cycle quickly adds up when you’re re-drawing 256 bytes every frame! The character buffer then looks like this, with the top line shifted. Now, with the hl register pair pointing at the correct location on the screen, I can then use the rl trick discussed above. This shifts a scanline from the bottom character row of the screen left by one pixel and draws my pixel in the rightmost position by moving it in from the carry flag. Looping 32 times over this moves the entire row by 1 pixel with my new pixel inserted at the right. I then repeat this step again using inc h to move down one scan line until the entire character column has been drawn. At this point, the character buffer looks like this, and all 8 lines of the bottom screen row have all been shifted left by one bit by looping over all 32 character columns. The whole thing then repeats another 7 times until the entire character has been smoothly scrolled onto the screen. So just to re-state this because I think it’s worth the emphasis - just to scroll a message across the bottom of the screen, we’re having to loop over and re-draw 256 pixels every frame! It shows how much faster assembly is than the BASIC interpreter built into the spectrum’s ROM. I found it amazing to see my code finally working and imagining all the bits flying around and registers flapping away. It really makes me appreciate just how much effort went into creating the games I played back then. Music The original Amiga module I converted. A proper 1980s ear-worm! Of course, any good intro needs some music. On the Amiga, the standard was the “tracker” module format, which has a long and involved history of different forks, versions and tweaks. A song is built up from a series of patterns, each made up of rows of instructions. Each instruction usually includes note or pitch information, which sample to play and various parameters such as volume, effects - or in the case of some instructions, which position in the song to jump to. In many ways, it’s almost like a programming language for music. Using ProTracker as an example, a typical note cell is 4 bytes, shown on-screen as something like C-2 01 C20. That breaks down into three fields: the note and octave (C-2), the sample number (01), and the effect and its parameter (C20 - in this case, “set the volume to $20”, which is half of the maximum $40). A close-up of Protracker playing a tune. Note the VUmeters - another classic demo effect I would pinch later on! For an impressive demonstration of what’s possible with a tracker and to see what the note data looks like on-screen, check out the work of h0ffman, and in particular his winning “Professional Tracker” entry from the Revision 2014 competition. He’s an undisputed master of the science (or art!) of tracking, and his “Secrets of Protracker” talk gives a great overview of what’s possible. The Amiga’s sound hardware far eclipsed anything else on the market at the time, and the ZX Spectrum is paltry by comparison. The original 48K models only offered a rudimentary “beeper”, but the later 128K models included a 3-channel AY-3-8912 sound chip. This was very popular in 8-bit micros of the time, and a more-or-less identical variant was also used in the Atari ST. While it only offers basic waveforms and sound processing, it does enable multi-channel “chip music”, and so it became the mainstay of the ZX Spectrum demoscene. It also spawned a variety of SoundTracker clones on the Spectrum itself, although most musicians these days prefer to work in modern tools like Vortex Tracker II, which runs natively under Windows. The Conversion Pex looking back on his demos - this starts at the point where he features my letter and plays a bit of my cover version! For my intro, I wanted to port an Amiga module - and I had the perfect one in mind. One of my favourite Amiga music disks was “Sounds Of Gnome”, released in 1989 by the Swedish duo “Mahoney & Kaktus”. Mahoney also wrote NoiseTracker, one of the first really widely used trackers on the Amiga, and the program that served as my teenage introduction to making computer music of my own. Back in 2018 I made a rock cover of one of the other tunes on that disk, and wrote to Mahoney (real name “Pex”) about it. He ended up showing my “music video” at a demo conference, which I am eternally grateful for! So it seemed only fitting that I mangle another song from that collection with a Spectrum conversion… Sorry, guys! I picked the track “Milkman” which was written by “Glue Master”, an associate of Mahoney’s who had a few of his tracks featured on the compilation. It’s an ideal track for converting: It’s short (only 7 patterns in total), most of those patterns share the same chord sequence, and it looked fairly easy to squeeze the original 4 channels down into the Spectrum’s 3. Plus, it’s an incredibly catchy tune and I knew it well - I’ve been listening to it for 35+ years and often jam along to it on guitar or bass. The conversion started off pretty easily: all I really had to do was copy the patterns note-for-note from the Amiga mod, which I had open in OpenMPT as my Windows tracker. Below is a session with both trackers open on the same pattern - the original Amiga mod on the left, and my 3-channel Spectrum conversion on the right. Note the 11.7 in the 3rd channel of the Spectrum version. This is a “pitch slide down” effect in Vortex Tracker syntax, which drops the pitch by about a quarter-tone. It’s repeated on the next line, so the overall effect is a slide down from the A# that leads in to the next bass note. The original Amiga mod does the same thing using its own syntax - those are the 202s you can see on the same notes, at the bottom of the 4th track. Using just a single “beep” sound for everything was enough to check everything was working, and after a few weeks of hacking on it and re-arranging things in the evenings, I had a basic version which sounded surprisingly good for a first draft: Download the MP3 instead. I have to give a special mention to this forum post which was a massive help getting me started. It’s probably the best introduction to Vortex Tracker I’ve found - many, many thanks to the original author for taking the time to write it all up! The real challenge was making it sound as close to the original as possible. That meant building up a library of better-sounding samples for the different parts, and making extensive use of Vortex Tracker’s “ornaments”. Vortex Tracker comes with a great selection of sounds, and fortunately I’m also a weirdo who likes to listen to chiptunes in the background while I work! So whenever I found something particularly enticing, I filed it away for later pilfering inspiration. A Vortex Tracker sample isn’t really a “sample” in the Amiga sense of a recorded sound - it’s a definition of how a note sounds, one line per frame. Each line can mix in a tone or the white noise generator (useful for snare effects and suchlike), along with various other parameters. Ornaments, on the other hand, give you extra control over what pitch a note plays - and this enables really useful tricks like “arpeggio chords”. For example, an ornament of 0,4 makes the channel switch rapidly (25 times a second) between the note being played and the one 4 semitones above it. It happens so quickly that it sounds much like two notes played at the same time, which means you can build up full-sounding triad chords using only two channels. Take the opening chord in my fully converted version, Cm: It plays C in the first channel and the minor 3rd (D#) in the second, then I apply an ornament to the second channel to bring it up 4 semitones to the 5th (G). That gives all three notes of the chord, while keeping the 3rd channel free for other parts. As well as making much more efficient use of the limited 3 channels, this also gives Spectrum music that characteristic “warbling” sound, which I find really appealing in a retro kind of way! Between tricks like this and samples that mix tones with white noise (to make a combined “bass-snare” sound, for example), I managed to condense the full track down to 3 channels. Many months later, I have to say I’m really pleased with the final result. Playback The original site that was home to Vortex Tracker, documentation and other resources appears to be offline at the moment. I managed to grab a copy of everything I needed from archive.org - you’ll want the PTxTools.7z file for the playback routines discussed here. The playback was simple in theory: As this overview notes, you basically just pull in the player routine, include your module file and call the playback routine once per frame in your main loop. The catch is, none of the source code versions supplied in the archive worked with my Pasmo assembler. Fortunately, there is a pre-compiled binary object shipped alongside the source (VTII10bG in the PTxTools/PT3Play directory). This just has to be loaded in at address $C000 - this cannot be changed, as it has lots of absolute addresses baked in. My code to handle module playback now looks something like this, with the binary module loaded in place: ; Define the start address of the player routine and functions we need player EQU $C000 ; hard-coded org address of binary player plinit EQU player+3 ; init function in docs, module address in hl plplay EQU player+5 ; play and move on one frame ; from the docs - this "analyzes PT3 module header and prepares ; corresponding note and volume tables" ld hl,moduleimg call plinit ; main loop hold: halt call plplay call scroll ; other effects / routines go here... jr hold moduleimg: incbin "assets/music.pt3" Starfield Download the video Well, that didn’t work as hoped. TNFStro - tortoise edition. This brings me to the main event. No self-respecting cracktro is complete without a starfield! My first, naive approach to the classic starfield effect was to scatter random stars across the screen, then move down each scan-line applying the same rl trick I’d used for the scroll-text. I figured I could get parallax by shifting each row a different amount: One pixel for the first row, two for the second, three for the third, then back to one again, all the way down the screen to give the impression of several “fields” of stars moving at different speeds. It turned out there were two major problems with that approach. Firstly, shifting an entire row smeared whatever else was on it along with the stars - which I’d expected, and had filed away as a problem for future me. The bigger issue was speed: by the time I’d finished the first three rows I’d already performed hundreds of shift operations, and it was nowhere near fast enough to fit inside a single frame. Everything crawled across the screen, and the music playback slowed down so much that you could actually hear the arpeggio ornaments warbling away as separate notes. Clearly I was going to have to keep track of the individual stars instead. I found this routine, which was more-or-less what I had in mind and started me off with the idea of a structure to hold the state of each star. It also uses a screen address lookup much like the one from “Break Into Program”. Download the video Dammit. Back with another funeral dirge rendition :( That got things moving, but the co-ordinate lookup and tracking proved expensive to do (it has to run for every star, every frame as well as keep track of the old position for erasing) and I’d been ignoring the bigger question how to display stuff on top of the starfield. My original idea was simply to draw the stars and then paint text over them each frame, which blew the frame budget just as comprehensively as my first attempt. If I wanted something more interesting than an empty starfield, I’d have to keep track of which cells were occupied by text and avoid drawing stars in them in the first place, rather than over-painting them every time. After lots of scribbling in my notepad and visualising rows of bits, I eventually solved both challenges by “cheating” fairly liberally. So now, I don’t generate a random starfield at all. Instead, I built a table of pre-calculated Y and X position pairs: stdata: defb 109, 25, 88,182, 42, 96, 155,226 defb 59,254, 137,169, 160, 84, 125,193 ; ... and so on, for 32 stars ... This saves faffing about with an RNG, and once I have a layout that looks good I know it’ll always look good. I can also “fudge” things and ensure no two stars share the same y co-ordinate which sidesteps a whole class of problems like stars sharing a scanline, and having to deal with them overtaking each other. The “random” layout is generated by a simple Ruby script: #!/usr/bin/env ruby PAIRS = 32 PER_LINE = 4 # pairs per line Y_RANGE = 40..175 # exclude rows taken up by my logo & bars X_RANGE = 0..255 raise "y range too small" if Y_RANGE.size < PAIRS # no duplicate y values ys = Y_RANGE.to_a.sample(PAIRS) pairs = ys.map { |y| [y, rand(X_RANGE)] } pairs.each_slice(PER_LINE) do |row| body = row.map { |y, x| format('%3d,%3d,', y, x) }.join(' ') puts "defb #{body.chomp(',')}" end When the routine initialises, I convert those co-ordinate pairs into a table that tracks the state of each star. So rather than storing a position and translating it to a screen address every frame, each of my star records now looks like this (offsets from the base address): STSCR EQU 0 ; screen address of the byte the star is in, e.g. $4123 STOCC EQU 2 ; a pointer to the "occupancy check" table cell STMASK EQU 4 ; the star's bit pattern, e.g. %00010000 STSTEP EQU 5 ; "speed" - num. of pixels this star travels per frame The key thing is that there are no X/Y co-ordinates at all. For each star I store the screen byte it sits in and the bit mask within that byte, so it’s already in the form the screen writes need. That means no address conversion per frame, just simple bit shifting and rotation. Moving a star is then dead simple. I rotate its mask left with rlca, and when the bit rotates out of bit 7, it lands back in bit 0 with the carry flag set. That’s the signal I need to move the star one character cell to the left, which is one of the “cheap moves” from earlier. Because I’m tracking which cell each star is in, the collision check comes almost for free. That’s what the second pointer is for: STOCC (Star Occupancy) moves in step with STSCR through a table holding one byte per character cell. The “typewriter” routine sets a byte to 1 for every cell containing a text character or the cursor, so checking whether a star is hidden behind text is a single or instruction. If the cell is occupied, I skip drawing the star but carry on rotating and stepping it along until it reaches a cell where it can be displayed: ld l,(ix+STOCC) ld h,(ix+STOCC+1) ld a,(hl) ; A = the occupancy check for this cell or a ; test A against zero jr nz,stmove ; it's occupied, so leave the screen alone Parallax is still handled by moving each star a different number of pixels per frame, but the cost is a lot lower this time as I’m only touching the cells the stars actually occupy, rather than every byte of every row. Download the video That’s looking more like it! The typewriter routine is a simple copy of the font data from memory, and is very cheap as we can use the “cheap moves” to quickly fetch the 8x8 block. I also skip drawing any space character so it leaves the “occupancy cell” unmarked. This means the stars keep drifting through the gaps in the text - you can see here the odd star passing in between the words. I switched to ALL CAPS writing for the final version which helps the effect even more as the font is quite blocky, so you don’t really notice stars skipping the spaces inside a glyph at all. Misc Notes A few odds and ends that didn’t warrant a section of their own… VU Meters Some sort of graphic display bouncing along with the audio playback was another common early demo trope. I did originally consider a real-time graphic equaliser, but that would have been far too complex, probably in terms of spare cycles per frame, and definitely in terms of my limited mental capacity at this point! So I settled for a simple set of bouncing meter bars instead. It’s worth a mention mostly because it gives me an excuse to link to the awesome chibiakumas website! I just love the late 90s web design on display - one can argue that readability and accessibility take a hit, but it’s just so much fun to encounter a site that still looks like this. It’s a clear labour of love and there’s a wealth of content there as well, if your eyeballs are up to it. Their guide on Making Sound with the AY-3-8910 was what showed me how to get the data I needed to draw the meter bars. The examples write a register number to $FFFD to select a register, and I realised that once a register is selected, I could use the in instruction to read its value back out instead of writing a new one. As the page says, R8, R9 and R10 hold the channel amplitudes, so they feed nicely into the height parameter for drawing a bar. Getting the amplitude for channel A is then as simple as: ld a,8 ; Channel A register call ayread ; ... rest of code here ... ayread: ld bc,$FFFD ; Register select port out (c),a in a,(c) ret Twinkle, Twinkle, Little Star This is a simple effect that makes the header logo a little more interesting. I’m not sure where I originally got the idea - I’ve definitely seen something like it on a few late 80s/early 90s demos, though. All I wanted was to pick random star dots in the logo background and animate them. Here’s the header image loaded in ZXPaintbrush with the character cell grid showing. By happy chance, there are four stars which each sit neatly inside a single character cell with enough surrounding space, so they can be used to give an animated “twinkle” effect by drawing a cross out from the centre point and back in again. These dots map to the following co-ordinates, which target the top scanline of each fully-grown star (two pixels up from the centre): twkdata: defb 1,5 ; char row 0, col 5, centre y=3 ink 5 defb 1,20 ; char row 0, col 20, centre y=3 ink 1 bright defb 26,7 ; char row 3, col 7, centre y=4 ink 4 defb 25,22 ; char row 3, col 22, centre y=3 ink 6 bright The “twinkle” routine itself is simple enough: it adds a pixel on either axis to grow the star, waits a few frames, then shrinks it back down again. The interesting bit was how I made the twinkle order pseudo-random. Remember that my starfield was entirely pre-calculated from a data table generated by a Ruby script, which meant I’d never actually needed a random number generator in assembly. From Jonathan Cauldwell’s “How To Write Spectrum Games” e-book, I found another great “hack” which simply walks through the start of the Spectrum’s ROM, picking out random-looking bytes as it goes. As he says, he uses it in most of his games, and it does the job in a really simple way - as long as you stay below the block of structured, repeating data that sits 8Kb into the ROM. twkrand: push hl ld hl,(twkseed) ld a,h and 31 ; keep it within the first 8k of ROM ld h,a ld a,(hl) ; the "random" byte inc hl ld (twkseed),hl pop hl ret Out Of Space About halfway through writing the intro, I hit a fairly predictable problem: I had run out of memory. In an attempt to keep everything self-contained in one block of code, I was pulling all of the assets (the screen background, the player binary, the music file etc.) into the program itself, and then using the ldir block copy instruction to move each one into its correct location. Since that meant effectively storing every block twice, and the player routine had a hard-coded start address of $C000, I quickly ran out of room and my code started crashing into the playback routine. The simplest option was to split things out into separate code blocks, load each one directly to the address it needed to live at, and wrap the lot in a multi-stage BASIC tape loader: 10 CLEAR 32767 20 BORDER 0: PAPER 0: INK 0: CLS 30 LOAD "" CODE 40 LOAD "" CODE 50 LOAD "" CODE 60 RANDOMIZE USR 32768 This loads the main program, then the player routine directly into $C000, and finally the background screen straight into screen memory. It does mean the background starts appearing before the demo has fully loaded, but as I imagine most people will be seeing this running in an emulator or on my TNFS site, it doesn’t really matter. Plus, at this point I was in danger of disappearing down yet more rabbit holes (I still really want to work out how to do multi-colour border effects!) and I’d never have got this thing released. The Zen Of Z80 This whole project is a really weird one to look back at. On the one hand, I’ve fulfilled a childhood ambition and have something I’m very proud of. It’s also pretty unusual these days to end up with a binary where you can say pretty much what every single byte is doing and the story of how it got there. But on the other hand - I now have an answer for why I don’t code in assembly: It’s because I value (what’s left) of my sanity! The thing is, in order to write code like this you have to start thinking like the machine itself. You have to visualise the registers and the status flags flapping away, graphics as lines of bits slotted into strange orders, loops and conditionals as binary logic and so on. Getting this close to the hardware definitely brought about a greater sense of attachment to the hardware of my childhood, and it’s easy to see why assembly coders and sceners get so passionate about it. That said, I’m in no hurry to start re-writing my body of work in assembly! The projects I really enjoy are those that give me an excuse to learn something for the sheer hell of it, and to slow down a little from the ever-increasing velocity and demands of the modern work world. Going back to Z80 code is probably about the furthest I could have taken this, short of going into hardware hacking. I found it at times an intriguing puzzle, often a frustrating exercise that nearly had me throwing in the towel at several stages. This project has certainly taken the longest to go from the “idea” stage of any given feature to any kind of meaningful output. True, I didn’t spend every day of the last year working on it - my actual coding time was probably only a few hours over a couple of evenings a week. But I’ve certainly been thinking about it and pondering it most days (as my dogs will attest). The biggest breakthroughs and “aha!” moments usually didn’t come from behind the keyboard; they came from scribbles in my notebook and thinking about processor states while doing something unrelated. It has however given me massive appreciation and respect for the coders who pull this stuff off and make it look easy. And that goes double for those kids and bedroom coding superstars who managed all of this back in the 80s, and had to work it all out by experiment themselves. I only managed to get this far because of numerous books, online tutorials, GitHub repositories with sample code, YouTube videos… I can’t imagine how much longer this would have all taken me if the only resources I had were my Spectrum, a cassette tape recorder and a notepad. So, as I draw a line under this project - for now! - I raise a glass, and in the words of h0ffman: “Shout out to the hackers, graphicians and protrackers.”
A decade ago, a little bit of history was made. I didn't realize it, but a colleague made a great point, one of those real mind-changing points that seem too obvious to admit same-day. But, the next day, calver.org was born. At the time my team maintained the Python infrastructure for eBay and PayPal, and we were stuck deciding whether we were really ready for a "major" 1.0 release. Semantic Versioning was the only game in town and "major" means "big", right?! Thankfully, a wiser colleague mentioned: Ubuntu and Twisted don't struggle with version number debates. They slap a date on it and keep shipping. In fact, their date-based versions were even better because you always knew where it stood, in terms of updatedness and support. The only problem is that no one really knew about it. Somehow, this problem solving versioning alternative, arguably as old as history itself, had gone nameless for millenia, conspiring to make me feel foolish in an office meeting. Never again! Ten years of adoption Fast forward 10 years, we've seen CalVer adopted by Apple, Nvidia, JetBrains, and countless others. (We have a timeline!) The site may have more inbound links than any other project of mine. Apple made the biggest jump, at WWDC 2025: iOS went from 18 to 26 macOS from 15 to 26 watchOS from 11 to 26 and visionOS from 2 to 26 All landing on one, consistent number like a car's model year. I still remember the texts from the Venn diagram fanbase of my friends who love Apple and reasonable versioning. No such texts from when NVIDIA announced calendar versions across the GPU Operator, RAPIDS, and its monthly NGC containers, but still very cool. Open source, too: Home Assistant, pip, CockroachDB, and yt-dlp all ship on dates, with plenty more on the users page. The conversation even reached the language core; PEP 2026 proposed versioning CPython as 3.YY, and it almost happened, too. And it's never too late, time marches on! Fixing the notation But I don't think I got every detail right from day 1. That's the main motivator for CalVer 26. It's high time to start righting a couple idiosyncratic token design choices, starting with some additions: Meaning Before 26.0 26.0 Full year YYYY YYYY Short year (6, 16) YY YY Zero-padded year (06, 16) 0Y 0Y Short month (1 ... 12) MM M Zero-padded month (01 ... 12) 0M 0M Short week (1 ... 52) WW W Zero-padded week (01 ... 52) 0W 0W Short day (1 ... 31) DD D Zero-padded day (01 ... 31) 0D 0D Seeing double First, the doubled letters. From the first version (16.6), MM and DD meant the unpadded month and day, which reads backwards to anyone who knows date formats (ISO 8601's YYYY-MM-DD, Java, moment.js, day.js), as some community members correctly pointed out. I was ready to flip them, until I checked what people actually use: most projects with a YY.MM.MICRO badge (conda, Twisted, Ansible's tooling) don't pad, and more than a dozen other version management tools (like bumpver and bump-my-version) implement the old meaning. So, it's too late to flip MM's meaning. Instead, 26.0 deprecates it and offers a more explicit and hopefully clearer option: M is the short month, 0M the padded one, and MM is a technically-retired synonym for M. In case you're wondering, the explicit 0M was me being overinspired by Ubuntu's approach, perhaps: 6.06 pads its month but not its year, and YY.0M says exactly that. To pad or not to pad I think it's worth a detour into why padding is even a thing anyways. It's become important now that new ecosystems have emerged that enforced SemVer formatting semantics, and I wanted clear guidance about on the spec site. SemVer forbids leading zeros outright, so Cargo rejects 26.04.0 and Go modules reject v26.04.0. Even Python's packaging spec normalizes leading zeros away, so you can tag 2026.08.19 if you want, but PyPI will still show 2026.8.19. NVIDIA's GPU Operator docs put it this way: "Zero padding is omitted for month to be still compatible with semantic versioning." CalVer was always intended to drop in where SemVer was used. So 26.0 recommends unpadded (YYYY.M.D) as a sane "pure" default for software libraries. But libraries are not the only objects of versioning schemes. The exception is a version that becomes a filename, an image tag, or an object-store key that gets listed lexically. There, padding keeps 26.10 sorted after 26.09, which is why Ubuntu, NixOS, and NVIDIA's own NGC containers pad. More evidence of teams designing their versions. We love to see it. Our FAQ has a longer discussion of the padding issue, as well. Optional segments There was never any rule against them, but 26.0 makes optional trailing segments more explicit with square brackets. Now, yt-dlp's scheme can finally be written down: YYYY.0M.0D[.MICRO]. For the CalVer badges I could find on GitHub, they all stay valid for now. I've got a note on the deprecated spellings and a new copy-paste badge section for new ones. What else is new? It's always a great time to add more citations to the site. A spec changelog; the spec now versions itself: 16.6, 19.7, 26.0. A FAQ: breaking changes, same-day releases, and padding. The users page, rebuilt by category, with past users of note (schemes change; that's fine) and tooling. Case studies: Apple and NVIDIA in, yt-dlp replacing youtube-dl. Much of the thinking behind these changes happened in the GitHub issue tracker over the years, and 19 or so issues close with this release. That's where ideas for CalVer should go, so by all means, open an issue, and we'll get it sorted! In due time, of course. In closing, I can't believe I still love belaboring these numbers so much. Thanks to all (but especially Mark, Glyph, Hugo, issue reporters, translators, and maintainers) for the discussion, and ultimately making the most timely versioning system a timeless classic. See also 2016 announcement Designing a version My Yap on Why CalVer beats Semver
Four sincere attempts at OKRs, four failures, and every time: "you didn't do it right." Maybe the framework doesn't fit you -- so how do you find one that does?
Multiple concurrent writers. Multiple processes. Same SQLite. No modifications.