Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from ntietz.com blog - technically a blog
Working on my cursed MIDI project, I needed a way to store the most recent messages without risking an unbounded amount of memory usage. I turned to the trusty ring buffer for this! I started by writing a very simple one myself in Rust, then I looked and what do you know, of course the standard library has an implementation. While I was working on this project, I was pairing with a friend. She asked me about ring buffers, and that's where this post came from! What's a ring buffer? Ring buffers are known by a few names. Circular queues and circular buffers are the other two I hear the most. But this name just has a nice ring to it. It's an array, basically: a buffer, or a queue, or a list of items stacked up against each other. You can put things into it and take things out of it the same as any buffer. But the front of it connects to the back, like any good ring1. Instead of adding on the end and popping from the end it, like a stack, you can add to one end and remove from the start, like a queue. And as you add or remove things, where the start and end of the list move around. Your data eats its own tail. This lets us keep a fixed number of elements in the buffer without running into reallocation. In a regular old buffer, if you use it as a queue—add to the end, remove from the front—then you'll eventually need to either reallocate the entire thing or shift all the elements over. Instead, a ring buffer lets you just keep adding from either end and removing from either end and you never have to reallocate! Uses for a ring buffer My MIDI program is one example of where you'd want a ring buffer, to keep the most recent items. There are some general situations where you'll run into this: You want a fixed number of the most recent things, like the last 50 items seen You want a queue, especially with a fixed maximum number of queued elements2 You want a buffer for data coming in with an upper bound, like with streaming audio, and you want to overwrite old data if the consumer can't keep up for a bit A lot of it comes down to performance, and streaming data. Something is producing data, something else is consuming it, and you want to make sure insertions and removals are fast. That was exactly my use: a MIDI device produces messages, my UI consumes them, but I don't want to fill up all my memory, forever, with more of them. How ring buffers work So how do they work? This is a normal, non-circular buffer: When you add something, you put it on the end. If you're using it as a stack, then you can pop things off the end. But if you're using it as a queue, you pop things off the front... And then you have to shuffle everything to the left. That's why we have ring buffers! I'll draw it as a straight line here, to show how it's laid out in memory. But the end loops back to the front logically, and we'll see how it wraps around. We start with an empty buffer, and start and end both point at the first element. When start == end, we know the buffer is empty. If we insert an element, we move end forward. And it gets repeated as we insert multiple times. If we remove an element, we move start forward. We can also start the buffer at any point, and it crosses over the end gracefully. Ring buffers are pretty simple when you get into their details, with just a few things to wrap your head around. It's an incredibly useful data structure! Rust options If you want to use one in Rust, what are your options? There's the standard library, which includes VecDeque. This implements a ring buffer, and the name comes from "Vec" (Rust's growable array type) combined with "Deque" (a double-ended queue). As this is in the standard library, this is quite accessible from most code, and it's a growable ring buffer. This means that the pop/push operations will always be efficient, but if your buffer is full it will result in a resize operation. The amortized running time will still be O(1) for insertion, but you incur the cost all at once when a resize happens, rather than at each insertion. You can enforce size limits in your code, if you want to avoid resize operations. Here's an example of how you could do that. You check if it's full first, and if so, you remove something to make space for the new element. let buffer: VecDeque<u32> = VecDeque::with_capacity(10); for thing in 0..15 { // if the buffer is already full, remove the first element to make space if buffer.len() == 10 { buffer.pop_front(); } buffer.push_back(thing); } There are also libraries that can enforce this for you! For example, circular-buffer implements a ring buffer. It has a fixed max capacity and won't resize on you, instead overwriting elements when you run out of space. The size is set at compile time, though, which can be great or can be a constraint that's hard to meet. There is also ringbuffer, which gives you a fixed size buffer that's heap allocated at runtime. That buys you some flexibility, with the drawback of being heap-based instead of stack-based. I'm definitely reaching for a library when I need a non-growable ring buffer in Rust now. There are some good choices, and it saves me the trouble of having to manually enforce size limits. 1 One of my favorite rings represents the sun and the moon, with the sun nestling inside this crescent moon shape. Unfortunately, the ring is open there. It makes for a very nice visual effect, but it kept getting snagged on things and bending. So it's not a very good ring for living an active hands-on life. 2 These can also be resizable! The Rust standard library one is. This comes with reallocation cost, which amortizes to a low cost but can be expensive in individual moments.
Last year, my coworker Suzanne and I got a case study accepted into an ethnography conference! She's an anthropologist by training, and I'm a software engineer. How'd we wind up there? The short version is I asked her what I thought was an interesting question, and she thought it was an interesting question, and we wound up doing research. We have a lot more to go on the original question—we're studying the relationship between the values we hold as engineers and our documentation practices—and this case study is the first step in that direction. The title of the case study is a mouthful: "Foundations of Practice: An Ethnographic Exploration of Software Documentation and its Cultural Implications for an Agile Startup". I mean, it's intended for a more academic audience, and in a field that's not software engineering. But what we studied is relevant for software engineers, so let's talk about it here in more familiar language! If you want to read the full case study, it's open access. I'll start with the recommendations we provided, share a little bit about our methodology, and finish us out with a little about where we can go from here. Our recommendations The case study concluded with recommendations. This is an area I'd like us to do a lot more work in, because recommendations based on a case study are based on one example, and they'd obviously be much stronger based on more data! But for the conference submission, they were a necessary component. I think that the specific recommendations we gave hold up based on my experience (I'm biased, so take this as it is). These were our core recommendations. Start with high-level documentation. If you don't have any docs today, or all your docs are just in a bad way, the biggest bang-for-buck is with your high-level systems docs and process documentation. Don't have a system architecture diagram? Make it! Don't have a CI process diagram? Make it! These help you get your bearings on everything quickly, and they're also usually much faster to make than detailed documentation of more focused pieces. Use design reviews! Here, "design review" means putting your architecture changes—or any proposal for a major code change—through a review process analogous to a code review. This is one of the things we implemented at my company over four years ago, and it's incredibly useful. It makes us think more deeply on our designs, so we make fewer errors in the design phase of any given feature. And it makes the design visible to the whole team, so we get much better knowledge transfer. If I could drop these into every software company, I would. They're that useful. Think about your audience. This is just plain good writing advice, and it bears repeating. Not every kind of documentation has the same audience, and if you don't take that into account you'll write bad docs that your end user won't be able to use effectively. You may use too much jargon for the lay user, or too little jargon for efficient communication with the engineers working on a feature. Maintain the docs, updating them as part of your workflow. I think we all know that docs should be updated. The recommendation here is that updating docs should be part of your workflow. If you run into something that's out of date, go ahead and fix it. If someone asks you a question because it wasn't in a given doc, go ahead and add it there. Document test plans. QA processes are woefully underappreciated in software engineering. I don't quite understand why QA engineering is so looked down upon, because y'all, that stuff is hard. At any rate, testing? That's something you should document! Assuming you run tests before you release (you do, don't you?) then you need to have some organization and documentation around what those tests are, why they're there, etc. This needs to be shared between both QA and product engineers since if the test fails, it's probably, you know, the product engineer's change that broke it—so the better those docs are, the faster they can find the issue. Make onboarding docs. We know that good onboarding is important for bringing new team members up to productive contribution quickly. And we know that documentation is a big part of that. So focus on your documentation as part of your onboarding strategy. Run post-mortems and retrospectives. Mistakes will happen, and when they do? It's imperative that we go back through why they happened and what changes (if any) are merited to our processes as a result. Maybe there's work to be done to prevent similar incidents, or there's something we could do to detect this quicker in the future. Whatever you find, also update your docs: point to the post-mortems in the design docs that led to these decisions so that if someone reads those, they'll also see the longer-term impacts of those decisions. Encourage collaboration on documentation. You can do this by involving multiple team members in creating and reviewing design docs. This is a tool for mentoring and professional development. It also improves all the documentation quality. If you involve team members at all levels (not just your senior engineers!), you'll be able to ensure the docs and processes meet everyone's actual needs. Prioritize critical, complex areas for docs. You can't document everything. We mentioned earlier that you want to start with a high-level documentation overview, but you also want to focus your efforts on things which are critical or particularly complex. This will pay off by reducing the difficulty of understanding these areas, reducing rates of defects and speeding up development. Eliminate ineffective docs, focus on friction points. If your team says that documentation isn't serving a useful purpose, trust them and get rid of it. Change what you do to do something more effective. Focus your documentation efforts on resolving friction points between teams or in the development process. The methodology Our research comprised two main pieces: a literature review and interviews with our engineering team. We did the literature review first, to get our bearings. And, honestly, I was hoping to find answers to what we ultimately were setting out to answer. I'd rather not have to do this research to get our answers! The existing papers gave us a good grounding. I won't repeat too much here—you can read the section in the case study if you're interested. The bulk of the new research we did was through interviews, and analysis of those interviews1. We interviewed nine of our software engineers who were present at a critical transition in the company's history, when we implemented a major new documentation process as a result of transitioning to remote work. Each interview was recorded with the consent of the participants. We then transcribed them and did qualitative coding to extract themes. We were looking for some pre-defined themes (collaboration, learning, and knowledge sharing) and we were also interested in emergent themes. From there, we were able to extract out five values that are represented by documentation practices at the company. These values (collaboration, learning, continuous improvement, accountability, and technical excellence) are exhibited through our documentation practices. They are also reinforced by the documentation practices, so it's not wholly a causal relationship one way or the other. At this point we were able to take the results of the interviews and combine that with the literature review to figure out recommendations based on our case study. These are, of course, not universal. And they're to some extent conjecture: by the nature of a case study, we're not able to ground this in data across multiple companies. But they're a launching off point for teams in need and for future research. What's next? Ultimately, we want to drive toward answering the bigger questions we have around software engineering and documentation. One of those is: a lot of software engineers say their documentation is bad, and we also say that we value good documentation, so why do we not make the documentation better? Why are we not able to live our what we value there? Or do we not value documentation? To answer these questions to a satisfying end, we'll need to do a larger study. That means we'll need to recruit participants across many companies, and we'll need to design a study that can cut to the heart of these issues. I'm not sure when we'll have the time or energy to approach that, but I know we've both been completely sucked in by these questions. So we'll get there eventually! 1 One of the most interesting questions Suzanne got when presenting our research was: how did you get permission to do this? The question surprised me, because I hadn't thought that other companies would make this really hard, but her peers at the conference certainly had a hard time doing similar projects! The answer to this is a few things: we didn't ask for budget (nothing needs approval there), our engineers volunteered their time for it (they were interested, too!), and I'm on the leadership team and was able to just talk to the other company leaders to clear it.
It's kind of dark times right now. And I'm definitely only talking about the days being short. It's pretty dark out right now, since it's the winter in the northern hemisphere. Every year, I start to realize somewhere around January that I'm tired, really tired, and don't want to do anything. I'm not sure that this is seasonal affective disorder, but I did start to explore it and found that bright light makes me feel a lot better in the winter. The problem is, I go through this discovery every year, it seems. My earlier attempts After I first learned that bright light can actually help you feel better in the winter, I got a Happy Light. That's the actual branding on it, and it might have helped, maybe? But it was really inconvenient to use. I got one of the more affordable ones, which meant a small panel I had to sit uncomfortably close to. And I was supposed to use it almost first thing in the morning for 30 minutes. That's... really not going to fit in my routine, especially now that I have two young kids who are raring to go right at 7am, and they do not want to deal with mom sitting in front of a lamp that early. So I just kind of stopped using it, and would poke at it in the drawer occasionally but not really do anything with it. Instead, I got a few more lamps in my office. These seemed to help, a little, but that might just be that I like my spaces well lit1. Going as bright as possible Somewhere along the line I saw a post talking about winter energy and light. And the author suggested that indoor lights just have to be bright. They recommended some high-watt LED bulbs. I went ahead and ordered some and... these are very bright. They're 100 watts of LEDs each, and I have two of them. I plugged them in to these plug-in fixtures that have inline switches. These seemed to really help me out a lot. On days I used them, I was more energetic and felt better. But I had a lot of days when I didn't use them, because using them was inconvenient. The main problems were that I had to turn on two switches manually, and they were too bright to look at. Turning them on doesn't sound like a lot, but doing it every day when you're already struggling can be just enough friction to avoid doing it! The brightness was the biggest issue, because they were blinding to look at and cast this reverse shadow out of my office. The brightness was the point, so now do I deal with that? Shading it Turning on the switches was easy. I put them on some smart outlets, set a schedule, and now they turn on and off at my predetermined times! Then I had to make a shade for the lamps. My main goal here is to diffuse the light a bit, so I was thinking of a small wooden frame wrapped in a light white fabric. That would spread the light out and avoid having any spots that are too bright to look at. I took measurements for where it was going to fit, then headed out to my workshop. Then I ripped a scrap board into 0.75"x0.75" stock. I ended up with just enough for the design. This is what I ended up with, after it was glued up. Here's a bonus detail shot of the extremely unnecessary joinery here. But hey, this joinery was incredibly satisfying, and the whole thing went together without any nails or screws. Joy makes it worthwhile. And then with the fabric attached2, it ended looking pretty nice. In its intended home, you can see that it does its job! It's now not painful to look straight at the lamp. You wouldn't want to do it, but it won't hurt you at least! Waiting for summer Of course, we need these bright lights during the winter, during the dark days. But summer will come again. Not all of us will see the long days. We'll lose some to the winter. But summer will come again, and we'll be able to go outside freely. Soak in the sun. Be ourselves. Dance in sunshowers. It's hard to see that in the darkest of times. I'm writing this for myself as much as I'm writing it for you. But believe me: you are strong, and I am strong, and together, we will survive. It seems like this winter is going to last a long time. And it probably will. But I'm not just talking about the winter, and the dark times are going to end, eventually. 1 Most places are very poorly lit and don't have enough lamps. You should have a lot of lamps in every room to make it as bright and cheery as possible! I refuse to believe otherwise, but my wife, a gremlin of the darkness3, does like rooms quite dim. 2 My wife helped with this step. She works with fabric a lot, and this was our first project combining our respective crafts. Certainly not our last! 3 She approved this phrasing.
Recently, a reader wrote to me and asked about my writing process and burnout. They had an image in their head that I could sit down at a computer and type up a full post on a given topic, but were unsure if that's the right approach when they start blogging. And they were concerned about how to keep things sustainable for themselves, too. I started to write back to them, but decided to ship my answer for everyone else, as well. Now, to be clear: this is my writing process. There are many other very valid approaches, such as what Gabriella wrote. But I think outlining my process here will at least help break some of the mystique around writing, because I certainly do not just sit down and plunk out a post. Well, it often looks like that if you're an outside observer1. But that's only what's visible. How I write The very first piece of any blog post for me is an idea, some inspiration, something I want to share. These ideas come from a variety of places. Some of them are from conversations with friends or coworkers. Others come from problems I'm solving in my own code. Or they come from things I'm reading. Very rarely, they come from just thinking about things, but those are other sources with the connection hidden by the passage of time. After an idea comes to me, I try to capture it quickly. If I don't, it's probably gone! I store these ideas in a note in Obsidian, which currently has 162 ideas in it2. I format this file as a bulleted list, with minimal categorization so that the friction for adding something new is as low as possible. When I revisit this list to plan things to write (I do this every few weeks), I'll remove things which are either completed or which I'm confident I am not interested in anymore. Ideas which are promoted from "want to write" to "definitely going to write" then get moved into my primary task tracking system, LunaTask. When I start typing an article, I move it from "later" or "next" to "in progress". But in reality, I start writing an article far before I start typing it. See, the thing is, I'm almost always thinking about topics I'm going to write about. Maybe this is an element of having ADHD, or maybe it's from my obsessive interests and my deep diving into things I'm curious about. But it started happening more when I started writing on a schedule. By writing on a schedule, having to publish weekly, I've gotten in the habit of thinking about things so that I can hit the ground running when I sit down to type. Once I've picked out an idea, then it sits in the back of my head and some background processing happens. Usually I'm not crystal clear on what I want to say on a topic, so that background process churns through it. After figuring out what I want to say, I move onto a little more active thought on how I want to say it and structure it. This happens while I'm on a run or a walk, while I'm on coffee breaks, between exercises I'm working on for my music lessons. There's a lot of background processing between snatches of active effort. Piece by piece, it crystallizes in my mind and I figure out what I want to say and how to say it. Then, finally, I sit at the keyboard. And this is where it really looks like I just sit down and write posts in one quick sitting. That's because by then, the ideas are fully formed and it's a matter of writing the words out—and after a few years of that, it's something I got pretty quick at. Take this blog post for example. I received the initial email on January 1st, and it sat in my inbox for a few days before I could read it and digest it. I started to write a reply, but realized it would work as a blog post. So it ended up in background processing for a week, then I started actively thinking about what I wanted to write. At that point, I created a blog post file with section headings as an outline. This post ended up being delayed a bit, since I had to practice content-driven development for a project that has a self-imposed deadline. But the core writing (though not the typing) was done in the week after opening the original email. And then when I finally sat down to write it, it was two 20-minute sessions of typing out the words3. Editing and publishing After the first draft of a post is written, there's a lot of detail work remaining. This draft is usually in the approximate shape of the final post, but it may be missing sections or have too much in some. I'll take this time to do a quick read over it by myself. Here I'm not checking for spelling errors, but checking if the structure is good and if it needs major work, or just copy edits. If I'm unsure of some aspects of a post, I ask for feedback. I'll reach out to one or two friends who I think would have useful feedback and would enjoy reading it early, and ask them if they can read it. This request comes with specific feedback requests, since those are easier to fulfill then a broad "what do you think?" These are something along the lines of "was it clear? what parts stuck out to you positively or negatively? is there anything you found unclear or inaccurate?" Once people get me feedback, I add an acknowledgement for them (with explicit consent for if/how to share their name and where to link). After I'm more sure of the structure and content of a post, I'll go through and do copy edits. I read through it, with a spellchecker enabled4, and fix any misspellings and improve the wording of sentences. Then it's just the final details. I write up the blurb that goes in my newsletter and on social media (Mastodon, Bluesky, and LinkedIn, currently). I make sure the date is correct, the title is a decent one, and the slug mostly matches the title. And I make sure I did do things like acknowledgements and spellchecking. Here's the full checklist I use for every post before publishing. - [ ] Edited? - [ ] Newsletter blurb written? - [ ] Spellchecked? - [ ] Is the date correct? - [ ] Is the title updated? - [ ] Does the slug match the title? - [ ] Are tags set? - [ ] Draft setting removed? - [ ] Are acknowledgements added? Once that checklist is complete, it's ready to go! That means that on Monday morning, I just have to run make deploy and it'll build and push out the article. Making it sustainable I've been writing at least one post a week since September 2022—over two years at this point, and about 200,000 words. I've avoided burning out on it, so far. That's an interesting thing, since I've burned out on personal projects in the past. What makes this different? One aspect is that it's something that's very enjoyable for me. I like writing, and the process of it. I've always liked writing, and having an outlet for it is highly motivating for me. I thrive in structure, so the self-imposed structure of publishing every week is helpful. This structure makes me abandon perfection and find smaller pieces that I can publish in a shorter amount of time, instead of clinging to a big idea and shipping all of it at once, or never. This is where content-driven development comes in for me, and the consistent schedule also leads me to create inspiration. I also don't put pressure on myself besides the deadline. That's the only immovable thing. Everything else—length, content, even quality5—is flexible. Some of my earliest posts from 2022 are of the form "here's what I did this week!" which got me into a consistent rhythm. The most important thing for me is to keep it fun. If I'm not finding something fun, I just move on to a different post or a different topic. I have some posts that would be good but have sat in my backlog for over a year, waiting to be written. Whenever I sit down to write those, my brain decides to find a bunch of other, better, posts to write instead! So far, it's been fun and motivating to write weekly. If that ever stops, and it becomes a job and something I dread, I'll change the format to be something that works better for me instead! 1 If this is what it looks like then hey, stop peeking through my window. 2 I keep them in a bulleted list, each starting with -, so I can find the count via: grep "\s*-" 'Blog post ideas.md' | wc -l 3 Drafting documents is where I feel like my typing speed is actually a real benefit. On a typing test this evening, I got 99% accuracy at 120wpm. This lets me get through a lot of words quickly. This is a big plus for me both as a writer and as a principal engineer. 4 I leave my spellchecker disabled the vast majority of the time, since the squiggles are distracting! It's not a big deal to misspell things most of the time, and I'd rather be able to focus on getting words out than noticing each mistake as I make it—talk about discouraging! 5 That said, I think it's hard to judge your own quality in the moment. As writers, we can be very critical of ourselves. We think we wrote something boring but others find it interesting. Or something we think is quite clever, others find uninspiring. Sometimes, when I push out something I think is lazy and quick and low-quality, I'm surprised to find a very eager reception.
More in programming
No newsletter next week, I'm teaching a TLA+ workshop. Speaking of which: I spend a lot of time thinking about formal methods (and TLA+ specifically) because it's where the source of almost all my revenue. But I don't share most of the details because 90% of my readers don't use FM and never will. I think it's more interesting to talk about ideas from FM that would be useful to people outside that field. For example, the idea of "property strength" translates to the idea that some tests are stronger than others. Another possible export is how FM approaches nondeterminism. A nondeterministic algorithm is one that, from the same starting conditions, has multiple possible outputs. This is nondeterministic: # Pseudocode def f() { return rand()+1; } When specifying systems, I may not encounter nondeterminism more often than in real systems, but I am definitely more aware of its presence. Modeling nondeterminism is a core part of formal specification. I mentally categorize nondeterminism into five buckets. Caveat, this is specifically about nondeterminism from the perspective of system modeling, not computer science as a whole. If I tried to include stuff on NFAs and amb operations this would be twice as long.1 1. True Randomness Programs that literally make calls to a random function and then use the results. This the simplest type of nondeterminism and one of the most ubiquitous. Most of the time, random isn't truly nondeterministic. Most of the time computer randomness is actually pseudorandom, meaning we seed a deterministic algorithm that behaves "randomly-enough" for some use. You could "lift" a nondeterministic random function into a deterministic one by adding a fixed seed to the starting state. # Python from random import random, seed def f(x): seed(x) return random() >>> f(3) 0.23796462709189137 >>> f(3) 0.23796462709189137 Often we don't do this because the point of randomness is to provide nondeterminism! We deliberately abstract out the starting state of the seed from our program, because it's easier to think about it as locally nondeterministic. (There's also "true" randomness, like using thermal noise as an entropy source, which I think are mainly used for cryptography and seeding PRNGs.) Most formal specification languages don't deal with randomness (though some deal with probability more broadly). Instead, we treat it as a nondeterministic choice: # software if rand > 0.001 then return a else crash # specification either return a or crash This is because we're looking at worst-case scenarios, so it doesn't matter if crash happens 50% of the time or 0.0001% of the time, it's still possible. 2. Concurrency # Pseudocode global x = 1, y = 0; def thread1() { x++; x++; x++; } def thread2() { y := x; } If thread1() and thread2() run sequentially, then (assuming the sequence is fixed) the final value of y is deterministic. If the two functions are started and run simultaneously, then depending on when thread2 executes y can be 1, 2, 3, or 4. Both functions are locally sequential, but running them concurrently leads to global nondeterminism. Concurrency is arguably the most dramatic source of nondeterminism. Small amounts of concurrency lead to huge explosions in the state space. We have words for the specific kinds of nondeterminism caused by concurrency, like "race condition" and "dirty write". Often we think about it as a separate topic from nondeterminism. To some extent it "overshadows" the other kinds: I have a much easier time teaching students about concurrency in models than nondeterminism in models. Many formal specification languages have special syntax/machinery for the concurrent aspects of a system, and generic syntax for other kinds of nondeterminism. In P that's choose. Others don't special-case concurrency, instead representing as it as nondeterministic choices by a global coordinator. This more flexible but also more inconvenient, as you have to implement process-local sequencing code yourself. 3. User Input One of the most famous and influential programming books is The C Programming Language by Kernighan and Ritchie. The first example of a nondeterministic program appears on page 14: For the newsletter readers who get text only emails,2 here's the program: #include /* copy input to output; 1st version */ main() { int c; c = getchar(); while (c != EOF) { putchar(c); c = getchar(); } } Yup, that's nondeterministic. Because the user can enter any string, any call of main() could have any output, meaning the number of possible outcomes is infinity. Okay that seems a little cheap, and I think it's because we tend to think of determinism in terms of how the user experiences the program. Yes, main() has an infinite number of user inputs, but for each input the user will experience only one possible output. It starts to feel more nondeterministic when modeling a long-standing system that's reacting to user input, for example a server that runs a script whenever the user uploads a file. This can be modeled with nondeterminism and concurrency: We have one execution that's the system, and one nondeterministic execution that represents the effects of our user. (One intrusive thought I sometimes have: any "yes/no" dialogue actually has three outcomes: yes, no, or the user getting up and walking away without picking a choice, permanently stalling the execution.) 4. External forces The more general version of "user input": anything where either 1) some part of the execution outcome depends on retrieving external information, or 2) the external world can change some state outside of your system. I call the distinction between internal and external components of the system the world and the machine. Simple examples: code that at some point reads an external temperature sensor. Unrelated code running on a system which quits programs if it gets too hot. API requests to a third party vendor. Code processing files but users can delete files before the script gets to them. Like with PRNGs, some of these cases don't have to be nondeterministic; we can argue that "the temperature" should be a virtual input into the function. Like with PRNGs, we treat it as nondeterministic because it's useful to think in that way. Also, what if the temperature changes between starting a function and reading it? External forces are also a source of nondeterminism as uncertainty. Measurements in the real world often comes with errors, so repeating a measurement twice can give two different answers. Sometimes operations fail for no discernable reason, or for a non-programmatic reason (like something physically blocks the sensor). All of these situations can be modeled in the same way as user input: a concurrent execution making nondeterministic choices. 5. Abstraction This is where nondeterminism in system models and in "real software" differ the most. I said earlier that pseudorandomness is arguably deterministic, but we abstract it into nondeterminism. More generally, nondeterminism hides implementation details of deterministic processes. In one consulting project, we had a machine that received a message, parsed a lot of data from the message, went into a complicated workflow, and then entered one of three states. The final state was totally deterministic on the content of the message, but the actual process of determining that final state took tons and tons of code. None of that mattered at the scope we were modeling, so we abstracted it all away: "on receiving message, nondeterministically enter state A, B, or C." Doing this makes the system easier to model. It also makes the model more sensitive to possible errors. What if the workflow is bugged and sends us to the wrong state? That's already covered by the nondeterministic choice! Nondeterministic abstraction gives us the potential to pick the worst-case scenario for our system, so we can prove it's robust even under those conditions. I know I beat the "nondeterminism as abstraction" drum a whole lot but that's because it's the insight from formal methods I personally value the most, that nondeterminism is a powerful tool to simplify reasoning about things. You can see the same approach in how I approach modeling users and external forces: complex realities black-boxed and simplified into nondeterministic forces on the system. Anyway, I hope this collection of ideas I got from formal methods are useful to my broader readership. Lemme know if it somehow helps you out! I realized after writing this that I already talked wrote an essay about nondeterminism in formal specification just under a year ago. I hope this one covers enough new ground to be interesting! ↩ There is a surprising number of you. ↩
I’ve been working on a transition to using light-dark() function in CSS. What this boils down to is, rather than CSS that looks like this: :root { color-scheme: light; --text: #000; } @media (prefers-color-scheme: dark) { :root { color-scheme: dark; --text: #fff; } } I now have this: :root { color-scheme: light; --text: light-dark(#000, #fff); } @media (prefers-color-scheme: dark) { :root { color-scheme: dark; } } That probably doesn’t look that interesting. That’s what I thought when I first learned about light-dark() — “Oh hey, that’s cool, but it’s just different syntax. Six of one, half dozen of another kind of thing.” But it does unlock some interesting ways to handling themeing which I will have to cover in another post. Suffice it to say, I think I’m starting to drink the light-dark() koolaid. Anyhow, using the above pattern, I want to compose CSS variables to make a light/dark theme based on a configurable hue. Something like this: :root { color-scheme: light; /* configurable via JS */ --accent-hue: 56; /* which then cascades to other derivations */ --accent: light-dark( hsl(var(--accent-hue) 50% 100%), hsl(var(--accent-hue) 50% 0%), ); } @media (prefers-color-scheme: dark) { :root { color-scheme: dark; } } The problem is that --accent-hue value doesn’t quite look right in dark mode. It needs more contrast. I need a slightly different hue for dark mode. So my thought is: I’ll put that value in a light-dark() function. :root { --accent-hue: light-dark(56, 47); --my-color: light-dark( hsl(var(--accent-hue) 50% 100%), hsl(var(--accent-hue) 50% 0%), ); } Unfortunately, that doesn’t work. You can’t put arbitrary values in light-dark(). It only accepts color values. I asked what you could do instead and Roma Komarov told me about CSS “space toggles”. I’d never heard about these, so I looked them up. First I found Chris Coyier’s article which made me feel good because even Chris admits he didn’t fully understand them. Then Christopher Kirk-Nielsen linked me to his article which helped me understand this idea of “space toggles” even more. I ended up following the pattern Christopher mentions in his article and it works like a charm in my implementation! The gist of the code works like this: When the user hasn’t specified a theme, default to “system” which is light by default, or dark if they’re on a device that supports prefers-color-scheme. When a user explicitly sets the color theme, set an attribute on the root element to denote that. /* Default preferences when "unset" or "system" */ :root { --LIGHT: initial; --DARK: ; color-scheme: light; } @media (prefers-color-scheme: dark) { :root { --LIGHT: ; --DARK: initial; color-scheme: dark; } } /* Handle explicit user overrides */ :root[data-theme-appearance="light"] { --LIGHT: initial; --DARK: ; color-scheme: light; } :root[data-theme-appearance="dark"] { --LIGHT: ; --DARK: initial; color-scheme: dark; } /* Now set my variables */ :root { /* Set the “space toggles’ */ --accent-hue: var(--LIGHT, 56) var(--DARK, 47); /* Then use them */ --my-color: light-dark( hsl(var(--accent-hue) 50% 90%), hsl(var(--accent-hue) 50% 10%), ); } So what is the value of --accent-hue? That line sort of reads like this: If --LIGHT has a value, return 56 else if --DARK has a value, return 47 And it works like a charm! Now I can set arbitrary values for things like accent color hue, saturation, and lightness, then leverage them elsewhere. And when the color scheme or accent color change, all these values recalculate and cascade through the entire website — cool! A Note on Minification A quick tip: if you’re minifying your HTML and you’re using this space toggle trick, beware of minifying your CSS! Stuff like this: selector { --ON: ; --OFF: initial; } Could get minified to: selector{--OFF:initial} And this “space toggles trick” won’t work at all. Trust me, I learned from experience. Email · Mastodon · Bluesky
Most of our cultural virtues, celebrated heroes, and catchy slogans align with the idea of "never give up". That's a good default! Most people are inclined to give up too easily, as soon as the going gets hard. But it's also worth remembering that sometimes you really should fold, admit defeat, and accept that your plan didn't work out. But how to distinguish between a bad plan and insufficient effort? It's not easy. Plenty of plans look foolish at first glance, especially to people without skin in the game. That's the essence of a disruptive startup: The idea ought to look a bit daft at first glance or it probably doesn't carry the counter-intuitive kernel needed to really pop. Yet it's also obviously true that not every daft idea holds the potential to be a disruptive startup. That's why even the best venture capital investors in the world are wrong far more than they're right. Not because they aren't smart, but because nobody is smart enough to predict (the disruption of) the future consistently. The best they can do is make long bets, and then hope enough of them pay off to fund the ones that don't. So far, so logical, so conventional. A million words have been written by a million VCs about how their shrewd eyes let them see those hidden disruptive kernels before anyone else could. Good for them. What I'm more interested in knowing more about is how and when you pivot from a promising bet to folding your hand. When do you accept that no amount of additional effort is going to get that turkey to soar? I'm asking because I don't have any great heuristics here, and I'd really like to know! Because the ability to fold your hand, and live to play your remaining chips another day, isn't just about startups. It's also about individual projects. It's about work methods. Hell, it's even about politics and societies at large. I'll give you just one small example. In 2017, Rails 5.1 shipped with new tooling for doing end-to-end system tests, using a headless browser to validate the functionality, as a user would in their own browser. Since then, we've spent an enormous amount of time and effort trying to make this approach work. Far too much time, if you ask me now. This year, we finished our decision to fold, and to give up on using these types of system tests on the scale we had previously thought made sense. In fact, just last week, we deleted 5,000 lines of code from the Basecamp code base by dropping literally all the system tests that we had carried so diligently for all these years. I really like this example, because it draws parallels to investing and entrepreneurship so well. The problem with our approach to system tests wasn't that it didn't work at all. If that had been the case, bailing on the approach would have been a no brainer long ago. The trouble was that it sorta-kinda did work! Some of the time. With great effort. But ultimately wasn't worth the squeeze. I've seen this trap snap on startups time and again. The idea finds some traction. Enough for the founders to muddle through for years and years. Stuck with an idea that sorta-kinda does work, but not well enough to be worth a decade of their life. That's a tragic trap. The only antidote I've found to this on the development side is time boxing. Programmers are just as liable as anyone to believe a flawed design can work if given just a bit more time. And then a bit more. And then just double of what we've already spent. The time box provides a hard stop. In Shape Up, it's six weeks. Do or die. Ship or don't. That works. But what's the right amount of time to give a startup or a methodology or a societal policy? There's obviously no universal answer, but I'd argue that whatever the answer, it's "less than you think, less than you want". Having the grit to stick with the effort when the going gets hard is a key trait of successful people. But having the humility to give up on good bets turned bad might be just as important.
Last year I wrote about using static websites for tiny archives. The idea is that I create tiny websites to store and describe my digital collections. There are several reasons I like this approach: HTML is flexible and lets me display data in a variety of ways; it’s likely to remain readable for a long time; it lets me add more context than a folder full of files. I’m converting more and more of my local data to be stored in static websites – paperwork I’ve scanned, screenshots I’ve taken, and web pages I’ve bookmarked. I really like this approach. I got a lot of positive feedback, but the most common reply was “please share some source code”. People wanted to see examples of the HTML and JavaScript I was using I deliberately omitted any code from the original post, because I wanted to focus on the concept, not the detail. I was trying to persuade you that static websites are a good idea for storing small archives and data sets, and I didn’t want to get distracted by the implementation. There’s also no single code base I could share – every site I build is different, and the code is often scrappy or poorly documented. I’ve built dozens of small sites this way, and there’s no site that serves as a good example of this approach – they’re all built differently, implement a subset of my ideas, or have hard-coded details. Even if I shared some source code, it would be difficult to read or understand what’s going on. However, there’s clearly an appetite for that sort of explanation, so this follow-up post will discuss the “how” rather than the “why”. There’s a lot of code, especially JavaScript, which I’ll explain in small digestible snippets. That’s another reason I didn’t describe this in the original post – I didn’t want anyone to feel overwhelmed or put off. A lot of what I’m describing here is nice-to-have, not essential. You can get started with something pretty simple. I’ll go through a feature at a time, as if we were building a new static site. I’ll use bookmarks as an example, but there’s nothing in this post that’s specific to bookmarking. If you’d like to see everything working together, check out the demo site. It includes the full code for all the sections in this post. Let’s dive in! Start with a hand-written HTML page (demo) Reduce repetition with JavaScript templates (demo) Add filtering to find specific items (demo) Introduce sorting to bring order to your data (demo) Use pagination to break up long lists (demo) Provide feedback with loading states and error handling (demo 1, demo 2) Test the code with QUnit and Playwright Manipulate the metadata with Python Store the website code in Git Closing thoughts demo) A website can be a single HTML file you edit by hand. Open a text editor like TextEdit or Notepad, copy-paste the following text, and save it in a file named bookmarks.html. <h1>Bookmarks</h1> <ul> <li><a href="https://estherschindler.medium.com/the-old-family-photos-project-lessons-in-creating-family-photos-that-people-want-to-keep-ea3909129943">Lessons in creating family photos that people want to keep, by Esther Schindler (2018)</a></li> <li><a href="https://www.theatlantic.com/technology/archive/2015/01/why-i-am-not-a-maker/384767/">Why I Am Not a Maker, by Debbie Chachra (The Atlantic, 2015)</a></li> <li><a href="https://meyerweb.com/eric/thoughts/2014/06/10/so-many-nevers/">So Many Nevers, by Eric Meyer (2014)</a></li> </ul> If you open this file in your web browser, you’ll see a list of three links. You can also check out my demo page to see this in action. This is an excellent way to build a website. If you stop here, you’ve got all the flexibility and portability of HTML, and this file will remain readable for a very long time. I build a lot of sites this way. I like it for small data sets that I know are never going to change, or which change very slowly. It’s simple, future-proof, and easy to edit if I ever need to. demo) As you store more data, it gets a bit tedious to keep copying the HTML markup for each item. Wouldn’t it be useful if we could push it into a reusable template? When a site gets bigger, I convert the metadata into JSON, then I use JavaScript and template literals to render it on the page. Let’s start with a simple example of metadata in JSON. My real data has more fields, like date saved or a list of keyword tags, but this is enough to get the idea: const bookmarks = [ { "url": "https://estherschindler.medium.com/the-old-family-photos-project-lessons-in-creating-family-photos-that-people-want-to-keep-ea3909129943", "title": "Lessons in creating family photos that people want to keep, by Esther Schindler (2018)" }, { "url": "https://www.theatlantic.com/technology/archive/2015/01/why-i-am-not-a-maker/384767/", "title": "Why I Am Not a Maker, by Debbie Chachra (The Atlantic, 2015)" }, { "url": "https://meyerweb.com/eric/thoughts/2014/06/10/so-many-nevers/", "title": "So Many Nevers, by Eric Meyer (2014)" } ]; Then I have a function that renders the data for a single bookmark as HTML: function Bookmark(bookmark) { return ` <li> <a href="${bookmark.url}">${bookmark.title}</a> </li> `; } Having a function that returns HTML is inspired by React and Next.js, where code is split into “components” that each render part of the web app. This function is simpler than what you’d get in React. Part of React’s behaviour is that it will re-render the page if the data changes, but my function won’t do that. That’s okay, because my data isn’t going to change. The HTML gets rendered once when the page loads, and that’s enough. I’m using a template literal because I find it simple and readable. It looks pretty close to the actual HTML, so I have a pretty good idea of what’s going to appear on the page. Template literals are dangerous if you’re getting data from an untrusted source – it could allow somebody to inject arbitrary HTML into your page – but I’m writing all my metadata, so I trust it. I know there are other ways to construct HTML in JavaScript, like document.createElement(), the <template> element, or Web Components – but template literals have always been sufficient for me, and I’ve never had a reason to explore other options. Now we have to call this function when the page loads, and render the list of bookmarks. Here’s the rest of the code: <script> window.addEventListener("DOMContentLoaded", () => { document.querySelector("#listOfBookmarks").innerHTML = bookmarks.map(Bookmark).join(""); }); </script> <h1>Bookmarks</h1> <ul id="listOfBookmarks"></ul> I’m listening for the DOMContentLoaded event, which occurs when the HTML page has been fully parsed. When that event occurs, it looks for <ul id="listOfBookmarks"> in the page, and inserts the HTML for the list of bookmarks. We have to wait for this event so the <ul> actually exists. If we tried to run it immediately, it might run before the <ul> exists, and then it wouldn’t know where to insert the HTML. I’m using querySelector() to find the <ul> I want to modify – this is a newer alternative to functions like getElementById(). It’s quite flexible, because I can target any CSS selector, and I find CSS rules easier to remember than the family of getElementBy* functions. Although it’s slightly slower in benchmarks, the difference is negligible and it’s easier for me to remember. If you want to see this page working, check out the demo page. I use this pattern as a starting point for a lot of my static sites – metadata in JSON, some functions that render HTML, and an event listener that renders the whole page after it loads. Once I have the basic site, I add data, render more HTML, and write CSS styles to make it look pretty. This is where I can have fun, and really customise each site. I keep tweaking until I have something I like. I’m ignoring CSS because that could be a whole other post, and there’s a vintage charm to unstyled HTML – it’s fine for what we’re discussing today. What else can we do? demo) As the list gets even longer, it’s useful to have a way to find specific items in the list – I don’t want to scroll the whole thing every time. I like adding keyword tags to my data, and then filtering for items with particular tags. If I add other metadata fields, I could filter on those too. Here’s a brief sketch of the sort of interface I like: I like to be able to define a series of filters, and apply them to focus on a specific subset of items. I like to combine multiple filters to refine my search, and to see a list of applied filters with a way to remove them, if I’ve filtered too far. I like to apply filters from a global menu, or to use controls on each item to find similar items. I use URL query parameters to store the list of currently-applied filters, for example: bookmarks.html?tag=animals&tag=wtf&publicationYear=2025 This means that any UI element that adds or removes a filter is a link to a new URL, so clicking it loads a new page, which triggers a complete re-render with the new filters. When I write filtering code, I try to make it as easy as possible to define new filters. Every site needs a slightly different set of filters, but the overall principle is always the same: here’s a long list of items, find the items that match these rules. Let’s start by expanding our data model to include a couple of new fields: const bookmarks = [ { "url": "https://estherschindler.medium.com/the-old-family-photos-project-lessons-in-creating-family-photos-that-people-want-to-keep-ea3909129943", "title": "Lessons in creating family photos that people want to keep, by Esther Schindler (2018)", "tags": ["photography", "preservation"], "publicationYear": "2018" }, … ]; Then we can define some filters we might use to narrow the list: const bookmarkFilters = [ { id: 'tag', label: 'tagged with', filterFn: (bookmark, tagName) => bookmark.tags.includes(tagName), }, { id: 'publicationYear', label: 'published in', filterFn: (bookmark, year) => bookmark.publicationYear === year, }, ]; Each filter has three fields: id matches the name of the associated URL query parameter label is how the filter will be described in the list of applied filters filterFn is a function that takes two arguments: a bookmark, and a filter value, and returns true/false depending on whether the bookmark matches this filter This list is the only place where I need to customise the filters for a particular site; the rest of the filtering code is completely generic. This means there’s only one place I need to make changes if I want to add or remove filters. The next piece of the filtering code is a generic function that filters a list of items, and takes the list of filters as an argument: /* * Filter a list of items. * * This function takes the list of items and available filters, and the * URL query parameters passed to the page. * * This function returns a list with the items that match these filters, * and a list of filters that have been applied. */ function filterItems({ items, filters, params }) { // By default, all items match, and no filters are applied. var matchingItems = items; var appliedFilters = []; // Go through the URL query params one by one, and look to // see if there's a matching filter. for (const [key, value] of params) { console.debug(`Checking query parameter ${key}`); const matchingFilter = filters.find(f => f.id === key); if (typeof matchingFilter === 'undefined') { continue; } // There's a matching filter! Go ahead and filter the // list of items to only those that match. console.debug(`Detected filter ${JSON.stringify(matchingFilter)}`); matchingItems = matchingItems.filter( item => matchingFilter.filterFn(item, value) ); // Construct a new query string that doesn't include // this filter. const altQuery = new URLSearchParams(params); altQuery.delete(key, value); const linkToRemove = "?" + altQuery.toString(); appliedFilters.push({ type: matchingFilter.id, label: matchingFilter.label, value, linkToRemove, }) } return { matchingItems, appliedFilters }; } This function doesn’t care what sort of items I’m passing, or what the actual filters are, so I can reuse it between different sites. It returns the list of matching items, and the list of applied filters. The latter allows me to show that list on the page. linkToRemove is a link to the same page with this filter removed, but keeping any other filters. This lets us provide a button that removes the filter. The final step is to wire this filtering into the page render. We need to make sure we only show items that match the filter, and show the user a list of applied filters. Here’s the new code: <script> window.addEventListener("DOMContentLoaded", () => { const params = new URLSearchParams(window.location.search); const { matchingItems: matchingBookmarks, appliedFilters } = filterItems({ items: bookmarks, filters: bookmarkFilters, params: params, }); document.querySelector("#appliedFilters").innerHTML = appliedFilters .map(f => `<li>${f.label}: ${f.value} <a href="${f.linkToRemove}">(remove)</a></li>`) .join(""); document.querySelector("#listOfBookmarks").innerHTML = matchingBookmarks.map(Bookmark).join(""); }); </script> <h1>Bookmarks</h1> <p>Applied filters:</p> <ul id="appliedFilters"></ul> <p>Bookmarks:</p> <ul id="listOfBookmarks"></ul> I stick to simple filters that can be phrased as a yes/no question, and I rely on my past self to have written sufficiently useful metadata. At least in static sites, I’ve never implemented anything like a fuzzy text search, where it’s less obvious whether a particular item should match. You can check out the filtering code on the demo page. demo) The next feature I usually implement is sorting. I build a dropdown menu with all the options, and picking one reloads the page with the new sort order. Here’s a quick design sketch: For example, I often sort by the date I saved an item, so I can find an item I saved recently. Another sort order I often use is “random”, which shuffles the items and is a fun way to explore the data. As with filters, I put the current sort order in a query parameter, for example: bookmarks.html?sortOrder=titleAtoZ As before, I want to write this in a generic way and share code between different sites. Let’s start by defining a list of sort options: const bookmarkSortOptions = [ { id: 'titleAtoZ', label: 'title (A to Z)', compareFn: (a, b) => a.title > b.title ? 1 : -1, }, { id: 'publicationYear', label: 'publication year (newest first)', compareFn: (a, b) => Number(b.publicationYear) - Number(a.publicationYear), }, ]; Each sort option has three fields: id is the value that will appear in the URL query parameter label is the human-readable label that will appear in the dropdown compareFn(a, b) is a function that compares two items, and will be passed directly to the JavaScript sort function. If it returns a negative value, then a sorts before b. If it returns a positve value, then a sorts after b. Next, we can define a function that will sort a list of items: /* * Sort a list of items. * * This function takes the list of items and available options, and the * URL query parameters passed to the page. * * It returns a list with the items in sorted order, and the * sort order that was applied. */ function sortItems({ items, sortOptions, params }) { // Did the user pass a sort order in the query parameters? const sortOrderId = getSortOrder(params); // What sort order are we using? // // Look for a matching sort option, or use the default if the sort // order is null/unrecognised. For now, use the first defined // sort order as the default. const defaultSort = sortOptions[0]; const selectedSort = sortOptions.find(s => s.id === sortOrderId) || defaultSort; console.debug(`Selected sort: ${JSON.stringify(selectedSort)}`); // Now apply the sort to the list of items. const sortedItems = items.sort(selectedSort.compareFn); return { sortedItems, appliedSortOrder: selectedSort }; } /* Get the current sort order from the URL query parameters. */ function getSortOrder(params) { return params.get("sortOrder"); } This function works with any list of items and sort orders, making it easy to reuse across different sites. I only have to define the list of sort orders once. This approach makes it easy to add new sort orders, and to write a component that renders a dropdown menu to pick the sort order: /* * Create a dropdown control to choose the sort order. When you pick * a different value, the page reloads with the new sort. */ function SortOrderDropdown({ sortOptions, appliedSortOrder }) { return ` <select onchange="setSortOrder(this.value)"> ${ sortOptions .map(({ id, label }) => ` <option value="${id}" ${id === appliedSortOrder.id ? 'selected' : ''}> ${label} </option> `) .join("") } </select> `; } function setSortOrder(sortOrderId) { const params = new URLSearchParams(window.location.search); params.set("sortOrder", sortOrderId); window.location.search = params.toString(); } Finally, we can wire the sorting code into the rest of the app. After filtering, we sort the items and then render the sorted list. We also show the sort controls on the page: <script> window.addEventListener("DOMContentLoaded", () => { const params = new URLSearchParams(window.location.search); const { matchingItems: matchingBookmarks, appliedFilters } = filterItems(…); … const { sortedItems: sortedBookmarks, appliedSortOrder } = sortItems({ items: matchingBookmarks, sortOptions: bookmarkSortOptions, params, }); document.querySelector("#sortOrder").innerHTML += SortOrderDropdown({ sortOptions: bookmarkSortOptions, appliedSortOrder }); document.querySelector("#listOfBookmarks").innerHTML = sortedBookmarks.map(Bookmark).join(""); }); </script> <p id="sortOrder">Sort by:</p> You can check out the sorting code on the demo page. demo) If you have a really long list of items, you may want to break them into multiple pages. This isn’t something I do very often. Modern web browsers are very performant, and you can put thousands of elements on the page without breaking a sweat. I’ve only had to add pagination in a couple of very image-heavy sites – if it’s a text-based site, I just show everything. (You may notice that, for example, there are no paginated lists anywhere on this site. By writing lean HTML, I can fit all my lists on a single page.) If I do want pagination, I stick to a classic design: As with other features, I use a URL query parameter to track the current page number: bookmarks.html?pageNumber=2 This code can be written in a completely generic way – it doesn’t have to care what sort of items we’re paginating. First, let’s write a function that will select a page of items for us. If we’re on page N, what items should we be showing? /* * Get a page of items. * * This function will reduce the list of items to the items that should * be shown on this particular page. */ function paginateItems({ items, pageNumber, pageSize }) { // Page numbers are 1-indexed, so page 1 corresponds to // the indices 0…(pageSize - 1). const startOfPage = (pageNumber - 1) * pageSize; const endOfPage = pageNumber * pageSize; const thisPage = items.slice(startOfPage, endOfPage); return { thisPage, totalPages: Math.ceil(items.length / pageSize), }; } In some of my sites, the page size is a suggestion rather than a hard rule. If there are 27 items and the page size is 25, I think it’s nicer to show all the items on one page than push a few items onto a second page which barely has anything on it. But that might reflect my general dislike of pagination, and it’s definitely a nice-to-have rather than a required feature. Once we know what page we’re on and how many pages there are, we can create a component to render some basic pagination controls: /* * Renders a list of pagination controls. * * This includes links to prev/next pages and the current page number. */ function PaginationControls({ pageNumber, totalPages, params }) { // If there are no pages, we don't need pagination controls. if (totalPages === 1) { return ""; } // Do we need a link to the previous page? Only if we're past page 1. if (pageNumber > 1) { const prevPageUrl = setPageNumber({ params, pageNumber: pageNumber - 1 }); prevPageLink = `<a href="${prevPageUrl}">← prev</a>`; } else { prevPageLink = null; } // Do we need a link to the next page? Only if we're before // the last page. if (pageNumber < totalPages) { const nextPageUrl = setPageNumber({ params, pageNumber: pageNumber + 1 }); nextPageLink = `<a href="${nextPageUrl}">next →</a>`; } else { nextPageLink = null; } const pageText = `Page ${pageNumber} of ${totalPages}`; // Construct the final result. return [prevPageLink, pageText, nextPageLink] .filter(p => p !== null) .join(" / "); } /* Returns a URL that points to the new page number. */ function setPageNumber({ params, pageNumber }) { const updatedParams = new URLSearchParams(params); updatedParams.set("pageNumber", pageNumber); return `?${updatedParams.toString()}`; } Finally, let’s wire this code into the rest of the app. We get the page number from the URL query parameters, paginate the list of filtered and sorted items, and show some pagination controls: <script> /* Get the current page number. */ function getPageNumber(params) { return Number(params.get("pageNumber")) || 1; } window.addEventListener("DOMContentLoaded", () => { const params = new URLSearchParams(window.location.search); const { matchingItems: matchingBookmarks, appliedFilters } = filterItems(…); const { sortedItems: sortedBookmarks, appliedSortOrder } = sortItems(…); const pageNumber = getPageNumber(params); const { thisPage: thisPageOfBookmarks, totalPages } = paginateItems({ items: sortedBookmarks, pageNumber, pageSize: 25, }); document.querySelector("#paginationControls").innerHTML += PaginationControls({ pageNumber, totalPages, params }); document.querySelector("#listOfBookmarks").innerHTML = thisPageOfBookmarks.map(Bookmark).join(""); }); </script> <p id="paginationControls">Pagination controls: </p> One thing that makes pagination a little tricky is that it affects filtering and sorting as well – when you change either of those, you probably want to reset to the first page. For example, if you’re filtering for animals and you’re on page 3, then you add a second filter for giraffes, you should reset to page 1. If you stay on page 3, it might be confusing if there are less than 3 pages of results with the new filter. The key to this is calling params.delete("pageNumber") when you update the URL query parameters. You can play with the pagination on the demo page. demo 1, demo 2) One problem with relying on JavaScript to render the page is that sometimes JavaScript goes wrong. For example, I write a lot of my metadata by hand, and a typo can create invalid JSON and break the page. There are also people who disable JavaScript, or sometimes it just doesn’t work. If I’m using the site, I can open the Developer Tools in my web browser and start debugging there – but that’s not a great experience. If you’re not expecting something to go wrong, it will just look like the page is taking a long time to load. We can do better. To start, we can add a <noscript> element that explains to users that they need to enable JavaScript. This will only be shown if they’ve disabled JavaScript: <noscript> <strong>You need to enable JavaScript to use this site!</strong> </noscript> I have a demo page which disables JavaScript, so you can see how the noscript tag behaves. This won’t help if JavaScript is broken rather than disabled, so we also need to add error handling. We can listen for the error event on the window, and report an error to the user – for example, if a script fails to load. <div id="errors"></div> <script> window.addEventListener("error", function(event) { document .querySelector('#errors') .innerHTML = `<strong>Something went wrong when loading the page!</strong>`; }); </script> We can also attach an onerror handler to specific script tags, which allows us to customise the error message – we can tell the user that a particular file failed to load. <script src="app.js" onerror="alert('Something went wrong while loading app.js')"></script> I have another demo page which has a basic error handler. Finally, I like to include a loading indicator, or some placeholder text that will be replaced when the page will finish loading – this tells the user where they can expect to see something load in. <ul id="listOfBookmarks">Loading…</ul> It’s somewhat rare for me to add a loading indicator or error handling, just because I’m the only user of my static sites, and it’s easier for me to use the developer tools when something breaks. But providing mechanisms for the user to understand what’s going on is crucial if you want to build static sites like this that other people will use. Test the code with QUnit and Playwright If I’m writing a very complicated viewer, it’s helpful to have tests. I’ve found two test frameworks that I particularly like for this purpose. QUnit is a JavaScript library that I use for unit testing – to me, that means testing individual functions and components. For example, QUnit was very helpful when I was writing the early iterations of the sorting and filtering code, and writing tests caught a number of mistakes. You can run QUnit in the browser, and it only requires two files, so I can test a project without creating a whole JavaScript build system or dependency tree. Here’s an example of a QUnit test: QUnit.test("sorts bookmarks by title", function(assert) { // Create three bookmarks with different titles const bookmarkA = { title: "Almanac for apples" }; const bookmarkC = { title: "Compendium of coconuts" }; const bookmarkP = { title: "Page about papayas" }; const params = new URLSearchParams("sortOrder=titleAtoZ"); // Pass the bookmarks in the wrong order, so they can't be sorted // correctly "by accident" const { sortedItems, appliedSortOrder } = sortItems({ items: [bookmarkC, bookmarkA, bookmarkP], sortOptions: bookmarkSortOptions, params, }); // Check the bookmarks have been sorted in the right order assert.deepEqual(sortedItems, [bookmarkA, bookmarkC, bookmarkP]); }); You can see this test running in the browser in my demo page. Playwright is a testing library that can open a web app in a real web browser, interact with the page, and check that the app behaves correctly. It’s often used for dynamic web apps, but it works just as well for static pages. For example, it can test that if you select a new sort order, the page reloads and show results in the correct order. Here’s an example of a simple test written with Playwright in Python: from playwright.sync_api import expect, sync_playwright with sync_playwright() as p: browser = p.webkit.launch() # Open the HTML file in the browser page = browser.new_page() page.goto('file:///Users/alexwlchan/Sites/sorting.html') # Look for an <li> element with one of the bookmarks -- this will # only appear if the page has rendered correctly. expect(page.get_by_text("So Many Nevers")).to_be_visible() browser.close() These tools are a great safety net for catching mistakes, but I don’t always need them. I only write tests for my more complicated sites – when the sorting/filtering code is particularly complex, there’s a lot of rendering code, or I anticipate making major changes in future. I don’t bother with tests when the site is simple and unlikely to change, and I can just do manual checks when I write it the first time. Tests are less useful if I know I’ll never make changes. This is getting away from the idea of a self-contained static website, because now I’m relying on third-party code, and for Playwright I need to maintain a working Python environment. I’m okay with this, because the website is still usable even if I can no longer run the tests. These are useful sidecar tools, but I only need them if I’m making changes. If I finish a site and I know I won’t change it again, I don’t need to worry about whether the tests will still work years later. Manipulate the metadata with Python For small sites, we could write all this JavaScript directly in <script> tags or in a single file. As we get more data, splitting the metadata and application logic makes everything easier to manage. One pattern I’ve adopted is to put all the item metadata into a single, standalone JavaScript file that assigns a single variable: const bookmarks = […]; and then load that file in the HTML page with a <script src="metadata.js"> element. I use JavaScript rather than pure JSON because browsers don’t allow fetching local JSON files via file://. If you open an HTML page without a web server, the browser will block requests to fetch a JSON file because of security restrictions. By storing data in a JavaScript file instead, I can load it with a simple <script> tag. I wrote a small Python library javascript-data-files that lets me interact with JSON stored this way. This allows me to write scripts that add data to the metadata file (like saving a new bookmark) or to verify the existing metadata (like checking that I have an archived copy of every bookmark). I’ll write more about this in future posts, because this one is long enough already. For example, let’s add a new bookmark to the metadata.js file: from javascript_data_files import read_js, write_js bookmarks = read_js("metadata.js", varname="bookmarks") bookmarks.append({ "url": "https://www.theguardian.com/lifeandstyle/2019/jan/13/ella-risbridger-john-underwood-friendship-life-new-family", "title": "When my world fell apart, my friends became my family, by Ella Risbridger (2019)" }) write_js("metadata.js", varname="bookmarks", value=bookmarks) We’re starting to blur the line between a static site and a static site generator. These scripts only work if I have a working Python environment, which is less future-proof than pure HTML. I’m happy with this compromise, because the website is fully functional without them – I only need to run these scripts if I’m modifying the metadata. If I stop making changes and the Python environment breaks, I can still read everything I’ve already saved. Store the website code in Git I create Git repositories for all of my local websites. This allows me to track changes, and it means I can experiment freely – I can always roll back if I break something. These Git repositories only live on my local machine. I run git init . in the folder, I create commits to record any changes, and that’s it. I don’t push the repository to GitHub or another remote Git server. (Although I do have backups of every site, of course.) Git has a lot of features for writing code in a collaborative environment, but I don’t need any of those here – I’m the only person working on these sites. Most of the time, I just use two commands: $ git add bookmarks.html $ git commit -m "Add filtering by author" This creates a labelled snapshot of my latest changes to bookmarks.html. I only track the text files in Git – the HTML, CSS, and JavaScript. I don’t track binary files like images and videos. Git struggles with those larger files, and I don’t edit those as much as the text files, so having them in version control is less useful. I write a gitignore file to ignore all of them. Closing thoughts There are lots of ideas here, but you don’t need to use all of them – most of my sites only use a few. Every site is different, and you can pick what makes most sense for your project. If you’re building a static site for a tiny archive, start with a simple HTML file. Add features like templates, sorting, and filtering incrementally as they become useful. You don’t need to add them all upfront – that can make things more complicated than they need to be. This approach can scale from simple collections to sophisticated archives. A static website built with HTML and JavaScript is easy to maintain and modify, has no external dependencies, and is future-proof against a lot of technological changes. I’ve come to love using static websites to store my local data. They’re flexible, resilient, and surprisingly powerful. I hope you’ll consider it too, and that these ideas help you get started. [If the formatting of this post looks odd in your feed reader, visit the original article]
Intellectual property is a really dumb idea. “But piracy is theft. Clean and simple. It’s smash and grab. It ain’t no different than smashing a window at Tiffany’s and grabbing merchandise.” - Joe Biden, 46th president of the USA Except it isn’t and Joe Biden is a senile moron. Because when you smash the windows and grab the stuff, Tiffany’s no longer has the stuff. With piracy, everyone has the stuff. It’s a lot more like taking a picture, which Tiffany’s probably encourages. Win-win cooperation. Wealth is being increasingly concentrated. What’s shocking to me is how much everyone still cares about money. Even the die-hard complain about capitalism type deeply cares, because the opposite of love isn’t hate, it’s indifference. I hate scammers, but I’m pretty indifferent to money. The best outcome of AI is if it delivers huge amounts of value to society but no profit to anyone. The old days of the Internet were this goldmine. The Internet delivered huge value but no profit, and that’s why it was good. Suddenly we had all these new powers. Then people figured out how to monetize it. It was a race to extract every tiny bit of value, and now we have today’s Internet. Can this play out differently with AI? Let’s build technology and open source software that market breaks everything. Let’s demoralize the scammers so hard that they don’t even try. Every loser and grifter will be gone from technology because there’s nothing to be gained there. They can play golf all day or something. If I ever figure out how to channel power like Elon, I will do this. Spin up open source projects in every sector to eliminate all the capturable value. This is what I’m trying to do with comma.ai and tinygrad. I dream of a day when company valuations halve when I create a GitHub repo. Someday.
