More from ntietz.com blog - technically a blog
It's kind of dark times right now. And I'm definitely only talking about the days being short. It's pretty dark out right now, since it's the winter in the northern hemisphere. Every year, I start to realize somewhere around January that I'm tired, really tired, and don't want to do anything. I'm not sure that this is seasonal affective disorder, but I did start to explore it and found that bright light makes me feel a lot better in the winter. The problem is, I go through this discovery every year, it seems. My earlier attempts After I first learned that bright light can actually help you feel better in the winter, I got a Happy Light. That's the actual branding on it, and it might have helped, maybe? But it was really inconvenient to use. I got one of the more affordable ones, which meant a small panel I had to sit uncomfortably close to. And I was supposed to use it almost first thing in the morning for 30 minutes. That's... really not going to fit in my routine, especially now that I have two young kids who are raring to go right at 7am, and they do not want to deal with mom sitting in front of a lamp that early. So I just kind of stopped using it, and would poke at it in the drawer occasionally but not really do anything with it. Instead, I got a few more lamps in my office. These seemed to help, a little, but that might just be that I like my spaces well lit1. Going as bright as possible Somewhere along the line I saw a post talking about winter energy and light. And the author suggested that indoor lights just have to be bright. They recommended some high-watt LED bulbs. I went ahead and ordered some and... these are very bright. They're 100 watts of LEDs each, and I have two of them. I plugged them in to these plug-in fixtures that have inline switches. These seemed to really help me out a lot. On days I used them, I was more energetic and felt better. But I had a lot of days when I didn't use them, because using them was inconvenient. The main problems were that I had to turn on two switches manually, and they were too bright to look at. Turning them on doesn't sound like a lot, but doing it every day when you're already struggling can be just enough friction to avoid doing it! The brightness was the biggest issue, because they were blinding to look at and cast this reverse shadow out of my office. The brightness was the point, so now do I deal with that? Shading it Turning on the switches was easy. I put them on some smart outlets, set a schedule, and now they turn on and off at my predetermined times! Then I had to make a shade for the lamps. My main goal here is to diffuse the light a bit, so I was thinking of a small wooden frame wrapped in a light white fabric. That would spread the light out and avoid having any spots that are too bright to look at. I took measurements for where it was going to fit, then headed out to my workshop. Then I ripped a scrap board into 0.75"x0.75" stock. I ended up with just enough for the design. This is what I ended up with, after it was glued up. Here's a bonus detail shot of the extremely unnecessary joinery here. But hey, this joinery was incredibly satisfying, and the whole thing went together without any nails or screws. Joy makes it worthwhile. And then with the fabric attached2, it ended looking pretty nice. In its intended home, you can see that it does its job! It's now not painful to look straight at the lamp. You wouldn't want to do it, but it won't hurt you at least! Waiting for summer Of course, we need these bright lights during the winter, during the dark days. But summer will come again. Not all of us will see the long days. We'll lose some to the winter. But summer will come again, and we'll be able to go outside freely. Soak in the sun. Be ourselves. Dance in sunshowers. It's hard to see that in the darkest of times. I'm writing this for myself as much as I'm writing it for you. But believe me: you are strong, and I am strong, and together, we will survive. It seems like this winter is going to last a long time. And it probably will. But I'm not just talking about the winter, and the dark times are going to end, eventually. 1 Most places are very poorly lit and don't have enough lamps. You should have a lot of lamps in every room to make it as bright and cheery as possible! I refuse to believe otherwise, but my wife, a gremlin of the darkness3, does like rooms quite dim. 2 My wife helped with this step. She works with fabric a lot, and this was our first project combining our respective crafts. Certainly not our last! 3 She approved this phrasing.
Recently, a reader wrote to me and asked about my writing process and burnout. They had an image in their head that I could sit down at a computer and type up a full post on a given topic, but were unsure if that's the right approach when they start blogging. And they were concerned about how to keep things sustainable for themselves, too. I started to write back to them, but decided to ship my answer for everyone else, as well. Now, to be clear: this is my writing process. There are many other very valid approaches, such as what Gabriella wrote. But I think outlining my process here will at least help break some of the mystique around writing, because I certainly do not just sit down and plunk out a post. Well, it often looks like that if you're an outside observer1. But that's only what's visible. How I write The very first piece of any blog post for me is an idea, some inspiration, something I want to share. These ideas come from a variety of places. Some of them are from conversations with friends or coworkers. Others come from problems I'm solving in my own code. Or they come from things I'm reading. Very rarely, they come from just thinking about things, but those are other sources with the connection hidden by the passage of time. After an idea comes to me, I try to capture it quickly. If I don't, it's probably gone! I store these ideas in a note in Obsidian, which currently has 162 ideas in it2. I format this file as a bulleted list, with minimal categorization so that the friction for adding something new is as low as possible. When I revisit this list to plan things to write (I do this every few weeks), I'll remove things which are either completed or which I'm confident I am not interested in anymore. Ideas which are promoted from "want to write" to "definitely going to write" then get moved into my primary task tracking system, LunaTask. When I start typing an article, I move it from "later" or "next" to "in progress". But in reality, I start writing an article far before I start typing it. See, the thing is, I'm almost always thinking about topics I'm going to write about. Maybe this is an element of having ADHD, or maybe it's from my obsessive interests and my deep diving into things I'm curious about. But it started happening more when I started writing on a schedule. By writing on a schedule, having to publish weekly, I've gotten in the habit of thinking about things so that I can hit the ground running when I sit down to type. Once I've picked out an idea, then it sits in the back of my head and some background processing happens. Usually I'm not crystal clear on what I want to say on a topic, so that background process churns through it. After figuring out what I want to say, I move onto a little more active thought on how I want to say it and structure it. This happens while I'm on a run or a walk, while I'm on coffee breaks, between exercises I'm working on for my music lessons. There's a lot of background processing between snatches of active effort. Piece by piece, it crystallizes in my mind and I figure out what I want to say and how to say it. Then, finally, I sit at the keyboard. And this is where it really looks like I just sit down and write posts in one quick sitting. That's because by then, the ideas are fully formed and it's a matter of writing the words out—and after a few years of that, it's something I got pretty quick at. Take this blog post for example. I received the initial email on January 1st, and it sat in my inbox for a few days before I could read it and digest it. I started to write a reply, but realized it would work as a blog post. So it ended up in background processing for a week, then I started actively thinking about what I wanted to write. At that point, I created a blog post file with section headings as an outline. This post ended up being delayed a bit, since I had to practice content-driven development for a project that has a self-imposed deadline. But the core writing (though not the typing) was done in the week after opening the original email. And then when I finally sat down to write it, it was two 20-minute sessions of typing out the words3. Editing and publishing After the first draft of a post is written, there's a lot of detail work remaining. This draft is usually in the approximate shape of the final post, but it may be missing sections or have too much in some. I'll take this time to do a quick read over it by myself. Here I'm not checking for spelling errors, but checking if the structure is good and if it needs major work, or just copy edits. If I'm unsure of some aspects of a post, I ask for feedback. I'll reach out to one or two friends who I think would have useful feedback and would enjoy reading it early, and ask them if they can read it. This request comes with specific feedback requests, since those are easier to fulfill then a broad "what do you think?" These are something along the lines of "was it clear? what parts stuck out to you positively or negatively? is there anything you found unclear or inaccurate?" Once people get me feedback, I add an acknowledgement for them (with explicit consent for if/how to share their name and where to link). After I'm more sure of the structure and content of a post, I'll go through and do copy edits. I read through it, with a spellchecker enabled4, and fix any misspellings and improve the wording of sentences. Then it's just the final details. I write up the blurb that goes in my newsletter and on social media (Mastodon, Bluesky, and LinkedIn, currently). I make sure the date is correct, the title is a decent one, and the slug mostly matches the title. And I make sure I did do things like acknowledgements and spellchecking. Here's the full checklist I use for every post before publishing. - [ ] Edited? - [ ] Newsletter blurb written? - [ ] Spellchecked? - [ ] Is the date correct? - [ ] Is the title updated? - [ ] Does the slug match the title? - [ ] Are tags set? - [ ] Draft setting removed? - [ ] Are acknowledgements added? Once that checklist is complete, it's ready to go! That means that on Monday morning, I just have to run make deploy and it'll build and push out the article. Making it sustainable I've been writing at least one post a week since September 2022—over two years at this point, and about 200,000 words. I've avoided burning out on it, so far. That's an interesting thing, since I've burned out on personal projects in the past. What makes this different? One aspect is that it's something that's very enjoyable for me. I like writing, and the process of it. I've always liked writing, and having an outlet for it is highly motivating for me. I thrive in structure, so the self-imposed structure of publishing every week is helpful. This structure makes me abandon perfection and find smaller pieces that I can publish in a shorter amount of time, instead of clinging to a big idea and shipping all of it at once, or never. This is where content-driven development comes in for me, and the consistent schedule also leads me to create inspiration. I also don't put pressure on myself besides the deadline. That's the only immovable thing. Everything else—length, content, even quality5—is flexible. Some of my earliest posts from 2022 are of the form "here's what I did this week!" which got me into a consistent rhythm. The most important thing for me is to keep it fun. If I'm not finding something fun, I just move on to a different post or a different topic. I have some posts that would be good but have sat in my backlog for over a year, waiting to be written. Whenever I sit down to write those, my brain decides to find a bunch of other, better, posts to write instead! So far, it's been fun and motivating to write weekly. If that ever stops, and it becomes a job and something I dread, I'll change the format to be something that works better for me instead! 1 If this is what it looks like then hey, stop peeking through my window. 2 I keep them in a bulleted list, each starting with -, so I can find the count via: grep "\s*-" 'Blog post ideas.md' | wc -l 3 Drafting documents is where I feel like my typing speed is actually a real benefit. On a typing test this evening, I got 99% accuracy at 120wpm. This lets me get through a lot of words quickly. This is a big plus for me both as a writer and as a principal engineer. 4 I leave my spellchecker disabled the vast majority of the time, since the squiggles are distracting! It's not a big deal to misspell things most of the time, and I'd rather be able to focus on getting words out than noticing each mistake as I make it—talk about discouraging! 5 That said, I think it's hard to judge your own quality in the moment. As writers, we can be very critical of ourselves. We think we wrote something boring but others find it interesting. Or something we think is quite clever, others find uninspiring. Sometimes, when I push out something I think is lazy and quick and low-quality, I'm surprised to find a very eager reception.
A project I'm working on (which is definitely not my SIGBOVIK submission for this year, and definitely not about computer ergonomics) requires me to use MIDI. And to do custom handling of it. So I need something that receives those MIDI events and handles them. But... I'm going to make mistakes along the way, and a terminal program isn't very interesting for a presentation. So of course, this program also needs a UI. This should be simple, right? Just a little UI to show things as they come in, should be easy, yeah? Hahahaha. Haha. Ha. Ha. Whoops. The initial plan Who am I kidding? There was no plan. I sat down with egui's docs open in a tab and started just writing a UI. After a few false starts with this—it turns out, sitting down without a plan is a recipe for not doing anything at all—I finally asked some friends to talk through it. I had two short meetings with two different friends. Talking it through with them forced me to figure out ahead of time what I wanted, which made the problems much clearer. Laying out requirements Our goal here is twofold: to provide a debug UI to show MIDI messages and connected devices, and to serve as scaffolding for future MIDI shenanigans. A few requirements fall out of this1: Display each incoming MIDI message in a list in the UI Display each connected MIDI device in a list in the UI Allow filtering of MIDI messages by type in the UI Provide a convenient hook to add subscribers, unrelated to the UI, which receive all MIDI messages Allow the subscribers to choose which device categories they receive messages from (these categories are things like "piano," "drums", or "wind synth") Dynamically detect MIDI devices, handling the attachment or removal of devices Minimize duplication of responsibility (cloning data is fine, but I want one source of truth for incoming data, not multiple) Now that we have some requirements, we can think about how this would be implemented. We'll need some sort of routing system. And the UI will need state, which I think should be separate from the state of the core application which handles the routing. That is, I want to make it so that the UI is a subscriber just like all the other subscribers are. Jumping ahead a bit, here's what it looks like when two of my MIDI devices are connected, after playing a few notes. Current architecture The architecture has three main components: the MIDI daemon, the message subscribers, and the GUI. The MIDI daemon is the core that handles detecting MIDI devices and sets up message routing. This daemon owns the connections for each port2, and periodically checks if devices are still connected or not. When it detects a new device, it maps it onto a type of device (is this a piano? a drum pad? a wind synth?) Each new device gets a listener, which will take every message and send it into the global routing. The message routing and device connection handling are done in the same loop, which is fine—originally I was separating them, but then I measured the timing and each refresh takes under 250 microseconds. That's more than fast enough for my purposes, and probably well within the latency requirements of most MIDI systems. The next piece is message subscribers. Each subscriber can specify which type of messages it wants to get, and then their logic is applied to all incoming messages. Right now there's just the GUI subscriber and some debug subscribers, but there'll eventually be a better debug subscriber and there'll be the core handlers that this whole project is written around. (Is this GUI part of a giant yak shave? Maaaaybeeeee.) The subscribers look pretty simple. Here's one that echoes every message it receives (where dbg_recv is its queue). You just receive from the queue, then do whatever you want with that information! std::thread::spawn(move || loop { match dbg_recv.recv() { Ok(m) => println!("debug: {m:?}"), Err(err) => println!("err: {err:?}"), } }); Finally we reach the GUI, which has its own receiver (marginally more complicated than the debug print one, but not much). The GUI has a background thread which handles message receiving, and stores these messages into state which the GUI uses. This is all separated out so we won't block a frame render if a message takes some time to handle. The GUI also contains state, in two pieces: the ports and MIDI messages are shared with the background thread, so they're in an Arc<Mutex>. And there is also the pure GUI state, like which fields are selected, which is not shared and is just used inside the GUI logic. I think there's a rule that you can't bandy around the word "architecture" without drawing at least one diagram, so here's one diagram. This is the flow of messages through the system. Messages come in from each device, go into the daemon, then get routed to where they belong. Ultimately, they end up stored in the GUI state for updates (by the daemon) and display (by the GUI). State of the code This one's not quite ready to be used, but the code is available. In particular, the full GUI code can be found in src/ui.rs. Here are the highlights! First let's look at some of the state handling. Here's the state for the daemon. CTM is a tuple of the connection id, timestamp, and message; I abbreviate it since it's just littered all over the place. pub struct State { midi: MidiInput, ports: Vec<MidiInputPort>, connections: HashMap<String, MidiInputConnection<(ConnectionId, Sender<CTM>)>>, conn_mapping: HashMap<ConnectionId, Category>, message_receive: Receiver<CTM>, message_send: Sender<CTM>, ports_receive: Receiver<Vec<MidiInputPort>>, ports_send: Sender<Vec<MidiInputPort>>, } And here's the state for the GUI. Data that's written once or only by the GUI is in the struct directly, and anything which is shared is inside an Arc<Mutex>. /// State used to display the UI. It's intended to be shared between the /// renderer and the daemon which updates the state. #[derive(Clone)] pub struct DisplayState { pub midi_input_ports: Arc<Mutex<Vec<MidiInputPort>>>, pub midi_messages: Arc<Mutex<VecDeque<CTM>>>, pub selected_ports: HashMap<String, bool>, pub max_messages: usize, pub only_note_on_off: Arc<AtomicBool>, } I'm using egui, which is an immediate-mode GUI library. That means we do things a little differently, and we build what is to be rendered on each frame instead of retaining it between frames. It's a model which is different from things like Qt and GTK, but feels pretty intuitive to me, since it's just imperative code! This comes through really clearly in the menu handling. Here's the code for the top menu bar. We show the top panel, and inside it we create a menu bar. Inside that menu bar, we create menu buttons, which have an if statement for the click hander. egui::TopBottomPanel::top("menu_bar_panel").show(ctx, |ui| { egui::menu::bar(ui, |ui| { ui.menu_button("File", |ui| { if ui.button("Quit").clicked() { ctx.send_viewport_cmd(egui::ViewportCommand::Close); } }); ui.menu_button("Help", |ui| { if ui.button("About").clicked() { // TODO: implement something } }); }); }); Notice how we handle clicks on buttons. We don't give it a callback—we just check if it's currently clicked and then take action from there. This is run each frame, and it just... works. After the top menu panel, we can add our left panel3. egui::SidePanel::left("instrument_panel").show(ctx, |ui| { ui.heading("Connections"); for port in ports.iter() { let port_name = // ...snip! let conn_id = port.id(); let selected = self.state.selected_ports.get(&conn_id).unwrap_or(&false); if ui .add(SelectableLabel::new(*selected, &port_name)) .clicked() { self.state.selected_ports.insert(conn_id, !selected); } } }); Once again, we see pretty readable code—though very unfamiliar, if you're not used to immediate mode. We add a heading inside the panel, then we iterate over the ports and for each one we render its label. If it's selected it'll show up in a different color, and if we click on it, the state should toggle4. The code for displaying the messages is largely the same, and you can check it out in the repo. It's longer, but only in tedious ways, so I'm omitting it here. What's next? Coming up, I'm going to focus on what I set out to in the first place and write the handlers. I have some fun logic to do for different MIDI messages! I'm also going to build another tab in this UI to show the state of those handlers. Their logic will be... interesting... and I want to have a visual representation of it. Both for presentation reasons, and also for debugging, so I can see what state they're in while I try to use those handlers. I think the next step is... encoding bytes in base 3 or base 45, then making my handlers interpret those. And also base 7 or 11. And then I'll be able to learn how to use this weird program I'm building. Sooo this has been fun, but back to work! 1 I really try to avoid bulleted lists when possible, but "list of requirements" is just about the perfect use case for them. 2 A MIDI port is essentially a physically connected, available device. It's attached to the system, but it's not one we're listening to yet. 3 The order you add panels matters for the final rendering result! This can feel a little different from what we're used to in a declarative model, but I really like it. 4 This is debounced by egui, I think! Otherwise it would result in lots of clicks, since we don't usually hold a button down for precisely the length of one frame render. 5 or in bass drum
Nearly every line of code I write for fun is in Rust. It's not because I need great performance, though that's a nice benefit. I write a lot of Rust because it's a joy to write code in. There is so much else to love about Rust beyond going fast without segfaults. Here are a few of my favorite things about it. Note that these are not unique to Rust by any stretch! Other languages have similar combinations of features. Expressive type safety There are two aspects of Rust's type system that I really enjoy: type safety and expressiveness. I got a taste of this expressiveness back when I learned Haskell, and had been seeking it. I found it in Rust. One of the other languages I use a fair amount at work1 is Go, and its type system is much harder for me to express ideas in. You can do it, but you're not getting the type system's help. Rust lets you put your design straight into types, with enums and structs and traits giving you a lot of room to maneuver. All the while, it's also giving you good type safety! I can express a lot in Python, but I don't trust the code as much without robust tests. You don't have a compiler checking your work! It's remarkably helpful having Rust's compiler by your side, making sure that you're using types correctly and satisfying constraints on things. To call back to data races, the type system is one of the reasons we can prevent those! There are traits that tell you whether or not data is safe to send to another thread or to share with another thread. If your language doesn't have the equivalent of these traits, then you're probably relying on the programmer to ensure those properties! That said, Rust's type system isn't an unmitigated good for me. It can take longer to get something up and running in Rust than in Python, for example, because of the rigidity of the type system: satisfy it or you don't run. And I find a lot of Rust that uses generics is very hard to read, feeling like it is a soup of traits. What we make generic is an implementation question, and a cultural question, so that isn't necessarily inherent to the language but does come strongly bundled to it. It doesn't crash out as much Okay, I have a beef with Go. They included Tony Hoare's "billion-dollar mistake": null pointers. Go gives you pointers, and they can be null2! This means that you can try to invoke methods on a null pointer, which can crash your program. In contrast, Rust tries very very hard to make you never crash. You can make null pointers, but you have to use unsafe and if you do that, well, you're taking on the risk. If you have something which is nullable, you'd use an Option of it and then the type system will make sure you handle both cases. The places you typically see crashes in Rust are when someone either intentionally panics, for an unrecoverable error, or when they unintentionally panic, if they use unwrap on an Option or Result. It's better to handle the other case explicitly. Fortunately, you can configure the linter, clippy, to deny code that uses unwrap (or expect)! If you add this to your Cargo.toml file, it will reject any code which uses unwrap. [lints.clippy] unwrap_used = "deny" Data race resistance It's so hard to write concurrent code that works correctly. Data races are one of the biggest factors contributing to this. Rust's data race prevention is an incredible help for writing concurrent code. Rust isn't immune to data races, but you have to work harder to make one happen. They're almost trivial to introduce in most languages, but in Rust, it's a lot harder! This happens because of the borrow checker, so it's harder to have multiple concurrent actors racing on the same data. You get more control, when you want With Rust, you know a lot more about what the CPU and memory will be doing than in many other languages. You can know this with C and C++, and newer systems programming languages like Zig. Rust is a little unique, to me, in being notably higher level than these languages while giving you ultimately the same amount of control (if you break glass enough, for some things). You're still subject to the operating system, most of the time, so you can't control the CPU and memory fully, but you get a lot more control than in Python. Or even than Go, another language used when you need good performance. This lets you predict what your code is going to do. You're not going to have surprise pauses for the garbage collector, and you're not going to have the runtime scheduler put some tasks off for a while. Instead, you know (or can determine) when memory will be deallocated. And you ultimately control when threads take tasks (though with async and the Tokio runtime, this gets much muddier and you do lose some of this control). This predictability is really nice. It's useful in production, but it's also just really pleasant and comforting. Mixing functional and imperative Rust lets you write in a functional programming style, and it also lets you write in an imperative programming style. Most idiomatic code tends toward functional style, but there's a lot of code that uses imperative style effectively as well! This is pretty unique in my experience, and I really like it. It means that I, the programmer, can pick the paradigm that best fits the problem at hand at any given moment. Code can be more expressive and clearer for the author and the team working on the code. One of the cool things here, too, is that the two paradigms effectively translate between each other! If you use iterators in Rust, they convert into the same compiled binary as the imperative code. You often don't lose any efficiency from using either approach, and you're truly free to express yourself! Helpful compiler errors A few other languages are renowned for their error message quality—Elm comes to mind. Rust is a standout here, as well. Earlier in my career, I was abused by C++. Besides all the production crashes and the associated stress of that, the compiler errors for it were absolutely inscrutable. If you messed up a template, you'd sometimes get thousands of lines of errors—from one missing semicolon. And those wouldn't tell you what the error was, but rather, what came after the mistake. In contrast, Rust's compiler error messages are usually pretty good at telling you exactly what the error is. They even provide suggestions for how to fix it, and where to read more about the error. Sometimes you get into a funny loop with these, where following the compiler's suggestions will lead to a loop of suggesting you change it another way and never get it to succeed, but that's fine. The fact that they're often useful is remarkable! It's fun! This is the big one for me. It's very subjective! I really like Rust for all the reasons listed above, and many more I've forgotten. There are certainly painful times, and learning to love the borrow checker is a process (but one made faster if you've been abused by C++ before). But on balance, using Rust has been great. It's fun having the ability to go ripping fast even when you don't need to. It's lovely having a type system that lets you express yourself (even if it lets you express yourself too much sometimes). The tooling is a joy. All around, there's so much to love about Rust. The performance and safety are great, but they're the tip of the iceberg, and it's a language worth considering even when you don't need top performance. 1 Go was introduced at work because I advocated for it! I'd probably use Go for fun sometimes if I were not getting enough of it at my day job. It's a remarkably useful language, just not my favorite type system. 2 They're expressed as nil in Go, which is the zero value, and is the equivalent of null elsewhere.
More in programming
One of my recent home organisation projects has been sorting out my LEGO collection. I have a bunch of sets which are mixed together in one messy box, and I’m trying to separate bricks back into distinct sets. My collection is nowhere near large enough to be worth sorting by individual parts, and I hope that breaking down by set will make it all easier to manage and store. I’ve been creating spreadsheets to track the parts in each set, and count them out as I find them. I briefly hinted at this in my post about looking at images in spreadsheets, where I included a screenshot of one of my inventory spreadsheets: These spreadsheets have been invaluable – I can see exactly what pieces I need, and what pieces I’m missing. Without them, I wouldn’t even attempt this. I’m about to pause this cleanup and work on some other things, but first I wanted to write some notes on how I’m creating these spreadsheets – I’ll probably want them again in the future. Getting a list of parts in a set There are various ways to get a list of parts in a LEGO set: Newer LEGO sets include a list of parts at the back of the printed instructions You can get a list from LEGO-owned website like LEGO.com or BrickLink There are community-maintained databases on sites like Rebrickable I decided to use the community maintained lists from Rebrickable – they seem very accurate in my experience, and you can download daily snapshots of their entire catalog database. The latter is very powerful, because now I can load the database into my tools of choice, and slice and dice the data in fun and interesting ways. Downloading their entire database is less than 15MB – which is to say, two-thirds the size of just opening the LEGO.com homepage. Bargain! Putting Rebrickable data in a SQLite database My tool of choice is SQLite. I slept on this for years, but I’ve come to realise just how powerful and useful it can be. A big part of what made me realise the power of SQLite is seeing Simon Willison’s work with datasette, and some of the cool things he’s built on top of SQLite. Simon also publishes a command-line tool sqlite-utils for manipulating SQLite databases, and that’s what I’ve been using to create my spreadsheets. Here’s my process: Create a Python virtual environment, and install sqlite-utils: python3 -m venv .venv source .venv/bin/activate pip install sqlite-utils At time of writing, the latest version of sqlite-utils is 3.38. Download the Rebrickable database tables I care about, uncompress them, and load them into a SQLite database: curl -O 'https://cdn.rebrickable.com/media/downloads/colors.csv.gz' curl -O 'https://cdn.rebrickable.com/media/downloads/parts.csv.gz' curl -O 'https://cdn.rebrickable.com/media/downloads/inventories.csv.gz' curl -O 'https://cdn.rebrickable.com/media/downloads/inventory_parts.csv.gz' gunzip colors.csv.gz gunzip parts.csv.gz gunzip inventories.csv.gz gunzip inventory_parts.csv.gz sqlite-utils insert lego_parts.db colors colors.csv --csv sqlite-utils insert lego_parts.db parts parts.csv --csv sqlite-utils insert lego_parts.db inventories inventories.csv --csv sqlite-utils insert lego_parts.db inventory_parts inventory_parts.csv --csv The inventory_parts table describes how many of each part there are in a set. “Set S contains 10 of part P in colour C.” The parts and colors table contains detailed information about each part and color. The inventories table matches the official LEGO set numbers to the inventory IDs in Rebrickable’s database. “The set sold by LEGO as 6616-1 has ID 4159 in the inventory table.” Run a SQLite query that gets information from the different tables to tell me about all the parts in a particular set: SELECT ip.img_url, ip.quantity, ip.is_spare, c.name as color, p.name, ip.part_num FROM inventory_parts ip JOIN inventories i ON ip.inventory_id = i.id JOIN parts p ON ip.part_num = p.part_num JOIN colors c ON ip.color_id = c.id WHERE i.set_num = '6616-1'; Or use sqlite-utils to export the query results as a spreadsheet: sqlite-utils lego_parts.db " SELECT ip.img_url, ip.quantity, ip.is_spare, c.name as color, p.name, ip.part_num FROM inventory_parts ip JOIN inventories i ON ip.inventory_id = i.id JOIN parts p ON ip.part_num = p.part_num JOIN colors c ON ip.color_id = c.id WHERE i.set_num = '6616-1';" --csv > 6616-1.csv Here are the first few lines of that CSV: img_url,quantity,is_spare,color,name,part_num https://cdn.rebrickable.com/media/parts/photos/9999/23064-9999-e6da02af-9e23-44cd-a475-16f30db9c527.jpg,1,False,[No Color/Any Color],Sticker Sheet for Set 6616-1,23064 https://cdn.rebrickable.com/media/parts/elements/4523412.jpg,2,False,White,Flag 2 x 2 Square [Thin Clips] with Chequered Print,2335pr0019 https://cdn.rebrickable.com/media/parts/photos/15/2335px13-15-33ae3ea3-9921-45fc-b7f0-0cd40203f749.jpg,2,False,White,Flag 2 x 2 Square [Thin Clips] with Octan Logo Print,2335pr0024 https://cdn.rebrickable.com/media/parts/elements/4141999.jpg,4,False,Green,Tile Special 1 x 2 Grille with Bottom Groove,2412b https://cdn.rebrickable.com/media/parts/elements/4125254.jpg,4,False,Orange,Tile Special 1 x 2 Grille with Bottom Groove,2412b Import that spreadsheet into Google Sheets, then add a couple of columns. I add a column image where every cell has the formula =IMAGE(…) that references the image URL. This gives me an inline image, so I know what that brick looks like. I add a new column quantity I have where every cell starts at 0, which is where I’ll count bricks as I find them. I add a new column remaining to find which counts the difference between quantity and quantity I have. Then I can highlight or filter for rows where this is non-zero, so I can see the bricks I still need to find. If you’re interested, here’s an example spreadsheet that has a clean inventory. It took me a while to refine the SQL query, but now I have it, I can create a new spreadsheet in less than a minute. One of the things I’ve realised over the last year or so is how powerful “get the data into SQLite” can be – it opens the door to all sorts of interesting queries and questions, with a relatively small amount of code required. I’m sure I could write a custom script just for this task, but it wouldn’t be as concise or flexible. [If the formatting of this post looks odd in your feed reader, visit the original article]
For some purpose, the DOGE people are burrowing their way into all US Federal Systems. Their complete control over the Treasury Department is entirely insane. Unless you intend to destroy everything, making arbitrary changes to complex computer systems will result in destruction, even if that was not your intention. No
Entering 2025, I decided to spend some time exploring the topic of agents. I started reading Anthropic’s Building effective agents, followed by Chip Huyen’s AI Engineering. I kicked off a major workstream at work on using agents, and I also decided to do a personal experiment of sorts. This is a general commentary on building that project. What I wanted to build was a simple chat interface where I could write prompts, select models, and have the model use tools as appropriate. My side goal was to build this using Cursor and generally avoid writing code directly as much as possible, but I found that generally slower than writing code in emacs while relying on 4o-mini to provide working examples to pull from. Similarly, while I initially envisioned building this in fullstack TypeScript via Cursor, I ultimately bailed into a stack that I’m more comfortable, and ended up using Python3, FastAPI, PostgreSQL, and SQLAlchemy with the async psycopg3 driver. It’s been a… while… since I started a brand new Python project, and used this project as an opportunity to get comfortable with Python3’s async/await mechanisms along with Python3’s typing along with mypy. Finally, I also wanted to experiment with Tailwind, and ended up using TailwindUI’s components to build the site. The working version supports everything I wanted: creating chats with models, and allowing those models to use function calling to use tools that I provide. The models are allowed to call any number of tools in pursuit of the problem they are solving. The tool usage is the most interesting part here for sure. The simplest tool I created was a get_temperature tool that provided a fake temperature for your location. This allowed me to ask questions like “What should I wear tomorrow in San Francisco, CA?” and get a useful respond. The code to add this function to my project was pretty straightforward, just three lines of Python and 25 lines of metadata to pass to the OpenAI API. def tool_get_current_weather(location: str|None=None, format: str|None=None) -> str: "Simple proof of concept tool." temp = random.randint(40, 90) if format == 'fahrenheit' else random.randint(10, 25) return f"It's going to be {temp} degrees {format} tomorrow." FUNCTION_REGISTRY['get_current_weather'] = tool_get_current_weather TOOL_USAGE_REGISTRY['get_current_weather'] = { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "format": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location.", }, }, "required": ["location", "format"], }, } } After getting this tool, the next tool I added was a simple URL retriever tool, which allowed the agent to grab a URL and use the content of that URL in its prompt. The implementation for this tool was similarly quite simple. def tool_get_url(url: str|None=None) -> str: if url is None: return '' url = str(url) response = requests.get(url) soup = BeautifulSoup(response.content, 'html.parser') content = soup.find('main') or soup.find('article') or soup.body if not content: return str(response.content) markdown = markdownify(str(content), heading_style="ATX").strip() return str(markdown) FUNCTION_REGISTRY['get_url'] = tool_get_url TOOL_USAGE_REGISTRY['get_url'] = { "type": "function", "function": { "name": "get_url", "description": "Retrieve the contents of a website via its URL.", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The complete URL, including protocol to retrieve. For example: \"https://lethain.com\"", } }, "required": ["url"], }, } } What’s pretty amazing is how much power you can add to your agent by adding such a trivial tool as retrieving a URL. You can similarly imagine adding tools for retrieving and commenting on Github pull requests and so, which could allow a very simple agent tool like this to become quite useful. Working on this project gave me a moderately compelling view of a near-term future where most engineers have simple application like this running that they can pipe events into from various systems (email, text, Github pull requests, calendars, etc), create triggers that map events to templates that feed into prompts, and execute those prompts with tool-aware agents. Combine that with ability for other agents to register themselves with you and expose the tools that they have access to (e.g. schedule an event with tool’s owner), and a bunch of interesting things become very accessible with a very modest amount of effort: You could schedule events between two busy people’s calendars, as if both of them had an assistant managing their calendar Reply to your own pull requests with new blog posts, providing feedback on typos and grammatical issues Crawl websites you care about and identify posts you might be interested in Ask the model to generate a system model using lethain:systems, run that model, then chart the responses Add a “planning tool” which allows the model to generate a plan to guide subsequent steps in a complex task. (e.g. getting my calendar, getting a friend’s calendar, suggesting a time we could meet) None of these are exactly lifesaving, but each is somewhat useful, and I imagine there are many more fairly obvious ideas that become easy once you have the necessary scaffolding to make this sort of thing easy. Altogether, I think that I am convinced at this points that agents, using current foundational models, are going to create a number of very interesting experiences that improve our day to day lives in small ways that are, in aggregate, pretty transformational. I’m less convinced that this is the way all software should work going forward though, but more thoughts on that over time. (A bunch of fun experiments happening at work, but early days on those.)
A lieutenant colonel in the Soviet Air Defense Forces prevented the end of human civilization on September 26th, 1983. His name was Stanislav Petrov. Protocol dictated that the Soviet Union would retaliate against any nuclear strikes sent by the United States. This was a policy of mutually assured destruction, a doctrine that compels a horrifying logical conclusion. The second and third stage effects of this type of exchange would be even more catastrophic. Allies for each side would likely be pulled into the conflict. The resulting nuclear winter was projected to lead to 2 billion deaths due to starvation. This is to say nothing about those who would have been unfortunate enough to have survived. Petrov’s job was to monitor Oko, the computerized warning systems built to centralize Soviet satellite communications. Around midnight, he received a report that one of the satellites had detected the infrared signature of a single launch of a United States ICBM. While Petrov was deciding what to do about this report, the system detected four more incoming missile launches. He had minutes to make a choice about what to do. It is impossible to imagine the amount of pressure placed on him at this moment. Source: Stanislav Petrov, Soviet officer credited with averting nuclear war, dies at 77 by Schwartzreport. Petrov lived in a world of deterministic systems. The technologies that powered these warning systems have outputs that are guaranteed, provided the proper inputs are provided. However, deterministic does not mean infallible. The only reason you are alive and reading this is because Petrov understood that the systems he observed were capable of error. He was suspicious of what he was seeing reported, and chose not to escalate a retaliatory strike. There were two factors guiding his decision: A surprise attack would most likely have used hundreds of missiles, and not just five. The allegedly foolproof Oko system was new and prone to errors. An error in a deterministic system can still lead to expected outputs being generated. For the Oko system, infrared reflections of the sun shining off of the tops of clouds created a false positive that was interpreted as detection of a nuclear launch event. Source: US-K History by Kosmonavtika. The concept of erroneous truth is a deep thing to internalize, as computerized systems are presented as omniscient, indefective, and absolute. Petrov’s rewards for this action were reprimands, reassignment, and denial of promotion. This was likely for embarrassing his superiors by the politically inconvenient shedding of light on issues with the Oko system. A coerced early retirement caused a nervous breakdown, likely him having to grapple with the weight of his decision. It was only in the 1990s—after the fall of the Soviet Union—that his actions were discovered internationally and celebrated. Stanislav Petrov was given the recognition that he deserved, including being honored by the United Nations, awarded the Dresden Peace Prize, featured in a documentary, and being able to visit a Minuteman Missile silo in the United States. On January 31st, 2025, OpenAI struck a deal with the United States government to use its AI product for nuclear weapon security. It is unclear how this technology will be used, where, and to what extent. It is also unclear how OpenAI’s systems function, as they are black box technologies. What is known is that LLM-generated responses—the product OpenAI sells—are non-deterministic. Non-deterministic systems don’t have guaranteed outputs from their inputs. In addition, LLM-based technology hallucinates—it invents content with no self-knowledge that it is a falsehood. Non-deterministic systems that are computerized also have the perception as being authoritative, the same as their deterministic peers. It is not a question of how the output is generated, it is one of the output being perceived to come from a machine. These are terrifying things to know. Consider not only the systems this technology is being applied to, but also the thoughtless speed of their integration. Then consider how we’ve historically been conditioned and rewarded to interpret the output of these systems, and then how we perceive and treat skeptics. We don’t live in a purely deterministic world of technology anymore. Stanislav Petrov died on September 18th, 2017, before this change occurred. I would be incredibly curious to know his thoughts about our current reality, as well as the increasing abdication of human monitoring of automated systems in favor of notably biased, supposed “AI solutions.” In acknowledging Petrov’s skepticism in a time of mania and political instability, we acknowledge a quote from former U.S. Secretary of Defense William J. Perry’s memoir about the incident: [Oko’s false positives] illustrates the immense danger of placing our fate in the hands of automated systems that are susceptible to failure and human beings who are fallible.
In our *Ambsheets* project, we are exploring a small extension to the familiar spreadsheet: **what if a single spreadsheet cell could hold multiple values at once**?