Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
1

Verifying my web archives with Playwright

from alexwlchan [alt+shift+b] in programming

To protect against link rot and disappearing content, I save local copies of web pages I like into my web archive. Ideally, this archive is completely self-contained. I want a local copy of each page’s HTML and all the assets it depends on (CSS, images, scripts, and so on). I don’t want my archived pages fetching resources from the original website, which could disappear at any time. Initially, I was verifying self-containedness by hand. I’d open each saved page in a web browser, and check the browser dev tools. I’d check the Console tab for errors and failed requests, and watch the Network tab for requests going out to the internet. That approach was tedious and error-prone, and I wanted something better. I now automate the process using Playwright, a testing framework that allows you to control web browsers. I open each page using Playwright, and check if it’s making external requests. Detecting external requests in Playwright When you open a page in Playwright, you can subscribe to request events which tell you when the page issues a request. In this Python snippet, we start Playwright, start a WebKit browser, and open a new page. We tell the page to append external requests to a list, and then load example.com. We see that example.com makes two requests on page load: one for https://example.com, and one for a JavaScript file. from contextlib import closing from playwright.sync_api import sync_playwright with ( sync_playwright() as p, closing(p.webkit.launch()) as browser, closing(browser.new_page()) as page, ): requests = [] page.on("request", lambda msg: requests.append(msg)) page.goto("https://example.com") print("requests:", requests) # requests: [<Request url='https://example.com/' method='GET'>, # <Request url='https://example.com/s.js' method='GET'>] In a more complex scenario, we might use Playwright to interact with the page after loading – for example, clicking a button or filling in a form, and then we could see...
2 weeks ago

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from alexwlchan

Organising my cross-stitch patterns

I have a lot of digital cross-stitch patterns and until recently they were a mess. My “collection” was a series of several hundred PDFs, disorganised and spread across my disk. Recently, I was looking for a pattern and it took me several minutes to find it. I had a brief moment of panic, thinking that I’d lost it, and the website this pattern came from is long since gone. Fortunately it did turn up, but it was clear I had to organise my pattern collection. Some people would use apps like Pattern Keeper or Flosshaven, but I just created another static website on my media drive. This is what it looks like: The patterns are arranged in a masonry layout with a preview thumbnail for each pattern, which is normally the first page of the PDF. If a pattern has multiple files – for example, different files for colour and black-and-white printing – there’s a list of links below the thumbnail. I can also add a source link to where I bought the pattern, if I want to find more patterns by the same author. Internally, it’s structured like my other static websites. The metadata for the patterns is stored in a JavaScript file, and then I have an HTML page which renders that metadata in this visual design. My pattern collection doesn’t change very often, so I’m adding new files by hand. There are no sorting or filtering options, because my collection is small enough that they’re unnecessary. I can scroll through the entire list in less than a minute and find whatever I need. I’d like to start making my own clothes soon, and I expect to copy this design to store clothing patterns. This collection now sits alongside my saved bookmarks, podcasts, TV shows, and more. I can keep all my patterns organised, and I won’t lose them in the depths of my hard drive. Technical tricks There are a couple of technical pieces I’m quite pleased with in this process: Masonry layout. I love the cascading grid style of a masonry layout, and I think they’re great when you have variable-sized content – like my pattern cards. I’ve been using them for over a decade; screenshots of my early web apps show my experiments with David Desandro’s Masonry.js library. Masonry layouts are now a native feature of CSS called “Grid Lanes”, which is simpler and more performant than the JavaScript approach. I’m excited to use these layouts in even more places. Thumbnails of PDFs. Most digital postage patterns are shipped as PDFs, and I’m creating the thumbnail images with my create_thumbnail tool. I had to extend my tool to support PDFs, which turned out to be straightforward with the pdf_oxide crate. I’d not come across Oxide before, but it’s a cross-language library for working with PDFs, and I can imagine using it in future. A coloured stripe for the palette. In the metadata for each project, I have the complete list of DMC thread colours and the pattern dimensions. I have a lookup table that converts DMC thread colours to hex codes, and then I’m able to create a linear gradient in CSS to render a coloured stripe of the pattern colours. Here’s an example from a pattern with four colours: border-bottom: 12px solid transparent; border-image: linear-gradient( to right, #7a451f 0%, #7a451f 25%, #985e33 25%, #985e33 50%, #e4bb8e 50%, #e4bb8e 75%, #361f0e 75%, #361f0e 100% ) 1; Building the HTML, CSS, and JavaScript was very quick – I have notes on how I build my media sites, and I’ve done it plenty of times before. I threw together the initial prototype in less than 15 minutes. The longer part was finding all the PDFs scattered across my disk – but now that’s done, and I know exactly where to find my patterns. [If the formatting of this post looks odd in your feed reader, visit the original article]

13th Oct 2026 • 1 votes
Abusing ID3 chapters to turn videos into glanceable podcasts

I listen to a lot of podcasts, and I like how they fit around other tasks. I press play, lock my phone, and put it down. I’m free to wash the dishes, fold the laundry, or shop for groceries. Unfortunately, more and more information is only published as a video. Technical talks, conference sessions, video essays – they don’t work in an audio-only podcast app. I could convert these videos to MP3 files, but that breaks down the moment a video isn’t pure spoken word. If a speaker says, “Look at this slide” or holds up a diagram, an audio-only file leaves me stranded. I don’t want to give up the podcast player I like, nor stare at a screen for an hour – but I do want the information in these videos. To solve this, I’m abusing my podcast player’s chapter support. This gives me the best of both worlds: I can listen to a video as audio-first, and glance at my lock screen if I need a moment of visual context. The idea: Chapters every few seconds MP3 files can have ID3 metadata, and ID3 metadata can include chapters. A chapter covers a particular time range, and it can have an associated title, description, and cover art. My podcast app of choice is Overcast, which can’t play videos, but it does have robust chapter support. I can jump between chapters, navigate a table of contents, and see per-chapter cover art. To get videos into Overcast, I’m creating MP3 files with a new chapter every few seconds, and the per-chapter cover art is a corresponding frame from the video. As I play the file, I get a slow, stop-motion-like rendition of the original video. If my phone is locked, I can glance at my lock screen and see the current frame in the Now Playing screen. Overcast is developed by Marco Arment, and I got this idea from Forecast, his app for adding chapters to podcasts. In particular, I was struck by its ability to create chapters that don’t display in the chapter list – ideal if I don’t want a table of contents with hundreds of entries. As I was developing my script, I compared my output to the output from Forecast to ensure I was creating the chapters correctly. The code: FFmpeg and Mutagen There are three steps in this process: Convert a video file to an MP3 Extract images from the video at a fixed interval Insert the images as hidden chapters in the MP3 file Let’s go through each in turn. 1. Convert a video file to an MP3 Converting a video file to an MP3 is a single FFmpeg command: ffmpeg -i video.mp4 audio.mp3 This is consistently the slowest step of the process, and I do wonder if I could use different settings or an alternative encoder to make it go faster – but it’s not slow enough to be worth further investigation. 2. Extract images from the video at a fixed interval Extracting images from a video needs a more complicated FFmpeg command: ffmpeg -i video.mp4 \ -vf 'fps=1/5,scale=iw*sar:ih,scale=min(iw\,945):min(ih\,945):force_original_aspect_ratio=decrease' \ thumbnail_%04d.jpg This extracts an image every 5 seconds, downscales any image larger than 945 pixels square (while preserving the original aspect ratio), and saves the results as sequentially numbered JPEG images (thumbnail_0001.png, thumbnail_0002.png, and so on). The key is the -vf flag, which defines two FFmpeg filters: The fps filter selects one frame every 5 seconds (fps=1/5). The first scale filter scales the width based on the sample aspect ratio (scale=iw*sar:ih). Without this filter, frames can be stretched and distorted. The second scale filter scales the input video, preserving the original aspect ratio (force_original_aspect_ratio=decrease), and ensuring the output images fit within 945×945px or the size of the input video, whichever is smaller. My limit is 945 pixels because that’s the largest size that cover art is shown on my iPhone. This filter still isn’t completely correct – it sometimes creates images from portrait videos that are smaller than I’m expecting – but it’s good enough. These are only thumbnails for glancing at, and if I want to change it later, I can always do the image resizing outside FFmpeg. 3. Insert the images as hidden chapters in the MP3 file Inserting the chapters into the MP3 file is more complicated. Although FFmpeg has basic support for ID3 metadata, as far as I know, it can’t insert chapters with per-chapter artwork. Instead, I’m going to reach for Python and the Mutagen library. Here’s the code to add a chapter to an MP3 file: from mutagen.id3 import APIC, CHAP, ID3, PictureType audio = ID3("audio.mp3") with open("thumbnail_0001.jpg", "rb") as f: img_data = f.read() image_frame = APIC(mime="image/jpeg", type=PictureType.OTHER, data=img_data) chapter_frame = CHAP( element_id="chp1", start_time=0, end_time=5 * 1000, sub_frames=[image_frame] ) audio.add(chapter_frame) audio.save() This creates a single chapter that lasts the first 5 seconds (0 to 5000 milliseconds), and the per-chapter cover art is thumbnail_0001.jpg. If we ran this in a loop, we could add images for every 5 second slice of the original video. This code is inserting two frames into the ID3 metadata: The CHAP (chapter) frame contains the timing information, and it can have subframes for metadata like title, chapter art, or associated URL. The APIC (attached picture) subframe contains information about a picture, which can either be a blob of image data or a URL to an image on the web. Normally, you’d also insert a CTOC frame which defines a table of contents, but I don’t want a TOC with hundreds of 5-second chapters, so I’m deliberately not doing this here. This is allowed by the ID3 spec – you’re not required to insert a CTOC frame if you’re using chapters, and you can have chapters that aren’t listed in your table of contents. To work out which frames I needed, I used Forecast to create some chapters by hand, and I inspected their frames. In particular, loading an MP3 and calling Mutagen’s pprint() method shows a human-readable list of frames, and then I could drill into the individual fields: from mutagen.id3 import ID3 audio = ID3("audio.mp3") print(audio.pprint()) I wrapped all this code in a project called glancecast, which allows you to convert a video file with a single command, with optional flags to set the frame length and chapter art size: $ python3 glancecast.py interesting_talk.mp4 interesting_talk.mp3 The process takes a minute or so to complete, most of which is spent transcoding the video file to MP3. The resulting MP3s are usually 40 to 50 MB in size, which is very reasonable. The outcome: How it looks in practice Here’s what one of these “glanceable” podcasts looks like in Overcast and on my lock screen: Maggie Appleton presented this talk over two years ago and it’s been on my “talks to watch” list ever since. Once I put it in Overcast? I listened to it in less than a day. It’s not a lot of extra information, but enough that I can quickly glance down and get the gist of what a speaker is saying. Both views update with a new frame every few seconds, or I can put my phone in my pocket and ignore the screen. I’ve used this approach for half a dozen videos so far, and I’m happy with the results. I expect to keep using it, because I have a long queue of videos I’ve been meaning to watch. If you’d like to try this, check out glancecast for the full code and instructions. [If the formatting of this post looks odd in your feed reader, visit the original article]

15th Sep 2026
How Tailscale tracked down a 16-year-old SQLite bug &rarr;

I wrote a post for the Tailscale blog about a long-running series of corruption incidents, and how they eventually led us to find an SQLite bug that predates my entire programming career. I’m incredibly proud of this, both the work and the blog post. Before Tailscale, I was coming from smaller teams where I didn’t get to tackle problems of this scale or complexity. This was exactly the sort of tricky, deep technical challenge I wanted to be part of (though I’d rather it hadn’t been quite so stressful)! I’m glad I got to play a small part in these incidents, and I learnt so much from the more experienced engineers I worked with. I never want to hear the words “SQLite corruption” again, but if I do, I’d want to have Tailscalars at my side. Writing the blog post has a blast, too. The piece transformed from a rough draft into a solid, engaging piece of writing, thanks to thoughtful feedback from many people at Tailscale. Most of my writing is self-edited, and it’s always a pleasure to work with a dedicated editor. Please check out the blog post if you haven’t read it already – I think it’s a fascinating technical story, and one readers of this site are bound to enjoy. [If the formatting of this post looks odd in your feed reader, visit the original article]

13th Aug 2026 • 2 votes
I don’t want to repeat repeat myself

Yesterday at work, a customer spotted a typo in our UI: “you can use the use the Tailscale CLI”. After the typo was fixed, I wanted to find other cases of accidentally repeated words or phrases. I used two regular expressions to search every codebase for unnecessary repetition. The first regex finds repeated words: \b([A-Za-z]+) \1\b Backfill product data from from Stripe Learn more about about inviting users Argument must be be one of host name, IP set name, IP prefix, or IP There’s a capturing group for a single word made up of letters ([A-Za-z]+), a space, then a backreference to the group. That expression is surrounded by word boundary assertions \b, which check that I’m at the start/end of a word – this avoids finding repeated character sequyences that within longer words, like “with the reason”. The second regex finds repeated phrases: \b([A-Za-z]+ [A-Za-z]+) \1\b Follow the steps in the in the "How to" section Log in to in to your account To configure federated identities federated identities using the Go SDK I’ve changed the capturing group, so now it looks for two words separated by a space. Sometimes repetition is useful, like when I really really went to emphasise a point, but often it’s just a typo. Cleaning up these mistakes has been a fun Friday cleanup task. [If the formatting of this post looks odd in your feed reader, visit the original article]

3rd Jul 2026 • 1 votes

More in programming

AI Homelab Part 4: Lowering the floor for Scout

Helping Scout focus and remember

5 hours ago • 1 votes
The Explorer Pattern for product invention and development.

tl;dr: the Explorer Pattern is one individual or team building a functional implementation that minimizes cross-team and cross-functional impacts as much as humanly possible, and then reconciling the details with other teams after that prototype is completed, accepting that this approach can require very substantial rework to that initial development. Why write this? Imprint previously called this the “Tech-led” or “Engineering-led” development model, but it’s pretty unclear to folks what that means. This writeup provides a more defined term, the “Explorer Pattern”, and explains in more detail how this works. Why do we need to adjust the way we work? Before 2026: Code is expensive, and we design process to minimize implementation and rework In and after 2026: Writing software is extremely cheap, but it’s a wider phenomenon: generating output is effectively-free across many dimensions. Code is nearly free — if we have appropriate constraints. Designs are nearly free — if they follow patterns. Analysis is nearly free — if we’ve instrumented things properly Reworking software is extremely cheap. Historically industry has optimized around avoiding engineering rework because engineering was the highest cost / narrowest constraint. In many companies this was interpreted as writing code being the constraint, which was never true, but many common practices were designed as if that were true (e.g. bringing engineering into problem-solving late “to protect their time”) Working on the system is much more effective than historically if we get automated integration and feedback loops right, e.g. not improving a one-off design but improving the design system. Any place where improving the higher-leverage process is slower than solving the one-off is a bug in our process Explorer Pattern: primary-and-replicas metaphor The easiest metaphor to think about the Explorer Pattern is database replication. Databases like MySQL and PostgreSQL have a primary database, where writes are performed. Every write is verified at that point, and once it’s verified, you know that the writes will flow downstream to various secondary replicas. Those downstream replicas can be delayed but they can’t be wrong because writes are verified upstream. (I’ve resisted the desire to talk about the CAP theorem.) A different model for database design is that used by Dynamo or Cassandra (or MongoDB to a certain extent), where you have a replica set, and have to get an acknowledgement from the majority of the replica set for a write to be strongly consistent. This is a very reliable setup, because it means you can tolerate losing a node, but it also introduces inconsistencies, and on a fixed amount of hardware they tend to be much, much slower than running a primary-secondary setup (or rely on bad performance metrics that don’t fsync to disk to create the illusion of throughput). The Explorer Pattern is the former primary-secondary model, where the vast majority of the work is done by the working individual or team. This makes all work internally consistent because it’s passed through the same small group. It doesn’t, however, mean that the work is perfect. For example, you can always insert data that meets the table’s requirements but is still inherently wrong (e.g. 200 is a valid integer, but it doesn’t mean that my credit limit was supposed to be 200, that depends on the surrounding application logic). Other teams become aware of the changes downstream, not synchronously upfront, and it’s instead the working group’s responsibility to both rationalize approach and create visibility into their work for other teams to engage in a non-blocking fashion. Example: Support for Financial Aggregators A concrete example of this practice at Imprint was rolling out financial aggregator support, e.g. allowing folks to connect to their Imprint powered accounts via Plaid. This has always been something folks want, but it’s particularly important in the realm of agentic commerce where more folks’ agents are analyzing their financial state for them. For this project, one of our engineering leaders (Ron), built the first pass of the feature at a low-burn while understanding the shape of the problem in a private alpha. He chatted through some of the details periodically to keep other teams aware, but it was a bit of a side project. However, once he found the full shape of the problem and had implemented a full solution, we cut the replica-lag across teams down to near-zero, identified and ran through the small number of remaining issues, and shipped it. Like many of our projects this year, it was never officially on a roadmap, and shipped a quality feature that our members have wanted for some time. How to follow the Explorer pattern The rough rules for following the Explorer pattern: Know your goal and sponsor: every initiative ultimately has to have a specific outcome it’s pursuing and a sponsor who’s bankrolling that pursuit. This doesn’t always need to be explicit, especially for smaller projects, but once a project goes beyond ~3 weeks of effort, if you can’t very explicitly identify who the stakeholder is sponsoring your project and they would not enthusiastically agree, then you are likely closer to “lost” than “exploring” Small working group is small: at most 5 people, and preferably smaller. Everyone in the group has to keep the entire project in their head for as long as possible to detect problems and opportunities Small working group is flexible: folks in small working groups have to “find a way to yes” rather than “find a way to no”. Exploring is about uncovering the fundamental shape of a problem rather than pretending we know the fundamental shape Publish progress in a public channel: explorers have to report home with the progress they are making so others are aware. This is both a Linear Project and a Slack channel. Make the narrowest viable changes to what already exists: when we introduce unnecessary changes, we invalidate others’ mental models about how the world works. When we haven’t made those changes with care, we’re not only invalidating others’ mental model, we are creating the expectation that we’ve done it very intentionally for a good reason, which has a chilling effect on others improving them later, which is very expensive, whereas the Explorer pattern depends on being as inexpensive as possible Exercise good judgment to identify reasonable hypotheses: discard bad ideas immediately, and then filter the rest by trying to implement them as quickly as possible. Once you see the shape of the implementation, you can figure out which of the remainders are interesting Hold assumptions constant to avoid getting stuck: the only way to navigate complex ambiguous problems is to hold things constant, even when you know they are imperfect. For example, to do headcount planning we have to know our revenue, to know our revenue we have to forecast it, to forecast it we have to know the day that partners will go live next year. We don’t truly know any of those things, so just anchoring on a plausible guess to unblock us is much more valuable than trying to get them accurate. A little prework is useful, but a lot is just artificial precision that’s a distraction from progress Hold space for being totally wrong: when we move fast, with low information, we always find the way out, but often it’s not the way out we anticipated when we got started. Being open to throwing it all away, and starting a second, third, fourth time Ship all the way to production behind a feature flag: features and products move fast when we get the pieces working every step from frontend to backend to data pulls. This is the first step of exploration. Just make sure it’s behind a feature flag that’s only enabled for internal folks!

8 hours ago • 1 votes
Value Creation vs Job Preservation

Comparing two different ways of work and working environments.

18 hours ago • 1 votes
Making a dumb TV just smart enough

I own an old, cheap TV. It has no Internet connection, no fancy smart features that make you want to install a Pi-hole, and no ask-me-again-later updates. However, it is connected to my own desktop computer through an HDMI cable, so I can watch movies and TV series from the couch. Setting it up used […] The post Making a dumb TV just smart enough appeared first on Quentin Santos.

2 days ago • 1 votes
Bikepacking across Kyushu

Five days and 540km on a bike across Kyushu, without a laptop. Notes on the peace of riding all day, convenience store dinners, and Japan's emptying countryside.

4 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in