Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
1

Developer variants

from Sedimental [alt+shift+b] in programming

Software development takes all kinds. I'm not talking about appearances or job titles. I'm talking about motivations and fulfillment. In my years of writing code and leading projects, I've come to learn a bit about how my teammates, and I, experience success, through a few manifest archetypes. The Developer-Mathematician Always a source of conversation, the Developer-Mathematician, seeks truth, pure and provable. They don't want to create software. They want to unearth timeless, universal absolutes that happen to be in the neighborhood of computers. Catch them crafting functional code, writing property-based tests, or exhaustively searching their bookmarks for that one paper on arXiv. To be honest, purity and formalism can chafe when building most software. Proofs are still more suited to dissertations than development. Still, it's good to strike a healthy balance between research and development. Make time to try new testing strategies, start a weekly paper club, and keep those fundamentals sharp. The Developer-Architect Less formal than the mathematician, but not always more practical, the Developer-Architect is brimming with potential. They want to create something original, important, and particularly elegant. They want to create something that outlasts them, something worthy of use, maintenance, and study. The creation need not be immortal or universal; the more of their mark that is left on it the better. Find them making high-concept pitches in response to clear gaps in the open-source ecosystem, or discussing best practices that are suspiciously similar to their own practices. If your Developer-Architect is low on ideas or recently saw one of their ideas superseded or implemented without them, they may become despondent. Software designers derive a lot of pleasure from the design process, but need to be reminded that architecture is far from the hardest part. To avoid turmoil and despondency, Developer-Architects must code their own implementations and...
9th Aug 2016

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Sedimental

Announcing FinFam

In my last post, I mentioned founding a startup. It's called FinFam, and we're building collaborative financial planning. The GitHub of money, if you will. Enough with the telling, time for the show! Here's a 3-minute demo: We launched beta this week and I couldn't be more excited. Let me tell you why. Where's this coming from? Growing up as the son of starving grad students, when it comes to money, I've been known to default to what we affectionately call "poor man brain." Cautious to a fault. But my student parents turned into scientists, so I'm eminently convinceable. I just need to see the math, or better yet, a spreadsheet. The problem? Making those spreadsheets. And more importantly, trusting them. A few years back, a friend was house hunting in the Bay Area. They came to me to talk through the famous rent-vs-buy problem. They didn't want an advisor to manage money or sell them products. They came to me because I worked in fintech and thus "knew money", even though I've only been involved in two home purchases, and wouldn't consider myself an expert. That experience crystallized something I'd been noticing everywhere: People trust their friends and family to have their best interests in mind... But can't trust them to have the best information. We can trust experts to have knowledge... but can't always trust their incentive alignment. In an era of ever-advancing information (and misinformation) glut, how do we get to a place of confidence in our hard-won justified true belief? Trust is social After 15+ years building fintech at PayPal and Stripe, I saw money movement simplified and commoditized. But the difficulty of transacting moved upstream. We made the how of buying easier, while advances in technology made the what, when, and why so much harder. From BNPL to crypto to vibecession, our economic realities aren't getting simpler. To compensate, 79% of young adults get financial guidance from social media (Forbes). Not because TikTok or YouTube has better models than Morgan Stanley, but because they trust the people sharing their stories. Millions of people are already collaborating on financial decisions. Privately on WhatsApp, obscurely on Discord, and full-blown publicly on Reddit: /r/personalfinance - 21 million members /r/financialindependence - 2.3 million /r/financialplanning - 1 million And dozens more subreddits and Internet forums (shoutout Bogleheads and Refinery29's Money Diaries). They're sharing detailed financial profiles with strangers on the internet, seeking advice from generous folks in full view. There's a fast-emerging story about AI here, and I've got whole posts dedicated to that coming soon. For now, suffice to say, we need the tools to catch up to the times. Enter FinFam FinFam[^name] lets families and friends collaborate on financial decisions with each other, using expert information without any commitments to said experts. We want to holistically solve the problem of financial decisionmaking using interaction models proven by GitHub + StackOverflow + app stores. How? First, creators publish interactive models, to FinFam's View marketplace. Then, you, a user who has a financial question or decision to make: Pick up a relevant expert view. If one doesn't exist, ask in FinFam's Q&A board. Plug in your own numbers and save it to a private workspace for your inner circle Discuss the results, with optional AI-assisted guidance as needed Move forward with confidence. We scale the expert knowledge while embracing the fundamentals of human social trust. Users get better decisions and peace of mind. Open-source meets fintech My years of work in open-source and wiki ecosystems showed me the power of collaborative, transparent tools. FinFam brings that same philosophy to personal finance. To further scale the knowledge, we make it possible for anyone to create a View. The View "source" format is XLSX, and can be edited with Google Sheets, Excel, or LibreOffice. Any published View can also be open-sourced. Just like with code, financial models are now reviewable, forkable, and improvable by the community. Curious users can check the community's math. Numbers and discussions happen with people you trust. Everything is private by default, shareable by design. What's next We launched beta this week and there are now daily spots available as we add capacity. You can sign up for early access here. I've been using it with friends and family for months now, and it has replaced Google Sheets for the financial decisions we face. There's so much more I want to share about the vision, the technology, and the journey so far. But this feels like we're off to a good start. Want to follow along? Subscribe to FinFam here and Sedimental here. If you're wondering about the name, just log in, go to your default space, create a thread, and ask Finn. ↩

19th Sep 2025 • 1 votes
What I've been up to in 2025

Been quiet around here. Time to change that! The short version up front: Since starting a family and leaving Stripe, I've pursued the dream that brought me to Silicon Valley. I've founded a startup. After taking some parental leave, helping found a Python non-profit, and a nice long visit back home, I was raring for a challenge. So these days, outside of family, I'm all in on something new. Contents Why now? Applications Monetary misunderstandings Showing vs Telling Why now? I've wanted to start my own business since building Access apps in high school. But, the reality of leaving my family and moving to study in the USA, combined with the technical and creative fulfillment of the software industry, took me on a scenic route through enterprise software, free culture, and open-source. That very same reality has since conspired to convince me to return to my original aspirations. I've lived through some exciting times in software, but nothing like now. This isn't something I imagined I'd be working on 10 years ago, but then again it's not something I thought possible even 3 years ago. What better time to be building and launching my most ambitious project ever? Full details on that are coming soon1. For now, here is a post about why. Applications To start my career, I worked on software infrastructure, security, observability, and developer productivity. But after eight years, around 2016, I started longing for something more human. You can see this start to come out in The Packaging Gradient. At a time where it seemed like everyone around me was talking about pip, pipenv, and PyPI, I couldn't help but remind people that the real end goal of software has always been the application (or even the appliance). This impulse came to a head with APA. Perhaps you, dear reader, have also been "lost in the sauce" of software: When you love computers and it dominates your thoughts, you might also spend most of your time thinking about the software that makes the software possible. Don't get me wrong. Languages, libraries, compilers, devtools, we need every bit of help we can get. But I fell in love with software for its potential to effect change in the world writ large. I started eyeing product. The famous full stack. That meant moving on from big tech, to a big startup, to a seed startup. One pandemic-fueled detour through a startup factory later, here we are. Finally, founding the startup. My own full stack. Monetary misunderstandings My 15+ year software engineering career can be summed up as: Building fintech software for pay Shipping open-source Python/wiki for free Professionally enabling commerce while avoiding it in my personal time. I was young and conflicted. Truthfully, I still harbor some reservations, but I have to build what I know. I know about software and money. "Money is the root of all evil." If you look at the state of say, open banking in the USA, or web3isgoinggreat, or just read Money Stuff, you probably agree something's off. Money changes people. But so does the lack thereof. I've watched more talented and deserving developers than myself befall a variety of fates. Hollowed out by monetary excess, blinded by greed, burned out by FOSS, literally working Doordash to keep the lights on. Dropping out of software completely. Shunning the world's favorite fungible has bad outcomes for individuals. Bless my friends at Tidelift, OSTIF, and other orgs working to sustain the maintainers. Paying maintainers is a worthy battle. We just need to open more fronts to navigate what's in store. Showing vs Telling Lately I've been thinking a lot about my favorite David Lynch (RIP) scene. It isn't from one of his films, it's this quote: "The film is the talking." I think it perfectly captures the auteur mindset. Words are extraneous. The consummate creative expresses themselves better in their native medium. Not that I mind words as a medium. After years of blogging and speaking, I've grown confident in my ability to tell. But now it's time for the show. For friends who can't wait a couple weeks, shoot me an email for early access. ↩

25th Aug 2025 • 1 votes
Intentional Creation

Reliably tap into your creativity with the 4 Cs: Consume, critique, curate, create. This is one of my oldest ideas, finally published on the GitHub ReadME Project blog, along with a profile, in June 2022. For more like this, follow me on Twitter or Mastodon. (You can also read it in 中文 here. Thanks Dominic Huang!) We all have creative potential. Whether it gets you up in the morning or keeps you up at night, you've felt its gnaw. Turning that potential into productivity can prove challenging in an internet-connected environment that offers a constant stream of consumables. How do we pick a direction? In this guide, you'll see how to distill the elements of creativity into four deliberate stages, and how to put the process to use: Contents Consume Critique Curate Create Debugging the process Putting it into practice These 4 Cs comprise a straightforward, adaptable approach that works well in both group and solo settings. You've already started on step 1. Read on to find out what to do next. Consume Step 1: Turn passive consumption into active research. From the moment you open your eyes in the morning, you're accosted with calls to consume. Articles, videos, podcasts, the newest Wordle variant. When consumption is the default mode of our modern computing environment, how is a builder supposed to build? To create, we must first recognize its inverse: consumption. Consumption is a useful stage, but can be dangerous if it’s terminal. An infinite loop in this stage kills any chance of creation. Little is created in a vacuum. Creation still starts with consumption, albeit consumption disarmed with an intention. Turn pure consumption into active research, punctuated with critique. Critique Step 2: Capture your reactions in critiques, and research no faster than you can react. As with so many software problems of our day, the answer is simple: React. No, not the JavaScript framework, but the human act of reaction. Intentional creation starts with giving yourself pause on new inputs. Seek a reaction from yourself. A semi-structured reaction, or critique, is a time-honored practice in creative fields, like architecture. Infinite scroll may prove challenging to overcome, but before turning your consideration to the next item in your feed, activate your critical senses. Draw some conclusions. Even unvetted, they're yours. If you're finding it hard to summon a critique, this is a clear sign you're consuming faster than you can reflect. If you're not reflecting, you're not learning. You may need to go deeper on individual items, or just take a break. Your critiques have never been easier to capture, whether typed in markdown, dictated to automatic transcription, or written down in a notepad on your desk. Try opening your editor or critique tool before opening any new resources. Feel free to open one now. If you're not reflecting, you're not learning. Curate Step 3: Curate critiques into collections that act as reservoirs of creative reference. Critiques are only proto-creative output. Writing anything helps prime the creative pump, but criticism is raw reaction. You want a refined synthesis. Once you've got enough critiques under your belt, curate the positive examples into a collection. From interior designers to lab researchers to club DJs, creators recognize the value of a structured, referenceable collection. Sometimes a situation calls for urgency or direction, and an organized, well-researched collection can offer an existing solution. Sometimes, in the context of a comprehensive collection, the lack of a referenceable solution is itself a signal that it's time to invent. Curated collections become artifacts unto themselves. I've helped create a few, including 0ver.org, seealso.org, and the Awesome Python Applications list. There’s more awesome out there beyond Awesome Lists, like explorabl.es, the Cooperpress newsletters, or the “swipe file” phenomenon used among designers and content creators. There's respectable work in curation. Still, curation is more important as a stepping stone to our original higher calling. Less is more. Create Step 4: Return to your curations regularly to discover your creative path forward. Collections of a certain size tend to produce interesting findings. Patterns and gaps emerge that inspire creative next steps. As an example, while researching approaches to Python packaging, a pattern emerged that led to one of my most popular concepts/blog posts/talks, The Packaging Gradient. Whole projects can be born out of connections made with collections. My framework Clastic, which was eventually used by teams at PayPal and Wiki Loves Monuments, came out of the curated combination of pytest dependency-injection semantics with werkzeug primitives. Realistically, the majority of creation happens below the threshold of standalone artifacts. For instance, when adding a feature to an existing system, a parallel approach in a different project serves as a useful guide. I've lost track of the number of times I've swiped techniques from Awesome Python Applications, including ones used to port my dayjob's 300k SLOC codebase from Python 2 to 3. Most creative outputs have a similar lineage. Only now we have an explicit process. Debugging the process It's easy to see creations we appreciate as towering achievements that sprung fully-formed from their creators' genius. But creation comes in fits and starts. If creation comes slowly, here are a few strategies to consider: Search for a natural split in an existing collection that's getting too big, and explore what makes it interesting. Revisit an old, contentious critique and re-react. What did you get right/wrong? Pick a particular exemplar and turn it into a case study. One beautiful aspect of FOSS projects is that going deep can mean getting involved. There's nothing like proximity to a problem to inspire creative thinking. More generally, be wary of one-size-fits-all solutions; while prescriptive techniques such as the Zettelkasten Method may work for some, creation is idiosyncratic. Embrace your own process. Putting it into practice When inspiration hits, connections can form so quickly that we take for granted what goes on. When inspiration proves less willing to strike, we can keep ourselves primed for creativity by ensuring all four activities continue in balance. There are a few notable benefits of intentional creation: When you've built something, the influences are well-documented. It can be easier to involve others when there's a clear creative thread to pull on. Sharing your critiques and curations invites collaboration with other creators and curators. Self-awareness. If you're not finding your critiques crystallizing into new thoughts and ideas for projects, that's a sign you're looking at the wrong stuff. Are you following your interests or passively consuming trending content? Practically, intentional creation means consciously spending less time on consumer sites, from Twitter to Hacker News, and more time taking notes, tagging bookmarks, and creating your own knowledge base. Attempt activities that are less entertainment and more you, ultimately closing the gap between you and your creative goals. If it sounds too simple, that's because it is. You're still accountable to you, that's the hard part. But hopefully you'll find some value in this simple hierarchy that lets you check in on your own activities and make adjustments toward a more creative end. Spend less time consuming, and more time on the other three Cs. Consume only enough to allow yourself to critique, curate, and create. If you made it this far, then start now. Step 2. Use any tool or service you like, from spreadsheets to YAML, and answer this: What's your critique?

4th Jan 2023 • 1 votes
Changing the Tires on a Moving Codebase

2020 was a year of reckonings. And for all that was beyond one’s control, as the year went on, I found myself pouring more and more into the one thing that felt within reach: futureproofing of the large enterprise web application I helped build, SimpleLegal. Now complete, this replatforming easily ranks in my most complex projects, and right now, holds the top spot for the happiest ending. That happiness comes at a cost, but with some the right approach that cost may not be as high as you think. Contents The Bottom Line The Setup The Outset The Traction Issues The Sentry Pivot The New Road Committing to transactions The truly atomic request Transactional test setup Better than best practices The utility of namespaces Coverage tools Flattening database migrations Easing onto the stack The Rollout The Aftermath The Bottom Line We took SimpleLegal’s primary product, a 300,000 line Django-1.11-Python 2.7-Redis-Postgres-10 codebase, to a Django 2.2-Python 3.8-Postgres-12 stack, on-schedule and without major site incidents. And it feels amazing. Speaking as tech lead on the project, what did it look like? For me, something like this: But as Director of Engineering, what did it cost? 3.5 dev years and just about $2 per line of code. And I'm especially proud of that result, because along the way, we also substantially improved the speed and reliability of both the site and development process itself. The product now has a bright future ahead, ready to shine in sales RFPs and compliance questionnaires. Most importantly, there’ll be no worrying about when to delicately break it to a candidate that they’ll be working with unsupported technology. In short, a large, solid investment that’s already paying for itself. If you just came here for the estimate we wish we had, you've got it. This post is all about how your team can achieve the same result, if not better. The Setup The story begins in 2013, when a freshly YC-incubated SimpleLegal made all the right decisions for a new SaaS LegalTech company: Python, Django, Postgres, Redis. In classic startup fashion, features came first, unless technology was a blocker. Packages were only upgraded incidentally. By 2019, the end of this technical runway had drawn near. While Python 2 may be getting extended support from various vendors, there were precious few volunteers in sight to do Django 1 CVE patches in 2021. A web framework’s a riskier attack surface, so we finally had our compliance forcing function, and it was time to pay off our tech debt. The Outset So began our Tech Refresh replatforming initiative, in Q4 2019. The goal: Upgrade the stack while still shipping features, like changing the tires of a moving car. We wanted to do it carefully, and that would take time. Here are some helpful ground rules for long-running projects: Any project that gets worked on 10+ hours per week deserves a 30-minute weekly sync. Every recurring meeting deserves a log. Put it in the invite. Use that Project Log to record progress, blockers, and decisions. It’s a marathon, not a sprint. Avoid relying on working nights, weekends, and holidays. We started with a sketch of a plan that, generously interpreted, ended up being about halfway correct. Some early guesses that turned into successes: Move to pip-tools and unpin dependencies based on extensive changelog analysis. Identify packages without py23 compatible versions. (Though we’ve since moved to poetry.) Add line coverage reporting to CI Revamp internal testing framework to allow devs to quickly write tests More on these below. Other plans weren’t so realistic: Take our CI from ~60% to 95% line coverage in 6 months Parallelized conversion of app packages over the course of 3 months Use low traffic times around USA holidays (Thanksgiving, Christmas, New Years) to gradually roll onto the new app before 2021. We were young! As naïve as we were, at least we knew it would be a lot of work. To help shoulder the burden, we scouted, hired, and trained three dedicated off-shore developers. The Traction Issues Even with added developers, by mid-2020 it was becoming obvious we were dreaming about 95% coverage, let alone 100%. Total coverage may be best practice, but 3.5 developers couldn’t cover enough ground. We were getting valuable tests, and even finding old bugs, but if we stuck with the letter of the plan, Django 2 would end up being a 2022 project. At 70%, we decided it was time to pivot. We realized that CI is more sensitive than most users for most of the site. So we focused in on testing the highest impact code. What’s high-impact? 1) the code that fails most visibly and 2) the code that’s hardest to retry. You can build an inventory of high-impact code in under a week by looking at traffic stats, batch job schedules, and asking your support staff. Around 80% of the codebase falls outside that high-traffic/high-impact list. What to do about that 80%? Lean in on error detection and fast time-to-fix. The Sentry Pivot One nice thing about startup life is that it’s easy to try new tools. One practice we’ve embraced at SimpleLegal is to reserve every 5th week for developers to work on the development process itself, like a coordinated 20% time. Even the best chef can’t cook five-star food in a messy kitchen. This was our way of cleaning up the shop and ultimately speeding up the ship. During one such period, someone had the genius idea to add dedicated error reporting to the system, using Sentry. Within a day or two, we had a site you could visit and get stack traces. It was pretty magical, and it wasn’t until Tech Refresh that we realized that while integration takes one dev-day, full adoption can take a team months. You see, adding Sentry to a mature-but-fast-moving system means one thing: noise. Our live site was erroring all the time. Most errors weren’t visible or didn’t block users, who in some cases had quietly learned to work around longstanding site quirks. Pretty quickly, our developers learned to treat Sentry as a repository of debugging information. A Sentry event on its own wasn’t something to be taken seriously in 2019. That changed in 2020, with the team responsible for delivering a seamless replatform needing Sentry to be something else: a responsive site quality tool. How did we get there? First step, enhance the data flowing into Sentry by following these best practices: Split up your products into separate Sentry projects. This includes your frontend and backend. Tag your releases. Don’t tag dev env deployments with the branch, it clutters up the Releases UI. Add a separate branch tag for searches. Split up your environments. This is critical for directing alerts. Our Sentry client environment is configured by domain conventions and Django’s sites framework. If it helps, here's a baseline, we use these environments: Production: Current official release. DevOps monitored. Sandbox: Current official release (some companies do next release). Used by customers to test changes. DevOps monitored. Demo/Sales: Previous official release. Mostly internal traffic, but external visibility at prospect demo time. DevOps monitored. Canary: Next official release. Otherwise known as staging. Internal traffic. Dev monitored. ProdQA: Current official release. Used internally to reproduce support issues. Dev monitored. QA: Dev branches, dev release, internal traffic. Unmonitored debugging data. Local test/CI: Not published to Sentry by default. With issues finally properly tagged and searchable, we used Sentry’s new Discover tool to export issues weekly, and prioritize legacy errors. To start, we focused on high-visibility production errors with non-internal human users. Our specific query: has:user !transaction:/api/* event.type:error !user.username:*@simplelegal.* We triaged into 4 categories: Quick fix (minor bug), Quick error (turn an opaque 500 error into a actionable 400 of some form), Spike (larger bug, requires research), and Silence (using Sentry’s ignore feature). Over 6 weeks we went from over 2500 weekly events down to less than 500. Further efforts have gotten us under 100 events per week, spread across a handful of issues, which is more than manageable for even a lean team. While "Sentry Zero" remains the ideal, we achieved and maintained the real goal of a responsive flow, in large part thanks to the Slack integration. Our team no longer hears about server errors from our Support team. In fact, these days, we let them know when a client is having trouble and we’ve got a ticket underway. And it really is important to develop close ties with your support team. Embedded in our strategy above was that CI is much more sensitive than a real user. While perfection is tempting, it’s not unrealistic to ask a bit of patience from an enterprise user, provided your support team is prepared. Sync with them weekly so surprise is minimized. If they’re feeling ambitious, you can teach them some Sentry basics, too. The New Road With noise virtually eliminated, we were ready to move fast. While the lean-in on fast-fixing Sentry issues was necessary, a strong reactive game is only useful if there are proactive changes being pushed. Here are some highlights we learned when making those changes: Committing to transactions Used properly, rollbacks can make it like errors never happened, the perfect complement to a fast-fix strategy. The truly atomic request Get as much as possible into the transactions. Turn on ATOMIC_REQUESTS, if you haven’t already. Some requests do more than change the database, though, like sending notifications and enqueuing background tasks. At SimpleLegal, we rearchitected to defer all side effects (except logging) until a successful response was being returned. Middleware can help, but mainly we achieved this by getting rid of our Redis queue, and switching to a PostgreSQL-backed task queue/broker. This arrangement ensures that if an error occurs, the transaction is rolled back, no tasks are enqueued, and the user gets a clean failure. We spot the breakage in Sentry, toggle over to the old site to unblock, and their next retry succeeds. Transactional test setup Transactionality also proved key to our testing strategy. SimpleLegal had long outgrown Django’s primitive fixture system. Most tests required complex Python to set up, making tests slow to write and slow to run. To speed up both writing and running, we wrapped the whole test session in a transaction, then, before any test cases run, we set up exemplary base states. Test cases used these base states as fixtures, and rolled back to the base state after every test case. See this conftest.py excerpt for details. Better than best practices Software scenarios vary so widely, there’s an art to knowing which advice isn’t for you. Here’s an assortment of cul de sacs we learned about firsthand. The utility of namespaces Given how code is divided into modules, packages, Django apps, etc., it may be tempting to treat those as units of work. Don’t start there. Code divisions can be pretty arbitrary, and it’s hard to know when you’ve pulled on a risky thread. Assuming there are automated refactorings, as in a 2to3 conversion, start by porting by type of transformation. That way, one need only review a command and a list of paths affected. Plus, automated fixes necessarily follow a pattern, meaning more people can fix bugs arising from the refactor. Coverage tools Coverage was a mixed bag for us. Obviously our coverage-first strategy wasn’t tenable, but it was still useful for prioritization and status checks. On a per-change basis, we found coverage tools to be somewhat unreliable. We never got to the bottom of why coverage acted nondeterministically, and we left the conclusion at, “off-the-shelf tools like codecov are probably not targeted at monorepos of our scale.” In running into coverage walls, we ended up exploring many other interpretations of coverage. For us, much higher-priority than line coverage were “route coverage” (i.e., every URL has at least one integration test) and “model repr coverage” (i.e., every model object had a useful text representation, useful for debugging in Sentry). With more time, we would have liked to build tools around those, and even around online-profiling based coverage statistics, to prioritize the highest traffic lines, not just the highest traffic routes. If you’ve heard of approaches to these ends, we’d love to discuss them with you. Flattening database migrations On the surface, reducing the number of files we needed to upgrade seems logical. Turns out, flattening migrations is a low-payoff strategy to get rid of files. Changing historical migration file structure complicated our rollout, while upgrading migrations we didn’t flatten was straightforward. Not to mention, if you just wanted the CI speedup, you can take the same page from the Open EdX Platform that we did: build a base DB cache that you check in every couple months. Turns out, you can learn a lot from open-source applications. Easing onto the stack If you have more than one application, use the smaller, simpler application to pilot changes. We were lucky enough to have a separate app whose tests ran faster, making for a tighter development loop we coul learn from. Likewise, if you have more than one production environment, start rollouts with the one with the least impact. Clone your CI jobs for the new stack, too. They’ll all fail, but resist the urge to mark them as optional. Instead, build a single-file inventory of all tests and their current testing state. We built a small extension for our test runner, pytest, which bulk skipped tests based on a status inventory file. Then, ratchet: unskip and fix a test, update the file, check that tests pass, and repeat. Much more convenient and scannable than pytest mark decorators spread throughout the codebase. See this conftest.py excerpt for details. The Rollout In Q4 2020, we doubled up on infrastructure to run the old and new sites in parallel, backed by the same database. We got into a loop of enabling traffic to the new stack, building a queue of Sentry issues to fix, and switching it back off, while tracking the time. After around 120 hours of new stack, strategically spread around the clock and week, enough organizational confidence had been built that we could leave the site on during our most critical hours: Mondays and Tuesdays at the beginning of the month. The sole hiccup was an AWS outage Thanksgiving week. At this point we were ahead of schedule, and enough confidence had been built in our fast-fix workflow that we didn’t need our original holiday testing windows. And for that, many thanks were given. We kept at the fast-fix crank until we were done. Done isn't when the new system has no errors, it's when traffic on the new system has fewer events than the old system. Then, fix forward, and start scheduling time to delete the scaffolding. The Aftermath So, once you’re on current LTS versions of Django, Python, Linux, and Postgres, job complete, right? Thankfully, tech debt never quite hits 0. While updating and replacing core technologies on a schedule is no small feat, replacing a rusty part with a shiny one doesn’t change a design. Architectural tech debt -- mistakes in abstractions, including the lack thereof -- can present an even greater challenge. Solutions to those problems don’t generalize between projects as cleanly, but they do benefit from up-to-date and error-free foundations. For all the projects looking to add tread to their technical tires, we hope this retrospective helps you confidently and pragmatically retrofit your stack for years to come. Finally, big thanks to Uvik for the talent connection, and the talent: Yaroslav, Serhii, and Oleh. Shoutouts to Kurt, Justin, and Chris, my fellow leads. And the cheers to business leadership at SimpleLegal and everywhere, for seeing the value in maintainability.

10th Mar 2021 • 1 votes
Thanks, 201X!

Thought I'd take a Sunday afternoon to reflect on, oh I don't know, a decade. Been a long ten years, but it's flown past. This particular decade happens to coincide with my first years of full-time professional software engineering. The Quantity I can't possibly summarize it all, and if I tried, it'd still be colored by what's on my mind right now. But I can point to the artifacts I tried to leave along the way: Twitter FWIW1 (2008+) ~20 Open-Source Projects (2012+) ~15 Hatnote Projects (2013+, follow us) ~25 entries on this blog (2015+) +7 here (2014-2016) Not including pythondoeswhat.com or blog.hatnote.com (or other posts on the blogs only real heads know) ~10 Talks (2016+) Lest I forget: O'Reilly's Enterprise Software with Python (2016) And several podcast/media appearances calver.org (2016) and 0ver.org (2018) (Versioning is a fun pastime) Pyninsula (2017+) - YouTube, Meetup, Email Announce Taking a chronological look at each of the above, I'm relieved to see obvious growth. If I were to highlight one resource, it would probably be the talks. Despite the stress of preparation and delivery, I'm least concerned with having a massive miscommunication when we're all in the room and I can see the points hitting home. It's impossible to pick a favorite, but Ask the Ecosystem (2019), the Restructuring Data lightning talk (2018), and The Packaging Gradient (2017) seem like audience faves from where I'm sitting. The Quality Each project, post, and talk had its own reward, but I guess I've got more than just those to show for the decade. On the more profit-driven side, I built tools and teams at PayPal, but once I could manage the risk, I got to dip into startups for the last few years. Lucky for me, it wasn't a total bust, and the wife and I bought a place in my favorite neighborhood (in the USA). Not a millionaire, but I'm hoping and working for a world where no one has to be. More recently, the Python Software Foundation made me a Fellow. This isn't something I can be nonchalant about, and I'm not going to understate how much this means, to me, working in a field like software, where concrete symbols of progress are alternatingly elusive and vanishing. Plus it's Python, and reciprocated love is nice. I have hundreds of people to thank for helping me reach this point, and I have to thank the PSF for dedicating the time to ramping up these awards. They've convinced me more than ever that we need more institutions to build this sort of advancement. To all of you, thank you. The Struggle I like to think I managed to do all of the above while staying away from industry hype, on the principle that massive speculative capital influx isn't where real value is added to society, and doesn't generate the kind of innovation that excites me. I may have been naïve, but I came to Silicon Valley with an idea about the transformative power of software. Changing times may illustrate a grittier interpretation than the one I had and have, but I continue to hold dear software's potential for positive impact. If you've felt that vision waver, let me tell you, you're not alone. In the past decade, I've seen too many engineers sucked in by new technologies and ventures, only to find themselves alienated from their work. Episodes ranging from an afternoon lost to debugging Docker/k8s clusters, to years of work disappearing at the end of a VC runway. Nothing has been harder to watch than those bedraggled-but-persistent idealists regroup, each time a bit more cynical than the last. Even if its seeming intractibility has taken it from the center stage, the burnout conversation continues to smolder, because there's no issue realer. I know; I released more ceramics than software back in 2014. Some problems can be solved by paying the maintainers, but I think the vastly bigger issue is around losing the human connection between the real effort software takes and the real benefits it brings, combined with FOSS's dearth of collaborators in supporting roles (QA, product/project/release management). That's why I'm incredibly thankful for the Wikimedia community for always being there, patient with schedules and issues, as long as the software got the job done. It can be a challenge to juggle projects, but I tell every budding engineer: find that direct connection to people who will appreciate your work, and avoid cynicism at all costs. There are some interesting prospects in the works, but I'm keeping this post retro. Besides, if 2029 rolls around and all I did was break even with 2009-19, I don't see how I can be disappointed. Thanks again for everything in 201X, and for sticking with me in 202X. Despite using Twitter for over a decade, the process of tweeting feels so perfunctory, and the service itself so tenuous, that I still can't bring myself to invest the time. I mostly use it to crosspost my blog posts or help friends promote their posts/projects. But until I start an email newsletter, or really get on top of yak.party, it's still the best I got for announcing where I'm speaking next. ↩

2nd Dec 2019 • 1 votes

More in programming

Lexxy 1.0 is here

Today we are releasing the version 1.0 of Lexxy. Lexxy is a rich text editor for Rails built on Lexical. It already powers Basecamp, Fizzy and many others, and it will become the default editor in Rails. I recently presented it in Rails World (slides, video coming soon). This is the article version of my talk. Trix hit a wall Trix has been our editor since 2015, and every Rails app’s editor since Action Text shipped in Rails 6. It’s small and reliable, and it has served millions of people for a decade. But in the last few years our customers kept asking for features like tables or code highlighting, and we kept struggling to deliver them. The reason is the Trix document model. A Trix document is a flat list of blocks. A block is a line of text with some labels attached, like quote, bullet list, bullet. There is no tree, and a block can never contain another block. Nesting is an illusion: at render time, adjacent blocks with the same labels get wrapped together. A flat list of blocks with very limited extensibility options That design bought a lot of simplicity, but you can’t express something like a table with it. Two cells next to each other would carry exactly the same labels, so Trix would merge them into one. The model can say “this bullet is one level deeper”. It cannot say “this cell is different from the cell next to it”. A tables issue has been opened since 2015! The model just can’t do it. The second problem was maintenance. An editor built on contenteditable behaves differently in every browser and even changes from time with operating system releases. In 2024, three iOS releases in a row broke typing, dictation or the caret in Trix, and each one cost us real effort to work around. Check this one as an example. Why Lexical This was a conversation we had at 37signals for years: 2022. We started a project to add tables to Trix. We gave up after a week. The document model can’t represent two-dimensional things. 2023. I built a proof of concept with Tiptap inside HEY. We liked it, but we never started a serious project with it. 2024. We built House, our own Markdown editor, for Writebook. Not WYSIWYG, but WYSIWYM: what you see is what you mean. A wonderful editor for long-form writing like books or technical documentation. 2025. We tried House in another product, and it didn’t fit. For most apps, WYSIWYG was just the right answer. 2025. We had the discussion again, and this time we looked at the whole field. Four years of the same conversation David ruled out Tiptap, CKEditor and the other commercial editors: an open source core with features kept proprietary, and a sales team behind them. We didn’t want our editor to depend on somebody else’s licensing decisions. Then we found Lexical: MIT, from Meta, and very powerful. I spent two weeks evaluating it, and in May we made the call to go with it. A tiny core. Pick the rest. Lexical’s core has zero dependencies and weighs forty-two kilobytes. In a way, it validates the approach that Trix pioneered. The document is an immutable state you never mutate directly, you just get new snapshots when performing updates; contenteditable is an input device and a rendering surface, never the source of truth. It has a DOM reconciler to update the actual DOM very efficiently, and other primitives to deal with handling commands and node transformations. Everything else, from lists to tables to markdown, is a package in the orbit. Lexxy uses thirteen of them. Lexical solved the maintenance problem too. Meta’s products like Facebook or Instagram use Lexical and their user count is in the hundreds of millions. This means that even small issues with new keyboards and devices are fixed fast by the dedicated Meta team that maintains it. Furthermore, Meta’s business is the products, not the editor. We much rather liked this structure of incentives for the long-term investment an editor represents. The iceberg The plan was simple. Pick Lexical, wire it up to Action Text, add a toolbar and ship it. Well, it didn’t go exactly like that. What you envision, and what's under the water Lexxy today is thirteen thousand lines of vanilla JavaScript on top of Lexical. A great editing experience is very hard to get right. An editor is a machine where the user can change the state in a thousand different ways. For example. you have two images one after the other and want to put the cursor between them, but there is nothing there to put a cursor in. Or somebody pastes from Google Docs, and you have to turn a pile of inline styles and empty spans into clean markup. And then Safari, and Android keyboards, and the clipboard, and undo, and… From the first pull request to Basecamp 5 The first pull request landed in May 2025. Fizzy launched with Lexxy in December, and Basecamp 5 in May this year. Basecamp was the real test: twenty years of content written with Trix, and people who use the editor all day, every day. Zoltán Hosszú and Samuel Péchèr were the key people who made this happen. The took a very green version of Lexxy, added a ton of features (including Tables) and polish, and they fixed countless bugs. They also pulled off a remarkable milestone: seamlessly switching millions of Basecamp users from Trix to Lexxy. What’s included? We didn’t want a to build Trix with tables. We had Lexical and we had agents to help, so we wanted to be ambitious here. We went for the whole package. Features In terms of major features: A color highlighter, built in instead of this being a custom Basecamp extension, as it was with Trix. Tables, with an interface we worked hard to keep simple and accessible. Markdown. You type it, you get rich text. Code blocks with syntax highlighting as you type, in more than twenty languages. Image galleries you can navigate and reorder with the keyboard. Prompts. Type a character, get a menu: mentions, emoji, or whatever your app needs. Links by pasting a URL over selected text. Previews of attachments like videos and PDFs, rendered as your app renders them. Action Text Native Lexxy is also Action Text native. Action Text stores attachments in a canonical format that Trix doesn’t speak, so it translates on save and again on render. We taught Lexxy to emit exactly that markup. What you see in the editor is what gets saved, and what gets saved is what your app renders. Your existing content, attachments and views keep working. That opened another door. Action Text now talks to an editor adapter, with an implementation for Trix and one for Lexxy, so switching is one line: config.action_text.editor = :lexxy. Here the credit goes to Sean Doyle, who started that pull request before Lexxy existed and took it to the finish line with our input. It ships with Rails 8.2, and we hope other editors will use it too. Extensibility And you can extend Lexxy. Extensions are built on Lexical’s own mechanism, and this is not a second-class API: Lexxy itself is thirteen extensions, tables included, and Basecamp has nine more. class MyExtension extends Lexxy.Extension { get enabled() { … } get allowedElements() { … } get lexicalExtension() { return this.defineExtension({ name: "my-extension", nodes: [ … ], register(editor) { … } }) } initializeToolbar(toolbar) { … } dispose() { … } } Lexxy.configure({ global: { extensions: [ MyExtension ] } }) My favorite of how extensible is Lexxy are voice notes in Basecamp: you record, you see the waveform while you talk, and it becomes a player inside the document. About a thousand lines, without forking or patching anything. Performance Lexxy is fast, because Lexical is fast. In a ten thousand word document, Trix takes thirty-eight milliseconds to process a keystroke. Lexxy takes four. Above fifty milliseconds, the editor starts feeling sluggish. Compared to Trix, Lexxy brought a whole new performance regime. Milliseconds per keystroke by document size Accessibility Accessibility in Trix was not great. In general, building accessible experiences for rich text editors built on top of contenteditable is quite hard. We had a dream team to help with Lexxy accessibility. Bruno Prieto worked with Michael Berger, our accessibility champion at 37signals, to bring the bar to where we wanted it to be. Bruno is an outstanding programmer who happens to be blind, so he knows one thing or two about accessibility, and he delivered. As a result, in Lexxy everything is reachable with the keyboard. The editor announces itself properly to screen readers, and it gets a thousand details right so that the editing experience using a screen reader is fantastic. You can learn more about accessibility in our docs. Security The latest AI models have resulted in an unprecedented explosion of vulnerabilities found, and we took this thread quite seriously. Lexxy counted with programmers of the caliber of Jeremy Daer and Mike Dalessio helping to make it more secure. We have put a lot of attention to sanitizing the editor contents, validating attachment URLs and making sure that the types of attachments and nodes the editor support are allow-listed. Lexxy also comes with preliminary Trusted Types support, to offer CSP-level control over certain DOM manipulation APIs. The trusted types policy is there, but we are not enforcing it everywhere yet. Agents We started Lexxy using Claude since day one. A main lesson was that an agent needs to drive the editor like a user does. The best decision we made in this project was moving the system tests from Capybara to Playwright: three browsers instead of one, a suite that runs in seconds, and a much more faithful clipboard, keyboard and focus. This represented a tremendous improvement in how agents could close the loop by themselves. Write a test, see it fail, fix it, see it pass. We have more than six hundred browser tests today. Moving the suite to Playwright changed how fast we could write tests With a solid testing foundation in place, we could start fixing bugs in large batches. As mentioned, getting a text editor right implies a ton of work, and the kind of backlog we got at some point would have have buried us in pre-agent times. We also used agents to validate the fixes: an agent reproduces the bug in the public Lexxy sandbox, checks that it’s gone with the branch applied, and labels the pull request. Agents were essential to get Lexxy done with the people and the deadlines we had: we are a small company, and the same people were shipping two products in parallel. 275 cards closed The new Rails default We believe Lexxy is the best rich text editor out there right now, and we are going to make it the default editor in Rails next. If you’re starting a Rails application today, use Lexxy. If you’re using Action Text with a standard configuration, switch. It’s one line, and we’ve worked hard to make it seamless.

an hour ago • 1 votes
George Dryden

Reading my recent computing retrospective, I realised there was a big section missing: the people in my life that made an impact and helped shape my career. Outside my immediate family, one person made an outsized contribution, and I’m fairly certain that without his influence my life would have taken a very different path. The fact that I’m still here in 2026, still writing code and being fortunate enough to have a career in something I love is testament to him. So I’d like to take a few moments to talk about my old secondary school teacher, George Dryden. Denied Back in 1995, I had a problem. I knew I wanted to study computing at university and build a career out of my passion, but there was a snag. For those unfamiliar with the UK schooling system, when you’re 15-16 you take a set of GCSE exams in a broad range of subjects. After that, you pick around 3 subjects to really focus on over a period of 2 years. These are called A Levels, and they are a big step up and are meant to prepare you for a degree-level course at university. Admission to university is also governed by these results - if you want to study computing, you’re going to need a computing A-Level, and most universities will only accept you (or “make an offer”) if you achieve a certain grade. And whilst I had taken computing at a GCSE level, my school did not offer a computing A-Level course. I instead had to settle on “Design & Technology”, which just didn’t inspire me. Instead of working on my portfolio and projects, I spent most of my time daydreaming and writing code on the Acorn Archimedes computers that were the staple of every 90s UK school. No disrespect to the teachers - they were all awesome - but it just wasn’t for me. I was miserable, and by the end of my first year, I was well on my way to failing outright with my entire future plans seemingly going up in smoke. Someone noticed That’s when George stepped in. He’d taught me computing right the way through my GCSEs, and with no A-Level course on offer, that was officially where his involvement was supposed to have ended. It didn’t. He had noticed my constant presence in the computing labs - before and after school, during lunch breaks, free “study” periods - working on some little pet project or digging into RISC OS internals. I remember him as warm, with a wicked, dry sense of humour, and a refreshingly spiky attitude to authority - I always got the sense he’d worked out for himself which rules were worth taking seriously and which ones weren’t. And he always had time for me. I spent years pestering him with questions that had nothing to do with anything on the syllabus, and he’d always find a way to answer them that actually made sense. He was just as supportive of my odd little obsessions. At one point I’d got deep into the BBS scene, which I thought was the coolest thing ever, and decided what the school really needed was an internal BBS running on its own network. So I wrote one. It was deeply cringeworthy, obviously - but George helped me put posters up around the school advertising it, and even gave it a mention in assembly one morning. I think about five people in total ever checked it out. It didn’t matter: a teacher had stood up in front of the entire school and treated my weird little project like it was worth something, and that was a hugely validating moment for me. Off the books He recognised the passion, and eventually he made a suggestion: What if I quit the Design and Technology course, and instead attempt the A-level course myself? Personally, I also suspect he was enjoying himself. There was some internal school politics behind why computing wasn’t offered at A-Level in the first place - I never knew the details - and I think the prospect of one of his students simply going out and getting the qualification anyway appealed to him on two separate levels. It would get me where I wanted to go, and it would wind up exactly the right people. It wouldn’t be easy, he warned. The school would be against it, plus it was a two-year course which I’d have to cram into one year. I’d have to do it all myself - studying, lesson planning, coursework - he couldn’t help me in an official capacity, but he said he’d advocate for me and help where he could. If I had assignments, he’d send them off to be graded and would give feedback in his own time. He’d enter me in for the exams and also gave me a set of keys to the computer lab so I could use it whenever I needed. It was the first time anybody outside my own family had really shown faith in my abilities and encouraged me to take a stand. It was a pivotal moment for me - I realised if I wanted something badly enough I would have to fight for it, but I could still make it happen. I didn’t have to take “NO” for an answer - a lesson I think I picked up from watching him as much as from anything he ever actually said to me. After a few weeks of dithering, I took the jump. I remember a few awkward meetings with the school administration but thanks to his behind-the-scenes work, I was soon following my dream. An intense year And yes, it was bloody hard work. I had to condense an entire two year course into under a year, be disciplined enough to produce my own study plan, and be critical enough of my own shortcomings that I could focus my study where it was needed. I pretty much lived and breathed it for months straight and was more-or-less a permanent fixture in the labs or school library poring over my course books. I’d make lists of questions and chat to George over lunch, and he’d provide guidance and encouragement. It was a lonely way to learn with no classmates to compare notes with, no lessons to turn up to, and right up until the end I had no real idea whether any of it was good enough - but bit by bit, it started to feel like something I could actually pull off. And sure enough, in the late spring of 1996, I sat down in an exam hall with my fellow students, the only one with an A-Level computing question paper in front of me. The final exam went by in a blur - I can only remember a few of the questions now (and a peculiar obsession with the Pascal language) - but I do remember the euphoria as the invigilator called “time’s up, pens down, close your papers NOW”. I had done it. A few nerve-wracking months later, my Mum drove me into school to pick up my results. I ripped open the envelope and saw it - I’d passed with an A grade! I literally ran up the stairs to George’s office next to the computing labs to thank him personally. I’d taken my camera into school to take a few last photos for memory’s sake and snapped this photo of him before I walked out the school gates for the last time: A different path Because of him, I managed to get into my university of choice, studying computing with a focus on networks. Because of that, I landed my first job working as a “webmaster”, and my career since has been one of the highlights of my life. All these years later, it’s a real privilege to be able to get up each morning and actively look forward to working in an industry I love. Without George stepping up for me and encouraging me to believe in myself, none of that would have happened. I wouldn’t have had the career I have, and I wouldn’t be where I am now. I met my wife when we both worked at a software company - she sat at the desk behind me - so even my home and family life can be traced back to that spring of 1996. And the A-Level itself was only half of what I took away from that year. The qualification opened the door to university, but the lesson that came with it was every bit as important: that a “no” isn’t always the end of it, and that sometimes the answer can be argued with. I’m so proud of what I managed to achieve all those years ago, and even more thankful to have had someone like George in my life to put me on the right track. Mr. Dryden I did see him again after I left. He drank in one of my local pubs - a pub I’d been going to for a good while before I was technically old enough to be in it - and I’d say hello if I spotted him in there, mostly in the months before I moved away to university. After that it was only a handful of times. For years I’d find myself scanning the room whenever I was back home and in for a pint, half expecting him to be at the bar. At some point I stopped seeing him altogether and eventually moved across the country. The trouble was I never really knew how to talk to him outside of school. He was always Mr. Dryden, or just “Sir”, I don’t think I ever once called him George to his face! I was (and still am if I’m honest) fairly socially awkward, and I never worked out how to phrase the thing I actually wanted to say: that he had changed the entire direction of my life, and I wasn’t sure he knew it. So instead I’d say hello, and ask how he was, talk about nothing much, and go back to my friends. Epilogue Sadly, 3 years ago now, I opened the latest issue of my old school alumni newsletter to read that he’d passed away. The photo at the start of this article was taken from his obituary article and I read that he’d had a long illness and had suffered from dementia at the end. I did write to him years ago by email - I don’t know if he ever got it, or was in any capacity to understand what he’d done for me, but I hope so. There’s an old saying by one of my favourite authors (Terry Pratchett) that “no one is finally dead until the ripples they cause in the world die away”. In one of his books, a character keeps the memory of his son alive by passing his name along a series of telegraph towers. It’s in that spirit that I’m writing this post - I debated it for many years as it’s very personal to me and I also have no contact with any of George’s family so I have no idea what they would make of it all. But even though it’s 30+ years ago now, I will never forget him or what he did for me - and at least now, if someone searches his name it’ll be recorded here for as long as I’m alive and running this site. Thank you, Sir. George Dryden 1942-2023

yesterday • 1 votes
How Copy-on-Write Works with Memory-Mapped Files

Let’s step inside the kernel and understand how it implements copy-on-write and what are its implications for the performance of user-space systems

yesterday • 1 votes
If we do not stop to help each other, what do we become?

Yesterday, I received this email as a response to You Can't Vibe Code Love. It's such a remarkable and powerful statement that I asked permission to share it here, in its entirety, with personal information redacted: Hey Jeff, Hope you and your family are doing well.

3 days ago • 1 votes
You are still valuable

A frustrated Reddit post about being a condom between an AI and production made the rounds in our team. Here is why I think the opposite is true and what it means for how we review code, plan work and think.

4 days ago • 2 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in