More from Sedimental
In my last post, I mentioned founding a startup. It's called FinFam, and we're building collaborative financial planning. The GitHub of money, if you will. Enough with the telling, time for the show! Here's a 3-minute demo: We launched beta this week and I couldn't be more excited. Let me tell you why. Where's this coming from? Growing up as the son of starving grad students, when it comes to money, I've been known to default to what we affectionately call "poor man brain." Cautious to a fault. But my student parents turned into scientists, so I'm eminently convinceable. I just need to see the math, or better yet, a spreadsheet. The problem? Making those spreadsheets. And more importantly, trusting them. A few years back, a friend was house hunting in the Bay Area. They came to me to talk through the famous rent-vs-buy problem. They didn't want an advisor to manage money or sell them products. They came to me because I worked in fintech and thus "knew money", even though I've only been involved in two home purchases, and wouldn't consider myself an expert. That experience crystallized something I'd been noticing everywhere: People trust their friends and family to have their best interests in mind... But can't trust them to have the best information. We can trust experts to have knowledge... but can't always trust their incentive alignment. In an era of ever-advancing information (and misinformation) glut, how do we get to a place of confidence in our hard-won justified true belief? Trust is social After 15+ years building fintech at PayPal and Stripe, I saw money movement simplified and commoditized. But the difficulty of transacting moved upstream. We made the how of buying easier, while advances in technology made the what, when, and why so much harder. From BNPL to crypto to vibecession, our economic realities aren't getting simpler. To compensate, 79% of young adults get financial guidance from social media (Forbes). Not because TikTok or YouTube has better models than Morgan Stanley, but because they trust the people sharing their stories. Millions of people are already collaborating on financial decisions. Privately on WhatsApp, obscurely on Discord, and full-blown publicly on Reddit: /r/personalfinance - 21 million members /r/financialindependence - 2.3 million /r/financialplanning - 1 million And dozens more subreddits and Internet forums (shoutout Bogleheads and Refinery29's Money Diaries). They're sharing detailed financial profiles with strangers on the internet, seeking advice from generous folks in full view. There's a fast-emerging story about AI here, and I've got whole posts dedicated to that coming soon. For now, suffice to say, we need the tools to catch up to the times. Enter FinFam FinFam[^name] lets families and friends collaborate on financial decisions with each other, using expert information without any commitments to said experts. We want to holistically solve the problem of financial decisionmaking using interaction models proven by GitHub + StackOverflow + app stores. How? First, creators publish interactive models, to FinFam's View marketplace. Then, you, a user who has a financial question or decision to make: Pick up a relevant expert view. If one doesn't exist, ask in FinFam's Q&A board. Plug in your own numbers and save it to a private workspace for your inner circle Discuss the results, with optional AI-assisted guidance as needed Move forward with confidence. We scale the expert knowledge while embracing the fundamentals of human social trust. Users get better decisions and peace of mind. Open-source meets fintech My years of work in open-source and wiki ecosystems showed me the power of collaborative, transparent tools. FinFam brings that same philosophy to personal finance. To further scale the knowledge, we make it possible for anyone to create a View. The View "source" format is XLSX, and can be edited with Google Sheets, Excel, or LibreOffice. Any published View can also be open-sourced. Just like with code, financial models are now reviewable, forkable, and improvable by the community. Curious users can check the community's math. Numbers and discussions happen with people you trust. Everything is private by default, shareable by design. What's next We launched beta this week and there are now daily spots available as we add capacity. You can sign up for early access here. I've been using it with friends and family for months now, and it has replaced Google Sheets for the financial decisions we face. There's so much more I want to share about the vision, the technology, and the journey so far. But this feels like we're off to a good start. Want to follow along? Subscribe to FinFam here and Sedimental here. If you're wondering about the name, just log in, go to your default space, create a thread, and ask Finn. ↩
Been quiet around here. Time to change that! The short version up front: Since starting a family and leaving Stripe, I've pursued the dream that brought me to Silicon Valley. I've founded a startup. After taking some parental leave, helping found a Python non-profit, and a nice long visit back home, I was raring for a challenge. So these days, outside of family, I'm all in on something new. Contents Why now? Applications Monetary misunderstandings Showing vs Telling Why now? I've wanted to start my own business since building Access apps in high school. But, the reality of leaving my family and moving to study in the USA, combined with the technical and creative fulfillment of the software industry, took me on a scenic route through enterprise software, free culture, and open-source. That very same reality has since conspired to convince me to return to my original aspirations. I've lived through some exciting times in software, but nothing like now. This isn't something I imagined I'd be working on 10 years ago, but then again it's not something I thought possible even 3 years ago. What better time to be building and launching my most ambitious project ever? Full details on that are coming soon1. For now, here is a post about why. Applications To start my career, I worked on software infrastructure, security, observability, and developer productivity. But after eight years, around 2016, I started longing for something more human. You can see this start to come out in The Packaging Gradient. At a time where it seemed like everyone around me was talking about pip, pipenv, and PyPI, I couldn't help but remind people that the real end goal of software has always been the application (or even the appliance). This impulse came to a head with APA. Perhaps you, dear reader, have also been "lost in the sauce" of software: When you love computers and it dominates your thoughts, you might also spend most of your time thinking about the software that makes the software possible. Don't get me wrong. Languages, libraries, compilers, devtools, we need every bit of help we can get. But I fell in love with software for its potential to effect change in the world writ large. I started eyeing product. The famous full stack. That meant moving on from big tech, to a big startup, to a seed startup. One pandemic-fueled detour through a startup factory later, here we are. Finally, founding the startup. My own full stack. Monetary misunderstandings My 15+ year software engineering career can be summed up as: Building fintech software for pay Shipping open-source Python/wiki for free Professionally enabling commerce while avoiding it in my personal time. I was young and conflicted. Truthfully, I still harbor some reservations, but I have to build what I know. I know about software and money. "Money is the root of all evil." If you look at the state of say, open banking in the USA, or web3isgoinggreat, or just read Money Stuff, you probably agree something's off. Money changes people. But so does the lack thereof. I've watched more talented and deserving developers than myself befall a variety of fates. Hollowed out by monetary excess, blinded by greed, burned out by FOSS, literally working Doordash to keep the lights on. Dropping out of software completely. Shunning the world's favorite fungible has bad outcomes for individuals. Bless my friends at Tidelift, OSTIF, and other orgs working to sustain the maintainers. Paying maintainers is a worthy battle. We just need to open more fronts to navigate what's in store. Showing vs Telling Lately I've been thinking a lot about my favorite David Lynch (RIP) scene. It isn't from one of his films, it's this quote: "The film is the talking." I think it perfectly captures the auteur mindset. Words are extraneous. The consummate creative expresses themselves better in their native medium. Not that I mind words as a medium. After years of blogging and speaking, I've grown confident in my ability to tell. But now it's time for the show. For friends who can't wait a couple weeks, shoot me an email for early access. ↩
Reliably tap into your creativity with the 4 Cs: Consume, critique, curate, create. This is one of my oldest ideas, finally published on the GitHub ReadME Project blog, along with a profile, in June 2022. For more like this, follow me on Twitter or Mastodon. (You can also read it in 中文 here. Thanks Dominic Huang!) We all have creative potential. Whether it gets you up in the morning or keeps you up at night, you've felt its gnaw. Turning that potential into productivity can prove challenging in an internet-connected environment that offers a constant stream of consumables. How do we pick a direction? In this guide, you'll see how to distill the elements of creativity into four deliberate stages, and how to put the process to use: Contents Consume Critique Curate Create Debugging the process Putting it into practice These 4 Cs comprise a straightforward, adaptable approach that works well in both group and solo settings. You've already started on step 1. Read on to find out what to do next. Consume Step 1: Turn passive consumption into active research. From the moment you open your eyes in the morning, you're accosted with calls to consume. Articles, videos, podcasts, the newest Wordle variant. When consumption is the default mode of our modern computing environment, how is a builder supposed to build? To create, we must first recognize its inverse: consumption. Consumption is a useful stage, but can be dangerous if it’s terminal. An infinite loop in this stage kills any chance of creation. Little is created in a vacuum. Creation still starts with consumption, albeit consumption disarmed with an intention. Turn pure consumption into active research, punctuated with critique. Critique Step 2: Capture your reactions in critiques, and research no faster than you can react. As with so many software problems of our day, the answer is simple: React. No, not the JavaScript framework, but the human act of reaction. Intentional creation starts with giving yourself pause on new inputs. Seek a reaction from yourself. A semi-structured reaction, or critique, is a time-honored practice in creative fields, like architecture. Infinite scroll may prove challenging to overcome, but before turning your consideration to the next item in your feed, activate your critical senses. Draw some conclusions. Even unvetted, they're yours. If you're finding it hard to summon a critique, this is a clear sign you're consuming faster than you can reflect. If you're not reflecting, you're not learning. You may need to go deeper on individual items, or just take a break. Your critiques have never been easier to capture, whether typed in markdown, dictated to automatic transcription, or written down in a notepad on your desk. Try opening your editor or critique tool before opening any new resources. Feel free to open one now. If you're not reflecting, you're not learning. Curate Step 3: Curate critiques into collections that act as reservoirs of creative reference. Critiques are only proto-creative output. Writing anything helps prime the creative pump, but criticism is raw reaction. You want a refined synthesis. Once you've got enough critiques under your belt, curate the positive examples into a collection. From interior designers to lab researchers to club DJs, creators recognize the value of a structured, referenceable collection. Sometimes a situation calls for urgency or direction, and an organized, well-researched collection can offer an existing solution. Sometimes, in the context of a comprehensive collection, the lack of a referenceable solution is itself a signal that it's time to invent. Curated collections become artifacts unto themselves. I've helped create a few, including 0ver.org, seealso.org, and the Awesome Python Applications list. There’s more awesome out there beyond Awesome Lists, like explorabl.es, the Cooperpress newsletters, or the “swipe file” phenomenon used among designers and content creators. There's respectable work in curation. Still, curation is more important as a stepping stone to our original higher calling. Less is more. Create Step 4: Return to your curations regularly to discover your creative path forward. Collections of a certain size tend to produce interesting findings. Patterns and gaps emerge that inspire creative next steps. As an example, while researching approaches to Python packaging, a pattern emerged that led to one of my most popular concepts/blog posts/talks, The Packaging Gradient. Whole projects can be born out of connections made with collections. My framework Clastic, which was eventually used by teams at PayPal and Wiki Loves Monuments, came out of the curated combination of pytest dependency-injection semantics with werkzeug primitives. Realistically, the majority of creation happens below the threshold of standalone artifacts. For instance, when adding a feature to an existing system, a parallel approach in a different project serves as a useful guide. I've lost track of the number of times I've swiped techniques from Awesome Python Applications, including ones used to port my dayjob's 300k SLOC codebase from Python 2 to 3. Most creative outputs have a similar lineage. Only now we have an explicit process. Debugging the process It's easy to see creations we appreciate as towering achievements that sprung fully-formed from their creators' genius. But creation comes in fits and starts. If creation comes slowly, here are a few strategies to consider: Search for a natural split in an existing collection that's getting too big, and explore what makes it interesting. Revisit an old, contentious critique and re-react. What did you get right/wrong? Pick a particular exemplar and turn it into a case study. One beautiful aspect of FOSS projects is that going deep can mean getting involved. There's nothing like proximity to a problem to inspire creative thinking. More generally, be wary of one-size-fits-all solutions; while prescriptive techniques such as the Zettelkasten Method may work for some, creation is idiosyncratic. Embrace your own process. Putting it into practice When inspiration hits, connections can form so quickly that we take for granted what goes on. When inspiration proves less willing to strike, we can keep ourselves primed for creativity by ensuring all four activities continue in balance. There are a few notable benefits of intentional creation: When you've built something, the influences are well-documented. It can be easier to involve others when there's a clear creative thread to pull on. Sharing your critiques and curations invites collaboration with other creators and curators. Self-awareness. If you're not finding your critiques crystallizing into new thoughts and ideas for projects, that's a sign you're looking at the wrong stuff. Are you following your interests or passively consuming trending content? Practically, intentional creation means consciously spending less time on consumer sites, from Twitter to Hacker News, and more time taking notes, tagging bookmarks, and creating your own knowledge base. Attempt activities that are less entertainment and more you, ultimately closing the gap between you and your creative goals. If it sounds too simple, that's because it is. You're still accountable to you, that's the hard part. But hopefully you'll find some value in this simple hierarchy that lets you check in on your own activities and make adjustments toward a more creative end. Spend less time consuming, and more time on the other three Cs. Consume only enough to allow yourself to critique, curate, and create. If you made it this far, then start now. Step 2. Use any tool or service you like, from spreadsheets to YAML, and answer this: What's your critique?
2020 was a year of reckonings. And for all that was beyond one’s control, as the year went on, I found myself pouring more and more into the one thing that felt within reach: futureproofing of the large enterprise web application I helped build, SimpleLegal. Now complete, this replatforming easily ranks in my most complex projects, and right now, holds the top spot for the happiest ending. That happiness comes at a cost, but with some the right approach that cost may not be as high as you think. Contents The Bottom Line The Setup The Outset The Traction Issues The Sentry Pivot The New Road Committing to transactions The truly atomic request Transactional test setup Better than best practices The utility of namespaces Coverage tools Flattening database migrations Easing onto the stack The Rollout The Aftermath The Bottom Line We took SimpleLegal’s primary product, a 300,000 line Django-1.11-Python 2.7-Redis-Postgres-10 codebase, to a Django 2.2-Python 3.8-Postgres-12 stack, on-schedule and without major site incidents. And it feels amazing. Speaking as tech lead on the project, what did it look like? For me, something like this: But as Director of Engineering, what did it cost? 3.5 dev years and just about $2 per line of code. And I'm especially proud of that result, because along the way, we also substantially improved the speed and reliability of both the site and development process itself. The product now has a bright future ahead, ready to shine in sales RFPs and compliance questionnaires. Most importantly, there’ll be no worrying about when to delicately break it to a candidate that they’ll be working with unsupported technology. In short, a large, solid investment that’s already paying for itself. If you just came here for the estimate we wish we had, you've got it. This post is all about how your team can achieve the same result, if not better. The Setup The story begins in 2013, when a freshly YC-incubated SimpleLegal made all the right decisions for a new SaaS LegalTech company: Python, Django, Postgres, Redis. In classic startup fashion, features came first, unless technology was a blocker. Packages were only upgraded incidentally. By 2019, the end of this technical runway had drawn near. While Python 2 may be getting extended support from various vendors, there were precious few volunteers in sight to do Django 1 CVE patches in 2021. A web framework’s a riskier attack surface, so we finally had our compliance forcing function, and it was time to pay off our tech debt. The Outset So began our Tech Refresh replatforming initiative, in Q4 2019. The goal: Upgrade the stack while still shipping features, like changing the tires of a moving car. We wanted to do it carefully, and that would take time. Here are some helpful ground rules for long-running projects: Any project that gets worked on 10+ hours per week deserves a 30-minute weekly sync. Every recurring meeting deserves a log. Put it in the invite. Use that Project Log to record progress, blockers, and decisions. It’s a marathon, not a sprint. Avoid relying on working nights, weekends, and holidays. We started with a sketch of a plan that, generously interpreted, ended up being about halfway correct. Some early guesses that turned into successes: Move to pip-tools and unpin dependencies based on extensive changelog analysis. Identify packages without py23 compatible versions. (Though we’ve since moved to poetry.) Add line coverage reporting to CI Revamp internal testing framework to allow devs to quickly write tests More on these below. Other plans weren’t so realistic: Take our CI from ~60% to 95% line coverage in 6 months Parallelized conversion of app packages over the course of 3 months Use low traffic times around USA holidays (Thanksgiving, Christmas, New Years) to gradually roll onto the new app before 2021. We were young! As naïve as we were, at least we knew it would be a lot of work. To help shoulder the burden, we scouted, hired, and trained three dedicated off-shore developers. The Traction Issues Even with added developers, by mid-2020 it was becoming obvious we were dreaming about 95% coverage, let alone 100%. Total coverage may be best practice, but 3.5 developers couldn’t cover enough ground. We were getting valuable tests, and even finding old bugs, but if we stuck with the letter of the plan, Django 2 would end up being a 2022 project. At 70%, we decided it was time to pivot. We realized that CI is more sensitive than most users for most of the site. So we focused in on testing the highest impact code. What’s high-impact? 1) the code that fails most visibly and 2) the code that’s hardest to retry. You can build an inventory of high-impact code in under a week by looking at traffic stats, batch job schedules, and asking your support staff. Around 80% of the codebase falls outside that high-traffic/high-impact list. What to do about that 80%? Lean in on error detection and fast time-to-fix. The Sentry Pivot One nice thing about startup life is that it’s easy to try new tools. One practice we’ve embraced at SimpleLegal is to reserve every 5th week for developers to work on the development process itself, like a coordinated 20% time. Even the best chef can’t cook five-star food in a messy kitchen. This was our way of cleaning up the shop and ultimately speeding up the ship. During one such period, someone had the genius idea to add dedicated error reporting to the system, using Sentry. Within a day or two, we had a site you could visit and get stack traces. It was pretty magical, and it wasn’t until Tech Refresh that we realized that while integration takes one dev-day, full adoption can take a team months. You see, adding Sentry to a mature-but-fast-moving system means one thing: noise. Our live site was erroring all the time. Most errors weren’t visible or didn’t block users, who in some cases had quietly learned to work around longstanding site quirks. Pretty quickly, our developers learned to treat Sentry as a repository of debugging information. A Sentry event on its own wasn’t something to be taken seriously in 2019. That changed in 2020, with the team responsible for delivering a seamless replatform needing Sentry to be something else: a responsive site quality tool. How did we get there? First step, enhance the data flowing into Sentry by following these best practices: Split up your products into separate Sentry projects. This includes your frontend and backend. Tag your releases. Don’t tag dev env deployments with the branch, it clutters up the Releases UI. Add a separate branch tag for searches. Split up your environments. This is critical for directing alerts. Our Sentry client environment is configured by domain conventions and Django’s sites framework. If it helps, here's a baseline, we use these environments: Production: Current official release. DevOps monitored. Sandbox: Current official release (some companies do next release). Used by customers to test changes. DevOps monitored. Demo/Sales: Previous official release. Mostly internal traffic, but external visibility at prospect demo time. DevOps monitored. Canary: Next official release. Otherwise known as staging. Internal traffic. Dev monitored. ProdQA: Current official release. Used internally to reproduce support issues. Dev monitored. QA: Dev branches, dev release, internal traffic. Unmonitored debugging data. Local test/CI: Not published to Sentry by default. With issues finally properly tagged and searchable, we used Sentry’s new Discover tool to export issues weekly, and prioritize legacy errors. To start, we focused on high-visibility production errors with non-internal human users. Our specific query: has:user !transaction:/api/* event.type:error !user.username:*@simplelegal.* We triaged into 4 categories: Quick fix (minor bug), Quick error (turn an opaque 500 error into a actionable 400 of some form), Spike (larger bug, requires research), and Silence (using Sentry’s ignore feature). Over 6 weeks we went from over 2500 weekly events down to less than 500. Further efforts have gotten us under 100 events per week, spread across a handful of issues, which is more than manageable for even a lean team. While "Sentry Zero" remains the ideal, we achieved and maintained the real goal of a responsive flow, in large part thanks to the Slack integration. Our team no longer hears about server errors from our Support team. In fact, these days, we let them know when a client is having trouble and we’ve got a ticket underway. And it really is important to develop close ties with your support team. Embedded in our strategy above was that CI is much more sensitive than a real user. While perfection is tempting, it’s not unrealistic to ask a bit of patience from an enterprise user, provided your support team is prepared. Sync with them weekly so surprise is minimized. If they’re feeling ambitious, you can teach them some Sentry basics, too. The New Road With noise virtually eliminated, we were ready to move fast. While the lean-in on fast-fixing Sentry issues was necessary, a strong reactive game is only useful if there are proactive changes being pushed. Here are some highlights we learned when making those changes: Committing to transactions Used properly, rollbacks can make it like errors never happened, the perfect complement to a fast-fix strategy. The truly atomic request Get as much as possible into the transactions. Turn on ATOMIC_REQUESTS, if you haven’t already. Some requests do more than change the database, though, like sending notifications and enqueuing background tasks. At SimpleLegal, we rearchitected to defer all side effects (except logging) until a successful response was being returned. Middleware can help, but mainly we achieved this by getting rid of our Redis queue, and switching to a PostgreSQL-backed task queue/broker. This arrangement ensures that if an error occurs, the transaction is rolled back, no tasks are enqueued, and the user gets a clean failure. We spot the breakage in Sentry, toggle over to the old site to unblock, and their next retry succeeds. Transactional test setup Transactionality also proved key to our testing strategy. SimpleLegal had long outgrown Django’s primitive fixture system. Most tests required complex Python to set up, making tests slow to write and slow to run. To speed up both writing and running, we wrapped the whole test session in a transaction, then, before any test cases run, we set up exemplary base states. Test cases used these base states as fixtures, and rolled back to the base state after every test case. See this conftest.py excerpt for details. Better than best practices Software scenarios vary so widely, there’s an art to knowing which advice isn’t for you. Here’s an assortment of cul de sacs we learned about firsthand. The utility of namespaces Given how code is divided into modules, packages, Django apps, etc., it may be tempting to treat those as units of work. Don’t start there. Code divisions can be pretty arbitrary, and it’s hard to know when you’ve pulled on a risky thread. Assuming there are automated refactorings, as in a 2to3 conversion, start by porting by type of transformation. That way, one need only review a command and a list of paths affected. Plus, automated fixes necessarily follow a pattern, meaning more people can fix bugs arising from the refactor. Coverage tools Coverage was a mixed bag for us. Obviously our coverage-first strategy wasn’t tenable, but it was still useful for prioritization and status checks. On a per-change basis, we found coverage tools to be somewhat unreliable. We never got to the bottom of why coverage acted nondeterministically, and we left the conclusion at, “off-the-shelf tools like codecov are probably not targeted at monorepos of our scale.” In running into coverage walls, we ended up exploring many other interpretations of coverage. For us, much higher-priority than line coverage were “route coverage” (i.e., every URL has at least one integration test) and “model repr coverage” (i.e., every model object had a useful text representation, useful for debugging in Sentry). With more time, we would have liked to build tools around those, and even around online-profiling based coverage statistics, to prioritize the highest traffic lines, not just the highest traffic routes. If you’ve heard of approaches to these ends, we’d love to discuss them with you. Flattening database migrations On the surface, reducing the number of files we needed to upgrade seems logical. Turns out, flattening migrations is a low-payoff strategy to get rid of files. Changing historical migration file structure complicated our rollout, while upgrading migrations we didn’t flatten was straightforward. Not to mention, if you just wanted the CI speedup, you can take the same page from the Open EdX Platform that we did: build a base DB cache that you check in every couple months. Turns out, you can learn a lot from open-source applications. Easing onto the stack If you have more than one application, use the smaller, simpler application to pilot changes. We were lucky enough to have a separate app whose tests ran faster, making for a tighter development loop we coul learn from. Likewise, if you have more than one production environment, start rollouts with the one with the least impact. Clone your CI jobs for the new stack, too. They’ll all fail, but resist the urge to mark them as optional. Instead, build a single-file inventory of all tests and their current testing state. We built a small extension for our test runner, pytest, which bulk skipped tests based on a status inventory file. Then, ratchet: unskip and fix a test, update the file, check that tests pass, and repeat. Much more convenient and scannable than pytest mark decorators spread throughout the codebase. See this conftest.py excerpt for details. The Rollout In Q4 2020, we doubled up on infrastructure to run the old and new sites in parallel, backed by the same database. We got into a loop of enabling traffic to the new stack, building a queue of Sentry issues to fix, and switching it back off, while tracking the time. After around 120 hours of new stack, strategically spread around the clock and week, enough organizational confidence had been built that we could leave the site on during our most critical hours: Mondays and Tuesdays at the beginning of the month. The sole hiccup was an AWS outage Thanksgiving week. At this point we were ahead of schedule, and enough confidence had been built in our fast-fix workflow that we didn’t need our original holiday testing windows. And for that, many thanks were given. We kept at the fast-fix crank until we were done. Done isn't when the new system has no errors, it's when traffic on the new system has fewer events than the old system. Then, fix forward, and start scheduling time to delete the scaffolding. The Aftermath So, once you’re on current LTS versions of Django, Python, Linux, and Postgres, job complete, right? Thankfully, tech debt never quite hits 0. While updating and replacing core technologies on a schedule is no small feat, replacing a rusty part with a shiny one doesn’t change a design. Architectural tech debt -- mistakes in abstractions, including the lack thereof -- can present an even greater challenge. Solutions to those problems don’t generalize between projects as cleanly, but they do benefit from up-to-date and error-free foundations. For all the projects looking to add tread to their technical tires, we hope this retrospective helps you confidently and pragmatically retrofit your stack for years to come. Finally, big thanks to Uvik for the talent connection, and the talent: Yaroslav, Serhii, and Oleh. Shoutouts to Kurt, Justin, and Chris, my fellow leads. And the cheers to business leadership at SimpleLegal and everywhere, for seeing the value in maintainability.
Thought I'd take a Sunday afternoon to reflect on, oh I don't know, a decade. Been a long ten years, but it's flown past. This particular decade happens to coincide with my first years of full-time professional software engineering. The Quantity I can't possibly summarize it all, and if I tried, it'd still be colored by what's on my mind right now. But I can point to the artifacts I tried to leave along the way: Twitter FWIW1 (2008+) ~20 Open-Source Projects (2012+) ~15 Hatnote Projects (2013+, follow us) ~25 entries on this blog (2015+) +7 here (2014-2016) Not including pythondoeswhat.com or blog.hatnote.com (or other posts on the blogs only real heads know) ~10 Talks (2016+) Lest I forget: O'Reilly's Enterprise Software with Python (2016) And several podcast/media appearances calver.org (2016) and 0ver.org (2018) (Versioning is a fun pastime) Pyninsula (2017+) - YouTube, Meetup, Email Announce Taking a chronological look at each of the above, I'm relieved to see obvious growth. If I were to highlight one resource, it would probably be the talks. Despite the stress of preparation and delivery, I'm least concerned with having a massive miscommunication when we're all in the room and I can see the points hitting home. It's impossible to pick a favorite, but Ask the Ecosystem (2019), the Restructuring Data lightning talk (2018), and The Packaging Gradient (2017) seem like audience faves from where I'm sitting. The Quality Each project, post, and talk had its own reward, but I guess I've got more than just those to show for the decade. On the more profit-driven side, I built tools and teams at PayPal, but once I could manage the risk, I got to dip into startups for the last few years. Lucky for me, it wasn't a total bust, and the wife and I bought a place in my favorite neighborhood (in the USA). Not a millionaire, but I'm hoping and working for a world where no one has to be. More recently, the Python Software Foundation made me a Fellow. This isn't something I can be nonchalant about, and I'm not going to understate how much this means, to me, working in a field like software, where concrete symbols of progress are alternatingly elusive and vanishing. Plus it's Python, and reciprocated love is nice. I have hundreds of people to thank for helping me reach this point, and I have to thank the PSF for dedicating the time to ramping up these awards. They've convinced me more than ever that we need more institutions to build this sort of advancement. To all of you, thank you. The Struggle I like to think I managed to do all of the above while staying away from industry hype, on the principle that massive speculative capital influx isn't where real value is added to society, and doesn't generate the kind of innovation that excites me. I may have been naïve, but I came to Silicon Valley with an idea about the transformative power of software. Changing times may illustrate a grittier interpretation than the one I had and have, but I continue to hold dear software's potential for positive impact. If you've felt that vision waver, let me tell you, you're not alone. In the past decade, I've seen too many engineers sucked in by new technologies and ventures, only to find themselves alienated from their work. Episodes ranging from an afternoon lost to debugging Docker/k8s clusters, to years of work disappearing at the end of a VC runway. Nothing has been harder to watch than those bedraggled-but-persistent idealists regroup, each time a bit more cynical than the last. Even if its seeming intractibility has taken it from the center stage, the burnout conversation continues to smolder, because there's no issue realer. I know; I released more ceramics than software back in 2014. Some problems can be solved by paying the maintainers, but I think the vastly bigger issue is around losing the human connection between the real effort software takes and the real benefits it brings, combined with FOSS's dearth of collaborators in supporting roles (QA, product/project/release management). That's why I'm incredibly thankful for the Wikimedia community for always being there, patient with schedules and issues, as long as the software got the job done. It can be a challenge to juggle projects, but I tell every budding engineer: find that direct connection to people who will appreciate your work, and avoid cynicism at all costs. There are some interesting prospects in the works, but I'm keeping this post retro. Besides, if 2029 rolls around and all I did was break even with 2009-19, I don't see how I can be disappointed. Thanks again for everything in 201X, and for sticking with me in 202X. Despite using Twitter for over a decade, the process of tweeting feels so perfunctory, and the service itself so tenuous, that I still can't bring myself to invest the time. I mostly use it to crosspost my blog posts or help friends promote their posts/projects. But until I start an email newsletter, or really get on top of yak.party, it's still the best I got for announcing where I'm speaking next. ↩
More in programming
I listen to a lot of podcasts, and I like how they fit around other tasks. I press play, lock my phone, and put it down. I’m free to wash the dishes, fold the laundry, or shop for groceries. Unfortunately, more and more information is only published as a video. Technical talks, conference sessions, video essays – they don’t work in an audio-only podcast app. I could convert these videos to MP3 files, but that breaks down the moment a video isn’t pure spoken word. If a speaker says, “Look at this slide” or holds up a diagram, an audio-only file leaves me stranded. I don’t want to give up the podcast player I like, nor stare at a screen for an hour – but I do want the information in these videos. To solve this, I’m abusing my podcast player’s chapter support. This gives me the best of both worlds: I can listen to a video as audio-first, and glance at my lock screen if I need a moment of visual context. The idea: Chapters every few seconds MP3 files can have ID3 metadata, and ID3 metadata can include chapters. A chapter covers a particular time range, and it can have an associated title, description, and cover art. My podcast app of choice is Overcast, which can’t play videos, but it does have robust chapter support. I can jump between chapters, navigate a table of contents, and see per-chapter cover art. To get videos into Overcast, I’m creating MP3 files with a new chapter every few seconds, and the per-chapter cover art is a corresponding frame from the video. As I play the file, I get a slow, stop-motion-like rendition of the original video. If my phone is locked, I can glance at my lock screen and see the current frame in the Now Playing screen. Overcast is developed by Marco Arment, and I got this idea from Forecast, his app for adding chapters to podcasts. In particular, I was struck by its ability to create chapters that don’t display in the chapter list – ideal if I don’t want a table of contents with hundreds of entries. As I was developing my script, I compared my output to the output from Forecast to ensure I was creating the chapters correctly. The code: FFmpeg and Mutagen There are three steps in this process: Convert a video file to an MP3 Extract images from the video at a fixed interval Insert the images as hidden chapters in the MP3 file Let’s go through each in turn. 1. Convert a video file to an MP3 Converting a video file to an MP3 is a single FFmpeg command: ffmpeg -i video.mp4 audio.mp3 This is consistently the slowest step of the process, and I do wonder if I could use different settings or an alternative encoder to make it go faster – but it’s not slow enough to be worth further investigation. 2. Extract images from the video at a fixed interval Extracting images from a video needs a more complicated FFmpeg command: ffmpeg -i video.mp4 \ -vf 'fps=1/5,scale=iw*sar:ih,scale=min(iw\,945):min(ih\,945):force_original_aspect_ratio=decrease' \ thumbnail_%04d.jpg This extracts an image every 5 seconds, downscales any image larger than 945 pixels square (while preserving the original aspect ratio), and saves the results as sequentially numbered JPEG images (thumbnail_0001.png, thumbnail_0002.png, and so on). The key is the -vf flag, which defines two FFmpeg filters: The fps filter selects one frame every 5 seconds (fps=1/5). The first scale filter scales the width based on the sample aspect ratio (scale=iw*sar:ih). Without this filter, frames can be stretched and distorted. The second scale filter scales the input video, preserving the original aspect ratio (force_original_aspect_ratio=decrease), and ensuring the output images fit within 945×945px or the size of the input video, whichever is smaller. My limit is 945 pixels because that’s the largest size that cover art is shown on my iPhone. This filter still isn’t completely correct – it sometimes creates images from portrait videos that are smaller than I’m expecting – but it’s good enough. These are only thumbnails for glancing at, and if I want to change it later, I can always do the image resizing outside FFmpeg. 3. Insert the images as hidden chapters in the MP3 file Inserting the chapters into the MP3 file is more complicated. Although FFmpeg has basic support for ID3 metadata, as far as I know, it can’t insert chapters with per-chapter artwork. Instead, I’m going to reach for Python and the Mutagen library. Here’s the code to add a chapter to an MP3 file: from mutagen.id3 import APIC, CHAP, ID3, PictureType audio = ID3("audio.mp3") with open("thumbnail_0001.jpg", "rb") as f: img_data = f.read() image_frame = APIC(mime="image/jpeg", type=PictureType.OTHER, data=img_data) chapter_frame = CHAP( element_id="chp1", start_time=0, end_time=5 * 1000, sub_frames=[image_frame] ) audio.add(chapter_frame) audio.save() This creates a single chapter that lasts the first 5 seconds (0 to 5000 milliseconds), and the per-chapter cover art is thumbnail_0001.jpg. If we ran this in a loop, we could add images for every 5 second slice of the original video. This code is inserting two frames into the ID3 metadata: The CHAP (chapter) frame contains the timing information, and it can have subframes for metadata like title, chapter art, or associated URL. The APIC (attached picture) subframe contains information about a picture, which can either be a blob of image data or a URL to an image on the web. Normally, you’d also insert a CTOC frame which defines a table of contents, but I don’t want a TOC with hundreds of 5-second chapters, so I’m deliberately not doing this here. This is allowed by the ID3 spec – you’re not required to insert a CTOC frame if you’re using chapters, and you can have chapters that aren’t listed in your table of contents. To work out which frames I needed, I used Forecast to create some chapters by hand, and I inspected their frames. In particular, loading an MP3 and calling Mutagen’s pprint() method shows a human-readable list of frames, and then I could drill into the individual fields: from mutagen.id3 import ID3 audio = ID3("audio.mp3") print(audio.pprint()) I wrapped all this code in a project called glancecast, which allows you to convert a video file with a single command, with optional flags to set the frame length and chapter art size: $ python3 glancecast.py interesting_talk.mp4 interesting_talk.mp3 The process takes a minute or so to complete, most of which is spent transcoding the video file to MP3. The resulting MP3s are usually 40 to 50 MB in size, which is very reasonable. The outcome: How it looks in practice Here’s what one of these “glanceable” podcasts looks like in Overcast and on my lock screen: Maggie Appleton presented this talk over two years ago and it’s been on my “talks to watch” list ever since. Once I put it in Overcast? I listened to it in less than a day. It’s not a lot of extra information, but enough that I can quickly glance down and get the gist of what a speaker is saying. Both views update with a new frame every few seconds, or I can put my phone in my pocket and ignore the screen. I’ve used this approach for half a dozen videos so far, and I’m happy with the results. I expect to keep using it, because I have a long queue of videos I’ve been meaning to watch. If you’d like to try this, check out glancecast for the full code and instructions. [If the formatting of this post looks odd in your feed reader, visit the original article]
Andrew Baker, the current Group CIO at Capitec Bank wrote an interesting piece on AI and open source, and how these tools that generate code according to one’s specification may replace the general reliance on open source implementations done by contributors around the world. I’d really recommend reading it. I have great admiration and respectContinue reading "AI Isn’t Replacing Open Source"
A framework for thinking about when AI involvement is additive or a violation
Why we need richer, thicker interfaces and better boundary objects for collaborative planning with agents
A look at 10 foundational pillars that enable agents to operate more competently and more efficiently in any codebase.