More from Notes on software development
Last month I completed my first year at EnterpriseDB. I'm on the team that built and maintains pglogical and who, over the years, contributed a good chunk of the logical replication functionality that exists in community Postgres. Most of my work, our work, is in C and Rust with tests in Perl and Python. Our focus these days is a descendant of pglogical called Postgres Distributed which supports replicating DDL, tunable consistency across the cluster, etc. This post is about how I got here. Black boxes I was a web developer from 2014-2021†. I wrote JavaScript and HTML and CSS and whatever server-side language: Python or Go or PHP. I was a hands-on engineering manager from 2017-2021. I was pretty clueless about databases and indeed database knowledge was not a serious part of any interview I did. Throughout that time (2014-2021) I wanted to move my career forward as quickly as possible so I spent much of my free time doing educational projects and writing about them on this blog (or previous incarnations of it). I learned how to write primitive HTTP servers, how to write little parsers and interpreters and compilers. It was a virtuous cycle because the internet (Hacker News anyway) liked reading these posts and I wanted to learn how the black boxes worked. But I shied away from data structures and algorithms (DSA) because they seemed complicated and useless to the work that I did. That is, until 2020 when an inbox page I built started loading more and more slowly as the inbox grew. My coworker pointed me at Use The Index, Luke and the DSA scales fell from my eyes. I wanted to understand this new black box so I built a little in-memory SQL database with support for indexes. I'm a college dropout so even while I was interested in compilers and interpreters earlier in my career I never dreamed I could get a job working on them. Only geniuses and PhDs did that work and I was neither. The idea of working on a database felt the same. However, I could work on little database side projects like I had done before on other topics, so I did. Or a series of explorations of Raft implementations, others' and my own. Startups From 2021-2023 I tried to start a company and when that didn't pan out I joined TigerBeetle as a cofounder to work on marketing and community. It was during this time I started the Software Internals Discord and /r/databasedevelopment which have since kind of exploded in popularity among professionals and academics in database and distributed systems. TigerBeetle was my first job at a database company, and while I contributed bits of code I was not a developer there. It was a way into the space. And indeed it was an incredible learning experience both on the cofounder side and on the database side. I wrote articles with King and Joran that helped teach and affirm for myself the basics of databases and consensus-based distributed systems. Holding out When I left TigerBeetle in 2023 I was still not sure if I could get a job as an actual database developer. My network had exploded since 2021 (when I started my own company that didn't pan out) so I had no trouble getting referrals at database companies. But my background kept leading hiring managers to suggest putting me on cloud teams doing orchestration in Go around a database rather than working on the database itself. I was unhappy with this type-casting so I held out while unemployed and continued to write posts and host virtual hackweeks messing with Postgres and MySQL. I started the first incarnation of the Software Internals Book Club during this time, reading Designing Data Intensive Applications with 5-10 other developers in Bryant Park. During this time I also started the NYC Systems Coffee Club. Postgres After about four months of searching I ended up with three good offers, all to do C and Rust development on Postgres (extensions) as an individual contributor. Working on extensions might sound like the definition of not-sexy, but Postgres APIs are so loosely abstracted it's really as if you're working on Postgres itself. You can mess with almost anything in Postgres so you have to be very aware of what you're doing. And when you can't mess with something in Postgres because an API doesn't yet exist, companies have the tendency to just fork Postgres so they can. (This tendency isn't specific to Postgres, almost every open-source database company seems to have a long-running internal fork or two of the database.) EnterpriseDB Two of the three offers were from early-stage startups and after more than 3 years being part of the earliest stages of startups I was happy for a break. But the third offer was from one of the biggest contributors to Postgres, a 20-year old company called EnterpriseDB. (You can probably come up with different rankings of companies using different metrics so I'm only saying EnterpriseDB is one of the biggest contributors.) It seemed like the best place to be to learn a lot and contribute something meaningful. My coworkers are a mix of Postgres veterans (people who contributed the WAL to Postgres, who contributed MVCC to Postgres, who contributed logical decoding and logical replication, who contributed parallel queries; the list goes on and on) but also my developer-coworkers are people who started at EnterpriseDB on technical support, or who were previously Postgres administrators. It's quite a mix. Relatively few geniuses or PhDs, despite what I used to think, but they certainly work hard and have hard-earned experience. Anyway, I've now been working at EnterpriseDB for over a year so I wanted to share this retrospective. I also wanted to cover what it's like coming from engineering management and founding companies to going back to being an individual contributor. (Spoiler: incredibly enjoyable.) But it has been hard enough to make myself write this much so I'm calling it a day. :) I wrote a post about the winding path I took from web developer to database developer over 10 years. pic.twitter.com/tf8bUDRzjV — Phil Eaton (@eatonphil) February 15, 2025 † From 2011-2014 I also did contract web development but this was part-time while I was in school.
A small standard library means an explosion in transitive dependencies. A more comprehensive standard library helps you minimize dependencies. Don't misunderstand me: in a real-world project, it is practically impossible to have zero dependencies. Armin Ronacher called for a vibe shift among programmers and I think that this actually exists already. Everyone I speak to on this topic has agreed that minimizing dependencies is ideal. Rust and JavaScript, with their incredibly minimal standard libraries, work against this ideal. Go, Python, Java, and C# in contrast have a decent standard library, which helps minimize the explosion of transitive dependencies. Examples I think the standard library should reasonably include: JSON, CSV, and Parquet support HTTP/2 support (which includes TLS, compression, random number generation, etc.) Support for asynchronous IO A logging abstraction A SQL client abstraction Key abstract data types (BTrees, hashmaps, sets, and growable arrays) Utilities for working with Unicode, time and timezones But I don't think it needs to include: Excel support PostgreSQL or Oracle clients Flatbuffers support Niche data structures Neither of these are intended to be complete lists, just examples. Walled gardens Minimal standard libraries force growing companies to build out their own internal collection of "standard libraries". As one example, Bloomberg did this with C++. And I've heard of companies doing this already with Rust. This allows larger companies to manage and minimize the explosion of transitive dependencies over time. All growing companies likely do something like this eventually. But again, smaller standard libraries incentivize companies to build this internal standard library earlier on. And the community benefits relatively little from these internal standard libraries. The community would benefit more if large organizations contributed back to an actual standard library. Smaller organizations do not have the capacity to build these internal standard libraries. Maybe the situation will lead to libraries like Boost for JavaScript and Rust programmers. That could be fine. Versioning A comprehensive standard library does not prevent the language developers from releasing new versions of the standard library. It is trivial to do this with naming like Go has done with the v2 pattern. math/rand/v2 is an example. Conclusion My concern about the standard library does not stop me from using Rust and JavaScript. Furthermore, they could choose to invest in the standard library at any time. We have already begun to see Bun and Deno to do exactly this. But it is clearly an area for improvement in Rust and JavaScript. And a mistake for other languages to avoid repeating. While zero dependencies is practically impossible, everyone I've spoken to agrees that minimizing dependencies is ideal. Rust and JavaScript work against this ideal. But they could change at any time. And Bun and Deno are already examples of this.https://t.co/qkSh6oW1Yd pic.twitter.com/mY1MNErZG7 — Phil Eaton (@eatonphil) January 25, 2025
This is an external post of mine. Click here if you are not redirected.
This is an external post of mine. Click here if you are not redirected.
More in technology
An update to my app, Quick Reviews, is rolling out now! After a lot of feature adds in the first few updates, I took this as a chance to clean up some bugs and improve a few elements that made the app a little annoying to use at times. Oh,
The age-old combination of physical locks and keys, although reliable, also comes with a few drawbacks, such as when you lose the key or you want to share access with someone else remotely. Davide Gomba has recognized this and built the MKR Keylock project as a way to address some of these shortcomings. Starting with an existing electronic […] The post MKR Keylock is an open-source IoT keypad for your front door appeared first on Arduino Blog.
Week 7 of my humanities crash course had me exploring ancient Mesopotamia with a side trip to northern India. I also watched an Iranian film that had me pondering the meaning of life. Readings This week, I read two short ancient texts: the Epic of Gilgamesh and the Dhammapada. Let’s tackle them in order. I’d never read any ancient Mesopotamian literature, so this was all new to me: the pantheon, story, style, etc. were thrillingly unfamiliar. Gilgamesh is around 1,500 years older than Homer, and it shows: there are lots of repetitive passages and what felt like archaic writing. But human nature hasn’t changed much in 4,700 years. People still love, hate, drink, eat, etc. – and they still fear death. Gilgamesh is the awe-inspiring, despotic king of Uruk. The gods answer his beleaguered subjects’ prayers in the form of Enkidu, a rival who becomes Gilgamesh’s friend. They embark on several heroic exploits and end up pissing off the gods. As a result, they condemn Enkidu to death. Despondent and fearing for his own death, Gilgamesh goes in search of the secret of immortality. His travels take him to Utnapishtim, immortal survivor of the great flood. Our hero finds a plant that restores youth, but loses it. By the end of the story, he accepts his fate as a mortal. The story moves fast and is surprisingly engaging. It includes early versions of ideas that would resurface later in the Bible. (Most obviously, Noah and the flood.) There’s also some material that probably wouldn’t pass muster in our prudish time. The Dhammapada is one of the central Buddhist scriptures. I was familiar with several of these texts but hadn’t read the whole thing. Gioa notes that he recommended it because of its length, but there are also obvious connections with Gilgamesh. For example, several verses in the Dhammapada deal with attachment. For example, here’s verse 215: From affection comes grief; Gilgamesh suffers from such an attachment. Here’s the moment of Enkidu’s death: He touched his heart but it did not beat, nor did he lift his eyes again. When Gilgamesh touched his heart it did not beat. So Gilgamesh laid a veil, as one veils the bride, over his friend. He began to rage like a lion, like a lioness robbed of her whelps. This way and that he paced round the bed, he tore out his hair and strewed it around. He dragged off his splendid robes and flung them down as though they were abominations. He wishes to hold on: Then Gilgamesh issued a proclamation through the land, he summoned them all, the coppersmiths, the goldsmiths, the stone-workers, and commanded them, ‘Make a statue of my friend.’ The statue was fashioned with a great weight of lapis lazuli for the breast and of gold for the body. A table of hard-wood was set out, and on it a bowl of carnelian filled with honey, and a bowl of lapis lazuli filled with butter. These he exposed and offered to the Sun; and weeping he went away. You can probably relate if you’ve ever lost someone dear. Human nature. Audiovisual Gioia recommended Stravinsky’s Rite of Spring and Wagner’s Overtures. I’m very familiar with both so I didn’t spend much time with either this week. For a new take on one of these familiar classics, check out Fazil Say’s astonishing piano version of the Rite of Spring. Here’s a short portion: Gioia also recommended looking at ancient Mesopotamian art. I didn’t spend as much time on this as I would’ve liked. That said, this introductory lecture provided context while highlighting major works of art and architecture: I took a different approach to cinema this week. Rather than go by an AI recommendation, I went down the old-fashioned route. (I.e., Google.) Specifically, I thought this would be a good opportunity to check out Iranian cinema. I’ve heard good things about Iranian films, but had never seen one. Googling led me to this article on Vulture. After reading through the list, I picked Abbas Kiarostami’s TASTE OF CHERRY. Yet again, I’ve gravitated towards a film about a middle-aged man in despair. (Is the Universe trying to tell me something?) Kiarostami effectively uses a minimalist style to explore what makes life meaningful despite (or perhaps because) of its finitude. Reflections There’s a pattern here. This week’s works dealt with core issues people have grappled with since we became people. The big one: how do we deal with death? Not just the impending death of everyone we love, but our own. Gilgamesh offers the traditional “Western” answer: “I can’t even.” So, fight it! He looks for a MacGuffin that’ll let him go on living and perhaps brings his loved ones back. It’s an idea that has had many progeny in our mythologies. And it’s not just the stuff of fiction: the impulse is still alive and well. (Pardon the pun.) The Buddha offers a different approach: non-attachment. It’ll be easier to let go if you don’t become enmeshed with things, people, and your own sense of being. So you train your mind so it won’t hang on. (Even the idea of “your mind” is suspect.) The price: not feeling either extreme. No despair, no elation. Kiarostami’s film suggests a third approach: accepting the inevitability of death while reveling in the experience of being alive. (You could argue this is part of the Buddhist way as well.) I won’t say more in case you haven’t seen TASTE OF CHERRY, but suffice it to say the film employs a clever structural trick to wake you from your slumber. Grappling with these kinds of issues is the point of studying the humanities. Yes, I know you’re busy. I’m busy too. But some day, the busyness will stop – as will everything else. I’m committed to living an examined life, and that requires thinking about stuff we’d rather put aside so we can get on with the next Zoom meeting. Notes on Note-taking I’m also committed to the other point of this humanities project: learning how to learn better in this AI age. This week, I continued tweaking my note-taking approach. I took notes in the Drafts app as I read, building an outline as I go through the week. I wrote down the main points I learned and things I’d like to share with you. I then elaborated this outline on one of my morning walks. My mind works better when my body is moving and clear from the day’s detritus. I also tweaked my note taking approach around the readings. I had an LLM summarize the reading and then used that as a refresher to write a summary in my own words. I’ve done the same in previous weeks. What’s different now is that I then pasted my summary into a ChatGPT window with a simple prompt: I read The Epic of Gilgamesh. What is wrong with this description of the story?: Gilgamesh is king of Uruk. He’s described as the strongest and most beautiful man in the world. He’s also something of a despot. He befriends Enkidu, a wild man who is almost as strong as Gilgamesh. They go on several adventures, which entail opposing the wishes of one of the Mesopotamian deities. Eventually, the gods are angered and decree Enkidu must die. Grief-stricken, Gilgamesh goes in search for the secret of eternal life, only to learn that human lives are limited. He returns to lead his people with this newfound wisdom. The LLM offered a helpful response that clarified nuances I’d missed: Gilgamesh wasn’t “something of a despot”; he was a tyrant. The gods created Enkidu as a counterbalance to answer his subjects’ calls for relief. The taming and channeling of this force of nature through initiation into human pleasures is an important aspect of the story I’d left out. Details about Gilgamesh and Enkidu’s transgressions against the gods. (These seemed less relevant for a high-level summary.) The fact Gilgamesh isn’t just searching for immortality because he’s grief-stricken over Enkidu’s death; he’s also fearing for his own life. The end of my summary was wrong; the book doesn’t suggest Gilgamesh changed as a result of his experiences. This last point is important. In writing my summary, I made stuff up that wasn’t in the book. I attribute my error to the fact I expect closure from my stories. Gilgamesh precedes Aristotle’s Poetics; its authors were under no compulsion to offer the hero a redemption arc. Which is to say, humans hallucinate too – and LLMs can correct us. Up Next Next week, we’re reading ancient Egyptian literature. I couldn’t find an ebook of the text suggested by Gioia, so I’m going with another Penguin book, Writings from Ancient Egypt. I studied some Egyptian architecture in college, and look forward to revisiting this part of the world and its history. Check out Gioia’s post for the full syllabus. I’ve started a YouTube playlist to bookmark all the videos I’m sharing in this course. And as a reminder, I’m also sharing these posts via Substack if you’d like to subscribe and comment.
99% of the time I want my software to be as fast as humanly possible. I want UI elements that respond quickly, and I want keyboard shortcuts to do as much as possible. But 1% of the time, for whatever reason, I just like when a computer takes a long
Last month I completed my first year at EnterpriseDB. I'm on the team that built and maintains pglogical and who, over the years, contributed a good chunk of the logical replication functionality that exists in community Postgres. Most of my work, our work, is in C and Rust with tests in Perl and Python. Our focus these days is a descendant of pglogical called Postgres Distributed which supports replicating DDL, tunable consistency across the cluster, etc. This post is about how I got here. Black boxes I was a web developer from 2014-2021†. I wrote JavaScript and HTML and CSS and whatever server-side language: Python or Go or PHP. I was a hands-on engineering manager from 2017-2021. I was pretty clueless about databases and indeed database knowledge was not a serious part of any interview I did. Throughout that time (2014-2021) I wanted to move my career forward as quickly as possible so I spent much of my free time doing educational projects and writing about them on this blog (or previous incarnations of it). I learned how to write primitive HTTP servers, how to write little parsers and interpreters and compilers. It was a virtuous cycle because the internet (Hacker News anyway) liked reading these posts and I wanted to learn how the black boxes worked. But I shied away from data structures and algorithms (DSA) because they seemed complicated and useless to the work that I did. That is, until 2020 when an inbox page I built started loading more and more slowly as the inbox grew. My coworker pointed me at Use The Index, Luke and the DSA scales fell from my eyes. I wanted to understand this new black box so I built a little in-memory SQL database with support for indexes. I'm a college dropout so even while I was interested in compilers and interpreters earlier in my career I never dreamed I could get a job working on them. Only geniuses and PhDs did that work and I was neither. The idea of working on a database felt the same. However, I could work on little database side projects like I had done before on other topics, so I did. Or a series of explorations of Raft implementations, others' and my own. Startups From 2021-2023 I tried to start a company and when that didn't pan out I joined TigerBeetle as a cofounder to work on marketing and community. It was during this time I started the Software Internals Discord and /r/databasedevelopment which have since kind of exploded in popularity among professionals and academics in database and distributed systems. TigerBeetle was my first job at a database company, and while I contributed bits of code I was not a developer there. It was a way into the space. And indeed it was an incredible learning experience both on the cofounder side and on the database side. I wrote articles with King and Joran that helped teach and affirm for myself the basics of databases and consensus-based distributed systems. Holding out When I left TigerBeetle in 2023 I was still not sure if I could get a job as an actual database developer. My network had exploded since 2021 (when I started my own company that didn't pan out) so I had no trouble getting referrals at database companies. But my background kept leading hiring managers to suggest putting me on cloud teams doing orchestration in Go around a database rather than working on the database itself. I was unhappy with this type-casting so I held out while unemployed and continued to write posts and host virtual hackweeks messing with Postgres and MySQL. I started the first incarnation of the Software Internals Book Club during this time, reading Designing Data Intensive Applications with 5-10 other developers in Bryant Park. During this time I also started the NYC Systems Coffee Club. Postgres After about four months of searching I ended up with three good offers, all to do C and Rust development on Postgres (extensions) as an individual contributor. Working on extensions might sound like the definition of not-sexy, but Postgres APIs are so loosely abstracted it's really as if you're working on Postgres itself. You can mess with almost anything in Postgres so you have to be very aware of what you're doing. And when you can't mess with something in Postgres because an API doesn't yet exist, companies have the tendency to just fork Postgres so they can. (This tendency isn't specific to Postgres, almost every open-source database company seems to have a long-running internal fork or two of the database.) EnterpriseDB Two of the three offers were from early-stage startups and after more than 3 years being part of the earliest stages of startups I was happy for a break. But the third offer was from one of the biggest contributors to Postgres, a 20-year old company called EnterpriseDB. (You can probably come up with different rankings of companies using different metrics so I'm only saying EnterpriseDB is one of the biggest contributors.) It seemed like the best place to be to learn a lot and contribute something meaningful. My coworkers are a mix of Postgres veterans (people who contributed the WAL to Postgres, who contributed MVCC to Postgres, who contributed logical decoding and logical replication, who contributed parallel queries; the list goes on and on) but also my developer-coworkers are people who started at EnterpriseDB on technical support, or who were previously Postgres administrators. It's quite a mix. Relatively few geniuses or PhDs, despite what I used to think, but they certainly work hard and have hard-earned experience. Anyway, I've now been working at EnterpriseDB for over a year so I wanted to share this retrospective. I also wanted to cover what it's like coming from engineering management and founding companies to going back to being an individual contributor. (Spoiler: incredibly enjoyable.) But it has been hard enough to make myself write this much so I'm calling it a day. :) I wrote a post about the winding path I took from web developer to database developer over 10 years. pic.twitter.com/tf8bUDRzjV — Phil Eaton (@eatonphil) February 15, 2025 † From 2011-2014 I also did contract web development but this was part-time while I was in school.