Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from Notes on software development

From web developer to database developer in 10 years

Last month I completed my first year at EnterpriseDB. I'm on the team that built and maintains pglogical and who, over the years, contributed a good chunk of the logical replication functionality that exists in community Postgres. Most of my work, our work, is in C and Rust with tests in Perl and Python. Our focus these days is a descendant of pglogical called Postgres Distributed which supports replicating DDL, tunable consistency across the cluster, etc. This post is about how I got here. Black boxes I was a web developer from 2014-2021†. I wrote JavaScript and HTML and CSS and whatever server-side language: Python or Go or PHP. I was a hands-on engineering manager from 2017-2021. I was pretty clueless about databases and indeed database knowledge was not a serious part of any interview I did. Throughout that time (2014-2021) I wanted to move my career forward as quickly as possible so I spent much of my free time doing educational projects and writing about them on this blog (or previous incarnations of it). I learned how to write primitive HTTP servers, how to write little parsers and interpreters and compilers. It was a virtuous cycle because the internet (Hacker News anyway) liked reading these posts and I wanted to learn how the black boxes worked. But I shied away from data structures and algorithms (DSA) because they seemed complicated and useless to the work that I did. That is, until 2020 when an inbox page I built started loading more and more slowly as the inbox grew. My coworker pointed me at Use The Index, Luke and the DSA scales fell from my eyes. I wanted to understand this new black box so I built a little in-memory SQL database with support for indexes. I'm a college dropout so even while I was interested in compilers and interpreters earlier in my career I never dreamed I could get a job working on them. Only geniuses and PhDs did that work and I was neither. The idea of working on a database felt the same. However, I could work on little database side projects like I had done before on other topics, so I did. Or a series of explorations of Raft implementations, others' and my own. Startups From 2021-2023 I tried to start a company and when that didn't pan out I joined TigerBeetle as a cofounder to work on marketing and community. It was during this time I started the Software Internals Discord and /r/databasedevelopment which have since kind of exploded in popularity among professionals and academics in database and distributed systems. TigerBeetle was my first job at a database company, and while I contributed bits of code I was not a developer there. It was a way into the space. And indeed it was an incredible learning experience both on the cofounder side and on the database side. I wrote articles with King and Joran that helped teach and affirm for myself the basics of databases and consensus-based distributed systems. Holding out When I left TigerBeetle in 2023 I was still not sure if I could get a job as an actual database developer. My network had exploded since 2021 (when I started my own company that didn't pan out) so I had no trouble getting referrals at database companies. But my background kept leading hiring managers to suggest putting me on cloud teams doing orchestration in Go around a database rather than working on the database itself. I was unhappy with this type-casting so I held out while unemployed and continued to write posts and host virtual hackweeks messing with Postgres and MySQL. I started the first incarnation of the Software Internals Book Club during this time, reading Designing Data Intensive Applications with 5-10 other developers in Bryant Park. During this time I also started the NYC Systems Coffee Club. Postgres After about four months of searching I ended up with three good offers, all to do C and Rust development on Postgres (extensions) as an individual contributor. Working on extensions might sound like the definition of not-sexy, but Postgres APIs are so loosely abstracted it's really as if you're working on Postgres itself. You can mess with almost anything in Postgres so you have to be very aware of what you're doing. And when you can't mess with something in Postgres because an API doesn't yet exist, companies have the tendency to just fork Postgres so they can. (This tendency isn't specific to Postgres, almost every open-source database company seems to have a long-running internal fork or two of the database.) EnterpriseDB Two of the three offers were from early-stage startups and after more than 3 years being part of the earliest stages of startups I was happy for a break. But the third offer was from one of the biggest contributors to Postgres, a 20-year old company called EnterpriseDB. (You can probably come up with different rankings of companies using different metrics so I'm only saying EnterpriseDB is one of the biggest contributors.) It seemed like the best place to be to learn a lot and contribute something meaningful. My coworkers are a mix of Postgres veterans (people who contributed the WAL to Postgres, who contributed MVCC to Postgres, who contributed logical decoding and logical replication, who contributed parallel queries; the list goes on and on) but also my developer-coworkers are people who started at EnterpriseDB on technical support, or who were previously Postgres administrators. It's quite a mix. Relatively few geniuses or PhDs, despite what I used to think, but they certainly work hard and have hard-earned experience. Anyway, I've now been working at EnterpriseDB for over a year so I wanted to share this retrospective. I also wanted to cover what it's like coming from engineering management and founding companies to going back to being an individual contributor. (Spoiler: incredibly enjoyable.) But it has been hard enough to make myself write this much so I'm calling it a day. :) I wrote a post about the winding path I took from web developer to database developer over 10 years. pic.twitter.com/tf8bUDRzjV — Phil Eaton (@eatonphil) February 15, 2025 † From 2011-2014 I also did contract web development but this was part-time while I was in school.

2 weeks ago 12 votes
Edit for clarity

I have the fortune to review a few important blog posts every year and the biggest value I add is to call out sentences or sections that make no sense. It is quite simple and you can do it too. Without clarity only those at your company in marketing and sales (whose job it is to work with what they get) will give you the courtesy of a cursory read and a like on LinkedIn. This is all that most corporate writing achieves. It is the norm and it is understandable. But if you want to reach an audience beyond those folks, you have to make sure you're not writing nonsense. And you, as reviewer and editor, have the chance to call out nonsense if you can get yourself to recognize it. Immune to nonsense But especially when editing blog posts at work, it is easy to gloss over things that make no sense because we are so constantly bombarded by things that make no sense. Maybe it's buzzwords or cliches, or simply lack of rapport. We become immune to nonsense. And even worse, without care, as we become more experienced, we become more fearful to say "I have no idea what you are talking about". We're afraid to look incompetent by admitting our confusion. This fear is understandable, but is itself stupid. And I will trust you to deal with this on your own. Read it out loud So as you review a post, read it out loud to yourself. And if you find yourself saying "what on earth are you talking about", add that as a comment as gently as you feel you should. It is not offensive to say this (depending on how you say it). It is surely the case that the author did not know they were making no sense. It is worse to not mention your confusion and allow the author to look like an idiot or a bore. Once you can call out what does not make sense to you, then read the post again and consider what would not make sense to someone without the context you have. Someone outside your company. Of course you need to make assumptions about the audience to a degree. It is likely your customers or prospects you have in mind. Not your friends or family. With the audience you have in mind, would what you're reading make any sense? Has the author given sufficient background or introduced relevant concepts before bringing up something new? Again this is a second step though. The first step is to make sure that the post makes sense to you. In almost every draft I read, at my company or not, there is something that does not make sense to me. Do two paragraphs need to be reordered because the first one accidentally depended on information mentioned in the second? Are you making ambiguous use of pronouns? And so on. In closing Clarity on its own will put you in the 99th percentile of writing. Beyond that it definitely still matters if you are compelling and original and whatnot. But too often it seems we focus on being exciting rather than being clear. But it doesn't matter if you've got something exciting if it makes no sense to your reader. This sounds like mundane guidance, but I have reviewed many posts that were reviewed by other people and no one else called out nonsense. I feel compelled to mention how important it is. Wrote a new post on the most important, and perhaps least done, thing you can do while reviewing a blog post: edit for clarity. pic.twitter.com/ODblOUzB3g — Phil Eaton (@eatonphil) January 29, 2025

a month ago 16 votes
An explosion of transitive dependencies

A small standard library means an explosion in transitive dependencies. A more comprehensive standard library helps you minimize dependencies. Don't misunderstand me: in a real-world project, it is practically impossible to have zero dependencies. Armin Ronacher called for a vibe shift among programmers and I think that this actually exists already. Everyone I speak to on this topic has agreed that minimizing dependencies is ideal. Rust and JavaScript, with their incredibly minimal standard libraries, work against this ideal. Go, Python, Java, and C# in contrast have a decent standard library, which helps minimize the explosion of transitive dependencies. Examples I think the standard library should reasonably include: JSON, CSV, and Parquet support HTTP/2 support (which includes TLS, compression, random number generation, etc.) Support for asynchronous IO A logging abstraction A SQL client abstraction Key abstract data types (BTrees, hashmaps, sets, and growable arrays) Utilities for working with Unicode, time and timezones But I don't think it needs to include: Excel support PostgreSQL or Oracle clients Flatbuffers support Niche data structures Neither of these are intended to be complete lists, just examples. Walled gardens Minimal standard libraries force growing companies to build out their own internal collection of "standard libraries". As one example, Bloomberg did this with C++. And I've heard of companies doing this already with Rust. This allows larger companies to manage and minimize the explosion of transitive dependencies over time. All growing companies likely do something like this eventually. But again, smaller standard libraries incentivize companies to build this internal standard library earlier on. And the community benefits relatively little from these internal standard libraries. The community would benefit more if large organizations contributed back to an actual standard library. Smaller organizations do not have the capacity to build these internal standard libraries. Maybe the situation will lead to libraries like Boost for JavaScript and Rust programmers. That could be fine. Versioning A comprehensive standard library does not prevent the language developers from releasing new versions of the standard library. It is trivial to do this with naming like Go has done with the v2 pattern. math/rand/v2 is an example. Conclusion My concern about the standard library does not stop me from using Rust and JavaScript. Furthermore, they could choose to invest in the standard library at any time. We have already begun to see Bun and Deno to do exactly this. But it is clearly an area for improvement in Rust and JavaScript. And a mistake for other languages to avoid repeating. While zero dependencies is practically impossible, everyone I've spoken to agrees that minimizing dependencies is ideal. Rust and JavaScript work against this ideal. But they could change at any time. And Bun and Deno are already examples of this.https://t.co/qkSh6oW1Yd pic.twitter.com/mY1MNErZG7 — Phil Eaton (@eatonphil) January 25, 2025

a month ago 16 votes
Embedding Python in Rust (for tests)

This is an external post of mine. Click here if you are not redirected.

a month ago 31 votes

More in technology

A lightweight file server running entirely on an Arduino Nano ESP32

Home file servers can be very useful for people who work across multiple devices and want easy access to their documents. And there are a lot of DIY build guides out there. But most of them are full-fledged NAS (network-attached storage) devices and they tend to rely on single-board computers. Those take a long time […] The post A lightweight file server running entirely on an Arduino Nano ESP32 appeared first on Arduino Blog.

4 hours ago 2 votes
The battle of the budget phones

Just days after I got my iPhone 16e, Apple’s (less) budget (than ever before) iPhone, Nothing is out here with new their new budget phones, the Phone (3a) and (3a) Pro. These models start at $379 and $459 respectively, so they certainly undercut the new iPhone, so let&

4 hours ago 1 votes
This vending machine draws generative art for just a euro

If you hear the term “generative art” today, you probably subconsciously add “AI” to the beginning without even thinking about it. But generative art techniques existed long before modern AI came along — they even predate digital computing altogether. Despite that long history, generative art remains interesting as consumers attempt to identify patterns in the […] The post This vending machine draws generative art for just a euro appeared first on Arduino Blog.

21 hours ago 1 votes
Apple’s new iPads are here, let’s break them down

Another day, another opportunity to rate my 2025 Apple predictions! iPad Here’s what I predicted would happen with the base iPad this year: I fully expect to see the 11th gen iPad in 2025, and I think it will come with a jump to the A17 Pro or

42 minutes ago 1 votes
The New Leverage: AI and the Power of Small Teams

This weekend, a small team in Latvia won an Oscar for a film they made using free software. That’s not just cool — it’s a sign of what’s coming. Sunday night was family movie night in my home. We picked a recent movie, FLOW. I’d heard good things about it and thought we’d enjoy it. What we didn’t know was that as we watched, the film won this year’s Academy Award as best animated feature. Afterwards, I saw this post from the movie’s director, Gints Zilbalodis: We established Dream Well Studio in Latvia for Flow. This room is the whole studio. Usually about 4-5 people were working at the same time including me. I was quite anxious about being in charge of a team, never having worked in any other studios before, but it worked out. pic.twitter.com/g39D6YxVWa — Gints Zilbalodis (@gintszilbalodis) January 26, 2025 Let that sink in: 4-5 people in a small room in Latvia led by a relatively inexperienced director used free software to make a movie that as of February 2025 had earned $20m and won an Oscar. I know it’s a bit more involved than that, but still – quite an accomplishment! But not unique. Billie Eilish and her brother Phineas produced her Grammy-winning debut album When We All Fall Asleep, Where Do We Go? in their home studio. And it’s not just cultural works such as movies and albums: small teams have built hugely successful products such as WhatsApp and Instagram. As computers and software get cheaper and more powerful, people can do more with less. And “more” here doesn’t mean just a bit better (pardon the pun) – it means among the world’s best. And as services and products continue migrating from the world of atoms to the world of bits, creators’ scope of action grows. This trend isn’t new. But with AI in the mix, things are about to go into overdrive. Zilbalodis and his collaborators could produce their film because someone else built Blender; they worked within its capabilities and constraints. But what if their vision exceeded what the software can do? Just a few years ago, the question likely wouldn’t even come up. Developing software calls for different abilities. Until recently, a small team had to choose: either make the movie or build the tools. AI changes that, since it enables small teams to “hire” virtual software developers. Of course, this principle extends beyond movies: anything that can be represented symbolically is in scope. And it’s not just creative abilities, such as writing, playing music, or drawing, but also more other business functions such as scheduling, legal consultations, financial transactions, etc. We’re not there yet. But if trends hold, we’ll soon see agent-driven systems do for other kinds of businesses what Blender did for Dream Well Studio. Have you dreamed of making a niche digital product to scratch an itch? That’s possible now. Soon, you’ll be able to build a business around it quickly, easily, and without needing lots of other humans in the mix. Many people have lost their jobs over the last three years. Those jobs likely won’t be replaced with AIs soon. But job markets aren’t on track to stability. If anything, they’re getting weirder. While it’s early days, AI promises some degree of resiliency. For people with entrepreneurial drive, it’s an exciting time: we can take ideas from vision to execution faster, cheaper, and at greater scale than ever. For others, it’ll be unsettling – or outright scary. We’re about to see a major shift in who can create, innovate, and compete in the market. The next big thing might not come from a giant company, but from a small team – or even an individual – using AI-powered tools. I expect an entrepreneurial surge driven by necessity and opportunity. How will you adapt?

9 hours ago 1 votes