Full Width [alt+shift+f] FOCUS MODE Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
44
There are a few interesting scenarios to keep in mind when writing applications (not just databases!) that interact with read and writes files, particularly in transactional contexts where you actually care about the integrity of the data and when you are editing data in place (versus copy-on-write for example). If I don't say otherwise I'm talking about behavior on Linux. The research version of this blog post is Parity Lost and Parity Regained and Characteristics, Impact, and Tolerance of Partial Disk Failures. These two papers also go into the frequency of some of the issues discussed here. These behaviors actually happen in real life! Thank you to Alex Miller and George Xanthakis for reviewing a draft of this post. Terminology Some of these terms are reused in different contexts, and sometimes they are reused because they effectively mean the same thing in a certain configuration. But I'll try to be explicit to avoid confusion. Sector The smallest amount of data that can be read...
5 months ago

Comments

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from Notes on software development

Burn your title

I've been a developer, a manager, a cofounder, and now I'm a developer again. I ran away from each position until being a founder because I felt like I was limited by what I was allowed to do. But I reached an enlightment of sorts during my career progression: that everyone around me was dying for someone to pick things up, for employees to show engagement and agency. We think of our titles as our limits. We're quick to say and believe, "that isn't my job". While in reality titles reflect the minimum expected of us, not the maximum that is open to us. Moving your career forward Trying to figure out what (new minimum) you must do to get promoted seems kind of backwards to me, reinforcing our sense of our own limits. Instead, at every stage in your career, focus on doing the intersection of: what you see needs to be done (that isn't being done) what you are capable of doing what you have the desire/energy (or would find fulfillment) doing And this is the path to promotion and a successful and interesting career. Burn your title. Burn your job description. I mean, keep your boss happy for sure. Keep your teammates happy by supporting them and building them up and communicating well. But don't wait to be officially made a lead or given a new title to do what otherwise fits into that intersection above. And if after doing this for some time, demonstrating this level of agency, you are not promoted, it just means you're not at the right company or right organization within your company and you should look elsewhere. What's more this work you did (at a company that doesn't appreciate your agency, if that happens to be the case) merely makes the case stronger for your successful interview at the next company. There's no downside. The cynical, and perhaps realistic, alternative to this is to do politics to get promoted. Or to do not do politics but to do things that don't align with your long-term goals. I'm not personally interested in either path so I'm not covering them here. I'm interested in the intersection of things that move me in the direction I want, things that are useful to the company, and things that I am capable of doing (in addition to whatever minimum work I must actually do). Examples Here's a peek at what this looks like for me as an individual contributor, a programmer, at EnterpriseDB. I started the EDB Engineering Newsletter because it seemed like we needed to do a better job telling the world the awesome things our engineering team is doing. (You know we're one of the biggest contributors to Postgres? Bruce Momjian, Robert Haas, Peter Eisentraut, etc. work here? The guy who implemented the WAL and MVCC in Postgres is my teammate?) Nobody asked me to do that. I started publishing blog views for the entire company once a month internally. Nobody asked me to do that. I wrote a number of internal docs and tutorials on the product because we were just obviously missing them. Nobody asked me to do that. I started a fortnightly incident review meeting for my team because it seemed like we were missing chances to update docs and teach each other. Nobody asked me to do that. I write a bunch of random posts for the company blog on what I've learned. Nobody asked me to do that. These are just a few of the random things that seemed like a good idea for me to do on top of my Actual Work as a developer, which I think I do a decent job of on its own. In closing Don't burn out. Don't do things you aren't asked for and don't find rewarding. Or that won't pave the way toward the career you want. I'm trying to be very careful not to advocate anything along those lines. But also don't wait to be asked to do something. Do what is interesting and obvious and rewarding to you. Interesting opportunities seem to come most reliably when you make them for yourself. Burn your title pic.twitter.com/4bQRPMX4EZ — Phil Eaton (@eatonphil) April 22, 2025

4 months ago 42 votes
Transactions are a protocol

Transactions are not an intrinsic part of a storage system. Any storage system can be made transactional: Redis, S3, the filesystem, etc. Delta Lake and Orleans demonstrated techniques to make S3 (or cloud storage in general) transactional. Epoxy demonstrated techniques to make Redis (and any other system) transactional. And of course there's always good old Two-Phase Commit. If you don't want to read those papers, I wrote about a simplified implementation of Delta Lake and also wrote about a simplified MVCC implementation over a generic key-value storage layer. It is both the beauty and the burden of transactions that they are not intrinsic to a storage system. Postgres and MySQL and SQLite have transactions. But you don't need to use them. It isn't possible to require you to use transactions. Many developers, myself a few years ago included, do not know why you should use them. (Hint: read Designing Data Intensive Applications.) And you can take it even further by ignoring the transaction layer of an existing transactional database and implement your own transaction layer as Convex has done (the Epoxy paper above also does this). It isn't entirely clear that you have a lot to lose by implementing your own transaction layer since the indexes you'd want on the version field of a value would only be as expensive or slow as any other secondary index in a transactional database. Though why you'd do this isn't entirely clear (I will like to read about this from Convex some time). It's useful to see transaction protocols as another tool in your system design tool chest when you care about consistency, atomicity, and isolation. Especially as you build systems that span data systems. Maybe, as Ben Hindman hinted at the last NYC Systems, even proprietary APIs will eventually provide something like two-phase commit so physical systems outside our control can become transactional too. Transactions are a protocol short new post pic.twitter.com/nTj5LZUpUr — Phil Eaton (@eatonphil) April 20, 2025

4 months ago 41 votes
Phil Eaton on Technical Blogging

This is an external post of mine. Click here if you are not redirected.

5 months ago 64 votes
Minimal downtime Postgres major version upgrades with EDB Postgres Distributed

This is an external post of mine. Click here if you are not redirected.

6 months ago 45 votes

More in technology

The unreasonable effectiveness of the pancake rule

Being chronically late to meetings sucks. Not only is it very rude, but you’re signalling that you don’t value your coworkers’ time. However, I’ve picked up a technique that works unreasonably well within a team.1 If you are late to the first meeting of the day three times within a quarter, then you will have to make pancakes for the whole team. Let’s say that you have a daily stand-up taking place at 10:00. Arriving at 10:00:59: completely OK. Arriving at 10:01:00: You’re one step closer to making pancakes! Keep in mind that you may hit some obstacles when implementing this rule, so feel free to adjust it. When proposing this idea in my current team, I learned that the office does not offer pancake-making facilities. The pancakes can be substituted for other types of cake or bringing in something else, as long as the team gives prior approval of that modification. The pancake strikes can also be pooled together and spent with your teammates if they wish to do so. If you’re struggling with your team being late to your daily meeting(s), then go ahead and add this rule to the working agreement. You do have a working agreement set up, right? Right? And a free security tech tip to close out: if you see an unlocked work laptop at the office, open your internal chat application of choice on it and try posting to a public channel that you’ll be bringing cake/beers/candy to the office. Works wonders for enforcing the habit of locking your laptop up when leaving the desk! to be fair, the sample size is two, but it has worked out really well in both! ↩︎

11 hours ago 6 votes
Turtle bots, Gestalt principles, and emergent art

In the worlds of programming and robotics, turtles are entities — either virtual or physical robots— that follow commands to move around a 2D plane. Those are usually very simple commands, such as “move forward 10 units” or “rotate 90 degrees clockwise,” and they help people learn some programming fundamentals (like Logo in the ’80s!) […] The post Turtle bots, Gestalt principles, and emergent art appeared first on Arduino Blog.

3 days ago 10 votes
Microsoft makes 6502 BASIC open source

It was probably going to happen sooner or later, but Microsoft has officially released the source code for 6502 BASIC. The specific revision is very Commodore-centric: it's the 1977 "8K" BASIC variant "1.1," which Commodore users know better as BASIC V2.0, the same BASIC used in the early PET and with later spot changes from Commodore (including removing Bill Gates' famous Easter egg) in the VIC-20 and Commodore 64. I put "8K" in quotes because the 40-bit Microsoft Binary Format version, which is most familiar as the native floating point format for most 8-bit BASICs derived from Microsoft's and all Commodore BASICs from the PET on up, actually starts at 9K in size. In the C64, because there is RAM and I/O between the BASIC ROM and the Kernal ROM, there is an extra JMP at the end of the BASIC ROM to continue to the routine in the lowest portions of the Kernal ROM. The jump doesn't exist in the VIC-20 where the ROM is contiguous and as a result everything past that point is shifted by three bytes on the C64, the length of the instruction. This is, of course, the same BASIC that Gates wanted a percentage of but Jack Tramiel famously refused to budge on the $25,000 one-time fee, claiming "I'm already married." Gates yielded to Tramiel, as most people did then, but I suspect the slight was never forgotten. Not until the 128 did Microsoft officially appear in the credits for Commodore BASIC, and then likely only as a way to push its bona fides as a low-end business computer. Microsoft's source release also includes changes from Commodore's own John Feagans, who rewrote the garbage collection routine, and was the original developer of the Commodore Kernal and later Magic Desk. The source code is all in one big file (typical for the time) and supports six machine models, the first most likely a vapourware 6502 system never finished by Canadian company Semi-Tech Microelectronics (STM) better known for the CP/M-based Pied Piper, then the Apple II, the Commodore (in this case PET 2001), the Ohio Scientific (OSI) Challenger, the Commodore/MOS KIM-1, and most intriguingly a PDP-10-based simulator written by Paul Allen. The source code, in fact, was cross-assembled on a PDP-10 using MACRO-10, and when assembled for the PDP-10 emulator it actually emits a PDP-10 executable that traps on every instruction into the simulator linked with it — an interesting way of effectively accomplishing threaded code. A similar setup was used for their 8080 emulator. Unfortunately, I don't believe Allen's code has been released anywhere, though I'd love to be proven wrong if people know otherwise. Note that they presently don't even mention the STM port in the Github README, possibly because no one was sure what it did. While MACRO-10 source for 6502 BASIC has circulated before and been analysed in detail, most notably by Michael Steil, this is nevertheless the first official release where it is truly open-source under the MIT license and Microsoft should be commended for doing so. This also makes it much easier to pull a BASIC up for your own 6502 homebrew system — there's nothing like the original.

5 days ago 15 votes
Byte Interviews IBM's Philip D. Estridge (1983)

They dig into the IBM PC.

a week ago 14 votes
The QNX Operating System

Quantum Software and the microkernel UNIX

a week ago 16 votes