Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Notes on software development
Transactions are not an intrinsic part of a storage system. Any storage system can be made transactional: Redis, S3, the filesystem, etc. Delta Lake and Orleans demonstrated techniques to make S3 (or cloud storage in general) transactional. Epoxy demonstrated techniques to make Redis (and any other system) transactional. And of course there's always good old Two-Phase Commit. If you don't want to read those papers, I wrote about a simplified implementation of Delta Lake and also wrote about a simplified MVCC implementation over a generic key-value storage layer. It is both the beauty and the burden of transactions that they are not intrinsic to a storage system. Postgres and MySQL and SQLite have transactions. But you don't need to use them. It isn't possible to require you to use transactions. Many developers, myself a few years ago included, do not know why you should use them. (Hint: read Designing Data Intensive Applications.) And you can take it even further by ignoring the transaction layer of an existing transactional database and implement your own transaction layer as Convex has done (the Epoxy paper above also does this). It isn't entirely clear that you have a lot to lose by implementing your own transaction layer since the indexes you'd want on the version field of a value would only be as expensive or slow as any other secondary index in a transactional database. Though why you'd do this isn't entirely clear (I will like to read about this from Convex some time). It's useful to see transaction protocols as another tool in your system design tool chest when you care about consistency, atomicity, and isolation. Especially as you build systems that span data systems. Maybe, as Ben Hindman hinted at the last NYC Systems, even proprietary APIs will eventually provide something like two-phase commit so physical systems outside our control can become transactional too. Transactions are a protocol short new post pic.twitter.com/nTj5LZUpUr — Phil Eaton (@eatonphil) April 20, 2025
There are a few interesting scenarios to keep in mind when writing applications (not just databases!) that interact with read and writes files, particularly in transactional contexts where you actually care about the integrity of the data and when you are editing data in place (versus copy-on-write for example). If I don't say otherwise I'm talking about behavior on Linux. The research version of this blog post is Parity Lost and Parity Regained and Characteristics, Impact, and Tolerance of Partial Disk Failures. These two papers also go into the frequency of some of the issues discussed here. These behaviors actually happen in real life! Thank you to Alex Miller and George Xanthakis for reviewing a draft of this post. Terminology Some of these terms are reused in different contexts, and sometimes they are reused because they effectively mean the same thing in a certain configuration. But I'll try to be explicit to avoid confusion. Sector The smallest amount of data that can be read and written atomically by hardware. It used to be 512 bytes, but on modern disks it is often 4KiB. There doesn't seem to be any safe assumption you can make about sector size, despite file system defaults (see below). You must check your disks to know. Block (filesystem/kernel view) Typically set to the sector size since only this block size is atomic. The default in ext4 is 4KiB. Page (kernel view) A disk block that is in memory. Any reads/writes less than the size of a block will read the entire block into kernel memory even if less than that amount is sent back to userland. Page (database/application view) The smallest amount of data the system (database, application, etc.) chooses to act on, when it's read or written or held in memory. The page size is some multiple of the filesystem/kernel block size (including the multiple being 1). SQLite's default page size is 4KiB. MySQL's default page size is 16KiB. Postgres's default page size is 8KiB. Things that go wrong The data didn't reach disk By default, file writes succeed when the data is copied into kernel memory (buffered IO). The man page for write(2) says: A successful return from write() does not make any guarantee that data has been committed to disk. On some filesystems, including NFS, it does not even guarantee that space has successfully been reserved for the data. In this case, some errors might be delayed until a future write(), fsync(2), or even close(2). The only way to be sure is to call fsync(2) after you are done writing all your data. If you don't call fsync on Linux the data isn't necessarily durably on disk, and if the system crashes or restarts before the disk writes the data to non-volatile storage, you may lose data. With O_DIRECT, file writes succeed when the data is copied to at least the disk cache. Alternatively you could open the file with O_DIRECT|O_SYNC (or O_DIRECT|O_DSYNC) and forgo fsync calls. fsync on macOS is a no-op. If you're confused, read Userland Disk I/O. Postgres, SQLite, MongoDB, MySQL fsync data before considering a transaction successful by default. RocksDB does not. The data was fsynced but fsync failed fsync isn't guaranteed to succeed. And when it fails you can't tell which write failed. It may not even be a failure of a write to a file that your process opened: Ideally, the kernel would report errors only on file descriptions on which writes were done that subsequently failed to be written back. The generic pagecache infrastructure does not track the file descriptions that have dirtied each individual page however, so determining which file descriptors should get back an error is not possible. Instead, the generic writeback error tracking infrastructure in the kernel settles for reporting errors to fsync on all file descriptions that were open at the time that the error occurred. In a situation with multiple writers, all of them will get back an error on a subsequent fsync, even if all of the writes done through that particular file descriptor succeeded (or even if there were no writes on that file descriptor at all). Don't be 2018-era Postgres. The only way to have known which exact write failed would be to open the file with O_DIRECT|O_SYNC (or O_DIRECT|O_DSYNC), though this is not the only way to handle fsync failures. The data was corrupted If you don't checksum your data on write and check the checksum on read (as well as periodic scrubbing a la ZFS) you will never be aware if and when the data gets corrupted and you will have to restore (who knows how far back in time) from backups if and when you notice. ZFS, MongoDB (WiredTiger), MySQL (InnoDB), and RocksDB checksum data by default. Postgres and SQLite do not (though databases created from Postgres 18+ will). You should probably turn on checksums on any system that supports it, regardless of the default. The data was partially written Only when the page size you write = block size of your filesystem = sector size of your disk is a write guaranteed to be atomic. If you need to write multiple sectors of data atomically there is the risk that some sectors are written and then the system crashes or restarts. This is called torn writes or torn pages. Postgres, SQLite, and MySQL (InnoDB) handle torn writes. Torn writes are by definition not relevant to immutable storage systems like RocksDB (and other LSM Tree or Copy-on-Write systems like MongoDB (WiredTiger)) unless writes (that update metadata) span sectors. If your file system duplicates all writes like MySQL (InnoDB) does (like you can with data=journal in ext4) you may also not have to worry about torn writes. On the other hand, this amplifies writes 2x. The data didn't reach disk, part 2 Sometimes fsync succeeds but the data isn't actually on disk because the disk is lying. These are called lost writes or phantom writes. You can be resilient to phantom writes by always reading back what you wrote (expensive) or versioning what you wrote. Databases and file systems generally do not seem to handle this situation. The data was written to the wrong place, read from the wrong place If you aren't including where data is supposed to be on disk as part of the checksum or page itself, you risk being unaware that you wrote data to the wrong place or that you read from the wrong place. This is called misdirected writes/reads. Databases and file systems generally do not seem to handle this situation. Further reading In increasing levels of paranoia (laudatory) follow ZFS, Andrea and Remzi Arpaci-Dusseau, and TigerBeetle.
This is an external post of mine. Click here if you are not redirected.
This is an external post of mine. Click here if you are not redirected.
More in technology
Fairphone has bad customer support. It’s not an issue with the individual customer support agents, I know how difficult their job is1, and I’m sure that they’re trying their best, but it’s a more systematic issue in the organization itself. It’s become so bad that Fairphone issued an open letter to the Fairphone community forum acknowledging the issue and steps they’re taking to fix it. Until then, I only have my experience to go by. I’ve contacted Fairphone customer support twice, once with a question about Fairphone 5 security updates not arriving in a timely manner, and another time with a request to refund the Fairphone Fairbuds XL as part of the 14-day policy. In both cases, I received an initial reply over 1 month later. It’s not that catastrophic for a non-critical query, but in situations where you have a technical issue with a product, this can become a huge inconvenience for the customer. I recently gave the Fairbuds XL a try because the reviews for it online were decent and I want to support the Fairphone project, but I found the sound profile very underwhelming and the noise cancelling did not work adequately.2 I decided to use the 14-day return policy that Fairphone advertise, which led to the worst customer care experience I’ve had so far.3 Here’s a complete timeline of the process on how to return a set of headphones to the manufacturer for a refund. 2025-02-10: initial purchase of the headphones 2025-02-14: I receive the headphones and test them out, with disappointing results 2025-02-16: I file a support ticket with Fairphone indicating that I wish to return the headphones according to their 14-day return policy 2025-02-25: I ask again about the refund after not hearing back from Faiprhone 2025-03-07: I receive an automated message that apologized for the delay and asked me to not make any additional tickets on the matter, which I had not been doing 2025-04-01: I start the chargeback process for the payment through my bank due to Fairphone support not replying over a month later 2025-04-29: Fairphone support finally responds with instructions on how to send back the device to receive a refund 2025-05-07: after acquiring packaging material and printing out three separate documents (UPS package card, invoice, Cordon Electronics sales voucher), I hand the headphones over to UPS 2025-05-15: I ask Fairphone about when the refund will be issued 2025-05-19 16:20 EEST: I receive a notice from Cordon Electronics confirming they have received the headphones 2025-05-19 17:50 EEST: I receive a notice from Cordon Electronics letting me know that they have started the process, whatever that means 2025-05-19 20:05 EEST: I receive a notice from Cordon Electronics saying that the repairs are done and they are now shipping the device back to me (!) 2025-05-19 20:14 EEST: I contact Fairphone support about this notice that I received, asking for a clarification 2025-05-19 20:24 EEST: I also send an e-mail to Cordon Electronics clarifying the situation and asking them to not send the device back to me, but instead return it to Fairphone for a refund 2025-05-20 14:42 EEST: Cordon Electronics informs me that they have already shipped the device and cannot reverse the decision 2025-05-21: Fairphone support responds, saying that it is being sent back due to a processing error, and that I should try to “refuse the order” 2025-05-22: I inform Fairphone support about the communication with Cordon Electronics 2025-05-27: Fairphone is aware of the chargeback that I initiated and they believe the refund is issued, however I have not yet received it 2025-05-27: I receive the headphones for the second time. 2025-05-28: I inform Fairphone support about the current status of the headphones and refund (still not received) 2025-05-28: Fairphone support recommends that I ask the bank about the status of the refund, I do so but don’t receive any useful information from them 2025-06-03: Fairphone support asks if I’ve received the refund yet 2025-06-04: I receive the refund through the dispute I raised through the bank. This is almost 4 months after the initial purchase took place. 2025-06-06: Fairphone sends me instructions on how to send back the headphones for the second time. 2025-06-12: I inform Fairphone that I have prepared the package and will post it next week due to limited access to a printer and the shipping company office 2025-06-16: I ship the device back to Fairphone again. There’s an element of human error in the whole experience, but the initial lack of communication amplified my frustrations and also contributed to my annoyances with my Fairphone 5 boiling over. And just like that, I’ve given up on Fairphone as a brand, and will be skeptical about buying any new products from them. I was what one would call a “brand evangelist” to them, sharing my good initial experiences with the phone to my friends, family, colleagues and the world at large, but bad experiences with customer care and the devices themselves have completely turned me off. If you have interacted with Fairphone support after this post is live, then please share your experiences in the Fairphone community forum, or reach out to me directly (with proof). I would love to update this post after getting confirmation that Fairphone has fixed the issues with their customer care and addressed the major shortcomings in their products. I don’t want to crap on Fairphone, I want them to do better. Repairability, sustainability and longevity still matter. I haven’t worked as a customer care agent, but I have worked in retail, so I roughly know what level of communication the agents are treated with, often unfairly. ↩︎ that experience reminded me of how big of a role music plays in my life. I’ve grown accustomed to using good sounding headphones and I immediately noticed all the little details being missing in my favourite music. ↩︎ until this point, the worst experience I had was with Elisa Eesti AS, a major ISP in Estonia. I wanted to use my own router-modem box that was identical to the rented one from the ISP, and that only got resolved 1.5 months later after I expressed intent to switch providers. Competition matters! ↩︎
Robot arms are very cool and can be quite useful, but they also tend to be expensive. That isn’t just markup either, because the components themselves are pricey. However, you can save a lot of money if you make some sacrifices and build everything yourself. In that case, you can follow Ruben Sanchez’s tutorial to […] The post Build your own 4DOF robotic arm on a budget appeared first on Arduino Blog.
Bona fides: Commodore 128DCR on my desk with a second 1571, Ultimate II+-L and a ZoomFloppy, three SX-64s I use for various projects, heaps of spare 128DCRs, breadbox 64s, 16s, Plus/4s and VIC-20s on standby, multiple Commodore collectables (blue-label PET 2001, C64GS, 116, TV Games, 1551, 1570), a couple A500s, an A3000 and a AmigaOS 3.9 QuikPak A4000T with '060 CPU, Picasso IV RTG card and Ethernet. I wrote for COMPUTE!'s Gazette (during the General Media years) and Loadstar. Here's me with Jack Tramiel and his son Leonard from a Computer History Museum event in 2007. It's on my wall. Retro Recipes video (not affiliated) stating that, in answer to a request for a very broad license to distribute under the Commodore name, Commodore Corporation BV instead simply proposed he buy them out, which would obviously transfer the trademark to him outright. Amiga News has a very nice summary. There was a time when Commodore intellectual property and the Commodore brand had substantial value, and that time probably ended around the mid-2000s. Prior to that point after Commodore went bankrupt in 1994, a lot of residual affection for the Amiga and the 64/128 still circulated, the AmigaOS still had viability for some applications and there might have been something to learn from the hardware, particularly the odder corners like the PA-RISC Hombre. That's why there was so much turmoil over the corpse, from Escom's abortive buyout to the split of the assets. Today the Commodore name (after many shifts and purchases and reorgs) is presently held by Commodore Corporation BV, a Netherlands company, who licenses it out. Pretty much the rest of it is split into the hardware patents (now with Acer after their buyout of Gateway 2000) and the remaining IP (Amiga Corporation, effectively Cloanto). The Commodore brand after the company's demise has had an exceptionally poor track record in the market. Many of us remember the 1999 Commodore 64 Web.it, licensed by Escom, which was a disastrously bad set-top 486 PC sold as an "Internet computer" whose only link to CBM was the Commodore name and a built-in 64 emulator. Reviewers savaged it and they've become collectors' items purely for the lulz. In 2007, Tulip licensee Commodore Gaming tried again with PC gaming rigs sold as the Commodore XX, GS, GX and G (are these computers or MPAA ratings?) and special wraps called C=kins (say it "skins"). I went to the launch party in L.A. — 8-Bit Weapon was there, hi Seth and Michelle! — and I even have one of their T-shirts around someplace. The company subsequently ran out of money and their most consequential legacy was the huge and heavily branded case. More recently, in 2010, another American company called itself Commodore USA LLC and tried developing new keyboard computers, most notably the (first) Commodore 64x. These were otherwise underpowered PCs using mini-ATX motherboards in breadboard-like cases where cooling was an obvious issue. They also tried selling "VICs" (which didn't look like VIC-20s) and "Amigas" (which were Intel i7 systems), and introduced their own Linux-based Commodore OS. Opinions were harsh and the company went under after its CEO died in 2012. Dishonourable mentions include Tulip-Yeahronimo's 2004 MP3 player line, sold as the (inexplicably) e-VIC, m-PET and f-PET, and the PET smartphone, a 2015 otherwise unremarkable Android device with its own collection of on-board emulators. No points for guessing how much of an impact those made. And none of this is really specific to Commodore, either: look at the shambling corpse of Atari SA, made to dance on decaying strings by the former Infogrames' principals. I mean, cryptocurrency and hotels straight out of Blade Runner — really? The exception to the rule was the 2004 C64DTV, a Tulip-licensed all-in-one direct-to-TV console containing a miniaturized and enhanced Commodore 64 designed by Jeri Ellsworth in a Competition Pro-style joystick. It played many built-in games from flash storage but more importantly could be easily modded into a distinct Commodore computer of its own, complete with keyboard and IEC serial ports, and VICE even emulates it. It sold well enough to go through two additional hardware revisions and the system turned up in other contemporary DTVs (like the DTV3 in the Hummer DTV game). There are also the 2019 "TheC64" machines, in both mini and full-size varieties (not affiliated), which are pretty much modern direct-to-TV systems in breadbin cases that run built-in games under emulation. The inclusion of USB "Comp Pro" styled joysticks is an obvious secondary homage to the C64DTV. Notably, Retro Games Ltd licensed the Commodore 64 ROMs from Cloanto but didn't license the Commodore trademark, so the name Commodore never appears anywhere on the box or the machine (though you decide if the trade dress is infringing). The remnant of the 64x was its case moulds, which were bought by My Retro Computer Ltd in the UK after Commodore USA LLC went under and that's where this story picks up, selling an officially licened new version of the 64x (also not affiliated) after Commodore Corporation BV granted permission in 2022. This new 64x comes in three pre-built configurations or as a bare case. By buying out the Commodore name they would get to sell these without the (frankly exorbitant) fees CC BV was charging and extend the brand to other existing Commodore re-creations like the Mega 65, but the video also has more nebulous aims, such as other retro Commodore products (Jeri Ellsworth herself appears in this video) or something I didn't quite follow about a Commodore charity arcade for children's hospitals, or other very enthusiastically expressed yet moderately unclear goals. I've been careful not to say there's no point in buying the Commodore trademark — I said there's not much. There is clearly a market for reimplementing classic Commodore hardware; Ellsworth herself proved it with the C64DTV, and current devices like the (also not affiliated with any) Mega 65, Ultimate64 and Kawari VIC-II still sell. But outside of the retro niche, Commodore as a brand name is pretty damn dead. Retro items sell only small numbers in boutique markets. Commodore PCs and Commodore smartphones don't sell because the Commodore name adds nothing now to a PC or handset, and the way we work with modern machines — for better or worse — is worlds different than how we worked with a 1982 home computer. No one expects to interact with, say, a Web page or a smartphone app in the same way we used a BASIC program or a 5.25" floppy. Maybe we should, but we don't. Furthermore, there's also the very pertinent question of how to steward such a community resource. The effort is clearly earnest, genuine and heartfelt, but that's not enough without governance. Letting these obviously hobbyist projects become full-fledged members of the extended Commodore family seems reasonable and even appropriate, but then there's the issue of preventing the Shenzhen back alley cloners from ripping them (and you) off. Plus, even these small products do make some money. What's FRAND in a situation like this? How would you enforce it? Should you enforce it? Does everyone who chips in get some fraction of a vote or some piece of the action? If the idea is only to allow the Commodore name to be applied to projects of sufficient quality and/or community benefit, who decides? Better to let it rest in peace and stop encouraging these bloodsuckers to drain what life and goodwill remain in the Commodore name. The crap products that came before only benefited the licensor and just make the brand more tawdry. CC BV only gets to do what it does because it's allowed to. TheC64 systems sold without the Commodore trademark because it was obvious what they were and what they do; Mega 65s and Ultimate64s are in the same boat. Commodore enthusiasts like me know what these systems are. We'll buy them on their merits, or not, whether the Commodore name is on the label, or not (and they will likely be cheaper if they don't). CC BV reportedly has been trying to sell off the trademark for awhile, which seems to hint that they too recognize the futility. Don't fall into their trap.
