Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Louwrentius - Solar
Skip to the bottom two paragraph for instructions on how to replace the battery. I bought my Bose SoundLink on-ear Bluetooth headphones for 250 Euros around 2017 and I really like them. They are small, light, comfortable and can easily fit in a coat pocket when folded. Up until now (about 7 years later) I have replaced the ear cushions in 2019 (€25) and 2024 (€18). Early 2025, battery capacity had deteriorated to a point where it became noticeable. The battery was clearly dying. Unfortunately these headphones aren't designed for easy battery replacement: Bose hasn't published instructions on how to replace the battery, doesn't offer a replacement battery and hasn't documented which battery type/model is used. The left 'head phone' has two Torx security screws and most people won't have the appropriate screwdriver for this size There is soldering involved I wanted to try a battery replacement anyway as I hate to throw away a perfectly good, working product just because the battery has worn out. Maybe at some point the headband needs replacing, but with a fresh battery, these headphones can last another 7 years. Let's prevent a bit of e-waste with a little bit of cost and effort. Most of all, the cost of this battery replacement is much lower than a new pair of headphones as the battery was €18 including taxes and shipping. Right to repair should include easy battery replacement Although my repair seemed to have worked out fine, it requires enough effort that most people won't even try. For this reason, I feel that it should be mandatory by law that: Batteries in any product must be user-replaceable (no special equipment or soldering required) Batteries must be provided by the vendor until 10 years after the last day the product was sold (unless it's a standard format like AA(A) or 18650). Batteries must be provided at max 10% of the cost of the original product The penalty for non-compliance should be high enough such that it won't be regarded as the cost of doing business For that matter, all components that may wear down over time should be user-replaceable. What you need to replace the battery Buy the exact battery type: ahb571935pct-01 (350mAh) (notice the three wires!) A Philips #0 screwdriver / bit A Torx T6H security screwdriver / bit (iFixit kits have them) A soldering iron Solder Heat shrink for 'very thin wire' Multimeter (optional) a bit of tape to 'cap off' bare battery leads Please note that I found another battery ahb571935pct-03 with similar specifications (capacity and voltage) but I don't know if it will fit. Putting the headphone ear cushion back on can actually be the hardest part of the process, you need to be firm and this process is documented by Bose. Battery replacement steps I took Make sure you don't short the wires on the old or new battery during replacement The battery is located in the left 'head phone'. Use a multimeter to check if your new battery isn't dead (should be 3+ volt) Remove the ear cushion from the left 'head phone' very gently as not to tear the rim Remove the two philips screws that keep the driver (speaker) in place Remove the two Torx screws (you may have to press a bit harder) Remove the speaker and be carefull not to snap the wire Gently remove the battery from the 'head phone' Cut the wires close to the old battery (one by one!) and cover the wires on the battery to prevent a short Strip the three wires from the headphones a tiny bit (just a few mm) Put a short piece of heat shrink on each of the three wires of the battery Solder each wire to the correct wire in the ear cup Adjust the location of the heat shrink over the freshly soldered joint. Use the soldering iron close to the heat shrink to shrink it (don't touch anything), this can take some time, be patient Check that the heat shrink is fixed in place and can't move Put the battery into it's specific location in the back of the 'head phone' Test the headphones briefly before reassembling the headphones Reassemble the 'head phone' (consider leaving out the two Torx screws) Dispose of the old battery in a responsible manner
My 4U 71 TiB ZFS NAS built with twenty-four 4 TB drives is over 10 years old and still going strong. Although now on its second motherboard and power supply, the system has yet to experience a single drive failure (knock on wood). Zero drive failures in ten years, how is that possible? Let's talk about the drives first The 4 TB HGST drives have roughly 6000 hours on them after ten years. You might think something's off and you'd be right. That's only about 250 days worth of runtime. And therein lies the secret of drive longevity (I think): Turn the server off when you're not using it. According to people on Hacker News I have my bearings wrong. The chance of having zero drive failures over 10 years for 24 drives is much higher than I thought it was. So this good result may not be related to turning my NAS off and keeping it off most off the time. My NAS is turned off by default. I only turn it on (remotely) when I need to use it. I use a script to turn the IoT power bar on and once the BMC (Baseboard Management Controller) is done booting, I use IPMI to turn on the NAS itself. But I could have used Wake-on-Lan too as an alternative. Once I'm done using the server, I run a small script that turns the server off, wait a few seconds and then turn the wall socket off. It wasn't enough for me to just turn off the server, but leave the motherboard, and thus the BMC powered, because that's just a constant 7 watts (about two Raspberry Pis at idle) being wasted (24/7). This process works for me because I run other services on low-power devices such as Raspberry Pi4s or servers that use much less power when idling than my 'big' NAS. This proces reduces my energy bill considerably (primary motivation) and also seems great for hard drive longevity. Although zero drive failures to date is awesome, N=24 is not very representative and I could just be very lucky. Yet, it was the same story with the predecessor of this NAS, a machine with 20 drives (1 TB Samsung Spinpoint F1s (remember those?)) and I also had zero drive failures during its operational lifespan (~5 years). The motherboard (died once) Although the drives are still ok, I had to replace the motherboard a few years ago. The failure mode of the motherboard was interesting: it was impossible to get into the BIOS and it would occasionally fail to boot. I tried the obvious like removing the CMOS battery and such but to no avail. Fortunately, the [motherboard]1 was still available on Ebay for a decent price so that ended up not being a big deal. ZFS ZFS worked fine for all these years. I've switched operating systems over the years and I never had an issue importing the pool back into the new OS install. If I would build a new storage server, I would definitely use ZFS again. I run a zpool scrub on the drives a few times a year2. The scrub has never found a single checksum error. I must have run so many scrubs, more than a petabyte of data must have been read from the drives (all drives combined) and ZFS didn't have to kick in. I'm not surprised by this result at all. Drives tend to fail most often in two modes: Total failure, drive isn't even detected Bad sectors (read or write failures) There is a third failure mode, but it's extremely rare: silent data corruption. Silent data corruption is 'silent' because a disk isn't aware it delivered corrupted data. Or the SATA connection didn't detect any checksum errors. However, due to all the low-level checksumming, this risk is extremely small. It's a real risk, don't get me wrong, but it's a small risk. To me, it's a risk you mostly care about at scale, in datacenters4 but for residential usage, it's totally reasonable to accept the risk3. But ZFS is not that difficult to learn and if you are well-versed in Linux or FreeBSD, it's absolutely worth checking out. Just remember! Sound levels (It's Oh So Quiet) This NAS is very quiet for a NAS (video with audio). But to get there, I had to do some work. The chassis contains three sturdy 12V fans that cool the 24 drive cages. These fans are extremely loud if they run at their default speed. But because they are so beefy, they are fairly quiet when they run at idle RPM5, yet they still provide enough airflow, most of the time. But running at idle speeds was not enough as the drives would heat up eventually, especially when they are being read from / written to. Fortunately, the particular Supermicro motherboard I bought at the time allows all fan headers to be controlled through Linux. So I decided to create a script that sets the fan speed according to the temperature of the hottest drive in the chassis. I actually visited a math-related subreddit and asked for an algorithm that would best fit my need to create a silent setup and also keep the drives cool. Somebody recommended to use a "PID controller", which I knew nothing about. So I wrote some Python, stole some example Python PID controller code, and tweaked the parameters to find a balance between sound and cooling performance. The script has worked very well over the years and kept the drives at 40C or below. PID controllers are awesome and I feel it should be used in much more equipment that controls fans, temperature, and so on, instead of 'dumb' on/of behaviour or less 'dumb' lookup tables. Networking I started out with quad-port gigabit network controllers and I used network bonding to get around 450 MB/s network transfer speeds between various systems. This setup required a ton of UTP cables so eventually I got bored with that and I bought some cheap Infiniband cards and that worked fine, I could reach around 700 MB/s between systems. As I decided to move away from Ubuntu and back to Debian, I faced a problem: the Infiniband cards didn't work anymore and I could not figure out how to fix it. So I decided to buy some second-hand 10Gbit Ethernet cards and those work totally fine to this day. The dead power supply When you turn this system on, all drives spin up at once (no staggered spinup) and that draws around 600W for a few seconds. I remember that the power supply was rated for 750W and the 12 volt rail would have been able to deliver enough power, but it would sometimes cut out at boot nonetheless. UPS (or lack thereof) For many years, I used a beefy UPS with the system, to protect against power failure, just to be able to shutdown cleanly during an outage. This worked fine, but I noticed that the UPS used another 10+ watts on top of the usage of the server and I decided it had to go. Losing the system due to power shenanigans is a risk I accept. Backups (or a lack thereof) My most important data is backed up trice. But a lot of data stored on this server isn't important enough for me to backup. I rely on replacement hardware and ZFS protecting against data loss due to drive failure. And if that's not enough, I'm out of luck. I've accepted that risk for 10 years. Maybe one day my luck will run out, but until then, I enjoy what I have. Future storage plans (or lack thereof) To be frank, I don't have any. I built this server back in the day because I didn't want to shuffle data around due to storage space constraints and I still have ample space left. I have a spare motherboard, CPU, Memory and a spare HBA card so I'm quite likely able to revive the system if something breaks. As hard drive sizes have increased tremendously, I may eventually move away from the 24-drive bay chassis into a smaller form-factor. It's possible to create the same amount of redundant storage space with only 6-8 hard drives with RAIDZ2 (RAID 6) redundancy. Yet, storage is always expensive. But another likely scenario is that in the coming years this system eventually dies and I decide not to replace it at all, and my storage hobby will come to an end. I needed the same board, because the server uses four PCIe slots: 3 x HBA and 1 x 10Gbit NIC. ↩ It takes ~20 hours to complete a scrub and it uses a ton of power while doing so. As I'm on a dynamic power tariff, I run it on 'cheap' days. ↩ every time I listen to ZFS enthusiasts you get the impression you are taking insane risks with your data if you don't run ZFS. I disagree, it all depends on context and circumstances. ↩ enterprise hard drives used in servers and SANs had larger sector sizes to accommodate even more checksumming data to prevent against silent data corruption. ↩ Because there is little airflow by default, I had to add a fan to cool the four PCIe cards (HBA and networking) or they would have gotten way too hot. ↩
Before Chat-GPT caused a sensation, big tech companies like Facebook and Apple were betting their future growth on virtual reality. But I'm convinced that virtual reality will never be a mainstream thing. If you ever used VR you know why: A heavy thing on your head that messes up your hair Nausea The focus on virtual reality felt like desperation to me. The desperation of big tech companies trying to find new growth, ideally a monopoly they control1, to satisfy the demands of shareholders. And then OpenAI dropped ChatGPT and all the big tech companies started to pivot so fast because in contrary to VR, AI doesn't involve making people nauseated and look silly. It's probably obvious that I feel it's not about AI itself. It is really about huge tech companies that have found a new way to sustain growth a bit longer, now that all other markets have been saturated. Flush with cash, they went nuts and bought up all the AI accelerator hardware2, which in turn uses unspeakable amounts of energy to train new large language models. Despite all the hype, current AI technology is at it's core a very sophisticated statistical model. It's all about probabilities, it can't actually reason. As I see it, work done by AI can't thus be trusted. Depending on the specific application, that may be less of an issue, but that is a fundamental limitation of current technology. And this gives me pause as it limits the application where it is most wanted: to control labour. To reduce the cost of headcount and to suppress wages. As AI tools become capable enough, it would be irresponsible towards shareholders not to pursue this direction. All this just to illustrate that the real value of AI is not for the average person in the street. The true value is for those bigger companies who can keep on growing, and the rest is just collateral damage. But I wonder: when the AI hype is over, what new hype will take it's place? I can't see it. I can't think of it. But I recognise that the internet created efficiencies that are convenient, yet social media weaponised this convenience to exploit our fundamental human weaknesses. As shareholder value rose, social media slowly chips away at the fabric of our society: trust. I've sold my Oculus Rift CV1 long ago, I lost hundreds of dollars of content but I refuse to create a Facebook/Meta account. ↩ climate change accelerators ↩
Introduction Victron Multiplus-II inverter/charges are configured with the veconfigure1 tool. Unforntunately this is a Windows-only tool, but there is still a way for Apple users to run this tool without any problems. Tip: if you've never worked with the Terminal app on MacOS, it might not be an easy process, but I've done my best to make it as simple as I can. A tool called 'Wine' makes it possible to run Windows applications on MacOS. There are some caveats, but none of those apply to veconfigure, this tool runs great! I won't cover in this tutorial how to make the MK-3 USB cable work. This tutorial is only meant for people who have a Cerbo GX or similar device, or run VenusOS, which can be used to remotely configure the Multipluss device(s). Step 1: install brew on macos Brew is a tool that can install additional software Visit https://brew.sh and copy the install command open the Terminal app on your mac and paste the command now press 'Enter' or return It can take a few minutes for 'brew' to install. Step 2: install wine Enter the following two commands in the terminal: brew tap homebrew/cask-versions brew install --cask --no-quarantine wine-stable Download Victron veconfigure Visit this page Scroll to the section "VE Configuration tools for VE.Bus Products" Click on the link "Ve Configuration Tools" You'll be asked if it's OK to download this file (VECSetup_B.exe) which is ok Start the veconfigure installer with wine Open a terminal window Run cd Enter the command wine Downloads\VECSetup_B.exe Observe that the veconfigure Windows setup installer starts Click on next, next, install and Finish veconfigure will run for the first time Click on the top left button on the video to enlarge These are the actual install steps: How to start veconfigure after you close the app Open a terminal window Run cd Run cd .wine/drive_c/Program\ Files\ \(x86\)/VE\ Configure\ tools/ Run wine VEConfig.exe Observe that veconfigure starts Allow veconfigure access to files in your Mac Download folder Open a terminal window Run cd run cd .wine/drive_c/ run ls -n ~/Downloads We just made the Downloads directory on your Mac accessible for the vedirect software. If you put the .RSVC files in the Downloads folder, you can edit them. Please follow the instructions for remote configuration of the Multiplus II. Click on the "Ve Configuration Tools" link in the "VE Configuration tools for VE.Bus Products" section. ↩
More in technology
Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’
You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.
Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together
This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.
