More from ./techtipsy
Seize the means of code production!
Man takes old computer, turns it into a home server again. Man happy.
Networking has long been my Achilles heel. I know the very basics, but the more complex areas of networking have been a bit puzzling to me. By the time I figured out how IPv4 works, I found IPv6 and that my ISP supports it. Back to square one. That didn’t stop me from learning some bits, and after 8+ years of self-hosting as a hobby, I’ve settled on a setup that works for me and overcomes common residential internet connection nuances, such as dynamic IPv4 addresses and changing IPv6 prefixes. I’m sharing these tips and tricks with the goal of helping out other hobbyists out there that happen to share a similar stack. Background My ISP is polite enough to provide a public IPv4 address, and allowing incoming traffic is a toggle in their online self-service. Not perfect, but at least you can do it. However, they charge about 6 EUR a month for the static IP address service, which I am not willing to pay out of principle. They also support IPv6, which is great, and they provide you a whole /56 slice of it to play with using IPv6 prefix delegation. Unfortunately they have configured the lease time for the prefix to be incredibly short: 26 minutes! A router reboot or short power outage usually results in the IPv4 address and IPv6 prefix changing, which is really annoying as my services become unavailable for a short time. Dynamic DNS A common way to overcome the dynamic IP address limitation is to sign up with a provider to set up a DNS record that changes whenever your home IP address changes. My domain registrar does not have this as a feature, and I’m not interested in using a different provider, so I went in a different direction and built a home-grown script that does the same thing. Initially, this script relied on a public service that tells you what your IP address is, and based on that I could check if things have changed and I need to update my DNS record. This approach has one glaring catastrophic failure mode though: that provider could lie to you one day and now you’ve pointed your DNS records at the attackers’ servers. :) I ignored that failure mode for a while, but once I learned about the effectiveness of LLM-based tooling, I decided to give it a go and to build a better solution that takes into account my setup and requirements, while at the same time saving me from the frustration of troubleshooting and debugging this in a late evening. I’m still very limited on available free time, so optimizing for that is a priority for me. My main networking gear runs OpenWRT, and it supports running shell scripts periodically in a crontab. The router has two WAN interfaces, one for IPv4 and one for IPv6. It already knows what IP address and prefix have been assigned to it, so I don’t have to rely on an external service provider for finding this out. Handling IPv4 addresses is simple: check the IPv4 address of the WAN interface. Query your existing DNS records, diff it, and if it has changed, push an update in a separate API call. Super simple! With IPv6, the approach is slightly different. Instead of the WAN interface, I have to get the IPv6 address of the target machine, and make sure that it’s routable over the public internet. When you’ve checked your network settings in an IPv6 network, you may have noticed a lot of different IP addresses there, with lots of letters thrown into the mix. Here’s an example from the machine that is serving you this blog (likely out of date though!): inet6 fdb3:6dad:6dce::f41/128 scope global dynamic noprefixroute inet6 fdb3:6dad:6dce:0:2e0:4cff:fe0c:9ddb/64 scope global noprefixroute inet6 2001:7d0:856c:4000::f41/128 scope global dynamic noprefixroute inet6 2001:7d0:856c:4000:2e0:4cff:fe0c:9ddb/64 scope global dynamic noprefixroute inet6 fe80::2e0:4cff:fe0c:9ddb/64 scope link noprefixroute The two relevant ones are the ones that start with 2001:, others are link-local or accessible over the local network only. The shorter one consists of the IPv6 prefix part, and then the unique bit at the end is a predictable suffix that the host gets. The other one also works, but is as far as I understand randomly generated and more difficult to predict when we get around to next sections. I know that there is probably a better way to do this, but I wanted to keep things simple enough so that I can troubleshoot them if needed. It may be possible to trigger this updater script on events that WAN and WAN6 interfaces send, but I have not validated this theory. There are many different ways to find the IPv6 address of a particular host, so the script I have just tries multiple approaches to find the one that we’re looking for. Here’s the script in case you’re interested in setting up something similar. It reads credentials from an .env file and is built around the Zone API. On OpenWRT, the only dependency that you need to install is curl, which to my surprise was not part of the default packages list, probably to save on space. One lesson I learned from a previous iteration of the script: if you trigger DNS record updates every minute, then Zone will actually reach out to you via e-mail telling you to cut that shit out, politely. It was just one missing if statement, and yet it caused some frustration to engineers far away. Sorry! Predictable IP addresses It’s a good idea to set up static IP addresses for your hosts, both for IPv4 addresses and IPv6 prefix delegation via DUID-s. The OpenWRT GUI LuCI makes it quite simple, just set the addresses as static on the landing page for the hosts that you are interested in forwarding traffic to, and you’re done! My recommendation here is to also set a predictable IPv6 suffix, otherwise all your IPv6 traffic rules may break once again due to this nuance. I like to make that host number the same for both IPv4 and IPv6, quick example: 192.168.1.2 2001:7d0:854f:8e00::2 Here’s a configuration snippet example from /etc/config/dhcp, look for the hostid option: config host option name 'mycoolserver' option ip '192.168.1.69' list mac '12:34:56:78:90:AB' option duid 'yourduidgoeshere' option hostid '69' Apply with service dnsmasq restart. In LuCI, as of OpenWRT 25.12, look for “IPv6 token”. Note that due to a bug, it doesn’t seem to be possible to set a numeric IPv6 token via GUI, which is why you will need to add it manually in CLI using the above approach. Port forwards, traffic rules, potato, potahtoh Whenever you want to make a local machine accessible on the internet for IPv4, the solution is simple: set up a port forward to that particular machine, and you’re done! It’s a common enough flow for people who’ve set up game servers and the like, and well understood by more novice users. With IPv6, port forwards don’t help. You’ll have to check one tab over at “Traffic rules” in OpenWRT GUI. It’s a common misconception that by using IPv6 you are exposing everything to the world as each device gets its own IPv6 address, but turns out that this is not the case in most common setups. By default, OpenWRT forwards only a few types of traffic to IPv6 hosts, such as ICMP packets that make ping work between devices over IPv6 across the public internet. If you are interested in allowing IPv6 clients to access services on your local server that has an IPv6 address, you will have to explicitly allow it by adding a new traffic rule. There’s one issue with this approach that a lot of users seem to run into: if the IPv6 prefix changes, then all my traffic rules that are pointing to a particular host are automatically broken! Luckily there is a clever workaround implemented on OpenWRT that bypasses this issue. Assuming that you followed the previous step and set yourself up with a predictable IPv6 suffix, when setting up a traffic rule, set the target device up as ::69/-64, just replace 69 with your actual suffix. The IPv6 prefix can now change, but the ports that you’ve made accessible on this specific host will remain working. At this point, you should be all set with a reasonably well working setup where you’ve handled the issues with dynamic IPv4 and IPv6 prefix, and you can access your services over the public internet even when things happen. Limitations One issue that this setup has is the fact that DNS change propagation takes time. Usually clients will pick up the new records within 5 minutes, but in my professional career I’ve seen some clients take up to 24 hours or longer to finally start sending traffic to the new DNS record. Whenever your IP address changes, there will be a mini-outage. Not catastrophic if you’re just hosting hobby projects and personal services at home, but I wouldn’t host anything mission-critical in such a setup. When your OpenWRT device is as underpowered as mine, then you may notice that the TLS encryption overhead when curl -ing around can be significant. I have set my dynamic DNS script to run every 5 minutes, and it shows up on the CPU usage graphs on my router. Wireguard all the things! I know that Tailscale is a popular method of connecting up your devices and making your personal services privately accessible over that, which significantly reduces your attack surface. Being behind a few updates or not being vigilant enough is less of an issue compared to exposing your services over the public internet. You don’t necessarily need Tailscale for that though! If you just need a way to access your services over a private and secure network, then setting up a dedicated mini PC or single-board computer is a very good starting point. Let it be the server, allow traffic to move between the clients over the Wireguard interface, and you’re all set! Alternatively, if you have an OpenWRT router, then you can do it right there, but I found the GUI management setup to be a bit clunky compared to deploying the plain configuration files to clients. When I did do that test, I discovered quickly that my router and its single ARM CPU core with no cryptography extensions is too slow for managing my Wireguard network, with speeds topping out at 20 Mbit/s. 20. The LattePanda IOTA can easily saturate its gigabit link, as does the ThinkPad T430, and even devices like the Orange Pi Zero can handle a theoretical maximum of about 240 Mbit/s over Wireguard measured using wg-bench. My current Wireguard host is the LattePanda V1, the most unstable computer in my fleet. With a USB adapter, it can push almost half a gigabit of traffic over Wireguard. If you’re like me, and you like hosting your services over Docker or Podman, then instead of listening on ports for all interfaces on your containers (default behaviour when setting up port forwards), I recommend listening only on the Wireguard interface. This makes the service only accessible over Wireguard, meaning that you only need to set up one port forward and traffic rule to connect to the Wireguard network, and then you have access to all of your services. The attack surface is significantly reduced, the whole Wireguard solution is stable and very small, and unless you leak your private key, you are reasonably secure! Here’s a snippet from a compose file showcasing how to set this up for IPv4 and IPv6: ports: - 10.69.69.12:2283:2283 - "[fded:abba:acca::12]:2283:2283" Want to make the service available over Wireguard and over the local network directly? Just add those to the list! Note that if your local address changes and you don’t update it in the compose file, your container will refuse to start up as it cannot listen to the interface any longer, but you can mitigate that with the static IP addresses step. When you are going with this route, it is unlikely but still possible that by the time the container starts up, the Wireguard interface is not yet up. To resolve this, you can use systemd to set Wireguard up as a dependency that you will have to wait for before the container starts up. I manage my Wireguard connection with wg-quick@interfacename service. You can set up a systemd override for Docker, or if you manage your Docker/Podman services via systemd, then you can set it up per-service using this pattern: # /etc/systemd/system/myimmichserver.service.d/override.conf [Unit] [email protected] [email protected] By the way, systemd overrides are also really useful for ensuring that your storage that your containers rely on is properly mounted. If my service requires the path /immich to be available and mounted, add something like this: BindsTo=immich.mount After=immich.mount If you unmount the mount point, it will also properly bring down the service. The service won’t start if the mount point is missing. I’ve had the issue with containers seeing blank mount points more times than I’d like to admit, and this has eliminated this issue for me. systemd has received a lot of hate online, and I don’t think it’s fair. The ease with which you can set up dependencies on your system, set up resource limits, make services more restricted to improve the security posture is great and allows me to and avoid all sorts of failure modes. Production services that I’m responsible for make use of these systemd features, with great results. For services that need to be public, such as Nextcloud and its public shareable links, this approach won’t work, obviously, but for things that only you and your family members use, this is a viable approach. Conclusion This setup works well enough for me to confidently host my blog and self-hosted services off of it. I’ve hit a lot of paper cuts and frustrations along the way, but after following this guide, you don’t have to do the same. Yes, VLAN-s are intentionally missing from this guide. I’ll get to them eventually, maybe by Q4 2037 given my lack of free time. And no, IPv6 isn’t complicated, it’s just different from what everyone is used to. If we started out with IPv6 right from the get-go, we wouldn’t be having dumb arguments online.
Keeping my focus has been challenging. It’s not a new phenomenon, and I suspect that there are contributing factors that have lead to the unfocused state dominating. For example, I’ve been that guy who wants to be on top of things, to be in the loop, to respond to urgent issues. It feels fantastic to be in that firefighter role as it gives me the feeling of having an impact, but it results in me being drained at the end of the day and often over-caffeinated. One day I was doing work on my laptop on a couch because hitting 30 apparently means that sleeping slightly incorrectly results in debilitating back pain. During that session, I was working on a larger task and making tons of tiny little changes that needed to be done in order to release a new feature. I was finally in the zone again, and it felt fantastic! That’s when I decided to start an experiment: can I improve my focus by giving up my big monitor? Results I’ve done this type of “experiment” a few times in the past when the power has gone out and my super duper ergonomic setup has become useless. No power, no USB-C dock, no monitor. It wasn’t that fun and my eyes hated reading text off of a laptop screen. A few things have changed since then: GNOME has working fractional scaling that you can simply enable in display settings ThinkPad displays have gotten better, with the picture being quite cromulent, and the 16:10 aspect ratio helps fit more on the screen the nature of my work has changed and will keep changing in the near future Almost a month in, I’ve had a pleasant experience with this experiment. I feel more focused. Yeah, that’s it. Am I actually more focused is up for debate, as I’m not sure how to measure it objectively.1 Working off of a single screen forces me to focus at what’s at hand. Alt-tabbing to a different app is quick, but just enough to deter me from doing it in meetings or other focused tasks. In my personal free time2, this has also resulted in computer use becoming more intentional. On a 34" ultrawide monitor, it was too easy to put YouTube running on the left side, and whatever else on the right. It was distracting and resulted in time being wasted doing nothing. Interestingly enough, making computer use more intentional was a trick that I tried when recovering from burnout, and it helped a lot. As a side effect, the power consumption of my whole home office setup is significantly smaller, as I don’t have to power my ultrawide monitor. That made up most of the power consumption, with peaks of up to 100W. I also don’t have to fight with my dock killing my whole network, because there is no dock. How to do it well If you’re just cleaning up your desk and plopping your laptop on there, you will likely have a bad time. The posture will be off, and depending on your laptop, the keyboard and touchpad combination can prove to be an ergonomic nightmare. At the very least, you should put your laptop up somewhere higher. Ideally, it should be using a stand that allows you to use your favourite wireless keyboard and mouse below it. A simple laptop stand could get you most of the way there, but the ideal solution is a freely adjustable monitor arm combined with a VESA-mounted laptop holder. This gives you the freedom to place the laptop exactly as you’d like while leaving the desk free for your peripherals. Most monitor arm laptop holders have side arms that keep it in place, but I found them to be extremely annoying, so I removed them by disassembling the holder and yanking out the side arms and springs. You may still need them if you are using a very aggressive vertical angle, but I hated having to give up one USB-A port and blocking about 25% of the exhaust fan also didn’t seem like a good idea. Mounting the laptop with the springy side arms was also awkward. If you’re using a desktop and have a big display, then intentionally using a smaller and cheaper one for a while may prove to be just as effective. If you’re using a laptop with a horrible display with poor viewing angles, glare and crappy resolution (which a lot of older ThinkPads have), then you can still try this out, but I suspect that you’ll not have a very good experience with it due to this reason alone. Exceptions to the rule I still prefer to do my gaming sessions on a big screen. It’s more immersive, and I can make out tiny details better, such as spotting a car in the distance while driving in the oncoming lane in Need for Speed Most Wanted. Conclusion I’m happy with this setup. That’s all I ever needed. go ahead, try to measure developer productivity objectively. Good luck! ↩︎ that’s what I call the time window between putting my son to sleep and midnight. ↩︎
More in technology
Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’
You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.
Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together
This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.