More from Victor Poughon
Some time ago, I saw this video by DIY Perks where they make artificial sunlight at home with a 500W LED and a gigantic (1.2m) parabolic reflector. I've been fascinated by this project ever since, and I wanted my own. Over the past year or so, I finally took the time to work on a similar project, but I had the idea for a different design. The issue with the parabolic reflector is that it takes a huge amount of space. Could I do something similar, but with a less bulky design? This is the story of my first attempt at this project - version 1 so to speak. Perhaps there will be a version 2 in the future. Enjoy the read! My idea - as others have had I'm sure - was to use an array of lenses laid out as a grid. Then, instead of a single light source, I would use a grid array of multiple LEDs, one per lens. In my mind, this would have two major advantages: Less bulky. The size of the device would be determined by the focal length of the individual lens elements, and because each would be small, the focal length could be small also, while maintaining a decent f number. Easier thermal management. Multiple light sources could be regular low power LEDs which wouldn't need special cooling. There would just be a lot of them, spread out over the entire device surface. Over the course of this project, I also intended to teach myself some manufacturing and 3D design, as I don't have any experience doing any of this. My background is software, and as you'll see I took a very software heavy approach to this. It was all a long learning journey for me, but in the end I used: Mostly build123d for CAD modeling, with some FreeCAD for final assembly checks and some experiments here and there - including with the cool OpticsWorkbench. KiCad for PCB design. Custom python code for simulating light and optimizing the optical system. (This custom code eventually became an entire open-source project for optimization-based optical design) JLCPCB for printing and assembling PCBs, and for manufacturing aluminum and plastic parts with their CNC service. TL;DR: I did it! Here is the finished device sitting on my desk today, at night: And here it is during the day (much less impressive!) Beware it's kinda hard to take good pictures of it, and I don't have the best photo gear. Here's also a video: (at night) Kinda cool that you can see a lens flare effect in the shape of the lens grid array. Technical specs Mechanical: Lens square side length: 30mm Effective Focal length: 55mm Array size: 6x6 = 36 LEDs Total size: 180x180mm Parts: Lenses: 1 biconvex lens array, 1 plano-convex lens array - custom made out of PMMA acrylic, CNC fabrication with vapor polish finish @ JLCCNC LEDs: LUXEON 2835 3V -- Ref: 2835HE. CRI: 95+, color temp: 4000K, 65mA. PCBs: Custom design Mounting hardware: custom design - aluminium 60601 for the CNC parts and mate black resin for the 3D printed parts Rayleigh diffuser: waterproof printing inkjet film General design and sizing To create artificial sunlight, you need four ingredients: Parallel light rays. The sun is so far away that light rays emitted from a point on the surface of the sun reach us essentially parallel. This is not to say that all light rays coming from the sun are parallel, as it still has a 0.5 deg apparent angular size. But they need to be pretty straight. Any light coming from an artificial light source like an LED will be going in all directions, so some optics is required. High color quality. A good indicator to look for on a datasheet is the color rendering index (CRI). 95+ is recommended to achieve a good effect. I'm sure there's more color science you could get into, but CRI is a great start for off the shelf parts. Rayleigh scattering, or an imitation of it. A LOT of power. Light intensity is the most important sizing constraint, so let's look at it first. Now, the sun is very bright. Like, ridiculously bright: around 100,000 lux. To achieve this with LEDs is by no means impossible, but it's a challenge. For this first version, I thought that targetting 10,000 lux would be quite enough because it would reduce the power consumption a lot for a first prototype, and also brightness perception is logarithmic. So one tenth of the intensity is really, perceptually, almost the same as full brightness. (In the end, I estimate my design only effectively achieved something between 1000 and 10000 lux). The general grid based design of this project really has two variables: the individual LED light output, in lumens the individual lens surface area in mm² After some research, I think values between 30 to 130 lumens are typical for high CRI surface mount LEDs. So, assuming this is what we are working with, what is the required lens size to achieve the brightness of the sun? We have to assume some non perfect efficiency for collimating the light. This will never be 100%, and in fact may be quite low if the focal length is high, because a lot of the light will be hitting the side walls instead of reaching the lens. The lens itself will also be absorbing some light. So taking a wild guess of 0.5 for the overall optical efficiency, and taking three lumens value of 30, 80 and 130, we get this plot: With that in mind, I selected 30mm as my lens square side length. Presumably, this would be small enough to achieve some effect, but not too small to make the lenses too hard to make. Lenses Focal length, and the lenses shape in general, is the next design consideration. The goal is to have perfectly parallel light rays. In theory, with a perfect point source and a perfect lens this is easy. Put the light source at the lens focal length, you're done. In practice, a lot of things make it harder to achieve with a lens. (This is where the parabolic reflector design is superior to a lens). A LED is not a point source A lens will not have perfect optical performance (i.e. aberrations) Mechanical reality of the device means that positioning and orientation will not be perfect A LED radiation pattern is not isotropic, meaning intensity will be greater at the lens center This is the radiation pattern characteristics diagram from my LED datasheet: I wrote some custom python code to simulate the optical system I had in mind, and find the best lens shape using numerical optimization. (This code eventually became an open-source project: torchlensmaker) After a lot of experimentation, I settled on a 2 lens design: Lens 1: Biconvex parabolic lens Lens 2: Planoconvex parabolic lens The effective focal length of this two lens system is about 55mm. Focal length is a key design parameter, and here I feel like more experimentation is needed. It's a big tradeoff consideration and has a huge impact on the system design. It impacts: The curvature of the lens surface, which is a key manufacturing point (you want to minimize curvature for manufacturing, which means maximizing focal length) The optical efficiency of the system due to the led radiance pattern (here you want to minimize focal length, to gather more of the emitted light) The device thickness (here I wanted a not-too-thick device, so to minimize focal length also) I used a two lens system mostly to reduce the surface curvature of the lens arrays. This reduces the manufacturing cost by a lot. High curvature lenses are more expensive in general, and this grid array design means that a high curvature lens will create sort of "valleys" in between the lenses. Because I was targetting CNC manufacturing, this is to be minimized to get a design that's even possible to machine. This is the optical simulation I had at the time I finalized the design and ordered the lenses. (Since then my simulation code has improved and I could likely do much better modeling today using the latest version of torchlensmaker): With some custom build123d code I was able to make the two lenses 3D models by stacking the lenses in a grid pattern and adding edges for mounting: Your browser does not support iframes. Your browser does not support iframes. What's really cool using build123d for 3D modeling, is that I can just change a python variable to change the size of the array, of the thickness of the lens, of anything else really. It's all parametric out of the box because it's regular Python code! This makes exploring the design space very efficient. I've never done 3D modeling any other way, but I can't imagine ever not having the power of programming with me if I ever do it again! I had the lenses manufactured out of PMMA acrylic at JLC with a vapor polish finish. Total cost for the lenses was about 55€ which is really not bad! One of the two main lens array, built by JLCCNC: LEDs I really wanted to use the 3030 G04 from YUJILEDS, but it's only sold on 5000 units reels that cost $1000 a piece... maybe for version 2 I will upgrade to those. For version 1, I settled on LUXEON 2835 3V. They are about 3 times less bright than the YUJILED, but they have good color rendering and the SMD package I was looking for. And importantly, the minimum order quantity was only 50 at JLC global sourcing. In the version 1 design, the grid is 6x6 which means 36 LEDs total. PCBs I designed a custom PCB with KiCAD. Each PCB holds 6 LEDs which are laid out as 2 segments of a 12V led strip in parallel. This allows to use a standard wall plug 12V power supply. The mechanical role of the PCB is very important in this design. Not only does it distribute power to the LEDs and regulate current, it also precisely positions the LEDs at the lens focal point. For this, exporting the PCB 3D model and importing it into FreeCAD was very useful to check that everything fits together: the PCB in the aluminum support baseplate, the holes on the light hoods, etc. My Python code exported the precise LED coordinates which I could input into KiCad's layout editor. I had the PCB printed and the components assembled by JLCPCB. It's very very cool to design an electronic board on your computer and get it fully assembled in the mail a few weeks later - no soldering required! (for this step anyway). Mechanical mounting parts To mount everything together I designed 3 parts: A baseplate, to hold the PCBs and the side walls. The PCBs are fitted below the baseplate, and light goes through holes drilled into the baseplate. There are also partial holes to allow for the thickness of the SMD resistors mounted on top of the PCBs, and finally two mounting holes per PCB. This is why it has so many holes :) Your browser does not support iframes. Side walls to hold the lenses using grooves in which to insert them, and a larger groove to secure in the baseplate. The baseplate side holes are threaded to support M2 screws securing the base of the walls. Again, JLCCNC did the drilling and threading of the holes at a great price. Your browser does not support iframes. Light hoods, a rectangle block with rectangular holes. It sits on top of the PCB to shape the light coming from each LED into a cone (or really a four sided pyramid). This is to make sure light from a given LED only reaches its matching lens on the lens array, and no other. Bleed light is inevitable, but at least this prevents direct leakage. Your browser does not support iframes. The hoods were 3D printed out of black resin, the walls and baseplate were CNC cut out of Aluminum 60601. I'm not a mechanical engineer so this process was... trial and error. Still the result is working so I'm quite happy with that. For a possible version 2, there's a lot I'll change in the mechanical design. But apart from the one design flaw I was able to fix manually with a drill (more on that below), everything fit together quite well on the first try. Rayleigh scattering The final ingredient is Rayleigh scattering. This is the physical phenomenon that makes the sky look blue, and it's important to achieve a convincing effect. In the DIY Perks video that inspired this project, they used a home made liquid solution with suspended particles of the correct size for Rayleigh scattering. Not super practical and I really wanted to find another solution (get it?). Thankfully, some time after the original video, someone on the diyperks forum discovered that inkjet print film achieves a very similar effect. A quick trip to a local office supply store was all I needed here! Amazing discovery. I didn't anticipate this step during the initial design phase, so the film is simply cut to the correct size and secured with black electrical tape. Assembly After a few weeks of design work, and another few weeks of waiting for the parts to arrive, it was finally time for assembly! On top of the individual 3D models made with build123d, I had a final assembly FreeCAD model with all parts fitted together, including the lenses: Note the green brackets that I initially planned to use. When actually assembling the walls to the baseplate, the solidity of the formed box was very high, I decided to drop the brackets entirely. This is why some extra unused holes remain on the side walls. This is all the parts just after unboxing (excluding the inkjet film, solder tin, screws, power supply, wiring, electrical tape): The only real design flaw was insufficient width of the grooves that hold the lenses. The lenses have an edge thickness of 1.2mm, which I had intended to fit into a 1.22mm groove. Turns out this was not enough, probably due to a combination of manufacturing tolerance and additional thickness added by the anodizing black matte surface finish of the aluminum part. The lenses didn't fit into the grooves! I don't have a very advanced tools at home, so my best solution to this was making the existing grooves wider by hand using a power drill. I bought a 1.5mm metal drill bit and achieved a decent result by doing 4 to 5 passes per groove. This took about 2-3h in total because I had to move the bit quite slow and could only machine about 1/4th of each groove depth at a time by moving the drill bit slowly accross, and there are 8 grooves total. Here's some more pictures of assembly below. The back side after soldering wires to the PCB power pins and a socket for the 12V power supply. The PCBs and hood pieces share a common mounting hole so only two screws per PCB-hood pair are used. The front side of the baseplate + PCB + hoods assembly, but without the lenses, powered on. Don't look at it directly :) It's interesting to note that in the picture above, all of the light you can see from the LEDs is actually "bleed light" and not useful light. None of the light visible above is the light that's intended to go into the lens and produce the sunlight effect. Testing with partial assembly of the walls and only 1 out of the 2 lenses: Testing the inkjet film layers with an avocado as a subject. I settled on using two layers of the inkjet film for the final build: Cost Overall I spent around 1000€ on this project. But this includes cost of tools I was missing, prototype parts that I had manufactured but discarded, bulk orders for parts like LEDs and PCBs which had a minium order quantity above what I need for 1 unit, and various supplies like screws, etc. The actual raw cost of parts only, without shipping, to build the final unit is hard to estimate. But I would say around 300€. The most expensive parts are the CNC parts (PMMA lenses and the aluminum baseplate and walls) accounting for about 2/3rd of the total price. The rest (PCBs, assembly service, LEDs, 3D printed plastic parts) was quite cheap. Conclusion As I write this the final piece is sitting on my desk and producing a pleasant soft white glow. It's definitely nice, and I'm very proud of the result - especially because this was by far the biggest build project I have ever done. Thanks to this project, I've learned a ton about PCB design, electronics and CNC manufacturing and optics. I even got so far down the side quest of learning optics that I started an open-source python project for modeling geometric optics. So, is it convincing as artificial sunlight? My honest answer to that is: partially. The geometric effect of the light source appearing at infinity works. As I pan and tilt my head from side to side, the illusion of light coming from way far behind the object is 100% a success. On top of that, if you look at it while moving your head into the light beam, my eyes get surprised - almost hurt - by the sudden intensity jump. This indicates that collimation is good and you can sort of see it in the video at the start of this post. However it's apparent that it's simply too weak. Don't get me wrong, it's still bright. I can't look at it directly without sunglasses, and honestly it's really hard to take a good picture of it because the contrast between the light it emits and the outside of it is very high. Another downside is that I can definitely make out the grid of lenses, as the intensity pattern clearly reveals the grid shape. This is quite a minor downside and not really unpleasant, and I'm sure it could be improved upon. If I were to ever work on a version 2, I would focus on: More power. My feeling is the light output needs to be 3 to 5 times stronger to get any closer to a convincing effect, and it's not crazy to aim for as much as 10x brighter than this prototype. More surface area. This prototype is 18cm x 18cm. So you only really get the effect if you are able to sit with the produced straight beam of light, which is quite narrow to resemble any kind of "fake window". A future version would need to be 2 to 4 times wider in my opinion. Better optical design. I still think a refraction based design is possible, but it requires very precise optical design and mechanical tolerances. My feeling is that a refraction based design, especially as a grid, is very sensitive to positioning and orientation of parts. I lack mechanical engineering skills in this area. However there are some really encouraging things that I really like about this grid based, refractive design: It's scalable. If I had built 4 identical items, I could literally stack them on top of each other and get more surface area. The "bezels" would be only 5% of the total light emitting area, and I'm sure this could be lowered. I also like that the inner design calls for repeated elements, as this introduces some economy of scale, even at the prototype level. The only part that's not trivially scalable is the lens grid. Maybe it could be injection molded for very large scale production, or for medium scale you could come up with a way to tile multiple lens grids into a larger overall grid pattern, adding some thin bezels for mounting. It's compact. The total size is 19cm x 19cm x 9cm. This is quite compact for a 5cm focal length and an effective lighting area of 18cm x 18cm. Reflective designs like the DIYPerks video or commercial products like CoeLux do not achieve this form factor. Thermal management is better by design. This is not really something I got into for this design, as it's quite underpowered. The whole thing runs comfortably on a 12V / 3A wall brick power supply. But this design offers great margin for scaling up because there isn't a single light source to cool down, but a number of LEDs proportional to the surface area. I suspect the main thermal issue when scaling up would be the cooling of the power supply itself, not of the lamp. As final thoughts, let me talk about the software heavy approach I had for this project. It's awesome. If I was starting a manufacturing company today, I would do it all code based. PCBs, 3D models, assembly, testing... I want code everywhere. The power of changing a parameter and having the entire design updated with a single script it so good. Run a script and get all the production data including GERBERs, BOM, 3D models, mechanical schematics, technical diagrams, automated tolerance and electrical checks... absolutely no manual steps between changing a design parameter and ready to send a new order to manufacturing. The PCB and CAD space is even evolving to use proper CI/CD tools which is really exciting. I don't know if I'll ever have the time to work on version 2 of this project, but it was great fun anyway! And now I have a cool unique lamp. Thank you for reading!
OK, this is going to sound crazy: I believe all clocks are 30 seconds late. Let's get a few things out of the way: this is neither about time zones, nor leap seconds. It's not about synchronization of clocks, and it's not about relativity or any obscure corner of physics. I'm talking about everyday clocks. The ones that you might have around in your house, like this one that sits on my desk: Or even the clock on your phone! Maybe they're all correctly set very precisely to a very accurate reference clock, but here I will argue that they are still exactly 30 seconds late. In other words: It would be more accurate if they were thirty seconds ahead. What are you on about? All the above clocks have one thing in common: they don't show seconds. They truncate down to the nearest lower whole minute, right? So when it's actually 14:15:45, they'll show 14:15. And when the actual time goes from 14:15:59 to 14:16:00, then that's when your clock changes from 14:15 to 14:16. They apply the floor function! Ok great. Now hear me out! Let's compute the average error between a usual (truncating) clock, and the precise current time. import datetime import numpy as np import matplotlib.pyplot as plt import matplotlib.dates as mdates # A datetime array from 14:00:15pm for 300 seconds true_time = np.array([datetime.datetime(2024, 12, 20, 14, 00, 15) + datetime.timedelta(seconds=i) for i in range(300)]) # The clock truncates seconds shown_time = [t.replace(second=0) for t in true_time] # The clock error is the shown time minus the true time error = [dt.total_seconds() for dt in (shown_time - true_time)] fig, ax = plt.subplots(figsize=(10, 3.5)) ax.plot(true_time, error, label="clock error", color="teal") ax.hlines(np.mean(error), xmin=true_time[0], xmax=true_time[-1], label="average", linestyle="--", color="orange") ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M')) ax.set_ylim([-60, 60]) ax.set_title("A normal clock") ax.legend() Sometimes the error is 1 or 2 seconds, like when the clock shows 14:02 and it's actually 14:02:01 Sometimes the error is almost 60 seconds, like when the clock shows 14:02 and it's actually 14:02:57 So, the average error of a truncating clock is 30 seconds. Let that sink in for a moment: the average error of a clock is 30 seconds. If clocks rounded to the nearest minute instead of truncating, the average error would be 0. Therefore all clocks are 30 seconds late! Similarly of course, all clocks that show seconds but not milliseconds are half a second late. # The rounding clock! aka +30s shown_time = [(t + datetime.timedelta(seconds=30)).replace(second=0) for t in true_time] # The clock error is the shown time minus the true time error = [dt.total_seconds() for dt in (shown_time - true_time)] fig, ax = plt.subplots(figsize=(10, 3.5)) ax.plot(true_time, error, label="clock error", color="teal") ax.hlines(np.mean(error), xmin=true_time[0], xmax=true_time[-1], label="average", linestyle="--", color="orange") ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M')) ax.set_ylim([-60, 60]) ax.set_title("A rounding clock") ax.legend() This is just a convention But this is just a convention! Truncating, rounding or even ceiling are all valid, as long as we pick one and stick to it. By your logic we should say that it's Tuesday as soon as it's one second past noon on Monday. Well... kind of. It would be weird if we rounded for years, months and days, that's for sure. I think most people think of those scales as intervals. In other words, July is a period of time, with a start and an end. So are years, centuries, seasons. We are inside of it or outside. Is it the same for the current minute of the current hour ? I'm not so sure. Basically I'm arguing that rounding for clocks would be more useful than flooring. This is especially apparent when you're trying to calculate "how much time until my next meeting?", and your next meeting is at noon. If it's 11:55, you would usually mentally subtract and conclude: the meeting is in 5 minutes. That's how I always do it myself anyway! But the most probable estimate given the available information is actually 4'30"! If you had a rounding clock instead, 5 minutes would be the correct estimation. In fact, as we go down the various scales of time all the way from years, months, days and we reach hours, that's when we start truncating! At hours! For example: if it's 10:43 and you are asked the time and (for some context appropriate reason) you reply with just hours, you would say it's 11! Personally, I would never say that it's 10 if the clock shows anything past 10:30. I realize that's probably very different in other cultures, though! And I'd be interested to learn about how different cultures approximate time in language. So, looking at the various scales of time and how I mentally view them: Years: truncate. If the full date has 2019 in it, I'll say "the year is 2019" - all the way to the end. Saying "it's 2020" in September 2019 is super weird. Months: truncate. It's January all the way until it's February. Days: truncate. It gets a little weird if it's past midnight and I haven't slept yet. Is tomorrow today? Or is it the day after that? I think collectively we still haven't quite figured that one out yet. And that's ok. Kind of like how "next Friday" becomes ambiguous on Thursdays. Some cultures use times beyond 24:00, like 25:30 to indicate one hour and a half past midnight, and I think that's lovely! Hours: That's when mentally, I switch to rounding! At 15:48 I definitely feel like it's pretty much 16:00. Minutes: At 15:48:57, I definitely feel like it's 15:49! But clocks switch back to flooring for for some reason! But mentally we round! This is why I feel like all clocks are 30 seconds late: because the truncating convention doesn't match my intuition at the minute scale. Seconds and below: definitely rounding. Please someone tell me I'm not crazy 🙃 EDIT 2025-01-07: This kinda blew up on HN and I want to clarify something: This is a tongue-in-cheek article! It's fun to think about, but in the end, this IS a convention and the most important thing about conventions is that we all follow the same ones. I'm obviously not going to move all my clocks forwards by 30 seconds and neither should you! In many ways truncating is a much better convention and that's why we use it. I've also been made aware that Big Ben is truly an analog clock and doesn't jump around every minute, so I've removed that example. My bad! The point still stands because some "analog" clocks actually jump every minute and are not truly analog, but I should have picked a better example photo.
For my torchlensmaker project, I need to represent lenses surface shapes parameterically. That is, I need equations that give the X and Y coordinates of the surface, as a function of a parameter, usually called t. The project supports different parametric shapes, but a very popular shape in optics is the arc of circle. So far I had been using the usual polar coordinates parameterization: X=Rcos(t)+RY=Rsin(t) This works great, you can represent circles that cross the origin at t=0, and it works for both signs of R (the radius) to represent the two possible curvature directions: import numpy as np import matplotlib.pyplot as plt def half_circle(R, color): t = np.linspace(np.pi/2, 3*np.pi/2, 1000) X = R * np.cos(t) + R Y = R * np.sin(t) plt.plot(X, Y, color=color) plt.gca().set_aspect("equal") half_circle(10, "orange") half_circle(-10, "navy") However, I want to do numerical optimization to find the best possible shape for lenses. This representation has two problems when used for optimization: When R is very large, t gets very small for a similar sized arc. This means that the scale of the parameter can span over a large range of orders of magnitudes, which is difficult to optimize for and can lead to precision issues. Additionally, representing a flat vertical line is not possible, as R would be infinite. But a flat surface is a very common case in optics, and not handling it well is a problem. Crossing over and changing the direction of curvature of the circle arc during optimization is not possible with this representation, because R would have to "wrap around" positive infinity and come back to negative infinity. The solution I came up with is double: Use curvature instead of radius. Curvature is the inverse of the radius: K=1/R. This means a line is a circle of curvature zero. Brilliant! Don't use the angle as the parameter, but the Y coordinate directly. This reduces the shapes we can model to only half-circles, but for optics that's not a problem! So we know that the Y equation is easy: Y=t. But what about X? Well, I'll spare you the derivation, but here it is: X=Kt21+1−t2K2 And that's a half-circle 🤩! We can also get the derivative with respect to t, which is needed when doing collision detection with Newton's method (more on that in a future article maybe!) X′=Kt1−t2K2 import numpy as np import matplotlib.pyplot as plt def half_circle(R, color): t = np.linspace(-10, 10, 1000) K = 1/R X = (K * t**2) / (1 + np.sqrt(1 - t**2 * K**2)) Y = t plt.plot(X, Y, color=color) plt.gca().set_aspect("equal") half_circle(10, "orange") half_circle(-10, "navy") Interestingly, to get some help because I suck at calculus double check my work, I asked both an LLM and Wolfram Alpha to derive the equation. Wolfram Alpha comes out on top with the final form, while LLM gets stuck and can't simplify all the way. They both show step-by-step derivation (although Wolfram Alpha's is behind a paywall). ChatGPT4o mini (top, derivation not shown) vs Wolfram Alpha (bottom):
More in technology
Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’
You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.
Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together
This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.