Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Victor Poughon
div.gradient { width: 650px; height: 60px; border-radius: 3px; margin-bottom: 10px; box-shadow: 5px 5px 5px rgb(0, 0, 0, 0.5); margin: 0 auto; } pre { max-width: 600px; } Colormaps are functions that map from [0, 1] to colors. I love colormaps! Maybe a little too much? There is a huge amount of very serious color science behind colormaps, and also a great Python package called colorcet which implements a lot of them. Would you ever need those as a CSS gradient? Probably not. Did I have fun writing a script to convert every 120 of them to a CSS gradient? Absolutely! Enjoy 😁 Note that these sample the actual colormap at N=101 points, and then your browser does linear interpolation to fill in the gaps. The actual colormaps are 255-entry tables. So not only is this horribly inefficient by using very verbose CSS syntax, it's also destroying a lot of the original colormap's information. This is just for fun, use the original data for anything serious. Also, I excluded the categorical colormaps because they don't really make sense as a gradient. Accent Blues BrBG BuGn BuPu CMRmap Dark2 GnBu Greens Grays / Greys OrRd Oranges PRGn Paired Pastel1 Pastel2 PiYG PuBu PuBuGn PuOr PuRd Purples RdBu RdGy RdPu RdYlBu RdYlGn Reds Set1 Set2 Set3 Spectral Wistia YlGn YlGnBu YlOrBr YlOrRd afmhot autumn bgy bgyw binary bjy bkr bky blues bmw bmy bone brg bwr bwy cet_CET_C1 cet_CET_C10 cet_CET_C10s cet_CET_C11 cet_CET_C11s cet_CET_C1s cet_CET_C2 cet_CET_C2s cet_CET_C3 cet_CET_C3s cet_CET_C4 cet_CET_C4s cet_CET_C5 cet_CET_C5s cet_CET_C6 cet_CET_C6s cet_CET_C7 cet_CET_C7s cet_CET_C8 cet_CET_C8s cet_CET_C9 cet_CET_C9s cet_CET_CBC1 cet_CET_CBC2 cet_CET_CBD1 cet_CET_CBD2 cet_CET_CBL1 cet_CET_CBL2 cet_CET_CBL3 cet_CET_CBL4 cet_CET_CBTC1 cet_CET_CBTC2 cet_CET_CBTD1 cet_CET_CBTL1 cet_CET_CBTL2 cet_CET_CBTL3 cet_CET_CBTL4 cet_CET_D1 cet_CET_D10 cet_CET_D11 cet_CET_D12 cet_CET_D13 cet_CET_D1A cet_CET_D2 cet_CET_D3 cet_CET_D4 cet_CET_D6 cet_CET_D7 cet_CET_D8 cet_CET_D9 cet_CET_I1 cet_CET_I2 cet_CET_I3 cet_CET_L1 cet_CET_L10 cet_CET_L11 cet_CET_L12 cet_CET_L13 cet_CET_L14 cet_CET_L15 cet_CET_L16 cet_CET_L17 cet_CET_L18 cet_CET_L19 cet_CET_L2 cet_CET_L20 cet_CET_L3 cet_CET_L4 cet_CET_L5 cet_CET_L6 cet_CET_L7 cet_CET_L8 cet_CET_L9 cet_CET_R1 cet_CET_R2 cet_CET_R3 cet_CET_R4 cet_bgy cet_bgyw cet_bjy cet_bkr cet_bky cet_blues cet_bmw cet_bmy cet_bwy cet_circle_mgbm_67_c31 cet_circle_mgbm_67_c31_s25 cet_colorwheel cet_coolwarm cet_cwr cet_cyclic_bgrmb_35_70_c75 cet_cyclic_bgrmb_35_70_c75_s25 cet_cyclic_grey_15_85_c0 cet_cyclic_grey_15_85_c0_s25 cet_cyclic_isoluminant cet_cyclic_mrybm_35_75_c68 cet_cyclic_mrybm_35_75_c68_s25 cet_cyclic_mybm_20_100_c48 cet_cyclic_mybm_20_100_c48_s25 cet_cyclic_mygbm_30_95_c78 cet_cyclic_mygbm_30_95_c78_s25 cet_cyclic_mygbm_50_90_c46 cet_cyclic_mygbm_50_90_c46_s25 cet_cyclic_protanopic_deuteranopic_bwyk_16_96_c31 cet_cyclic_protanopic_deuteranopic_wywb_55_96_c33 cet_cyclic_rygcbmr_50_90_c64 cet_cyclic_rygcbmr_50_90_c64_s25 cet_cyclic_tritanopic_cwrk_40_100_c20 cet_cyclic_tritanopic_wrwc_70_100_c20 cet_cyclic_wrkbw_10_90_c43 cet_cyclic_wrkbw_10_90_c43_s25 cet_cyclic_wrwbw_40_90_c42 cet_cyclic_wrwbw_40_90_c42_s25 cet_cyclic_ymcgy_60_90_c67 cet_cyclic_ymcgy_60_90_c67_s25 cet_dimgray cet_diverging_bkr_55_10_c35 cet_diverging_bky_60_10_c30 cet_diverging_bwg_20_95_c41 cet_diverging_bwr_20_95_c54 cet_diverging_bwr_40_95_c42 cet_diverging_bwr_55_98_c37 cet_diverging_cwm_80_100_c22 cet_diverging_gkr_60_10_c40 cet_diverging_gwr_55_95_c38 cet_diverging_gwv_55_95_c39 cet_diverging_isoluminant_cjm_75_c23 cet_diverging_isoluminant_cjm_75_c24 cet_diverging_isoluminant_cjo_70_c25 cet_diverging_linear_bjr_30_55_c53 cet_diverging_linear_bjy_30_90_c45 cet_diverging_linear_protanopic_deuteranopic_bjy_57_89_c34 cet_diverging_protanopic_deuteranopic_bwy_60_95_c32 cet_diverging_rainbow_bgymr_45_85_c67 cet_diverging_tritanopic_cwr_75_98_c20 cet_fire cet_gouldian cet_gray cet_gwv cet_isolum cet_isoluminant_cgo_70_c39 cet_isoluminant_cgo_80_c38 cet_isoluminant_cm_70_c39 cet_kb cet_kbc cet_kbgyw cet_kg cet_kgy cet_kr cet_linear_bgy_10_95_c74 cet_linear_bgyw_15_100_c67 cet_linear_bgyw_15_100_c68 cet_linear_bgyw_20_98_c66 cet_linear_blue_5_95_c73 cet_linear_blue_95_50_c20 cet_linear_bmw_5_95_c86 cet_linear_bmw_5_95_c89 cet_linear_bmy_10_95_c71 cet_linear_bmy_10_95_c78 cet_linear_gow_60_85_c27 cet_linear_gow_65_90_c35 cet_linear_green_5_95_c69 cet_linear_grey_0_100_c0 cet_linear_grey_10_95_c0 cet_linear_kbc_5_95_c73 cet_linear_kbgoy_20_95_c57 cet_linear_kbgyw_10_98_c63 cet_linear_kbgyw_5_98_c62 cet_linear_kgy_5_95_c69 cet_linear_kry_0_97_c73 cet_linear_kry_5_95_c72 cet_linear_kry_5_98_c75 cet_linear_kryw_0_100_c71 cet_linear_kryw_5_100_c64 cet_linear_kryw_5_100_c67 cet_linear_protanopic_deuteranopic_kbjyw_5_95_c25 cet_linear_protanopic_deuteranopic_kbw_5_95_c34 cet_linear_protanopic_deuteranopic_kbw_5_98_c40 cet_linear_protanopic_deuteranopic_kyw_5_95_c49 cet_linear_ternary_blue_0_44_c57 cet_linear_ternary_green_0_46_c42 cet_linear_ternary_red_0_50_c52 cet_linear_tritanopic_kcw_5_95_c22 cet_linear_tritanopic_krjcw_5_95_c24 cet_linear_tritanopic_krjcw_5_98_c46 cet_linear_tritanopic_krw_5_95_c46 cet_linear_wcmr_100_45_c42 cet_linear_worb_100_25_c53 cet_linear_wyor_100_45_c55 cet_rainbow cet_rainbow4 cet_rainbow_bgyr_10_90_c83 cet_rainbow_bgyr_35_85_c72 cet_rainbow_bgyr_35_85_c73 cet_rainbow_bgyrm_35_85_c69 cet_rainbow_bgyrm_35_85_c71 CET_C10 / circle_mgbm_67_c31 CET_C10s / circle_mgbm_67_c31_s25 cividis colorwheel cool coolwarm copper cubehelix cwr CET_C11 / cyclic_bgrmb_35_70_c75 CET_C11s / cyclic_bgrmb_35_70_c75_s25 CET_C5 / cyclic_grey_15_85_c0 CET_C5s / cyclic_grey_15_85_c0_s25 cyclic_isoluminant CET_C1 / cyclic_mrybm_35_75_c68 CET_C1s / cyclic_mrybm_35_75_c68_s25 CET_C9 / cyclic_mybm_20_100_c48 CET_C9s / cyclic_mybm_20_100_c48_s25 CET_C2 / cyclic_mygbm_30_95_c78 CET_C2s / cyclic_mygbm_30_95_c78_s25 CET_C8 / cyclic_mygbm_50_90_c46 CET_C8s / cyclic_mygbm_50_90_c46_s25 CET_CBC1 / cyclic_protanopic_deuteranopic_bwyk_16_96_c31 CET_CBC2 / cyclic_protanopic_deuteranopic_wywb_55_96_c33 CET_C6 / cyclic_rygcbmr_50_90_c64 CET_C6s / cyclic_rygcbmr_50_90_c64_s25 CET_CBTC1 / cyclic_tritanopic_cwrk_40_100_c20 CET_CBTC2 / cyclic_tritanopic_wrwc_70_100_c20 CET_C3 / cyclic_wrkbw_10_90_c43 CET_C3s / cyclic_wrkbw_10_90_c43_s25 CET_C4 / cyclic_wrwbw_40_90_c42 CET_C4s / cyclic_wrwbw_40_90_c42_s25 CET_C7 / cyclic_ymcgy_60_90_c67 CET_C7s / cyclic_ymcgy_60_90_c67_s25 dimgray CET_D4 / diverging_bkr_55_10_c35 CET_D6 / diverging_bky_60_10_c30 CET_D13 / diverging_bwg_20_95_c41 CET_D1A / diverging_bwr_20_95_c54 CET_D1 / diverging_bwr_40_95_c42 CET_D9 / diverging_bwr_55_98_c37 CET_D10 / diverging_cwm_80_100_c22 diverging_gkr_60_10_c40 CET_D3 / diverging_gwr_55_95_c38 CET_D2 / diverging_gwv_55_95_c39 CET_D12 / diverging_isoluminant_cjm_75_c23 diverging_isoluminant_cjm_75_c24 CET_D11 / diverging_isoluminant_cjo_70_c25 CET_D8 / diverging_linear_bjr_30_55_c53 CET_D7 / diverging_linear_bjy_30_90_c45 CET_CBD2 / diverging_linear_protanopic_deuteranopic_bjy_57_89_c34 CET_CBD1 / diverging_protanopic_deuteranopic_bwy_60_95_c32 CET_R3 / diverging_rainbow_bgymr_45_85_c67 CET_CBTD1 / diverging_tritanopic_cwr_75_98_c20 fire flag gist_earth gist_gray / gist_grey gist_heat gist_ncar gist_rainbow gist_stern gist_yarg / gist_yerg gnuplot gnuplot2 gouldian gray / grey gwv hot hsv inferno isolum CET_I1 / isoluminant_cgo_70_c39 CET_I2 / isoluminant_cgo_80_c38 CET_I3 / isoluminant_cm_70_c39 jet kb kbc kbgyw kg kgy kr linear_bgy_10_95_c74 linear_bgyw_15_100_c67 linear_bgyw_15_100_c68 CET_L9 / linear_bgyw_20_98_c66 linear_blue_5_95_c73 CET_L12 / linear_blue_95_50_c20 CET_L7 / linear_bmw_5_95_c86 linear_bmw_5_95_c89 CET_L8 / linear_bmy_10_95_c71 linear_bmy_10_95_c78 CET_L10 / linear_gow_60_85_c27 CET_L11 / linear_gow_65_90_c35 linear_green_5_95_c69 CET_L1 / linear_grey_0_100_c0 CET_L2 / linear_grey_10_95_c0 CET_L6 / linear_kbc_5_95_c73 CET_L20 / linear_kbgoy_20_95_c57 linear_kbgyw_10_98_c63 CET_L16 / linear_kbgyw_5_98_c62 CET_L5 / linear_kgy_5_95_c69 CET_L4 / linear_kry_0_97_c73 linear_kry_5_95_c72 linear_kry_5_98_c75 CET_L3 / linear_kryw_0_100_c71 linear_kryw_5_100_c64 linear_kryw_5_100_c67 CET_CBL1 / linear_protanopic_deuteranopic_kbjyw_5_95_c25 CET_CBL3 / linear_protanopic_deuteranopic_kbw_5_95_c34 CET_CBL2 / linear_protanopic_deuteranopic_kbw_5_98_c40 CET_CBL4 / linear_protanopic_deuteranopic_kyw_5_95_c49 CET_L15 / linear_ternary_blue_0_44_c57 CET_L14 / linear_ternary_green_0_46_c42 CET_L13 / linear_ternary_red_0_50_c52 CET_CBTL3 / linear_tritanopic_kcw_5_95_c22 CET_CBTL2 / linear_tritanopic_krjcw_5_95_c24 CET_CBTL1 / linear_tritanopic_krjcw_5_98_c46 CET_CBTL4 / linear_tritanopic_krw_5_95_c46 CET_L19 / linear_wcmr_100_45_c42 CET_L17 / linear_worb_100_25_c53 CET_L18 / linear_wyor_100_45_c55 magma nipy_spectral ocean pink plasma prism rainbow rainbow4 CET_R4 / rainbow_bgyr_10_90_c83 CET_R2 / rainbow_bgyr_35_85_c72 rainbow_bgyr_35_85_c73 CET_R1 / rainbow_bgyrm_35_85_c69 rainbow_bgyrm_35_85_c71 seismic spring summer tab10 tab20 tab20b tab20c terrain turbo twilight twilight_shifted viridis winter
OK, this is going to sound crazy: I believe all clocks are 30 seconds late. Let's get a few things out of the way: this is neither about time zones, nor leap seconds. It's not about synchronization of clocks, and it's not about relativity or any obscure corner of physics. I'm talking about everyday clocks. The ones that you might have around in your house, like this one that sits on my desk: Or even the clock on your phone! Maybe they're all correctly set very precisely to a very accurate reference clock, but here I will argue that they are still exactly 30 seconds late. In other words: It would be more accurate if they were thirty seconds ahead. What are you on about? All the above clocks have one thing in common: they don't show seconds. They truncate down to the nearest lower whole minute, right? So when it's actually 14:15:45, they'll show 14:15. And when the actual time goes from 14:15:59 to 14:16:00, then that's when your clock changes from 14:15 to 14:16. They apply the floor function! Ok great. Now hear me out! Let's compute the average error between a usual (truncating) clock, and the precise current time. import datetime import numpy as np import matplotlib.pyplot as plt import matplotlib.dates as mdates # A datetime array from 14:00:15pm for 300 seconds true_time = np.array([datetime.datetime(2024, 12, 20, 14, 00, 15) + datetime.timedelta(seconds=i) for i in range(300)]) # The clock truncates seconds shown_time = [t.replace(second=0) for t in true_time] # The clock error is the shown time minus the true time error = [dt.total_seconds() for dt in (shown_time - true_time)] fig, ax = plt.subplots(figsize=(10, 3.5)) ax.plot(true_time, error, label="clock error", color="teal") ax.hlines(np.mean(error), xmin=true_time[0], xmax=true_time[-1], label="average", linestyle="--", color="orange") ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M')) ax.set_ylim([-60, 60]) ax.set_title("A normal clock") ax.legend() Sometimes the error is 1 or 2 seconds, like when the clock shows 14:02 and it's actually 14:02:01 Sometimes the error is almost 60 seconds, like when the clock shows 14:02 and it's actually 14:02:57 So, the average error of a truncating clock is 30 seconds. Let that sink in for a moment: the average error of a clock is 30 seconds. If clocks rounded to the nearest minute instead of truncating, the average error would be 0. Therefore all clocks are 30 seconds late! Similarly of course, all clocks that show seconds but not milliseconds are half a second late. # The rounding clock! aka +30s shown_time = [(t + datetime.timedelta(seconds=30)).replace(second=0) for t in true_time] # The clock error is the shown time minus the true time error = [dt.total_seconds() for dt in (shown_time - true_time)] fig, ax = plt.subplots(figsize=(10, 3.5)) ax.plot(true_time, error, label="clock error", color="teal") ax.hlines(np.mean(error), xmin=true_time[0], xmax=true_time[-1], label="average", linestyle="--", color="orange") ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M')) ax.set_ylim([-60, 60]) ax.set_title("A rounding clock") ax.legend() This is just a convention But this is just a convention! Truncating, rounding or even ceiling are all valid, as long as we pick one and stick to it. By your logic we should say that it's Tuesday as soon as it's one second past noon on Monday. Well... kind of. It would be weird if we rounded for years, months and days, that's for sure. I think most people think of those scales as intervals. In other words, July is a period of time, with a start and an end. So are years, centuries, seasons. We are inside of it or outside. Is it the same for the current minute of the current hour ? I'm not so sure. Basically I'm arguing that rounding for clocks would be more useful than flooring. This is especially apparent when you're trying to calculate "how much time until my next meeting?", and your next meeting is at noon. If it's 11:55, you would usually mentally subtract and conclude: the meeting is in 5 minutes. That's how I always do it myself anyway! But the most probable estimate given the available information is actually 4'30"! If you had a rounding clock instead, 5 minutes would be the correct estimation. In fact, as we go down the various scales of time all the way from years, months, days and we reach hours, that's when we start truncating! At hours! For example: if it's 10:43 and you are asked the time and (for some context appropriate reason) you reply with just hours, you would say it's 11! Personally, I would never say that it's 10 if the clock shows anything past 10:30. I realize that's probably very different in other cultures, though! And I'd be interested to learn about how different cultures approximate time in language. So, looking at the various scales of time and how I mentally view them: Years: truncate. If the full date has 2019 in it, I'll say "the year is 2019" - all the way to the end. Saying "it's 2020" in September 2019 is super weird. Months: truncate. It's January all the way until it's February. Days: truncate. It gets a little weird if it's past midnight and I haven't slept yet. Is tomorrow today? Or is it the day after that? I think collectively we still haven't quite figured that one out yet. And that's ok. Kind of like how "next Friday" becomes ambiguous on Thursdays. Some cultures use times beyond 24:00, like 25:30 to indicate one hour and a half past midnight, and I think that's lovely! Hours: That's when mentally, I switch to rounding! At 15:48 I definitely feel like it's pretty much 16:00. Minutes: At 15:48:57, I definitely feel like it's 15:49! But clocks switch back to flooring for for some reason! But mentally we round! This is why I feel like all clocks are 30 seconds late: because the truncating convention doesn't match my intuition at the minute scale. Seconds and below: definitely rounding. Please someone tell me I'm not crazy 🙃 EDIT 2025-01-07: This kinda blew up on HN and I want to clarify something: This is a tongue-in-cheek article! It's fun to think about, but in the end, this IS a convention and the most important thing about conventions is that we all follow the same ones. I'm obviously not going to move all my clocks forwards by 30 seconds and neither should you! In many ways truncating is a much better convention and that's why we use it. I've also been made aware that Big Ben is truly an analog clock and doesn't jump around every minute, so I've removed that example. My bad! The point still stands because some "analog" clocks actually jump every minute and are not truly analog, but I should have picked a better example photo.
For my torchlensmaker project, I need to represent lenses surface shapes parameterically. That is, I need equations that give the X and Y coordinates of the surface, as a function of a parameter, usually called t. The project supports different parametric shapes, but a very popular shape in optics is the arc of circle. So far I had been using the usual polar coordinates parameterization: X=Rcos(t)+RY=Rsin(t) This works great, you can represent circles that cross the origin at t=0, and it works for both signs of R (the radius) to represent the two possible curvature directions: import numpy as np import matplotlib.pyplot as plt def half_circle(R, color): t = np.linspace(np.pi/2, 3*np.pi/2, 1000) X = R * np.cos(t) + R Y = R * np.sin(t) plt.plot(X, Y, color=color) plt.gca().set_aspect("equal") half_circle(10, "orange") half_circle(-10, "navy") However, I want to do numerical optimization to find the best possible shape for lenses. This representation has two problems when used for optimization: When R is very large, t gets very small for a similar sized arc. This means that the scale of the parameter can span over a large range of orders of magnitudes, which is difficult to optimize for and can lead to precision issues. Additionally, representing a flat vertical line is not possible, as R would be infinite. But a flat surface is a very common case in optics, and not handling it well is a problem. Crossing over and changing the direction of curvature of the circle arc during optimization is not possible with this representation, because R would have to "wrap around" positive infinity and come back to negative infinity. The solution I came up with is double: Use curvature instead of radius. Curvature is the inverse of the radius: K=1/R. This means a line is a circle of curvature zero. Brilliant! Don't use the angle as the parameter, but the Y coordinate directly. This reduces the shapes we can model to only half-circles, but for optics that's not a problem! So we know that the Y equation is easy: Y=t. But what about X? Well, I'll spare you the derivation, but here it is: X=Kt21+1−t2K2 And that's a half-circle 🤩! We can also get the derivative with respect to t, which is needed when doing collision detection with Newton's method (more on that in a future article maybe!) X′=Kt1−t2K2 import numpy as np import matplotlib.pyplot as plt def half_circle(R, color): t = np.linspace(-10, 10, 1000) K = 1/R X = (K * t**2) / (1 + np.sqrt(1 - t**2 * K**2)) Y = t plt.plot(X, Y, color=color) plt.gca().set_aspect("equal") half_circle(10, "orange") half_circle(-10, "navy") Interestingly, to get some help because I suck at calculus double check my work, I asked both an LLM and Wolfram Alpha to derive the equation. Wolfram Alpha comes out on top with the final form, while LLM gets stuck and can't simplify all the way. They both show step-by-step derivation (although Wolfram Alpha's is behind a paywall). ChatGPT4o mini (top, derivation not shown) vs Wolfram Alpha (bottom):
More in technology
Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’
You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.
Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together
This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.
