Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Applied Cartography
One of the most useful and janky internal tools we have in Buttondown’s codebase is a codegen pipeline called “autogen”. There is nothing “auto” about autogen: it is a series of scripts that munges a bunch of data into a bunch of different formats, to generate things like our API clients and code snippets and storybooks. Some of this data is stateful, and therefore requires a database, and therefore requires migrations — you see how this kind of thing can grow somewhat labrynthine. Each individual script is pretty simple, but as we’ve found more and more things to glom onto autogen. This, to be clear, is a good thing. It’s really nice to have automatic, consistent data and types everywhere, so that we literally cannot change the API without also pushing a concomitant change to the API docs. With each glom, though, the wall-clock time of running autogen increases — and so I found myself staring down the barrel at a 50second script running whenever we wanted to make any sort of non-trivial change to our schema. Fifty seconds was too many seconds. I set a budget of ten seconds — still a long time, but significantly less onerous — and began digging in at low-hanging fruit. There was a lot. A few that come to mind: We split up our vite config so we could only run the portion that we needed (cross-piling and minifying our CSS bundles; We disabled all Sentry and perf-tracing stuff that was getting enabled as part of the standard build; We no-oped all of the Python-land data generation if it was already there, since that stateful data didn’t change very often. This was all great, but we were still left with 15 seconds of wall clock time. Profiling each individual cog in the script revealed that the problem was essentially “it’s Python”: four items in the script ran Django commands, and just spinning up the Django process and running autodiscovery took around two seconds. Ouch! The impulse was to cut down that runtime. A great post by Adam led us to discover the biggest culprit was our Stripe imports, and we timeboxed a bit of time to try and get rid of them, either by deferring the imports or excising the library; neither seemed particularly feasible. Then, suddenly, the answer seemed obvious. If we have four scripts where the fixed cost of invoking Django is the long pole, why not simply combine the scripts? And that’s exactly what we did: if len(sys.argv) > 1 and "," in sys.argv[1]: commands = sys.argv[1].split(",") original_argv = sys.argv.copy() for command in commands: sys.argv[1] = command execute_from_command_line(sys.argv) sys.argv = original_argv else: execute_from_command_line(sys.argv)
Here is a confession: I am a very strong proponent of a robust test suite being perhaps the single most important asset of a codebase, but when it comes to auxiliary services like admin sites or CLIs when it comes to testing I tend to ask for forgiveness more than I ask for permission. Django's admin site is no different: and, because Django's admin DSL is very magic-string-y, there's a lot of stuff that never gets caught by CI or mypy until a lovely CS agent informs me that something is blowing up in their face. Take this example, which bites me more often than I care to admit: from django.contrib import admin from stripe.models import StripeCustomer class StripeCustomer(models.Model): id = models.CharField(max_length=100, unique=True) username = models.CharField(max_length=100, unique=True) email_address = models.EmailField() creation_date = models.DateTimeField(auto_now=True) @admin.register(StripeCustomer) class StripeCustomerAdmin(admin.ModelAdmin): list_display = ( "id", "username", "email", "creation_date", ) search_fields = ( "username", "email", ) One thing that has made my life slightly easier in this respect is a parametric test that just makes sure we can render the empty state for every single admin view. Code snippet first, explanation after: from django.urls import get_resolver, reverse def extract_routes(resolver: URLResolver) -> iter[str]: keys = [key for key in resolver.reverse_dict.keys() if isinstance(key, str)] key_to_route = {key: resolver.reverse_dict.getlist(key)[0][0][0] for key in keys} for key in keys: yield key for key, (prefix, subresolver) in resolver.namespace_dict.items(): for route in extract_routes(subresolver): yield f"{key}:{route.name}" def is_django_admin_route(route_name: str): # Matches, e.g., `admin:emails_event_changelist`. return route_name.split(":").endswith("changelist") ADMIN_URL_ROUTE = "buttondown.urls.admin" DJANGO_ADMIN_CHANGELIST_ROUTES = [ route.name for route in extract_routes(get_resolver(ADMIN_URL_ROUTE)) if is_django_admin_route(route.name) ] # The fixture is overkill for this example, but I'm copying this from the actual codebase. @pytest.fixture def superuser_client(superuser: User, client: Client) -> Client: client.force_login(superuser) return client @pytest.mark.parametrize( "url", DJANGO_ADMIN_CHANGELIST_ROUTES ) def test_can_render_route(superuser_client: Any, url: str) -> None: url = reverse(url, args=[]) response = superuser_client.get(url) assert response.status_code == 200 Okay, a bit of a mouthful, but the final test itself is very clean and tidy and catches a lot of stuff. That extract_routes implementation looks scary and magical, and it is — I use a more robust implementation in django-typescript-routes, which itself we gratefully purloined from django-js-reverse. Lots of scary indexing, but its held up well for a while. The fixture and parametrize assumes usage of pytest (you should use pytest!) but it's trivially rewritable to use subTest instead.
When we added support for complex filtering in Buttondown, I spent a long time trying to come up with a schema for filters that felt sufficiently ergonomic and future-proof. I had a few constraints, all of which were reasonable: It needed to be JSON-serializable, and trivially parsable by both the front-end and back-end. It needed to be arbitrarily extendible across a number of domains (you could filter subscribers, but also you might want to filter emails or other models.) It needed to be able to handle both and and or logic (folks tagged foo and bar as well as folded tagged foo or bar). It needed to handle nested logic (folks tagged foo and folks tagged bar or baz.) The solution I landed upon is not, I’m sure, a novel one, but googling “recursive filter schema” was unsuccessful and I am really happy with the result so here it is in case you need something like this: @dataclass class FilterGroup: filters: list[Filter] groups: list[FilterGroup] predicate: "and" | "or" @dataclass class Filter: field: str operator: "less_than" | "greater_than" | "equals" | "not_equals" | "contains" | "not_contains" value: str And there you have it. Simple, easily serializable/type-safe, can handle everything you throw at it. For example, a filter for all folks younger than 18 or older than 60 and retired: FilterGroup( predicate="or", filters=[ Field( field="age", operator="less_than", value="18" ) ], groups=[ FilterGroup( predicate="and", filters=[ Field( field="age", operator="greater_than", value="60" ), Field( field="status", operator="equals", value="retired" ) ] groups=[], ) ] )
If there's been one through line in changes to Buttondown's architecture over the past six months or so, it's been the removal and consolidation of dependencies: on the front-end, back-end, and in paid services. I built our own very spartan version of Metabase, Notion, and Storybook; we vended a half-dozen or so Django packages that were not worth the overhead of pulling from PyPI (and rewrote another half-dozen or so, which we will open-source in due time); we ripped out c3, our visualization library, and built our own; we ripped out vuedraggable and a headlessui and a slew more of otherwise-underwhelming frontend packages in favor of purpose-built (faster, smaller, less-flexible) versions. [1] There are a few reasons for this: Both Buttondown as an application and I as a developer have now been around long enough to be scarred by big ecosystem changes. Python has gone through both the 2.x to 3.x transition and, more recently, the untyped to typed transition; Vue has gone from 2.x to 3.x. The academic problem of "what happens if this language completely changes?" is no longer academic, and packages that we installed back in 2018 slowly succumbed to bitrot. It's more obvious to me now than a few years ago that pulling in dependencies incurs a non-trivial learning cost for folks paratrooping into the codebase. A wrapper library around fetch might be marginally easier to invoke once you get used to it, but it's a meaningful bump in the learning curve to adapt to it for the first time. It is easier than ever to build 60% of a tool, which is problematic in many respects but useful if you know exactly which 60% you care about. (Internal tools like Storybook or Metabase are great examples of this. It was a fun and trivial exercise to get Claude to build a tool that did everything I wanted Metabase to do, and save me $120/mo in the process.) We still use a lot of very heavy, very complex stuff that we're very happy with. Our editor sits on top of tiptap (and therefore ProseMirror); we use marked and turndown liberally, because they're fast and robust. On the Python side, our number of non-infrastructural packages is smaller but still meaningful (beautifulsoup, for instance, and django-allauth / django-anymail which are both worth their weight in gold). But the bar for pulling in a small dependency is much higher than it was, say, twelve months ago. My current white whale is to finally get rid of axios. 39 call sites to go! ↩︎
After many wonderful years of working out of my home office (see Workspaces), I've now "expanded" [1] into an office of my own. 406 W Franklin St #201 is now the Richmond-area headquarters of Buttondown. Send me gifts! The move is a bittersweet one; it was a great joy to be so close to Haley and Lucy (and, of course, Telly), and the flexibility of being able to hop off a call and then take the dog for a walk or hold Lucy for a while was very, very nice. At the same time, for the first time in my life that flexibility has become a little bit of a burden! It turns out it is very hard to concentrate on responding to emails when your alternative is to play with your daughter giggling in the adjoining room; similarly, as Buttondown grows and as more and more of my time is spent on calls, it turns out long-winded demos and onboarding calls are logistically trickier when it is Nap Time a scant six feet away. And, beyond that, it's felt harder and harder to turn my brain off for the day: when there is always more work to be done, it's hard not to poke away at a stubborn pull request or jot down some strategy notes instead of being more present for my family (or even for myself, in a non-work capacity.) So, I leased an office. The space is pretty cool: it's downtown in the sweet spot of a little more than a mile away from the house: trivially walkable (or bikeable, as the above photo suggests) but far enough away to give me a good bit of mental space. The building is an old manor (turned dormitory, turned office building). I've got a bay window with plenty of light but no views; I've got a nice ethernet connection and a Mac Mini with very few things installed; I've got a big Ikea desk and a printer; I've got an alarm on my phone for 4:50pm, informing me that it's time to go home, where my world becomes once again lively and lovely, full of noise and joy and laughter. Air quote because I'm fairly confident this office is actually smaller than the home office. ↩︎
More in technology
WordPress 6.8 Cecil is out, and it’s a great release. It’s unbelievable that it’s already been downloaded over 6 million times as I write this. That feeling never gets old. It’s a funny time in WordPress because there are a lot of really interesting open questions: Some of these broad changes are mixed. At one … Continue reading 6.8 →
We’re very excited to share that the Arduino AI Assistant is now available in the Arduino Cloud Editor! This expert coding companion truly understands your project and board, and can generate and fix your code in seconds. We know that many of you already use other AI tools to assist with coding, but switching back […] The post Code faster with the new Arduino AI Assistant appeared first on Arduino Blog.
Securing My Web Infrastructure A few months ago, I very briefly mentioned that I've migrated all my web infrastructure off Cloudflare, as well as having built a custom web service to host it all. I call this new web service WebCentral and I'd like to talk about some of the steps I've taken and lessons I've learned about how I secure my infrastructure. Building a Threat Model Before you can work to secure any service, you need to understand what your threat model is. This sounds more complicated than it really is; all you must do is consider what your risks how, how likely those risks are to be realized, and what the potential damage or impact those risks could have. My websites don't store or process any user data, so I'm not terribly concerned about exfiltration, instead my primary risks are unauthorized access to the server, exploitation of my code, and denial of service. Although the risks of denial of service are self-explanatory, the primary risk I see needing to protect against is malicious code running on the machine. Malicious actors are always looking for places to run their cryptocurrency miners or spam botnets, and falling victim to that is simply out of the question. While I can do my best to try and ensure I'm writing secure code, there's always going to be the possibility that I or someone else makes a mistake that turns into an exploitable weakness. Therefore, my focus is on minimizing the potential impact should this occur. VPS Security The server that powers the very blog you're reading is a VPS, virtual private server, hosted by Azure. A VPS is just a fancy way to say a virtual machine that you have mostly total control over. A secure web service must start with a secure server hosting it, so let's go into detail about all the steps I take to keep the server safe. Network Security Minimizing the internet-facing exposure is critical for any VPS and can be one of the most effective ways to keep a machine safe. My rule is simple, no open ports other than what is required for user traffic. In effect this only means one thing: I cannot expose SSH to the internet. Doing so protects me against a wide range of threats and also reduces the impact from scanners (more on them later). While Azure itself offers several of ways to interact with a running VPS, I've chosen to disable most of those features and instead rely on my own. I personally need to be able to access the machine over SSH, however, so how do I do that if SSH is blocked? I use a VPN. On my home network is a WireGuard VPN server as well as a Dynamic DNS setup to work-around my rotating residential IP address. The VM will try to connect to the WireGuard VPN on my home network and establish a private tunnel between them. Since the VM is the one initiating the connection (acting as a client) no port must be exposed. With this configuration I can effortlessly access and manage the machine without needing to expose SSH to the internet. I'm also experimenting with, but have not yet fully rolled out, an outbound firewall. Outbound firewalls are far, far more difficult to set up than inbound because you must first have a very good understanding of what and where your machine talks to. OS-Level Security Although the internet footprint of my VPS is restricted to only HTTP and HTTPS, I still must face the risk of someone exploiting a vulnerability in my code. I've taken a few steps to help minimize the impact from a compromise to my web application's security. Automatic Updates First is some of the most basic things everyone should be doing, automatic updates & reboots. Every day I download and install any updates and restart the VPS if needed. All of this is trivially easy with a cron job and built-in tooling. I use this script that runs using a cron job: #!/bin/bash # Check for updates dnf check-update > /dev/null if [[ $? == 0 ]]; then # Nothing to update exit 0 fi # Install updates dnf -y update # Check if need to reboot dnf needs-restarting -r if [[ $? == 1 ]]; then reboot fi Low-Privileged Accounts Second, the actual process serving traffic does not run as root, instead it runs as a dedicated service user without a shell and without sudo permission. Doing this limits the abilities of what an attacker might be able to do, should they somehow have the ability to execute shell code on the machine. A challenge with using non-root users for web services is a specific security restriction enforced by Linux: only the root user can bind to port at or below 1024. Thankfully, however, SystemD services can be granted additional capabilities, one of which is the capability to bind to privileged ports. A single line in the service file is all it takes to overcome this challenge. Filesystem Isolation Lastly, the process also uses a virtualized root filesystem, a process known as chroot(). Chrooting is a method where the Linux kernel effectively lies to the process about where the root of the filesystem is by prepending a path to every call to access the filesystem. To chroot a process, you provide a directory that will act as the filesystem root for that process, meaning if the process were to try and list of contents of the root (/), they'd instead be listing the contents of the directory you specified. When configured properly, this has the effect of an filesystem allowlist - the process is only allowed to access data in the filesystem that you have specifically granted for it, and all of this without complicated permissions. It's important to note, however, that chrooting is often misunderstood as a more involved security control, because it's often incorrectly called a "jail" - referring to BSD's jails. Chrooting a process only isolates the filesystem from the process, but nothing else. In my specific use case it serves as an added layer of protection to guard against simple path transversal bugs. If an attacker were somehow able to trick the server into serving a sensitive file like /etc/passwd, it would fail because that file doesn't exist as far as the process knows. For those wondering, my SystemD service file looks like this: [Unit] Description=webcentral After=syslog.target After=network.target [Service] # I utilize systemd's heartbeat feature, sd-notify Type=notify NotifyAccess=main WatchdogSec=5 # This is the directory that serves as the virtual root for the process RootDirectory=/opt/webcentral/root # The working directory for the process, this is automatically mapped to the # virtual root so while the process sees this path, in actuality it would be # /opt/webcentral/root/opt/webcentral WorkingDirectory=/opt/webcentral # Additional directories to pass through to the process BindReadOnlyPaths=/etc/letsencrypt # Remember all of the paths here are being mapped to the virtual root ExecStart=/opt/webcentral/live/webcentral -d /opt/webcentral/data --production ExecReload=/bin/kill -USR2 "$MAINPID" TimeoutSec=5000 Restart=on-failure # The low-privilege service user to run the process as User=webcentral Group=webcentral # The additional capability to allow this process to bind to privileged ports CapabilityBoundingSet=CAP_NET_BIND_SERVICE [Install] WantedBy=default.target To quickly summarize: Remote Access (SSH) is blocked from the internet, a VPN must be used to access the VM, updates are automatically installed on the VM, the web process itself runs as a low-privileged service account, and the same process is chroot()-ed to shield the VMs filesystem. Service Availability Now it's time to shift focus away from the VPS to the application itself. One of, if not the, biggest benefits of running my own entire web server means that I can deeply integrate security controls how I best see fit. For this, I focus on detection and rejection of malicious clients. Being on the internet means you will be constantly exposed to malicious traffic - it's just a fact of life. The overwhelming majority of this traffic is just scanners, people going over every available IP address and looking widely known and exploitable vulnerabilities, things like leaving credentials out in the open or web shells. Generally, these scanners are one and done - you'll see a small handful of requests from a single address and then never again. I find that trying to block or prevent these scanners is a bit of a fool's errand, however by tracking these scanners over time I can begin to identify patterns to proactively block them early, saving resources. Why this matters is not because of the one-and-done scanners, but instead the malicious ones, the ones that don't just send a handful of requests - they send hundreds, if not thousands, all at once. These scanners risk degrading the service for others by occupying server resources that would better be used for legitimate visitors. To detect malicious hosts, I employ some basic heuristic by focusing on the headers sent by the client, and the paths they're trying to access. Banned Paths Having collected months of data from the traffic I served, I was able to identify some of the most common paths these scanners are looking for. One of the more common treds I see if scanning for weak and vulnerable WordPress configurations. WordPress is an incredibly common content management platform, which also makes it a prime target for attackers. Since I don't use WordPress (and perhaps you shouldn't either...) this made it a good candidate for scanner tracking. Therefore, any request where the path contains any of: "wp-admin", "wp-content", "wp-includes", or "xmlrpc.php" are flagged as malicious and recorded. User Agents The User Agent header is data sent by your web browser to the server that provides a vague description of the browser and the device it's running on. For example, my user agent when I wrote this post is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:128.0) Gecko/20100101 Firefox/128.0 All this really tells the server is that I'm on a Mac running macOS 15 and using Firefox 128. One of the most effective measures I've found to block malicious traffic early is to do some very basic filtering by user agent. The simplest and most effective measure thus far has been to block requests that have no user agent header. I also have a growing list of bogus user agent values, where the header looks valid - but if you check the version numbers of the system or browser, nothing lines up. IP Firewall When clients start getting a bit too rowdy, they get put into the naughty corner temporarily blocked from connecting. Blocked connections happen during the TCP handshake, saving resources as we skip the TLS negotiation. Addresses are blocked 24 hours, and I found this time to be perfectly adequate as most clients quickly give up and move on. ASN Blocks In some extreme situations, it's necessary to block entire services and all of their addresses from accessing my server. This happens when a network provider, such as an ISP, VPN, or cloud provider, fails to do their job in preventing abuse of their services and malicious find home there. Cloud providers have a responsibility to ensure that if a malicious customer is using their service, they would terminate their accounts and stop providing their services. For the most part, these cloud providers do a decent enough job at that. Some providers, however, don't care - at all - and quickly become popular amongst malicious actors. Cloudflare and Alibaba are two great examples. Because of the sheer volume of malicious traffic and total lack of valid user traffic, I block all of Cloudflare and Alibaba's address space. Specifically, I block AS13335 and AS45102. Putting It All Together Summarized, this is the path a request takes when connecting to my server: Upon recieving a TCP connection, the IP address of the client is checked if it's either in a blocked ASN or is individually blocked. If so, the request is quickly rejected. Otherwise, TLS is negotiated, allowing the server to see the details of the actual HTTP request. We then check if the request is for a banned path, or has a banned user agent, if so the IP is blocked for 24 hours and the request is rejected, otherwise the request is served as normal. The Result I feel this graph speaks for itself: This graph shows the number of requests that were blocked per minute. These bursts are the malicious scanners that I'm working to block, and all of these were successful defences against them. This will be a never-ending fight, but that's part of the fun, innit?
