Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Applied Cartography
One of the most useful and janky internal tools we have in Buttondown’s codebase is a codegen pipeline called “autogen”. There is nothing “auto” about autogen: it is a series of scripts that munges a bunch of data into a bunch of different formats, to generate things like our API clients and code snippets and storybooks. Some of this data is stateful, and therefore requires a database, and therefore requires migrations — you see how this kind of thing can grow somewhat labrynthine. Each individual script is pretty simple, but as we’ve found more and more things to glom onto autogen. This, to be clear, is a good thing. It’s really nice to have automatic, consistent data and types everywhere, so that we literally cannot change the API without also pushing a concomitant change to the API docs. With each glom, though, the wall-clock time of running autogen increases — and so I found myself staring down the barrel at a 50second script running whenever we wanted to make any sort of non-trivial change to our schema. Fifty seconds was too many seconds. I set a budget of ten seconds — still a long time, but significantly less onerous — and began digging in at low-hanging fruit. There was a lot. A few that come to mind: We split up our vite config so we could only run the portion that we needed (cross-piling and minifying our CSS bundles; We disabled all Sentry and perf-tracing stuff that was getting enabled as part of the standard build; We no-oped all of the Python-land data generation if it was already there, since that stateful data didn’t change very often. This was all great, but we were still left with 15 seconds of wall clock time. Profiling each individual cog in the script revealed that the problem was essentially “it’s Python”: four items in the script ran Django commands, and just spinning up the Django process and running autodiscovery took around two seconds. Ouch! The impulse was to cut down that runtime. A great post by Adam led us to discover the biggest culprit was our Stripe imports, and we timeboxed a bit of time to try and get rid of them, either by deferring the imports or excising the library; neither seemed particularly feasible. Then, suddenly, the answer seemed obvious. If we have four scripts where the fixed cost of invoking Django is the long pole, why not simply combine the scripts? And that’s exactly what we did: if len(sys.argv) > 1 and "," in sys.argv[1]: commands = sys.argv[1].split(",") original_argv = sys.argv.copy() for command in commands: sys.argv[1] = command execute_from_command_line(sys.argv) sys.argv = original_argv else: execute_from_command_line(sys.argv)
Here is a confession: I am a very strong proponent of a robust test suite being perhaps the single most important asset of a codebase, but when it comes to auxiliary services like admin sites or CLIs when it comes to testing I tend to ask for forgiveness more than I ask for permission. Django's admin site is no different: and, because Django's admin DSL is very magic-string-y, there's a lot of stuff that never gets caught by CI or mypy until a lovely CS agent informs me that something is blowing up in their face. Take this example, which bites me more often than I care to admit: from django.contrib import admin from stripe.models import StripeCustomer class StripeCustomer(models.Model): id = models.CharField(max_length=100, unique=True) username = models.CharField(max_length=100, unique=True) email_address = models.EmailField() creation_date = models.DateTimeField(auto_now=True) @admin.register(StripeCustomer) class StripeCustomerAdmin(admin.ModelAdmin): list_display = ( "id", "username", "email", "creation_date", ) search_fields = ( "username", "email", ) One thing that has made my life slightly easier in this respect is a parametric test that just makes sure we can render the empty state for every single admin view. Code snippet first, explanation after: from django.urls import get_resolver, reverse def extract_routes(resolver: URLResolver) -> iter[str]: keys = [key for key in resolver.reverse_dict.keys() if isinstance(key, str)] key_to_route = {key: resolver.reverse_dict.getlist(key)[0][0][0] for key in keys} for key in keys: yield key for key, (prefix, subresolver) in resolver.namespace_dict.items(): for route in extract_routes(subresolver): yield f"{key}:{route.name}" def is_django_admin_route(route_name: str): # Matches, e.g., `admin:emails_event_changelist`. return route_name.split(":").endswith("changelist") ADMIN_URL_ROUTE = "buttondown.urls.admin" DJANGO_ADMIN_CHANGELIST_ROUTES = [ route.name for route in extract_routes(get_resolver(ADMIN_URL_ROUTE)) if is_django_admin_route(route.name) ] # The fixture is overkill for this example, but I'm copying this from the actual codebase. @pytest.fixture def superuser_client(superuser: User, client: Client) -> Client: client.force_login(superuser) return client @pytest.mark.parametrize( "url", DJANGO_ADMIN_CHANGELIST_ROUTES ) def test_can_render_route(superuser_client: Any, url: str) -> None: url = reverse(url, args=[]) response = superuser_client.get(url) assert response.status_code == 200 Okay, a bit of a mouthful, but the final test itself is very clean and tidy and catches a lot of stuff. That extract_routes implementation looks scary and magical, and it is — I use a more robust implementation in django-typescript-routes, which itself we gratefully purloined from django-js-reverse. Lots of scary indexing, but its held up well for a while. The fixture and parametrize assumes usage of pytest (you should use pytest!) but it's trivially rewritable to use subTest instead.
When we added support for complex filtering in Buttondown, I spent a long time trying to come up with a schema for filters that felt sufficiently ergonomic and future-proof. I had a few constraints, all of which were reasonable: It needed to be JSON-serializable, and trivially parsable by both the front-end and back-end. It needed to be arbitrarily extendible across a number of domains (you could filter subscribers, but also you might want to filter emails or other models.) It needed to be able to handle both and and or logic (folks tagged foo and bar as well as folded tagged foo or bar). It needed to handle nested logic (folks tagged foo and folks tagged bar or baz.) The solution I landed upon is not, I’m sure, a novel one, but googling “recursive filter schema” was unsuccessful and I am really happy with the result so here it is in case you need something like this: @dataclass class FilterGroup: filters: list[Filter] groups: list[FilterGroup] predicate: "and" | "or" @dataclass class Filter: field: str operator: "less_than" | "greater_than" | "equals" | "not_equals" | "contains" | "not_contains" value: str And there you have it. Simple, easily serializable/type-safe, can handle everything you throw at it. For example, a filter for all folks younger than 18 or older than 60 and retired: FilterGroup( predicate="or", filters=[ Field( field="age", operator="less_than", value="18" ) ], groups=[ FilterGroup( predicate="and", filters=[ Field( field="age", operator="greater_than", value="60" ), Field( field="status", operator="equals", value="retired" ) ] groups=[], ) ] )
If there's been one through line in changes to Buttondown's architecture over the past six months or so, it's been the removal and consolidation of dependencies: on the front-end, back-end, and in paid services. I built our own very spartan version of Metabase, Notion, and Storybook; we vended a half-dozen or so Django packages that were not worth the overhead of pulling from PyPI (and rewrote another half-dozen or so, which we will open-source in due time); we ripped out c3, our visualization library, and built our own; we ripped out vuedraggable and a headlessui and a slew more of otherwise-underwhelming frontend packages in favor of purpose-built (faster, smaller, less-flexible) versions. [1] There are a few reasons for this: Both Buttondown as an application and I as a developer have now been around long enough to be scarred by big ecosystem changes. Python has gone through both the 2.x to 3.x transition and, more recently, the untyped to typed transition; Vue has gone from 2.x to 3.x. The academic problem of "what happens if this language completely changes?" is no longer academic, and packages that we installed back in 2018 slowly succumbed to bitrot. It's more obvious to me now than a few years ago that pulling in dependencies incurs a non-trivial learning cost for folks paratrooping into the codebase. A wrapper library around fetch might be marginally easier to invoke once you get used to it, but it's a meaningful bump in the learning curve to adapt to it for the first time. It is easier than ever to build 60% of a tool, which is problematic in many respects but useful if you know exactly which 60% you care about. (Internal tools like Storybook or Metabase are great examples of this. It was a fun and trivial exercise to get Claude to build a tool that did everything I wanted Metabase to do, and save me $120/mo in the process.) We still use a lot of very heavy, very complex stuff that we're very happy with. Our editor sits on top of tiptap (and therefore ProseMirror); we use marked and turndown liberally, because they're fast and robust. On the Python side, our number of non-infrastructural packages is smaller but still meaningful (beautifulsoup, for instance, and django-allauth / django-anymail which are both worth their weight in gold). But the bar for pulling in a small dependency is much higher than it was, say, twelve months ago. My current white whale is to finally get rid of axios. 39 call sites to go! ↩︎
More in technology
I uploaded YouTube videos from time to time, and a fun comment I often get is “Whoa, this is in 8K!”. Even better, I’ve had comments from the like, seven people with 8K TVs that the video looks awesome on their TV. And you guessed it, I don’t record my videos in 8K! I record them in 4K and upscale them to 8K after the fact. There’s no shortage of AI video upscaling tools today, but they’re of varying quality, and some are great but quite expensive. The legendary Finn Voorhees created a really cool too though, called fx-upscale, that smartly leverages Apple’s built-in MetalFX framework. For the unfamiliar, this library is an extensive of Apple’s Metal graphics library, and adds functionality similar to NVIDIA’s DLSS where it intelligently upscales video using machine learning (AI), so rather than just stretching an image, it uses a model to try to infer what the frame would look like at a higher resolution. It’s primarily geared toward video game use, but Finn’s library shows it does an excellent job for video too. I think this is a really killer utility, and use it for all my videos. I even have a license for Topaz Video AI, which arguably works better, but takes an order of magnitude longer. For instance my recent 38 minute, 4K video took about an hour to render to 8K via fx-upscale on my M1 Pro MacBook Pro, but would take over 24 hours with Topaz Video AI. # Install with homebrew brew install finnvoor/tools/fx-upscale # Outputs a file named my-video Upscaled.mov fx-upscale my-video.mov --width 7680 --codec h265 Anyway, just wanted to give a tip toward a really cool tool! Finn’s even got a [version in the Mac App Store called Unsqueeze](https://apps.apple.com/ca/app/unsqueeze/id6475134617 Unsqueeze) with an actual GUI that’s even easier to use, but I really like the command line version because you get a bit more control over the output. 8K is kinda overkill for most use cases, so to be clear you can go from like, 1080p to 4K as well if you’re so inclined. I just really like 8K for the future proofing of it all, in however many years when 8K TVs are more common I’ll be able to have some of my videos already able to take advantage of that. And it takes long enough to upscale that I’d be surprised to see TVs or YouTube offering that upscaling natively in a way that looks as good given the amount of compute required currently. Obviously very zoomed in to show the difference easier If you ask me, for indie creators, even when 8K displays are more common, the future of recording still probably won’t be in native 8K. 4K recording gives so much detail still that have more than enough details to allow AI to do a compelling upscale to 8K. I think for my next camera I’m going to aim for recording in 6K (so I can still reframe in post), and then continue to output the final result in 4K to be AI upscaled. I’m coming for you, Lumix S1ii.
Talks about the famous Dragon's Lair
totally unreasonable price for a completely untested item, as-was, no returns, with no power supply, no wiring harness and no auxiliary daughterboards. At the end of this article, we'll have it fully playable and wired up to a standard ATX power supply, a composite monitor and off-the-shelf Atari joysticks, and because this board was used for other related games from that era, the process should work with only minor changes on other contemporary Gremlin arcade classics like Blockade, Hustle and Comotion [sic]. It's time for a Refurb Weekend. a July 1982 San Diego Reader article, the locally famous alternative paper I always snitched a copy of when I was downtown, and of which I found a marginally better copy to make these scans. There's also an exceptional multipart history of Gremlin you can read but for now we'll just hit the highlights as they pertain to today's project. ported to V1 Unix and has a simpler three-digit variant Bagels which was even ported to the KIM-1. Unfortunately his friends didn't have minicomputers of their own, so Hauck painstakingly put together a complete re-creation from discrete logic so they could play too, later licensed to Milton Bradley as their COMP IV handheld. Hauck had also been experimenting with processor-controlled video games, developing a simple homebrew unit based around the then-new Intel 8080 CPU that could connect to his television set and play blackjack. Fogleman met Hauck by chance at a component vendor's office and hired him on to enhance the wall game line, but Hauck persisted in his experiments, and additionally presented Fogleman with a new and different machine: a two-player game played with buttons on a video TV display, where each player left a boxy solid trail in an attempt to crowd out the other. To run the fast action on its relatively slow ~2MHz CPU and small amount of RAM, a character generator circuit made from logic chips painted a 256x224 display from 32 8x8 tiles in ROM specified by a 32x28 screen matrix, allowing for more sophisticated shapes and relieving the processor of having to draw the screen itself. (Does this sound like an early 8-bit computer? Hold that thought.) patent application was too late and too slow to stop the ripoffs. (For the record, Atari programmer Dennis Koble was adamant he didn't steal the idea from Gremlin, saying he had seen similar "snake" games on CompuServe and ARPANET, but Nolan Bushnell nevertheless later offered Gremlin $100,000 in "consolation" which the company refused.) Meanwhile, Blockade orders evaporated and Gremlin's attempts to ramp up production couldn't save it, leaving the company with thousands of unused circuit boards, game cabinets and video monitors. While lawsuits against the copycats slowly lumbered forward, Hauck decided to reprogram the existing Blockade hardware to play new games, starting with converting the Comotion board into Hustle in 1977 where players could also nab targets for additional points. The company ensured they had a thousand units ready to ship before even announcing it and sales were enough to recoup at least some of the lost investment. Hauck subsequently created a reworked version of the board with the same CPU for the more advanced game Depthcharge, initially testing poorly with players until the controls were simplified. This game was licensed to Taito as Sub Hunter and the board reworked again for the target shooter Safari, also in 1977, and also licensed by Taito. For 1978, Gremlin made one last release using the Hustle-Comotion board. This game was Blasto. present world record is 8,730), but in two player mode the players can also shoot each other for an even bigger point award. This means two-player games rapidly turn into active hunts, with a smaller bonus awarded to a player as well if the other gets nailed by a mine. shown above with a screenshot of the interactive on-board assembler. Noval also produced an education-targeted system called the Telemath, based on the 760 hardware, which was briefly deployed in a few San Diego Unified elementary schools. Alas, they were long gone before we arrived. Industry observers were impressed by the specs and baffled by the desk. Although the base price of $2995 [about $16,300] was quite reasonable considering its capabilities, you couldn't buy it without its hulking enclosure, which made it a home computer only to the sort of people who would buy a home PDP-8. (Raises hand.) Later upgrades with a Z80 and a full 32K didn't make it any more attractive to buyers and Noval barely sold about a dozen. Some of the rest remained at Gremlin as development systems (since they practically were already), and an intact upgraded unit with aftermarket floppy drives lives at the Computer History Museum. The failure of Noval didn't kill Gremlin outright, but Fogleman was concerned the company lacked sufficient capital to compete more strongly in the rapidly expanding video game market, and Noval didn't provide it. With wall game sales fading fast and cash flow crunched, the company was slowly approaching bankruptcy by the time Blasto hit arcades. At the same time, Sega Enterprises, Inc., then owned by conglomerate Gulf + Western (who also then owned Paramount Pictures), was looking for a quick way to revive its failing North American division which was only surviving on the strength of its aggressively promoted mall arcades. Sega needed development resources to bring out new games States-side, and Gremlin needed money. In September 1978 Fogleman agreed to make Gremlin a Sega subsidiary in return for an undisclosed number of shares, and became a vice chairman. Sega was willing to do just about anything to achieve supremacy on this side of the Pacific. In addition to infusing cash into Gremlin to make new games (as Gremlin/Sega) and distribute others from their Japanese peers and partners (as Sega/Gremlin), Sega also perceived a market opportunity in licensing arcade ports to the growing home computer segment. Texas Instruments' 99/4 had just hit the market in 1979 to howls there was hardly any software, and their close partner Milton Bradley was looking for marketable concepts for cartridge games. Blasto had simple fast action and a good name in the arcades, required only character graphics (well within the 9918 video chip's capabilities) and worked for both one or two players, and Sega had no problem blessing a home port of an older property for cheap. Milton Bradley picked up the license to Hustle as well. Bob Harris for completion, and TI house programmer Kevin Kenney wrote some additional features. 1 to 40 (obviously some thought was given to using the same PCB as much as possible). The power header is also a 10-pin block and the audio and video headers are 4-pin. Oddly, the manual doesn't say anywhere what the measurements are, so I checked them with calipers and got a pitch of around 0.15", which sounds very much like a common 0.156" header. I ordered a small pack of those as an experiment. 0002 because of the control changes: if you have an 814-0001, then you have a prototype. The MAME driver makes reference to an Amutech Mine Sweeper which is a direct and compatible ripoff of this board — despite the game type, it's not based on Depthcharge.) listed with the part numbers for the cocktail, but the ROM contents expected in the hashes actually correspond to the upright. Bipolar ROMs and PROMs are, as the name suggests, built with NPN bipolar junction transistors instead of today's far more common MOSFETs ("MOS transistors"). This makes them lower density but also faster: these particular bipolar PROMs have access times of 55-60ns as opposed to EPROMs or flash ROMs of similar capacity which may be multiple times slower depending on the chip and process. For many applications this doesn't matter much, but in some tightly-timed systems the speed difference can make it difficult to replace bipolar PROMs with more convenient EPROMs, and most modern-day chip programmers can't generate the higher voltage needed to program them (you're basically blowing a whole bunch of microscopic Nichrome metal fuses). Although modern CMOS PROMs are available at comparable speeds, bipolars were once very common, including in military environments where they could be manufactured to tolerate unusually harsh operating conditions. The incomparable Ken Shirriff has a die photo and article on the MMI 5300, an open-collector chip which is one of the military-spec parts from this line. Model 745 KSR and bubble memory Model 763 ASR, use AMD 8080s! The Intel 8080A is a refined version of the original Intel 8080 that works properly with more standard TTL devices (the original could only handle low-power TTL); the "NL" tag is TI's designation for a plastic regular-duty DIP. Its clock source is a 20.79MHz crystal at Y1 which is divided down by ten to yield the nominal clock rate of 2.079MHz, slightly above its maximum rating of 2MHz but stable enough at that speed. The later Intel 8080A-1 could be clocked up to 3.125MHz and of course the successor Intel 8085 and Zilog Z80 processors could run faster still. An interesting absence on this board is an Intel 8224 or equivalent to generate the 8080A's two-phase clock: that's done directly off the crystal oscillator with discrete logic, an elegant (and likely cheaper) design by Hauck. The video output also uses the same crystal. Next to the CPU are pads for the RAM chips. You saw six of them in the last picture under the second character ROM (316-0100M), all 2102 (1Kbit) static RAM. These were the chips I was most expecting to fail, having seen bad SRAM in other systems like my KIM-1. The ones here are 450ns Fairchild 21021 SRAMs in the 21021PC plastic case and "commercial" temperature range, and six of them adds up to 768 bytes of memory. NOS examples and equivalents are fortunately not difficult to find. Closer to the CPU in this picture, however, are two more RAM chip pads that are empty except for tiny factory-installed jumpers. On the Hustle and Blasto boards (both), they remain otherwise unpopulated, and there is an additional jumper between E4 and E5 also visible in the last picture. The Comotion board, however, has an additional 256 bytes of RAM here (as two more 1024x1 SRAMs). On that board these pads have RAM, there are no jumpers on the pads, and the jumper is now between E3 (ground) and E5. This jumper is also on Blockade, even though it has only five 2102s and three dummy jumpers on the other pads. That said, the games don't seem to care how much RAM is present as long as the minimum is: the current MAME driver gives all of them the full 1K. this 8080 system which uses a regulator). Tracing the schematic out further, the -12V line is also used with the +5V and +12V lines to run the video circuit. These are all part of the 10-pin power header. almost this exact sequence of voltages? An AT power supply connector! If we're clever about how we put the two halves on, we can get nearly the right lines in the right places. The six-pin AT P9 connector reversed is +5V, +5V, +5V, -5V, ground, ground, so we can cut the -5V to be the key. The six-pin AT P8 connector not reversed is power-good, +5V (or NC), +12V, -12V, ground, ground, so we cut the +5V to be the key, and cut the power-good line and one of the dangling grounds and wire ground to the power-good pin. Fortunately I had a couple spare AT-to-ATX converter cables from when we redid the AT power supply on the Alpha Micro Eagle 300. connectors since we're going to modify them anyway. A quick couple drops of light-cured cyanoacrylate into the key hole ... Something's alive! An LED glows! Time now for the video connector to see if we can get a picture! a nice 6502 reset circuit). The board does have its own reset circuit, of a sort. You'll notice here that the coin start is wired to the same line, and the manual even makes reference to this ("The circuitry in this game has been arranged so that the insertion of a quarter through the coin mechanism will reset the restart [sic] in the system. This clears up temporary problems caused by power line disturbances, static, etc."). We'll of course be dealing with the coin mechanism a little later, but that doesn't solve the problem of bringing the machine into the attract mode when powered on. I also have doubts that people would have blithely put coins into a machine that was obviously on the fritz. pair is up and down, or left and right, but not which one is exactly which because that depends on the joystick construction. We'll come back to this. Enterprises) to emphasize the brand name more strongly. The company entered a rapid decline with the video game crash of 1983 and the manufacturing assets were sold to Bally Midway with certain publishing rights, but the original Gremlin IP and game development teams stayed with Sega Electronics and remained part of Gulf+Western until they were disbanded. The brand is still retained as part of CBS Media Ventures today though modern Paramount Global doesn't currently use the label for its original purpose. In 1987 the old wall game line was briefly reincarnated under license, also called Gremlin Industries and with some former Gremlin employees, but only released a small number of new machines before folding. Meanwhile, Sega Enterprises separated from Gulf+Western in a 1984 management buyout by original founder David Rosen, Japanese executive Hayao Nakayama and their backers. This Sega is what people consider Sega today, now part of Sega Sammy Holdings, and the rights to the original Gremlin games — including Blasto — are under it. Lane Hauck's last recorded game at Gremlin/Sega was the classic Carnival in 1980 (I played this first on the Intellivision). After leaving the company, he held positions at various companies including San Diego-based projector manufacturer Proxima (notoriously later merging with InFocus), Cypress Semiconductor and its AgigA Tech subsidiary (both now part of Infineon), and Maxim Integrated Products (now part of Analog Devices), and works as a consultant today. I'm not done with Blasto. While I still enjoy playing the TI-99/4A port, there are ... improvements to be made, particularly the fact it's single fire, and it was never ported to anything else. I have ideas, I've been working on it off and on for a year or so and all the main gameplay code is written, so I just have to finish the graphics and music. You'll get to play it. And the arcade board? Well, we have a working game and a working harness that I can build off. I need a better sound amplifier, the "boom" circuit deserves a proper subwoofer, and I should fake up a little circuit using the power-good line from the ATX power supply to substitute for the power interrupt board. Most of all, though, we really need to get it a proper display and cabinet. That's naturally going to need a budget rather larger than my typical projects and I'm already saving up for it. Suggestions for a nice upright cab with display, buttons and joysticks that I can rewire — and afford! — are solicited. On both those counts, to be continued.
Hard data is hard to find, but roughly 100 million books were published prior to the 21st century. Of those, a significant portion were never available in a digital format and haven’t yet been digitized, which means their content is effectively inaccessible to most people today. To bring that content into the digital world, Redditor […] The post This machine automatically scans books from cover to cover appeared first on Arduino Blog.
