More from Sebastian Witowski
For a long time, the two most popular articles on my blog were lists of macOS apps and CLI tools, which I've been using. Coincidentally, these kinds of lists are something I love to read myself. I can always find some cool, new tool in such articles. It doesn't matter that I will probably immediately forget about this tool and never use it again. Nor does it matter that 80% of the items on those lists are always the same. The promise of being able to do some obscure task in a slightly more efficient way is all I need to get excited each time I find another "Best 15 MacBook apps for Python developers" article. However, those lists tend to get outdated pretty quickly. Every year or two, a new process manager, disk space analyzer, file manager, or other tool gets released that's faster, fancier, and more colorful. So, after almost five years, it's about time I updated my original macOS apps article and wrote down which tools I use now. I still use quite a lot of programs listed in the original article. For some, I found better replacements. And, of course, there are a few interesting new items. Some tools are paid (I marked those with a 💰 icon next to their name), and some have both a free tier and a paid one (I marked those with 🆓/💰). All the other ones are free to use. And, obviously, there are no affiliate links, and no one paid me to have their tool listed here - I use them because I like them. Table of contents # This is a long list, so here is a table of contents with short summaries of what each app does: Alfred - Spotlight on steroids Raycast - A more capable and modern alternative to Alfred Bartender 5 - An app for showing and hiding icons on the menu bar iTerm2 - A terminal app BetterTouchTool - An app for customizing all sorts of input devices for your Mac Karabiner Elements - An app for customizing your keyboard Magnet - A windows manager Monosnap -An app for taking screenshots and annotating them on the fly Kap - An app for recording videos of your screen QSpace Pro - A file manager Obsidian - A note-taking app for editing Markdown files Carbon Copy Cloner - An app for making computer backups Ferdium - A chat aggregator Todoist - An app for managing TODO lists Toggl - An app for tracking time when working for clients Velja - A browser picker Spark - An email client Figma - My preferred graphic editing tool 1Password - A password manager Nucleo app - An app for managing icons n8n - An automation tool (free alternative to Zapier) PDFGear - An app for editing PDF files The Unarchiver - An app for opening archived files OBS Studio - A video recording tool KeyCastr - A keystroke visualizer for showing which keys you press when recording screencasts Audacity - An audio editing tool HandBrake - An app for transcoding videos Captin - A tool that shows a popup when you toggle Caps Lock Balance Lock - An app for fixing the balance of the headphones MOS - An app for fixing smooth scrolling on a Logitech mouse noTunes - An app for preventing Apple Music from opening Other applications Apps I like but don't use anymore Apps I want to try Alfred 🆓/💰 # It's like Spotlight, but on steroids. I launch it with ⌘+Space, and then I can type some letters or numbers to quickly start applications, solve simple calculations, convert currencies, find files, and more. To make the most out of Alfred, you need to buy the Powerpack. It's a paid extension that allows you to use even more features, including the most important ones - workflows. Written mainly by the community, workflows are plugins that add additional functionality. Here are some of the workflows that I like the most: Calculate Anything - one of the workflows that I use the most. It allows me to calculate many more things than just some mathematical operations. It can do currency or unit conversion, operations on dates, and more. And it does a good job with processing natural language, so no matter if I write "100 euro to pln" or "100eur pln", I get the correct answer. Calm notifications - toggles the "Do Not Disturb" mode when I don't want to be distracted by any notifications. Coffee coffee - when enabled, it prevents your computer from going to sleep. I was previously using a free app called Amphetamine that does the same but with some additional customization options. And then I realized I didn't need any of those additional options. All I needed was to occasionally prevent my computer from sleeping. Fun fact - by looking at the source code of this workflow, I've noticed that it's calling the caffeinate command. It turns out that macOS has a built-in tool that can prevent your computer from sleeping. So, you don't even need any external tools. Check out caffeinate's manpage. DeepL Translate - for quick translations using DeepL. I don't use it for long translations, but mostly when I'm looking for synonyms of an English word. Emoji Mate - an emoji finder. I can type ; followed by the name or a description of an emoji, find an emoji that matches it, and copy it to the clipboard. UTF Search - a UTF characters finder (similar to Emoji mate mentioned above). I can type : followed by the name of a UTF character and then copy it to the clipboard. Gist - I use it to quickly create GitHub gists from the content of my clipboard. If I have some code copied in the clipboard, I can run the gist command, and this creates a GitHub gist that I can share with someone. I used this feature a lot when I was publishing on Medium.com, and I needed to create gists for my code, as Medium didn't have good support for code snippets. Network - show and change network settings. I use it to quickly get my IP address. Reminders for Alfred - add new items to the Reminders app. This is a very old plugin that is no longer available in Alfred's plugins gallery, so you have to install it from GitHub. And even though the author says that it's no longer actively maintained, it still works. I like this plugin because it's very good at parsing natural language. I can type "r 5 minutes leave home" or "r 10:50 leave home", and both commands will create a "leave home" reminder at a specific time. Temporary email - I can type tmpmail, and this workflow will create a temporary email with one of the various temporary email services. It's indispensable for all those "subscribe to our newsletter to get 10% off for your order" online shops, where I just want to shop once and never come back. TOD Todoist - lets me quickly add tasks to the Todoist app. I use this workflow very often, especially in the middle of a call, to add new tasks before I forget about them. So it's critical for me to have a plugin that can easily parse natural language, just like Todoist does (writing "buy milk tod p1" creates a "buy milk" task, sets the priority to 1, and the due date to today). When the workflow I was using previously (Alfred Workflow Todoist) stopped working, and the only replacement I found (AlfreDo) required me to learn some unintuitive syntax for adding tasks, I briefly started using a different launcher just to have my simple workflow for adding new tasks back. Luckily, later, I found "TOD Todoist" and got my simple way of adding TODO tasks back. alfred-toggl - I use Toggl to track the time when I'm working, and "alfred-toggl" allows me to interact with Toggl. I can start and stop timers or generate reports showing how long I worked today or this week. Raycast 🆓/💰 # Raycast is another launcher like Alfred, but it's much more actively developed and already much more capable than Alfred. On top of its vast catalog of features, there are many community plugins to further extend its functionality. I installed Raycast to quickly create new TODO tasks when my old Todoist plugin in Alfred stopped working. Nowadays, I use Raycast mostly for its Emoji Picker and Floating Notes features. Raycast looks very promising, and most importantly, it has a generous free plan. So, where's the catch, and why am I still using Alfred? Because Raycast is a VC-backed startup, and I'm not really convinced that their business model is sustainable. There is a paid Raycast Pro version that initially included AI features[1]. Now I see that the paid plan offers more features at the expense of the free plan (unlimited clipboard history, etc.) For now, the free version is fantastic. But I don't want to get vendor-locked when the money becomes tight and the free plan gets even more limited. Plus, I already paid for Alfred's Powerpack license, and I got used to using it. Alfred works fine for me, even though it's much less actively developed, and I wish it had more integrations with other tools (or at least stopped breaking community workflows when they release new versions). Bartender 5 💰 # Bartender lets you hide icons on the menu bar, so you can keep only the most important ones visible, while the rest can be displayed when you click Bartender's icon. Here's how my menu bar looks most of the time: And here's how it would look without the Bartender app: I bought Bartender to replace Dozer when I got a new MacBook with a notch because Dozer didn't work well with the notch - some icons were hiding under the notch, and I had no way to click them. Dozer has not been updated for quite some time, so support for MacBooks with a notch was never implemented. However, if I were looking for an app to hide menu bar icons today, I would first check out the free Ice app. Why? Because the owner of Bartender sold the app and never informed the users, which was a shady move. After the app was sold, it started including new features like a new analytics framework because, well, why shouldn't a simple app for hiding icons spy on its users, right? I hope by the time I write the next edition of a list of my tools, I will switch to something else, but for now, I didn't have the time to switch. iTerm2 # This is still my terminal app. It works great, stores its configuration nicely in my Dropbox backup folder, and I'm not planning to change it to anything else any time soon. New terminal apps come and go (warp looked promising until it became an "AI-powered intelligent terminal", took money from VC funds, and started collecting telemetry data by default), but for me, iTerm is perfect the way it is. BetterTouchTool 💰 # A feature-packed app for customizing all kinds of inputs for your computer. You can define keyboard shortcuts, mouse and trackpad gestures, control Stream Deck, manage your computer from your phone using BTT Remote, customize the content of the Touch Bar, and more. I initially bought it many years ago to customize the Touch Bar (as I described in the previous version of this article), and nowadays, I still use it to define some more advanced keyboard shortcuts. I have shortcuts that will mute and unmute my microphone (no matter if I'm using Teams or Zoom), pause and play music, start some of the most common apps, etc. I describe additional keyboard shortcuts in more detail in the You Don't Need Stream Deck, You Need Macros article. Karabiner Elements # A powerful and stable keyboard customizer for macOS, as its tagline says. And boy, it is powerful indeed. Karabiner Elements covers all my keyboard customization needs (although, for convenience, I still use BetterTouchTool for some more advanced keyboard shortcuts): Remapping keys Changing the behavior of function keys on specific keyboards (I want them to act as function keys on an external keyboard and, at the same time, act as media keys for MacBook's built-in keyboard) Creating complex mappings (pressing both Shift keys to toggle Caps Lock) Another great thing about Karabiner Elements is that it stores the configuration files as JSON. So I can save them in a git repository and have a well-documented history of all the changes I made. Magnet 💰 # A tool for managing windows with keyboard shortcuts. It allows me to move windows between monitors or split them into halves, quarters, or even sixths without having to drag them with the mouse. I paid for this tool only because I didn't do my research, and I didn't check if there were free apps that could do the same. But luckily, there are, and if I were looking for a windows manager for macOS today, I would use Rectangle instead. If you're looking for a more advanced tiling windows manager similar to i3 or xmonad, there is also Amethyst. Monosnap 🆓/💰 # I use Monosnap to take screenshots, crop, and annotate them on the fly. Then, I can send those screenshots to someone by simply dragging them into a chat app or an email. Before Monosnap, I used Skitch, but it seems Skitch is now abandoned. Even though Monosnap has a paid plan, the 2GB of storage in the free plan is generous enough that I don't think I'll ever become a paying customer[2]. Monosnap also supports recording videos of your screen, but I got used to using another tool for this. Kap # A simple and free tool for recording videos of my screen. The video can then be trimmed in Kap and exported to one of many supported formats (including MP4 and GIF). It's a great tool for when I want to share something happening on my screen. Even though the aforementioned Monosnap also supports recording videos, I use Kap out of habit. But if you're already using Monosnap to record videos, I don't think Kap has enough features to justify the change. QSpace Pro 💰 # For a long time, I have been looking for a file manager that supports multiple panes (which surprisingly is still not supported in macOS' default file manager - Finder). And I randomly found QSpace Pro, which works pretty well. It's a paid app, but it's cheaper than Forklift (another dual-pane file manager that some people recommended), and it doesn't require a subscription like the Path Finder (seriously, a subscription model for a friggin file manager?!). It offers plenty of keyboard shortcuts, and with plugins, I can connect to external services like Dropbox or Google Drive. Obsidian 🆓/💰 # My go-to note-taking app. I use it for all my note-taking needs: writing articles, creating mind maps of my presentations, planning my year, etc. I switched from Evernote to Obsidian long ago, even before Evernote became practically useless with the free plan. Obsidian uses Markdown files stored locally on my computer, so I don't have to worry about losing access to my notes or having to go through a difficult migration process if I ever need to switch to a different tool. I wrote a detailed article on how I use Obsidian: How I Organize My Notes With Obsidian. Even though Obsidian comes with paid plugins, I don't use any of them. Carbon Copy Cloner 💰 # This is my preferred tool for backups. I don't know if I did something wrong, but when I tried to use Time Machine with a non-Apple NAS, I was constantly running into issues. The backups were getting corrupted, and I was losing access to the old versions of the files. Carbon Copy Cloner solved all my problems with backups. I use it to back up specific folders during the night. The main selling point of making backups using Time Machine was that I could, in theory, set up a new computer from this backup[3]. But I know I won't need that. If I ever lose/break my computer, I will get a new one and install applications manually. Then, I will restore specific folders (those that Carbon Copy Cloner backs up for me every night). Once, I tried to set up a new MacBook from an old one using the Migration Assistant, but it copied so much unnecessary old trash that eventually, I ended up reinstalling everything from scratch. Ferdium # A chat aggregator. I use Telegram, WhatsApp, and Discord to communicate with people. And in the past, I used even more applications, including multiple Slack channels. Ferdium allows me to have all those different chat services in one app instead of starting a separate app for each one. What I don't like about Ferdium is that every service I add starts a separate electron application. With many chats, RAM usage goes through the roof. But I didn't find a workaround for this problem (except for not using so many chats). Regardless of this issue, Ferdium is still the best app I've used so far (after trying Franz, Rambox, One Chat, and - now discontinued - Ferdi). Todoist 🆓/💰 # I use Todoist to manage my TODO tasks, and I really like it. The free plan is still perfectly suitable for me because I created a lot of projects before the number of projects in the free plan was severely limited. Now, I can rename and reuse old projects without having to create new ones. If I ever run out of projects or need some paid features, like reminders, I will definitely pay for the paid plan instead of looking for a replacement, because Todoist is a very solid app. Toggl 🆓/💰 # A time-tracking app. Ever since I started freelancing, I've been using Toggl to create time sheets for my clients. The free plan is all I need from the app. I love the feature of stopping the timer after a predefined period of inactivity, so I never have to worry when I get distracted and leave my computer for longer without stopping the timer. Velja # It's a browser picker application. It lets you define rules and open links from specific apps or specific URLs in different browsers, not just the default one. I need it because I use the same computer for client and personal work and I want all the client-related links (JIRA, GitLab, etc.) to open in my "work" browser (which is currently Brave, because I need to be able to run Teams in the browser and Firefox was having some problems with that[4]). Spark 🆓/💰 # I think I'm slowly running out of built-in macOS apps that I didn't replace with something else... Before macOS Ventura, Apple Mail was missing one feature critically important to me - "send later". That's why I looked for an alternative, and I found Spark. I'm still using the old, "Classic" version of Spark, not the new one that is needlessly packed with AI features. This email client also solved some other small annoyances. For example, Apple Mail was occasionally messing up my flags - I used different flag colors for different emails, and sometimes it would randomly change the colors of the flags or put emails that used different flags under one flag. Pinned emails (that appear at the top of the inbox), "undo send", or "snooze" were all additional quality-of-life improvements that Apple Mail didn't have at that time. So I stayed with Spark mail until now. However, you should be aware of some privacy concerns regarding Spark. For example, it stores email credentials on its servers so that it can send emails later, even when your laptop is turned off. Seeing that Apple Mail now has most of the features I was missing before, I might give it another go one day. Figma 🆓/💰 # Even though Figma is considered primarily a prototyping tool, I use it for all my simple graphic editing needs. It's easy to use for simple tasks (adding text to an image, creating patterns from a few icons, etc.), and unlike GIMP, which I used before, I don't have to search for tutorials explaining how to do all those simple things. Don't get me wrong, GIMP is awesome, but I don't use it often enough to learn how to use it properly. And Figma is much more intuitive for casual usage. 1Password 💰 # My password manager since I switched away from LastPass a few years ago. Why am I not using the password manager built into macOS? Because my wife is not using Apple devices, and we share some passwords and important documents. There are some good alternatives to 1Password, like Bitwarden, that a lot of people recommend, but so far, I'm a happy 1Password's user. Its UX is very pleasant. I especially like that I can use it to store and fill in OTP tokens. Each year when my subscription is supposed to renew, I'm thinking about switching to something else, but I always end up extending my subscription. When it comes to something as important as a password manager, I just don't think it's worth replacing a tool that works with a completely new one just to save a few bucks. Nucleo app # I use Nucleo app to manage the icons I have saved on my computer (I have a library of over 10,000 SVG icons that I downloaded when I had a subscription at one of the premium icon services). Nucleo itself is quite an expensive icon bundle, but their app is free to use and works very well for finding a specific icon in a large pool of locally stored icons. Nucleo app looks very similar to the Iconset app I used in the past. But I moved away from using Iconset when it became a paid app. n8n 🆓/💰 # n8n is an automation tool - a fair-code distribution alternative to Zapier. Technically, this is not a macOS app, but it's a very useful app that you can install on your local computer or on an external server with a Docker image. I use it for simple things (like deploying this blog every morning to publish scheduled posts) and also some more advanced ones (like managing subscription-based access to a side project followed by sending Telegram notifications and emails that I described here). Currently, n8n is being developed by a company funded with VC money, and that rarely ends up well for free-to-use projects. If things go south, I'll switch to Huginn for my automations. PDFGear # Occasionally, I need to edit a PDF file. And I don't mean simply annotating it (which I can do with the built-in Preview app) but actually changing the text written in a PDF. Instead of paying Adobe a monthly fee for this feature, I found PDFGear - a free PDF editor with plenty of advanced features that fulfills all my occasional needs when working with PDF files. The Unarchiver # A free-to-use application for opening any type of archive on a Mac. OBS Studio # If you're a video streamer, this tool needs no introduction. OBS Studio is probably the best open-source recording and streaming software. I used it on a few occasions when I needed to simultaneously record a video from multiple input sources (screen-sharing plus camera). The most interesting use I found for it was when we were doing SCRUM Poker to estimate sprint tickets. Instead of waving my phone in front of the camera like a caveman, I connected a text document to my video feed. Whenever I typed a number, this number appeared over my head. It was a pretty stupid way to spend the CPU power (using OBS increases the CPU usage), but it was totally worth it! KeyCastr # A keystroke visualizer. I use it when I'm recording a screencast, and I want to display the keys I'm pressing. A very nice tool with plenty of customization options. Audacity # Every once in a while, I need to do some audio processing (crop an audio recording, remove background noise, or just convert a recording to another format), and Audacity is still the best tool for working with audio. At first glance, it might look overwhelming, but because it's been around for a long time, the internet is full of guides on how to do most of the basic processing. HandBrake # An open-source tool that can convert a video between different formats while performing some additional processing. I use it to perform resizing and removing audio from videos while converting them to a more space-efficient format. Install-and-forget apps # There are a few small applications that I installed one day, allowed them to run on startup, and basically forgot about them. They do their job perfectly while running in the background. Captin # A very simple tool that shows the Caps Lock status. I use a quite quirky way of enabling Caps Lock (pressing both Shift keys at the same time toggles the Caps Lock on and off), so I installed Captin to get a popup confirmation that Caps Lock has been successfully enabled or disabled. Balance Lock # If your headphones randomly change the balance settings (so the sound is audible more in one ear than in the other) - which is something that macOS is guilty of doing, Balance Lock is the app you need. It will automatically center the balance of the audio back each time macOS messes it up. MOS # I switched back from using a trackpad to a mouse, only to realize that Logitech's LogiOption+ app sucks. It turned out that to get a smooth scrolling experience for a Logitech mouse, I needed a 3rd party app. Luckily, I found a free app called MOS that solved my problems. Another useful feature it has is "per-application mouse scroll setting" that I use for the VPN application called Citrix when remotely connecting to my work PC (the default scroll settings caused scrolling in Citrix to be very janky, so I had to adjust them). noTunes # Another small utility to fix Apple's stupidity (oh, it rhymes!). This tool prevents Apple Music from launching when I press "play" on my AirPods if the Spotify app is not running. Because, you know, I have never EVER used Apple Music, but it totally makes sense for macOS to think that today is the day I finally want to play music from this app. Yes, macOS. Today is definitely the day when we play something from my impressive iTunes library of zero songs. Other, less thrilling applications # Some other, more well-known (but also more mundane) apps that I use include: Elgato's Stream Deck software - because I need to control my Stream Deck somehow. I'm considering switching to BetterTouchTool to control my Stream Deck, though. Inkscape - for when I need to edit an SVG file. I haven't edited an SVG file in ages, but if I were to do this today, I would probably try Figma first. f.lux - adapts the brightness of my screen depending on the time. Early in the day or late at night, it will make the colors warmer to make it less taxing on my eyes. TeamViewer - for those occasional times when I need to help my parents with something on their computer. Spotify 💰 - for listening to music. Visual Studio Code - my code editor. Everyone knows what VS Code is. VLC - one of the most popular video players for any operating system. I've been using it on every computer I have owned for as long as I can remember, and I never had any issues. But I've heard good things about Infuse, and I already use it on my iPhone, so maybe one day I'll switch to this app on my MacBook, too. Apps I like but don't use anymore # I've stopped using some of the apps listed in the previous version of this article. Even though I still think they are great, I simply don't have use for them right now: Anki 🆓/💰 - in my opinion, Anki is still the best app for creating and reviewing flash cards. I used it almost every day when I was learning Japanese a few years ago, and I loved it, even though the design looked crude and a bit outdated. The iOS app is paid, but the Android or web versions are free. I stopped using Anki because I'm currently not learning any new language. Amphetamine - an application that prevents your computer from going to sleep. I stopped using it because, in a rare instance, when I need to prevent my MacBook from sleeping, I use Alfred's "coffee coffee" workflow. It uses the built-in caffeinate script to prevent my computer from sleeping. Apps I want to try # Finally, there are a few applications that look very interesting, but I haven't found a use case for them yet. I'm making a note about them, and hopefully, one day, I'll give them a try: Orbstack - seems to be a nice and supposedly faster alternative to Docker Desktop. However, I currently don't use Docker for any of my projects. MeetingBar - a simple app that sits in your menu bar and reminds you of upcoming meetings. You can also join the meeting directly from this tool. It looks very cool, but most of my daily meetings happen in Teams, which I have to use through the browser or a VPN. If I ever get back to a standalone Teams client or start using different videoconferencing tools, I'll give the MeetingBar app a try. Bruno - looks like a pretty cool alternative to Postman for testing APIs. Thanks for reading and see you in the next edition of this list (hopefully sooner than in five years)! If you have suggestions regarding other awesome macOS apps I should try, or if you have your own list of favorite tools, please share them in the comments (or email me using the contact form). And if you're interested in the CLI tools I'm using, I'll soon publish an updated version of my old list of CLI tools. Because that's what everyone needs in a launcher - AI! ↩︎ I actually don't use the cloud storage for screenshots. I annotate and send them without storing them in the cloud. The only time I had to use cloud storage was when I couldn't share the screenshot with someone directly, and I needed to store it somewhere online for a couple of days. ↩︎ Of course, as long as the data doesn't get corrupted in the meantime. ↩︎ Let me tell you a funny story about the most popular videoconferencing application in the world. Many large organizations use it, so obviously, it needs extensive admin-level settings that allow security teams to prevent users from using the app on unauthorized operating systems or through unauthorized browsers. For example, some companies might say that using Teams on macOS is forbidden because Windows is the only "blessed" operating system where you can use this app. If you try to open Teams in your browser on macOS, you will get an error message saying that your administrator prevented you from using Teams on this operating system. But how does the most widely used videoconferencing tool check what operating system you're using? Surely, it has to be something so advanced that the users wouldn't be able to easily tamper with it, right? Definitely, it wouldn't be just checking your browser's user agent of your browser because even a ten-year-old can change the user agent with one of the many free-to-use plugins. Right, Microsoft? You wouldn't be just checking the browser's user agent, would you? Would you?! ↩︎
I like makefiles. I first used a makefile more than ten years ago. Even back then, it looked like some ancient technology used by the graybeard Linux wizards. Years passed, and new build tools came and went, but I kept seeing makefiles still used here and there. I got used to them because they were part of some projects I joined. At some point, I started to like them. Today, they are often the first automation tool I use when I start a new project. The reason I like makefiles is that they often follow an unwritten convention of implementing the same set of commands to get you up and running. When I find a project I know nothing about, and I see a Makefile file inside, chances are that I can run make or make build followed by make install, and I will get this project built and set up on my computer. Or at least I will get information on other steps I need to include. I try to apply the same rule in my projects. If I open a folder with one of my old projects and run make dev, this will perform all the necessary steps to build the project and spin up a dev server. That's convenient because throughout the years, I used many different technologies, and each had different commands to build or deploy a project. I have old projects written in Jekyll, Hugo, 11ty, and all sorts of different Python web frameworks. With makefiles, when I come back to a project I haven't touched for months (or years), I don't have to remember the command to start a dev server with, let's say, Jekyll. I just run make dev, and this, in turn, fires up the corresponding Bundler commands. Even if I use tools like Docker or gulp in my project, I still use makefiles to orchestrate those tools. For example, I often write a make build command that builds all the necessary Docker images, passing additional parameters specific to a given project. My makefiles are simple. I don't use conditional statements, flags or any other fancy features. Most of the tasks (they are technically called targets, but I always call them tasks in my head) consist of one or more shell commands. I could write bash scripts with a couple of functions instead, but makefiles are easier and faster to write. Some common tasks that most of my personal projects[1] contain include: dev to start the development server build to build the project (if a build step is necessary) deploy to deploy/publish the project And that's really it. Sometimes, I include additional tasks like watch to automatically rerun the build task when I change any of the source files. But many of my projects can be managed with just two or three Make commands. This blog that you're reading right now has a simple makefile with just one target: dev: npm run dev And a more advanced project of mine uses the following makefile to run the dev server, watch for changes, build, encrypt and deploy the website: # Run dev server dev: bundle exec jekyll serve --unpublished -w --config _config.yml,_config-dev.yml --livereload # Build assets build: npm run gulp build # Watch a specific folder and process assets watch: npm run gulp watch -- --wip # Build the website locally, encrypt and deploy to Netlify server deploy: JEKYLL_ENV=production bundle exec jekyll build; \ make encrypt; \ netlify deploy --prod # Encrypt the "_site" folder encrypt: npx staticrypt _site/*.html -r -d _site In both of the above examples, I'm ignoring the existence of phony targets, which you might want to add if you have a file called dev, build, watch, deploy, or encrypt, as many kind readers on Hacker News suggested. Otherwise, this Makefile won't work as expected. GNU Make (the software that runs makefiles) is quite ubiquitous. If you're on Linux, you probably already have it installed. Even on my MacBook, I don't remember installing it explicitly. It must have come with some other tools that I installed in the past. Make is simple and doesn't require as many additional dependencies as some other build tools. This can be useful if you need a tool that will work in a restricted environment where installing additional packages is difficult or impossible for security reasons. Make will probably be already present in that environment. And if not, you can just take the commands from the makefile and run them manually in the shell. If gulp is not available on your server, you can't really take the JavaScript code and paste that into the terminal. I'm not against other build tools. I like other build tools too. I'm excited when I find a new one that is better and faster than the one I was using before. But I will still use Make to orchestrate them because it gives me a set of familiar commands to manage all sorts of different setups with different tools. By "personal", I mean projects where the deployment process is much simpler than production-grade stuff. ↩︎
In 2020, I built my first side project. I scratched my own itch, then started selling it, and since then, the project has earned me over $15,000. But a few months after releasing it, I was so tired of the maintenance that I wanted to shut it down. Here is a story about my short entrepreneurial adventure. Gumroad dashboard with the earnings from my project In the second part of 2020, I had some spare time. I just finished a project for a client, and I've decided to dedicate a few months to making a Python course – just to try something new. At the same time, I became interested in using technical analysis to trade stocks. Following one of the "internet investment gurus" (yes, I know how bad it sounds), I started day-trading stocks following some "mystical knowledge" on how to leverage price formation called "harmonic patterns" to figure out when to buy or sell (look, I'm sorry, I also cringe when I write those words). Like many traders investors, I was using TradingView, the most popular free platform for technical analysis. Compared to the tools offered by stock brokers at that time, TradingView was far superior with its user-friendly web interface and a plethora of tools. You could even write your own scripts using an abomination of a scripting language called PineScript to implement all sorts of additional tools, graphs, and metrics for your charts. But there was no script that would draw harmonic patterns for me, and after a few days of drawing them manually, I got tired and decided to automate the process. It took me a few weeks to learn the language and implement a script to automatically draw harmonic patterns on the chart. The script significantly decreased the time I had to spend searching for stocks to trade. Then, I wrote another script - one that was drawing potential future patterns. Great! If those scripts work for me, maybe someone will pay money to use them too. After all, the idea of harmonic patterns wasn't new – the earliest harmonic pattern, the "Gartley Pattern", was first described in 1935. There are books, articles, and videos on this topic, so for sure, I wasn't the only one using them. Luckily for me, TradingView supports selling access to scripts, although in a slightly convoluted way. With the most expensive subscription, if you publish your script with "invite only" access, you can control which users can use it. So, I bought that subscription and created a landing page connected to Gumroad to accept payments for monthly and yearly subscriptions. Promoting my scripts # Now, I needed to find a way to tell TradingView's users about my scripts, as those "invite only" scripts are not easy to find. TradingView doesn't promote them in the search results; instead, it prioritizes free-to-use scripts with publicly available source code. And even if someone found my scripts or my website, I needed to convince them that my scripts are useful and do what they are supposed to do. Even though I offered a 14-day money-back guarantee on any subscription, people were reluctant to give their credit card details on some random website. So, I've created a couple of smaller scripts by reusing large chunks of code from the existing scripts. Their source code was still hidden, but people could use the scripts for free - with limited functionality: you could find only some types of price formations there was a limit to how many past days you could check there were no automatic notifications when a new pattern appeared Still, people liked them and started using them. When they asked about missing features like the notifications or detecting more patterns, I directed them to the paid scripts. Then, I published some "ideas" on TradingView - screenshots of popular stocks with drawings of the existing and potential future harmonic patterns generated with my scripts. Finally, I created a YouTube channel where I published videos showing all the features of my scripts (this was partially so I didn't have to explain over and over again how they work). And I started waiting for the cash to roll in. Nothing happened for the first two weeks, but then I got my first sale! Someone bought access to one of the scripts for a month. For the first time in my life, I got paid for selling something on the internet. It felt amazing! Later, they requested a refund because they didn't find the script helpful, but to this day, I clearly remember the excitement of laying on the sofa in the evening and getting an email from Gumroad saying, "Hey, someone just paid you $9 for the thing you've built". The first sale felt more rewarding than any other that followed, although hitting a $1,000 revenue per month was also a nice milestone. Slowly, more people started using my free scripts, and some also bought the paid ones. To strike while the iron was still hot, I posted a message saying that anyone interested can leave a comment under the paid scripts, and I will give them a free 1-week trial. So people started doing that, and each time, I would log in to TradingView and manually grant them access for a week. Some of those trials turned into paid subscriptions, but most didn't. Slowly but steadily, the number of subscribers started increasing. Needy customers? Checked. Fraud? Checked. Next stop - the "burn out" station # As more people used my scripts, I started getting more emails with questions and requests for new features. Many of the emails contained requests for free access to the scripts. Others had very basic questions, answers to which were given in the description of each script. Some people wanted to buy the source code: Or get access to it for free. Of course, for personal use. Sure thing random person from the Internet, I totally trust that you won't share or publish my script as your own. There were people requesting that I add some very specific features because someone they watched on YouTube recommended it. Or that I add some kind of feature that I could not really understand, but it surely made a lot of sense in their mind: Quite a few folks were asking for some "hot tips" on how to trade something or if that something was going to go up or down (almost always, this something was cryptocurrencies or forex). Even worse, sometimes such requests were accompanied by a backstory like "I'm a poor student and want to make some money" or "I've lost some money trading, but your script looks very cool, and I will use it to make money back". Why on earth would you bet your money on some random tool you don't even understand? And then ask some random stranger on the internet for financial advice?! This was sad. I built a tool for people who knew what harmonic patterns were. People who had an investment strategy and only needed a tool to automate the drawing of the charts that they would normally draw by hand. Not some gung-ho investors whose entire investment strategy consists of "this random script I found five minutes ago says that stock X will go up, so I buy." Sometimes, I would get a job offer. Yay! For writing PineScript for a living. Nay! Writing PineScript full time? No, thanks. Or a collaboration proposal from someone who had a "large group of active traders": Somehow all those claims from 'people with large communities' never materialized beyond testing the trial. I even had a chance to practice foreign languages! Disputes # Sometimes, people would open a "dispute". A dispute means someone complained to PayPal that their credit card was wrongfully charged and then it's up to PayPal to decide if they are right (and refund them the full amount of money + charge me additional $20 for "chargeback fees") or if they are not (then I get to keep the payment). In general, I try to be as accommodating to customers as possible. I'm selling software, so unlike with a physical product, I have unlimited stock of my products; producing new copies costs me nothing, and I don't have costs related to handling shipments or returns. So if someone writes to me and asks, "Hey, I forgot to cancel my subscription, and it renewed for another month", or "It's past the 14-day money-back guarantee, but it turns out that the script is not useful for me", I give them their money back. I want to be nice to people. I started selling those tools to help others. I don't want to charge them for a product they don't like. But I also don't want people to open disputes. If you have too many disputes, you risk that PayPal will freeze your account. Sometimes, being nice to people is not enough. Especially if you don't get a chance to show your goodwill because you're sleeping. My first dispute came from a customer who sent me a message in the middle of the night asking how to cancel their subscription and, half an hour later, proceeded to open a dispute. Even though all they had to do was check the emails from Gumroad that contained a link to manage their subscription. And those are short emails with a few lines of text, so it's not that the link is hidden somewhere - at least other people who wanted to cancel their subscription managed to find it. Or they could at least send me an email without opening a dispute right away. Luckily, after I explained that I could (and wanted to) refund their payment, provided they closed the dispute first, the customer cooperated with me. So we closed the dispute, and shortly afterwards, they got their money back. A few more disputes happened throughout the years but were less thrilling. Usually, someone would open a dispute through PayPal, not answer my messages (or PayPal's), and then PayPal would dismiss that dispute as unfounded. And yes, fraud # Being polite and accommodating will take you a long way and make your online interactions much more enjoyable. It also means that, at some point, someone will try to take advantage of you. For me, it happened three times. Most of the time, someone tried using a stolen credit card, but Gumroad caught that and cancelled the payment. But the first time a fraud happened, it caught me a bit off guard. I don't have many screenshots left because that person deleted their TradingView account, but here's how it went. First, Mr. Scammer (that's not his real name 😉) asked for free access to the script. That's not uncommon - sometimes, people send me requests like that. When I kindly but firmly denied his request, he tried to pay, but his payment was declined with a message saying he should contact Gumroad. Again, this didn't sound like a red flag, more like a technical issue. Eventually, I got a confirmation email that he had paid for the subscription. Two months later, a "sale has been disputed" email dropped. The dispute was initiated by Mr. Scammer. I asked him why he initiated the dispute because he never mentioned wanting to cancel his subscription or getting a refund. This is when things started to be very fishy: Weird grammar errors were not a red flag. They are common in emails I get from existing and potential customers, so it doesn't automatically mean an email is a scam. But a story of your wife controlling the finances and you forgetting to tell her about buying a script for TradingView, which made her open a dispute? That's even weirder than the "I'm a student who can't afford $15, but I really need this script for my day-trading" emails I sometimes get. While waiting for the dispute to be resolved, Mr. Scammer tried to buy the subscription one more time, this time with less success: So, was his "wife" not only controlling the finances but also using stolen credit cards? I wrote him that he's permanently banned from buying any scripts from me, and the next time he tries, I will immediately revoke his access. That stopped any further attempts. That was the most interesting case of fraud. Usually, when people got caught, they ignored my messages instead of coming up with a creative story: Burn out # Answering emails and checking TradingView comments to give out trial access took a bit of time every day. Not much, but I tried to answer at least every day. It was a minor annoyance when I wasn't working full-time. But when I had to do this after a full day of work, and I got yet another email asking to add some magical method for determining take profit levels that was invented by a crypto-trading YouTuber with 50 followers[1], I had enough. The worst part was that after a few months, I wasn't even using those scripts anymore. I had fun day trading when I wasn't working full-time. It was a nice break from recording videos and a motivation to learn about financial markets. But when a new contract started, I stopped day trading due to the lack of time. But I still had existing customers that I needed to take care of and potential new customers asking me all sorts of questions. I was growing tired of maintaining this project. Sometimes, I would spend an hour answering many detailed questions from a potential customer only to never hear back from them. At times, someone would leave a 1-star review on Gumroad because I didn't want to add some weird functionality they requested. On days like that, I wanted to close down this project, refund the money to the existing clients, and delete my TradingView account. Still, the project was bringing in a bit of money every month. I didn't want to kill the goose that laid tiny golden eggs. So, I've decided that I'm going to sell my "goose". Selling my project. Well, trying to... # I looked for platforms where I could list my project for sale. I found a few and decided to submit an ad to IndieMaker and Transferslot (which looked much more active three years ago). I prepared a detailed description of the project and shared details like the number of views on my YouTube channel, unique visitors to the website, the number of followers and subscribers on various platforms, and, of course, the revenue. I even shared some ideas on how to move this project forward. For example, getting one of the free scripts promoted in the weekly TradingView newsletter could accelerate the sales of paid scripts. Then, following some random advice on the internet, I came up with a number to ask for that was around 5x the revenue from last year. I then sent my ad for publication. IndieMaker published my ad, and shortly after, I received an inquiry from a potential buyer. But after answering questions about the Monthly Recurring Revenue growth and what assets come with the project, there was no follow-up. I got more messages later, but those were just spam ("Hey, I'm interested in your project, send us a list of your inventory"). Transferslot never published my ad. I think the website was already abandoned when I submitted it. If you have any experience selling projects online (and if you don't, this article from Patrick McKenzie is the best way to understand how it works), you're probably shaking your head by now. People buying software businesses look for simple, low-risk, boring-tech projects with a proven track record of stable or increasing profits. No one wants to buy a project written in an esoteric language (PineScript) running on a proprietary platform (TradingView) that targets a very specific niche of traders. After a couple of weeks with no more replies, I had to decide what to do next. I could submit my ad to bigger platforms like Flippa and pay to have it listed there. But I felt that this project was so niche that I wouldn't find a potential buyer on other platforms either. Especially a potential buyer who knew the PineScript language necessary to maintain and improve the scripts' source code. While waiting for more bids from potential buyers, I started thinking about what I could do to ease the burden of this project. I'm an engineer, goddamit! We're not meant to click buttons in the browser or send the same email over and over again! Turning the autopilot on # I checked my options and decided to automate the following parts of my work: Gumroad payment should automatically grant access to the script on TradingView and send a "welcome" email. This would cover the most time-critical part of my work, so I didn't have to start my day from logging in to TradingView and granting access to people who bought subscriptions during the night. A chatbot to manage subscriptions so that I could do all the manual work (removing access from people who cancelled subscriptions, extending trial access, etc.) with simple commands from my phone without logging in to TradingView. Finally, I could create an online form where people could request trial access by leaving their TradingView username. This form would then trigger a Python script granting them one week's access to the selected TradingView script. I implemented all those features using n8n to connect various services together - webhooks, a Telegram bot, Python scripts, sending emails, etc. Here's an article describing how I did this. Once I had the automation in place, users could request free trial access through an online form or buy a subscription on Gumroad, and everything would be handled automatically. From time to time, when they cancelled their subscription, I had to send a short message to my Telegram bot to set an expiration date for their access. The hardest part was to stop caring so much about this project and disconnect myself. I like helping people. That's why I became a programmer - to build things that help me or help others. When I get an email, I try to answer it as best as I can. Years of working with clients taught me to explain things in a simple and easy-to-understand way. So, I spent hours patiently answering questions from potential customers only to never hear back from them. And even though I posted a link to the form where people could request trial access, plenty of folks didn't bother reading the description and kept commenting or sending me messages requesting trials. This had to stop. I started by checking TradingView messages only on the weekends and granting trial access in bulk. For every comment about trial access, I replied "I gave you access now, but it would have been faster if you used the form I posted", hoping that others would see my reply and use the form next time. I still paid special attention to my existing clients and answered their emails every evening. But if someone was not an existing client, I would only respond to their questions on weekends. Eventually, I stopped checking the TradingView comments or messages. If people really wanted a trial, they would eventually read the two-sentence instruction, click the link, and fill in the form. Did this affect my sales significantly? I don't think so. Most of the people asking basic questions would never convert to paying customers anyway. They were window shopping. They found a script that seemed to predict price changes, and they wanted to know how to make money with it. And the theory behind how it works and why it's supposed to work? Nah, they didn't care. They had ten more random scripts to test out today. Did disconnecting from my project affect my sanity? Oh, hell yes! The project went on autopilot, and it no longer felt like a dreaded chore when I was checking my emails or logging in to my TradingView account. It lost a lot of traction because I stopped updating it, so people think it's not useful anymore (even though it does its job as well as it did a few years ago). Its revenue declined in the past years. But I don't care. I'm happy again. What's next for this project? I guess I will keep running it until I decide that including it in my monthly accounting is not worth the effort, which is when I will shut it down. But for now, the $200 I get every month with almost no work is a nice passive income. Things I've learned # Don't let a side project burn you out # The dreaded truth that most programmers learn the hard way is that releasing a product is just the beginning. Once it's out there, you have to maintain it: fix bugs, implement new features, deal with disputes or frauds, and answer a LOT of emails (often from curious window shoppers who never convert to paying customers). This maintenance is easier if you have more products and you do support in bulk. Or if your project can replace your full-time job and maintaining it is the only thing you do during the day. But if you do this on top of your regular job, it's a terrible time sink. You need to set some boundaries. Otherwise, working on your project will take all your available time. Working on a side project is much more fun if you're also one of its users and genuinely want to keep improving it. It's incredibly smooth if you love your project, and the perspective of working on it full-time sounds like a dream job. And accordingly, it's much harder to support a project when you move on to doing something else and you're no longer interested in maintaining it, but you still have existing customers you need to keep happy. If your side project is not a source of genuine joy and you have other priorities, it's important to set some boundaries. I've decided to focus my support only on the paying customers. I also stopped adding new features and told myself that unless my scripts break, I'm not touching the code. Even though I still had a list of additional improvements I wanted to make one day, I deleted that list. I already had paying customers, which means people find value in what I've built. Sure, I can add new features to attract new customers. But that means even more coding and maintenance for a chance to attract new customers. Use a merchant of record # If you're planning to sell products globally, consider using a "merchant of record" like Gumroad or Paddle. This way, you won't have to figure out how much tax to charge for each country (basically, you only sell to Gumroad, and they resell your product to the final customer). Figuring out taxes was the last thing I wanted to do when selling my products. I went with Gumroad, which - at the time of writing - charges a 10% fee, plus around 3% of PayPal/Stripe fees on top of that. There might be cheaper options, so do your research before choosing one. Once you get some recurring payments rolling in, you can't easily move your existing customers to a different company. When the Gumroad fees increased throughout the years, I always thought that migrating to another service was too much of a hassle, so that's why I'm still using Gumroad. Selling stuff to customers is hard # I had this one guy where the conversation looked like this: Hey, amazing script, I rated it 5 stars! Can you implement the take profit levels for me? I'm sorry, but I can't. The original methodology doesn't specify exact take profit levels. It also doesn't make sense because take profits should be manually determined, for example, based on support or resistance levels visible on the chart. Different people will use different take profit levels based on their risk appetite. Ok, but can you please put them for me? I follow this [insert some YouTube crypto day-trader], and he uses [some very specific take profit levels]. I would like the script to draw them for me. Look, I can't do this. Those take profit levels are very specific to your trading system. They won't work for most other people, but they will slow down the script for everyone. Ok, bro. He cancelled his subscription a few days later and lowered his 5-star review to 2 stars. Business-to-consumer (B2C) is difficult (not that business-to-business is much easier - it has a different set of challenges). But this doesn't mean that B2C is not worth pursuing. There are many people who did great selling to customers. One of the most well-known is Pieter Levels (@levelsio), creator of projects like NomadList, RemoteOK, and, more recently, PhotoAI. According to his Twitter profile stats, he makes an astonishing $200k Monthly Recurring Revenue from his projects. But B2C, while easier to get into than establishing a relationship with a company in a B2B business model, means that you will need many more customers to have the revenue levels that B2B can bring. And more customers mean more work. Be nice # Try to be nice to people. I mean, being nice to people in general will make the world a better place. But when dealing with customers, never assume they do something in bad faith. Sure, some folks will try to take advantage of you. But many "problems" are just a matter of misunderstanding. People will request a chargeback because they think it's the only way to get a refund for a subscription they forgot about. If someone asks me for a refund, then no matter what's the reason, I give them their money back and wish them good luck with their investments. Fighting disgruntled customers over $20 is not a good way to spend time. If you're nice to people, they are more likely to use and recommend your products. One of the reasons why Amazon got so popular early in the days was because of its extraordinary customer support. Ten years ago, when my Kindle stopped turning on (which might or might not have been caused by accidental yet excessive force applied to it), Amazon's support said "no worries, we'll send you a new one for free". Someone would send me a free product instead of requesting to prove that it broke by itself? This blew me away! Especially since I was living in Poland, where the strong post-communism roots mean that if a seller is not mean to you, then this already counts as them being nice.😉 So, I wanted to do the same with my customers. I tried to accommodate their needs, write detailed explanations to their questions, and generously issue refunds or even discounts when requested. When someone complained that something didn't work because it didn't work the way they wanted, I suggested alternative scripts and offered refunds. And that paid back. I received a lot of nice messages from customers, and even though I was getting tired of supporting my project, getting positive feedback from time to time kept me going for all those years. And that's the story of my first side project. It was fun at the beginning, frustrating in the middle, and now it's mostly a passive income that will eventually dry up. Was it worth the money? Not really. Spending equal time working for a client would probably earn me more. But the experience I gained from this side project will be invaluable when I build the next one. If you're interested in the technical setup I used to automate most of this project, check out my Web Automation With n8n, Telegram, Online Forms, and a Bit of Python article. It was always the crypto or forex crowd that was asking for weird stuff. ↩︎
One day, during one of those meetings that did not require my full attention, I was doing some work in the background. I muted my microphone, and I was coding away[1]. Suddenly, someone asked me a question. "Now, where is that goddamn Teams window so I can unmute my microphone?!" - I asked myself, frantically going through my MacOS desktops. I use multiple desktops (or workspaces, as some other operating systems call them), each with one to three apps opened. So, instead of shuffling 20 apps opened on one desktop, I shuffle ten desktops scattered across three monitors. It took a few seconds of awkward silence before I found that Teams window. And it wasn't even the first time this happened to me. "There has to be a better way!" - I thought, again, not for the first time. I've decided it's time to figure out how to set up a "global mute/unmute" button that would work regardless of whether Teams (or any other videoconferencing app) has the focus. One way to solve this problem was to use Elgato's Stream Deck. I've been eyeing it for some time, as many developers claim it's a very useful device. Is Stream Deck really indispensable for a software developer? # "Stream deck is a fantastic tool, and every developer should have one (or three)!" is something I've heard here and there. I've put it on a shopping list for Black Friday, hoping I can get a good deal. But as Black Friday was getting close, I asked myself - do I really need Stream Deck? My main driver was to have a global mute/unmute button for those Teams calls when I'm doing something else and suddenly need to unmute myself. Pressing a button to unmute my microphone, say what I need to say, and mute it again without shuffling through multiple windows and desktops would be bliss. But that doesn't justify shelling out over $100 (or less, for the "mini" version) and putting another device on my already cluttered desk. So, I've started reading about other things people do with Stream Deck, especially if they are not streamers but developers like me. There are some good reasons to buy Stream Deck. For example, if you SSH to many servers, talk with many people on Slack, generate different Stripe test cases, switch between many Spotify playlists, work on different VS Code projects, and push code to different GitHub repositories - and all that on a daily basis, then you will probably benefit from using a device that can reduce all those tasks to a press of a button or two. It sounds more useful for people who prefer to click things and press buttons instead of typing all the commands in the terminal. It also works well for those who spend most of their time in one piece of software like AutoCAD or Photoshop, where creating and assigning macros to hardware buttons will save them time. Not to mention that in this line of work, you're already using a drawing tablet, so you're used to using different devices apart from a keyboard. There are some weird but ok reasons to buy it. Some people use it for home automation - ok, but I'm not going to run to my office to toggle lights in the bedroom. Some use it to start applications - fine, but for me, firing up a launcher like Spotlight/Alfred, typing 2-3 first letters of the app I want to start and pressing enter seems equally fast as pressing a button on an external device[2]. But a typical software developer spends most of their day pressing keys on a keyboard. Everything that Stream Deck does can also be achieved by pressing one or more keys. Some actions will trigger a keyboard shortcut, while others are equivalent to typing commands in the terminal or running a script. Stream Deck is a glorified macro recorder. It comes with a nice app that makes it easier to record those macros. It has colorful icons that catch your eye and pressing a physical button is easier than remembering a shortcut for a specific macro. But most of its functionality can be achieved by recording macros on your computer. If you have a manageable number of macros (or a very good memory of all the shortcuts you defined), it might actually be easier to use your keyboard to trigger the macros. Stream Deck is not a big device (especially if you go for the mini version that has six buttons), but it's another device you would need to carry with you if you go somewhere. A keyboard, on the other hand, is usually an integrated part of your laptop. Where your laptop goes, your keyboard goes too. Actually, I bought the Stream Deck after writing and implementing all the tips from this article. I got a good deal for it, and I could always return it if I saw that I wasn't using it that much. After a couple of days, I liked it and decided to keep it. While I still think you can write keyboard macros that can do the same things as an external device, remembering how to trigger those macros gets harder the more macros you have. A device like Stream Deck (or one of its cheaper or more customizable alternatives) is just more convenient for me. Pressing a colorful button with a label takes no mental effort. On the other hand, trying to remember how to trigger some less often-used shortcuts can be a bit more demanding, especially if I'm in the middle of thinking about something else. I decided to keep the Stream Deck because I found several additional uses for it (that I will describe in another article). If I were to use it only to mute/unmute my microphone, I would definitely stick with the macro on my keyboard instead. How do I find unused key combinations? # One of the great things about Stream Deck or a similar device is that all of their buttons come with different key codes. There is no risk that pressing a button on a device like Stream Deck will trigger a different action, for example, from the currently opened application. But if you want to start assigning custom shortcuts on your keyboard, this might be a problem. When defining keyboard shortcuts that will trigger your macros, you need to find a combination of keys not used by any application. Otherwise, if you try to use a shortcut, the current application will intercept it and perform a completely different action. Assigning custom actions to shortcuts like "cmd+s" ("ctrl+s" for Linux/Windows users) or "cmd+w" is off the table because, in most programs, they are already used to save the file or close the app. Luckily, there are some combinations of modifiers that are rarely used. These are called the Hyper key (shift+cmd+ctrl+option on Mac and shift+windows+ctrl+alt on Windows) and the Meh key (shift+ctrl+option on Mac and shift+ctrl+alt on Windows). Both combinations are not the easiest to press; for example, for a keyboard shortcut that uses the Hyper key, you need to press four modifier keys and another key assigned to a shortcut. But many external keyboards have additional keys to which you can assign the Hyper or Meh keys (that's what I did with my Kinesis Advantage keyboard). Even the built-in keyboards have keys you can live without (the "Caps Lock" key is fairly unused - unless you need to shout a lot on the internet) and can be converted into a special key. I decided that before I buy the Stream Deck, I will first try to implement some useful macros. I used BetterTouchTool or BTT for short - my preferred way to define macros on a MacBook (I bought it years ago to customize the touchbar on my previous laptop, and I've been using it ever since). I configured the "Esc" key to act as the Hyper (so "shift+cmd+ctrl+alt" ) and "Insert" to act as the Meh (so "shift+ctrl+alt"). Both those keys were previously unused - I'm using "Caps Lock" as the Esc key and the "Insert" key is basically a second backslash key on the Kinesis Advantage2 keyboard. An unused key on my keyboard that could be used as Meh Next, I've decided to make an easy-to-remember split of functionality of those custom shortcuts. The Hyper key would be used in shortcuts that start some applications, and the Meh key would be used for all the other macros. Ideally, the first letter of the application's name could be used as the shortcut to open it. So, for example, Hyper + o would open the Obsidian app, Hyper + i would start iTerm, and Hyper + f would start Finder. Of course, not every app could have a shortcut like that (e.g. if two apps started with the same letter, one would need a different shortcut). But I've decided to limit the number of application shortcuts to only those I use multiple times per day. If an app is already opened, pressing its shortcut will switch to the desktop with that app and bring it to the front. Next, it was time for more advanced macros that I would use with the Meh modifier: Meh + m: mutes/unmutes the microphone. This shortcut triggers a "Run Apple Script (blocking)" from BTT that runs an applescript to toggle the current microphone's volume between 0 and 100 and displays a notification. To make sure I don't get confused about whether my microphone is muted or not, I've added this BTT widget to display an icon in the menu bar indicating the microphone's status. Meh + p: triggers the "Play/Pause" action in BTT to toggle the currently playing music or any other video or audio. I could achieve the same by pressing the button on my headphones, but still, my fingers are on the keyboard, so pressing a keyboard shortcut is faster. Meh + n: sets the "Do not disturb" status for 1 hour. This hides all the notifications and is very useful during screen-sharing sessions, ensuring no weird message pops up in the middle of a presentation. I have an Apple Shortcut that sets the "Do not disturb" status and accepts a parameter defining how long it should be set. This shortcut comes with the Calm notifications workflow for Alfred, and all I have to do is trigger it from BTT with the "Run Shortcut from Shortcuts App" action (optionally followed by the "Show notification" action to get a popup confirming the correct execution of the shortcut). Meh + t: brings the Teams window to the foreground. This is done by using the "Show/Hide Specific Application" action in BTT, and it saves me from figuring out where my Teams window is during calls. Those are the most useful shortcuts I could think of. This setup was not without flaws, and I solved them when I bought the Stream Deck. But it was good enough to start. What were the flaws? For instance, the play/pause button usually works for the last piece of media content played, but sometimes it's a bit random. This was an issue when I had the Spotify app opened (which is always opened) and a YouTube video paused in one of the browser tabs. Sometimes, the macro would work on Spotify, when I wanted it to start the video, and sometimes it would start a video somewhere in my browser when I wanted it to stop the music [3]. That was quite annoying, and with Stream Deck, I've solved this problem by having a separate button for Spotify and another one for other media content. Or let's take the mute/unmute functionality. It worked well as a macro, but with Stream Deck, I can get a plugin that controls the Teams client directly and toggles the mute/unmute setting in the video call. This approach is more convenient because everyone on the call can see an icon indicating that I'm muted (and, when I try to say something, they can tell me, "Hey, you're muted"). An "operating system-wide mute" is not reflected in Teams, and I've noticed that it's more confusing to people during the calls. Having a few macros is great if you need to simplify some common actions on your computer. But as I wrote in the side note about my experience after buying the Stream Deck, the more macros you have, the more you have to think about how to trigger each one. This can be distracting when you are in the middle of a task requiring a lot of focus. So, I gave Stream Deck a chance and ended up keeping it. Check it out to see my Stream Deck setup and my conclusions after using it for a few months. Try to get a mechanical keyboard, and you will quickly understand why muting is necessary. ↩︎ I was wrong with this statement. Pressing a button to open an app is sometimes more convenient. Especially when I'm doing something else on my main screen and I can press the button on an external device without losing focus. ↩︎ And DJ "Spotify mixed with a random YouTube video" is terrible. ↩︎
Earlier this year, I hit GitLab's 10GB storage limit with one of my git repositories. I'm writing a family blog where I document our various trips and activities, hoping that in 20 years, it will be a great memento (like a photo album combined with a journal in a digital form, with search functionality, videos, GIFs, etc.). The blog started as a simple Git repository that I hosted on GitLab, but after adding hundreds of pictures, I hit the maximum repository size, and it was time to move images and movies somewhere else. Looking for a possible solution, I wanted to choose something that integrates seamlessly with my existing git workflow, so the final choice was between git LFS and git-annex. The reviews were mixed, but git-annex was more favored by the audience. The deal breaker was that if I decided to stop using it in the future, I would just need to run git annex uninit to get all the images back in place. That means losing the history of image changes, but I don't care about that anyway - I only care about not losing the most recent version of each file. So I've decided to go with git-annex. Unlike my wife, I rarely look back at the old pictures and walk down memory lane. But I know that when I get old, those memories will be among the most precious things left in my life. And even today, I can't remember trips I took ten years ago or large parts of my student life. That's why the only non-work related subscription for which I pay a significant amount of money is the 1 second everyday app. It lets you record a one-second-long clip of your day and then join multiple clips together to get a video of how your month/year/life looked like. If you pay for the premium plan (that costs around $70 per year), you can record longer clips, add multiple clips per day and - most importantly for me - have your clips backed up. And while it costs me more than, let's say, my Spotify subscription that I use for multiple hours per day, I often forget to record my "1 second" every day. I remember using it mostly when I'm on holidays. So why do I pay for an app that I forget to use? Because I love the idea behind the app. It lets me generate 1-minute-long videos full of great memories from vacations. And with the paid plan, I get backups, which means I won't lose the recorded memories if I lose my phone. I've looked for alternative solutions and even thought about making all those recordings myself. Still, I've decided that the time I will spend cutting and editing my videos is not worth saving 70 bucks per year. If you like the idea of recording small clips of your life, give the app a try - it has a free plan that might be enough for you. Disclaimer: I'm not related to that company in any way; I just like their app in the context of my family blog. git-annex is a tool that synchronizes files across different repositories. And by repositories, I mean folders, external drives or even 3rd party services like Google Drive, Dropbox, Amazon s3, and others (synchronization with 3rd parties usually happens through rclone, rsync, and similar tools). It also integrates nicely with git, so you can manage "large files" through git-annex and all the other smaller files through git. You can even go into full automatic mode and have git-annex handle all the files in your folder. Then, each time something changes, git-annex automatically commits the changes and propagates them to other repositories. git-annex home page That was the good news. The bad news is that git-annex is a complex beast with a steep learning curve unless you settle on the "relinquish all the control and let the git-annex daemon handle all the changes for you" mode (which I didn't want to do since I like to control exactly how my git repository works)[1]. It's much more complex than git. And surprisingly, there are not many tutorials clearly explaining how to do what I wanted to do (which is - to store images from my repository on Google Drive and a NAS server). The tutorial from the documentation is a great read, and I recommend you read that first. But it focuses on adding remote repositories on a USB drive or in another folder. There is a brief mention of special remotes, but for people like me, who just learned about git-annex, it's not clear if a special remote will ever be needed. I update my family blog once a year - I spend 2-3 weeks working on it and then leave it until the next year. So, I decided to write down everything I learned when setting up git-annex for the first time, including how to use it and how to troubleshoot common problems. This will make my life easier when I get back to git-annex next year and forget everything I did last time. Maybe my notes will make your life easier too? I've spent a couple of days trying to understand how git-annex works and made many mistakes and invalid assumptions along the way. A few times, it was easier to start from scratch with newly gained knowledge than try to fix the mess I did. I used a test repository until I felt I had a good grasp of how git-annex works. I still don't understand many of its details, but at this point, I have a setup that works well for me. I also have a lot of respect for the creators of this tool and all the contributors. git-annex is an incredibly versatile tool that supports a plethora of different scenarios, backends, backup modes, etc. But using it is an order of magnitude harder than using git alone. How to set up Google Drive (or any other external backup service) with git-annex # The first task I embarked on was to enable storing files on my Google Drive account. The steps that I describe work for any other service you can connect to through rclone, including most of the popular file storage and backup services. To enable using rclone with git-annex, follow the instructions on the git-annex-remote-rclone website. When setting up rclone, you will at some point have to choose the "scope", with the available options being "drive", "drive.file", "drive.appfolder", etc. I initially chose "drive.appfolder", which stands for "Allows read and write access to the Application Data folder". However, this means that you won't see your files when you open Google Drive in the web UI - they will be hidden in a folder specific to an application you create in the setup part. You can interact with those files using rclone directly, but I prefer to see my files in the web UI to make sure they are still there. So later, I changed the scope to "drive", which gives rclone access to all your files on Google Drive. When you change the scope, remember to authenticate rclone with remote again (that's part of the rclone setup instructions)! I didn't do this (and only replaced "drive.appfolder" with "drive" in rclone.conf), and then I couldn't figure out why git-annex is still storing my files in the Application folder. Once I re-authenticated and obtained a new token, everything worked fine. Once rclone is set up, use the example command from the instructions to enable that special remote in git-annex. Here is the command I used: git annex initremote gdrive \ type=external \ externaltype=rclone \ target=GoogleDrive \ prefix=git-annex-travel-blog \ chunk=50MiB \ encryption=shared mac=HMACSHA512 \ rclone_layout=lower How to back up git-annex to a locally mounted NAS server # I have a Synology NAS that I'm using as the 2nd backup for my project. The way I configured it is that I mount it on my computer and access it through the /Volumes/gitannex folder. That's one way to do it, but there are other solutions. If you check the git-annex documentation, you can see that they suggest using ssh to connect to NAS on their Synology NAS and git annex page. I don't use that approach because I'm lazy (I really don't want to install additional stuff on my NAS, set SSH keys, etc.), and I want the simplest solution possible (I already have a NAS mounted on my computer most of the time anyway). Here is how I add my NAS folder to git-annex: git annex initremote nas \ type=directory \ directory=/Volumes/gitannex/travel-blog/ \ encryption=none Things that surprised me the most about git-annex # Figuring out how to use git-annex is more difficult than understanding a typical piece of software created nowadays. Everything you need to know is in the documentation, but it's not served in the most easy-to-digest way. The majority of projects today come with a quick start guide explaining how to use them, often in a step-by-step form, so I can just run some commands and set up everything. And if that doesn't work, I can search the internet for some tutorials on how to do X in a given software. I should get back enough blogs to quickly figure out what to do. git-annex is old school. It doesn't go easy on people like me who live by the rule of "5 hours of debugging can save you 5 minutes of reading the documentation". The knowledge is in the documentation, but just like with Linux man pages, you need to know what you want to do. And to know what you want to do, you must first read the documentation carefully to understand how git-annex works. After multiple days of trying to set it up and rereading the same few pages of documentation, I kept finding new information as more pieces were falling into place to form a better understanding of this tool. Some assumptions that I made initially were revised. Sometimes - even multiple times (I went from manually managing my files to the automatic mode and back). I didn't know what to expect from git-annex and how it was supposed to work. On the one hand, I was expecting something like rsync that would simply move files from one place on the internet (my computer) to another (some server). On the other hand, it had to somehow integrate with git's workflow - track changes, update files, roll back to previous versions, etc. Here's my journey on the path of surprises. First, I installed git-annex and found the walkthrough. I followed it, and after running git annex add *.jpg, I saw that my images were converted into symlinks. Fantastic! That will let me keep the repository size small. Now, what do I do with those symlinks? The walkthrough doesn't really explain how to make them work with Google Drive (apart from vaguely mentioning special remotes). Instead, it focuses on having multiple git repositories in different places. Ok, cool, but after finishing the walkthrough, I was not even a single step closer to figuring out how to set up git-annex for my use case. I found the git-annex-remote-rclone plugin that can use rclone to sync files with Google Drive and act as a special remote. Cool! I've set it up, and then I was able to send files there using git annex copy --to gdrive and retrieve them using git annex copy --from gdrive. Great, it seemed I was all set. It was a bit cumbersome because I had to add files to git-annex by hand with commands like git annex add *.jpg, and when I added files, they became locked, preventing me from editing them unless I unlocked them. But I could live with that. However, the next day, I found out that there was a workflows page explaining that I didn't have to do all that by hand! So I fired up git annex webapp that starts the "assistant" - a daemon that automatically detects changes in the repository, adds files to git-annex, and creates commits. I gave it a try, and it worked, but I no longer had symlinks in my project. All the images were reported in my terminal as full-size. Ok, that's not what I wanted to have. But when I pushed those files to GitLab, I could see that in the repository their size was tiny (hundreds of bytes). When I tried to open one of the images, I saw that inside, there was a text that looked like this: /annex/objects/SHA256E-s144178--dc75f9d82cc334de0445f744bcb.jpeg. What the heck?! The text looked like a symlink, but unlike when I ran git annex add, my terminal didn't see those files as symlinks! Turns out that what I saw were pointer files - the second way git-annex stores objects (the first way being symlinks). Pointer files are kind of like symlinks, and even though they look strange to me because they appear as normal files in the terminal, the important thing is that GitLab sees them as tiny files (which is good; they don't use space on GitLab). And my code editor opens them as normal files too. Running 'git-annex webapp' starts a website for configuring the assistant Great success! From now on, I will use the git-annex assistant! It has the additional benefit that my files are no longer locked. I can edit them freely, and the assistant automatically commits every change. Here is some more information about the pros and cons of unlocked files - the unlocked mode is a successor to the direct mode. All seemed to work great, except that when I created a text file, git assistant also added it to git-annex. That's not what I wanted! I don't want my text files to be converted to symlinks or pointer files. So how do I make git-annex stick with processing only images and videos, leaving all the other files alone? Googling for "git-annex filter file types" didn't immediately return any useful answers. It turns out that I had to change how I think about git-annex. It's supposed to work with "large files", so what I had to do was to declare that images and videos should be considered as large files. One way to do this is to define large files as larger than some threshold, e.g. git config annex.largefiles 'largerthan=100kb'. This might work fine, but I prefer to determine which files git-annex should process based on their extensions because I know exactly what extensions my images and videos have. That seems easier than trying to guess what file size will catch all the media files but not other files. I set this up by creating the following .gitattributes file in the root folder of my project: * annex.largefiles=nothing *.jpg annex.largefiles=anything *.mp4 annex.largefiles=anything The above settings are parsed from the top, so we first declare that no files should be treated as large files, and then add specific file extensions to the list of large files. Alternatively, you could add file types to the list of preferred_content. According to the description from the documentation, "preferred content" is supposed to specify what types of files should go into which remote repository. But I couldn't find any information on how to set preferred content in the .gitattributes file, and passing a long list of filetypes that is stored somewhere in the configuration file seemed less optimal than defining this setting in .gitattributes that I can easily access and edit. Great, now I can start the assistant, and it will only watch for specific file types (all the file types that do not match its filters will be managed with git, not git-annex). I thought this would be my final solution, but I ended up disliking the mess that git-annex was making to my git log. So I went back to manually committing files with my own commit messages, which brings me to... My final workflow # Even though using the git-annex assistant to watch and automatically commit all my changes sounds tempting, it makes my git log look like this: commit 430b796f2559a99b8a01f1a69ce55dfe3c50f75e Author: Sebastian Witowski <[email protected]> Date: Sat Jan 20 20:54:35 2024 +0100 git-annex in [email protected]:~/workspace/test2/test-repo commit f44a6e40440cc083e3c15bf6f9b40ef2ee818e8f Author: Sebastian Witowski <[email protected]> Date: Sat Jan 20 20:01:56 2024 +0100 git-annex in [email protected]:~/workspace/test2/test-repo commit 95090b5861a0afc9de20796c6a7c3bb7ea696649 Author: Sebastian Witowski <[email protected] Date: Sat Jan 20 20:00:34 2024 +0100 git-annex in [email protected]:~/workspace/test2/test-repo ... And I don't like that. Sure, I can squash and edit the commit messages, but I prefer to use git-annex the same way I use git - so control manually when and what I commit. In the end, I disabled the assistant. Now, when I work on my project, and I want to commit changes, I do this: # Add all files to the index # git-annex will figure out which files go into annex and which go into git based on the filters I defined in .gitattributes git annex add # Commit changes git commit # Push changes to git and send files to remotes git annex sync --content Short and sweet. Setting up git-annex from scratch and basic usage # Let's distil all I've learned into a short guide explaining how to set up git-annex from scratch and how to use it. Take my advice with a grain of salt! git-annex supports many different workflows, and what I'm describing in this article is a workflow that works for me. Also, I've used git-annex barely for a couple of evenings. For sure, there are more optimal ways to do some things (or at least more optimal for your scenario, dear reader). I hope I'm not messing up something seriously, but this I will see in a couple of months/years. My scenario doesn't require very strict version control of images, and I'm fine with keeping only the latest versions. So if I mess something up, I can just take the most recent version of all the files and start a new git repository from scratch. And git-annex allows you to disable it rather easily - you just run git annex uninit, and it turns all symlinks back to the original files. That said, let's dig into setting up git-annex from scratch. If your starting point is a folder with neither git nor git-annex, then first, you need to initialize both of them: git init git annex init Next, set up a remote Google Drive (or some other external storage) using the git-annex-remote-rclone plugin. The following commands will add a new remote called "gdrive" using the "GoogleDrive" configuration from rclone.conf (so make sure to use the same name when you configure rclone) and store files on Google Drive in the my-app1 folder: git annex initremote gdrive \ type=external \ externaltype=rclone \ target=GoogleDrive \ prefix=my-app1 \ chunk=50MiB \ encryption=shared \ mac=HMACSHA512 \ rclone_layout=lower # [Optional step] Verify that the remote is correctly set up and we can push/pull files to/from there # This will take a few minutes and might fail/hang up (it failed for me, but I ignored that) git annex testremote gdrive # Sync the information about our new git-annex remote to the git repository git annex sync The git annex sync command above is important. When you initialize a new remote, this command will store information about that remote in .git, so when you clone this repository on another computer, you can enable that remote with git annex enableremote gdrive instead of setting it up from scratch. Now, you can either manually commit files and send them to/pull them from a remote or use one of the automatic ways to do all that. As I explained before, I do everything manually to have better control over when I commit and what I write in the commit messages. Here's how to add and synchronize files: # Add all JPG files to git annex git annex add *.jpg # Copy all git-annex'ed files to gdrive git annex copy . --to gdrive Now, if you need to create another repository (let's say you have a new computer and you want to set up your project there), run the following commands: git clone <link-to-the-repository> . git annex init # Retrieve information about initialized remotes git annex sync git annex enableremote gdrive git annex copy --from gdrive And that should make all the annexed files available. To add more remotes, for example, in a different folder or with a different rclone configuration, just follow the steps again to initialize it (git annex initremote) and synchronize the changes. Once you have more remotes enabled, instead of manually pushing and pulling files from remotes, you can synchronize the state of the files using git annex sync --content command. It will push all your changes to the remotes that don't have their own copy of the files (and also pull those files that are on any of the remotes but not on your computer). Troubleshooting and "how-to" guides # Here is a list of common tasks you might have to do when using git-annex. Remove a remote # How to remove a remote that is in a wrong state (you misconfigured something, and want to remove that remote and try again)? Following this answer, you can mark it as dead and propagate that information to other repos: git annex dead my-badly-configured-remote git annex sync Synchronize changes to a remote # How to synchronize changes after one of the remotes was not available during the previous sync (e.g. NAS was not mounted)? If you're using git-annex assistant, you can wait until the changes synchronize automatically. But if you don't use it or if you want to force the synchronization, run: git annex sync --content Edit a locked file # How to edit a locked file? By default, when git-annex replaces the file with a symlink, it locks it, so you can't modify it. To modify such a file, you need to unlock it, change it, and add it back to the annex: git annex unlock image.jpg <edit the file> git annex add image.jpg Remove unused files # How to remove unused files from git-annex? Removing a file from git doesn't automatically remove the copies from git-annex and the remotes. First of all, be careful, as the method I suggest removes all files not referenced by a branch or a tag. I use the --force option that bypasses the check if at least one copy of the file is left in some other remote (in case you want to restore that file in the future). This basically means that you lose the possibility of going back to the previous state of a file unless it's a file in a commit that was tagged. I don't care about the previous versions of my media files, so this is fine for me. # Find all unused files locally git annex unused # Delete them git annex dropunused --force all # Find all unused files on a remote git annex unused --from gdrive # Delete them on a remote git annex dropunused --force all --from gdrive I'm using the --force option to force the removal. Otherwise, git-annex tries to make sure that at least one copy of this file exists. And that often results in getting "Unable to lock down 1 copy of file necessary to safely drop it" errors because Google Drive accessed through rclone doesn't support file locking needed to ensure that a copy exists. The git annex dropunused command is not needed if you remember to remove files from the annex with the git annex drop command each time you remove a file from the repository. But I'm too lazy to remember to do that and prefer to remove files by hand (e.g. by pressing "Delete" in my IDE) and clean up git-annex afterwards. Additionally, to verify that the files are really unused, I like to check some of them by finding which file points to the given symlink: find -L /directory/with/images -samefile /path/to/file/in/git-annex For any /path/to/file/in/git-annex file reported by git annex unused, there should be no match from the above find command. List where files are stored # How to see where all the files are stored? There are two ways. A more compact list: git annex list (recording state in git...) here |origin ||SynologyNAS |||web ||||bittorrent |||||GoogleDrive |||||| X_X__X _assets/images/posts/2019-02-27-file1.jpg X_X__X _assets/images/posts/2019-03-04-fiel2.jpg ... Or a bit more verbose version: # More verbose list git annex whereis whereis _assets/images/posts/2019-02-27-file1.jpg (3 copies) 2a241183-7b15-433d-af55-ddecbe876408 -- [nas] c9501338-8824-4d32-82e9-4e9039590943 -- [gdrive] debde584-293f-4bfc-a024-b28bdeb7a00d -- [email protected]:~/workspace/travel-blog [here] ok whereis _assets/images/posts/2019-03-04-file2.jpg (3 copies) 2a241183-7b15-433d-af55-ddecbe876408 -- [nas] c9501338-8824-4d32-82e9-4e9039590943 -- [gdrive] debde584-293f-4bfc-a024-b28bdeb7a00d -- [email protected]:~/workspace/travel-blog [here] ok ... Conclusions # I hope this little git-annex tutorial will be useful to some of you (or at least to me when I need to remind myself how to use it next year). If you have any additional tips for using git-annex, please leave them in a comment. I'm sure I'm still missing a lot of cool features hidden in the corners of the documentation. Would I use git-annex for important projects where being able to retrieve old versions of files is actually crucial? Probably not. git-annex is a good tool, but it's also a complex one. I'm sure some smart DevOps engineers find it easy to use git-annex efficiently without worrying about losing any data. But I'm not that kind of a person. I'm comfortable using git, but I don't have enough experience using git-annex to use it for anything else but a hobby project. If I mess it up, I can always recover the original photos directly from my phone. What else would I use instead? Surprisingly, there are not that many great options - probably because version control of large files is not a common problem (except for the game dev industry, where you need to version your assets that can take hundreds of GBs). For many years, the main alternative has been a commercial tool called Perforce (no matter if you asked that question 8 years ago or last year). But if you do want to relinquish the control and let git-annex take the wheel, you can stop reading here and check out this article on how to set up git-annex with two commands: https://bryan-murdock.blogspot.com/2020/03/git-annex-is-great.html. ↩︎
More in programming
Download Now View on pouet.net Intro In the late 80s, I spent an embarrassing number of hours hunched over my ZX Spectrum, typing in Sinclair BASIC programs and rudimentary assembler routines copied from magazines, trying to re-create the Amiga crack intros I’d seen round at a friend’s house. Obviously, they never came even slightly close - but now, many decades later, I’ve finally managed it! I’m now the proud author of a hand-rolled retro demoscene intro that runs on a stock 128Kb Spectrum. It features all those late 80s tropes: A smooth scrolltext (with ‘leet custom fonts!), a three-channel AY soundtrack, bouncing VU meters and the essential parallax starfield. It’s currently 10287 bytes of code, or 19552 bytes as a complete tape image - which, given I’ve been working on this since December 13th last year, means my average rate of code-production is a paltry ~ 455 bytes per week! This blog post is a technical overview of how I got there, some cool tricks I picked up, and everything I had to learn (and re-learn) about the Speccy and Z80 assembly along the way. It simultaneously rates as one of my favourite side-projects ever, and also the one that most often nearly triggered a foot put through my monitor in frustration. I’ll reflect on all that at the end, but for now if you just want to see the finished product, the download button is at the top of the article and the included .tap cassette tape image should load into any emulator. There’s also a YouTube capture, links to the production on pouet.net, and you can run it from within a web browser by visiting my TNFS site portal - check option 4 for recent uploads. Be Kind, Rewind But let’s skip back briefly to the late 80s. Coming from the world of an 8-bit, cassette tape loading computer, seeing an Amiga for the first time was like encountering something from outer space. Like everyone who saw one back in the day, I was blown away by the amazing graphics and sound on display, not to mention the library of ground-breaking games. But I found myself even more captivated by the mysterious (and totally badass) “intros” displayed at the beginning of pirated games. Created by the software piracy groups that had cracked the copy protection and distributed the floppy-disk based games, they were mini works of art in their own right - music, graphics and usually some cool-looking effect to show off the group’s prowess. There’s a great collection of these intros (or “cracktros”) archived on YouTube. Watch a few of those videos and you’ll quickly notice the “design” tropes that almost every cracktro shares: a cool, edgy logo at the top of the screen (bonus points if it’s dripping blood or in any way “metal”), some boastful text displayed on top of an effect of some sort - usually a scrolling parallax starfield, or rotating 3d object - and a further scroll-text full of shout outs, greetings to fellow pseudonymous crackers and adverts for illegal BBS systems galore. Needless to say, to a kid in the 80s/early 90s this was possibly the coolest thing ever. I thought this one was the absolute bomb when I first saw it. Sine-scroller, pirate BBS adverts, cool logo with a sword - and of course, a starfield. There’s something for everyone here. The music from this is burned into my brain. I think pretty much everyone of my classmates knew this intro given the popularity of Lemmings in the early 90s. Skid Row as a group are still active to this day and this is probably the quintessential Amiga cracktro of the period. Endless Piracy. Edgy. Another cool logo, sine-wave scroller, starfield and boasting text. The classics never go out of style. Another great combination of all the essential ingredients. Cool logo, even cooler text font, scrolling starfield and a scroll-text running along the bottom. Vintage stuff. Logo. Text. Scroller. But no starfield - otherwise a solid effort from Anthrox here :) The demoscene grew out of these intros. The cracking eventually faded into the background, and coders, “graphicians” and musicians all contributed to feature-length productions that showcased their abilities, competed with other groups and pushed the machine to its limits. In amazing feats of ingenuity and pure “code as an art form”, the scene lives on today, with groups producing breathtaking productions like this and this - bear in mind, both of those are running on a 7Mhz home computer equipped with a floppy disk, 1Mb of memory and a sound chip from 1985. But much as I admire the sheer skill and dedication that takes, it’s the early “single screen” scrolly-text-and-starfield cracktros I’ve always found captivating. They took on a sort of mystical quality for me, hinting at a scene and a culture I could only ever experience from the outside - which is exactly why I spent all those hours in Sinclair BASIC, trying in vain to re-create them, and dreaming of the day I’d eventually have an Amiga of my own. The Jump Back I eventually did trade up to a range of Amiga systems, joined a few “scene” groups myself and threw myself into a variety of activities until exams and the end of my teenage years beckoned. I dabbled with Z80 code back then, and at one point knew enough 68k assembly to bluff my way through - partly from the Amiga’s heyday, partly from a few “embedded systems” modules at University where I had to program a set of traffic lights or lift-controllers. Given there’s not much call for that sort of thing at my $DAYJOB, though, it had all pretty much faded from my memory. I have kept working on a bunch of projects in my spare time, and one of my favourites is my TNFS site - sort of an old-school BBS system or interactive “website” for the Spectrum, which grew out of a fun experiment to see how far I could push Sinclair BASIC. It relies on a modern network card fitted to the Spectrum, which is how I came to be chatting to “Desertkun”, another TNFS-protocol enthusiast. He’s now producing a great new WiFi network card for the Speccy, and was kind enough to send me a few of the prototypes to test out for him. I was massively inspired by his hardware and low-level coding chops, and during one of our video calls where we’d been comparing our development setups, he asked why I didn’t write any Z80 assembly myself. A throw-away comment, but the challenge was set! Now, I couldn’t face re-writing the whole site in asm, but I figured I could produce a self-contained intro to promote it in the style of the early “BBStros”, and finally be one of the cool kids with a scene production of my own :) I mean, how hard could it be ? (ooooh - foreshadowing!) Dev Environment My development environment through this was pretty straightforward, and is the same stack I use for developing my TNFS site. I just used a Makefile-driven build system which ran my code through the Pasmo assembler, my fork of zmakebas to build BASIC files, and the FUSE emulator for testing. For final testing on the real hardware, I uploaded my .tap files to my local TNFS server where I could run them on my +3. A few additional tools I have to call out because they ended up making a big difference - one is the amazing web-based “asm-editor” I discovered while looking up Z80 instructions. Not only does this site have a great reference of all the Z80 instructions, but it also has an editor where you can run example code and see how the memory and/or various registers change state. Take for example, the djnz instruction - you can build it and step through the example loop construct one instruction at a time. Or try my example of an add instruction in the screenshot to see how the carry flag works when a bit gets shifted out of a register. Although it uses a slightly different syntax to Pasmo (so I couldn’t drop my code directly in it), it massively helped when manually stepping through things. And lastly, probably the most significant debugging tools I used on this project - a notepad and pen, and my two dogs (say “Hi!” to Biscuit & Pebble…) I’ll write some more about this in the wrap-up, but this really was a highly unusual project for me, coming from decades of much higher-level languages and constructs. Most of my z80 code I only ever really understood after drawing out the bytes and bitmasks, keeping a table of register flags and drawing chunks of screen memory out by hand. And I found my evening walks with the two dogs provided ideal rubber duck exercises, although I’m not sure what they both made of my protracted mutterings about the hl register pair. Either way, debugging-by-dog is highly recommended! The Starting Point So, let’s get started. I followed a great “Hello, world!” example from futurice.com to validate my build environment. Beyond that, the first thing I wanted to do was load in my logo and background image I’d use as a “stage” for the remainder of the planned intro. I wasn’t really sure what I was going to do at this point, just that I wanted my logo at the top, and a scrolltext along the bottom row of the screen. As my TNFS site header already looked kinda demoscene-y, I made a quick edit putting in place two blue guide-bars inside which I’d put further content. Memory map courtesy of breakintoprogram.co.uk A Spectrum SCREEN$ file (usually displayed as loading screens) is just a dump of screen memory, starting from byte 16384, and running for 6912 bytes. The weird layout of this is very relevant later, but in order to display it all you need to do is load your file into that memory location. After a few hour’s work of reviewing Speccy internals and following various guides, I had a very simple starting point for my demo that served as the basis for everything else that came after. org $8000 screen EQU $4000 ; address of the first byte of screen memory, 16384 in dec scrlen EQU 6912 ; total length of the screen memory inc. attributes ula EQU $FE ; I/O port for the ULA to control the border black EQU 0 ; Spectrum colour number, the exact same ones BASIC uses start: ld a,black ; Set the border colour to black using the ULA out (ula),a ; Copy my logo header picture to screen memory ld hl,header ; HL = location of my screen file in this code ld de,screen ; DE = start of screen memory ld bc,scrlen ; BC = 6912 bytes to copy ldir ; block copy, see https://jnz.dk/z80/ldir.html ; busy loop - rest of intro goes here :P hold: halt ; sleep until the next interrupt (1/50 second) jr hold ; then jump back and sleep again, for ever header: incbin "assets/header.scr" ; my logo header/background end start Not much to look at now, but this simple starting point introduced me to important concepts like interfacing with the Spectrum’s hardware (ULA chip to start with, and AY sound chip later) via the out instruction, how to pull in binary files using Pasmo’s INCBIN directive, using the Z80’s block copy ldir instruction, the use of register pairs for 16-bit addressing, and simple branching to labels. Hello World The next step was to print something on the screen and explore some of the Speccy’s built-in ROM routines, which are somewhat analagous to syscalls on modern operating systems. There are plenty of disassemblies and guides to the Spectrum ROM available on the internet, one of the best commented and linked one is at https://skoolkid.github.io/rom/. Using this as a reference alongside the various “Hello, world!” tutorials showed me how this sort of thing is built up: at EQU 22 ; "at" control character used to position the cursor row EQU 11 ; coordinates where to print the message col EQU 9 prstring EQU $203C ; where the pr_string routine lives in ROM ld de,msg ; DE = address of the first character ld bc,msglen ; BC = how many characters to print call prstring ; print the lot ; busy loop is the same... msg: db at,row,col ; equivalent of "PRINT AT 11,9" in BASIC db "Hello, world!" msglen EQU $-msg ; 16 bytes in all, pasmo shortcut, $=program counter I ended up mostly not using the ROM routines in my code as they dragged in all sorts of things I didn’t need, but it was massively useful to understand how they worked as they introduced me to yet more commonly-used constructs. For example, the PR_STRING routine defined here shows how you can build a loop over each character. If you’re just starting out, I highly recommend going over the 8 lines in that routine with a notepad and pen, working out the contents of the registers and following the loop constructs. I also made a version of it in asm-editor you can try. If you click that link, you’ll see a version of the code loaded into the web editor. Click “Build”, and then step through the code using the “Step” button - although no text will be actually displayed (I skipped over the call to the Spectrum ROM “restart” vector that does this), you’ll see the various states of the registers and for example the BC register counting down as each byte of the string is processed. Adding Some Style Problem is, the Speccy’s built-in font is not exactly dripping with demoscene credibility. I wanted to use some stylish fonts and this turned out to be super easy - a quick search turned up Jim Blimey’s awesome font collection, which included a ready-to-use set of assembler files and some instructions which soon led me to these additions in my code: ; CHARS is the address the ROM's printing routine reads to find the font. ; Picking our own font is simply a matter of over-writing this chars EQU $5C36 ; from Jim's docs - POKE 23607=5C36 in hex ld hl,myfont-256 ; HL = font address - 256 as per docs ld (chars),hl ; from now on the ROM draws with it! ; rest of code is the same myfont: include "textfont.asm" ; "Cartoon" font from jimblimey.com There we go! Now that’s looking a little more elite. Sorry, I mean 31337. Strap in folks, we’re leaving lamer city… So at this point, a month or so in to my dabblings and I’d started to get a reasonable understanding of common idioms and was starting to feel like this was a project that might actually go somewhere. The problem started when I moved on to trying to implement actual demo effects: It became very apparent, very quickly, just how much I didn’t know about how this all worked. Not to mention the fact that at this level, you’re really closely exposed to the physical hardware intricacies, including the design decisions and trade-offs that the hardware designers had to make when working with the extremely limited specs of the early 1980s. Cursed Screen Layout The first such problem - and the cause of probably 90% of my pain - is the Speccy’s extremely weird screen layout. Rather than standard bitmapped display you might expect on more modern hardware, it is instead split into 3 sections, each with a set of interleaved rows (so if you keep adding 1 to a screen address, eventually instead of reaching the next row, you’ll instead end up 8 rows down, and so on until you hit the bottom of the section and then loop back to the row underneath where you started!) followed by a block of “attribute” bytes which determine the colours of each 8x8 block. The attribute blocks are the reason why each 8x8 character cell can only have two colours (a PAPER and INK attribute) which led to the garish “colour-clash” of spectrum games and distinctive style where artists try to keep inside 8x8 blocks as much as possible. But it was the layout of the bitmap portion that is such a mind twister. Download the video You can see this illustrated whenever a game reads in a loading screen from tape. You can see each section built up with the interleaved rows one at a time, and then finally the colour and other attributes (like flashing ). There’s a lot of detail at the fantastic Break Into Program site. In the defense of the designers, they were working with extremely limited hardware and I don’t think the Speccy was ever imagined as a games machine. When you start shuffling bytes around, you realise that this approach is really optimised for displaying text which is neatly arranged into 8x8 character boundaries. For example, using this “scrambled” bit order, if you have a screen address in the hl register pair, you can quickly get to the next pixel line of a given address by simply incrementing the “high” byte. There’s a bunch of other “cheap moves” you can make by simply shifting bits around: Move Instruction Right 8 pixels inc l Left 8 pixels dec l Down 1 pixel line inc h Up 1 pixel line dec h Down 1 character row add 32 to l It was a reasonable choice for a machine that would probably spend a lot of it’s time displaying BASIC program listings - it’s only when you start trying to write demo or sprite routines that it starts to feel really weird. This also means you’ll pretty quickly need a routine to convert X,Y addresses to the relevant memory addresses. Again, L Break Into Program has a useful routine and if you disassemble more or less any spectrum game or demo, you’ll find a version of this procedure somewhere. And yes, this all became horribly relevant later on… The Scroller Download the video Admittedly, not the most interesting demonstration - but this was the first time I had made something move on the Spectrum’s screen with assembly! I watched this with pride more times than I care to admit :) I had a snazzy logo and “Hello World” text, but that doesn’t make for a very interesting intro! The first effect I wanted was the scrolltext - without that, it’s not really an intro. Now, the Amiga had all kinds of hardware sprites, blitter operations, custom chips but the Spectrum had none of that. I was going to have to work out how to do things the hard way - “banging on the metal”, literally by shifting bits around the screen memory by hand and having the CPU do all the work. This is where understanding the Speccy’s weird screen and character cell layout is so important. It turns out there’s a neat trick as long as I stay inside one character row - there’s a RL instruction, which takes a byte and “rotates” it left. It inserts whatever is in the carry flag into the rightmost bit, and whatever gets shifted out ends up in the carry bit again. That means if you string a bunch of them together, you can move a set of pixels across the screen - the moment they “fall off” one character cell, they end up in the carry flag and then the next rl instruction inserts them at the beginning of the next character cell to the left. Repeat 32 times, and you’ve just moved a pixel (or set of pixels) across a line of the screen: linestart EQU $4880 ; leftmost byte of the line 12 rows down lineend EQU linestart+31 ; rightmost byte column ($489F) linelen EQU 32 ; one whole line is 32 bytes (character cells) across dot EQU %00000001 ; the dot "shape", just one pixel lit at the far right end newdot: ld a,dot ; load the dot shape (only rightmost pixel lit) into a ld (lineend),a ; store it in the rightmost byte of the line, pixel is now lit frame: halt ; wait for the next frame, otherwise it whizzes across too fast to see ld hl,lineend ; hl = last (rightmost) byte of the line ld b,linelen ; loop counter, 32 bytes to work through or a ; trick to clear the carry flag, or against itself leaves a register ; unchanged but clears various bits including the carry flag ; see https://jnz.dk/z80/or_r.html shift: rl (hl) ; rotate this byte left (00000001 -> 00000010) dec hl ; point hl at the next byte to the left djnz shift ; loop construct: b=b-1 and then jump back to shift label until b=0 jr c,newdot ; now we've shifted the whole line, so if the carry flag is set now it means ; the dot has just left the screen, so we jump back to the start jr frame ; otherwise it is still somewhere on the screen, so keep shifting! Download the video Hell yeah. It’s almost a proper scrolltext now! Wrapping this in another loop that increments h each time (remember - that gets us moving down a scanline) gets us moving an entire character cell. Here you can see it moving an entire row of my “Hello world” message. There was however a lot more work needed to make a proper scrolltext, with each letter appearing smoothly. I started off by just printing an entire character at a time on the right-hand edge of the screen - it worked, but was pretty ugly as you saw each new character appear at once in the right-hand side of the screen, scrolling 8 pixels over and then the next character flashed up. Instead, I wanted to make each character appear pixel by pixel and that required a bit more bit juggling. The routine I ended up with effectively streams each character in, line by line, pixel by pixel using the font data directly instead of calling the ROM print character routine. It does this by looking up the ASCII code for a character, and then fetching the corresponding font representation into a 8-byte buffer, one byte for each line of data. The font data starts with the space character (character code 32), so a lookup is simply font code base address + (ascii code - 32) * 8 to get to the bits for that letter. Arranged into an 8x8 grid, the bits for the letter H in the font data look like this. For each row of bits, I shift the line left using a neat trick. The byte is loaded into the a register, and then by adding itself to itself - add a,a we shift the value left (in binary, multiplying by 2 moves everything over one bit) so that the pixel I want to draw ends up dropping into the carry flag. This is an idiomatic way of doing it in Z80 code as it’s faster and smaller than the sla instruction. And every cycle quickly adds up when you’re re-drawing 256 bytes every frame! The character buffer then looks like this, with the top line shifted. Now, with the hl register pair pointing at the correct location on the screen, I can then use the rl trick discussed above. This shifts a scanline from the bottom character row of the screen left by one pixel and draws my pixel in the rightmost position by moving it in from the carry flag. Looping 32 times over this moves the entire row by 1 pixel with my new pixel inserted at the right. I then repeat this step again using inc h to move down one scan line until the entire character column has been drawn. At this point, the character buffer looks like this, and all 8 lines of the bottom screen row have all been shifted left by one bit by looping over all 32 character columns. The whole thing then repeats another 7 times until the entire character has been smoothly scrolled onto the screen. So just to re-state this because I think it’s worth the emphasis - just to scroll a message across the bottom of the screen, we’re having to loop over and re-draw 256 pixels every frame! It shows how much faster assembly is than the BASIC interpreter built into the spectrum’s ROM. I found it amazing to see my code finally working and imagining all the bits flying around and registers flapping away. It really makes me appreciate just how much effort went into creating the games I played back then. Music The original Amiga module I converted. A proper 1980s ear-worm! Of course, any good intro needs some music. On the Amiga, the standard was the “tracker” module format, which has a long and involved history of different forks, versions and tweaks. A song is built up from a series of patterns, each made up of rows of instructions. Each instruction usually includes note or pitch information, which sample to play and various parameters such as volume, effects - or in the case of some instructions, which position in the song to jump to. In many ways, it’s almost like a programming language for music. Using ProTracker as an example, a typical note cell is 4 bytes, shown on-screen as something like C-2 01 C20. That breaks down into three fields: the note and octave (C-2), the sample number (01), and the effect and its parameter (C20 - in this case, “set the volume to $20”, which is half of the maximum $40). A close-up of Protracker playing a tune. Note the VUmeters - another classic demo effect I would pinch later on! For an impressive demonstration of what’s possible with a tracker and to see what the note data looks like on-screen, check out the work of h0ffman, and in particular his winning “Professional Tracker” entry from the Revision 2014 competition. He’s an undisputed master of the science (or art!) of tracking, and his “Secrets of Protracker” talk gives a great overview of what’s possible. The Amiga’s sound hardware far eclipsed anything else on the market at the time, and the ZX Spectrum is paltry by comparison. The original 48K models only offered a rudimentary “beeper”, but the later 128K models included a 3-channel AY-3-8912 sound chip. This was very popular in 8-bit micros of the time, and a more-or-less identical variant was also used in the Atari ST. While it only offers basic waveforms and sound processing, it does enable multi-channel “chip music”, and so it became the mainstay of the ZX Spectrum demoscene. It also spawned a variety of SoundTracker clones on the Spectrum itself, although most musicians these days prefer to work in modern tools like Vortex Tracker II, which runs natively under Windows. The Conversion Pex looking back on his demos - this starts at the point where he features my letter and plays a bit of my cover version! For my intro, I wanted to port an Amiga module - and I had the perfect one in mind. One of my favourite Amiga music disks was “Sounds Of Gnome”, released in 1989 by the Swedish duo “Mahoney & Kaktus”. Mahoney also wrote NoiseTracker, one of the first really widely used trackers on the Amiga, and the program that served as my teenage introduction to making computer music of my own. Back in 2018 I made a rock cover of one of the other tunes on that disk, and wrote to Mahoney (real name “Pex”) about it. He ended up showing my “music video” at a demo conference, which I am eternally grateful for! So it seemed only fitting that I mangle another song from that collection with a Spectrum conversion… Sorry, guys! I picked the track “Milkman” which was written by “Glue Master”, an associate of Mahoney’s who had a few of his tracks featured on the compilation. It’s an ideal track for converting: It’s short (only 7 patterns in total), most of those patterns share the same chord sequence, and it looked fairly easy to squeeze the original 4 channels down into the Spectrum’s 3. Plus, it’s an incredibly catchy tune and I knew it well - I’ve been listening to it for 35+ years and often jam along to it on guitar or bass. The conversion started off pretty easily: all I really had to do was copy the patterns note-for-note from the Amiga mod, which I had open in OpenMPT as my Windows tracker. Below is a session with both trackers open on the same pattern - the original Amiga mod on the left, and my 3-channel Spectrum conversion on the right. Note the 11.7 in the 3rd channel of the Spectrum version. This is a “pitch slide down” effect in Vortex Tracker syntax, which drops the pitch by about a quarter-tone. It’s repeated on the next line, so the overall effect is a slide down from the A# that leads in to the next bass note. The original Amiga mod does the same thing using its own syntax - those are the 202s you can see on the same notes, at the bottom of the 4th track. Using just a single “beep” sound for everything was enough to check everything was working, and after a few weeks of hacking on it and re-arranging things in the evenings, I had a basic version which sounded surprisingly good for a first draft: Download the MP3 instead. I have to give a special mention to this forum post which was a massive help getting me started. It’s probably the best introduction to Vortex Tracker I’ve found - many, many thanks to the original author for taking the time to write it all up! The real challenge was making it sound as close to the original as possible. That meant building up a library of better-sounding samples for the different parts, and making extensive use of Vortex Tracker’s “ornaments”. Vortex Tracker comes with a great selection of sounds, and fortunately I’m also a weirdo who likes to listen to chiptunes in the background while I work! So whenever I found something particularly enticing, I filed it away for later pilfering inspiration. A Vortex Tracker sample isn’t really a “sample” in the Amiga sense of a recorded sound - it’s a definition of how a note sounds, one line per frame. Each line can mix in a tone or the white noise generator (useful for snare effects and suchlike), along with various other parameters. Ornaments, on the other hand, give you extra control over what pitch a note plays - and this enables really useful tricks like “arpeggio chords”. For example, an ornament of 0,4 makes the channel switch rapidly (25 times a second) between the note being played and the one 4 semitones above it. It happens so quickly that it sounds much like two notes played at the same time, which means you can build up full-sounding triad chords using only two channels. Take the opening chord in my fully converted version, Cm: It plays C in the first channel and the minor 3rd (D#) in the second, then I apply an ornament to the second channel to bring it up 4 semitones to the 5th (G). That gives all three notes of the chord, while keeping the 3rd channel free for other parts. As well as making much more efficient use of the limited 3 channels, this also gives Spectrum music that characteristic “warbling” sound, which I find really appealing in a retro kind of way! Between tricks like this and samples that mix tones with white noise (to make a combined “bass-snare” sound, for example), I managed to condense the full track down to 3 channels. Many months later, I have to say I’m really pleased with the final result. Playback The original site that was home to Vortex Tracker, documentation and other resources appears to be offline at the moment. I managed to grab a copy of everything I needed from archive.org - you’ll want the PTxTools.7z file for the playback routines discussed here. The playback was simple in theory: As this overview notes, you basically just pull in the player routine, include your module file and call the playback routine once per frame in your main loop. The catch is, none of the source code versions supplied in the archive worked with my Pasmo assembler. Fortunately, there is a pre-compiled binary object shipped alongside the source (VTII10bG in the PTxTools/PT3Play directory). This just has to be loaded in at address $C000 - this cannot be changed, as it has lots of absolute addresses baked in. My code to handle module playback now looks something like this, with the binary module loaded in place: ; Define the start address of the player routine and functions we need player EQU $C000 ; hard-coded org address of binary player plinit EQU player+3 ; init function in docs, module address in hl plplay EQU player+5 ; play and move on one frame ; from the docs - this "analyzes PT3 module header and prepares ; corresponding note and volume tables" ld hl,moduleimg call plinit ; main loop hold: halt call plplay call scroll ; other effects / routines go here... jr hold moduleimg: incbin "assets/music.pt3" Starfield Download the video Well, that didn’t work as hoped. TNFStro - tortoise edition. This brings me to the main event. No self-respecting cracktro is complete without a starfield! My first, naive approach to the classic starfield effect was to scatter random stars across the screen, then move down each scan-line applying the same rl trick I’d used for the scroll-text. I figured I could get parallax by shifting each row a different amount: One pixel for the first row, two for the second, three for the third, then back to one again, all the way down the screen to give the impression of several “fields” of stars moving at different speeds. It turned out there were two major problems with that approach. Firstly, shifting an entire row smeared whatever else was on it along with the stars - which I’d expected, and had filed away as a problem for future me. The bigger issue was speed: by the time I’d finished the first three rows I’d already performed hundreds of shift operations, and it was nowhere near fast enough to fit inside a single frame. Everything crawled across the screen, and the music playback slowed down so much that you could actually hear the arpeggio ornaments warbling away as separate notes. Clearly I was going to have to keep track of the individual stars instead. I found this routine, which was more-or-less what I had in mind and started me off with the idea of a structure to hold the state of each star. It also uses a screen address lookup much like the one from “Break Into Program”. Download the video Dammit. Back with another funeral dirge rendition :( That got things moving, but the co-ordinate lookup and tracking proved expensive to do (it has to run for every star, every frame as well as keep track of the old position for erasing) and I’d been ignoring the bigger question how to display stuff on top of the starfield. My original idea was simply to draw the stars and then paint text over them each frame, which blew the frame budget just as comprehensively as my first attempt. If I wanted something more interesting than an empty starfield, I’d have to keep track of which cells were occupied by text and avoid drawing stars in them in the first place, rather than over-painting them every time. After lots of scribbling in my notepad and visualising rows of bits, I eventually solved both challenges by “cheating” fairly liberally. So now, I don’t generate a random starfield at all. Instead, I built a table of pre-calculated Y and X position pairs: stdata: defb 109, 25, 88,182, 42, 96, 155,226 defb 59,254, 137,169, 160, 84, 125,193 ; ... and so on, for 32 stars ... This saves faffing about with an RNG, and once I have a layout that looks good I know it’ll always look good. I can also “fudge” things and ensure no two stars share the same y co-ordinate which sidesteps a whole class of problems like stars sharing a scanline, and having to deal with them overtaking each other. The “random” layout is generated by a simple Ruby script: #!/usr/bin/env ruby PAIRS = 32 PER_LINE = 4 # pairs per line Y_RANGE = 40..175 # exclude rows taken up by my logo & bars X_RANGE = 0..255 raise "y range too small" if Y_RANGE.size < PAIRS # no duplicate y values ys = Y_RANGE.to_a.sample(PAIRS) pairs = ys.map { |y| [y, rand(X_RANGE)] } pairs.each_slice(PER_LINE) do |row| body = row.map { |y, x| format('%3d,%3d,', y, x) }.join(' ') puts "defb #{body.chomp(',')}" end When the routine initialises, I convert those co-ordinate pairs into a table that tracks the state of each star. So rather than storing a position and translating it to a screen address every frame, each of my star records now looks like this (offsets from the base address): STSCR EQU 0 ; screen address of the byte the star is in, e.g. $4123 STOCC EQU 2 ; a pointer to the "occupancy check" table cell STMASK EQU 4 ; the star's bit pattern, e.g. %00010000 STSTEP EQU 5 ; "speed" - num. of pixels this star travels per frame The key thing is that there are no X/Y co-ordinates at all. For each star I store the screen byte it sits in and the bit mask within that byte, so it’s already in the form the screen writes need. That means no address conversion per frame, just simple bit shifting and rotation. Moving a star is then dead simple. I rotate its mask left with rlca, and when the bit rotates out of bit 7, it lands back in bit 0 with the carry flag set. That’s the signal I need to move the star one character cell to the left, which is one of the “cheap moves” from earlier. Because I’m tracking which cell each star is in, the collision check comes almost for free. That’s what the second pointer is for: STOCC (Star Occupancy) moves in step with STSCR through a table holding one byte per character cell. The “typewriter” routine sets a byte to 1 for every cell containing a text character or the cursor, so checking whether a star is hidden behind text is a single or instruction. If the cell is occupied, I skip drawing the star but carry on rotating and stepping it along until it reaches a cell where it can be displayed: ld l,(ix+STOCC) ld h,(ix+STOCC+1) ld a,(hl) ; A = the occupancy check for this cell or a ; test A against zero jr nz,stmove ; it's occupied, so leave the screen alone Parallax is still handled by moving each star a different number of pixels per frame, but the cost is a lot lower this time as I’m only touching the cells the stars actually occupy, rather than every byte of every row. Download the video That’s looking more like it! The typewriter routine is a simple copy of the font data from memory, and is very cheap as we can use the “cheap moves” to quickly fetch the 8x8 block. I also skip drawing any space character so it leaves the “occupancy cell” unmarked. This means the stars keep drifting through the gaps in the text - you can see here the odd star passing in between the words. I switched to ALL CAPS writing for the final version which helps the effect even more as the font is quite blocky, so you don’t really notice stars skipping the spaces inside a glyph at all. Misc Notes A few odds and ends that didn’t warrant a section of their own… VU Meters Some sort of graphic display bouncing along with the audio playback was another common early demo trope. I did originally consider a real-time graphic equaliser, but that would have been far too complex, probably in terms of spare cycles per frame, and definitely in terms of my limited mental capacity at this point! So I settled for a simple set of bouncing meter bars instead. It’s worth a mention mostly because it gives me an excuse to link to the awesome chibiakumas website! I just love the late 90s web design on display - one can argue that readability and accessibility take a hit, but it’s just so much fun to encounter a site that still looks like this. It’s a clear labour of love and there’s a wealth of content there as well, if your eyeballs are up to it. Their guide on Making Sound with the AY-3-8910 was what showed me how to get the data I needed to draw the meter bars. The examples write a register number to $FFFD to select a register, and I realised that once a register is selected, I could use the in instruction to read its value back out instead of writing a new one. As the page says, R8, R9 and R10 hold the channel amplitudes, so they feed nicely into the height parameter for drawing a bar. Getting the amplitude for channel A is then as simple as: ld a,8 ; Channel A register call ayread ; ... rest of code here ... ayread: ld bc,$FFFD ; Register select port out (c),a in a,(c) ret Twinkle, Twinkle, Little Star This is a simple effect that makes the header logo a little more interesting. I’m not sure where I originally got the idea - I’ve definitely seen something like it on a few late 80s/early 90s demos, though. All I wanted was to pick random star dots in the logo background and animate them. Here’s the header image loaded in ZXPaintbrush with the character cell grid showing. By happy chance, there are four stars which each sit neatly inside a single character cell with enough surrounding space, so they can be used to give an animated “twinkle” effect by drawing a cross out from the centre point and back in again. These dots map to the following co-ordinates, which target the top scanline of each fully-grown star (two pixels up from the centre): twkdata: defb 1,5 ; char row 0, col 5, centre y=3 ink 5 defb 1,20 ; char row 0, col 20, centre y=3 ink 1 bright defb 26,7 ; char row 3, col 7, centre y=4 ink 4 defb 25,22 ; char row 3, col 22, centre y=3 ink 6 bright The “twinkle” routine itself is simple enough: it adds a pixel on either axis to grow the star, waits a few frames, then shrinks it back down again. The interesting bit was how I made the twinkle order pseudo-random. Remember that my starfield was entirely pre-calculated from a data table generated by a Ruby script, which meant I’d never actually needed a random number generator in assembly. From Jonathan Cauldwell’s “How To Write Spectrum Games” e-book, I found another great “hack” which simply walks through the start of the Spectrum’s ROM, picking out random-looking bytes as it goes. As he says, he uses it in most of his games, and it does the job in a really simple way - as long as you stay below the block of structured, repeating data that sits 8Kb into the ROM. twkrand: push hl ld hl,(twkseed) ld a,h and 31 ; keep it within the first 8k of ROM ld h,a ld a,(hl) ; the "random" byte inc hl ld (twkseed),hl pop hl ret Out Of Space About halfway through writing the intro, I hit a fairly predictable problem: I had run out of memory. In an attempt to keep everything self-contained in one block of code, I was pulling all of the assets (the screen background, the player binary, the music file etc.) into the program itself, and then using the ldir block copy instruction to move each one into its correct location. Since that meant effectively storing every block twice, and the player routine had a hard-coded start address of $C000, I quickly ran out of room and my code started crashing into the playback routine. The simplest option was to split things out into separate code blocks, load each one directly to the address it needed to live at, and wrap the lot in a multi-stage BASIC tape loader: 10 CLEAR 32767 20 BORDER 0: PAPER 0: INK 0: CLS 30 LOAD "" CODE 40 LOAD "" CODE 50 LOAD "" CODE 60 RANDOMIZE USR 32768 This loads the main program, then the player routine directly into $C000, and finally the background screen straight into screen memory. It does mean the background starts appearing before the demo has fully loaded, but as I imagine most people will be seeing this running in an emulator or on my TNFS site, it doesn’t really matter. Plus, at this point I was in danger of disappearing down yet more rabbit holes (I still really want to work out how to do multi-colour border effects!) and I’d never have got this thing released. The Zen Of Z80 This whole project is a really weird one to look back at. On the one hand, I’ve fulfilled a childhood ambition and have something I’m very proud of. It’s also pretty unusual these days to end up with a binary where you can say pretty much what every single byte is doing and the story of how it got there. But on the other hand - I now have an answer for why I don’t code in assembly: It’s because I value (what’s left) of my sanity! The thing is, in order to write code like this you have to start thinking like the machine itself. You have to visualise the registers and the status flags flapping away, graphics as lines of bits slotted into strange orders, loops and conditionals as binary logic and so on. Getting this close to the hardware definitely brought about a greater sense of attachment to the hardware of my childhood, and it’s easy to see why assembly coders and sceners get so passionate about it. That said, I’m in no hurry to start re-writing my body of work in assembly! The projects I really enjoy are those that give me an excuse to learn something for the sheer hell of it, and to slow down a little from the ever-increasing velocity and demands of the modern work world. Going back to Z80 code is probably about the furthest I could have taken this, short of going into hardware hacking. I found it at times an intriguing puzzle, often a frustrating exercise that nearly had me throwing in the towel at several stages. This project has certainly taken the longest to go from the “idea” stage of any given feature to any kind of meaningful output. True, I didn’t spend every day of the last year working on it - my actual coding time was probably only a few hours over a couple of evenings a week. But I’ve certainly been thinking about it and pondering it most days (as my dogs will attest). The biggest breakthroughs and “aha!” moments usually didn’t come from behind the keyboard; they came from scribbles in my notebook and thinking about processor states while doing something unrelated. It has however given me massive appreciation and respect for the coders who pull this stuff off and make it look easy. And that goes double for those kids and bedroom coding superstars who managed all of this back in the 80s, and had to work it all out by experiment themselves. I only managed to get this far because of numerous books, online tutorials, GitHub repositories with sample code, YouTube videos… I can’t imagine how much longer this would have all taken me if the only resources I had were my Spectrum, a cassette tape recorder and a notepad. So, as I draw a line under this project - for now! - I raise a glass, and in the words of h0ffman: “Shout out to the hackers, graphicians and protrackers.”
A decade ago, a little bit of history was made. I didn't realize it, but a colleague made a great point, one of those real mind-changing points that seem too obvious to admit same-day. But, the next day, calver.org was born. At the time my team maintained the Python infrastructure for eBay and PayPal, and we were stuck deciding whether we were really ready for a "major" 1.0 release. Semantic Versioning was the only game in town and "major" means "big", right?! Thankfully, a wiser colleague mentioned: Ubuntu and Twisted don't struggle with version number debates. They slap a date on it and keep shipping. In fact, their date-based versions were even better because you always knew where it stood, in terms of updatedness and support. The only problem is that no one really knew about it. Somehow, this problem solving versioning alternative, arguably as old as history itself, had gone nameless for millenia, conspiring to make me feel foolish in an office meeting. Never again! Ten years of adoption Fast forward 10 years, we've seen CalVer adopted by Apple, Nvidia, JetBrains, and countless others. (We have a timeline!) The site may have more inbound links than any other project of mine. Apple made the biggest jump, at WWDC 2025: iOS went from 18 to 26 macOS from 15 to 26 watchOS from 11 to 26 and visionOS from 2 to 26 All landing on one, consistent number like a car's model year. I still remember the texts from the Venn diagram fanbase of my friends who love Apple and reasonable versioning. No such texts from when NVIDIA announced calendar versions across the GPU Operator, RAPIDS, and its monthly NGC containers, but still very cool. Open source, too: Home Assistant, pip, CockroachDB, and yt-dlp all ship on dates, with plenty more on the users page. The conversation even reached the language core; PEP 2026 proposed versioning CPython as 3.YY, and it almost happened, too. And it's never too late, time marches on! Fixing the notation But I don't think I got every detail right from day 1. That's the main motivator for CalVer 26. It's high time to start righting a couple idiosyncratic token design choices, starting with some additions: Meaning Before 26.0 26.0 Full year YYYY YYYY Short year (6, 16) YY YY Zero-padded year (06, 16) 0Y 0Y Short month (1 ... 12) MM M Zero-padded month (01 ... 12) 0M 0M Short week (1 ... 52) WW W Zero-padded week (01 ... 52) 0W 0W Short day (1 ... 31) DD D Zero-padded day (01 ... 31) 0D 0D Seeing double First, the doubled letters. From the first version (16.6), MM and DD meant the unpadded month and day, which reads backwards to anyone who knows date formats (ISO 8601's YYYY-MM-DD, Java, moment.js, day.js), as some community members correctly pointed out. I was ready to flip them, until I checked what people actually use: most projects with a YY.MM.MICRO badge (conda, Twisted, Ansible's tooling) don't pad, and more than a dozen other version management tools (like bumpver and bump-my-version) implement the old meaning. So, it's too late to flip MM's meaning. Instead, 26.0 deprecates it and offers a more explicit and hopefully clearer option: M is the short month, 0M the padded one, and MM is a technically-retired synonym for M. In case you're wondering, the explicit 0M was me being overinspired by Ubuntu's approach, perhaps: 6.06 pads its month but not its year, and YY.0M says exactly that. To pad or not to pad I think it's worth a detour into why padding is even a thing anyways. It's become important now that new ecosystems have emerged that enforced SemVer formatting semantics, and I wanted clear guidance about on the spec site. SemVer forbids leading zeros outright, so Cargo rejects 26.04.0 and Go modules reject v26.04.0. Even Python's packaging spec normalizes leading zeros away, so you can tag 2026.08.19 if you want, but PyPI will still show 2026.8.19. NVIDIA's GPU Operator docs put it this way: "Zero padding is omitted for month to be still compatible with semantic versioning." CalVer was always intended to drop in where SemVer was used. So 26.0 recommends unpadded (YYYY.M.D) as a sane "pure" default for software libraries. But libraries are not the only objects of versioning schemes. The exception is a version that becomes a filename, an image tag, or an object-store key that gets listed lexically. There, padding keeps 26.10 sorted after 26.09, which is why Ubuntu, NixOS, and NVIDIA's own NGC containers pad. More evidence of teams designing their versions. We love to see it. Our FAQ has a longer discussion of the padding issue, as well. Optional segments There was never any rule against them, but 26.0 makes optional trailing segments more explicit with square brackets. Now, yt-dlp's scheme can finally be written down: YYYY.0M.0D[.MICRO]. For the CalVer badges I could find on GitHub, they all stay valid for now. I've got a note on the deprecated spellings and a new copy-paste badge section for new ones. What else is new? It's always a great time to add more citations to the site. A spec changelog; the spec now versions itself: 16.6, 19.7, 26.0. A FAQ: breaking changes, same-day releases, and padding. The users page, rebuilt by category, with past users of note (schemes change; that's fine) and tooling. Case studies: Apple and NVIDIA in, yt-dlp replacing youtube-dl. Much of the thinking behind these changes happened in the GitHub issue tracker over the years, and 19 or so issues close with this release. That's where ideas for CalVer should go, so by all means, open an issue, and we'll get it sorted! In due time, of course. In closing, I can't believe I still love belaboring these numbers so much. Thanks to all (but especially Mark, Glyph, Hugo, issue reporters, translators, and maintainers) for the discussion, and ultimately making the most timely versioning system a timeless classic. See also 2016 announcement Designing a version My Yap on Why CalVer beats Semver
Four sincere attempts at OKRs, four failures, and every time: "you didn't do it right." Maybe the framework doesn't fit you -- so how do you find one that does?
Multiple concurrent writers. Multiple processes. Same SQLite. No modifications.
Brilliant jerks, ZIRP-era managers, and how psychological safety lost the plot. Part 3 of my conversation with Dr. Cat Hicks.