Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
1
If you have functions that do a lot of mathematical operations, use NumPy or rely heavily on loops, then there is a way to speed them up significantly with one line of code. Ok, two lines if you count the import. Numba and the @jit decorator # Meet Numba and its @jit decorator. It changes how your code is compiled, often improving its performance. You don't have to install any special tools (just the numba pip package), you don't have to tweak any parameters. All you have to do is: Add the @jit decorator to a function Check if it's faster Let's see an example of code before and after applying Numba's optimization. # numba_testing.py import math def compute(): # Bunch of dummy math operations result = 0 for number in range(1_000_000): double = number * 2 result += math.sqrt(double) + double return result The only purpose of this code is to do some calculations and to "be slow." Let's see how slow (benchmarks are done with Python 3.8 - I describe the whole setup in the Introduction article): $ python -m timeit -s "from numba_testing import compute" "compute()" 1 loop, best of 5: 217 msec per loop Now, we add @jit to our code. The body of the function stays the same, and the only difference is the decorator. Don't forget to install Numba package with pip (pip install numba). # numba_testing.py import math from numba import jit @jit def compute_jit(): # Bunch of dummy math operations result = 0 for number in range(1_000_000): double = number * 2 result += math.sqrt(double) + double return result Let's measure the execution time once more: $ python -m timeit -s "from numba_testing import compute_jit" "compute_jit()" 200 loops, best of 5: 1.76 msec per loop Using @jit decorator gave us a 120x speedup (217 / 1.76 = 123.295)! That's a huge improvement for such a simple change! How did I discover Numba? # I first learned about Numba when I was doing code challenges from the Advent of Code a few years ago. I...
3rd Sep 2020

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Sebastian Witowski

The macOS Apps I'm Using

For a long time, the two most popular articles on my blog were lists of macOS apps and CLI tools, which I've been using. Coincidentally, these kinds of lists are something I love to read myself. I can always find some cool, new tool in such articles. It doesn't matter that I will probably immediately forget about this tool and never use it again. Nor does it matter that 80% of the items on those lists are always the same. The promise of being able to do some obscure task in a slightly more efficient way is all I need to get excited each time I find another "Best 15 MacBook apps for Python developers" article. However, those lists tend to get outdated pretty quickly. Every year or two, a new process manager, disk space analyzer, file manager, or other tool gets released that's faster, fancier, and more colorful. So, after almost five years, it's about time I updated my original macOS apps article and wrote down which tools I use now. I still use quite a lot of programs listed in the original article. For some, I found better replacements. And, of course, there are a few interesting new items. Some tools are paid (I marked those with a 💰 icon next to their name), and some have both a free tier and a paid one (I marked those with 🆓/💰). All the other ones are free to use. And, obviously, there are no affiliate links, and no one paid me to have their tool listed here - I use them because I like them. Table of contents # This is a long list, so here is a table of contents with short summaries of what each app does: Alfred - Spotlight on steroids Raycast - A more capable and modern alternative to Alfred Bartender 5 - An app for showing and hiding icons on the menu bar iTerm2 - A terminal app BetterTouchTool - An app for customizing all sorts of input devices for your Mac Karabiner Elements - An app for customizing your keyboard Magnet - A windows manager Monosnap -An app for taking screenshots and annotating them on the fly Kap - An app for recording videos of your screen QSpace Pro - A file manager Obsidian - A note-taking app for editing Markdown files Carbon Copy Cloner - An app for making computer backups Ferdium - A chat aggregator Todoist - An app for managing TODO lists Toggl - An app for tracking time when working for clients Velja - A browser picker Spark - An email client Figma - My preferred graphic editing tool 1Password - A password manager Nucleo app - An app for managing icons n8n - An automation tool (free alternative to Zapier) PDFGear - An app for editing PDF files The Unarchiver - An app for opening archived files OBS Studio - A video recording tool KeyCastr - A keystroke visualizer for showing which keys you press when recording screencasts Audacity - An audio editing tool HandBrake - An app for transcoding videos Captin - A tool that shows a popup when you toggle Caps Lock Balance Lock - An app for fixing the balance of the headphones MOS - An app for fixing smooth scrolling on a Logitech mouse noTunes - An app for preventing Apple Music from opening Other applications Apps I like but don't use anymore Apps I want to try Alfred 🆓/💰 # It's like Spotlight, but on steroids. I launch it with ⌘+Space, and then I can type some letters or numbers to quickly start applications, solve simple calculations, convert currencies, find files, and more. To make the most out of Alfred, you need to buy the Powerpack. It's a paid extension that allows you to use even more features, including the most important ones - workflows. Written mainly by the community, workflows are plugins that add additional functionality. Here are some of the workflows that I like the most: Calculate Anything - one of the workflows that I use the most. It allows me to calculate many more things than just some mathematical operations. It can do currency or unit conversion, operations on dates, and more. And it does a good job with processing natural language, so no matter if I write "100 euro to pln" or "100eur pln", I get the correct answer. Calm notifications - toggles the "Do Not Disturb" mode when I don't want to be distracted by any notifications. Coffee coffee - when enabled, it prevents your computer from going to sleep. I was previously using a free app called Amphetamine that does the same but with some additional customization options. And then I realized I didn't need any of those additional options. All I needed was to occasionally prevent my computer from sleeping. Fun fact - by looking at the source code of this workflow, I've noticed that it's calling the caffeinate command. It turns out that macOS has a built-in tool that can prevent your computer from sleeping. So, you don't even need any external tools. Check out caffeinate's manpage. DeepL Translate - for quick translations using DeepL. I don't use it for long translations, but mostly when I'm looking for synonyms of an English word. Emoji Mate - an emoji finder. I can type ; followed by the name or a description of an emoji, find an emoji that matches it, and copy it to the clipboard. UTF Search - a UTF characters finder (similar to Emoji mate mentioned above). I can type : followed by the name of a UTF character and then copy it to the clipboard. Gist - I use it to quickly create GitHub gists from the content of my clipboard. If I have some code copied in the clipboard, I can run the gist command, and this creates a GitHub gist that I can share with someone. I used this feature a lot when I was publishing on Medium.com, and I needed to create gists for my code, as Medium didn't have good support for code snippets. Network - show and change network settings. I use it to quickly get my IP address. Reminders for Alfred - add new items to the Reminders app. This is a very old plugin that is no longer available in Alfred's plugins gallery, so you have to install it from GitHub. And even though the author says that it's no longer actively maintained, it still works. I like this plugin because it's very good at parsing natural language. I can type "r 5 minutes leave home" or "r 10:50 leave home", and both commands will create a "leave home" reminder at a specific time. Temporary email - I can type tmpmail, and this workflow will create a temporary email with one of the various temporary email services. It's indispensable for all those "subscribe to our newsletter to get 10% off for your order" online shops, where I just want to shop once and never come back. TOD Todoist - lets me quickly add tasks to the Todoist app. I use this workflow very often, especially in the middle of a call, to add new tasks before I forget about them. So it's critical for me to have a plugin that can easily parse natural language, just like Todoist does (writing "buy milk tod p1" creates a "buy milk" task, sets the priority to 1, and the due date to today). When the workflow I was using previously (Alfred Workflow Todoist) stopped working, and the only replacement I found (AlfreDo) required me to learn some unintuitive syntax for adding tasks, I briefly started using a different launcher just to have my simple workflow for adding new tasks back. Luckily, later, I found "TOD Todoist" and got my simple way of adding TODO tasks back. alfred-toggl - I use Toggl to track the time when I'm working, and "alfred-toggl" allows me to interact with Toggl. I can start and stop timers or generate reports showing how long I worked today or this week. Raycast 🆓/💰 # Raycast is another launcher like Alfred, but it's much more actively developed and already much more capable than Alfred. On top of its vast catalog of features, there are many community plugins to further extend its functionality. I installed Raycast to quickly create new TODO tasks when my old Todoist plugin in Alfred stopped working. Nowadays, I use Raycast mostly for its Emoji Picker and Floating Notes features. Raycast looks very promising, and most importantly, it has a generous free plan. So, where's the catch, and why am I still using Alfred? Because Raycast is a VC-backed startup, and I'm not really convinced that their business model is sustainable. There is a paid Raycast Pro version that initially included AI features[1]. Now I see that the paid plan offers more features at the expense of the free plan (unlimited clipboard history, etc.) For now, the free version is fantastic. But I don't want to get vendor-locked when the money becomes tight and the free plan gets even more limited. Plus, I already paid for Alfred's Powerpack license, and I got used to using it. Alfred works fine for me, even though it's much less actively developed, and I wish it had more integrations with other tools (or at least stopped breaking community workflows when they release new versions). Bartender 5 💰 # Bartender lets you hide icons on the menu bar, so you can keep only the most important ones visible, while the rest can be displayed when you click Bartender's icon. Here's how my menu bar looks most of the time: And here's how it would look without the Bartender app: I bought Bartender to replace Dozer when I got a new MacBook with a notch because Dozer didn't work well with the notch - some icons were hiding under the notch, and I had no way to click them. Dozer has not been updated for quite some time, so support for MacBooks with a notch was never implemented. However, if I were looking for an app to hide menu bar icons today, I would first check out the free Ice app. Why? Because the owner of Bartender sold the app and never informed the users, which was a shady move. After the app was sold, it started including new features like a new analytics framework because, well, why shouldn't a simple app for hiding icons spy on its users, right? I hope by the time I write the next edition of a list of my tools, I will switch to something else, but for now, I didn't have the time to switch. iTerm2 # This is still my terminal app. It works great, stores its configuration nicely in my Dropbox backup folder, and I'm not planning to change it to anything else any time soon. New terminal apps come and go (warp looked promising until it became an "AI-powered intelligent terminal", took money from VC funds, and started collecting telemetry data by default), but for me, iTerm is perfect the way it is. BetterTouchTool 💰 # A feature-packed app for customizing all kinds of inputs for your computer. You can define keyboard shortcuts, mouse and trackpad gestures, control Stream Deck, manage your computer from your phone using BTT Remote, customize the content of the Touch Bar, and more. I initially bought it many years ago to customize the Touch Bar (as I described in the previous version of this article), and nowadays, I still use it to define some more advanced keyboard shortcuts. I have shortcuts that will mute and unmute my microphone (no matter if I'm using Teams or Zoom), pause and play music, start some of the most common apps, etc. I describe additional keyboard shortcuts in more detail in the You Don't Need Stream Deck, You Need Macros article. Karabiner Elements # A powerful and stable keyboard customizer for macOS, as its tagline says. And boy, it is powerful indeed. Karabiner Elements covers all my keyboard customization needs (although, for convenience, I still use BetterTouchTool for some more advanced keyboard shortcuts): Remapping keys Changing the behavior of function keys on specific keyboards (I want them to act as function keys on an external keyboard and, at the same time, act as media keys for MacBook's built-in keyboard) Creating complex mappings (pressing both Shift keys to toggle Caps Lock) Another great thing about Karabiner Elements is that it stores the configuration files as JSON. So I can save them in a git repository and have a well-documented history of all the changes I made. Magnet 💰 # A tool for managing windows with keyboard shortcuts. It allows me to move windows between monitors or split them into halves, quarters, or even sixths without having to drag them with the mouse. I paid for this tool only because I didn't do my research, and I didn't check if there were free apps that could do the same. But luckily, there are, and if I were looking for a windows manager for macOS today, I would use Rectangle instead. If you're looking for a more advanced tiling windows manager similar to i3 or xmonad, there is also Amethyst. Monosnap 🆓/💰 # I use Monosnap to take screenshots, crop, and annotate them on the fly. Then, I can send those screenshots to someone by simply dragging them into a chat app or an email. Before Monosnap, I used Skitch, but it seems Skitch is now abandoned. Even though Monosnap has a paid plan, the 2GB of storage in the free plan is generous enough that I don't think I'll ever become a paying customer[2]. Monosnap also supports recording videos of your screen, but I got used to using another tool for this. Kap # A simple and free tool for recording videos of my screen. The video can then be trimmed in Kap and exported to one of many supported formats (including MP4 and GIF). It's a great tool for when I want to share something happening on my screen. Even though the aforementioned Monosnap also supports recording videos, I use Kap out of habit. But if you're already using Monosnap to record videos, I don't think Kap has enough features to justify the change. QSpace Pro 💰 # For a long time, I have been looking for a file manager that supports multiple panes (which surprisingly is still not supported in macOS' default file manager - Finder). And I randomly found QSpace Pro, which works pretty well. It's a paid app, but it's cheaper than Forklift (another dual-pane file manager that some people recommended), and it doesn't require a subscription like the Path Finder (seriously, a subscription model for a friggin file manager?!). It offers plenty of keyboard shortcuts, and with plugins, I can connect to external services like Dropbox or Google Drive. Obsidian 🆓/💰 # My go-to note-taking app. I use it for all my note-taking needs: writing articles, creating mind maps of my presentations, planning my year, etc. I switched from Evernote to Obsidian long ago, even before Evernote became practically useless with the free plan. Obsidian uses Markdown files stored locally on my computer, so I don't have to worry about losing access to my notes or having to go through a difficult migration process if I ever need to switch to a different tool. I wrote a detailed article on how I use Obsidian: How I Organize My Notes With Obsidian. Even though Obsidian comes with paid plugins, I don't use any of them. Carbon Copy Cloner 💰 # This is my preferred tool for backups. I don't know if I did something wrong, but when I tried to use Time Machine with a non-Apple NAS, I was constantly running into issues. The backups were getting corrupted, and I was losing access to the old versions of the files. Carbon Copy Cloner solved all my problems with backups. I use it to back up specific folders during the night. The main selling point of making backups using Time Machine was that I could, in theory, set up a new computer from this backup[3]. But I know I won't need that. If I ever lose/break my computer, I will get a new one and install applications manually. Then, I will restore specific folders (those that Carbon Copy Cloner backs up for me every night). Once, I tried to set up a new MacBook from an old one using the Migration Assistant, but it copied so much unnecessary old trash that eventually, I ended up reinstalling everything from scratch. Ferdium # A chat aggregator. I use Telegram, WhatsApp, and Discord to communicate with people. And in the past, I used even more applications, including multiple Slack channels. Ferdium allows me to have all those different chat services in one app instead of starting a separate app for each one. What I don't like about Ferdium is that every service I add starts a separate electron application. With many chats, RAM usage goes through the roof. But I didn't find a workaround for this problem (except for not using so many chats). Regardless of this issue, Ferdium is still the best app I've used so far (after trying Franz, Rambox, One Chat, and - now discontinued - Ferdi). Todoist 🆓/💰 # I use Todoist to manage my TODO tasks, and I really like it. The free plan is still perfectly suitable for me because I created a lot of projects before the number of projects in the free plan was severely limited. Now, I can rename and reuse old projects without having to create new ones. If I ever run out of projects or need some paid features, like reminders, I will definitely pay for the paid plan instead of looking for a replacement, because Todoist is a very solid app. Toggl 🆓/💰 # A time-tracking app. Ever since I started freelancing, I've been using Toggl to create time sheets for my clients. The free plan is all I need from the app. I love the feature of stopping the timer after a predefined period of inactivity, so I never have to worry when I get distracted and leave my computer for longer without stopping the timer. Velja # It's a browser picker application. It lets you define rules and open links from specific apps or specific URLs in different browsers, not just the default one. I need it because I use the same computer for client and personal work and I want all the client-related links (JIRA, GitLab, etc.) to open in my "work" browser (which is currently Brave, because I need to be able to run Teams in the browser and Firefox was having some problems with that[4]). Spark 🆓/💰 # I think I'm slowly running out of built-in macOS apps that I didn't replace with something else... Before macOS Ventura, Apple Mail was missing one feature critically important to me - "send later". That's why I looked for an alternative, and I found Spark. I'm still using the old, "Classic" version of Spark, not the new one that is needlessly packed with AI features. This email client also solved some other small annoyances. For example, Apple Mail was occasionally messing up my flags - I used different flag colors for different emails, and sometimes it would randomly change the colors of the flags or put emails that used different flags under one flag. Pinned emails (that appear at the top of the inbox), "undo send", or "snooze" were all additional quality-of-life improvements that Apple Mail didn't have at that time. So I stayed with Spark mail until now. However, you should be aware of some privacy concerns regarding Spark. For example, it stores email credentials on its servers so that it can send emails later, even when your laptop is turned off. Seeing that Apple Mail now has most of the features I was missing before, I might give it another go one day. Figma 🆓/💰 # Even though Figma is considered primarily a prototyping tool, I use it for all my simple graphic editing needs. It's easy to use for simple tasks (adding text to an image, creating patterns from a few icons, etc.), and unlike GIMP, which I used before, I don't have to search for tutorials explaining how to do all those simple things. Don't get me wrong, GIMP is awesome, but I don't use it often enough to learn how to use it properly. And Figma is much more intuitive for casual usage. 1Password 💰 # My password manager since I switched away from LastPass a few years ago. Why am I not using the password manager built into macOS? Because my wife is not using Apple devices, and we share some passwords and important documents. There are some good alternatives to 1Password, like Bitwarden, that a lot of people recommend, but so far, I'm a happy 1Password's user. Its UX is very pleasant. I especially like that I can use it to store and fill in OTP tokens. Each year when my subscription is supposed to renew, I'm thinking about switching to something else, but I always end up extending my subscription. When it comes to something as important as a password manager, I just don't think it's worth replacing a tool that works with a completely new one just to save a few bucks. Nucleo app # I use Nucleo app to manage the icons I have saved on my computer (I have a library of over 10,000 SVG icons that I downloaded when I had a subscription at one of the premium icon services). Nucleo itself is quite an expensive icon bundle, but their app is free to use and works very well for finding a specific icon in a large pool of locally stored icons. Nucleo app looks very similar to the Iconset app I used in the past. But I moved away from using Iconset when it became a paid app. n8n 🆓/💰 # n8n is an automation tool - a fair-code distribution alternative to Zapier. Technically, this is not a macOS app, but it's a very useful app that you can install on your local computer or on an external server with a Docker image. I use it for simple things (like deploying this blog every morning to publish scheduled posts) and also some more advanced ones (like managing subscription-based access to a side project followed by sending Telegram notifications and emails that I described here). Currently, n8n is being developed by a company funded with VC money, and that rarely ends up well for free-to-use projects. If things go south, I'll switch to Huginn for my automations. PDFGear # Occasionally, I need to edit a PDF file. And I don't mean simply annotating it (which I can do with the built-in Preview app) but actually changing the text written in a PDF. Instead of paying Adobe a monthly fee for this feature, I found PDFGear - a free PDF editor with plenty of advanced features that fulfills all my occasional needs when working with PDF files. The Unarchiver # A free-to-use application for opening any type of archive on a Mac. OBS Studio # If you're a video streamer, this tool needs no introduction. OBS Studio is probably the best open-source recording and streaming software. I used it on a few occasions when I needed to simultaneously record a video from multiple input sources (screen-sharing plus camera). The most interesting use I found for it was when we were doing SCRUM Poker to estimate sprint tickets. Instead of waving my phone in front of the camera like a caveman, I connected a text document to my video feed. Whenever I typed a number, this number appeared over my head. It was a pretty stupid way to spend the CPU power (using OBS increases the CPU usage), but it was totally worth it! KeyCastr # A keystroke visualizer. I use it when I'm recording a screencast, and I want to display the keys I'm pressing. A very nice tool with plenty of customization options. Audacity # Every once in a while, I need to do some audio processing (crop an audio recording, remove background noise, or just convert a recording to another format), and Audacity is still the best tool for working with audio. At first glance, it might look overwhelming, but because it's been around for a long time, the internet is full of guides on how to do most of the basic processing. HandBrake # An open-source tool that can convert a video between different formats while performing some additional processing. I use it to perform resizing and removing audio from videos while converting them to a more space-efficient format. Install-and-forget apps # There are a few small applications that I installed one day, allowed them to run on startup, and basically forgot about them. They do their job perfectly while running in the background. Captin # A very simple tool that shows the Caps Lock status. I use a quite quirky way of enabling Caps Lock (pressing both Shift keys at the same time toggles the Caps Lock on and off), so I installed Captin to get a popup confirmation that Caps Lock has been successfully enabled or disabled. Balance Lock # If your headphones randomly change the balance settings (so the sound is audible more in one ear than in the other) - which is something that macOS is guilty of doing, Balance Lock is the app you need. It will automatically center the balance of the audio back each time macOS messes it up. MOS # I switched back from using a trackpad to a mouse, only to realize that Logitech's LogiOption+ app sucks. It turned out that to get a smooth scrolling experience for a Logitech mouse, I needed a 3rd party app. Luckily, I found a free app called MOS that solved my problems. Another useful feature it has is "per-application mouse scroll setting" that I use for the VPN application called Citrix when remotely connecting to my work PC (the default scroll settings caused scrolling in Citrix to be very janky, so I had to adjust them). noTunes # Another small utility to fix Apple's stupidity (oh, it rhymes!). This tool prevents Apple Music from launching when I press "play" on my AirPods if the Spotify app is not running. Because, you know, I have never EVER used Apple Music, but it totally makes sense for macOS to think that today is the day I finally want to play music from this app. Yes, macOS. Today is definitely the day when we play something from my impressive iTunes library of zero songs. Other, less thrilling applications # Some other, more well-known (but also more mundane) apps that I use include: Elgato's Stream Deck software - because I need to control my Stream Deck somehow. I'm considering switching to BetterTouchTool to control my Stream Deck, though. Inkscape - for when I need to edit an SVG file. I haven't edited an SVG file in ages, but if I were to do this today, I would probably try Figma first. f.lux - adapts the brightness of my screen depending on the time. Early in the day or late at night, it will make the colors warmer to make it less taxing on my eyes. TeamViewer - for those occasional times when I need to help my parents with something on their computer. Spotify 💰 - for listening to music. Visual Studio Code - my code editor. Everyone knows what VS Code is. VLC - one of the most popular video players for any operating system. I've been using it on every computer I have owned for as long as I can remember, and I never had any issues. But I've heard good things about Infuse, and I already use it on my iPhone, so maybe one day I'll switch to this app on my MacBook, too. Apps I like but don't use anymore # I've stopped using some of the apps listed in the previous version of this article. Even though I still think they are great, I simply don't have use for them right now: Anki 🆓/💰 - in my opinion, Anki is still the best app for creating and reviewing flash cards. I used it almost every day when I was learning Japanese a few years ago, and I loved it, even though the design looked crude and a bit outdated. The iOS app is paid, but the Android or web versions are free. I stopped using Anki because I'm currently not learning any new language. Amphetamine - an application that prevents your computer from going to sleep. I stopped using it because, in a rare instance, when I need to prevent my MacBook from sleeping, I use Alfred's "coffee coffee" workflow. It uses the built-in caffeinate script to prevent my computer from sleeping. Apps I want to try # Finally, there are a few applications that look very interesting, but I haven't found a use case for them yet. I'm making a note about them, and hopefully, one day, I'll give them a try: Orbstack - seems to be a nice and supposedly faster alternative to Docker Desktop. However, I currently don't use Docker for any of my projects. MeetingBar - a simple app that sits in your menu bar and reminds you of upcoming meetings. You can also join the meeting directly from this tool. It looks very cool, but most of my daily meetings happen in Teams, which I have to use through the browser or a VPN. If I ever get back to a standalone Teams client or start using different videoconferencing tools, I'll give the MeetingBar app a try. Bruno - looks like a pretty cool alternative to Postman for testing APIs. Thanks for reading and see you in the next edition of this list (hopefully sooner than in five years)! If you have suggestions regarding other awesome macOS apps I should try, or if you have your own list of favorite tools, please share them in the comments (or email me using the contact form). And if you're interested in the CLI tools I'm using, I'll soon publish an updated version of my old list of CLI tools. Because that's what everyone needs in a launcher - AI! ↩︎ I actually don't use the cloud storage for screenshots. I annotate and send them without storing them in the cloud. The only time I had to use cloud storage was when I couldn't share the screenshot with someone directly, and I needed to store it somewhere online for a couple of days. ↩︎ Of course, as long as the data doesn't get corrupted in the meantime. ↩︎ Let me tell you a funny story about the most popular videoconferencing application in the world. Many large organizations use it, so obviously, it needs extensive admin-level settings that allow security teams to prevent users from using the app on unauthorized operating systems or through unauthorized browsers. For example, some companies might say that using Teams on macOS is forbidden because Windows is the only "blessed" operating system where you can use this app. If you try to open Teams in your browser on macOS, you will get an error message saying that your administrator prevented you from using Teams on this operating system. But how does the most widely used videoconferencing tool check what operating system you're using? Surely, it has to be something so advanced that the users wouldn't be able to easily tamper with it, right? Definitely, it wouldn't be just checking your browser's user agent of your browser because even a ten-year-old can change the user agent with one of the many free-to-use plugins. Right, Microsoft? You wouldn't be just checking the browser's user agent, would you? Would you?! ↩︎

16th Nov 2024 1 votes
I Like Makefiles

I like makefiles. I first used a makefile more than ten years ago. Even back then, it looked like some ancient technology used by the graybeard Linux wizards. Years passed, and new build tools came and went, but I kept seeing makefiles still used here and there. I got used to them because they were part of some projects I joined. At some point, I started to like them. Today, they are often the first automation tool I use when I start a new project. The reason I like makefiles is that they often follow an unwritten convention of implementing the same set of commands to get you up and running. When I find a project I know nothing about, and I see a Makefile file inside, chances are that I can run make or make build followed by make install, and I will get this project built and set up on my computer. Or at least I will get information on other steps I need to include. I try to apply the same rule in my projects. If I open a folder with one of my old projects and run make dev, this will perform all the necessary steps to build the project and spin up a dev server. That's convenient because throughout the years, I used many different technologies, and each had different commands to build or deploy a project. I have old projects written in Jekyll, Hugo, 11ty, and all sorts of different Python web frameworks. With makefiles, when I come back to a project I haven't touched for months (or years), I don't have to remember the command to start a dev server with, let's say, Jekyll. I just run make dev, and this, in turn, fires up the corresponding Bundler commands. Even if I use tools like Docker or gulp in my project, I still use makefiles to orchestrate those tools. For example, I often write a make build command that builds all the necessary Docker images, passing additional parameters specific to a given project. My makefiles are simple. I don't use conditional statements, flags or any other fancy features. Most of the tasks (they are technically called targets, but I always call them tasks in my head) consist of one or more shell commands. I could write bash scripts with a couple of functions instead, but makefiles are easier and faster to write. Some common tasks that most of my personal projects[1] contain include: dev to start the development server build to build the project (if a build step is necessary) deploy to deploy/publish the project And that's really it. Sometimes, I include additional tasks like watch to automatically rerun the build task when I change any of the source files. But many of my projects can be managed with just two or three Make commands. This blog that you're reading right now has a simple makefile with just one target: dev: npm run dev And a more advanced project of mine uses the following makefile to run the dev server, watch for changes, build, encrypt and deploy the website: # Run dev server dev: bundle exec jekyll serve --unpublished -w --config _config.yml,_config-dev.yml --livereload # Build assets build: npm run gulp build # Watch a specific folder and process assets watch: npm run gulp watch -- --wip # Build the website locally, encrypt and deploy to Netlify server deploy: JEKYLL_ENV=production bundle exec jekyll build; \ make encrypt; \ netlify deploy --prod # Encrypt the "_site" folder encrypt: npx staticrypt _site/*.html -r -d _site In both of the above examples, I'm ignoring the existence of phony targets, which you might want to add if you have a file called dev, build, watch, deploy, or encrypt, as many kind readers on Hacker News suggested. Otherwise, this Makefile won't work as expected. GNU Make (the software that runs makefiles) is quite ubiquitous. If you're on Linux, you probably already have it installed. Even on my MacBook, I don't remember installing it explicitly. It must have come with some other tools that I installed in the past. Make is simple and doesn't require as many additional dependencies as some other build tools. This can be useful if you need a tool that will work in a restricted environment where installing additional packages is difficult or impossible for security reasons. Make will probably be already present in that environment. And if not, you can just take the commands from the makefile and run them manually in the shell. If gulp is not available on your server, you can't really take the JavaScript code and paste that into the terminal. I'm not against other build tools. I like other build tools too. I'm excited when I find a new one that is better and faster than the one I was using before. But I will still use Make to orchestrate them because it gives me a set of familiar commands to manage all sorts of different setups with different tools. By "personal", I mean projects where the deployment process is much simpler than production-grade stuff. ↩︎

18th Sep 2024 1 votes
I've Built My First Successful Side Project, and I Hate It

In 2020, I built my first side project. I scratched my own itch, then started selling it, and since then, the project has earned me over $15,000. But a few months after releasing it, I was so tired of the maintenance that I wanted to shut it down. Here is a story about my short entrepreneurial adventure. Gumroad dashboard with the earnings from my project In the second part of 2020, I had some spare time. I just finished a project for a client, and I've decided to dedicate a few months to making a Python course – just to try something new. At the same time, I became interested in using technical analysis to trade stocks. Following one of the "internet investment gurus" (yes, I know how bad it sounds), I started day-trading stocks following some "mystical knowledge" on how to leverage price formation called "harmonic patterns" to figure out when to buy or sell (look, I'm sorry, I also cringe when I write those words). Like many traders investors, I was using TradingView, the most popular free platform for technical analysis. Compared to the tools offered by stock brokers at that time, TradingView was far superior with its user-friendly web interface and a plethora of tools. You could even write your own scripts using an abomination of a scripting language called PineScript to implement all sorts of additional tools, graphs, and metrics for your charts. But there was no script that would draw harmonic patterns for me, and after a few days of drawing them manually, I got tired and decided to automate the process. It took me a few weeks to learn the language and implement a script to automatically draw harmonic patterns on the chart. The script significantly decreased the time I had to spend searching for stocks to trade. Then, I wrote another script - one that was drawing potential future patterns. Great! If those scripts work for me, maybe someone will pay money to use them too. After all, the idea of harmonic patterns wasn't new – the earliest harmonic pattern, the "Gartley Pattern", was first described in 1935. There are books, articles, and videos on this topic, so for sure, I wasn't the only one using them. Luckily for me, TradingView supports selling access to scripts, although in a slightly convoluted way. With the most expensive subscription, if you publish your script with "invite only" access, you can control which users can use it. So, I bought that subscription and created a landing page connected to Gumroad to accept payments for monthly and yearly subscriptions. Promoting my scripts # Now, I needed to find a way to tell TradingView's users about my scripts, as those "invite only" scripts are not easy to find. TradingView doesn't promote them in the search results; instead, it prioritizes free-to-use scripts with publicly available source code. And even if someone found my scripts or my website, I needed to convince them that my scripts are useful and do what they are supposed to do. Even though I offered a 14-day money-back guarantee on any subscription, people were reluctant to give their credit card details on some random website. So, I've created a couple of smaller scripts by reusing large chunks of code from the existing scripts. Their source code was still hidden, but people could use the scripts for free - with limited functionality: you could find only some types of price formations there was a limit to how many past days you could check there were no automatic notifications when a new pattern appeared Still, people liked them and started using them. When they asked about missing features like the notifications or detecting more patterns, I directed them to the paid scripts. Then, I published some "ideas" on TradingView - screenshots of popular stocks with drawings of the existing and potential future harmonic patterns generated with my scripts. Finally, I created a YouTube channel where I published videos showing all the features of my scripts (this was partially so I didn't have to explain over and over again how they work). And I started waiting for the cash to roll in. Nothing happened for the first two weeks, but then I got my first sale! Someone bought access to one of the scripts for a month. For the first time in my life, I got paid for selling something on the internet. It felt amazing! Later, they requested a refund because they didn't find the script helpful, but to this day, I clearly remember the excitement of laying on the sofa in the evening and getting an email from Gumroad saying, "Hey, someone just paid you $9 for the thing you've built". The first sale felt more rewarding than any other that followed, although hitting a $1,000 revenue per month was also a nice milestone. Slowly, more people started using my free scripts, and some also bought the paid ones. To strike while the iron was still hot, I posted a message saying that anyone interested can leave a comment under the paid scripts, and I will give them a free 1-week trial. So people started doing that, and each time, I would log in to TradingView and manually grant them access for a week. Some of those trials turned into paid subscriptions, but most didn't. Slowly but steadily, the number of subscribers started increasing. Needy customers? Checked. Fraud? Checked. Next stop - the "burn out" station # As more people used my scripts, I started getting more emails with questions and requests for new features. Many of the emails contained requests for free access to the scripts. Others had very basic questions, answers to which were given in the description of each script. Some people wanted to buy the source code: Or get access to it for free. Of course, for personal use. Sure thing random person from the Internet, I totally trust that you won't share or publish my script as your own. There were people requesting that I add some very specific features because someone they watched on YouTube recommended it. Or that I add some kind of feature that I could not really understand, but it surely made a lot of sense in their mind: Quite a few folks were asking for some "hot tips" on how to trade something or if that something was going to go up or down (almost always, this something was cryptocurrencies or forex). Even worse, sometimes such requests were accompanied by a backstory like "I'm a poor student and want to make some money" or "I've lost some money trading, but your script looks very cool, and I will use it to make money back". Why on earth would you bet your money on some random tool you don't even understand? And then ask some random stranger on the internet for financial advice?! This was sad. I built a tool for people who knew what harmonic patterns were. People who had an investment strategy and only needed a tool to automate the drawing of the charts that they would normally draw by hand. Not some gung-ho investors whose entire investment strategy consists of "this random script I found five minutes ago says that stock X will go up, so I buy." Sometimes, I would get a job offer. Yay! For writing PineScript for a living. Nay! Writing PineScript full time? No, thanks. Or a collaboration proposal from someone who had a "large group of active traders": Somehow all those claims from 'people with large communities' never materialized beyond testing the trial. I even had a chance to practice foreign languages! Disputes # Sometimes, people would open a "dispute". A dispute means someone complained to PayPal that their credit card was wrongfully charged and then it's up to PayPal to decide if they are right (and refund them the full amount of money + charge me additional $20 for "chargeback fees") or if they are not (then I get to keep the payment). In general, I try to be as accommodating to customers as possible. I'm selling software, so unlike with a physical product, I have unlimited stock of my products; producing new copies costs me nothing, and I don't have costs related to handling shipments or returns. So if someone writes to me and asks, "Hey, I forgot to cancel my subscription, and it renewed for another month", or "It's past the 14-day money-back guarantee, but it turns out that the script is not useful for me", I give them their money back. I want to be nice to people. I started selling those tools to help others. I don't want to charge them for a product they don't like. But I also don't want people to open disputes. If you have too many disputes, you risk that PayPal will freeze your account. Sometimes, being nice to people is not enough. Especially if you don't get a chance to show your goodwill because you're sleeping. My first dispute came from a customer who sent me a message in the middle of the night asking how to cancel their subscription and, half an hour later, proceeded to open a dispute. Even though all they had to do was check the emails from Gumroad that contained a link to manage their subscription. And those are short emails with a few lines of text, so it's not that the link is hidden somewhere - at least other people who wanted to cancel their subscription managed to find it. Or they could at least send me an email without opening a dispute right away. Luckily, after I explained that I could (and wanted to) refund their payment, provided they closed the dispute first, the customer cooperated with me. So we closed the dispute, and shortly afterwards, they got their money back. A few more disputes happened throughout the years but were less thrilling. Usually, someone would open a dispute through PayPal, not answer my messages (or PayPal's), and then PayPal would dismiss that dispute as unfounded. And yes, fraud # Being polite and accommodating will take you a long way and make your online interactions much more enjoyable. It also means that, at some point, someone will try to take advantage of you. For me, it happened three times. Most of the time, someone tried using a stolen credit card, but Gumroad caught that and cancelled the payment. But the first time a fraud happened, it caught me a bit off guard. I don't have many screenshots left because that person deleted their TradingView account, but here's how it went. First, Mr. Scammer (that's not his real name 😉) asked for free access to the script. That's not uncommon - sometimes, people send me requests like that. When I kindly but firmly denied his request, he tried to pay, but his payment was declined with a message saying he should contact Gumroad. Again, this didn't sound like a red flag, more like a technical issue. Eventually, I got a confirmation email that he had paid for the subscription. Two months later, a "sale has been disputed" email dropped. The dispute was initiated by Mr. Scammer. I asked him why he initiated the dispute because he never mentioned wanting to cancel his subscription or getting a refund. This is when things started to be very fishy: Weird grammar errors were not a red flag. They are common in emails I get from existing and potential customers, so it doesn't automatically mean an email is a scam. But a story of your wife controlling the finances and you forgetting to tell her about buying a script for TradingView, which made her open a dispute? That's even weirder than the "I'm a student who can't afford $15, but I really need this script for my day-trading" emails I sometimes get. While waiting for the dispute to be resolved, Mr. Scammer tried to buy the subscription one more time, this time with less success: So, was his "wife" not only controlling the finances but also using stolen credit cards? I wrote him that he's permanently banned from buying any scripts from me, and the next time he tries, I will immediately revoke his access. That stopped any further attempts. That was the most interesting case of fraud. Usually, when people got caught, they ignored my messages instead of coming up with a creative story: Burn out # Answering emails and checking TradingView comments to give out trial access took a bit of time every day. Not much, but I tried to answer at least every day. It was a minor annoyance when I wasn't working full-time. But when I had to do this after a full day of work, and I got yet another email asking to add some magical method for determining take profit levels that was invented by a crypto-trading YouTuber with 50 followers[1], I had enough. The worst part was that after a few months, I wasn't even using those scripts anymore. I had fun day trading when I wasn't working full-time. It was a nice break from recording videos and a motivation to learn about financial markets. But when a new contract started, I stopped day trading due to the lack of time. But I still had existing customers that I needed to take care of and potential new customers asking me all sorts of questions. I was growing tired of maintaining this project. Sometimes, I would spend an hour answering many detailed questions from a potential customer only to never hear back from them. At times, someone would leave a 1-star review on Gumroad because I didn't want to add some weird functionality they requested. On days like that, I wanted to close down this project, refund the money to the existing clients, and delete my TradingView account. Still, the project was bringing in a bit of money every month. I didn't want to kill the goose that laid tiny golden eggs. So, I've decided that I'm going to sell my "goose". Selling my project. Well, trying to... # I looked for platforms where I could list my project for sale. I found a few and decided to submit an ad to IndieMaker and Transferslot (which looked much more active three years ago). I prepared a detailed description of the project and shared details like the number of views on my YouTube channel, unique visitors to the website, the number of followers and subscribers on various platforms, and, of course, the revenue. I even shared some ideas on how to move this project forward. For example, getting one of the free scripts promoted in the weekly TradingView newsletter could accelerate the sales of paid scripts. Then, following some random advice on the internet, I came up with a number to ask for that was around 5x the revenue from last year. I then sent my ad for publication. IndieMaker published my ad, and shortly after, I received an inquiry from a potential buyer. But after answering questions about the Monthly Recurring Revenue growth and what assets come with the project, there was no follow-up. I got more messages later, but those were just spam ("Hey, I'm interested in your project, send us a list of your inventory"). Transferslot never published my ad. I think the website was already abandoned when I submitted it. If you have any experience selling projects online (and if you don't, this article from Patrick McKenzie is the best way to understand how it works), you're probably shaking your head by now. People buying software businesses look for simple, low-risk, boring-tech projects with a proven track record of stable or increasing profits. No one wants to buy a project written in an esoteric language (PineScript) running on a proprietary platform (TradingView) that targets a very specific niche of traders. After a couple of weeks with no more replies, I had to decide what to do next. I could submit my ad to bigger platforms like Flippa and pay to have it listed there. But I felt that this project was so niche that I wouldn't find a potential buyer on other platforms either. Especially a potential buyer who knew the PineScript language necessary to maintain and improve the scripts' source code. While waiting for more bids from potential buyers, I started thinking about what I could do to ease the burden of this project. I'm an engineer, goddamit! We're not meant to click buttons in the browser or send the same email over and over again! Turning the autopilot on # I checked my options and decided to automate the following parts of my work: Gumroad payment should automatically grant access to the script on TradingView and send a "welcome" email. This would cover the most time-critical part of my work, so I didn't have to start my day from logging in to TradingView and granting access to people who bought subscriptions during the night. A chatbot to manage subscriptions so that I could do all the manual work (removing access from people who cancelled subscriptions, extending trial access, etc.) with simple commands from my phone without logging in to TradingView. Finally, I could create an online form where people could request trial access by leaving their TradingView username. This form would then trigger a Python script granting them one week's access to the selected TradingView script. I implemented all those features using n8n to connect various services together - webhooks, a Telegram bot, Python scripts, sending emails, etc. Here's an article describing how I did this. Once I had the automation in place, users could request free trial access through an online form or buy a subscription on Gumroad, and everything would be handled automatically. From time to time, when they cancelled their subscription, I had to send a short message to my Telegram bot to set an expiration date for their access. The hardest part was to stop caring so much about this project and disconnect myself. I like helping people. That's why I became a programmer - to build things that help me or help others. When I get an email, I try to answer it as best as I can. Years of working with clients taught me to explain things in a simple and easy-to-understand way. So, I spent hours patiently answering questions from potential customers only to never hear back from them. And even though I posted a link to the form where people could request trial access, plenty of folks didn't bother reading the description and kept commenting or sending me messages requesting trials. This had to stop. I started by checking TradingView messages only on the weekends and granting trial access in bulk. For every comment about trial access, I replied "I gave you access now, but it would have been faster if you used the form I posted", hoping that others would see my reply and use the form next time. I still paid special attention to my existing clients and answered their emails every evening. But if someone was not an existing client, I would only respond to their questions on weekends. Eventually, I stopped checking the TradingView comments or messages. If people really wanted a trial, they would eventually read the two-sentence instruction, click the link, and fill in the form. Did this affect my sales significantly? I don't think so. Most of the people asking basic questions would never convert to paying customers anyway. They were window shopping. They found a script that seemed to predict price changes, and they wanted to know how to make money with it. And the theory behind how it works and why it's supposed to work? Nah, they didn't care. They had ten more random scripts to test out today. Did disconnecting from my project affect my sanity? Oh, hell yes! The project went on autopilot, and it no longer felt like a dreaded chore when I was checking my emails or logging in to my TradingView account. It lost a lot of traction because I stopped updating it, so people think it's not useful anymore (even though it does its job as well as it did a few years ago). Its revenue declined in the past years. But I don't care. I'm happy again. What's next for this project? I guess I will keep running it until I decide that including it in my monthly accounting is not worth the effort, which is when I will shut it down. But for now, the $200 I get every month with almost no work is a nice passive income. Things I've learned # Don't let a side project burn you out # The dreaded truth that most programmers learn the hard way is that releasing a product is just the beginning. Once it's out there, you have to maintain it: fix bugs, implement new features, deal with disputes or frauds, and answer a LOT of emails (often from curious window shoppers who never convert to paying customers). This maintenance is easier if you have more products and you do support in bulk. Or if your project can replace your full-time job and maintaining it is the only thing you do during the day. But if you do this on top of your regular job, it's a terrible time sink. You need to set some boundaries. Otherwise, working on your project will take all your available time. Working on a side project is much more fun if you're also one of its users and genuinely want to keep improving it. It's incredibly smooth if you love your project, and the perspective of working on it full-time sounds like a dream job. And accordingly, it's much harder to support a project when you move on to doing something else and you're no longer interested in maintaining it, but you still have existing customers you need to keep happy. If your side project is not a source of genuine joy and you have other priorities, it's important to set some boundaries. I've decided to focus my support only on the paying customers. I also stopped adding new features and told myself that unless my scripts break, I'm not touching the code. Even though I still had a list of additional improvements I wanted to make one day, I deleted that list. I already had paying customers, which means people find value in what I've built. Sure, I can add new features to attract new customers. But that means even more coding and maintenance for a chance to attract new customers. Use a merchant of record # If you're planning to sell products globally, consider using a "merchant of record" like Gumroad or Paddle. This way, you won't have to figure out how much tax to charge for each country (basically, you only sell to Gumroad, and they resell your product to the final customer). Figuring out taxes was the last thing I wanted to do when selling my products. I went with Gumroad, which - at the time of writing - charges a 10% fee, plus around 3% of PayPal/Stripe fees on top of that. There might be cheaper options, so do your research before choosing one. Once you get some recurring payments rolling in, you can't easily move your existing customers to a different company. When the Gumroad fees increased throughout the years, I always thought that migrating to another service was too much of a hassle, so that's why I'm still using Gumroad. Selling stuff to customers is hard # I had this one guy where the conversation looked like this: Hey, amazing script, I rated it 5 stars! Can you implement the take profit levels for me? I'm sorry, but I can't. The original methodology doesn't specify exact take profit levels. It also doesn't make sense because take profits should be manually determined, for example, based on support or resistance levels visible on the chart. Different people will use different take profit levels based on their risk appetite. Ok, but can you please put them for me? I follow this [insert some YouTube crypto day-trader], and he uses [some very specific take profit levels]. I would like the script to draw them for me. Look, I can't do this. Those take profit levels are very specific to your trading system. They won't work for most other people, but they will slow down the script for everyone. Ok, bro. He cancelled his subscription a few days later and lowered his 5-star review to 2 stars. Business-to-consumer (B2C) is difficult (not that business-to-business is much easier - it has a different set of challenges). But this doesn't mean that B2C is not worth pursuing. There are many people who did great selling to customers. One of the most well-known is Pieter Levels (@levelsio), creator of projects like NomadList, RemoteOK, and, more recently, PhotoAI. According to his Twitter profile stats, he makes an astonishing $200k Monthly Recurring Revenue from his projects. But B2C, while easier to get into than establishing a relationship with a company in a B2B business model, means that you will need many more customers to have the revenue levels that B2B can bring. And more customers mean more work. Be nice # Try to be nice to people. I mean, being nice to people in general will make the world a better place. But when dealing with customers, never assume they do something in bad faith. Sure, some folks will try to take advantage of you. But many "problems" are just a matter of misunderstanding. People will request a chargeback because they think it's the only way to get a refund for a subscription they forgot about. If someone asks me for a refund, then no matter what's the reason, I give them their money back and wish them good luck with their investments. Fighting disgruntled customers over $20 is not a good way to spend time. If you're nice to people, they are more likely to use and recommend your products. One of the reasons why Amazon got so popular early in the days was because of its extraordinary customer support. Ten years ago, when my Kindle stopped turning on (which might or might not have been caused by accidental yet excessive force applied to it), Amazon's support said "no worries, we'll send you a new one for free". Someone would send me a free product instead of requesting to prove that it broke by itself? This blew me away! Especially since I was living in Poland, where the strong post-communism roots mean that if a seller is not mean to you, then this already counts as them being nice.😉 So, I wanted to do the same with my customers. I tried to accommodate their needs, write detailed explanations to their questions, and generously issue refunds or even discounts when requested. When someone complained that something didn't work because it didn't work the way they wanted, I suggested alternative scripts and offered refunds. And that paid back. I received a lot of nice messages from customers, and even though I was getting tired of supporting my project, getting positive feedback from time to time kept me going for all those years. And that's the story of my first side project. It was fun at the beginning, frustrating in the middle, and now it's mostly a passive income that will eventually dry up. Was it worth the money? Not really. Spending equal time working for a client would probably earn me more. But the experience I gained from this side project will be invaluable when I build the next one. If you're interested in the technical setup I used to automate most of this project, check out my Web Automation With n8n, Telegram, Online Forms, and a Bit of Python article. It was always the crypto or forex crowd that was asking for weird stuff. ↩︎

19th Aug 2024 1 votes
You Don't Need Stream Deck, You Need Macros

One day, during one of those meetings that did not require my full attention, I was doing some work in the background. I muted my microphone, and I was coding away[1]. Suddenly, someone asked me a question. "Now, where is that goddamn Teams window so I can unmute my microphone?!" - I asked myself, frantically going through my MacOS desktops. I use multiple desktops (or workspaces, as some other operating systems call them), each with one to three apps opened. So, instead of shuffling 20 apps opened on one desktop, I shuffle ten desktops scattered across three monitors. It took a few seconds of awkward silence before I found that Teams window. And it wasn't even the first time this happened to me. "There has to be a better way!" - I thought, again, not for the first time. I've decided it's time to figure out how to set up a "global mute/unmute" button that would work regardless of whether Teams (or any other videoconferencing app) has the focus. One way to solve this problem was to use Elgato's Stream Deck. I've been eyeing it for some time, as many developers claim it's a very useful device. Is Stream Deck really indispensable for a software developer? # "Stream deck is a fantastic tool, and every developer should have one (or three)!" is something I've heard here and there. I've put it on a shopping list for Black Friday, hoping I can get a good deal. But as Black Friday was getting close, I asked myself - do I really need Stream Deck? My main driver was to have a global mute/unmute button for those Teams calls when I'm doing something else and suddenly need to unmute myself. Pressing a button to unmute my microphone, say what I need to say, and mute it again without shuffling through multiple windows and desktops would be bliss. But that doesn't justify shelling out over $100 (or less, for the "mini" version) and putting another device on my already cluttered desk. So, I've started reading about other things people do with Stream Deck, especially if they are not streamers but developers like me. There are some good reasons to buy Stream Deck. For example, if you SSH to many servers, talk with many people on Slack, generate different Stripe test cases, switch between many Spotify playlists, work on different VS Code projects, and push code to different GitHub repositories - and all that on a daily basis, then you will probably benefit from using a device that can reduce all those tasks to a press of a button or two. It sounds more useful for people who prefer to click things and press buttons instead of typing all the commands in the terminal. It also works well for those who spend most of their time in one piece of software like AutoCAD or Photoshop, where creating and assigning macros to hardware buttons will save them time. Not to mention that in this line of work, you're already using a drawing tablet, so you're used to using different devices apart from a keyboard. There are some weird but ok reasons to buy it. Some people use it for home automation - ok, but I'm not going to run to my office to toggle lights in the bedroom. Some use it to start applications - fine, but for me, firing up a launcher like Spotlight/Alfred, typing 2-3 first letters of the app I want to start and pressing enter seems equally fast as pressing a button on an external device[2]. But a typical software developer spends most of their day pressing keys on a keyboard. Everything that Stream Deck does can also be achieved by pressing one or more keys. Some actions will trigger a keyboard shortcut, while others are equivalent to typing commands in the terminal or running a script. Stream Deck is a glorified macro recorder. It comes with a nice app that makes it easier to record those macros. It has colorful icons that catch your eye and pressing a physical button is easier than remembering a shortcut for a specific macro. But most of its functionality can be achieved by recording macros on your computer. If you have a manageable number of macros (or a very good memory of all the shortcuts you defined), it might actually be easier to use your keyboard to trigger the macros. Stream Deck is not a big device (especially if you go for the mini version that has six buttons), but it's another device you would need to carry with you if you go somewhere. A keyboard, on the other hand, is usually an integrated part of your laptop. Where your laptop goes, your keyboard goes too. Actually, I bought the Stream Deck after writing and implementing all the tips from this article. I got a good deal for it, and I could always return it if I saw that I wasn't using it that much. After a couple of days, I liked it and decided to keep it. While I still think you can write keyboard macros that can do the same things as an external device, remembering how to trigger those macros gets harder the more macros you have. A device like Stream Deck (or one of its cheaper or more customizable alternatives) is just more convenient for me. Pressing a colorful button with a label takes no mental effort. On the other hand, trying to remember how to trigger some less often-used shortcuts can be a bit more demanding, especially if I'm in the middle of thinking about something else. I decided to keep the Stream Deck because I found several additional uses for it (that I will describe in another article). If I were to use it only to mute/unmute my microphone, I would definitely stick with the macro on my keyboard instead. How do I find unused key combinations? # One of the great things about Stream Deck or a similar device is that all of their buttons come with different key codes. There is no risk that pressing a button on a device like Stream Deck will trigger a different action, for example, from the currently opened application. But if you want to start assigning custom shortcuts on your keyboard, this might be a problem. When defining keyboard shortcuts that will trigger your macros, you need to find a combination of keys not used by any application. Otherwise, if you try to use a shortcut, the current application will intercept it and perform a completely different action. Assigning custom actions to shortcuts like "cmd+s" ("ctrl+s" for Linux/Windows users) or "cmd+w" is off the table because, in most programs, they are already used to save the file or close the app. Luckily, there are some combinations of modifiers that are rarely used. These are called the Hyper key (shift+cmd+ctrl+option on Mac and shift+windows+ctrl+alt on Windows) and the Meh key (shift+ctrl+option on Mac and shift+ctrl+alt on Windows). Both combinations are not the easiest to press; for example, for a keyboard shortcut that uses the Hyper key, you need to press four modifier keys and another key assigned to a shortcut. But many external keyboards have additional keys to which you can assign the Hyper or Meh keys (that's what I did with my Kinesis Advantage keyboard). Even the built-in keyboards have keys you can live without (the "Caps Lock" key is fairly unused - unless you need to shout a lot on the internet) and can be converted into a special key. I decided that before I buy the Stream Deck, I will first try to implement some useful macros. I used BetterTouchTool or BTT for short - my preferred way to define macros on a MacBook (I bought it years ago to customize the touchbar on my previous laptop, and I've been using it ever since). I configured the "Esc" key to act as the Hyper (so "shift+cmd+ctrl+alt" ) and "Insert" to act as the Meh (so "shift+ctrl+alt"). Both those keys were previously unused - I'm using "Caps Lock" as the Esc key and the "Insert" key is basically a second backslash key on the Kinesis Advantage2 keyboard. An unused key on my keyboard that could be used as Meh Next, I've decided to make an easy-to-remember split of functionality of those custom shortcuts. The Hyper key would be used in shortcuts that start some applications, and the Meh key would be used for all the other macros. Ideally, the first letter of the application's name could be used as the shortcut to open it. So, for example, Hyper + o would open the Obsidian app, Hyper + i would start iTerm, and Hyper + f would start Finder. Of course, not every app could have a shortcut like that (e.g. if two apps started with the same letter, one would need a different shortcut). But I've decided to limit the number of application shortcuts to only those I use multiple times per day. If an app is already opened, pressing its shortcut will switch to the desktop with that app and bring it to the front. Next, it was time for more advanced macros that I would use with the Meh modifier: Meh + m: mutes/unmutes the microphone. This shortcut triggers a "Run Apple Script (blocking)" from BTT that runs an applescript to toggle the current microphone's volume between 0 and 100 and displays a notification. To make sure I don't get confused about whether my microphone is muted or not, I've added this BTT widget to display an icon in the menu bar indicating the microphone's status. Meh + p: triggers the "Play/Pause" action in BTT to toggle the currently playing music or any other video or audio. I could achieve the same by pressing the button on my headphones, but still, my fingers are on the keyboard, so pressing a keyboard shortcut is faster. Meh + n: sets the "Do not disturb" status for 1 hour. This hides all the notifications and is very useful during screen-sharing sessions, ensuring no weird message pops up in the middle of a presentation. I have an Apple Shortcut that sets the "Do not disturb" status and accepts a parameter defining how long it should be set. This shortcut comes with the Calm notifications workflow for Alfred, and all I have to do is trigger it from BTT with the "Run Shortcut from Shortcuts App" action (optionally followed by the "Show notification" action to get a popup confirming the correct execution of the shortcut). Meh + t: brings the Teams window to the foreground. This is done by using the "Show/Hide Specific Application" action in BTT, and it saves me from figuring out where my Teams window is during calls. Those are the most useful shortcuts I could think of. This setup was not without flaws, and I solved them when I bought the Stream Deck. But it was good enough to start. What were the flaws? For instance, the play/pause button usually works for the last piece of media content played, but sometimes it's a bit random. This was an issue when I had the Spotify app opened (which is always opened) and a YouTube video paused in one of the browser tabs. Sometimes, the macro would work on Spotify, when I wanted it to start the video, and sometimes it would start a video somewhere in my browser when I wanted it to stop the music [3]. That was quite annoying, and with Stream Deck, I've solved this problem by having a separate button for Spotify and another one for other media content. Or let's take the mute/unmute functionality. It worked well as a macro, but with Stream Deck, I can get a plugin that controls the Teams client directly and toggles the mute/unmute setting in the video call. This approach is more convenient because everyone on the call can see an icon indicating that I'm muted (and, when I try to say something, they can tell me, "Hey, you're muted"). An "operating system-wide mute" is not reflected in Teams, and I've noticed that it's more confusing to people during the calls. Having a few macros is great if you need to simplify some common actions on your computer. But as I wrote in the side note about my experience after buying the Stream Deck, the more macros you have, the more you have to think about how to trigger each one. This can be distracting when you are in the middle of a task requiring a lot of focus. So, I gave Stream Deck a chance and ended up keeping it. Check it out to see my Stream Deck setup and my conclusions after using it for a few months. Try to get a mechanical keyboard, and you will quickly understand why muting is necessary. ↩︎ I was wrong with this statement. Pressing a button to open an app is sometimes more convenient. Especially when I'm doing something else on my main screen and I can press the button on an external device without losing focus. ↩︎ And DJ "Spotify mixed with a random YouTube video" is terrible. ↩︎

9th May 2024 1 votes
Managing Gigabytes of Images with git-annex

Earlier this year, I hit GitLab's 10GB storage limit with one of my git repositories. I'm writing a family blog where I document our various trips and activities, hoping that in 20 years, it will be a great memento (like a photo album combined with a journal in a digital form, with search functionality, videos, GIFs, etc.). The blog started as a simple Git repository that I hosted on GitLab, but after adding hundreds of pictures, I hit the maximum repository size, and it was time to move images and movies somewhere else. Looking for a possible solution, I wanted to choose something that integrates seamlessly with my existing git workflow, so the final choice was between git LFS and git-annex. The reviews were mixed, but git-annex was more favored by the audience. The deal breaker was that if I decided to stop using it in the future, I would just need to run git annex uninit to get all the images back in place. That means losing the history of image changes, but I don't care about that anyway - I only care about not losing the most recent version of each file. So I've decided to go with git-annex. Unlike my wife, I rarely look back at the old pictures and walk down memory lane. But I know that when I get old, those memories will be among the most precious things left in my life. And even today, I can't remember trips I took ten years ago or large parts of my student life. That's why the only non-work related subscription for which I pay a significant amount of money is the 1 second everyday app. It lets you record a one-second-long clip of your day and then join multiple clips together to get a video of how your month/year/life looked like. If you pay for the premium plan (that costs around $70 per year), you can record longer clips, add multiple clips per day and - most importantly for me - have your clips backed up. And while it costs me more than, let's say, my Spotify subscription that I use for multiple hours per day, I often forget to record my "1 second" every day. I remember using it mostly when I'm on holidays. So why do I pay for an app that I forget to use? Because I love the idea behind the app. It lets me generate 1-minute-long videos full of great memories from vacations. And with the paid plan, I get backups, which means I won't lose the recorded memories if I lose my phone. I've looked for alternative solutions and even thought about making all those recordings myself. Still, I've decided that the time I will spend cutting and editing my videos is not worth saving 70 bucks per year. If you like the idea of recording small clips of your life, give the app a try - it has a free plan that might be enough for you. Disclaimer: I'm not related to that company in any way; I just like their app in the context of my family blog. git-annex is a tool that synchronizes files across different repositories. And by repositories, I mean folders, external drives or even 3rd party services like Google Drive, Dropbox, Amazon s3, and others (synchronization with 3rd parties usually happens through rclone, rsync, and similar tools). It also integrates nicely with git, so you can manage "large files" through git-annex and all the other smaller files through git. You can even go into full automatic mode and have git-annex handle all the files in your folder. Then, each time something changes, git-annex automatically commits the changes and propagates them to other repositories. git-annex home page That was the good news. The bad news is that git-annex is a complex beast with a steep learning curve unless you settle on the "relinquish all the control and let the git-annex daemon handle all the changes for you" mode (which I didn't want to do since I like to control exactly how my git repository works)[1]. It's much more complex than git. And surprisingly, there are not many tutorials clearly explaining how to do what I wanted to do (which is - to store images from my repository on Google Drive and a NAS server). The tutorial from the documentation is a great read, and I recommend you read that first. But it focuses on adding remote repositories on a USB drive or in another folder. There is a brief mention of special remotes, but for people like me, who just learned about git-annex, it's not clear if a special remote will ever be needed. I update my family blog once a year - I spend 2-3 weeks working on it and then leave it until the next year. So, I decided to write down everything I learned when setting up git-annex for the first time, including how to use it and how to troubleshoot common problems. This will make my life easier when I get back to git-annex next year and forget everything I did last time. Maybe my notes will make your life easier too? I've spent a couple of days trying to understand how git-annex works and made many mistakes and invalid assumptions along the way. A few times, it was easier to start from scratch with newly gained knowledge than try to fix the mess I did. I used a test repository until I felt I had a good grasp of how git-annex works. I still don't understand many of its details, but at this point, I have a setup that works well for me. I also have a lot of respect for the creators of this tool and all the contributors. git-annex is an incredibly versatile tool that supports a plethora of different scenarios, backends, backup modes, etc. But using it is an order of magnitude harder than using git alone. How to set up Google Drive (or any other external backup service) with git-annex # The first task I embarked on was to enable storing files on my Google Drive account. The steps that I describe work for any other service you can connect to through rclone, including most of the popular file storage and backup services. To enable using rclone with git-annex, follow the instructions on the git-annex-remote-rclone website. When setting up rclone, you will at some point have to choose the "scope", with the available options being "drive", "drive.file", "drive.appfolder", etc. I initially chose "drive.appfolder", which stands for "Allows read and write access to the Application Data folder". However, this means that you won't see your files when you open Google Drive in the web UI - they will be hidden in a folder specific to an application you create in the setup part. You can interact with those files using rclone directly, but I prefer to see my files in the web UI to make sure they are still there. So later, I changed the scope to "drive", which gives rclone access to all your files on Google Drive. When you change the scope, remember to authenticate rclone with remote again (that's part of the rclone setup instructions)! I didn't do this (and only replaced "drive.appfolder" with "drive" in rclone.conf), and then I couldn't figure out why git-annex is still storing my files in the Application folder. Once I re-authenticated and obtained a new token, everything worked fine. Once rclone is set up, use the example command from the instructions to enable that special remote in git-annex. Here is the command I used: git annex initremote gdrive \ type=external \ externaltype=rclone \ target=GoogleDrive \ prefix=git-annex-travel-blog \ chunk=50MiB \ encryption=shared mac=HMACSHA512 \ rclone_layout=lower How to back up git-annex to a locally mounted NAS server # I have a Synology NAS that I'm using as the 2nd backup for my project. The way I configured it is that I mount it on my computer and access it through the /Volumes/gitannex folder. That's one way to do it, but there are other solutions. If you check the git-annex documentation, you can see that they suggest using ssh to connect to NAS on their Synology NAS and git annex page. I don't use that approach because I'm lazy (I really don't want to install additional stuff on my NAS, set SSH keys, etc.), and I want the simplest solution possible (I already have a NAS mounted on my computer most of the time anyway). Here is how I add my NAS folder to git-annex: git annex initremote nas \ type=directory \ directory=/Volumes/gitannex/travel-blog/ \ encryption=none Things that surprised me the most about git-annex # Figuring out how to use git-annex is more difficult than understanding a typical piece of software created nowadays. Everything you need to know is in the documentation, but it's not served in the most easy-to-digest way. The majority of projects today come with a quick start guide explaining how to use them, often in a step-by-step form, so I can just run some commands and set up everything. And if that doesn't work, I can search the internet for some tutorials on how to do X in a given software. I should get back enough blogs to quickly figure out what to do. git-annex is old school. It doesn't go easy on people like me who live by the rule of "5 hours of debugging can save you 5 minutes of reading the documentation". The knowledge is in the documentation, but just like with Linux man pages, you need to know what you want to do. And to know what you want to do, you must first read the documentation carefully to understand how git-annex works. After multiple days of trying to set it up and rereading the same few pages of documentation, I kept finding new information as more pieces were falling into place to form a better understanding of this tool. Some assumptions that I made initially were revised. Sometimes - even multiple times (I went from manually managing my files to the automatic mode and back). I didn't know what to expect from git-annex and how it was supposed to work. On the one hand, I was expecting something like rsync that would simply move files from one place on the internet (my computer) to another (some server). On the other hand, it had to somehow integrate with git's workflow - track changes, update files, roll back to previous versions, etc. Here's my journey on the path of surprises. First, I installed git-annex and found the walkthrough. I followed it, and after running git annex add *.jpg, I saw that my images were converted into symlinks. Fantastic! That will let me keep the repository size small. Now, what do I do with those symlinks? The walkthrough doesn't really explain how to make them work with Google Drive (apart from vaguely mentioning special remotes). Instead, it focuses on having multiple git repositories in different places. Ok, cool, but after finishing the walkthrough, I was not even a single step closer to figuring out how to set up git-annex for my use case. I found the git-annex-remote-rclone plugin that can use rclone to sync files with Google Drive and act as a special remote. Cool! I've set it up, and then I was able to send files there using git annex copy --to gdrive and retrieve them using git annex copy --from gdrive. Great, it seemed I was all set. It was a bit cumbersome because I had to add files to git-annex by hand with commands like git annex add *.jpg, and when I added files, they became locked, preventing me from editing them unless I unlocked them. But I could live with that. However, the next day, I found out that there was a workflows page explaining that I didn't have to do all that by hand! So I fired up git annex webapp that starts the "assistant" - a daemon that automatically detects changes in the repository, adds files to git-annex, and creates commits. I gave it a try, and it worked, but I no longer had symlinks in my project. All the images were reported in my terminal as full-size. Ok, that's not what I wanted to have. But when I pushed those files to GitLab, I could see that in the repository their size was tiny (hundreds of bytes). When I tried to open one of the images, I saw that inside, there was a text that looked like this: /annex/objects/SHA256E-s144178--dc75f9d82cc334de0445f744bcb.jpeg. What the heck?! The text looked like a symlink, but unlike when I ran git annex add, my terminal didn't see those files as symlinks! Turns out that what I saw were pointer files - the second way git-annex stores objects (the first way being symlinks). Pointer files are kind of like symlinks, and even though they look strange to me because they appear as normal files in the terminal, the important thing is that GitLab sees them as tiny files (which is good; they don't use space on GitLab). And my code editor opens them as normal files too. Running 'git-annex webapp' starts a website for configuring the assistant Great success! From now on, I will use the git-annex assistant! It has the additional benefit that my files are no longer locked. I can edit them freely, and the assistant automatically commits every change. Here is some more information about the pros and cons of unlocked files - the unlocked mode is a successor to the direct mode. All seemed to work great, except that when I created a text file, git assistant also added it to git-annex. That's not what I wanted! I don't want my text files to be converted to symlinks or pointer files. So how do I make git-annex stick with processing only images and videos, leaving all the other files alone? Googling for "git-annex filter file types" didn't immediately return any useful answers. It turns out that I had to change how I think about git-annex. It's supposed to work with "large files", so what I had to do was to declare that images and videos should be considered as large files. One way to do this is to define large files as larger than some threshold, e.g. git config annex.largefiles 'largerthan=100kb'. This might work fine, but I prefer to determine which files git-annex should process based on their extensions because I know exactly what extensions my images and videos have. That seems easier than trying to guess what file size will catch all the media files but not other files. I set this up by creating the following .gitattributes file in the root folder of my project: * annex.largefiles=nothing *.jpg annex.largefiles=anything *.mp4 annex.largefiles=anything The above settings are parsed from the top, so we first declare that no files should be treated as large files, and then add specific file extensions to the list of large files. Alternatively, you could add file types to the list of preferred_content. According to the description from the documentation, "preferred content" is supposed to specify what types of files should go into which remote repository. But I couldn't find any information on how to set preferred content in the .gitattributes file, and passing a long list of filetypes that is stored somewhere in the configuration file seemed less optimal than defining this setting in .gitattributes that I can easily access and edit. Great, now I can start the assistant, and it will only watch for specific file types (all the file types that do not match its filters will be managed with git, not git-annex). I thought this would be my final solution, but I ended up disliking the mess that git-annex was making to my git log. So I went back to manually committing files with my own commit messages, which brings me to... My final workflow # Even though using the git-annex assistant to watch and automatically commit all my changes sounds tempting, it makes my git log look like this: commit 430b796f2559a99b8a01f1a69ce55dfe3c50f75e Author: Sebastian Witowski <[email protected]> Date: Sat Jan 20 20:54:35 2024 +0100 git-annex in [email protected]:~/workspace/test2/test-repo commit f44a6e40440cc083e3c15bf6f9b40ef2ee818e8f Author: Sebastian Witowski <[email protected]> Date: Sat Jan 20 20:01:56 2024 +0100 git-annex in [email protected]:~/workspace/test2/test-repo commit 95090b5861a0afc9de20796c6a7c3bb7ea696649 Author: Sebastian Witowski <[email protected] Date: Sat Jan 20 20:00:34 2024 +0100 git-annex in [email protected]:~/workspace/test2/test-repo ... And I don't like that. Sure, I can squash and edit the commit messages, but I prefer to use git-annex the same way I use git - so control manually when and what I commit. In the end, I disabled the assistant. Now, when I work on my project, and I want to commit changes, I do this: # Add all files to the index # git-annex will figure out which files go into annex and which go into git based on the filters I defined in .gitattributes git annex add # Commit changes git commit # Push changes to git and send files to remotes git annex sync --content Short and sweet. Setting up git-annex from scratch and basic usage # Let's distil all I've learned into a short guide explaining how to set up git-annex from scratch and how to use it. Take my advice with a grain of salt! git-annex supports many different workflows, and what I'm describing in this article is a workflow that works for me. Also, I've used git-annex barely for a couple of evenings. For sure, there are more optimal ways to do some things (or at least more optimal for your scenario, dear reader). I hope I'm not messing up something seriously, but this I will see in a couple of months/years. My scenario doesn't require very strict version control of images, and I'm fine with keeping only the latest versions. So if I mess something up, I can just take the most recent version of all the files and start a new git repository from scratch. And git-annex allows you to disable it rather easily - you just run git annex uninit, and it turns all symlinks back to the original files. That said, let's dig into setting up git-annex from scratch. If your starting point is a folder with neither git nor git-annex, then first, you need to initialize both of them: git init git annex init Next, set up a remote Google Drive (or some other external storage) using the git-annex-remote-rclone plugin. The following commands will add a new remote called "gdrive" using the "GoogleDrive" configuration from rclone.conf (so make sure to use the same name when you configure rclone) and store files on Google Drive in the my-app1 folder: git annex initremote gdrive \ type=external \ externaltype=rclone \ target=GoogleDrive \ prefix=my-app1 \ chunk=50MiB \ encryption=shared \ mac=HMACSHA512 \ rclone_layout=lower # [Optional step] Verify that the remote is correctly set up and we can push/pull files to/from there # This will take a few minutes and might fail/hang up (it failed for me, but I ignored that) git annex testremote gdrive # Sync the information about our new git-annex remote to the git repository git annex sync The git annex sync command above is important. When you initialize a new remote, this command will store information about that remote in .git, so when you clone this repository on another computer, you can enable that remote with git annex enableremote gdrive instead of setting it up from scratch. Now, you can either manually commit files and send them to/pull them from a remote or use one of the automatic ways to do all that. As I explained before, I do everything manually to have better control over when I commit and what I write in the commit messages. Here's how to add and synchronize files: # Add all JPG files to git annex git annex add *.jpg # Copy all git-annex'ed files to gdrive git annex copy . --to gdrive Now, if you need to create another repository (let's say you have a new computer and you want to set up your project there), run the following commands: git clone <link-to-the-repository> . git annex init # Retrieve information about initialized remotes git annex sync git annex enableremote gdrive git annex copy --from gdrive And that should make all the annexed files available. To add more remotes, for example, in a different folder or with a different rclone configuration, just follow the steps again to initialize it (git annex initremote) and synchronize the changes. Once you have more remotes enabled, instead of manually pushing and pulling files from remotes, you can synchronize the state of the files using git annex sync --content command. It will push all your changes to the remotes that don't have their own copy of the files (and also pull those files that are on any of the remotes but not on your computer). Troubleshooting and "how-to" guides # Here is a list of common tasks you might have to do when using git-annex. Remove a remote # How to remove a remote that is in a wrong state (you misconfigured something, and want to remove that remote and try again)? Following this answer, you can mark it as dead and propagate that information to other repos: git annex dead my-badly-configured-remote git annex sync Synchronize changes to a remote # How to synchronize changes after one of the remotes was not available during the previous sync (e.g. NAS was not mounted)? If you're using git-annex assistant, you can wait until the changes synchronize automatically. But if you don't use it or if you want to force the synchronization, run: git annex sync --content Edit a locked file # How to edit a locked file? By default, when git-annex replaces the file with a symlink, it locks it, so you can't modify it. To modify such a file, you need to unlock it, change it, and add it back to the annex: git annex unlock image.jpg <edit the file> git annex add image.jpg Remove unused files # How to remove unused files from git-annex? Removing a file from git doesn't automatically remove the copies from git-annex and the remotes. First of all, be careful, as the method I suggest removes all files not referenced by a branch or a tag. I use the --force option that bypasses the check if at least one copy of the file is left in some other remote (in case you want to restore that file in the future). This basically means that you lose the possibility of going back to the previous state of a file unless it's a file in a commit that was tagged. I don't care about the previous versions of my media files, so this is fine for me. # Find all unused files locally git annex unused # Delete them git annex dropunused --force all # Find all unused files on a remote git annex unused --from gdrive # Delete them on a remote git annex dropunused --force all --from gdrive I'm using the --force option to force the removal. Otherwise, git-annex tries to make sure that at least one copy of this file exists. And that often results in getting "Unable to lock down 1 copy of file necessary to safely drop it" errors because Google Drive accessed through rclone doesn't support file locking needed to ensure that a copy exists. The git annex dropunused command is not needed if you remember to remove files from the annex with the git annex drop command each time you remove a file from the repository. But I'm too lazy to remember to do that and prefer to remove files by hand (e.g. by pressing "Delete" in my IDE) and clean up git-annex afterwards. Additionally, to verify that the files are really unused, I like to check some of them by finding which file points to the given symlink: find -L /directory/with/images -samefile /path/to/file/in/git-annex For any /path/to/file/in/git-annex file reported by git annex unused, there should be no match from the above find command. List where files are stored # How to see where all the files are stored? There are two ways. A more compact list: git annex list (recording state in git...) here |origin ||SynologyNAS |||web ||||bittorrent |||||GoogleDrive |||||| X_X__X _assets/images/posts/2019-02-27-file1.jpg X_X__X _assets/images/posts/2019-03-04-fiel2.jpg ... Or a bit more verbose version: # More verbose list git annex whereis whereis _assets/images/posts/2019-02-27-file1.jpg (3 copies) 2a241183-7b15-433d-af55-ddecbe876408 -- [nas] c9501338-8824-4d32-82e9-4e9039590943 -- [gdrive] debde584-293f-4bfc-a024-b28bdeb7a00d -- [email protected]:~/workspace/travel-blog [here] ok whereis _assets/images/posts/2019-03-04-file2.jpg (3 copies) 2a241183-7b15-433d-af55-ddecbe876408 -- [nas] c9501338-8824-4d32-82e9-4e9039590943 -- [gdrive] debde584-293f-4bfc-a024-b28bdeb7a00d -- [email protected]:~/workspace/travel-blog [here] ok ... Conclusions # I hope this little git-annex tutorial will be useful to some of you (or at least to me when I need to remind myself how to use it next year). If you have any additional tips for using git-annex, please leave them in a comment. I'm sure I'm still missing a lot of cool features hidden in the corners of the documentation. Would I use git-annex for important projects where being able to retrieve old versions of files is actually crucial? Probably not. git-annex is a good tool, but it's also a complex one. I'm sure some smart DevOps engineers find it easy to use git-annex efficiently without worrying about losing any data. But I'm not that kind of a person. I'm comfortable using git, but I don't have enough experience using git-annex to use it for anything else but a hobby project. If I mess it up, I can always recover the original photos directly from my phone. What else would I use instead? Surprisingly, there are not that many great options - probably because version control of large files is not a common problem (except for the game dev industry, where you need to version your assets that can take hundreds of GBs). For many years, the main alternative has been a commercial tool called Perforce (no matter if you asked that question 8 years ago or last year). But if you do want to relinquish the control and let git-annex take the wheel, you can stop reading here and check out this article on how to set up git-annex with two commands: https://bryan-murdock.blogspot.com/2020/03/git-annex-is-great.html. ↩︎

13th Mar 2024 1 votes

More in programming

Abusing ID3 chapters to turn videos into glanceable podcasts

I listen to a lot of podcasts, and I like how they fit around other tasks. I press play, lock my phone, and put it down. I’m free to wash the dishes, fold the laundry, or shop for groceries. Unfortunately, more and more information is only published as a video. Technical talks, conference sessions, video essays – they don’t work in an audio-only podcast app. I could convert these videos to MP3 files, but that breaks down the moment a video isn’t pure spoken word. If a speaker says, “Look at this slide” or holds up a diagram, an audio-only file leaves me stranded. I don’t want to give up the podcast player I like, nor stare at a screen for an hour – but I do want the information in these videos. To solve this, I’m abusing my podcast player’s chapter support. This gives me the best of both worlds: I can listen to a video as audio-first, and glance at my lock screen if I need a moment of visual context. The idea: Chapters every few seconds MP3 files can have ID3 metadata, and ID3 metadata can include chapters. A chapter covers a particular time range, and it can have an associated title, description, and cover art. My podcast app of choice is Overcast, which can’t play videos, but it does have robust chapter support. I can jump between chapters, navigate a table of contents, and see per-chapter cover art. To get videos into Overcast, I’m creating MP3 files with a new chapter every few seconds, and the per-chapter cover art is a corresponding frame from the video. As I play the file, I get a slow, stop-motion-like rendition of the original video. If my phone is locked, I can glance at my lock screen and see the current frame in the Now Playing screen. Overcast is developed by Marco Arment, and I got this idea from Forecast, his app for adding chapters to podcasts. In particular, I was struck by its ability to create chapters that don’t display in the chapter list – ideal if I don’t want a table of contents with hundreds of entries. As I was developing my script, I compared my output to the output from Forecast to ensure I was creating the chapters correctly. The code: FFmpeg and Mutagen There are three steps in this process: Convert a video file to an MP3 Extract images from the video at a fixed interval Insert the images as hidden chapters in the MP3 file Let’s go through each in turn. 1. Convert a video file to an MP3 Converting a video file to an MP3 is a single FFmpeg command: ffmpeg -i video.mp4 audio.mp3 This is consistently the slowest step of the process, and I do wonder if I could use different settings or an alternative encoder to make it go faster – but it’s not slow enough to be worth further investigation. 2. Extract images from the video at a fixed interval Extracting images from a video needs a more complicated FFmpeg command: ffmpeg -i video.mp4 \ -vf 'fps=1/5,scale=iw*sar:ih,scale=min(iw\,945):min(ih\,945):force_original_aspect_ratio=decrease' \ thumbnail_%04d.jpg This extracts an image every 5 seconds, downscales any image larger than 945 pixels square (while preserving the original aspect ratio), and saves the results as sequentially numbered JPEG images (thumbnail_0001.png, thumbnail_0002.png, and so on). The key is the -vf flag, which defines two FFmpeg filters: The fps filter selects one frame every 5 seconds (fps=1/5). The first scale filter scales the width based on the sample aspect ratio (scale=iw*sar:ih). Without this filter, frames can be stretched and distorted. The second scale filter scales the input video, preserving the original aspect ratio (force_original_aspect_ratio=decrease), and ensuring the output images fit within 945×945px or the size of the input video, whichever is smaller. My limit is 945 pixels because that’s the largest size that cover art is shown on my iPhone. This filter still isn’t completely correct – it sometimes creates images from portrait videos that are smaller than I’m expecting – but it’s good enough. These are only thumbnails for glancing at, and if I want to change it later, I can always do the image resizing outside FFmpeg. 3. Insert the images as hidden chapters in the MP3 file Inserting the chapters into the MP3 file is more complicated. Although FFmpeg has basic support for ID3 metadata, as far as I know, it can’t insert chapters with per-chapter artwork. Instead, I’m going to reach for Python and the Mutagen library. Here’s the code to add a chapter to an MP3 file: from mutagen.id3 import APIC, CHAP, ID3, PictureType audio = ID3("audio.mp3") with open("thumbnail_0001.jpg", "rb") as f: img_data = f.read() image_frame = APIC(mime="image/jpeg", type=PictureType.OTHER, data=img_data) chapter_frame = CHAP( element_id="chp1", start_time=0, end_time=5 * 1000, sub_frames=[image_frame] ) audio.add(chapter_frame) audio.save() This creates a single chapter that lasts the first 5 seconds (0 to 5000 milliseconds), and the per-chapter cover art is thumbnail_0001.jpg. If we ran this in a loop, we could add images for every 5 second slice of the original video. This code is inserting two frames into the ID3 metadata: The CHAP (chapter) frame contains the timing information, and it can have subframes for metadata like title, chapter art, or associated URL. The APIC (attached picture) subframe contains information about a picture, which can either be a blob of image data or a URL to an image on the web. Normally, you’d also insert a CTOC frame which defines a table of contents, but I don’t want a TOC with hundreds of 5-second chapters, so I’m deliberately not doing this here. This is allowed by the ID3 spec – you’re not required to insert a CTOC frame if you’re using chapters, and you can have chapters that aren’t listed in your table of contents. To work out which frames I needed, I used Forecast to create some chapters by hand, and I inspected their frames. In particular, loading an MP3 and calling Mutagen’s pprint() method shows a human-readable list of frames, and then I could drill into the individual fields: from mutagen.id3 import ID3 audio = ID3("audio.mp3") print(audio.pprint()) I wrapped all this code in a project called glancecast, which allows you to convert a video file with a single command, with optional flags to set the frame length and chapter art size: $ python3 glancecast.py interesting_talk.mp4 interesting_talk.mp3 The process takes a minute or so to complete, most of which is spent transcoding the video file to MP3. The resulting MP3s are usually 40 to 50 MB in size, which is very reasonable. The outcome: How it looks in practice Here’s what one of these “glanceable” podcasts looks like in Overcast and on my lock screen: Maggie Appleton presented this talk over two years ago and it’s been on my “talks to watch” list ever since. Once I put it in Overcast? I listened to it in less than a day. It’s not a lot of extra information, but enough that I can quickly glance down and get the gist of what a speaker is saying. Both views update with a new frame every few seconds, or I can put my phone in my pocket and ignore the screen. I’ve used this approach for half a dozen videos so far, and I’m happy with the results. I expect to keep using it, because I have a long queue of videos I’ve been meaning to watch. If you’d like to try this, check out glancecast for the full code and instructions. [If the formatting of this post looks odd in your feed reader, visit the original article]

17 hours ago 1 votes
AI Isn’t Replacing Open Source

Andrew Baker, the current Group CIO at Capitec Bank wrote an interesting piece on AI and open source, and how these tools that generate code according to one’s specification may replace the general reliance on open source implementations done by contributors around the world. I’d really recommend reading it. I have great admiration and respectContinue reading "AI Isn’t Replacing Open Source"

yesterday 1 votes
Confessions of an Unrepentant Slop Snob

A framework for thinking about when AI involvement is additive or a violation

2 days ago 1 votes
Planning with Agents: Divided Worlds, Boundary Objects, and Thicker Interfaces

Why we need richer, thicker interfaces and better boundary objects for collaborative planning with agents

2 days ago 1 votes
Foundations of Agent Friendly Codebases

A look at 10 foundational pillars that enable agents to operate more competently and more efficiently in any codebase.

3 days ago 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in