More from markround.com
Earlier today, I got an email alerting me to an angrier than usual comment on this website. It was a proper keyboard warrior rant accusing me of all sorts of misdeads revolving around “forcing ads down people’s throats”. I replied saying that there had never been any ads on this site, never will be and I detest the enshitification trend of the modern Internet too. I also have found much of today’s web unbearable without tools such as Pi-Hole and a VPN; I use Firefox with adblockers whenever possible and generally speaking, if a site forces me to disable my ad-blocker I’ll simply stop visiting. Then I had a sinking feeling. Years ago, when I migrated this site from a PHP codebase to a static site generator, I’d enabled Disqus comments as it (at the time) seemed a reasonable alternative to dealing with all the spam, moderation, flame wars and handling user data that comes with a comments engine. I’d never really paid that much attention to it as it never got that much use, and I certainly had never noticed any ads or other junk being injected in amongst my content. But then I go to somewhat extreme lengths to avoid that sort of crap, and had a horrible feeling that maybe I’d simply just never noticed anything un-toward as I’d been blocking it. So I downloaded Google Chrome (ugh!), and browsed to this site without any kind of protection or blocking, and was confronted with an absolute abomination of link-farm “chumbox” adverts littering the bottom half of the page. Even though I’m currently on holiday, I immediately disabled the Disqus integration (yay for GitOps and CI/CD pipelines!) and can only apologise for the eye sore and god-awful mess that I was unwittingly inflicting on people. I’m not sure exactly when Disqus got that bad. It certainly wasn’t when I initially set it up, but I guess this is another example of why we can’t have nice things. So to my angry anonymous poster: You were right, I apologise (but you were still kind of a dick about it), and holy crap do I despise what we’ve done to the web.
Discussion on Hacker News Discussion on lobste.rs If you ever get a chance to look through the classic Amiga OS source-code still floating around some murky corners of the internet, it is a thing of beauty and astonishing capabilities. It’s an inspirational piece of computing history with unmatched capabilities for the time. Remember, this was all originally on a computer released in the 1980s with 512Kb memory, a 7Mhz 68000 16-bit CPU, and a single floppy drive with 880Kb storage. On these limited specs, AmigaOS provided a pre-emptive multi-tasking operating system, a full set of GUI primatives and built-in “Workbench” interface, expansion card auto-configuration and a fully-featured filesystem with some unique and powerful capabilities. Although to be fair, the AmigaDOS parts do literally come from a different time (and possibly planet) - but more on that later. Oh and of course, there was that amazing chipset that meant even that humble base can do things like this - while PCs of the time were basically office boxes that occasionally bleeped and home computers still loaded games from cassette tape. There’s understandably a lot of on-line interest in those parts of the Amiga as they’re the most impressive in an obvious “wow!” way. But while that was what drew me to the Amiga when I was a kid (and the demo/cracking/bbs scene heavily influenced me) I’ve always been more of a systems geek at heart. I’ve always loved building tools and platforms, and have long been fascinated with the world of operating systems. Apart from reading through the source code (where that’s legally available, of course…) I think there’s no better way to explore and understand a system - and the mindset that produced it - than to develop for it. What follows is a brain-dump of what I’ve learned about developing for the AmigaOS, both on classic 68k-powered hardware to modern PowerPC systems like the X5000. I’ll cover development environments, modern workflows like CI builds on containerised infrastructure, distribution of packages and even a look back in time before C existed, thanks to AmigaDOS’s odd heritage. Table Of Contents SetCmd SDK Updates Editors Native hardware Modern development AmigaDOS is weird Distribution Archive Documentation Installer File Sites External Documentation The way forward is back ? Wrap-up SetCmd There’s plenty of guides and videos on setting up an old-school game or demo-coding environment, but all of what follows is in the context of developing a systems tool in C as that’s the language of AmigaOS. I started a real-life project partly to solve a small problem I had (switching between different versions of commands/tools at the Amiga CLI) but mainly to explore and dig deeper into the OS that influenced me so much as a teenager. SetCmd was the result, and is a very simple AmigaOS 4 PowerPC package. I’m working (very slowly) on porting it to run on classic AmigaOS and variants but it has to be said this is my first time writing C in any meaningful capacity beyond wrestling with pointers at University. The source code is on GitLab if you want to take a look but bear in mind despite having owned Amigas since they were released, I’m a total newbie at most of this! I wrote it to have fun, explore the AmigaOS, set up build environments and figure out how to package it up for re-distribution. I have written a bit about my development setup in the past, but things have changed a fair bit since then - so without further ado, here’s my development environment and thoughts in 2023. SDK Updates Whilst things do move at a glacial pace in the world of AmigaOS 4/PPC, there have been a few big updates. A-EON’s Enhancer Software has had several releases, each adding new applications and developer APIs. As well as shipping their own versions of key Amiga OS applications and utilities, they also now are installing several core AmigaDOS command replacements. I tend to skip the installation of these as I’ve encountered a few edge cases where they don’t quite behave like the original OS 4 commands, but from recent discussions online it appears as if they are preparing for their own “clean-room” re-implementation and modernisation of Amiga OS 4. Presumably in order to free themselves from the eternal legal shenanigans with Hyperion et al. I’m not going to get into that raging dumpster fire here, but it’ll be interesting to see what comes of this. On the Hyperion side, they released a big SDK Update for OS 4 including updated GCC toolchains, cross-compilers, profilers and loads of updated SDKs. Bearing in mind the ancient GCC 4.x toolchain that had been in place for years it was great to have a more modern environment. On the classic Amiga front, Hyperion have also been pushing ahead with their updated AmigaOS 3.x OS for 68k-powered Amigas. Now on version 3.2.2.1 (I’ve got my boxed set of CD and floppy disks) there have also been several NDK (“Native Developer Kit”) Updates providing updated APIs and tools. AmiKit also released a great “all-in-one” environment called DevPack which includes a huge range of languages (C, Assembly, Amos, Lua, Basic…) and NDKs all configured and ready to go. As a quick and easy way of setting up a development environment on classic Amigas, it’s hard to beat and saves a lot of manual downloading, configuring and glue-ing everything together. Editors In my last update 3 years ago, I’d more or-less settled on using a GUI VIM derivative. While I’m still a die-hard VIM user at $DAYJOB, I really appreciate the modern comforts of e.g. VSCode. Thanks to the amazing work of George Sokianos, there is now a OS 4 package of the awesome Lite-XL editor along with a comprehensive set of plugins. Here’s what a hacking session on my X5000 looks like: In that session you can see alongside LiteXL two terminal windows: I’m compiling and running the PowerPC and classic 68k versions of setcmd thanks to the cross-compilers and native “Petunia” JIT 68k emulation built into OS 4. More on that later, but while we’re talking about classic Amigas… Native hardware Emulation is fine, but nothing beats running on the actual hardware! In the lead image to this article you can see my treasured Amiga 1200 (with older 8-bit friend in the background running my TNFS site) expanded with an Apollo Vampire accelerator. An Ethernet or Wifi adapter is more or less essential though when it comes to transferring data around and fortunately the Vampire card is capable of network connectivity, high-resolution display and other niceities but can easily be switched back to a more “stock” environment. I do still occasionally use my licensed copy of CubicIDE but due to the age of this architecture, I tend to keep my tools light and have settled on the simple Jano editor or sometimes CygnusEd for old time’s sake. My build toolchain is provided by Devpack and it’s included VBCC compiler. I use the vbcc_target_m68k-amigaos target with this makefile to build: Modern development I’m (sadly) not always in front of my Amigas, but these days a modern laptop and cloud-native tools offer a lot of flexibility particularly with the advanced state of emulation. I use VSCode as my editor, and a containerised cross-compiler toolchain built by - again! - George Sokianos to target both 68k and PPC platforms. I can build my project on any system capable of running OCI containers, e.g. docker run \ --user $UID:$GID \ -v ./:/opt/code \ walkero/docker4amigavbcc:latest-m68k-amd64 \ make -f makefile.docker Testing and running the code is made easy by the very advanced state of emulators. On Windows, WinUAE is the gold standard and can emulate everything from an original 1985-vintage A1000 up to modern systems with PowerPC accelerators, graphics cards and other devices. I have it multi-booting into clean Hyperion and Commodore/classic OS environments, with my source code directory shared as a virtual hard-drive: I can compile in seconds with Docker, and then straight away test the resulting binary in my emulated Amiga. Source-code is kept up to date between systems using Git; on the Amiga X5000 I use the port of SimpleGit, which is now bundled with the latest Hyperion SDK under SDK:c/sgit. I haven’t yet found a suitable Git solution for the classic Amigas, so on those I use a makeshift AmigaDOS shell script that uses Backup to copy files over to a network mount in an rsync -like fashion. I also keep meaning to test running AmigaOS 4.1 under QEMU as support for this has greatly improved and looks a lot simpler than the currently convoluted process of getting “classic OS 4” running on an emulated 68k Amiga with PPC accelerator configured. But for now, the WinUAE approach is working pretty well. Another advantage of having a containerised build-chain is that combined with Git and Drone running on my personal Kubernetes clusters, I can build and package my code with a simple git push wherever I am: AmigaDOS is weird Although AmigaOS is frequently lauded for it’s sophistication and elegance, there is a notable “oddness” about the AmigaDOS components which handle storage I/O, devices and filesystems. The original developers of the Amiga had an ambitious DOS system planned, but in the end Commodore had to purchase the Tripos operating system and port parts of it to the Amiga due to deadline challenges. This mismatch is all the more pronounced as Tripos was written in BCPL - which in turn, influenced the B programming language which begat the C we all know and… well, tolerate, in my case. So it really is looking back into computing history and remnants of this still remain even in the “modern” AmigaOS 4.x and other derivatives. Once you start diving into AmigaDOS code, you end up face-to-face with this legacy and need to convert back and forth with BCPL and C data-types. For example, BCPL strings are not NULL-terminated, instead they have a length in the first byte and then the characters follow. And pointers are similarly alien. This is why my code has stuff like this littered through it: // Convert the new node to a BPTR new_node_bptr = MKBADDR(new_node); // Set the new path cli->cli_CommandDir = new_node_bptr; As the NDK include file dos.h explains: “All BCPL data must be long word aligned. BCPL pointers are the long word address (i.e byte address divided by 4 (»2))”. It also includes helper functions like MKBADDR to help with the conversion as most DOS system calls use BCPL pointers in arguments: /* Convert BPTR to typical C pointer */ #define BADDR(x) ((APTR)((ULONG)(x) << 2)) /* Convert address into a BPTR */ #define MKBADDR(x) (((LONG)(x)) >> 2) All in all, a fascinating look back into an obscure branch of computing history, but it hasn’t furthered my appreciation of pointers any! Distribution If you want to distribute your project to a wider audience, there’s a few Amiga-specific things you can do that makes life much easier for people running AmigaOS. These all make use of some pretty cool bits of Amiga technology and have been widely adopted by native software since their introduction in the early 1990s. Archive Just use LHA format archives. It’s the standard compression tool on Amigas and even though there are modern (and technically better) alternatives, .lha files are as ubiquitous as e.g. .zip or .tar.gz packages on other systems and can also be handled by low-spec machines. There are ports of CLI tools and GUIs available on all platforms to handle these archives and while the syntax can be a little different, it’s quite easy to use. See my AmigaDOS script that builds the SetCmd release artifact for a practical example. Documentation Along with a basic README.txt, it’s a great practice to distribute more detailed documentation in AmigaGuide format. Another area the Amiga was way ahead of it’s time - AmigaGuide is a hyper-text format commonly used for application manuals although people have even used it to publish disk magazines! Introduced in 1992, the bundled tools on any AmigaOS (or clone/derivative) can read and use AmigaGuide as standard so you can include formatting, links and other content in your documentation. You can see the AmigaGuide docs I include with SetCmd in the screenshot above, or view the source to see what the syntax looks like. It’s pretty simple to write and is much like any other markdown or formatting code. You can set basic parameters: 1 2 @Width 72 @wordwrap Documents have “Nodes” which can be linked to, e.g. 1 2 @Node About "About SetCMD" @{"About" Link About} And formatting is much like HTML with opening and closing tags: @{b}@{u} Bold and Underlined! @{uu}@{ub} Installer A fantastic addition to AmigaOS, the system installer utility reads a developer-provided script which handles copying files, comparing versions, modifying system scripts and so on, in a standardized fashion. You can pass useful information and configuration which controls the Installer tool through standard Amiga tooltypes, and it uses a sort of LISP-ey syntax which again runs on all Amigas and derivatives. The syntax does some getting used to, but the best source of documentation is the AmigaGuide documentation found in the Installer dev package. As an example, here’s an excerpt of the block of code which copies the setcmd program file over to a previously created directory: 1 2 3 4 5 6 7 8 (copyfiles (source "setcmd") (dest #dname) (prompt ("Copy SetCmd program file?")) (confirm "expert") (all) (help @copyfiles-help) ) And for running commands, you can use the run command, along with cat (short for “concatenate”) to build up the command string : (run (cat "C:MakeLink FROM " #dname "/cmds/setcmd/release TO SETCMD:setcmd SOFT")) I found the best approach was to examine other Installer scripts to get a feel for common practices and idioms. Here’s my simplistic Install_SetCmd script, and if you want to see something more complex, there’s always the AmigaOS installation scripts, or the Qt installer for OS 4 which taught me a lot. File Sites The Amiga doesn’t have a universal package manager, so files are usually downloaded manually and installed from the .lha archives. The go-to place for Amiga software for all systems is AmiNet. It’s the biggest repository of Amiga packages on the internet (and, at one point in the mid-90s was actually the largest software repository of any platform) and now also supports hosting packages for Amiga OS 4, MorphOS and AROS alongside classic 68k fare. There are also smaller, platform-focused sites for each platform e.g. os4depot.net for OS 4, morphos-storage.net for MorphOS and so on. Getting your package uploaded and accepted into the repository is broadly the same for all these: You FTP your package up according to the naming standards, and supply a Readme file which provides the required metadata like this excerpt in AmiNet format: 1 2 3 4 5 6 Short: Switch between versions of software Author: amiga@markround.com (Mark Dastmalchi-Round) Type: util/shell Version: 1.1.0 Architecture: ppc-amigaos >= 4.0.0 Distribution: Aminet There are some Amiga-native GUI tools that assist with creating these files, but the specs e.g. for os4depot.net are pretty straight forward. And here’s the end result - My package available on os4depot.net and aminet. External Documentation When trying to learn or re-learn everything from C to AmigaDOS scripting, I found a few great resources. However, as with most things in Amiga-land, there’s an extraordinarily high “bus factor” for many websites and my biggest recommendation is to use native tools or save local copies of anything you find! With that said, here’s my essential Amiga bookmarks: Autodocs references. There’s lots of websites where you can browse the auto-generated docs from the SDK header files, like this with clickable links to jump between things. If I’m actually at an Amiga though, there are some useful native tools I prefer that can index and search through the local headers. The screenshot above shows the standard “AutoDoc Reader” freeware tool viewing the equivalent of a man page for the AmigaDOS library, alongside the AmigaGuide Installer language reference. http://www.pjhutchison.org/tutorial/amiga_c.html - amazing site. This is what inspired me to pick up a compiler again and get to work. There’s a great refresher on the C language itself, and then it dives into Amiga-specific coding with everything from low-level library access, sound and GUI programming and more. Amiga OS Dev wiki is a goldmine, although it can take a little searching to find what you’re after. It’s mostly OS 4-focused but because all Amiga systems share a common ancestor it’s usually pretty applicable to all platforms. Specific articles that I found useful include: OS 4 Migration Guide Programming in the Amiga Environment Fundamental Types https://www.amiga-news.de is a great news aggregator site for all things Amiga, and also has a bunch of exclusive articles on programming in the AmigaOS environment - Like this recent article on GUI programming. Well worth a read - thanks to Daniel Reimann for pointing out the great content to me! And lastly, there are great threads I constantly found on amigans.net which is where a lot of OS 4/”Next Gen” technical discussion happens. For classic systems, I found the Coders discussions on the English Amiga Board an invaluable resource. The way forward is back ? When I started this project, it was really a way to get acquainted with my new X5000. Since then, I’ve decided to port my codebase back to the classic Amiga, as well as explore porting over to other Amiga-like systems such as MorphOS and AROS. This leads to some choices: From a packaging and distribution point of view, a 68k binary is pretty much the universal standard in Amiga land. It can run natively on classic Amigas, and modern systems like AmigaOS 4.x and MorphOS can run 68k binaries through translation. In a method similar to how Apple has handled the transition between processors in the Mac, it’s a pretty seamless experience and I run a lot of classic 68k software on my X5000. As long as you aren’t “banging on the metal” it works really well and integrates smoothly with the rest of the system. The original 68k AmigaOS from Commodore is also pretty much the standard for source-code compatibility; code targetting this release can be built on most of the derivatives and later systems with very little (if any) modification. On AmigaOS 4 for example, you can simply add -D__USE_INLINE__ to your makefiles and in theory build from a common codebase. If you start the other way as I did and write initially targetting AmigaOS 4, it’s harder to port to other systems. For example, I originally followed the AmigaOS 4 programming style which favours prefixing library calls with interface names. This isn’t compatible with any other system, so the easiest way to port this to more Amiga-like platforms is to refactor this code back to the classic style of calling system functions. I do plan on building platform-specific binaries using #defines so I can for example use functions like dos.library/AddCmdPathNode on OS 4 that I otherwise have to manually implement, and while a lot of higher-level layers (like e.g. MUI for building graphical applications) are shared across platforms this is probably the best bet for adding specific features from one platform that aren’t available on others. Honestly though, at this point if you want to just get started I’d have to suggest you target classic AmigaOS compatibility and build a 68k binary. I’d personally target AmigaOS 3.x or 2.1 if you want to support a wider range of truly vintage systems; 1.x is facinating from a retro-geek perspective but lacks a lot of the nice features that came with later systems. Everything else like the installer, archive format, documentation format and so on is cross-platform anyway and supported from OS 2.1 and up. MorphOS, AROS and OS 4 are really fun systems to explore, and I highly recommend checking them out if this article has whetted your appetite (and you can find a system to run them on!) but classic is the easiest way to get your code out to the wider world and ironically provides a better code-base for future porting and native binaries than my “working backwards” approach. Wrap-up So that’s about the sum total of what I’ve picked up over the last few years, anyway! I still enjoy working on my Amigas when I get some “hacking on code in the evening” time, and in particular I find AmigaOS 4 on my X5000 a refreshing blend of retro appeal and just about enough modern convenience to use it for development tasks, or even for writing this article itself. My A1200 continues to impress me with how much utility there is in such a small box and is a wonderful distraction from the modern era of bloated systems and applications. It is perhaps an evolutionary dead-end, but it’s still a lot of fun and is one of the rare occasions these days where I feel actually in control of my computer. Working backwards in time from OS 4.1 to my classic Amigas has also really given me a greater insight and appreciation for what the Amiga engineers managed to pull off back then. If you’re in any way interested in computer history - or simply want to give something truly different a try - you should definitely check out AmigaOS. I hope this quick type BRAIN: > WEB: dump provides you with some good starting points, and maybe gets you coding too!
Discussion on Hacker News Discussion on lobste.rs I’ve long since been a die-hard BeOS fan and have been running the open-source recreation Haiku for many years. I think it’s interesting to explore the “alternative OS” world and consider some great ideas that for whatever reason never caught on elsewhere. The way Haiku handles package management and its alternative approach to an “immutable system” is one of those ideas I find really cool. Here’s what it looks like from a desktop user’s perspective - there’s all the usual stuff like an “app store”, package updater, repositories of packages and so on: It’s all there and works well - it’s easily as smooth as any desktop Linux experience. However, it’s the implementation details behind the scenes that make it so interesting to me. Haiku takes a refreshingly new approach to package management: Despite the user experience “feeling” like a traditional package manager - say, something like apt or dnf - it has seamless support for: Immutable system directories Rollback to previous states User-managed packages separated from system packages And a whole bunch of infrastructure and tooling to support multiple package repositories, building packages from source and more. As a geek, I find it beautifully elegant and an idea that I’d love to see other platforms exploring. Let’s take a closer look, starting with how Haiku handles the split between “system” and “user” directories. Filesystem layout Running a df command from the Haiku shell (BASH is the default, but others like ZSH can be easily installed) shows the following: ~> df -h Mount Type Total Free Flags Device ----------------- --------- --------- --------- ------- ------------------------ /boot bfs 232.9 GiB 219.6 GiB QAM-P-W /dev/disk/scsi/0/0/0/0 /boot/system packagefs 4.0 KiB 4.0 KiB QAM-P-- /boot/home/config packagefs 4.0 KiB 4.0 KiB QAM-P-- In Haiku, the root / is a ram-based virtual filesystem set up by the kernel when it’s booted. All other filesystems and devices are mounted under /, so /boot refers to the entire boot volume - and not a boot partition as is the case with e.g. most Linux installs. The /boot/system mountpoint is essentially the system directory which makes up the Haiku OS and installed applications. In the root directory, there are a bunch of symlinks that point there for convenience: ~> ls -l / total 6 lrwxrwxrwx 1 user root 16 Feb 13 14:18 bin -> /boot/system/bin drwxr-xr-x 1 user root 2048 Mar 31 2021 boot drwxr-xr-x 1 user root 0 Feb 13 14:18 dev lrwxrwxrwx 1 user root 25 Feb 13 14:18 etc -> /boot/system/settings/etc lrwxrwxrwx 1 user root 5 Feb 13 14:18 Haiku -> /boot lrwxrwxrwx 1 user root 26 Feb 13 14:18 packages -> /boot/system/package-links lrwxrwxrwx 1 user root 12 Feb 13 14:18 system -> /boot/system lrwxrwxrwx 1 user root 22 Feb 13 14:18 tmp -> /boot/system/cache/tmp lrwxrwxrwx 1 user root 16 Feb 13 14:18 var -> /boot/system/var So we can access e.g. /boot/system as /system and so on. Note that this mount point is backed by packagefs - this is provided by a virtual filesystem that presents a merged view of all the packages installed. It’s sort of like an overlay filesystem (commonly used by container runtimes) in the Linux world. This mount-point is read only: ~> touch /system/test touch: cannot touch '/system/test': Read-only file system But we can however write to anything in our home directory, which ensures a clean separation of system and user data/configuration. NOTE : Due to its BeOS ancestry, Haiku is not currently a multi-user system so there is only one /boot/home directory and the user is effectively the sole administrator account. As I understand it, the scaffolding is present to support multiple users, but it won’t be a priority until after R1 is released. However, this opens up the later possibility for packages to be installed and configured on a per-user basis. The Package Daemon and packagefs There’s a great overview of how all these components fit together in the developer documentation, but here’s my lay-persons understanding of it… Haiku has many packages available, ranging from system components, development libraries and end-user applications. There’s even recent ports of Wine, LibreOffice and KDE Applications. These are available as .hpkg files which are placed in a special packages directory, and the packagefs service mounts the contents into e.g. the /boot/system mountpoint. Unlike traditional package management tools, the contents of the package are not unarchived and copied into the filesystem; When you’re looking at /boot/system, you’re essentially looking at a collection of multiple packages and their contents, all dynamically mounted and made available on-the-fly as a read-only, virtual filesystem. This is why /boot/system is read-only: It’s not a “real” filesystem and only exists as a virtual union of all the different packages that are activated at that point in time. NOTE : There are some exceptions to this, as some directories such as packages, cache, var etc. are writeable. There are called “shine-through directories” which reside on the underlying BFS volume. You can read more about these in the developer documentation. And while we’re geeking out, BeFS itself is definitely also worth investigating! When packagefs detects a new .hpkg file has been copied to the packages directory, it performs some checks such as searching for dependencies or conflicts. If everything is OK it “activates” the package, and the contents are then available. Installing a package Here’s an example of installing a package using the CLI pkgman tool, showing what happens behind the scenes. First, let’s install an example package, in this case something simple like the awesome CLI Pipe Viewer tool. It’s very much like running apt-get, yum or other similar tools on a Linux system: ~> pkgman install pv Validating checksum for Haiku...done. 100% repochecksum-1 [64 bytes] Validating checksum for HaikuPorts...done. The following changes will be made: in system: install package pv-1.6.6-2 from repository HaikuPorts 100% pv-1.6.6-2-x86_64.hpkg [40.57 KiB] Validating checksum for https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/pv-1.6.6-2-x86_64.hpkg...done. [system] Applying changes ... [system] Changes applied. Old activation state backed up in "state_2023-02-13_14:33:27" [system] Cleaning up ... [system] Done. If we now look at /system/packages we can see the downloaded .hpkg file (which can also be inspected with CLI or Desktop tools): ~> ls -l /system/packages/pv-1.6.6-2-x86_64.hpkg -rw-r--r-- 1 user root 41543 Feb 13 14:33 /system/packages/pv-1.6.6-2-x86_64.hpkg Sure enough, the package daemon picked it up and mounted its contents so they are available through the merged packagefs under /boot/system. Here’s where the pv binary now appears installed: ~> ls -l /system/bin/pv -r-xr-xr-x 1 user root 70136 Aug 6 2018 /system/bin/pv Uninstalling is as simple as pkgman uninstall pv which removes the .hpkg file, and the contents then “disappear” from /boot/system. State and Rollback What’s really nice about this, is that it enables a simple and elegant way of rolling-back your system state to a previous set of packages or even OS releases. If you check the last output from the pkgman install pv command above, you’ll see there’s a message saying that an “old activation state” is being backed up to a newly-created directory. The package manager does this at the start of every transaction, and if we check that directory we’ll see a simple plain-text file called activated-packages: ~> head -n5 /system/packages/administrative/state_2023-02-13_14\:33\:27/activated-packages netcat-1.10-4-x86_64.hpkg ncurses6-6.3-2-x86_64.hpkg mpfr3-3.1.6-6-x86_64.hpkg mesa_swpipe-22.0.5-2-x86_64.hpkg mesa_devel-22.0.5-2-x86_64.hpkg This contains a record of the exact package versions that were installed at that time. If any packages were to be upgraded, then the state directory will also contain a set of packages to facilitate roll-back. Here’s what it looks like from the Haiku Desktop: Along with a listing of old packages in a state directory, the currently installed packages are shown on the top left. You can see for example, that BASH in the current system is at 5.1 but in an old backup state from 2021 it was 5.0. You can also clean these old state directories up according to your needs - such as only keeping the last 30 days of state to preserve disk space. Tying this all together, the Haiku Boot Loader can make use of activation lists and saved packages to get back to any particular state very easily - all it has to do is pick a backup package activation file, and only activate the packages found in it when booting. You can select a backup state from the “Select Boot Volume” option in the boot loader and your system will boot back to the previous state using the archived packages and activation list. Your virtual /boot/system directory will then be reverted to the desired state - and this all happens on-the-fly at boot time; there is no long roll-back process or destructive operations and you can reboot into different states at will. User packages So far, I’ve just been talking about the packagefs mountpoint at /boot/system, but there’s also another mountpoint shown in my first df -h command which lives under /boot/home/config. Here’s what exists under that directory: ~/config> tree -d /boot/home/config -L 1 /boot/home/config ├── apps ├── cache ├── data ├── non-packaged ├── packages ├── settings └── var This is more-or-less a copy of the system directories, but they are specific to my user account. This means I can use this location to install software and test new packages out without “polluting” the system directories. And when Haiku gets full multi-user support this also means each user can have their own distinct set of packages available. A couple of points of interest as an aside: For non-native software packages (e.g. something installed with ./configure && make install) you can use ~/config/non-packaged. This is somewhat similar to /usr/local on Unix systems. The settings directory is where Haiku-packaged software keeps local configuration. To use an analogy again, it’s sort of like $XDG_CONFIG_HOME on other Unix-like systems. Haiku software tends to make use of this directory: For example, instead of SSH keeping configuration under ~/.ssh, it’s now stored under a directory in ~/config/settings: ~/config> ls -l settings/ssh/ total 32 -rw------- 1 user root 1238 Mar 31 2021 authorized_keys -rw------- 1 user root 476 Mar 31 2021 config -rw------- 1 user root 1679 Mar 31 2021 id_rsa -rw------- 1 user root 410 Mar 31 2021 id_rsa.pub -rw------- 1 user root 1790 Dec 18 2021 known_hosts Building your own packages I’ll finish off by showing an example of building a user package and installing it using Haikuporter and the community HaikuPorts collection. These tools can be used to build and customise a massive amount of software for Haiku ranging from old BeOS tools to a complete LibreOffice installation. NOTE : Think of building HaikuPorts from source like the FreeBSD “ports” collection. All the HaikuPorts packages are available through their repository as binaries through pkgman or the HaikuDepot graphical desktop application. As an end-user, you typically do not need to use Haikuporter unless you want to customise a package, create a new one, or submit patches/bug-fixes. I’m just using it as an example and because it’s cool! After setting up Haiku Ports and Haikuporter by following the instructions, I can now build a package from source. I’ll use the GUI FTP client “FTP Positive” as an example: ~/haikuports/haiku-apps> alias hp="haikuporter -S -j8 --no-source-packages --get-dependencies" ~/haikuports/haiku-apps> hp ftppositive Checking if any dependency-infos need to be updated ... Looking for stale dependency-infos ... ---------------------------------------------------------------------- haiku-apps::ftppositive-1.2.2 /boot/home/haikuports/haiku-apps/ftppositive/ftppositive-1.2.2.recipe ---------------------------------------------------------------------- Skipping download of source for 48a5acdfe0981697018abf151a82802f4f3e500e.tar.gz Validating checksum of 48a5acdfe0981697018abf151a82802f4f3e500e.tar.gz Unpacking source of 48a5acdfe0981697018abf151a82802f4f3e500e.tar.gz ... ... Build output truncated ... mimesetting files for package ftppositive-1.2.2-7-x86_64.hpkg ... creating package ftppositive-1.2.2-7-x86_64.hpkg ... ----- Package Info ---------------- header size: 80 heap size: 211523 TOC size: 1110 package attributes size: 695 total size: 211603 ----------------------------------- waiting for build package ftppositive-1.2.2-7 to be deactivated grabbing ftppositive-1.2.2-7-x86_64.hpkg and moving it to /boot/home/haikuports/packages/ftppositive-1.2.2-7-x86_64.hpkg The last lines of output show me that a .hpkg file has been created. I can then simply copy the package to my local ~/config/packages directory to “activate” it. Once again, packagefs will check it and then add the contents to the /boot/home/config directory. Here’s what it looks like on the Haiku desktop: Note the top two windows “stuck” together using the built-in stacking and tiling window manager! You can see the package has been activated after copying it into the user’s packages directory. It’s installed the application to /boot/home/config/apps/FtpPositive/ instead of the system directories and has added a DeskBar menu entry which has also been picked up and “merged” into the global system Applications menu - where it appears alongside all the system-installed packages. Conclusion Like the rest of Haiku, package management is a refreshingly different and well thought-out experience. Given how niche an OS it is, it still surprises me how polished the system is - I have it running natively on an old Lenovo ThinkStation SFF PC and it absolutely flies. Apart from my Amigas it’s easily my favourite “hacking on code in the evening” system and for many tasks is perfectly capable of being a daily-driver. There’s a fascinating world of alternative OSes out there, many of them following entirely different paradigms than the current mainstream world of Windows/Mac/Linux. I’ve had the good fortune to be exposed to these different ways of thinking all the way back to my school days in the UK, where RISC OS was commonplace in the classroom and Amigas and Ataris ruled the playground. From this once-rich polyculture of competing processor architectures and platforms, the world seemingly consolidated itself around an x86 or ARM processor running something based on Windows or Unix. Which gets the job done, but it’s y’know… a little boring. There are so many interesting - and some downright crazy - ideas that either failed in the commercial marketplace, were never given a proper chance (yeah, I’m still salty about Commodore killing the Amiga) or only found a hobbyist following. But here’s the thing: Just because they never got wide adoption doesn’t mean they were wrong. I think it’s great that people are exploring new ideas, continuing the lineage of legacy systems, and simply creating stuff because they damn well feel like it and it’s fun! There’s lots of really interesting code out there and it makes you wonder what an alternate time-line would look like where Apple did use BeOS as the basis for their next operating system. What if VMS had “won” over UNIX? What if Commodore had known what to do with the Amiga ? Anyway, I hope this has whetted your appetite for all things Alt-OS, and Haiku in particular. If you haven’t yet tried it, there’s detailed instructions on their website, and I highly recommend taking it for a spin. Happy Hacking!
In Part 3 I covered the backend server processes and protocols, CI/CD pipelines and unit tests I used to build the TNFS site. In this (much shorter) part, I’d like to take a step back from the hardcore geekery, and wrap up with my thoughts on the whole thing. Other Sites But before that, I’m going to explore a little part of the rest of the TNFS universe. After all, this project is intended to build a community site, and the Speccy has one of the friendliest retro computing communities out there. My site isn’t the only one out there - there’s a whole network of TNFS servers on the public internet, and the protocol has also been adopted for 8-bit Atari systems. There’s currently no central directory as such (although work is underway to create an index system using DNS TXT records) but there’s a forum thread that gets regular updates, and I have a links section on my site’s main menu. To give you an idea of some of the great content others have built, here’s a quick overview and screenshots of some of my favourite Speccy TNFS sites… Spectranet-related vexed4.alioth.net A very useful site to have bookmarked in one of the Spectranet’s slots. Hosted by the developer of the Spectranet’s firmware, It has some nice lists of classic and modern (2000s-era) games, demos and also some internet utilities including IRC and Twitter gateways. The first option in the menu is an online firmware update utility for Spectranet cards. The updater doesn’t work on an emulated Spectranet but works great on the real hardware - this is one of the first sites you should visit! The weird and wonderful tnfs.bytedelight.com This site takes up the default slot in Spectranet cards purchased from ByteDelight.com and serves as a demonstration. When you first configure your card, it’ll boot straight into this site which displays a snazzy “SkyNet” intro sequence informing you that you Speccy has been taken over, and all your base are belong to them. zx.zapto.org This wonderful fever-dream of a site was produced by a “p13z”, a regular on the Spectrum Computing Forums. It’s an impressive demonstration of what can be achieved using plain Sinclair BASIC - It’s sort of an interactive adventure where you start off by driving away from a police car against a neat parallax-scrolling background. You then end up wandering around various locations including a car-park frequented by “doggers” (and a telephone box which lets you connect to other TNFS sites), some standing stones where you can sample some “banging mushrooms” and other weird delights. Utterly bonkers in a classic Speccy kinda way… Gateways zx.desertkun.in Home of the Channels project, which provides a ZX Spectrum browser for forums and imageboards. If you boot into this site, set zx.desertkun.in as your proxy in the opening screen, and you can connect to internet forums and message boards including the Spectrum Computing Forums! There’s a Docker Image and source code available so you can run and customise things on your own systems. This uses a similar approach to my site where the heavy-lifting (in this case, TLS processing and connecting to/parsing the data from websites) is shifted to a more-capable modern environment. Awesome work! irata.online Now this one is a proper rabbit hole you could spend days exploring. It’s a fascinating part of computer history I had never encountered before: A visionary system that hosted some of the earliest ever online message boards, as well as apparently inspired the creation of Castle Wolfenstein and Lotus Notes! Their website explains it: “IRATA.ONLINE is provided for the benefit of retro-computing users to have a place to socialize, and develop interesting multi-user, interactive, and graphical games and social applications. It descends from the historical PLATO system, a massive time-sharing system that lasted from 1962 until NovaNET was closed in 2015.” The TNFS site hosts a terminal application that connects to the PLATO system (other retro systems are also supported) and you can even browse it through the web at https://js.irata.online/. Amazing stuff, and well worth some time checking out. nihirash.net This TNFS site hosts the uGophy Gopher client for the Spectrum. This lets you browse “gopherspace” from your speccy - try out gopher.floodgap.com as a starting point. You can use a web browser to access the HTTP proxy at https://gopher.floodgap.com/gopher/ to get a useful list of links to other Gopher sites still in operation, including the Veronica-2 search engine. zxnet.co.uk As well as a couple of “experiments” (a multi-player Tank game and music Jukebox), the zxnet.co.uk TNFS site hosts a copy of snapCterm (see below) and an IRC gateway. The IRC gateway allows you to connect to any public IRC server network and chat with other users - try libera.chat to get started, and use their excellent online help guides to find some interesting channels to join. snapCterm SnapCTerm is a terminal emulator that provides the basics of an ANSI 80 column terminal for the plus machines. It supports ASCII characters as well as ANSI escape and colour codes. This means you can use it to connect to Telnet servers e.g. a Linux system running telnetd which makes it possible to access the many old-school BBS systems still running. This is how we used to “social network”, kids, before this new fangled web thing took over… There’s a great curated list of systems to connect to at https://www.telnetbbsguide.com/ including all those “elite” sites that used to be advertised all over Amiga demo/warez scene productions. You can see in the screenshot a connection starting to The Sanctuary BBS, running on AmiExpress (still being updated!) and which used to be Fairlight’s World Headquarters back in the day. SnapCTerm is available on my TNFS site (2nd menu page, option 3) as well as on sites like zxnet.co.uk. File sites tnfs.millhill.org This is an archive of the zx.kupo.be TNFS site, which at one point hosted one of the largest collections of games and files for the Speccy. Sadly, Adam Colley who ran the site passed away in 2020. Kupo was one of the first sites I connected to when I first got my Spectranet card and looking through the code gave me the inspiration to start my own site. His work is archived here by someone who was in close contact with him before he passed, and it’s fitting that his site lives on and will be remembered by the community. retrojen.org As well as a nice little “blizzard” intro, retrojen.org hosts around 500 classic Spectrum scene demo tapes. Use the QAOP keys to move around the menu system, ENTER to select a title, 1 to jump to the first page and F to find a production by the release date. A great little archive of some old gems! Thanks for the recent post on my message wall, too :) szeliga.zapto.org This site hosts a curated collection of modern (mid-2000s onwards) games and titles released for the Speccy. There’s some great games that showcase the best of the current Speccy scene. Many of these titles are also available on my site, but it’s great to have them all arranged in a curated date order like this. I highly recommend trying out 2019’s “Yazzie”, a fantastic platformer/puzzle game that is ridiculously addictive. Wrap Up Since the site went online in January 2021 it’s grown to nearly 10k lines of code, handled over 11,500 connections and has gathered a small community of users chatting, playing games and leaving messages. It’s connected me to a wonderful (if slightly crazy) network of people who care about this little squidgey-keyed black box from the 80s as much as I do. It’s been a real pleasure seeing first users come to the site and I have lots of plans still for the future! Just a few of the bits and pieces in various Git branches right now: Proper pagination indication on all menus (page x of y type stuff) Message Board improvements (big speed boost and longer retention) Reply to comments - this opens up the possibility of a proper message board system! More files to upload and more curated lists I’m open to suggestions! If you have any ideas or fancy writing a text article for the site, let me know… In the words of one of the messages from a user: “It’s mad being online with a 40yr old computer designed around a tape recorder.” Perhaps it’s even crazier to drag the whole shebang into the modern age and cobble together infra-as-code pipelines, unit testing and gratuitous amounts of Kubernetes around it all. I did use a lot of my current-day practical knowledge during this project, but the most enjoyable parts where when I was doing stuff like studying the decades-old intricies of the VAL$ function. Sometimes, introducing constraints like an ancient 8-bit processor with only 48Kb of usable RAM can really force you to think “outside the box” and produce the most creative hacks. Plus, in this day and age it’s a lot of fun to do something just for the sheer hell of it! I’m also definitely going to add Sinclair BASIC to my “skills” in LinkedIn now ;) Before I close, I’d just like to add a note of thanks to everyone who’s used my site, suggested features, left messages or helped me out with my many technical queries on forums. It’s been an absolute blast and I look forward to the next 40 years of Spectrum hacking! -Mark, February 2022.
More in programming
Once you’ve written your strategy’s exploration, the next step is working on its diagnosis. Diagnosis is understanding the constraints and challenges your strategy needs to address. In particular, it’s about doing that understanding while slowing yourself down from deciding how to solve the problem at hand before you know the problem’s nuances and constraints. If you ever find yourself wanting to skip the diagnosis phase–let’s get to the solution already!–then maybe it’s worth acknowledging that every strategy that I’ve seen fail, did so due to a lazy or inaccurate diagnosis. It’s very challenging to fail with a proper diagnosis, and almost impossible to succeed without one. The topics this chapter will cover are: Why diagnosis is the foundation of effective strategy, on which effective policy depends. Conversely, how skipping the diagnosis phase consistently ruins strategies A step-by-step approach to diagnosing your strategy’s circumstances How to incorporate data into your diagnosis effectively, and where to focus on adding data Dealing with controversial elements of your diagnosis, such as pointing out that your own executive is one of the challenges to solve Why it’s more effective to view difficulties as part of the problem to be solved, rather than a blocking issue that prevents making forward progress The near impossibility of an effective diagnosis if you don’t bring humility and self-awareness to the process Into the details we go! This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts. Diagnosis is strategy’s foundation One of the challenges in evaluating strategy is that, after the fact, many effective strategies are so obvious that they’re pretty boring. Similarly, most ineffective strategies are so clearly flawed that their authors look lazy. That’s because, as a strategy is operated, the reality around it becomes clear. When you’re writing your strategy, you don’t know if you can convince your colleagues to adopt a new approach to specifying APIs, but a year later you know very definitively whether it’s possible. Building your strategy’s diagnosis is your attempt to correctly recognize the context that the strategy needs to solve before deciding on the policies to address that context. Done well, the subsequent steps of writing strategy often feel like an afterthought, which is why I think of diagnosis as strategy’s foundation. Where exploration was an evaluation-free activity, diagnosis is all about evaluation. How do teams feel today? Why did that project fail? Why did the last strategy go poorly? What will be the distractions to overcome to make this new strategy successful? That said, not all evaluation is equal. If you state your judgment directly, it’s easy to dispute. An effective diagnosis is hard to argue against, because it’s a web of interconnected observations, facts, and data. Even for folks who dislike your conclusions, the weight of evidence should be hard to shift. Strategy testing, explored in the Refinement section, takes advantage of the reality that it’s easier to diagnose by doing than by speculating. It proposes a recursive diagnosis process until you have real-world evidence that the strategy is working. How to develop your diagnosis Your strategy is almost certain to fail unless you start from an effective diagnosis, but how to build a diagnosis is often left unspecified. That’s because, for most folks, building the diagnosis is indeed a dark art: unspecified, undiscussion, and uncontrollable. I’ve been guilty of this as well, with The Engineering Executive’s Primer’s chapter on strategy staying silent on the details of how to diagnose for your strategy. So, yes, there is some truth to the idea that forming your diagnosis is an emergent, organic process rather than a structured, mechanical one. However, over time I’ve come to adopt a fairly structured approach: Braindump, starting from a blank sheet of paper, write down your best understanding of the circumstances that inform your current strategy. Then set that piece of paper aside for the moment. Summarize exploration on a new piece of paper, review the contents of your exploration. Pull in every piece of diagnosis from similar situations that resonates with you. This is true for both internal and external works! For each diagnosis, tag whether it fits perfectly, or needs to be adjusted for your current circumstances. Then, once again, set the piece of paper aside. Mine for distinct perspectives on yet another blank page, talking to different stakeholders and colleagues who you know are likely to disagree with your early thinking. Your goal is not to agree with this feedback. Instead, it’s to understand their view. The Crux by Richard Rumelt anchors diagnosis in this approach, emphasizing the importance of “testing, adjusting, and changing the frame, or point of view.” Synthesize views into one internally consistent perspective. Sometimes the different perspectives you’ve gathered don’t mesh well. They might well explicitly differ in what they believe the underlying problem is, as is typical in tension between platform and product engineering teams. The goal is to competently represent each of these perspectives in the diagnosis, even the ones you disagree with, so that later on you can evaluate your proposed approach against each of them. When synthesizing feedback goes poorly, it tends to fail in one of two ways. First, the author’s opinion shines through so strongly that it renders the author suspect. Your goal is never to agree with every team’s perspective, just as your diagnosis should typically avoid crowning any perspective as correct: a reader should generally be appraised of the details and unaware of the author. The second common issue is when a group tries to jointly own the synthesis, but create a fractured perspective rather than a unified one. I generally find that having one author who is accountable for representing all views works best to address both of these issues. Test drafts across perspectives. Once you’ve written your initial diagnosis, you want to sit down with the people who you expect to disagree most fervently. Iterate with them until they agree that you’ve accurately captured their perspective. It might be that they disagree with some other view points, but they should be able to agree that others hold those views. They might argue that the data you’ve included doesn’t capture their full reality, in which case you can caveat the data by saying that their team disagrees that it’s a comprehensive lens. Don’t worry about getting the details perfectly right in your initial diagnosis. You’re trying to get the right crumbs to feed into the next phase, strategy refinement. Allowing yourself to be directionally correct, rather than perfectly correct, makes it possible to cover a broad territory quickly. Getting caught up in perfecting details is an easy way to anchor yourself into one perspective prematurely. At this point, I hope you’re starting to predict how I’ll conclude any recipe for strategy creation: if these steps feel overly mechanical to you, adjust them to something that feels more natural and authentic. There’s no perfect way to understand complex problems. That said, if you feel uncertain, or are skeptical of your own track record, I do encourage you to start with the above approach as a launching point. Incorporating data into your diagnosis The strategy for Navigating Private Equity ownership’s diagnosis includes a number of details to help readers understand the status quo. For example the section on headcount growth explains headcount growth, how it compares to the prior year, and providing a mental model for readers to translate engineering headcount into engineering headcount costs: Our Engineering headcount costs have grown by 15% YoY this year, and 18% YoY the prior year. Headcount grew 7% and 9% respectively, with the difference between headcount and headcount costs explained by salary band adjustments (4%), a focus on hiring senior roles (3%), and increased hiring in higher cost geographic regions (1%). If everyone evaluating a strategy shares the same foundational data, then evaluating the strategy becomes vastly simpler. Data is also your mechanism for supporting or critiquing the various views that you’ve gathered when drafting your diagnosis; to an impartial reader, data will speak louder than passion. If you’re confident that a perspective is true, then include a data narrative that supports it. If you believe another perspective is overstated, then include data that the reader will require to come to the same conclusion. Do your best to include data analysis with a link out to the full data, rather than requiring readers to interpret the data themselves while they are reading. As your strategy document travels further, there will be inevitable requests for different cuts of data to help readers understand your thinking, and this is somewhat preventable by linking to your original sources. If much of the data you want doesn’t exist today, that’s a fairly common scenario for strategy work: if the data to make the decision easy already existed, you probably would have already made a decision rather than needing to run a structured thinking process. The next chapter on refining strategy covers a number of tools that are useful for building confidence in low-data environments. Whisper the controversial parts At one time, the company I worked at rolled out a bar raiser program styled after Amazon’s, where there was an interviewer from outside the team that had to approve every hire. I spent some time arguing against adding this additional step as I didn’t understand what we were solving for, and I was surprised at how disinterested management was about knowing if the new process actually improved outcomes. What I didn’t realize until much later was that most of the senior leadership distrusted one of their peers, and had rolled out the bar raiser program solely to create a mechanism to control that manager’s hiring bar when the CTO was disinterested holding that leader accountable. (I also learned that these leaders didn’t care much about implementing this policy, resulting in bar raiser rejections being frequently ignored, but that’s a discussion for the Operations for strategy chapter.) This is a good example of a strategy that does make sense with the full diagnosis, but makes little sense without it, and where stating part of the diagnosis out loud is nearly impossible. Even senior leaders are not generally allowed to write a document that says, “The Director of Product Engineering is a bad hiring manager.” When you’re writing a strategy, you’ll often find yourself trying to choose between two awkward options: Say something awkward or uncomfortable about your company or someone working within it Omit a critical piece of your diagnosis that’s necessary to understand the wider thinking Whenever you encounter this sort of debate, my advice is to find a way to include the diagnosis, but to reframe it into a palatable statement that avoids casting blame too narrowly. I think it’s helpful to discuss a few concrete examples of this, starting with the strategy for navigating private equity, whose diagnosis includes: Based on general practice, it seems likely that our new Private Equity ownership will expect us to reduce R&D headcount costs through a reduction. However, we don’t have any concrete details to make a structured decision on this, and our approach would vary significantly depending on the size of the reduction. There are many things the authors of this strategy likely feel about their state of reality. First, they are probably upset about the fact that their new private equity ownership is likely to eliminate colleagues. Second, they are likely upset that there is no clear plan around what they need to do, so they are stuck preparing for a wide range of potential outcomes. However they feel, they don’t say any of that, they stick to precise, factual statements. For a second example, we can look to the Uber service migration strategy: Within infrastructure engineering, there is a team of four engineers responsible for service provisioning today. While our organization is growing at a similar rate as product engineering, none of that additional headcount is being allocated directly to the team working on service provisioning. We do not anticipate this changing. The team didn’t agree that their headcount should not be growing, but it was the reality they were operating in. They acknowledged their reality as a factual statement, without any additional commentary about that statement. In both of these examples, they found a professional, non-judgmental way to acknowledge the circumstances they were solving. The authors would have preferred that the leaders behind those decisions take explicit accountability for them, but it would have undermined the strategy work had they attempted to do it within their strategy writeup. Excluding critical parts of your diagnosis makes your strategies particularly hard to evaluate, copy or recreate. Find a way to say things politely to make the strategy effective. As always, strategies are much more about realities than ideals. Reframe blockers as part of diagnosis When I work on strategy with early-career leaders, an idea that comes up a lot is that an identified problem means that strategy is not possible. For example, they might argue that doing strategy work is impossible at their current company because the executive team changes their mind too often. That core insight is almost certainly true, but it’s much more powerful to reframe that as a diagnosis: if we don’t find a way to show concrete progress quickly, and use that to excite the executive team, our strategy is likely to fail. This transforms the thing preventing your strategy into a condition your strategy needs to address. Whenever you run into a reason why your strategy seems unlikely to work, or why strategy overall seems difficult, you’ve found an important piece of your diagnosis to include. There are never reasons why strategy simply cannot succeed, only diagnoses you’ve failed to recognize. For example, we knew in our work on Uber’s service provisioning strategy that we weren’t getting more headcount for the team, the product engineering team was going to continue growing rapidly, and that engineering leadership was unwilling to constrain how product engineering worked. Rather than preventing us from implementing a strategy, those components clarified what sort of approach could actually succeed. The role of self-awareness Every problem of today is partially rooted in the decisions of yesterday. If you’ve been with your organization for any duration at all, this means that you are directly or indirectly responsible for a portion of the problems that your diagnosis ought to recognize. This means that recognizing the impact of your prior actions in your diagnosis is a powerful demonstration of self-awareness. It also suggests that your next strategy’s success is rooted in your self-awareness about your prior choices. Don’t be afraid to recognize the failures in your past work. While changing your mind without new data is a sign of chaotic leadership, changing your mind with new data is a sign of thoughtful leadership. Summary Because diagnosis is the foundation of effective strategy, I’ve always found it the most intimidating phase of strategy work. While I think that’s a somewhat unavoidable reality, my hope is that this chapter has somewhat prepared you for that challenge. The four most important things to remember are simply: form your diagnosis before deciding how to solve it, try especially hard to capture perspectives you initially disagree with, supplement intuition with data where you can, and accept that sometimes you’re missing the data you need to fully understand. The last piece in particular, is why many good strategies never get shared, and the topic we’ll address in the next chapter on strategy refinement.
A Live, Interactive Course for Systems Engineers
I’m sitting in a small coffee shop in Brooklyn. I have a warm drink, and it’s just started to snow outside. I’m visiting New York to see Operation Mincemeat on Broadway – I was at the dress rehearsal yesterday, and I’ll be at the opening preview tonight. I’ve seen this show more times than I care to count, and I hope US theater-goers love it as much as Brits. The people who make the show will tell you that it’s about a bunch of misfits who thought they could do something ridiculous, who had the audacity to believe in something unlikely. That’s certainly one way to see it. The musical tells the true story of a group of British spies who tried to fool Hitler with a dead body, fake papers, and an outrageous plan that could easily have failed. Decades later, the show’s creators would mirror that same spirit of unlikely ambition. Four friends, armed with their creativity, determination, and a wardrobe full of hats, created a new musical in a small London theatre. And after a series of transfers, they’re about to open the show under the bright lights of Broadway. But when I watch the show, I see a story about friendship. It’s about how we need our friends to help us, to inspire us, to push us to be the best versions of ourselves. I see the swaggering leader who needs a team to help him truly achieve. The nervous scientist who stands up for himself with the support of his friends. The enthusiastic secretary who learns wisdom and resilience from her elder. And so, I suppose, it’s fitting that I’m not in New York on my own. I’m here with friends – dozens of wonderful people who I met through this ridiculous show. At first, I was just an audience member. I sat in my seat, I watched the show, and I laughed and cried with equal measure. After the show, I waited at stage door to thank the cast. Then I came to see the show a second time. And a third. And a fourth. After a few trips, I started to see familiar faces waiting with me at stage door. So before the cast came out, we started chatting. Those conversations became a Twitter community, then a Discord, then a WhatsApp. We swapped fan art, merch, and stories of our favourite moments. We went to other shows together, and we hung out outside the theatre. I spent New Year’s Eve with a few of these friends, sitting on somebody’s floor and laughing about a bowl of limes like it was the funniest thing in the world. And now we’re together in New York. Meeting this kind, funny, and creative group of people might seem as unlikely as the premise of Mincemeat itself. But I believed it was possible, and here we are. I feel so lucky to have met these people, to take this ridiculous trip, to share these precious days with them. I know what a privilege this is – the time, the money, the ability to say let’s do this and make it happen. How many people can gather a dozen friends for even a single evening, let alone a trip halfway round the world? You might think it’s silly to travel this far for a theatre show, especially one we’ve seen plenty of times in London. Some people would never see the same show twice, and most of us are comfortably into double or triple-figures. Whenever somebody asks why, I don’t have a good answer. Because it’s fun? Because it’s moving? Because I enjoy it? I feel the need to justify it, as if there’s some logical reason that will make all of this okay. But maybe I don’t have to. Maybe joy doesn’t need justification. A theatre show doesn’t happen without people who care. Neither does a friendship. So much of our culture tells us that it’s not cool to care. It’s better to be detached, dismissive, disinterested. Enthusiasm is cringe. Sincerity is weakness. I’ve certainly felt that pressure – the urge to play it cool, to pretend I’m above it all. To act as if I only enjoy something a “normal” amount. Well, fuck that. I don’t know where the drive to be detached comes from. Maybe it’s to protect ourselves, a way to guard against disappointment. Maybe it’s to seem sophisticated, as if having passions makes us childish or less mature. Or perhaps it’s about control – if we stay detached, we never have to depend on others, we never have to trust in something bigger than ourselves. Being detached means you can’t get hurt – but you’ll also miss out on so much joy. I’m a big fan of being a big fan of things. So many of the best things in my life have come from caring, from letting myself be involved, from finding people who are a big fan of the same things as me. If I pretended not to care, I wouldn’t have any of that. Caring – deeply, foolishly, vulnerably – is how I connect with people. My friends and I care about this show, we care about each other, and we care about our joy. That care and love for each other is what brought us together, and without it we wouldn’t be here in this city. I know this is a once-in-a-lifetime trip. So many stars had to align – for us to meet, for the show we love to be successful, for us to be able to travel together. But if we didn’t care, none of those stars would have aligned. I know so many other friends who would have loved to be here but can’t be, for all kinds of reasons. Their absence isn’t for lack of caring, and they want the show to do well whether or not they’re here. I know they care, and that’s the important thing. To butcher Tennyson: I think it’s better to care about something you cannot affect, than to care about nothing at all. In a world that’s full of cynicism and spite and hatred, I feel that now more than ever. I’d recommend you go to the show if you haven’t already, but that’s not really the point of this post. Maybe you’ve already seen Operation Mincemeat, and it wasn’t for you. Maybe you’re not a theatre kid. Maybe you aren’t into musicals, or history, or war stories. That’s okay. I don’t mind if you care about different things to me. (Imagine how boring the world would be if we all cared about the same things!) But I want you to care about something. I want you to find it, find people who care about it too, and hold on to them. Because right now, in this city, with these people, at this show? I’m so glad I did. And I hope you find that sort of happiness too. Some of the people who made this trip special. Photo by Chloe, and taken from her Twitter. Timing note: I wrote this on February 15th, but I delayed posting it because I didn’t want to highlight the fact I was away from home. [If the formatting of this post looks odd in your feed reader, visit the original article]
One of the biggest mistakes that new startup founders make is trying to get away from the customer-facing roles too early. Whether it's customer support or it's sales, it's an incredible advantage to have the founders doing that work directly, and for much longer than they find comfortable. The absolute worst thing you can do is hire a sales person or a customer service agent too early. You'll miss all the golden nuggets that customers throw at you for free when they're rejecting your pitch or complaining about the product. Seeing these reasons paraphrased or summarized destroy all the nutrients in their insights. You want that whole-grain feedback straight from the customers' mouth! When we launched Basecamp in 2004, Jason was doing all the customer service himself. And he kept doing it like that for three years!! By the time we hired our first customer service agent, Jason was doing 150 emails/day. The business was doing millions of dollars in ARR. And Basecamp got infinitely, better both as a market proposition and as a product, because Jason could funnel all that feedback into decisions and positioning. For a long time after that, we did "Everyone on Support". Frequently rotating programmers, designers, and founders through a day of answering emails directly to customers. The dividends of doing this were almost as high as having Jason run it all in the early years. We fixed an incredible number of minor niggles and annoying bugs because programmers found it easier to solve the problem than to apologize for why it was there. It's not easy doing this! Customers often offer their valuable insights wrapped in rude language, unreasonable demands, and bad suggestions. That's why many founders quit the business of dealing with them at the first opportunity. That's why few companies ever do "Everyone On Support". That's why there's such eagerness to reduce support to an AI-only interaction. But quitting dealing with customers early, not just in support but also in sales, is an incredible handicap for any startup. You don't have to do everything that every customer demands of you, but you should certainly listen to them. And you can't listen well if the sound is being muffled by early layers of indirection.