More from Sometimes It Works :: simonhamp.me
So what's going on? This is a reflection on this post (of the same title) that I thoroughly enjoyed, by Flavio Copes. "I am the genuine article, therefore I don't have to try. I just have to be. You, on the other hand, have to try any passing bandwagon, because what else have you got?" Life Isn't All Ha Ha Hee Hee It seems that a great many people are feeling the pinch of a tech job market squeeze that seems to have lasted for over 2 years now. A lot of folks seem to be quick to dump the blame for this squarely on "AI". And what can you expect when the tech CEOs are all doing their stupid dance, touting the latest fadtech as the breakthrough that justifies them laying off swathes of employees in short-sighted attempts to prop up shareholder value? A more self-destructive spiral I have not seen. But the truth is masked by this conclusion. I don't believe for one second—and I don't think a lot of other popcorn-crunching devs believe it either—that "AI" is remotely close to the claims being made by these flaccid excuses for billionaires. Something else is afoot. "First, there’s a lot of people, and I mean a LOT, that went into programming and didn’t have a genuine passion for the job. For them, it’s just a way to make money." Hard agree! I've seen some of these people. I don't believe it's a bad reason to get into any line of work, but in my experience, they have made for some of the worst developers. Not because of a lack of technical capability (I truly believe almost anyone can learn to program well), but the money motivator only takes you so far. Like any job worth doing, it is hard. The rewards come to those who continue to put in the effort. If you've based your decision to become a programmer on seeing someone else claiming success on social media, I can only tell you that you've been duped. The vast majority do not have this story. And even some of the ones that appear to, don't. "Very few jobs out there give those kind of perks: high pay and comfortable life." It's all relative. My personal experience with programming hasn't been "high pay... comfortable life". I've worked damn hard for a very long time. I still don't feel sufficiently compensated and I've just had my best year ever. And that as a freelancer! I've never seen programming (or really any possible career path) as "easy". Currently "AI" makes my life a bit easier, but nothing about using it in and out every day to build things convinces me that it's going to do away with my job. "To them, those people that care about the craft, AI is not a problem at all." I totally agree. ¶So what's going on? "post-COVID companies hired like crazy" "the end of the zero-interest rate period" "increased competition" from the potential of a globally shifted workforce "investor pressure" "AI" Look, there's not just one answer. That's clear. And these are just part of the picture. I have another theory: a huge factor at play here is that, for most of these businesses, their revenue is based on ads. Therefore, consumers (for the most part) are the driving force of a lot of the top tech companies' bottom lines. But the consumers are getting sick and tired of: Privacy violations. Crypto scams. Income disparity between the workers and the C-suite. Ads! Ads! Ads! everywhere, all of the time. The lies and manipulation of it all. Many of these tech companies haven't innovated anything meaningful in a long time. They're jumping on AI right now because they're all hoping they'll discover the next big breakthrough and unlock megabucks. And in the meantime they prop themselves up by touting eye-watering sales figures (and net losses) of their next "always-on, internet connected, touchscreen, AI-powered, super agent whosawidget." Or their "giant, face-hugging, reality-distortion spectacles that no person in their right mind would be caught dead wearing in public" But are any of these "advances" really moving the needle on solving the bigger problems we're facing as a species? No. So the ads and PR BS isn't working as well as it used. If your business model is to be a big promotional platform that relies on real human beings to view the content that your advertisers want to promote, then you have to bring real value. And if you're the advertiser whose whole business model depends on crazy ad spend to penetrate into and reach your target market of what you hope are real people, then you need a lot of cheap money to get there. It's a poisoned lake. These big fish have done it to themselves. Sure, they're big, so they'll survive for a while, but it's still terminal. The good news is that, in their wake will come true innovators. I expect the next few years to be something of the beginnings of a rebirth period in this constant boom-bust circus. And I genuinely believe that "AI" will enable a lot of that. It is already enabling small teams to move faster whilst staying lean. In time, these little fish will grow and once again find opportunity in hiring fresh talent. Get in with those companies now while they're still small by seeking them out and starting a conversation. It might not be a job right away, but it's building the relationship. "You have to have a network... Local meetups, and local conferences... Go a few days before... hang out before the conference... Go to the after parties. That’s where you actually meet and get to know people." This is the way. You know, I've never once applied to FAANG. I've never felt worthy. In most cases, I don't meet the requirements as I don't have a degree. And honestly, the companies and their leaders do not align with my values at all, so I'm extremely disinclined to apply. But imagine if someone like me, a "self-taught" nobody dev, with no recognisable education, only having a connection or two, skips the queue ahead of you and lands a job at the next great innovative tech company. All for the sake of saying hello and getting to know people. This is why, especially in the age of "AI", cultivating genuine relationships with real people is so important. Work on silly side projects (if you have the capacity). Read! Share what you're working on. Share what you enjoy. Retweet. Reply. Write about it—in the abstract if you have to. Make videos. Do a podcast. Go meet new people, in and out of your field. Don't do it for the algorithm, or the likes and subscribes. Do it for the genuine relationships. Because in a world of fakes and forgeries, the genuine article sticks out like a sore thumb!
Avoid the shiny Avoid comparison Some examples Laravel Build tools Typescript and JS frameworks The Web Platform What I keep up with This morning, I watched a video advertising a course aimed at developers. One of the first sentiments proclaimed is that of feeling left behind, that tech is moving too fast to keep up. I think we're probably one of only a few industries globally that have this problem. I don't see baristas learning how to use every kind of coffee machine, or carpenters buying and using all the different types of wood saw. I don't see how they could. I'm sure those industries don't move anywhere near as fast as tech, so it may be a bit of an unfair comparison, but that has led me to a really interesting axiom: After 20 years in tech, I've learned that slow tech is good tech. I'm happier when I'm not trying to be on the bleeding edge. ¶Avoid the shiny I realised over time that I have been quite a lot more dismissive of "the new shiny" than a lot of other devs. In the back of my mind I was always kinda worried that I was being left behind because I wasn't learning X or getting experience with Y. But I've learned over the years that it truly doesn't matter. Why? Most importantly: users won't notice. Second: many companies don't have the capacity or desire to be on the bleeding edge. Finally: my brain still works, I can learn new things any time, when I really need to. And I learn faster now anyway. ¶Avoid comparison It's led me to believe that "falling behind" is a made-up concept designed to sell you stuff you probably don't need. 🌶️ Or maybe it's just human nature, similar to "keeping up with the Jones's". We're competitive, we compare ourselves to others very quickly. It's a bad habit though. I'm here to tell you that it's not just ok to fall (a little bit) behind, it can even be A Good Thing™. I've been falling behind for my entire career. 😂 I'm probably more behind now than I've ever been. But I'm also doing better now - in many ways - than I ever have. ¶Some examples ¶Laravel I didn't use Laravel until a couple years after its first release. When I picked up v4.2 it was using Composer and had started the transition to Symfony components. It had DI and all sorts of other goodies that earlier versions lacked. ¶Build tools I never used Bower/Gulp/Grunt/Yarn; I settled on NPM after the war was over. I haven't switched to Bun (and probably won't). I barely touched Webpack thanks to Laravel Mix. I use Vite, but also hardly use it directly, thanks to Laravel's first-party Vite plugin. I waited so long on all of this stuff that when I actually needed it, the choices were easy. Sure, I wasn't there at the front lines; I didn't invent React. I didn't build Vite. I didn't write the Laravel plugin. But then, I didn't need to, I wasn't building the frontend to the biggest online social network ever or a toolchain for other developers. ¶Typescript and JS frameworks I looked at Typescript once. You can just write plain JavaScript and get the same results. I slept on Coffeescript, Backbone, Angular, React, Vue, Svelte etc. Though I've used some of them briefly during my time, I never went deep on any of them. I stuck with jQuery for a long time because it worked and I knew how to build things rapidly with it. Importantly, I know the value of these other tools and when to use them. That hasn't stopped me being able to work on teams that use them heavily. But I've learned that it probably won't need to be me that's the person who's working day-in, day-out with them. And I'm ok with that. I use Alpine and Livewire now, as they were built to work with the tools I know and love using, each with a small footprint and easy to learn. They're more than enough for my needs—and many of my clients! And you know what? Some of my clients still use jQuery. 🤷♂️ ¶The Web Platform I love seeing the latest features in HTML, CSS & JS. I'll play with them, but very rarely deploy them. It usually takes some time before new features are available on ALL browsers and devices. And even then, billions of potential users are still running older versions. Yes, it still tickles my curiosity learning new things. It's intellectually satisfying. And, yes, proficiency in multiple tools can make you a more valuable asset. 💰 I'm not saying "you should not...", I'm just saying "you don't need to". ¶What I keep up with The only things that have been really important for me to stay up to date on are the core technologies I use: PHP & Laravel. I make it a point to keep my apps up to date with close to the latest releases - mainly for security and performance reasons, but also so I can use the latest and greatest features. 😛 But I rarely upgrade apps in production to the very latest versions as soon as they're available... I always leave it a few weeks. In that way, my work is not dictated by someone else's release schedule! That's not to say I don't keep abreast of what's coming in future releases of those tools; they're core to the service I provide and the tools I build, so I would be remiss not to. Being aware of what's coming is the priority there. But I don't have to have hands-on experience with everything. Testing early ("beta") releases against existing code is a useful exercise from time to time, but not always required. So basically I only need to regularly monitor two technologies. Easily done with a reasonably well-curated Twitter or a couple of RSS feeds. Sure, I keep my finger on the pulse of all the other tech I use, glancing occasionally out the corner of my eye and paying attention when the sources I am more focused on mention them But by keeping it simple, I can focus on delivering the most value rather than spinning my wheels on all of the superfluous things adjacent to the. Everything else can wait. Don't try to learn and keep up to date with everything; pick your battles!
Some history, some context When does an SPA make sense? So, why are SPAs bad again? Your front-end and back-end get decoupled! It may hurt customisability Performance will suffer So what's the alternative? I came across this interesting article by @gregnavis the other day. I guess it's from a few years ago now, judging by some of the other posts on his blog. But it still holds up. It's maybe even more relevant now. The article is entitled The Architecture No One Needs and it makes a simple and clear case, arguing that SPA's are more expensive than a standard multi-page app (which may or may not be a monolith). I'm going to use SPA throughout this post to mean the whole umbrella of ways you might be building a front-end application that is not server-side rendered. I think this is in line with Greg's intent too. I don't want to split hairs over whether a particular framework can be used to build front-end apps that aren't strictly SPAs. Since I read it, I haven't stopped thinking about this article. I found myself agreeing with all of Greg's stated points and it made me realise I actually have a strong opinion about this topic. I believe Greg is right, and as time rolls on I think I'm becoming more bullish in my stance on SPAs too. ¶Some history, some context My foray into the dirty, hubbub streets of front-end frameworks came about because of Laravel Nova and Statamic. They both use Vue, so I learned Vue. Of course, I looked around. But React made me retch. Angular almost made me want to buy a katana just to perform seppuku. (Of course I'm being hyperbolic.) I hear things about all of these and more thanks to Twitter. I can say some of it is good, but most of it continues to push me away. I stuck with Vue. I actually like Vue a lot, even if v3 has caused a bit of headache—it's actually better for it in my opinion and yes I do like the composition API. Overall though, I'm definitely heading more towards preferring not to have to build my front-end using node/deno/bun or whatever tool becomes popular today. That said, you just try and pry Tailwind from my frigid, rigid fingers! I'm quite firmly in the Livewire camp now. In another life, I may even like HTMX? I've built a few SPAs, but I can probably count them on one hand. But not only will I probably not build another one, I think you shouldn't either. I'm strongly encouraging my clients not to. I do think SPAs have a place, but it is almost certainly not in your project. Yes, I know this argument has been made before and probably far more eloquently than I'm going to, but you know I just felt like I needed to get this out of my head in my own way. ¶When does an SPA make sense? Let's get this out of the way first before I dunk on SPAs some more. They do have a place: I believe that the main advantage of an SPA is/was/has always been the decoupling of the front-end from the back-end. As time goes on and engineers specialise in areas that interest them the most, roles become more well-defined. This is why we have 'front-end' and 'back-end' and 'full stack' engineers. Although there is principally a lot of overlap (this is all programming at the end of the day and many concepts are similar), the domain—the environment that the engineer is most familiar with—is what determines their preference. Some engineers will prefer back-end because they don't want to think about or deal with a certain class of bugs or issues that arise from the rapidly-evolving world of front-end development. They may be uncomfortable using 3 or 4 different languages at the same time to get their work done, or they groan as soon as there's another major version of some framework which is going to require a load of refactoring that doesn't add immediate value to your product. Consider: every user of your app could be using a different version of a particular browser, which uses one kind of rendering engine, and a specific level of conformity to various web standards e.g. ECMAScript (the official standard that underpins what most of us think of as JavaScript) or CSS. Keeping on top of these variations and differences across desktop and mobile is enough to make anyone's head spin. And other engineers will prefer the front-end for its stateful nature, or because they're more comfortable with JavaScript/TypeScript, they grok CSS, and they love the intersection of code and design. Or maybe they just dislike dealing with databases, concurrency, queues, messaging and APIs a lot more than they dislike wild browser evolution. In any case, whatever size your team is, there will be these preferences. For example, I consider myself a full-stack developer, but honestly I like to avoid front-end work as much as possible, so will generally take the easy route there. Developing an SPA could allow you/your business to split the work of front-end and back-end into separate teams, which may help each team focus so they can do what they do best and be the happiest they can be. Which is the most important metric that will make an appreciable difference to your bottom-line long term. Being intentional about this will see you hit Conway's Law head-on and potentially tackle that beast in the best way, as long as you build up the necessary lines of communication. It also allows you to scale the two parts of the system separately—both in terms of team scaling and resource scaling—which, if you ever need that flexibility, could end up saving you from a certain group of headaches. And I'm gonna be honest, building distributed systems like this is a cool problem to solve and will be a point of growth for many of your engineers. ¶So, why are SPAs bad again? Yeh, so far this all sounds great, right? Well, just letting ol' Conway right into your living room isn't exactly the best idea. Aside of all the points that Greg makes in his article (if you haven't read it yet, go read it), I want to present three extra reasons why I think an SPA is a bad idea. ¶Your front-end and back-end get decoupled! Your back-end and front-end are always coupled. So trying to split them in anything but the most extreme circumstances is an exercise in futility. I think this is probably the worst part of this whole story. If your back-end team want to move in one direction, they've got to align with the front-end team. If timings and priorities don't work out, it's going to force someone to either put a hold on some work that really needs sorting out or do some grunt work just to patch over a hole that's about to appear. This is communication overhead. It adds risk. It adds complexity. It adds meetings into engineers' calendars. It adds friction, and stress, and distraction. It flies in the face of that number one principle: let your teams focus and be happy. This literally costs you money one way or another, cost that you could avoid. Deployments get unavoidably riskier in ways that are super difficult to test because testing distributed systems is really hard. Again, this might all be fine, in the most extreme cases, where you need the decoupling. Then this extra expense, and complexity, and churn-causing evil, is just a necessary evil that you have to learn to swallow and live with. But I've got x engineers, y1 are front-end, y2 are back-end. What do I do? I would strongly recommend that one of your engineering groups need to roll up their sleeves and get on learning the other group's code, tooling and responsibilities. This will have multiple benefits: career progression and learning opportunities, increased bus factor, fewer meetings and more collaboration. Sure there will be challenges too, but they won't be as big or as painful as the other challenges you'd face with an SPA. ¶It may hurt customisability As I mentioned earlier, the reason I got into Vue at all was because other tech I was using required me to. In both Statamic and Laravel Nova, the choice of Vue—well, not specifically Vue, but rather a reactive front-end framework—made sense because at the time it really was the best way to build flexible, reactive front-ends. And both of those tools needed that power and have become fantastic tools because of it. But there is one pain point that it's created that's quite hard to escape: the customisation story for each of these is harder because of it. How so? Basically, because each tool needs to build the assets to ship their product. And once they're built it's hard for third-parties to build on what's already there. How is it harder? Let's say I'm creating a Statamic add-on that allows CMS owners to post to social media from their control panel. As Statamic uses Vue and already has a bunch of components I can leverage, I am going to use some of them. But I'm also going to add some of my own functionality that doesn't already exist within Statamic. Now what happens? Well I build the Javascript... but wait. I can't change the bundled JS that's part of Statamic core. I have to build my own JS and load it at the right time, something I'm not in control of. Thankfully the Statamic team (building on tooling from the Vue & JS community) have worked hard to make this relatively easy, but my tool choice is now limited by what they support - if they're using Vue 2, I have to use Vue 2; if I don't like Vite or Webpack, tough luck. And on top of that, the builds have to happen at the client's end, which means we're now offloading some of the responsibility of making this whole thing work to people who don't need or want to know anything about this stuff. They just want to install your thing and get on with their jobs and lives. Why is this such a pain though? It used to be (in other platforms) that I could just load some extra JS file into the admin interface and do what I want. Honestly, we probably should never have been doing that either. Hands up, how many of you have seen a WordPress installation that tries to load 2 or more different versions of jQuery? 🙋🏼♂️ So these build tools go some way to alleviating some problems, but in the process have introduced so many layers of protection and abstraction that it presents a brain-melting, Japanese puzzle box to unlock. And all this JavaScript flying around is really unsafe, because JavaScript is completely malleable on the client side. That's meant library creators have had to go to some unusual lengths to protect the state of the application and encapsulate the code in attempts to make it safer and more portable. I won't pretend to understand all of the requirements, pre-requisites and implications for why built JavaScript assets are packaged up in a complex soup of function calls and obfuscated code, but suffice to say this makes building on top of pre-existing code that much harder. The web standards track is working to make this easier: we have Web Components and module starting to come through which should alleviate some of this. But if you're not building with those standard in play—I understand, it might not be possible because of browser support etc—and you want to expose your user's to third-party plugins/add-ons, then you've got to figure out how to make it easy for other developers. Some of that's going to require the specific implementation, the other part is going to be documentation. No matter how you cut it, it's going to be harder to get right than if you had server-side rendered views that you allow your third-party developers to load at runtime. ¶Performance will suffer This isn't really an extra reason as Greg did touch on this a little already, but I wanted to go harder on performance. You should never choose to build an SPA because of some supposed performance benefit. That is the wrong hill to try and defend for many reasons, but primarily because you've got the whole of the web stack—on horses, with bazookas—nipping at your heels. Sure, it may take a little while for web standards to get ratified and then rolled out, but the reality is that it's only a matter of time. We now have wide browser support for QUIC / HTTP/3 (which brings faster downloads and reduced server load) and things like 103 Early Hints response headers (which let us tell browsers what they should prioritise pre-loading), making the standard, non-SPA web even more performant. (And yes, some of this benefits SPAs too!) Sure, you can argue some of this advancement may have been driven by SPAs and their apparent benefits. But there's some inevitability to all of this (both the appearance of SPAs and the advancement of HTTP) which makes the whole argument moot in my opinion. As adoption and overall performance of the web platform increases, SPAs will even start to feel slow in comparison. Some feel slow already! That's because so much of the heavy lifting is left to the userland threads of the in-browser JavaScript engine instead of the lower-level compiled languages, the ones used to build the core browser engine itself (C++, Rust, Swift etc). That translates to a poorer experience in your app, and a penalty for your users. While it's not impossible for JavaScript to be as fast or faster than the actual browser it's running in, it's just such a long way for it to get there it's a no-brainer at this stage to let the browser do what the browser does best instead of trying to replicate all of that in JS. So don't! Use JavaScript the way it was intended: as a sugar-coating to enhance the pages, not to build entire pages. I mean, you wouldn't eat a donut made entirely out of sugar, would you? Again, in the extreme cases, maybe you would. Maybe for this donut-sized/-shaped sugar torus, you have an appropriately-sized dough-only counterpart, both of which you consume in close proximity... I donut-know where this analogy is going. ¶So what's the alternative? You, dear reader, are not in the most extreme case. Probably not even close! And you may never be. So, if you haven't started building an SPA, don't! Keep your code together in a single application where the front-end is rendered by the back-end, and then you can test and deploy it as a single unit. And yes, you can even do that without Docker 😱 Create it as a fully server-rendered, multi-page application with page reloads and everything. Go on! I double dare you. If you really want/need the reactivity, try something like Livewire, Hotwire or HTMX. You can do this all the way up to many millions of users per day and it will be fine, which you are a long way from. Trust me, your front-end will never meaningfully need to move faster than your back-end, and vice-versa. If you're already running an SPA and are contemplating bringing the two parts of the donut back together, do it! Bite your lip, close your eyes, and just do it. Relevant
Job hunting Freelancing Side Projects NativePHP Spred Veliance & Laradir Streaming Personal Things This year has been quite a ride! 🎢 It's felt like the first real year since 2020 that things have been somewhat "normal". However, looking back, it's been anything but normal. ¶Job hunting Although I finished my job at Elvie back at the beginning of November 2022, I allowed myself a couple of months off before fully getting into recruitment mode. I did a lot of interviews and general job-hunting busy-work at the beginning of 2023. For all my efforts, I had only one job offer come out of it, despite getting to final stage interviews on quite a few exciting roles. Sadly that offer would have forced us to move back to the UK, which wasn't something I was ready to do. We will move back at some point, but I had hoped to make more of a 'success' of living in another country and not throw in the towel at the first sign of rocky terrain. However, after pouring so much time and energy into job-hunting and really feeling like I wasn't getting anywhere (and steadily coming to the end of our savings), it was time to look for other sources of income. This meant switching from the comfort of a regular salary to freelancing... ¶Freelancing Freelancing work picked up and was going well for most of the summer. I was able to go to Laracon US (although the journey there was an experience) and enjoy a few days of nerding out with a bunch of old and new friends in the Laravel community, which was awesome. I went straight from there to Sicily for a holiday with my wife and some friends. Although we had more travel troubles thanks to the ridiculous Mediterranean heatwave that occurred, and we were at one point legitimately concerned about our well-being due to wildfires, it was a welcome break. But despite paying well, freelancing isn't my dream. I've spent too long in my career doing this kind of work, and when I joined Elvie back in 2017 it was a dream-come-true to move away from endless projects and to focus on refining systems over the long-haul. I have always enjoyed a few aspects of working with clients: learning their business and their problems, solving new and interesting challenges, and meeting new people. But I find the lack of control or influence frustrating. Moving back to freelancing, I feel like I've let go of my dream and hastily picked up client work again... something I'd wanted to escape for so long. However, it has allowed me to work with some incredible teams. Most notably I've enjoyed working with Spruce, Manual and more recently Tighten. One good thing from freelancing is that it has afforded me some extra freedom, allowing me to explore some of my side projects. ¶Side Projects Aside of the hundreds of hours I spent interviewing, sucking at Rocket League and not really getting any better at guitar, my time out of work also gave me the headspace to think about some side projects that I'd wanted to explore for a while. ¶NativePHP One idea was to create completely distributable Laravel apps by bundling PHP within Tauri. After a bit of banging my head against the wall, I finally nailed it, thanks largely to the static-php-cli package, which seemed to have gone somewhat unnoticed and in need of a lot of love. But Jerry, the maintainer, was super helpful and back in March I was able to get a statically-compiled version of PHP up and running with enough of the needed extensions baked-in to run a Laravel app. Then I tweeted about it and things went a little crazy. Marcel Pociot from BeyondCode reached out to me and NativePHP was born. (Although I wasn't involved with its development, Laravel Herd also came out of this and I was using it for about a month before it was released at Laracon US) We launched NativePHP publicly at Laracon US—Marcel changed the topic of his talk specifically to focus on NativePHP, which was super cool. He repeated the talk at Laracon AU. The interest in NativePHP has blown me away honestly, but it's also been somewhat frustrating. While I'd love to focus all my attention on it, I can't because I have a family and we need to eat etc. This has really brought the stark reality of open source home for me, where hundreds and thousands of people want something, but only a tiny, tiny proportion of them are prepared to put their money towards it. I get that it's a bit of a Catch-22 situation: if it's not production-ready, it's hard to justify contributing towards it because you likely can't make any money from it yet. Unfortunately, that has meant it's had to take a back seat while I go and do paid work or build up other projects that can generate some income for me. ¶Spred Another idea I'd be knocking around for a while was a payment solution that did away with using debit/credit cards altogether. My previous boss and mentor got me onto the idea and I was keen to explore it. Spred came out of that and it's been a bit of a vehicle for me to explore building something in public. While the idea is solid, I haven't yet figured out why it doesn't have wide appeal. Fee-free payments seems like a huge deal, but I'm struggling to find businesses that are prepared to try it out. I've only had a handful of payments run through the service so far. ¶Veliance & Laradir I started Veliance because I was freelancing and wanted to build up a directory of other Laravel freelancers. Having gone from being part of a team of people that I thoroughly enjoyed working with to suddenly being 'on my own' was a little daunting. Having some like-minded individuals that I could tap into felt appropriate. It wasn't an app, it was just a Notion page hooked up with a Typeform or two. A lot of it was manual and I didn't expect it to explode or anything—we're talking about a niche within a niche here—so the workload was never going to be intense. While I was spreading the message about Veliance, someone pointed out Laradir. It looked like they'd had a similar idea and had already started to build out an app. I reached out to them to see if we could collab... 🦗 🏏 I watched the site over the months—the number of registered user accounts kept climbing steadily and I thought they must be onto something. They had an 'open metrics' page, so I was able to see the signups going up, but revenue wasn't really moving (maybe the cron job broke?) And then, out of the blue, they got back to me. We chatted and I learned that the team behind it had kind of moved onto other things and didn't have the budget to carry it further. The opportunity to acquire the site came up and so I made an offer. And they accepted! This is the first time I've acquired a website or app. I couldn't be happier with how this went and I'm excited to do it again. It all went super smoothly and now I'm running Laradir. And I'm super excited to keep on developing that platform to help even more Laravel developers find work. Where Veliance was focused solely on Laravel freelancers, Laradir is for all Laravel engineers. I only managed to get about 70 devs signed up to Veliance, but Laradir already had over 700 when I acquired it. Since then it's grown to almost 2,000 (1,988 as of today)! These aren't huge numbers, but it's a start. I think there are plenty more to come. If you haven't joined yet, please do! ¶Streaming Something I've been wanting to do for a while is live-stream programming. I wasn't able to do it at Elvie because it just wasn't ever going to be that kind of org. But now that I'm a free agent, I wanted to give it a meaningful shot. I'm not a showman. I just enjoy writing code. I'd love it if it helps someone, but honestly, I largely expect that someone to only be me. That said, I had a few goals: Create some accountability to help me focus and get things done. Do it regularly and meaningfully, even if no one is watching. Keep the overall commitment low. See if it's something I enjoy doing. I think it's passed on all counts: It has given me a reason to build different side projects in public, which is nerve-wracking and exciting. I've limited it to just Fridays for now. And I find that around 2 hours is enough time. But I'm trying to keep it a regular slot every week. I haven't gone out and bought any new gear to do it—I use my iPhone camera (Continuity Camera) and I bought a Røde lavalier. I haven't decked out my office or installed soundproofing or bought an expensive mic. And I am really enjoying it! It's honestly quite tiring, but it's worth it. I think every stream I've done has been better the previous one, both in my comfort level doing the stream but also in what I'm able to work on and the progress we make together. And somehow I've got 127 subscribers on my channel, which I honestly never believed would happen! Thanks to everyone who's been on the show, said hi and subscribed. I really appreciate your support 💜 ¶Personal Things I'm proud of many of the things I've done this year. But it's also been quite hard. That takes a toll. I've felt quite tired and frustrated by a lot of things—more than normal. I consider myself to be quite patient, but I feel like that's wearing thinner... I'm becoming a bit of a 'grumpy old man' and honestly I have to have a strong word with myself every now and then. I am extremely fortunate all things considered: I have a beautiful and caring wife, a sometimes-obedient pup, and a modest set of relative luxuries... a nice home, a car, eating out, the occasional holiday etc. My wife has worked insanely hard on her business too. It's been a really tough year for her. And even in the last couple of months, we've had some pretty big ups followed by some equally low downs. All this combined has had us talking more and more seriously about a move back to London. Although life after the pandemic (is it ever really over?) seems to be returning to something like what it was prior, there's still quite a deep unsettling feeling that I don't think I've had before. I feel like I used to have a pretty good idea of what the next year would bring, but I honestly have no clue for 2024. I do feel a bit more positive about the year ahead than I did at this time last year, but the levels of uncertainty are still quite high. The best I can do is stake a claim on trying to achieve a few goals: Drastically improve my Spanish! Grow Laradir so that it becomes self-sustaining. Launch Spred so that anyone can sign up and start using it. Continue streaming and enjoying it. Get NativePHP out of alpha and into production—especially the Tauri driver. A stretch goal might be that all of the above do enough to financially support us that I can scale back freelancing. For sure there will be some unexpected curveballs to throw some of this off track. Hopefully they're good surprises and not bad surprises 🤷🏻♂️
More in programming
A clip of me singing a funny song from Gilbert and Sullivan’s Ruddigore back in 2013
In this video, we look at why fork() needs copy-on-write, how it works inside the kernel, and a memory usage problem that Instagram encountered with Python.
Comments require commitment, but they’re worth it.
Basecamp 5 runs on Puma in cluster mode: one master process with preload_app! and 63 single-threaded workers per host, deployed as a Docker container with Kamal. We serve Basecamp from several sites. Each site has its own web hosts and a read replica of the database, and writes go to a single primary database in one of them. On our busiest hosts, each deploy left up to 2,000 requests waiting while the new workers warmed up. We reduced those queues by running signed-in requests through the app in the Puma master, before it forked the workers. Why 63 single-threaded workers? Basecamp has always served web requests from processes rather than threads. It ran on Unicorn, which only does processes, until we moved to Puma in January 2025, and we kept the same setup: workers (Concurrent.physical_processor_count * 1.3).ceil threads 1, 1 preload_app! On a 48-core host that’s 63 workers, each handling one request at a time. We chose 1.3 after benchmarking HEY in 2023, when we moved our apps out of the cloud and onto our own hardware. We tested several combinations of workers and threads with a mix of GET and POST requests on a 32-vCPU VM. Every multithreaded configuration we tested was slower and handled fewer requests than single-threaded workers. Adding workers beyond about 1.2 to 1.3 per vCPU brought little benefit. The threaded workers spent a lot of their time waiting for Ruby’s global VM lock. That made single-threaded workers a good fit for this workload, and we use the same setup for Basecamp. An app that spends more time waiting on its database or other services may benefit from more threads, so benchmark your own app. The other reason is the app itself. Basecamp has class-level state in places and has never needed to be thread-safe. With one request per process, it still doesn’t. Processes do use more memory than threads, and preload_app! reduces the difference. The master loads the app once and the workers share its memory through copy-on-write until they write to it. Shopify’s comparison of Ruby execution models explains the trade-off well. In the HEY benchmark the best setup came to about 260 MB of PSS per core, where PSS counts each shared page once, split between the processes using it, and the gap to a threaded setup was smaller than we’d expected. What Puma does on each host when a container starts: one master, then 63 forked workers that share its memory until they write to it. Two things about this setup matter for the rest of the post. A worker that’s compiling or loading something is fully blocked — there’s no other thread to pick up the next request. And whatever the master has in memory before it forks, all 63 workers share. Whatever they build after the fork, they build 63 times. What happens when we deploy Kamal starts the new container alongside the old one, and kamal-proxy moves the host’s traffic across as soon as the health check passes. At that moment, the new workers have handled health checks but no customer requests. preload_app! means the master loads the app once and the workers inherit it through fork. That covers the code. It doesn’t cover anything Ruby and Rails set up on first use: YJIT compiled code. YJIT compiles a method once it’s been called a certain number of times. The master calls very little during boot, so every worker compiles the same methods again on its own first requests. Compiled templates. Action View turns each ERB template into a Ruby method the first time it’s rendered. The schema cache. Active Record reads each model’s columns from the database the first time that model is used. Inline caches and memoized values throughout Ruby, Rails and the app. All 63 workers did all of this at once, while serving the traffic the old container had been handling a second earlier. In the test environment with YJIT on, the first request to a project page on a cold process took 652 ms, 151 ms of it YJIT compiling. The same request to a warm process took 28 ms. In production, CPU time per request peaked at around 200 ms while kamal-proxy moved traffic to the new container, against about 30 ms once the workers had warmed up. A host with spare CPU absorbs this. Every one of our web hosts has 48 cores and 63 workers, but each Amsterdam host serves around 250 requests per second, against 25 to 60 at our other sites. In Amsterdam the slow first requests turned into a queue. At a peak-hour deploy, the Puma backlog on an Amsterdam host reached anywhere from 250 to 2,238 requests, and kamal-proxy’s p99 response time hit about 10 seconds. Eron, our Director of Operations, had been tracking this since June. Another server in Amsterdam would help, but it would take weeks to arrive, so we also wanted to make deploys cheaper on the hardware we already had. What didn’t work We tried a few things first. In June, Donal tested the first two on a single Amsterdam host, comparing it with its neighbors, and they ruled out two likely causes. Warming each worker’s database connections. Puma’s before_fork hook clears the master’s connections, and each worker opened its own on its first request. Opening them in before_worker_boot instead made no difference. Queries on a freshly booted production host were already under a millisecond, so connections weren’t the problem. A synthetic request in each worker. Next, each worker made a few requests in before_worker_boot to an internal controller that touched every model. That ran the middleware, routing and Active Record paths, but it ran them in 63 workers at once — exactly the CPU spike we were trying to avoid. And a request with no real data renders no real views, so most of the app stayed cold. Spreading YJIT compilation out. Delaying YJIT in each worker by a random interval spread the compiling out over a few minutes, but every worker still ran interpreted until its delay ended. The queue didn’t change. Reforking from a warm worker. This is what Shopify’s Pitchfork does: let one worker serve traffic until it’s warm, then fork the others from it. Puma has an experimental version called fork_worker, and on beta it worked — the reforked workers were warm after three to five requests, where fresh ones took up to 30 seconds. But with fork_worker the template is worker 0, and it keeps serving requests. If it exits, the workers waiting to be forked never start (puma/puma#3596). If it gets no traffic, the refork never happens, which is what we saw on beta. Instacart have a mold_worker patch that promotes a warm worker to a template that stops serving, but it isn’t in a Puma release. We have a branch of it, and we may come back to it. That last experiment did show us where the fix was, though. Everything a warm worker has that a cold one lacks is in its memory, and fork copies memory. The master already has the app loaded. It just never runs it. Run the requests in the master So now, before the master binds its socket and forks, it makes the app’s own requests, in-process, the way a signed-in user would. Rack has a hook for exactly this. Rack::Builder#warmup takes a block that’s called once with the built app, before the server starts. rails server builds the app from config.ru, so the change to boot is one line: require_relative "config/environment" warmup { WarmUp.configured.run } if ENV["WARM_UP"] run Rails.application With preload_app! this runs in the master, and the workers inherit whatever it did. Puma binds its socket after the app is built, so until the warm-up finishes the health check’s connection is refused and kamal-proxy keeps retrying. No request reaches a worker that hasn’t been warmed. The warm-up has three steps. After precompiling the views, it gives the page requests and schema loading a shared 20-second budget, checked before each page or model. 1. Precompile the views actionview_precompiler reads every template for its render calls and compiles each one with the locals it’s passed. For us that’s 1,394 templates in about two seconds. A first request to a project page then compiles 2 templates instead of 44. 2. Request the pages, signed in A small browser class makes the requests through Rack::MockRequest, with the two cookies a real sign-in sets, then goes back for each page’s lazy Turbo frames: class WarmUp::Browser def initialize(signed_in_as:) @client = Rack::MockRequest.new(Rails.application) @headers = { "HTTP_USER_AGENT" => "Basecamp warm-up", "HTTP_COOKIE" => cookie_for(signed_in_as), "bc3.warm_up" => true } end def visit(path) page = get(path) frames_in(page).each { |id, src| get(src, "HTTP_TURBO_FRAME" => id) } end private def get(path, headers = {}) @client.get("https://#{host}#{path}", @headers.merge(headers)) end def frames_in(page) Nokogiri::HTML5(page.body).css("turbo-frame[src]").map { |frame| [ frame["id"], frame["src"] ] } end end The requests are signed in. The user is a monitoring account we already use for automated checks, and the pages are its own project, Campfire, to-dos, documents and messages. Public pages weren’t enough: after warming up with signed-out pages only, the first signed-in request to the projects page still took 131 ms, because authentication, the signed-in controllers and their views had never run. With signed-in pages it took 40 ms. cookie_for writes the same signed cookie the sign-in controller does, using the app’s own cookie jar, so there’s no API token and no secret to store. The frames are followed. The busiest HTML requests in production aren’t pages at all but Turbo frames — the sidebar badge, the inbox, the navigation menus. The browser parses each page and requests its <turbo-frame src> URLs with the Turbo-Frame header, so those controllers and views get warmed too. Our first four pages turned into 60 requests. The requests are excluded from rate limiting. They are internal, so they do not count against the rate limits that apply to real visitors. 3. Load the rest of the schema The page requests load the schema for the models they touch. The last step loads the rest, from the read replica: ApplicationRecord.reading do models.lazy.take_while { time_left? }.each { |model| model.load_schema if model.table_exists? } end The step checks 261 models and loads any schema information still missing. Those database round trips add up when the primary is far away: outside a request, Active Record uses the writing role, and from a host a long way from the primary each round trip is tens of milliseconds. Reading from the local replica brings the step down from about 20 seconds to 3.5. The pages go first because they load most of the schema anyway. If the time budget runs out, the step stops, logs how many models it got through, and the workers load the rest on first use like they always did. Rails can also load the schema from a dumped cache file at boot (bin/rails db:schema:cache:dump), which would make this step unnecessary. We don’t ship one in our image yet, because the dump needs a database to read from at build time, and we have several databases to cover. It’s on the list. What to close before the fork Running requests in the master opens things the master never opened before, and every worker inherits them. Two processes writing to the same socket will corrupt each other’s traffic, so you need to know what’s open before you fork. The way to find out is to list the master’s open file descriptors — ls -l /proc/<pid>/fd — before and after a warm-up, in an environment set up like production. Development wasn’t enough for us: it stores files on disk, so our S3 connections only showed up in production. Then, for each thing that’s open, check how its library handles a fork. We found three kinds: Already handled. Plenty of libraries detect a fork on their own, either by recording the PID they connected from and reconnecting in the child, by opening per-process files, or by resetting their thread pools. Redis clients, metrics libraries and concurrency libraries tend to be in this group. Check, but you probably don’t need to do anything. Already closed. Database connections are the classic one, and most Puma configs already clear them in before_fork. Anything else that’s opened per process — we have a SQLite cache the workers open on boot — needs closing when the warm-up finishes. Needs a new step. HTTP clients with keep-alive connections are the ones to look for: cloud SDKs with connection pools, tracing exporters, error reporters. They usually have no fork handling at all. We empty the aws-sdk connection pools in before_fork, and we run the warm-up untraced so the OpenTelemetry exporter never opens its connection to Tempo in the first place. Once that’s done, before_fork finishes with Process.warmup, which Ruby 3.3 added for this purpose: a major GC, a heap compaction, and every surviving object promoted to the old generation, so the memory pages the workers share change as little as possible afterwards. Choosing the pages The first list was the four pages that ran the busiest requests on beta. Once the warm-up was live, production showed us which endpoints were still cold. For one deploy, we compared each endpoint’s mean duration in the six minutes after kamal-proxy moved traffic to the new container with the same endpoint an hour later, then multiplied the difference by the number of requests in those six minutes. That gives the extra time each endpoint cost us because it was cold: Endpoint Cold Warm Requests in 6 min Extra seconds Campfire 246 ms 70 ms 6,490 1,140 Projects (JSON API) 84 ms 50 ms 22,077 771 Docs & Files 262 ms 177 ms 4,996 421 To-dos tool 205 ms 113 ms 4,018 371 To-dos (JSON API) 33 ms 16 ms 18,738 320 The pages already in the warm-up showed what to expect: the project page kept a 36 ms gap after a deploy, and the to-do page 10 ms. We’ve proposed adding these five requests, and expect them to add about five to seven seconds to the page step. The two JSON endpoints were a surprise. The warm-up’s page list had no API requests in it, so nothing on the API path had run before the first real request: not the API controllers, and not the Jbuilder templates rendering real records. Precompiling the views covers JSON templates too, but it isn’t a substitute for running the request. Results The warm-up is on for all 68 web hosts. With the first four pages it took 12 to 16 seconds per host: about 2 seconds to precompile the views, 7 to 9 for the 60 requests, and 3.5 for the schema. Deploys take that much longer per host, and we raised the deploy timeout from 30 to 60 seconds to cover it. In Amsterdam, at a peak-hour deploy: During deploy Before After Peak Puma backlog per host 250–2,238 requests 19–223 requests Peak kamal-proxy p99 about 10 s 2.4–4.8 s Peak CPU time per request 201–214 ms 88–132 ms Peak database time per request 56–69 ms 39–47 ms The same eight hosts at three deploys on 1 October, an hour apart, as the warm-up went from one host to four to all eight. The deploy in the middle, with four hosts warmed and four not, shows why every host needed the warm-up. Each warmed host recovered faster on its own: mean request duration peaked at 130 to 173 ms, against 203 to 311 ms on the hosts that weren’t warmed. But the backlog on all eight was about the same, because they were all waiting on the same database. Mean request duration on each host at the 07:21 UTC deploy. Blue hosts warmed up in the master before forking, orange hosts did not. Memory came down too. The workers now share compiled templates, YJIT code and the schema with the master instead of each building their own copy. On beta, the view precompiler alone took a busy worker’s private memory from 174–202 MB to 119–135 MB. Thirty minutes after the deploy, the web containers used about 39 GB less memory than the previous day’s containers at the same age and traffic. Amsterdam served most of our traffic at the times we tested. In Amsterdam, each new container used about 2 GB less just after traffic moved to it, which lowers the peak while the old and new containers overlap. Working with Claude Claude Code helped throughout. It combed through the per-worker backlogs and per-endpoint timings in Prometheus and Loki after each deploy, worked out the cold-versus-warm cost of each endpoint, and prepared the changes and the pull request descriptions with the benchmarks in them. We decided what to try, deployed it and read the results. If you do this Warm the master before it forks. Compile common code and templates and load their schema in the master, so workers inherit that work. With preload_app!, Rack::Builder#warmup runs before the workers start accepting traffic. Use the app’s real requests. Public pages, internal endpoints and synthetic queries warm the paths they run and nothing else. Signed-in requests to real records, frames included, run what production runs. Measure the cold penalty per endpoint. The difference between an endpoint’s cold and warm duration, times its request count after a deploy, ranks the pages worth adding. Ours weren’t the ones we’d have guessed, and two of them were JSON. Check what the warm-up leaves open. List the master’s file descriptors after a warm-up and account for every one before the fork. Two of ours needed changes. Set a time budget. A warm-up that runs long on one slow host fails the deploy on that host. Ours gives the page requests and schema loading a shared 20-second budget, checked before each page or model, puts the most valuable pages first, and logs what it skipped. Reforking from a warm worker, as Pitchfork does, solves the same problem continuously rather than once at boot, and it would warm paths no fixed list of pages covers. We may still get there: our branch brings Instacart’s mold_worker up to date with Puma’s main branch and fixes the bugs we found in it. But warming the master works with the Puma we already run, took a few days to implement, and substantially reduced the queues after deployment.