More from Joel Gascoigne's blog
The significance of Bluesky and decentralized social media I'm delighted to share that we have introduced support for Bluesky in Buffer. This is an important moment for us as a company, and there are a number of reasons that adding Bluesky is personally meaningful for me. With Bluesky, we now support the three major social networks pushing forward a new era of decentralized social media: Mastodon, Threads and Bluesky. We have been intentional about moving fast to add these channels to our tool. Supporting independence and ownership in social media Buffer has now existed for almost 14 years, and throughout that time I've seen a lot change in social media, and in our space of tools to support people and businesses with social. We're an outlier as a product and company that has existed for that kind of timeframe with leadership and values left in tact. We've had to work hard at times to maintain control over our destiny. In 2018, we made the decision to spend $3.3M to buy out the majority of our VC investors and be able to go our long-term path. We have continued to carry out buybacks each year since 2018, and at this stage we are majority founder and team owned. One of the things I'm proudest of is that we still wholeheartedly serve individuals and creators, and have not gone up-market as many other long-running companies in our space have done. We've been fortunate to be able to scale to 56,000 paying customers and over $18M in annual revenue while taking our own unique path. Through intentional choices over the years, we have maintained a level of optionality over our future that most do not have. This independence is something I don't take for granted. Keeping ownership of our company, and through that ownership having an ability to boldly go in the direction we believe is best for customers and the team, is very important to me. This is why, as a business, we feel so philosophically aligned with rising new decentralized social media networks, such as Bluesky and Mastodon. These networks have been started with a belief that individuals should maintain ownership over their content and the connection to their audience. They have data portability baked in from the beginning. When you use these networks, you are much more likely to be able to maintain control over your content and audience than if you use social networks owned by large corporations with complex ownership structures of their own, and often with public markets to answer to. The larger social networks provide a level of distribution that's worth tapping into, but I strongly encourage investing a portion of your energy into networks where you will be able to maintain ownership long-term. At Buffer, we will be doing everything we can to support the growth of new decentralized social media options, because we believe that individuals and small businesses should maintain control over their content and the connection to their audience. The resurgence of the open web with social media protocols I have been eagerly observing the emergence and growth of social media protocols, in particular with ActivityPub (and Mastodon as the prominent implementation), and AT Protocol from Bluesky. Open standards in social media could be as powerful as open standards have been for direct and private communication (email). What I find exciting about the development of these open standards, and more importantly the adoption of them and traction of social networks which support them, is that they can bring forth a new era of open standards for the web. The Internet was built upon open standards — HTTP, URL, TCP/IP, DNS, HTML. A vast many valuable internet businesses have built on these "shoulders of giants." ActivityPub and AT Protocol are built with open standards philosophies, and could similarly enable a new playground of innovation, with openness, ownership and interoperability at their core. I personally miss the earlier days of social media where the APIs had much greater parity with what could be done natively on the platforms. When I started Buffer, the Twitter and Facebook APIs were close to feature-complete, and brought about a lot of innovation in third-party development on top of those APIs. This is how Buffer was born, along with many other products in our space. Over time, we saw an era of closed APIs with reduced transparency and ownership of content and audiences. Mastodon and Bluesky bring the opportunity for a new era of innovation in our space, which I am welcoming with open arms. More innovation in the social media management space will be better for customers, and frankly makes for more exciting work to do. Bluesky is bringing innovation back to social media If you haven't had a chance to take a look at some of Bluesky's recent product and platform announcements, I highly recommend that you go and read them. In particular, what they've done with introducing custom feeds as well as starter packs gets me very excited about some real innovation from a social network. When I saw starter packs introduced, it immediately felt like a no-brainer feature for a social network, and such a powerful thing, especially for an emerging social network, to offer. Starter packs allow anyone to create a "getting started pack" for a new Bluesky user. This can include a set of recommended follows, and up to three recommended custom feeds (more on those below). This enables their passionate users to be able to personalize an introduction for people not yet on Bluesky. It's a smart way to activate users to play a meaningful role in onboarding new people to the network and grounding them with an existing community to interact with. Of course, Bluesky benefits by likely getting more people onto their new network than they would otherwise. Custom feeds are an incredible innovation that put the choice of algorithm for the social network in the hands of the wide range of users and different niche communities that exist on the network. The way that the Bluesky team have built custom feeds enables a ton of flexibility for the types of content alogrithms can serve up, and creates a marketplace for browsing and enabling different custom feeds you can choose to view. Something I've observed from the Bluesky team is their commitment to, and intentionality around, building tools for the governance of the network itself. It's very meaningful that on Bluesky you can choose your own algorithm and you can adopt an algorithm that someone else has written, or create your own algorithm for what content shows up in your feed. And I think it's very smart that Bluesky has done this — because it's both innovation and it's strong strategy because it's a highly defensible move which many of the other networks would not be able offer. It would be very unlikely for the commercial social networks to move away from the company, the network themselves, holding on to ownership of the algorithm and what is served up to you. I had a wonderful conversation with Rose Wang from the Bluesky team a couple of weeks ago and one of the topics we got into was around the values that are embedded in the Bluesky team and the work they're trying to do. It was clear to me how thoughtful and intentional they are being around the governance of the network and the flexibility they're building in to allow users to really shape the community and what is important to them. Something I appreciate about Bluesky is that their goal is to create a social network not controlled by a single company, while also ensuring that it comes together as a cohesive and easy-to-use experience. Decentralized social media can be daunting and feel complex and inaccessible to people initially, and so I think intentional work going into the simplicity of the experience is paramount. With great innovation from the Bluesky team such as starter packs and custom feeds, along with their focus on simplicity, I strongly encourage you to go and take a look at this new social network. This is a platform and community that's worth taking a deeper look at, participating in and investing time into. Join us in participating in a new era of decentralized social media By supporting Bluesky, along with Mastodon and Threads, we are playing our part in moving forward this promising new era of social media. Many of us in the team have been personally drawn to these networks for their special and supportive communities. We're here to see decentralized social media grow and become more meaningful for more people across the world. That's why we've put our scale, brand and resources into building awareness and providing tools to make participating on these new social networks more streamlined. I encourage you to add Bluesky to your channels in Buffer, and start participating in the social network today. Learn more and get started by visiting our Bluesky page. Photo by Kumiko SHIMIZU on Unsplash.
Build Week at Buffer: What it is and how we’re approaching it Note: this was originally posted on the Buffer blog. We’ve dedicated the week of August 22nd to a brand new internal initiative called Build Week. We’ll all be putting aside our regular work for a single week to come together in small groups and work on ideas that can benefit customers or us as a company, ideally with something of value shipped or in place by the end. The inspiration for Build Week Before building Buffer, I had several formative experiences attending “build a startup in a weekend”-type events. Two I attended were run by Launch48, and another was Startup Weekend. Anyone could sign up to attend no matter what skill set or experience level they would bring. As long as you were willing to roll up your sleeves, build something, and contribute in any way, you’d be very welcome. The focus was on building something rapidly from end to end, within the space of a weekend. Teams would be capped to a small number, around three to five people per team, so the groups could move quickly with decision making. Once the teams were formed, you’d get to work and start doing research, building, and marketing (often all in parallel) to move as fast as possible in building a minimum viable product and achieving a level of validation. At the end of the weekend, teams would present what they achieved, what they validated, and what they learned. Through these events, I met people, formed strong bonds, and stayed in contact for years with them afterward. Some teams even became startups. It felt like highly accelerated learning, and it was intense but fun, very energizing and inspiring. I’ve been thinking about how this could translate to Buffer and why it would be so powerful for us in our current season, which is where Build Week comes in. What is Build Week? Build Week is a week at Buffer where we’ll form teams, work with people we don’t typically work with, and work together on an idea we feel called towards. The highest level goals of Build Week are to inject into the company and team a spirit of shipping, creativity, and innovation, making progress and decisions rapidly, comfort with uncertainty, and ultimately going from idea to usable value out in the world in the space of a week. When it comes to the type of projects we’ll work on and the skill sets required to accomplish them, the goal is for those to be far-reaching. While it may seem like Build Week would be more suited to engineers specifically, our goal is to achieve the outcome that everyone realizes they are and can be a Builder. Ultimately, being a Builder in Buffer Build Week will mean that you are part of a team that successfully makes a change that brings value, and it happens in the short period of a week. Everyone on the team has something to bring to this goal, and I'm excited by the various projects that will be worked on. How we’re approaching Build Week With our high-level vision and ideas for Build Week, several months ago we got to work to bring this concept to life and make it happen. The first thing we did was form a team to plan and design Build Week itself. Staying true to our vision for Build Week itself, where we want to have small teams of people who don’t normally work together, this is also how we approached forming the Build Week Planning team. With this team in place, we started meeting weekly. Overall, it has been a small time commitment of 45 minutes per week to plan and design Build Week. As we got closer to the actual week, we started meeting for longer and having real working sessions. Our final design for Build Week consisted of three key stages: Idea Gathering, Team Formation and Build Week. For the Idea Gathering stage, we created a Trello board where anyone in the team could contribute an idea. We used voting and commenting on the cards, which helped narrow the ideas to those that would be worked on during Build Week. We gave people a few days to submit ideas and received 78 total contributions. This was a big win and a clear indication of a big appetite for Build Week within the company. The Team Formation stage was a trickier problem to solve and determine the process for. Initially, we had hoped that this could be entirely organic, with people gravitating towards an idea and joining up with people who are also excited to work on that idea. Ultimately, we realized that if we approached it this way, we would likely struggle with our goal of having people work with folks they don’t normally work with, and we wouldn’t have enough control over other aspects, such as the time zones within each team. All of this could jeopardize the success of Build Week itself. So we arrived at a hybrid, where we created a Google Form for people to submit their top 3 choices of ideas they’d like to work on. With that information, we determined the teams and made every effort to put people in a team they had put down as a choice. And the final stage is, of course, Build Week itself! The teams have now been formed, and we created a Slack channel for each team to start organizing themselves. We are providing some very lightweight guidance, and we will have a few required deliverables, but other than that, we are leaving it to each team to determine the best way to work together to create value during the week. If you're a Buffer customer, one small note that as we embrace this company-wide event and time together, we will be shifting our focus slightly away from the support inbox. We will still be responding to your questions and problems with Buffer; however, we may be slightly slower than usual. We also won't be publishing any new content on the blog. We’re confident that this time for the team to bond and build various projects of value will ultimately benefit all Buffer customers. Why right now is the time for Build Week at Buffer 2022 has been a different year for Buffer. We’re in a position of flatter to declining revenue, and we’ve been working hard to find our path back to healthy, sustainable growth. One key element of this effort has been actively embracing being a smaller company. We’re still a small company, and we serve small businesses. Unless we lean into this, we will lose many of our advantages. We want to drive more connection across the team in a time where we’ve felt it lacking for the past couple of years. While we’ve been remote for most of our 11+ years of existence, we’ve always found a ton of value from company retreats where we all meet in person, and we’ve suffered during the pandemic where we’ve not been able to have these events. Build Week is an opportunity for us to do that with a whole new concept and event rather than trying to do it with something like a virtual retreat which would likely never be able to live up to our previous retreat experiences. There’s a big opportunity for exchanging context and ideas of current Buffer challenges within teams where the teams are cross-functional and with people who don’t normally work together. This could help us for months afterward. Build Week can also be a time where strong bonds, both in work and personally, are formed. My dream would be that after Build Week, people within their teams hit each other up in Slack and jump on a spontaneous catch-up call once in a while because they’ve become close during the week. We’ve had engineering hack weeks for a long time now. Those have been awesome in their way, but they have been very contained to engineering. And while those events created a lot of value, they often lacked perspectives that would have enhanced the work, such as customer advocacy, design, culture, or operational perspectives. As a company, we want to challenge some of the processes we have built up over the past few years. Build Week is like a blank canvas – we clear out a whole week and then diligently decide what we need in terms of structure and process to make this concept thrive and no more. This can act as inspiration for us going forward, where we can use the week as an example of rethinking process and questioning the ways we do things. The opportunity that comes with Build Week If we are successful with Build Week, I am confident that we will surprise ourselves with just how much value is created by the whole company in that one week alone. In embracing being a small company, we’re currently striving to challenge ourselves by moving at a faster pace without over-working. I think this is possible, and the completely different nature of how we work together in Build Week could give us ideas for what we can adjust to work more effectively and productively together in our regular flow of work. The opportunity for value creation within Build Week goes far beyond product features or improvements. Build Week will be a time for us to build anything that serves either customers or the team in pursuit of our vision and mission, or strengthens and upholds our values. We can stretch ourselves in the possibilities – there could be a marketing campaign, a data report, improving an existing process in the company, rethinking our tools, creating a new element of transparency, bringing our customers together, etc. Wish us luck! I believe Build Week can be one of the most fun, high-energy weeks we’ve had in years. I expect we can come out of the week on a high that can fuel us with motivation and enjoyment of our work for months. That is a worthy goal and something I think we can achieve with a little creativity and the right group of people designing and planning the event. Of course, part of the beauty of Build Week itself is that just like all the ideas and the freedom to choose how you work in a team, we don’t know everything we’ll learn as a company by doing this. It could be chaotic, there could be challenges, and there will undoubtedly be many insights, but we will be better off for having gone through the process. Please wish us all luck as we head into next week. There’s a lot of excitement in the company to create value. We hope to have new features to share with you in the coming weeks, and we’ll be back soon with a post sharing how it went. Have you tried something like Build Week before? If so, how did it go? I’d love to hear from you on Twitter. Photo by C Dustin on Unsplash.
Our vision for location-independent salaries at Buffer Note: this was originally posted on the Buffer blog. I’m happy to share that we’ve established a long-term goal that salaries at Buffer will not be based on location. We made our first step towards this last year, when we moved from four cost-of-living based location bands for salaries to two bands. We did this by eliminating the lower two location bands The change we made resulted in salary increases for 55 of 85 team members, with the increase being on average $10,265. When the time is right, we will be eliminating the concept of cost-of-living based location bands entirely, which will lead to a simpler approach to providing generous, fair and transparent salaries at Buffer. In this post I’m sharing my thinking behind this change and our approach to pay overall. Location and Salaries It’s been interesting to see the conversation about location and salaries unfold both within Buffer and beyond. We’ve heard from many teammates over the years about the pros and cons of the location factor, and of course we’ve watched with interest as this became a regular topic of conversation within the larger remote work community. I've had many healthy debates with other remote leaders, and there are arguments for eliminating a location component which I haven’t agreed with. I don’t believe pay differences across locations is unethical, and it has made a lot of sense for us in the past. However, the last few years have seen a lot of change for remote teams. A change like this isn't to be made lightly, and at our scale comes with considerations. Our Compensation Philosophy Compensation is always slowly evolving as companies and markets mature and change. We’ve been through several major iterations of our salary formula, and myriad small tweaks throughout the last 8 or so years since we launched the initial version. Part of the fun of having a salary formula is knowing that it’s never going to be “done.” Knowing that the iterations would continue, Caryn, our VP of Finance, and I worked together to establish our compensation philosophy and document our principles on compensation to help us determine what should always be true even as the salary formula changes over time. We arrived at four principles that guide our decisions around compensation. We strive for Buffer’s approach to salary, equity, and benefits to be: Transparent Simple Fair Generous These are the tenets that have guided us through compensation decisions over the years. After we articulated them as our compensation principles, we were able to look at the location factor of our formula with new clarity. There are a few key considerations that were part of our discussions and my decision to put Buffer on a path towards removing our location factor from salaries that I'll go into more detail about next. Transparency, Simplicity, and Trust Our salary formula is one of the fundamental reasons that we can share our salaries transparently. Having a spreadsheet of team salaries is a huge step toward transparency, but true transparency is reached when the formula is simple, straightforward, easy to understand, and importantly, easy to use. In one of our earlier versions of the salary formula, we calculated the cost-of-living multiplier for every new location when we made an offer. That was cumbersome, and it meant that a candidate couldn’t truly know their salary range until we calculated that. This was improved greatly when we moved to the concept of “cost-of-living bands.”. After that, different cities and towns could more easily be classified into each band. This massively increased the transparency of the formula, and I think it helped create a lot more trust in this system. Anyone could relatively easily understand which band their location fit into, and with that knowledge understand the exact salary they'd receive at Buffer. This type of immediate understanding of the salary formula, and ability to run calculations yourself, is where transparency really gains an extra level of impact and drives trust within and beyond the team. However, with our four cost-of-living bands, there were still decisions to be made around where locations fall, and this has been the topic of much healthy and productive debate over the years. The conversations around locations falling between the Average and High bands is what led us to introduce the Intermediate band. And with four choices of location, it has meant there is some disparity in salaries across the team. With the benefits that come from the powerful combination of transparency and simplicity, alongside the increased trust that is fostered with more parity across the team, I’m choosing to drive Buffer’s salary formula in the direction of eventually having no cost-of-living factor. Freedom and Flexibility We’ve long taken approaches to work which have been grounded in the ideal of an increased level of freedom and flexibility as a team member. When I started Buffer, I wanted greater freedom and a better quality of life than I felt would be possible by working at a company. That came in various forms, including location freedom, flexibility of working hours, and financial freedom. And as we’ve built the company, I’ve been proud that we’ve built a culture where every single team member can experience an unusual and refreshing level of freedom and flexibility. Since the earliest days, one of our most fondly held values has been to Improve Consistently, and in particular this line: “We choose to be where we are the happiest and most productive”. This is a value that has supported and encouraged teammates to travel and try living in different cities, in search of that “happiest and most productive” place. It has enabled people to find work they love and great co-workers, from a hometown near family where it would be hard to find a local company that can offer that same experience and challenge. It has also enabled people to travel in order to support their partner in an important career change involving a move, something which allows an often stressful change to happen much more smoothly, since you can keep working at Buffer from anywhere in the world. Having a culture that has supported moving freely across the globe has been a powerful level of freedom and flexibility. That freedom has been matched with a salary system which adjusts compensation to accommodate those changes in a fair and appropriate way. However, knowing that your salary will fluctuate and can decrease due to a choice to be somewhere else, does limit that freedom and the ability to make a decision to move. Moving towards a salary formula with parity across all locations, will enable an even greater level of freedom and flexibility. It feels clear to me that choosing to move is a personal or a family decision, and it is ideal if Buffer salaries are structured in a way that honor and support that reality. I’m excited that working towards removing our cost-of-living differences will help significantly reduce the friction involved in making a potentially positively life-changing decision to live in a different city or country. Results, Independence, and Reward At Buffer, we are not on the typical hyper-growth VC path. This comes with some constraints: we don’t have tens of millions in funding and unlimited capital to deploy in an attempt to find a rapid path to $100m and going public (thankfully, that’s not our goal). This path also means that our experiences as teammates in a variety of ways are directly tied to whether we are successfully serving existing and new customers. For example, the level of benefits, ability to travel (in normal times), and competitiveness of compensation, are very much driven by our revenue growth and profitability. But, this is independence too. The thing we often need to remind ourselves of, is that while we may feel more constrained at times, we have full freedom of what we do with the success we achieve. Making a choice like this is one example of that. It is my intention as founder / CEO that as we succeed together as a company, we all benefit from that success and see adjustments that improve our quality of life and create wealth. We are in a position of profitability which allows us to take a significant step towards removing the cost-of-living factor from our salary framework, which I believe serves those goals. And removing it entirely will be determined by us successfully executing on our strategy and serving customers well. Reducing Cost-of-Living Bands The way our salary formula works is that we benchmark a teammate’s role based on market data at the 50th percentile for the software industry in San Francisco and then multiply that by the cost-of-living band. So, a Product Marketer benchmark at the 50th percentile of the San Francisco market data is $108,838. Depending on the teammate’s location this would be multiplied by a cost-of-living band (Low, Average, Intermediate or High). For example, if they lived Boulder, Colorado, a city with Average cost-of-living, the benchmark would be multiplied by 0.85 for a salary of $92,512. To best reflect our compensation philosophy, company values, and the path we want for Buffer, we have eliminated the Low and Average cost-of-living bands. What we’ve done is brought all Low (.75 multiplier) and Average (.85 multiplier) salaries up to Intermediate (.9 multiplier), which we now call our Global band. This is what resulted in 55 teammates seeing on average an increase to their salary of $10,265. Our two bands are now Global (.9 multiplier) and High (1.0 multiplier). This change is based on my vision for Buffer and how being a part of this team affects each of us as individually, as well as the direction I believe the world is going. I’m excited about the change first and foremost because it supports our goal of having a transparent, simple, fair, and generous approach to compensation. This is also a move that raised salaries right away for more than half of the team. This point in particular gives me a lot of joy because I want compensation to be one of the incredible parts of working at Buffer. Money isn’t everything, and we all need kind and smart colleagues, a psychologically safe environment, and to work on challenging and interesting problems, in order to be fulfilled at work. Beyond that, however, money really impacts life choices, and that’s ultimately what I want for every Bufferoo; the freedom to choose their own lifestyle and make choices for themselves and their families’ long-term health and happiness. It’s important to me that people who choose to spend their years at Buffer will have the freedom to make their own choices to have a great life. And, for our teammates who live in much lower cost-of-living areas, a Buffer salary could end up being truly life changing. I’m really happy with that outcome. The decision was also impacted by the direction that I believe the world is going (and, the direction we want to help it go). Remote is in full swing, and it’s increasingly breaking down geographical borders. I believe this is a great thing. Looking ahead 10 or even 5 years, it seems to me that we’re going to see a big rebalancing, or correction, that’s going to happen. I believe it’s important to be ahead of these types of shifts, and be proactively choosing the path that’s appropriate and energizing for us. What next? Our plan is to eventually get to one single location band, essentially eliminating the cost-of-living factor from the salary formula altogether. This will be possible once we can afford to make this change and sustain our commitment to profitability. So, this will be driven by the long-term results we create from our hard work, creativity in the market, and commitment to customers. What questions does this spark for you? Send me a tweet with your thoughts. Photo by Javier Allegue Barros on Unsplash.
More in programming
A clip of me singing a funny song from Gilbert and Sullivan’s Ruddigore back in 2013
In this video, we look at why fork() needs copy-on-write, how it works inside the kernel, and a memory usage problem that Instagram encountered with Python.
Comments require commitment, but they’re worth it.
Basecamp 5 runs on Puma in cluster mode: one master process with preload_app! and 63 single-threaded workers per host, deployed as a Docker container with Kamal. We serve Basecamp from several sites. Each site has its own web hosts and a read replica of the database, and writes go to a single primary database in one of them. On our busiest hosts, each deploy left up to 2,000 requests waiting while the new workers warmed up. We reduced those queues by running signed-in requests through the app in the Puma master, before it forked the workers. Why 63 single-threaded workers? Basecamp has always served web requests from processes rather than threads. It ran on Unicorn, which only does processes, until we moved to Puma in January 2025, and we kept the same setup: workers (Concurrent.physical_processor_count * 1.3).ceil threads 1, 1 preload_app! On a 48-core host that’s 63 workers, each handling one request at a time. We chose 1.3 after benchmarking HEY in 2023, when we moved our apps out of the cloud and onto our own hardware. We tested several combinations of workers and threads with a mix of GET and POST requests on a 32-vCPU VM. Every multithreaded configuration we tested was slower and handled fewer requests than single-threaded workers. Adding workers beyond about 1.2 to 1.3 per vCPU brought little benefit. The threaded workers spent a lot of their time waiting for Ruby’s global VM lock. That made single-threaded workers a good fit for this workload, and we use the same setup for Basecamp. An app that spends more time waiting on its database or other services may benefit from more threads, so benchmark your own app. The other reason is the app itself. Basecamp has class-level state in places and has never needed to be thread-safe. With one request per process, it still doesn’t. Processes do use more memory than threads, and preload_app! reduces the difference. The master loads the app once and the workers share its memory through copy-on-write until they write to it. Shopify’s comparison of Ruby execution models explains the trade-off well. In the HEY benchmark the best setup came to about 260 MB of PSS per core, where PSS counts each shared page once, split between the processes using it, and the gap to a threaded setup was smaller than we’d expected. What Puma does on each host when a container starts: one master, then 63 forked workers that share its memory until they write to it. Two things about this setup matter for the rest of the post. A worker that’s compiling or loading something is fully blocked — there’s no other thread to pick up the next request. And whatever the master has in memory before it forks, all 63 workers share. Whatever they build after the fork, they build 63 times. What happens when we deploy Kamal starts the new container alongside the old one, and kamal-proxy moves the host’s traffic across as soon as the health check passes. At that moment, the new workers have handled health checks but no customer requests. preload_app! means the master loads the app once and the workers inherit it through fork. That covers the code. It doesn’t cover anything Ruby and Rails set up on first use: YJIT compiled code. YJIT compiles a method once it’s been called a certain number of times. The master calls very little during boot, so every worker compiles the same methods again on its own first requests. Compiled templates. Action View turns each ERB template into a Ruby method the first time it’s rendered. The schema cache. Active Record reads each model’s columns from the database the first time that model is used. Inline caches and memoized values throughout Ruby, Rails and the app. All 63 workers did all of this at once, while serving the traffic the old container had been handling a second earlier. In the test environment with YJIT on, the first request to a project page on a cold process took 652 ms, 151 ms of it YJIT compiling. The same request to a warm process took 28 ms. In production, CPU time per request peaked at around 200 ms while kamal-proxy moved traffic to the new container, against about 30 ms once the workers had warmed up. A host with spare CPU absorbs this. Every one of our web hosts has 48 cores and 63 workers, but each Amsterdam host serves around 250 requests per second, against 25 to 60 at our other sites. In Amsterdam the slow first requests turned into a queue. At a peak-hour deploy, the Puma backlog on an Amsterdam host reached anywhere from 250 to 2,238 requests, and kamal-proxy’s p99 response time hit about 10 seconds. Eron, our Director of Operations, had been tracking this since June. Another server in Amsterdam would help, but it would take weeks to arrive, so we also wanted to make deploys cheaper on the hardware we already had. What didn’t work We tried a few things first. In June, Donal tested the first two on a single Amsterdam host, comparing it with its neighbors, and they ruled out two likely causes. Warming each worker’s database connections. Puma’s before_fork hook clears the master’s connections, and each worker opened its own on its first request. Opening them in before_worker_boot instead made no difference. Queries on a freshly booted production host were already under a millisecond, so connections weren’t the problem. A synthetic request in each worker. Next, each worker made a few requests in before_worker_boot to an internal controller that touched every model. That ran the middleware, routing and Active Record paths, but it ran them in 63 workers at once — exactly the CPU spike we were trying to avoid. And a request with no real data renders no real views, so most of the app stayed cold. Spreading YJIT compilation out. Delaying YJIT in each worker by a random interval spread the compiling out over a few minutes, but every worker still ran interpreted until its delay ended. The queue didn’t change. Reforking from a warm worker. This is what Shopify’s Pitchfork does: let one worker serve traffic until it’s warm, then fork the others from it. Puma has an experimental version called fork_worker, and on beta it worked — the reforked workers were warm after three to five requests, where fresh ones took up to 30 seconds. But with fork_worker the template is worker 0, and it keeps serving requests. If it exits, the workers waiting to be forked never start (puma/puma#3596). If it gets no traffic, the refork never happens, which is what we saw on beta. Instacart have a mold_worker patch that promotes a warm worker to a template that stops serving, but it isn’t in a Puma release. We have a branch of it, and we may come back to it. That last experiment did show us where the fix was, though. Everything a warm worker has that a cold one lacks is in its memory, and fork copies memory. The master already has the app loaded. It just never runs it. Run the requests in the master So now, before the master binds its socket and forks, it makes the app’s own requests, in-process, the way a signed-in user would. Rack has a hook for exactly this. Rack::Builder#warmup takes a block that’s called once with the built app, before the server starts. rails server builds the app from config.ru, so the change to boot is one line: require_relative "config/environment" warmup { WarmUp.configured.run } if ENV["WARM_UP"] run Rails.application With preload_app! this runs in the master, and the workers inherit whatever it did. Puma binds its socket after the app is built, so until the warm-up finishes the health check’s connection is refused and kamal-proxy keeps retrying. No request reaches a worker that hasn’t been warmed. The warm-up has three steps. After precompiling the views, it gives the page requests and schema loading a shared 20-second budget, checked before each page or model. 1. Precompile the views actionview_precompiler reads every template for its render calls and compiles each one with the locals it’s passed. For us that’s 1,394 templates in about two seconds. A first request to a project page then compiles 2 templates instead of 44. 2. Request the pages, signed in A small browser class makes the requests through Rack::MockRequest, with the two cookies a real sign-in sets, then goes back for each page’s lazy Turbo frames: class WarmUp::Browser def initialize(signed_in_as:) @client = Rack::MockRequest.new(Rails.application) @headers = { "HTTP_USER_AGENT" => "Basecamp warm-up", "HTTP_COOKIE" => cookie_for(signed_in_as), "bc3.warm_up" => true } end def visit(path) page = get(path) frames_in(page).each { |id, src| get(src, "HTTP_TURBO_FRAME" => id) } end private def get(path, headers = {}) @client.get("https://#{host}#{path}", @headers.merge(headers)) end def frames_in(page) Nokogiri::HTML5(page.body).css("turbo-frame[src]").map { |frame| [ frame["id"], frame["src"] ] } end end The requests are signed in. The user is a monitoring account we already use for automated checks, and the pages are its own project, Campfire, to-dos, documents and messages. Public pages weren’t enough: after warming up with signed-out pages only, the first signed-in request to the projects page still took 131 ms, because authentication, the signed-in controllers and their views had never run. With signed-in pages it took 40 ms. cookie_for writes the same signed cookie the sign-in controller does, using the app’s own cookie jar, so there’s no API token and no secret to store. The frames are followed. The busiest HTML requests in production aren’t pages at all but Turbo frames — the sidebar badge, the inbox, the navigation menus. The browser parses each page and requests its <turbo-frame src> URLs with the Turbo-Frame header, so those controllers and views get warmed too. Our first four pages turned into 60 requests. The requests are excluded from rate limiting. They are internal, so they do not count against the rate limits that apply to real visitors. 3. Load the rest of the schema The page requests load the schema for the models they touch. The last step loads the rest, from the read replica: ApplicationRecord.reading do models.lazy.take_while { time_left? }.each { |model| model.load_schema if model.table_exists? } end The step checks 261 models and loads any schema information still missing. Those database round trips add up when the primary is far away: outside a request, Active Record uses the writing role, and from a host a long way from the primary each round trip is tens of milliseconds. Reading from the local replica brings the step down from about 20 seconds to 3.5. The pages go first because they load most of the schema anyway. If the time budget runs out, the step stops, logs how many models it got through, and the workers load the rest on first use like they always did. Rails can also load the schema from a dumped cache file at boot (bin/rails db:schema:cache:dump), which would make this step unnecessary. We don’t ship one in our image yet, because the dump needs a database to read from at build time, and we have several databases to cover. It’s on the list. What to close before the fork Running requests in the master opens things the master never opened before, and every worker inherits them. Two processes writing to the same socket will corrupt each other’s traffic, so you need to know what’s open before you fork. The way to find out is to list the master’s open file descriptors — ls -l /proc/<pid>/fd — before and after a warm-up, in an environment set up like production. Development wasn’t enough for us: it stores files on disk, so our S3 connections only showed up in production. Then, for each thing that’s open, check how its library handles a fork. We found three kinds: Already handled. Plenty of libraries detect a fork on their own, either by recording the PID they connected from and reconnecting in the child, by opening per-process files, or by resetting their thread pools. Redis clients, metrics libraries and concurrency libraries tend to be in this group. Check, but you probably don’t need to do anything. Already closed. Database connections are the classic one, and most Puma configs already clear them in before_fork. Anything else that’s opened per process — we have a SQLite cache the workers open on boot — needs closing when the warm-up finishes. Needs a new step. HTTP clients with keep-alive connections are the ones to look for: cloud SDKs with connection pools, tracing exporters, error reporters. They usually have no fork handling at all. We empty the aws-sdk connection pools in before_fork, and we run the warm-up untraced so the OpenTelemetry exporter never opens its connection to Tempo in the first place. Once that’s done, before_fork finishes with Process.warmup, which Ruby 3.3 added for this purpose: a major GC, a heap compaction, and every surviving object promoted to the old generation, so the memory pages the workers share change as little as possible afterwards. Choosing the pages The first list was the four pages that ran the busiest requests on beta. Once the warm-up was live, production showed us which endpoints were still cold. For one deploy, we compared each endpoint’s mean duration in the six minutes after kamal-proxy moved traffic to the new container with the same endpoint an hour later, then multiplied the difference by the number of requests in those six minutes. That gives the extra time each endpoint cost us because it was cold: Endpoint Cold Warm Requests in 6 min Extra seconds Campfire 246 ms 70 ms 6,490 1,140 Projects (JSON API) 84 ms 50 ms 22,077 771 Docs & Files 262 ms 177 ms 4,996 421 To-dos tool 205 ms 113 ms 4,018 371 To-dos (JSON API) 33 ms 16 ms 18,738 320 The pages already in the warm-up showed what to expect: the project page kept a 36 ms gap after a deploy, and the to-do page 10 ms. We’ve proposed adding these five requests, and expect them to add about five to seven seconds to the page step. The two JSON endpoints were a surprise. The warm-up’s page list had no API requests in it, so nothing on the API path had run before the first real request: not the API controllers, and not the Jbuilder templates rendering real records. Precompiling the views covers JSON templates too, but it isn’t a substitute for running the request. Results The warm-up is on for all 68 web hosts. With the first four pages it took 12 to 16 seconds per host: about 2 seconds to precompile the views, 7 to 9 for the 60 requests, and 3.5 for the schema. Deploys take that much longer per host, and we raised the deploy timeout from 30 to 60 seconds to cover it. In Amsterdam, at a peak-hour deploy: During deploy Before After Peak Puma backlog per host 250–2,238 requests 19–223 requests Peak kamal-proxy p99 about 10 s 2.4–4.8 s Peak CPU time per request 201–214 ms 88–132 ms Peak database time per request 56–69 ms 39–47 ms The same eight hosts at three deploys on 1 October, an hour apart, as the warm-up went from one host to four to all eight. The deploy in the middle, with four hosts warmed and four not, shows why every host needed the warm-up. Each warmed host recovered faster on its own: mean request duration peaked at 130 to 173 ms, against 203 to 311 ms on the hosts that weren’t warmed. But the backlog on all eight was about the same, because they were all waiting on the same database. Mean request duration on each host at the 07:21 UTC deploy. Blue hosts warmed up in the master before forking, orange hosts did not. Memory came down too. The workers now share compiled templates, YJIT code and the schema with the master instead of each building their own copy. On beta, the view precompiler alone took a busy worker’s private memory from 174–202 MB to 119–135 MB. Thirty minutes after the deploy, the web containers used about 39 GB less memory than the previous day’s containers at the same age and traffic. Amsterdam served most of our traffic at the times we tested. In Amsterdam, each new container used about 2 GB less just after traffic moved to it, which lowers the peak while the old and new containers overlap. Working with Claude Claude Code helped throughout. It combed through the per-worker backlogs and per-endpoint timings in Prometheus and Loki after each deploy, worked out the cold-versus-warm cost of each endpoint, and prepared the changes and the pull request descriptions with the benchmarks in them. We decided what to try, deployed it and read the results. If you do this Warm the master before it forks. Compile common code and templates and load their schema in the master, so workers inherit that work. With preload_app!, Rack::Builder#warmup runs before the workers start accepting traffic. Use the app’s real requests. Public pages, internal endpoints and synthetic queries warm the paths they run and nothing else. Signed-in requests to real records, frames included, run what production runs. Measure the cold penalty per endpoint. The difference between an endpoint’s cold and warm duration, times its request count after a deploy, ranks the pages worth adding. Ours weren’t the ones we’d have guessed, and two of them were JSON. Check what the warm-up leaves open. List the master’s file descriptors after a warm-up and account for every one before the fork. Two of ours needed changes. Set a time budget. A warm-up that runs long on one slow host fails the deploy on that host. Ours gives the page requests and schema loading a shared 20-second budget, checked before each page or model, puts the most valuable pages first, and logs what it skipped. Reforking from a warm worker, as Pitchfork does, solves the same problem continuously rather than once at boot, and it would warm paths no fixed list of pages covers. We may still get there: our branch brings Instacart’s mold_worker up to date with Puma’s main branch and fixes the bugs we found in it. But warming the master works with the Puma we already run, took a few days to implement, and substantially reduced the queues after deployment.