Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Cognitive Computations
With the recent update to OpenAI's Terms of Use on October 23, 2024, there’s been a flurry of online discussions around what these terms mean for developers, businesses, and everyday users of AI tools like ChatGPT. Much of the conversation, especiall...
Gratitude to https://tensorwave.com/ for giving me access to their excellent servers! Few have tried this and fewer have succeeded. I've been marginally successful after a significant amount of effort, so it deserves a blog post. Know that you are in for rough waters. And even when you arrive - There are lots of optimizations tailored for nVidia GPUs so, even though the hardware may be just as strong spec-wise, in my experience so far, it still may take 2-3 times as long to train on equivalient AMD hardware. (though if you are a super hacker maybe you can fix it!) Right now I'm using Axolotl. Though I am probably going to give LlamaFactory a solid try in the near future. There's also LitGpt and TRL. But I kind of rely on the dataset features and especially the sample packing of Axolotl. But more and more LlamaFactory is interesting me, it supports new features really fast. (like GaLore is the new hotness at the moment). This blog post will be about getting Axolotl up and running in AMD, and I may do one about LlamaFactory if there is demand. I am using Ubuntu 22.04 LTS, and you should too. (unless this blog post is really old by the time you read it). Otherwise you can use this post as a general guide. Here are all the environment variables I ended up setting in my .bashrc and I'm not exactly sure which ones are needed. You better set them all just in case. export GPU_ARCHS="gfx90a" # mi210 - use the right code for your GPUexport ROCM_TARGET="gfx90a"export HIP_PATH="/opt/rocm-6.0.0"export ROCM_PATH="/opt/rocm-6.0.0"export ROCM_HOME="/opt/rocm-6.0.0"export HIP_PLATFORM=amdexport DS_BUILD_CPU_ADAM=1 export TORCH_HIP_ARCH_LIST="gfx90a" Part 1: Driver, ROCm, HIP Clean everything out. There shouldn't be any trace of nvidia, cuda, amd, hip, rocm, anything like that. This is not necessarily a simple task, and of course it totally depends on the current state of your system. and I had to use like 4 of my daily Claude Opus questions to accomplish this. (sad face) By the way Anthropic Claude Opus is the new king of interactive troubleshooting. By far. Bravo. Don't nerf it pretty please! Here are some things I had to do, that might help you: sudo apt autoremove rocm-core sudo apt remove amdgpu-dkms sudo dpkg --remove --force-all amdgpu-dkms sudo apt purge amdgpu-dkms sudo apt remove --purge nvidia* sudo apt remove --purge cuda* sudo apt remove --purge rocm-* hip-* sudo apt remove --purge amdgpu-* xserver-xorg-video-amdgpu sudo apt clean sudo reboot sudo dpkg --remove amdgpu-install sudo apt remove --purge amdgpu-* xserver-xorg-video-amdgpu sudo apt autoremove sudo apt clean rm ~/amdgpu-install_*.deb sudo reboot sudo rm /etc/apt/sources.list.d/amdgpu.list sudo rm /etc/apt/sources.list.d/rocm.list sudo rm /etc/apt/sources.list.d/cuda.list sudo apt-key del A4B469963BF863CC sudo apt update sudo apt remove --purge nvidia-* cuda-* rocm-* hip-* amdgpu-* sudo apt autoremove sudo apt clean sudo rm -rf /etc/OpenCL /etc/OpenCL.conf /etc/amd /etc/rocm.d /usr/lib/x86_64-linux-gnu/amdgpu /usr/lib/x86_64-linux-gnu/rocm /opt/rocm-* /opt/amdgpu-pro-* /usr/lib/x86_64-linux-gnu/amdvlk sudo reboot I love Linux (smile with tear) Now finally do like sudo apt-get updatesudo apt-get upgrade and sudo apt-get dist-upgrade and make sure there's no errors or warnings! You should be good to begin your journey. Install AMD drivers, ROCm, HIP wgethttps://repo.radeon.com/amdgpu-install/23.40.2/ubuntu/jammy/amdgpu-install_6.0.60002-1_all.deb (at time of this writing). But you should double check here. And the install instructions here. sudo apt-get install ./amdgpu-install_6.0.60002-1_all.deb sudo apt-get update sudo amdgpu-install -y --accept-eula --opencl=rocr --vulkan=amdvlk --usecase=workstation,rocm,rocmdev,rocmdevtools,lrt,opencl,openclsdk,hip,hiplibsdk,mllib,mlsdk If you get error messages (I did) try to fix them. I had to do this: sudo dpkg --remove --force-all libvdpau1 sudo apt clean sudo apt update sudo apt --fix-broken install sudo apt upgrade and then, again, I had to run sudo amdgpu-install -y --accept-eula --opencl=rocr --vulkan=amdvlk --usecase=workstation,rocm,rocmdev,rocmdevtools,lrt,opencl,openclsdk,hip,hiplibsdk,mllib,mlsdk Check Installation rocm-smirocminfo/opt/rocm/bin/hipconfig --full I hope that worked for you - if not, I suggest asking Claude Opus about the error messages to help you figure it out. If that doesn't work, reach out to the community. Part 2: Pytorch, BitsAndBytes, Flash Attention, DeepSpeed, Axolotl Conda mkdir -p ~/miniconda3wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.shbash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3rm -rf ~/miniconda3/miniconda.sh~/miniconda3/bin/conda init bash Exit your shell and enter it again. conda create -n axolotl python=3.12conda activate axolotl Pytorch I tried the official install command from pytorch's website, and it didn't work for me. Here is what did work: pip install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/rocm6.0python -c "import torch; print(torch.version.hip)" This tests both Torch, and Torch's ability to interface with HIP. If it worked, it will print HIP version. Otherwise, it will print None. BitsAndBytes BitsAndBytes is by Tim Dettmers, an absolute hero among men. It lets us finetune in 4-bits. It gives us qLoRA. It brings AI to the masses. There is a fork of BitsAndBytes that supports ROCm. This is provided not by Tim Dettmers, and not by AMD, but by a vigilante superhero, Arlo-Phoenix. In appreciation, here is a portrait ChatGPT made for Arlo-Phoenix, vigilante superhero. I hope you like it, if you see this Arlo-Phoenix. <3 git clone https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6cd bitsandbytes-rocm-5.6git checkout rocmROCM_TARGET=gfx90a make hip # use the ROCM_TARGET for your GPUpip install . Flash Attention This fork is maintained by AMD git clone --recursive https://github.com/ROCmSoftwarePlatform/flash-attention.gitcd flash-attentionexport GPU_ARCHS="gfx90a" # use the GPU_ARCHS for your GPUpip install . DeepSpeed Microsoft included AMD support in DeepSpeed proper, but there's still some undocumented fussiness to get it working, and there is a bug I found with DeepSpeed, I had to modify it to get it to work. git clone https://github.com/microsoft/DeepSpeedcd DeepSpeedgit checkout v0.14.0 # but check the tags for newer version Now, you gotta modify this file: vim op_builder/builder.py Replace the function assert_no_cuda_mismatch with this: (unless they fixed it yet) def assert_no_cuda_mismatch(name=""): cuda_available = torch.cuda.is_available() if not cuda_available and not torch.version.hip: # Print a warning message indicating no CUDA or ROCm support print(f"Warning: {name} requires CUDA or ROCm support, but neither is available.") return False else: # Check CUDA version if available if cuda_available: cuda_major, cuda_minor = installed_cuda_version(name) sys_cuda_version = f'{cuda_major}.{cuda_minor}' torch_cuda_version = torch.version.cuda if torch_cuda_version is not None: torch_cuda_version = ".".join(torch_cuda_version.split('.')[:2]) if sys_cuda_version != torch_cuda_version: if (cuda_major in cuda_minor_mismatch_ok and sys_cuda_version in cuda_minor_mismatch_ok[cuda_major] and torch_cuda_version in cuda_minor_mismatch_ok[cuda_major]): print(f"Installed CUDA version {sys_cuda_version} does not match the " f"version torch was compiled with {torch.version.cuda} " "but since the APIs are compatible, accepting this combination") return True elif os.getenv("DS_SKIP_CUDA_CHECK", "0") == "1": print( f"{WARNING} DeepSpeed Op Builder: Installed CUDA version {sys_cuda_version} does not match the " f"version torch was compiled with {torch.version.cuda}." "Detected `DS_SKIP_CUDA_CHECK=1`: Allowing this combination of CUDA, but it may result in unexpected behavior." ) return True raise CUDAMismatchException( f">- DeepSpeed Op Builder: Installed CUDA version {sys_cuda_version} does not match the " f"version torch was compiled with {torch.version.cuda}, unable to compile " "cuda/cpp extensions without a matching cuda version.") else: print(f"Warning: {name} requires CUDA support, but torch.version.cuda is None.") return False return True pip install -r requirements/requirements.txtHIP_PLATFORM="amd" DS_BUILD_CPU_ADAM=1 TORCH_HIP_ARCH_LIST="gfx90a" python setup.py install Axolotl Installing Axolotl might overwrite BitsAndBytes, DeepSpeed, and PyTorch. Be prepared for things to break, they do often. Your choice is either modify the setup.py and requirements.txt (if you are confident to change those things) or pay attention to what libraries get deleted and reinstalled, and just delete them again and reinstall the correct ROCm version that you installed earlier. If Axolotl complains about incorrect versions - just ignore it, you know better than Axolotl. Right now, Axolotl's Flash Attention implementation has a hard dependency on Xformers for its SwiGLU implementation, and Xformers doesn't work with ROCm, you can't even install it. So, we are gonna have to hack axolotl to remove that dependency. https://github.com/OpenAccess-AI-Collective/axolotl.gitcd axolotl from requirements.txt remove xformers==0.0.22 from setup.py make this change (remove any mention of xformers) $ git diff setup.pydiff --git a/setup.py b/setup.pyindex 40dd0a6..235f1d0 100644--- a/setup.py+++ b/setup.py@@ -30,7 +30,7 @@ def parse_requirements(): try: if "Darwin" in platform.system():- _install_requires.pop(_install_requires.index("xformers==0.0.22"))+ print("hi") else: torch_version = version("torch") _install_requires.append(f"torch=={torch_version}")@@ -45,9 +45,6 @@ def parse_requirements(): else: raise ValueError("Invalid version format")- if (major, minor) >= (2, 1):- _install_requires.pop(_install_requires.index("xformers==0.0.22"))- _install_requires.append("xformers>=0.0.23") except PackageNotFoundError: pass And then in src/axolotl/monkeypatch/llama_attn_hijack_flash.py make this change: --- a/src/axolotl/monkeypatch/llama_attn_hijack_flash.py+++ b/src/axolotl/monkeypatch/llama_attn_hijack_flash.py@@ -22,7 +22,9 @@ from transformers.models.llama.modeling_llama import ( apply_rotary_pos_emb, repeat_kv, )-from xformers.ops import SwiGLU+class SwiGLU:+ def __init__():+ print("hi") from axolotl.monkeypatch.utils import get_cu_seqlens_from_pos_ids, set_module_name@@ -45,15 +47,7 @@ LOG = logging.getLogger("axolotl") def is_xformers_swiglu_available() -> bool:- from xformers.ops.common import get_xformers_operator-- try:- get_xformers_operator("swiglu_packedw")()- return True- except RuntimeError as exc:- if "No such operator xformers::swiglu_packedw " in str(exc):- return False- return True+ return False Now you can install axolotl pip install -e .accelerate launch -m axolotl.cli.train examples/openllama-3b/lora.yml Welcome to finetuning on ROCm!
Wanna chat with Dolphin locally? (no internet connection needed) Here is the easy way - Ollama. install ollama. after you finsh you should be able to run ollama from the command line. Also you will see the ollama icon up top like this: Iff you are curious - anytime you see that icon, that means ollama is running in the background and it also has a port open (11434) that can accept api calls. Sadly, as of this writing there is no swagger. Back to business. download your dolphin. You can choose which size you want from here. I am choosing size Q5_K_M which is just over 5gb. mkdir ~/models cd ~/models wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf Now you need a Modelfile. vim Modelfile.dolphin.mistral Add the following text to Modelfile.dolphin.mistral FROM ./dolphin-2.2.1-mistral-7b.Q5_K_M.gguf TEMPLATE """{{ if and .First .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}<|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ SYSTEM """You are Dolphin, an uncensored, unbiased AI assistant""" PARAMETER stop <|im_end|> Now you can use Ollama to install this model. ollama create dolphin.mistral -f Modelfile.dolphin.mistral Now look, you can run it from the command line. Which is cool enough. But we are just getting started. If you want, you can install samantha too so you have two models to play with. wget https://huggingface.co/TheBloke/samantha-1.2-mistral-7B-GGUF/resolve/main/sama ntha-1.2-mistral-7b.Q5_K_M.gguf vim Modelfile.samantha.mistral And enter the following into Modelfile.samantha.mistral FROM ./samantha-1.2-mistral-7b.Q5_K_M.gguf TEMPLATE """{{ if and .First .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}<|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ SYSTEM """You are Samantha, an AI companion""" PARAMETER stop <|im_end|> Then install the model ollama create samantha -f Modelfile.samantha.mistral And now you can also chat with Samantha from the command line. Cool yeah? We are just getting started. Let's get Ollama Web UI installed. cd ~ git clone https://github.com/ollama-webui/ollama-webui.git cd ollama-webui npm i npm run dev Now you can open that link http://localhost:5173 in your web browser. now you can choose dolphin or samantha from the dropdown (I have installed a few others too) Well talking to these models from the command line and the web ui is just the beginning. Also, frameworks such as langchain, llamaindex, litellm, autogen, memgpt all can integrate with ollama. Now you can really play with these models. Here is a fun idea that I will leave as an exercise - given some query, ask dolphin to decide whether a question about coding, a request for companionship, or something else. If it is a request for companionship then send it to Samantha. If it is a coding question, send it to deepseek-coder. Otherwise, send it to Dolphin. And just like that, you have your own MoE.
When a16z generously sponsored Dolphin, I had some compute budget, and because the original dolphin-13b was a flop, I had some time to go back to the drawing board. When I was ready to train the next iteration, I reconsidered whether to rent or buy the compute for the build. I ultimately decided to buy, because I have the skill and interest, I'm good at finding deals, and owning a cluster would give me the ability to continue executing on future projects beyond Dolphin, not to mention the satisfaction of building the AI end-to-end, like a baker baking bread from scratch. Artisan AI. What to build? I am building 4 servers of 8x AMD Instinct MI100 and 4 servers of 8x NVIDIA GeForce RTX 4090. But for now, I'm just going to get 2 servers up and running at a time. Later, after I've built out all of the servers and tested them, then I will get the full 8 server setup running. (that'll require extra electrical work) I went with the MI100s because I got a killer deal with Rhino Technology (who have an excellent sales and technical team, I highly recommend) on some refurbished Gigabyte G482-Z53 servers that came preinstalled each with 4x MI100s. As a scrappy guy building in my garage, I gotta roll with the deals that I can find. And the servers support 8x PCI-e gen4 x16. Exactly what I needed. This capability is pretty hard to find, by the way. Usually, the PCI slots are bifurcated and don't get all 16 lanes. And for training AI models for bandwidth I really need each card to get all 16 lanes. The servers also came preinstalled with dual AMD EPYC 7742 64-core CPUs (wow!) and 256GB RAM. Which is plenty to start with. I had a very good start to my cluster with these servers. So I'm starting with the easy ones, the MI100s. They are easy because they fit in the server, unlike the 4090s, which are larger and won't fit in the chassis. My inspiration: https://www.pugetsystems.com/labs/articles/1-7x-nvidia-geforce-rtx-4090-gpu-scaling/ https://nonint.com/2022/05/30/my-deep-learning-rig/ The care and feeding of servers Then I had to do some math. I planned to limit each card to 300 watts each. So I figured, I need 240 volts and 25 amps per server. For the cooling, for each pair of servers I got a 25000 BTU air conditionter (240v, 30 amps) So for my "building" phase when I only need to power 2 servers at a time, I will have 2 breakers. One 240v 50 amps for the two servers, and one 240v 30 amps for the air conditioner. Later, when all the servers are ready for operation, I will need 4x 50 amp breakers and 4x 30 amp breakers. And I figured with the amps and the length of the wire, I needed an 8-gauge 3 conductor wire that I got from the hardware store. And a couple of flush-mounted outlets, one 6-50 for the pair of servers and one 6-30 for the air conditioner. I found these PDUs that can each handle 50 amps and power 2 servers that each have 3 power supplies. Perfect. As I have never done this before, it took a few weeks of researching and ordering things to get all of this put together before I was able to power up the first server. Getting Ubuntu Server installed Pretty straightforward, just put Ubuntu on a thumb drive and boot from that. Of course, I needed to get a monitor and a keyboard working too. And one of my servers turned out to have a nonfunctional VGA output so i switched to another server for now. Getting it on the network So I didn't wanna run a wire ALL the way to my living room. I bought these TP-Link AC600 wireless adapters but of course they didn't just work when I plug them in. So I had to first get the server on the network so I could install the driver. So I hooked it up with ethernet to my workstation and used Windows wifi sharing to bridge the wifi to the ethernet. That got the server on the internet. After that, I was able to download updates. I installed links2 web browser because Ubuntu Server has no GUI. After that, I was able to install the driver for the TP-Link AC600 and get it connected. Then I give the mac address an assigned ip address on my router's dhcp, so that I can forward port 22 and dynamic dns so I can SSH to it from the outside. (of course, I add my public ed25519 key and disable password login) This server will act as my bastion, I will ssh from this to the other servers in my cluster. Installing drivers, ROCm, and HIP I was told I should use the docker image, but I couldn't get that working. Instead I installed the drivers from the repository. The trick is this: don't try to install multiple versions of ROCm, just install version 5.7, and install the nightly version of Pytorch that works with 5.7. That's the combination that made everything work for me. Actually, AMD's install experience is better than NVidia's. Also you gotta install NVidia's Cuda too before you install HIP. https://docs.amd.com/projects/HIP/en/docs-5.3.0/how_to_guides/install.html Inference with Oobabooga Setup Oobabooga as normal, using requirements-rocm.txt then duplicate this drive for the other servers Future Plans I am going to make any changes required to get Axolotl running on these servers. I am going to put a Lustre cluster on these servers, I plan to do 7x 2tb ssd on each server and using 100GbE so it should be fast enough and able to handle the nodes saving and loading checkpoints. This is very important for multinode training. I am going to train more versions of dolphin and other models using these servers. I am eventually going to train my own base model.
More in programming
A clip of me singing a funny song from Gilbert and Sullivan’s Ruddigore back in 2013
In this video, we look at why fork() needs copy-on-write, how it works inside the kernel, and a memory usage problem that Instagram encountered with Python.
Comments require commitment, but they’re worth it.
Basecamp 5 runs on Puma in cluster mode: one master process with preload_app! and 63 single-threaded workers per host, deployed as a Docker container with Kamal. We serve Basecamp from several sites. Each site has its own web hosts and a read replica of the database, and writes go to a single primary database in one of them. On our busiest hosts, each deploy left up to 2,000 requests waiting while the new workers warmed up. We reduced those queues by running signed-in requests through the app in the Puma master, before it forked the workers. Why 63 single-threaded workers? Basecamp has always served web requests from processes rather than threads. It ran on Unicorn, which only does processes, until we moved to Puma in January 2025, and we kept the same setup: workers (Concurrent.physical_processor_count * 1.3).ceil threads 1, 1 preload_app! On a 48-core host that’s 63 workers, each handling one request at a time. We chose 1.3 after benchmarking HEY in 2023, when we moved our apps out of the cloud and onto our own hardware. We tested several combinations of workers and threads with a mix of GET and POST requests on a 32-vCPU VM. Every multithreaded configuration we tested was slower and handled fewer requests than single-threaded workers. Adding workers beyond about 1.2 to 1.3 per vCPU brought little benefit. The threaded workers spent a lot of their time waiting for Ruby’s global VM lock. That made single-threaded workers a good fit for this workload, and we use the same setup for Basecamp. An app that spends more time waiting on its database or other services may benefit from more threads, so benchmark your own app. The other reason is the app itself. Basecamp has class-level state in places and has never needed to be thread-safe. With one request per process, it still doesn’t. Processes do use more memory than threads, and preload_app! reduces the difference. The master loads the app once and the workers share its memory through copy-on-write until they write to it. Shopify’s comparison of Ruby execution models explains the trade-off well. In the HEY benchmark the best setup came to about 260 MB of PSS per core, where PSS counts each shared page once, split between the processes using it, and the gap to a threaded setup was smaller than we’d expected. What Puma does on each host when a container starts: one master, then 63 forked workers that share its memory until they write to it. Two things about this setup matter for the rest of the post. A worker that’s compiling or loading something is fully blocked — there’s no other thread to pick up the next request. And whatever the master has in memory before it forks, all 63 workers share. Whatever they build after the fork, they build 63 times. What happens when we deploy Kamal starts the new container alongside the old one, and kamal-proxy moves the host’s traffic across as soon as the health check passes. At that moment, the new workers have handled health checks but no customer requests. preload_app! means the master loads the app once and the workers inherit it through fork. That covers the code. It doesn’t cover anything Ruby and Rails set up on first use: YJIT compiled code. YJIT compiles a method once it’s been called a certain number of times. The master calls very little during boot, so every worker compiles the same methods again on its own first requests. Compiled templates. Action View turns each ERB template into a Ruby method the first time it’s rendered. The schema cache. Active Record reads each model’s columns from the database the first time that model is used. Inline caches and memoized values throughout Ruby, Rails and the app. All 63 workers did all of this at once, while serving the traffic the old container had been handling a second earlier. In the test environment with YJIT on, the first request to a project page on a cold process took 652 ms, 151 ms of it YJIT compiling. The same request to a warm process took 28 ms. In production, CPU time per request peaked at around 200 ms while kamal-proxy moved traffic to the new container, against about 30 ms once the workers had warmed up. A host with spare CPU absorbs this. Every one of our web hosts has 48 cores and 63 workers, but each Amsterdam host serves around 250 requests per second, against 25 to 60 at our other sites. In Amsterdam the slow first requests turned into a queue. At a peak-hour deploy, the Puma backlog on an Amsterdam host reached anywhere from 250 to 2,238 requests, and kamal-proxy’s p99 response time hit about 10 seconds. Eron, our Director of Operations, had been tracking this since June. Another server in Amsterdam would help, but it would take weeks to arrive, so we also wanted to make deploys cheaper on the hardware we already had. What didn’t work We tried a few things first. In June, Donal tested the first two on a single Amsterdam host, comparing it with its neighbors, and they ruled out two likely causes. Warming each worker’s database connections. Puma’s before_fork hook clears the master’s connections, and each worker opened its own on its first request. Opening them in before_worker_boot instead made no difference. Queries on a freshly booted production host were already under a millisecond, so connections weren’t the problem. A synthetic request in each worker. Next, each worker made a few requests in before_worker_boot to an internal controller that touched every model. That ran the middleware, routing and Active Record paths, but it ran them in 63 workers at once — exactly the CPU spike we were trying to avoid. And a request with no real data renders no real views, so most of the app stayed cold. Spreading YJIT compilation out. Delaying YJIT in each worker by a random interval spread the compiling out over a few minutes, but every worker still ran interpreted until its delay ended. The queue didn’t change. Reforking from a warm worker. This is what Shopify’s Pitchfork does: let one worker serve traffic until it’s warm, then fork the others from it. Puma has an experimental version called fork_worker, and on beta it worked — the reforked workers were warm after three to five requests, where fresh ones took up to 30 seconds. But with fork_worker the template is worker 0, and it keeps serving requests. If it exits, the workers waiting to be forked never start (puma/puma#3596). If it gets no traffic, the refork never happens, which is what we saw on beta. Instacart have a mold_worker patch that promotes a warm worker to a template that stops serving, but it isn’t in a Puma release. We have a branch of it, and we may come back to it. That last experiment did show us where the fix was, though. Everything a warm worker has that a cold one lacks is in its memory, and fork copies memory. The master already has the app loaded. It just never runs it. Run the requests in the master So now, before the master binds its socket and forks, it makes the app’s own requests, in-process, the way a signed-in user would. Rack has a hook for exactly this. Rack::Builder#warmup takes a block that’s called once with the built app, before the server starts. rails server builds the app from config.ru, so the change to boot is one line: require_relative "config/environment" warmup { WarmUp.configured.run } if ENV["WARM_UP"] run Rails.application With preload_app! this runs in the master, and the workers inherit whatever it did. Puma binds its socket after the app is built, so until the warm-up finishes the health check’s connection is refused and kamal-proxy keeps retrying. No request reaches a worker that hasn’t been warmed. The warm-up has three steps. After precompiling the views, it gives the page requests and schema loading a shared 20-second budget, checked before each page or model. 1. Precompile the views actionview_precompiler reads every template for its render calls and compiles each one with the locals it’s passed. For us that’s 1,394 templates in about two seconds. A first request to a project page then compiles 2 templates instead of 44. 2. Request the pages, signed in A small browser class makes the requests through Rack::MockRequest, with the two cookies a real sign-in sets, then goes back for each page’s lazy Turbo frames: class WarmUp::Browser def initialize(signed_in_as:) @client = Rack::MockRequest.new(Rails.application) @headers = { "HTTP_USER_AGENT" => "Basecamp warm-up", "HTTP_COOKIE" => cookie_for(signed_in_as), "bc3.warm_up" => true } end def visit(path) page = get(path) frames_in(page).each { |id, src| get(src, "HTTP_TURBO_FRAME" => id) } end private def get(path, headers = {}) @client.get("https://#{host}#{path}", @headers.merge(headers)) end def frames_in(page) Nokogiri::HTML5(page.body).css("turbo-frame[src]").map { |frame| [ frame["id"], frame["src"] ] } end end The requests are signed in. The user is a monitoring account we already use for automated checks, and the pages are its own project, Campfire, to-dos, documents and messages. Public pages weren’t enough: after warming up with signed-out pages only, the first signed-in request to the projects page still took 131 ms, because authentication, the signed-in controllers and their views had never run. With signed-in pages it took 40 ms. cookie_for writes the same signed cookie the sign-in controller does, using the app’s own cookie jar, so there’s no API token and no secret to store. The frames are followed. The busiest HTML requests in production aren’t pages at all but Turbo frames — the sidebar badge, the inbox, the navigation menus. The browser parses each page and requests its <turbo-frame src> URLs with the Turbo-Frame header, so those controllers and views get warmed too. Our first four pages turned into 60 requests. The requests are excluded from rate limiting. They are internal, so they do not count against the rate limits that apply to real visitors. 3. Load the rest of the schema The page requests load the schema for the models they touch. The last step loads the rest, from the read replica: ApplicationRecord.reading do models.lazy.take_while { time_left? }.each { |model| model.load_schema if model.table_exists? } end The step checks 261 models and loads any schema information still missing. Those database round trips add up when the primary is far away: outside a request, Active Record uses the writing role, and from a host a long way from the primary each round trip is tens of milliseconds. Reading from the local replica brings the step down from about 20 seconds to 3.5. The pages go first because they load most of the schema anyway. If the time budget runs out, the step stops, logs how many models it got through, and the workers load the rest on first use like they always did. Rails can also load the schema from a dumped cache file at boot (bin/rails db:schema:cache:dump), which would make this step unnecessary. We don’t ship one in our image yet, because the dump needs a database to read from at build time, and we have several databases to cover. It’s on the list. What to close before the fork Running requests in the master opens things the master never opened before, and every worker inherits them. Two processes writing to the same socket will corrupt each other’s traffic, so you need to know what’s open before you fork. The way to find out is to list the master’s open file descriptors — ls -l /proc/<pid>/fd — before and after a warm-up, in an environment set up like production. Development wasn’t enough for us: it stores files on disk, so our S3 connections only showed up in production. Then, for each thing that’s open, check how its library handles a fork. We found three kinds: Already handled. Plenty of libraries detect a fork on their own, either by recording the PID they connected from and reconnecting in the child, by opening per-process files, or by resetting their thread pools. Redis clients, metrics libraries and concurrency libraries tend to be in this group. Check, but you probably don’t need to do anything. Already closed. Database connections are the classic one, and most Puma configs already clear them in before_fork. Anything else that’s opened per process — we have a SQLite cache the workers open on boot — needs closing when the warm-up finishes. Needs a new step. HTTP clients with keep-alive connections are the ones to look for: cloud SDKs with connection pools, tracing exporters, error reporters. They usually have no fork handling at all. We empty the aws-sdk connection pools in before_fork, and we run the warm-up untraced so the OpenTelemetry exporter never opens its connection to Tempo in the first place. Once that’s done, before_fork finishes with Process.warmup, which Ruby 3.3 added for this purpose: a major GC, a heap compaction, and every surviving object promoted to the old generation, so the memory pages the workers share change as little as possible afterwards. Choosing the pages The first list was the four pages that ran the busiest requests on beta. Once the warm-up was live, production showed us which endpoints were still cold. For one deploy, we compared each endpoint’s mean duration in the six minutes after kamal-proxy moved traffic to the new container with the same endpoint an hour later, then multiplied the difference by the number of requests in those six minutes. That gives the extra time each endpoint cost us because it was cold: Endpoint Cold Warm Requests in 6 min Extra seconds Campfire 246 ms 70 ms 6,490 1,140 Projects (JSON API) 84 ms 50 ms 22,077 771 Docs & Files 262 ms 177 ms 4,996 421 To-dos tool 205 ms 113 ms 4,018 371 To-dos (JSON API) 33 ms 16 ms 18,738 320 The pages already in the warm-up showed what to expect: the project page kept a 36 ms gap after a deploy, and the to-do page 10 ms. We’ve proposed adding these five requests, and expect them to add about five to seven seconds to the page step. The two JSON endpoints were a surprise. The warm-up’s page list had no API requests in it, so nothing on the API path had run before the first real request: not the API controllers, and not the Jbuilder templates rendering real records. Precompiling the views covers JSON templates too, but it isn’t a substitute for running the request. Results The warm-up is on for all 68 web hosts. With the first four pages it took 12 to 16 seconds per host: about 2 seconds to precompile the views, 7 to 9 for the 60 requests, and 3.5 for the schema. Deploys take that much longer per host, and we raised the deploy timeout from 30 to 60 seconds to cover it. In Amsterdam, at a peak-hour deploy: During deploy Before After Peak Puma backlog per host 250–2,238 requests 19–223 requests Peak kamal-proxy p99 about 10 s 2.4–4.8 s Peak CPU time per request 201–214 ms 88–132 ms Peak database time per request 56–69 ms 39–47 ms The same eight hosts at three deploys on 1 October, an hour apart, as the warm-up went from one host to four to all eight. The deploy in the middle, with four hosts warmed and four not, shows why every host needed the warm-up. Each warmed host recovered faster on its own: mean request duration peaked at 130 to 173 ms, against 203 to 311 ms on the hosts that weren’t warmed. But the backlog on all eight was about the same, because they were all waiting on the same database. Mean request duration on each host at the 07:21 UTC deploy. Blue hosts warmed up in the master before forking, orange hosts did not. Memory came down too. The workers now share compiled templates, YJIT code and the schema with the master instead of each building their own copy. On beta, the view precompiler alone took a busy worker’s private memory from 174–202 MB to 119–135 MB. Thirty minutes after the deploy, the web containers used about 39 GB less memory than the previous day’s containers at the same age and traffic. Amsterdam served most of our traffic at the times we tested. In Amsterdam, each new container used about 2 GB less just after traffic moved to it, which lowers the peak while the old and new containers overlap. Working with Claude Claude Code helped throughout. It combed through the per-worker backlogs and per-endpoint timings in Prometheus and Loki after each deploy, worked out the cold-versus-warm cost of each endpoint, and prepared the changes and the pull request descriptions with the benchmarks in them. We decided what to try, deployed it and read the results. If you do this Warm the master before it forks. Compile common code and templates and load their schema in the master, so workers inherit that work. With preload_app!, Rack::Builder#warmup runs before the workers start accepting traffic. Use the app’s real requests. Public pages, internal endpoints and synthetic queries warm the paths they run and nothing else. Signed-in requests to real records, frames included, run what production runs. Measure the cold penalty per endpoint. The difference between an endpoint’s cold and warm duration, times its request count after a deploy, ranks the pages worth adding. Ours weren’t the ones we’d have guessed, and two of them were JSON. Check what the warm-up leaves open. List the master’s file descriptors after a warm-up and account for every one before the fork. Two of ours needed changes. Set a time budget. A warm-up that runs long on one slow host fails the deploy on that host. Ours gives the page requests and schema loading a shared 20-second budget, checked before each page or model, puts the most valuable pages first, and logs what it skipped. Reforking from a warm worker, as Pitchfork does, solves the same problem continuously rather than once at boot, and it would warm paths no fixed list of pages covers. We may still get there: our branch brings Instacart’s mold_worker up to date with Puma’s main branch and fixes the bugs we found in it. But warming the master works with the Puma we already run, took a few days to implement, and substantially reduced the queues after deployment.
