A Pika author was editing the same post in two browsers. They added six images in Browser A, then later they referenced an old tab editing the post in Browser B. That copy of the editor did not include those images. Browser B saved and removed all six images from Active Storage. A short time later, Browser A saved its newer copy of the post and put the image references back, but at that point the originally uploaded files were gone due to Browser B saving the post without the images. This exposed two separate problems: Missing images produced uncached 404 responses that the editor requested repeatedly. Action Text purged an image as soon as one save removed it from the body. Here’s how we fixed those problems. Why Action Text deleted the images Action Text keeps a rich text record’s embeds synchronized with the attachments in its body. When a submitted body no longer contains an image, Action Text removes its ActiveStorage::Attachment. Active Storage then purges the dependent blob, deleting both its database record and its file. That behavior makes sense when there is only one current copy of a document. It is unsafe when the same post can be open in multiple tabs, browsers, or devices. One stale save can delete an image that another editor is still using. Why the missing images flooded the server Browser A still had the six images in its editor, each pointing at an Active Storage redirect URL: /rails/active_storage/blobs/redirect/:signed_id/:filename Pika uses the ProseMirror-based Rhino editor. Rhino rebuilt an image’s DOM node when its caption or selection changed. Each new <img> element requested its URL again. Healthy images did not cause trouble since Active Storage makes its redirect cacheable. For a purged blob, Rails returned a 404 without Cache-Control, so the browser kept sending the request to Rails instead of reusing the failure. In a gallery, typing could re-request several missing images per keystroke. In these scenarios, this issue led to hundreds of requests hitting our servers. Fix 1: Cache permanent Active Storage 404s A purged blob’s redirect URL can never start working again. Blob IDs are not reused, and uploading the image again creates a new blob with a new signed URL. Sinc there is no concern of a 404 becoming stale, we made a specific 404 cacheable for the same period as a successful redirect: # app/controllers/concerns/active_storage_missing_blob_cache.rb module ActiveStorageMissingBlobCache extend ActiveSupport::Concern included do rescue_from ActiveRecord::RecordNotFound, with: :render_cacheable_not_found end private def render_cacheable_not_found expires_in ActiveStorage.service_urls_expire_in render file: Rails.public_path.join("404.html"), layout: false, status: :not_found end end We included the concern only in the controller serving these blob redirects: # config/initializers/active_storage_missing_blob_cache.rb Rails.application.config.to_prepare do ActiveStorage::Blobs::RedirectController.include(ActiveStorageMissingBlobCache) end (We also offered an update to Rhino that avoids it being so aggressive re-requesting images.) Fix 2: Detach dropped images instead of purging them When a save removes an image from an Action Text body, we now remove the attachment while temporarily keeping its blob. Another browser can save the image again and reattach that same blob. If nothing brings it back, a cleanup job purges it later. First, we added a timestamp for when the image was dropped: class AddDetachedAtToActiveStorageBlobs < ActiveRecord::Migration[8.1] def change add_column :active_storage_blobs, :detached_at, :datetime end end We then intercepted the private callback Active Storage uses to purge a dependent blob: # config/initializers/detach_dropped_embeds.rb module DetachDroppedEmbeds private def purge_dependent_blob if dropped_from_rich_text? blob&.update_columns(detached_at: Time.now) else super end end def dropped_from_rich_text? record.is_a?(ActionText::RichText) && !destroyed_by_association end end ActiveSupport.on_load(:active_storage_attachment) do prepend DetachDroppedEmbeds end The destroyed_by_association check preserves intentional deletion. Removing an image from a saved body detaches it, but deleting the post and its rich text still purges its images normally. Purge detached blobs after a grace period The cleanup query considers both when the blob was created and when it was detached: ActiveSupport.on_load(:active_storage_blob) do scope :unattached_since, ->(time) { unattached.where( created_at: ..time, detached_at: [nil, ..time] ) } end That scope handles both kinds of unattached blobs: A new upload that has never been attached becomes eligible based on created_at. An image removed from an old post becomes eligible based on detached_at. Pika gives both a 30-day grace period: class PurgeUnattachedUploadsJob < ApplicationJob GRACE_PERIOD = 30.days def perform ActiveStorage::Blob.unattached_since(GRACE_PERIOD.ago).find_each do |blob| blob.purge_later end end end If another browser saves the image during that period, it is attached again and the unattached scope excludes it from cleanup. The result Replaying the original two-browser edit now works safely: Browser B saves a body without the six images. The attachments are removed, but their blobs are only marked as detached. Browser A saves its newer body and reattaches the existing blobs. The images remain available and do not need to be uploaded again. If no editor brings a detached image back, the cleanup job eventually purges it. Much better! Reply by email

Back in March we added Album Search in Album Whale. This was a very nice update because it meant you no longer had to first grab an album share link from a music service (e.g. Spotify, Apple Music, Bandcamp, etc) in order to save an album to a list. You can just head straight to Album Whale and do it all there. Today we’ve added another feature to Album Whale with a similar goal in mind, this time on the other end of the spectrum: now you can listen to an album right in Album Whale! No more needing to jump over to a music service first to do so. Hover over any album artwork and you’ll find a green ▶︎ play button, which opens a dialog containing music service embeds for that album, like this: While amazing for discovering new music quickly, I’ve also found it very useful for my own To Try list. Now this entire flow can happen within Album Whale after hearing of a new album I want to listen to at some point: Add it to my To Try list in Album Whale using search Listen to it right in Album Whale If I like it, copy it to another list in Album Whale (oh ya, this is another small feature we added!) A couple more notes: For albums added to Album Whale using a Bandcamp share link, those green play buttons just link back to Bandcamp. At least for Spotify, I found I needed to be logged into Spotify on the web to have their embeds in Album Whale play the whole album, not just a preview. This might also be true for other music services. We hope you discover many great new tunes this summer! Listen on Album Whale Reply by email

Did you know that we, Good Enough, make a little website called Album Whale where you can make beautiful lists of albums to share with your friends, the world, or just yourself? It’s true! We haven’t made an update there in quite some time (it’s pretty good enough as-is), but I was inspired this week. Adding an album to a list previously required you to first grab an album share link from a music service of your choice (e.g. Spotify, Apple Music, Bandcamp, etc). This has always been a bit cumbersome. I’m excited to share you can now just search for an album right away in Album Whale! Like this: 🔍 💿 🎉 No need to first go somewhere else, you can jump right to Album Whale when you want to save an album to a list. I’ve found this really made my private “To Try” list more useful. Behind-the-scenes we’re using MusicFetch, which supports most of the big music services. We aren’t sure, but there may be some albums that aren’t found via search? If so you can still paste in a link as you always could. Reply by email

Continuous integration is a great thing, and having tests and security checks run before every deploy is also a great thing. But if you’re a developer who has been shipping production code for more than a week, you definitely understand how much it can all feel like a house of cards that tumbles down nearly every day. The Good Enough suite of products have been using GitHub Actions to make sure our automated test suites run before each deployment. The (mostly free) servers GitHub offers are predictably slow, with the Pika test suite generally taking close to ten minutes to run. (To that you say, “Delete most of your system tests!” Alas, due to Pika’s lovely editor, we unfortunately have to maintain quite a few system tests for the service.) Even when upgrading, and paying for, a higher-strength GitHub Action server we were seeing runs approaching eight minutes for Pika. That’s already no fun, but even worse is the fact that our system tests were a bit flaky in the GitHub Actions environment. We eventually got the hint that running system tests in parallel just isn’t possible, but even running them one test at a time would lead to odd failures in part because of how slow things  move in the Actions environment. So imagine the cycle of trying to deploy a Pika update and needing to run continuous integration two, three, or four times. Frustration! There’s got to be a better way! There is. Hopefully. With the arrival of Rails 8.1 came the option to set up local CI. As a team of two wanting to move a little more quickly and with a little less frustration, this seems like a perfect fit. Here’s how I’ve set it up for Pika… ci.rb: # Run using bin/ci CI.run do step "Setup", "bin/setup --skip-server" step "Security: Gem audit", "bin/bundler-audit" step "Security: Brakeman code analysis", "bin/brakeman --quiet --no-pager --exit-on-warn --exit-on-error --confidence-level 2" step "Security: Importmap vulnerability audit", "bin/importmap audit" step "Tests: Rails", "bin/rails test" step "Tests: System", "bin/rails test:system" step "Tests: Seeds", "env RAILS_ENV=test bin/rails db:seed:replant" # Set a green GitHub commit status to unblock PR merge. # Requires the `gh` CLI and `gh extension install basecamp/gh-signoff`. if success? step "Signoff: All systems go. Ready for merge and deploy.", "gh signoff" else failure "Signoff: CI failed. Do not merge or deploy.", "Fix the issues and try again." end end In order for the importmap vulnerability audit to run successfully, I needed to update our gemfile with openssl: group :development, :test do gem "openssl" end Here’s an excerpt of Pika’s application_system_test_case.rb: ENV["PARALLEL_WORKERS"] ||= "1" # System tests seem less flakey when not run in parallel require "test_helper" class ApplicationSystemTestCase < ActionDispatch::SystemTestCase browser_options = Selenium::WebDriver::Chrome::Options.new.tap do |opts| opts.add_argument("--window-size=1200,800") opts.add_argument("--disable-extensions") # Disable non-foreground tabs from getting a lower process priority opts.add_argument("--disable-renderer-backgrounding") # Normally, Chrome will treat a 'foreground' tab instead as backgrounded if the surrounding # window is occluded (aka visually covered) by another window. This flag disables that. opts.add_argument("--disable-backgrounding-occluded-windows") # Suppress all permission prompts by automatically denying them. opts.add_argument("--deny-permission-prompts") opts.add_argument("--enable-automation") end Capybara.register_driver :chrome_headless do |app| browser_options.add_argument("--headless") Capybara::Selenium::Driver.new(app, browser: :chrome, options: browser_options) end Capybara.register_driver :chrome do |app| Capybara::Selenium::Driver.new(app, browser: :chrome, options: browser_options) end if ENV["SYSTEM_TESTS_BROWSER"] driven_by :chrome, screen_size: [ 1200, 1000 ] else driven_by :chrome_headless, screen_size: [ 1200, 1000 ] end end Prerequisites to run local CI: brew install gh gh auth login gh extension install basecamp/gh-signoff Run: gh signoff install This installs the GitHub command-line interface, installs the signoff extension for GitHub command-line, and turns on the signoff requirement in your repo. Here’s the process: Get all your changes pushed to a branch and make a PR Make sure your local environment doesn’t have any lingering file changes or CI will fail Run bin/ci Upon successful completion of local CI, signoff will land on your branch, and you can merge and push to main. If you ever need to move quickly, say in an emergency situation: > gh signoff create -f > git push Since Lettini and I are both super-duper admins in our GitHub account, we needed one more update to protect us from willy-nilly pushing to main. I had to update a setting on GitHub in each repository. I clicked on Do not allow bypassing the above settings in repo > branches > Branch protection rules > main > edit: It’s not all rainbows and unicorns In an ideal world, hands-off CI is a really great thing. It will take a bit for these steps to become muscle memory. I hope they do! System tests are still notoriously flaky, but running tests only in our local environments means we shouldn’t have to account for both general flakiness and super-slow-test-running flakiness. GitHub has a useful feature called Dependabot, which can apply security updates to your dependencies and create a pull request that’s often ready to merge. Sometimes we’ve just clicked that merge button in the past, feeling confident because our test suite had already run in GitHub Actions. Now we’ll have to pull down those branches to go through a local CI and signoff step in order to merge things. If local CI doesn’t end up fitting us, I’ve also discovered there are faster, GitHub-Action-based alternatives for automating CI, such as Blacksmith. These services also have historically been cheaper than increasing server power at GitHub, though recent policy changes at GitHub have changed that math. And a thank you I’d be remiss if I didn’t thank 37signals for opening up their Fizzy repository. This helped me to really streamline our application_system_test_case.rb, which had become a Frankenstein’s monster of a thing as I troubleshot system test issues over the years.