Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
13

How to Actually Integrate Angular and Nestjs

from These Yaks Ain't Gonna Shave Themselves [alt+shift+b] in programming

I don’t know who needs to hear this. But your frontend and backend systems don’t need to be completely separate. I started anew side project recently. You know, one of things that allows me to tinker with new technology but will probably never be finished. I’m using Angular for the frontend and Nestjs for the backend. All good. But then I go to do something that I thought was very normal and common and run into a wall. I want to integrate the two frameworks. I want to serve my initial html with nestjs and add script tags so that Angular takes over the frontend. This will allow me to do dynamic things on the backend and frontend however I want. But also deploy the system all as one cohesive product. Apparently this is no longer How Things Are Done. I literally could not find documentation on how to do this. When you read the docs and blog posts, everybody expects you to just have two systems that run entirely independently. Here’s the server for your backend and here’s the entirely different server for your frontend. Hashtag winning! When I google for “integrate angular and nestjs”, nobody knows what I’m talking about. On the surface, this seems like is a great technical blog post from LogRocket. It says “I will teach you how. First, set up two separate servers…” I think I know why the community has ended up in this place. But that’s a rant for another blog post. Let me try to explain what I’m talking about. Angular is designed as an a frontend framework (let’s set aside SSR for now). The primary output of an Angular build is javascript and css files that are meant to run in the browser. When you run ng build, you’ll get a set of files put into your output folder. Usually the folder is dist/<your_project_name>. Let’s look at what’s in there. polotek $> ls -la dist/my-angular-project -rw-r--r-- 1 polotek staff 12K Sep 13 14:15 3rdpartylicenses.txt -rw-r--r-- 1 polotek staff 948B Sep 13 14:15 favicon.ico -rw-r--r-- 1 polotek staff 573B Sep 13 14:15...
15th Sep 2023

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from These Yaks Ain't Gonna Shave Themselves

Running Pull Requests in Github Codespaces

Earlier this week, I was having a lot of trouble understanding how to get Dev Containers working. Dev Containers are the required way to enable Github Codespaces, which was my actual goal. I finally got to something that works, so I want to document what I learned. Dev Containers is a standard developed my Microsoft for using docker containers to run dev environments for a project. I use Docker for most of my projects these days. I think it’s a great tool both for simplifying local development and for packaging and deploying my projects. In the current project, I already have a Dockerfile and a docker-compose.yml file. I deploy to fly.io which will also run my docker build to deploy the project. This works fine for me, and I had never used Dev Containers. It seemed overly complicated (and I was right about that). I want to give more of the context about why I’ve been pursuing github codespaces, because I think it’s relevant. But if you wanna skip to the tips for getting Dev Containers and Codespaces working, feel free. (I’m also going to talk positively about using LLMs to code, so if that makes you want to skip too, there’s no hard feelings). Background Here’s the idea. I have a solo project that I’m working on, and instead of hacking away on the main branch, I’m using a full feature branch and pull request workflow to keep my changes organized. This might seem like overkill, but it’s helpful for me because I tend to bounce around and leave things half done. I can run things locally and test different things by switching between branches. But I’m also doing a lot more AI-assisted development. Locally I use Claude Code. But I’ve been experimenting with allowing Github Copilot to create PRs. I create a github issue, assign it to copilot, and it spins up in the cloud and does work. Afterwards it opens a pull request for me to review. So even though this is a “solo” project, I end up with pull requests that I didn’t write myself and I need to validate. Giving tasks to copilot in the cloud is cool for a couple of reasons: It’s better than setting an LLM lose on my laptop unsupervised. I’m still working on how to use them safely. When I’m using them locally, I babysit diligently and approve every command it wants to run. But this gets tedious, and that can only lead to cutting corners. With github codespaces, I can make it github’s problem. I get to work on the project even if I can’t sit in front of my computer. When I get excited about a project, I tend to think about it constantly. It’s tough to compartmentalize when I’m having ideas but I’ve got other things to do. The workflow of creating issues and assigning them to copilot helps me move forward even in-between work sessions. I plan to give more of my thoughts on AI-assisted development in a future post. The result is I’ve got PRs from copilot that I need to review. I can look at the diffs, but we know that even if it “looks good to me”,LLMs can and do make subtle mistakes and require extra scrutiny. What I want to do is actually spin up the app with the changes and kick the tires. Github codespaces seemed perfect for this. But it took a lot of trial and error to get to something that works for me. Leaning on docker and docker compose We need to take a slight detour before diving into Dev Containers. We will be using docker and docker compose to help us bypass some of the complexity. So this post assumes that you have a working setup using docker-compose.yml. I’m not going to go into detail about that here. If you’re interested, maybe I can do a future blog post. Your docker compose setup should also include a database if you need one. Most apps like mine require a database to connect to. There are lots of ways you can go about this locally. In fact, if you’ve got a local setup using docker compose, you’ve probably already figured this out. But there will also need to be a database available once we upload to a github codespaces environment. So for that purpose, this post also assumes that you have a database container alongside your app container. It usually looks something like this for me (stripped down for brevity). services: app: image: my-app container_name: my-app build: context: . dockerfile: Dockerfile ports: - "3000:3000" env_file: - .env postgres: image: postgres:17-alpine container_name: my-app-db environment: POSTGRES_DB: my_app POSTGRES_USER: postgres POSTGRES_PASSWORD: password ports: - "5432:5432" volumes: - ./backend/data/db:/var/lib/postgresql/data Note: There are a lot of other details to configuring a non-trivial docker compose environment. That’s out of the scope of this post. Sorry. This creates an app that uses my project files to build a docker image. The database is pulled from an existing docker image that runs postgres. The resulting app can reference postgres as the hostname of the database. Docker automatically creates an internal network where the two containers can see each other. Because we’re going to have Dev Containers read our docker-compose.yml from the start, this should work as expected. Getting a Dev Container working This walkthrough assumes you have a few prerequisites already installed: vs code Dev Containers extension for vs code Docker Note: The Dev Containers extension uses your local docker system to build and run containers. If you run into early issues, check that docker is running and that vs code has permission to access it. Reading the docs for Dev Containers seemed straightforward enough. I used the vs code worflow to create a devcontainer.json file. But nothing worked out of the box. It came with it’s own docker-compose.yml file. It wanted me to use a standard Microsoft docker image for the environment rather than the one specified in my own Dockerfile. I’m pretty sure we’re mostly expected to run their pre-built templates for dev containers. You pick the one that seems to match your project most closely, and it’ll probably work by just injecting your code into it. But if you want to be more opinionated about your build, you’re gonna have a bad time. Eventually, I abandoned trying to figure this out and instead configured the dev container to just use the existing docker setup I aleady had. Open your command palette and select “Dev Containers: Add Dev Container Configuration Files…”. You might run through a couple of options like whether to add the dev container files to the workspace or the user config. I chose workspace. Then you’ll be prompted to choose what approach you want to take to create the dev container configuration. You’ll see the option to create it from a predefined template. But what you want to do is select the From 'Dockerfile' option or the From 'docker-compose.yml' option. The trick is that these options only appear if these files are already present in your project. When I tried to recreate what I did for this blog post, I got tripped up by that. If you wanna get started with this before you tackle the docker stuff, you can just touch Dockerfile docker-compose.yml in your project root and that’s enough to enable the menu items. However, the setup may error at some point if there’s nothing in the files. Creating a Dev Container config through vs code The second confusing thing is it asks you to “chose a service”. There is very little context here for what service to choose and why. You’ll see this prompt, because you have to pick one of the services to be the “primary” one for the dev container. Say your docker compose file specifies multple containers, like a web app, a database, and maybe a background worker. You probably want to choose whichever one is the main app. So for example, I selected app, which is the name in my docker-compose.yml that runs the node server. When you run through these setup prompts, you end up with a devcontainer.json that is pretty different from the one you see from the standard tutorials. Instead of having a bunch of setup for your project in the config file, it just references your docker-compose.yml file. // For format details, see https://aka.ms/devcontainer.json. For config options, see the // README at: https://github.com/devcontainers/templates/tree/main/src/docker-existing-docker-compose { "name": "Project name", // Update the 'dockerComposeFile' list if you have more compose files or use different names. // The .devcontainer/docker-compose.yml file contains any overrides you need/want to make. "dockerComposeFile": [ "../docker-compose.yml", "docker-compose.yml" ], // The 'service' property is the name of the service for the container that VS Code should // use. Update this value and .devcontainer/docker-compose.yml to the real service name. "service": "app", ... Notice that you’ll see two docker-compose.yml files. The first one is your existing file. Assuming it’s in the root of your project, it gets referenced from inside the .devcontainer folder. The second one is a custom override created by the dev container setup. It lives inside the .devcontainer folder. Leave that as-is. From here, you can add “features” to the dev container and any other things you might want from the setup wizard. For example, I added the PostgreSQL client so that I can use it to connect to my database from inside the dev container. Finally, you’ll choose to expose any ports that your app exposes. These might be added automatically if it exists in your Dockerfiles. Here is the full version of the devcontainer.json that currently works for my project: // For format details, see https://aka.ms/devcontainer.json. For config options, see the // README at: https://github.com/devcontainers/templates/tree/main/src/docker-existing-docker-compose { "name": "Project name", // Update the 'dockerComposeFile' list if you have more compose files or use different names. // The .devcontainer/docker-compose.yml file contains any overrides you need/want to make. "dockerComposeFile": [ "../docker-compose.yml", "docker-compose.yml" ], // The 'service' property is the name of the service for the container that VS Code should // use. Update this value and .devcontainer/docker-compose.yml to the real service name. "service": "app", // The optional 'workspaceFolder' property is the path VS Code should open by default when // connected. This is typically a file mount in .devcontainer/docker-compose.yml "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}", // Features to add to the dev container. More info: https://containers.dev/features. "features": { "ghcr.io/devcontainers/features/aws-cli:1": {}, "ghcr.io/devcontainers/features/github-cli:1": {}, "ghcr.io/devcontainers/features/node:1": {}, "ghcr.io/devcontainers-extra/features/ripgrep:1": {}, "ghcr.io/robbert229/devcontainer-features/postgresql-client:1": {} }, // Use 'forwardPorts' to make a list of ports inside the container available locally. "forwardPorts": [ 3000 ] // Uncomment the next line if you want start specific services in your Docker Compose config. // "runServices": [], // Uncomment the next line if you want to keep your containers running after VS Code shuts down. // "shutdownAction": "none", // Uncomment the next line to run commands after the container is created. // "postCreateCommand": "cat /etc/os-release", // Configure tool-specific properties. // "customizations": {}, // Uncomment to connect as an existing user other than the container default. More info: https://aka.ms/dev-containers-non-root. // "remoteUser": "devcontainer" } Note: As of this writing, it looks like you can only get Node version 22 as a feature in the official Microsoft images. If you need other versions of node, you can get them from third-party community builds. From here you should be able to tell vs code to Open Workspace in Container. You’ll see it spin up and you’ll see a bunch of logs. If it’s working properly, you’ll see it build an image from your dockerfile and then spin up the containers. Eventually the vs code terminal will open, and you’ll be dropped into a bash prompt inside your dev container. Success! But obviously it’s not gonna work the first time. I had to debug several issues first. Required files should be optional Because I was only using docker compose for local development up to this point, I had my local .env file referenced in the docker-compose.yml under the env_file property. services: app: image: my-app container_name: my-app build: context: . dockerfile: Dockerfile ports: - "3000:3000" env_file: - backend/.env The problem is that by default, this file is required. If it’s not present, docker compose will error. When you go to build the dev container, this file may not be present. Probably because you put it in .gitignore so you didn’t accidently check it into version control. This is the right thing. So the fix here is to use the extended format to specify that the env_file is optional. services: app: image: my-app container_name: my-app build: context: . dockerfile: Dockerfile ports: - "3000:3000" env_file: - path: backend/.env required: false Adding your env variables back to the build Because Dev Containers won’t have your .env file, you’ll need to add the environment variables directly to the docker-compose.yml. Yes this is redundant, but I don’t know a cleaner alternative. Make sure you’re only adding standard env variables and not secrets. Secrets will go elsewhere, but they’re fine in the .env for now. According to the docker compose docs, you can have both env_file and environment specified in your config, but entries in environment will always take precedence. So you can have your .env file for local development and the environment entries for when it gets pushed to github. services: app: image: my-app container_name: my-app build: context: . dockerfile: Dockerfile ports: - "3000:3000" environment: PORT: "3000" AWS_REGION: us-west-2 S3_BUCKET_NAME: bucket-dev SENTRY_DSN: "..." env_file: - path: backend/.env required: false Note: You can also add environment variables directly to the devcontainer.json. But I didn’t try this and YMMV. Build args are different from environment variables My project uses Hono for the backend. The frontend uses the Vue framework and gets built for production using Vite. If you’ve dealt with Vite, you know it has a specific workflow for how to get environment variables into your frontend. When you prefix your environment variables with VITE_, they get built into your frontend bundle at build time. For example if you use S3_BUCKET_NAME on the backend, you have to use VITE_S3_BUCKET_NAME in frontend code. You may be more familiar with NEXT_S3_BUCKET_NAME in Nextjs. Same deal. When you use vite locally, it knows how to find the build variables in your .env file, and it just works. But when you’re using docker, these variables need to be present when the Dockerfile is being built. This can be a pain to figure out even without the added complexity of Dev Containers. I’ll just give you the answer here. The short version is that you reference the the vite variables as docker “build args”, then you inject them into the environment during the docker build. ARG VITE_S3_BUCKET_NAME ENV VITE_S3_BUCKET_NAME=$VITE_S3_BUCKET_NAME RUN npm run build ... Then you need to add these build args to your docker-compose.yml in the build section so they get passed into the Dockerfile execution. Yes this is redundant, but I don’t know a cleaner alternative. And once again, make sure you’re only adding standard build variables and not secrets. services: app: image: my-app container_name: my-app build: context: . dockerfile: Dockerfile args: VITE_S3_BUCKET_NAME: "bucket-dev" VITE_SENTRY_DSN: "..." ports: - "3000:3000" environment: PORT: "3000" AWS_REGION: us-west-2 S3_BUCKET_NAME: bucket-dev SENTRY_DSN: "..." env_file: - path: backend/.env required: false Note: You can also add build args manually using the cli. docker build --build-arg "VITE_S3_BUCKET_NAME=..." There may be other changes you need to make to your docker compose setup. But these are the major issues I ran into that related to Dev Containers specifically. If you’re lucky, you should be able to spin up a dev container for your project locally and try it out. Next we have to get this to work with github codespaces. Which of course requires even more steps. Note: If your dev container spins up, but your app doesn’t work, make sure the app can connect to the database, and make you’ve actually populated the database. It’s probably still empty! Getting Your dev Container working in Github Codespaces Once you’ve got a dev container working locally, you can check all of the config files into github and push it. This includes the .devcontainer folder and any changes you had to make to Dockerfile and docker-compose.yml. I did this in a branch at first. Mostly because I wasn’t confident enough yet to drop it into main. But also because I was explicitly looking for this to work on pull requests. Creating a codespace Github will recognize your .devcontainer folder in your project and allow you to create a codespace. But if you’re working in a private repo, you may need to make sure to enable codespaces first. Then you can go to your repo page and look for codespaces under the green “Code” button on the top right. Click to create a codespace, and make sure you select the appropriate branch that you want to use. The codespace will clone that git branch into the environment before running the build. The Codespaces docs do a decent job of introducing this stuff, so I won’t go into detail here. We’ve done most of the heavy lifting with the local Dev Container setup. Where to create new codespaces Adding secrets You can try to spin up a codespace in github, but it probably won’t quite work yet. Codespaces works almost like using Dev Containers locally. It can find your environment variables in your docker-compose.yml files. But it can’t find any secrets. Those have to be injected by github. Codespaces has it’s own separate space for variables and secrets. See this screenshot of the settings menu. Where to add secrets in github settings Once you’ve got the secrets in, try rebuilding your codespace to pick them up. Running the app When using docker containers, you usually specify a default command to run when the container starts up. This is usually where you start your web server. By default, Dev Containers overrides your command. Instead they add a simple command that just sleeps and keeps the codespace up and running so you can connect to it. What this means is that your app is not running by default. You’ll have to go into the codespace terminal and run it. For me it’s just an npm script. npm run start This is important because you want github to create a unique url for you to access your app from the web browser. That doesn’t happen until the codespace detects that a port is being used. You can leave your terminal running with your app, or you can put it in the background and manage it with something like systemd. Depends on how sophisticated you wanna get. Wrapping up Hooray! Hopefully you’ve successfully got a github codespace running your app. Github will give you a unique, generated url to access it. And you should be able to spin one up against any branch or pull request. I hope this saves someone some headaches when getting Github Codespaces working. Once you get it going, it’s very cool and useful. Unfortunately I don’t have any advice for getting this working if you don’t already have your own docker setup. But the docker approach isn’t well documented in the searches I did, so this is my contribution to fixing that.

23rd Oct 2025 • 24 votes
Getting A Local Mastodon Setup In Docker

This is the first in probably a series of posts as I dig into the technical aspects of mastodon. My goal is to get a better understanding of the design of ActivityPub and how mastodon itself is designed to use ActivityPub. Eventually I want to learn enough to maybe do some hacking and create some of the experiences I want that mastodon doesn’t support today. The first milestone is just getting a mastodon instance set up on my laptop. I’m gonna give some background and context. If you want to skip straight to the meat of things, here’s an anchor link. Some background Mastodon is a complex application with lots of moving parts. For now, all I want is to get something running so I can poke at it. Docker should be a great tool for this. Because a lot of that complexity can be packaged up in pre-built images. I tried several times using using the official docs and various other alternative projects to get a working mastodon instance in docker. But I kept running into problems that were hard to understand and harder to resolve. I have a lot to learn about all the various pieces of mastodon and how they fit together. But I understand docker pretty well. So after some experimenting, I was able to get an instance running on my own. The rest of this post will be dedicated to explaining what I did and what I learned along the way. One final note. I know many folks work hard to write docs and provide an out of the box dev experience that works. This isn’t meant to dismiss that hard work. It just didn’t work for me. I’m certainly going to share this experience with the mastodon team. Hopefully these lessons can make the experience better for others in the future. The approach Here’s the outline of what we’re doing. We’re going to use a modified version of the docker-compose.yml that comes in the official mastodon repo. It doesn’t work out of the box. So I had to make some heavy tweaks. As of this writing, the mastodon docs seem to want people to use an alternate setup based on Dev Containers. I found that very confusing, and it didn’t work for me at all. Once we have all of the docker images we need, all of the headaches are in configuring them to work together. Most of mastodon is a ruby on rails app with a database. But there is also a node app to handle streaming updates, redis for caching and background jobs, and we need to handle file storage. We will do the minimum configuration to get all of that set up and able to talk to each other. There is also support for sending emails and optional search capabilities. These are not required just to get something working, so we’ll ignore them for now. It’s also worth noting that if we want to develop code in mastodon, we need to put our rails app in development mode. That introduces another layer of headaches and errors that I haven’t figured out yet. So that will be a later milestone. For now, all of this will be in “production” mode by default. That’s how the docker image comes packaged. Keep it simple. There are still many assumptions here. I am running on Mac OS with Apple Silicon (M3). If you’re trying this out, you may run into different issues depending your environment. Pre-requisites We need docker. And a relatively new version. The first thing I did was ditch the version: 3 specifier in the docker-compose.yml. Using versions in these files is deprecated, and we can use some newer features of docker compose. I have v4.30.0 of Docker Desktop for Mac. We also need caddy. Mastodon instances require a domain in most cases. This is mostly about identity and security. It would be bad if an actor on mastodon could change their identity very easily just by pretending to be a different domain or account. There are ways around this, but I couldn’t get any of them to work for me. That complicates our setup. Because we can’t just use localhost in the browser. We need a domain, which means we also need HTTPS support. Modern browsers require it by default unless you jump through a bunch of hoops. Caddy gives us all of that out of the box really easily. It will be the only thing running outside of docker. There’s only one caveat with caddy. The way that it is able to do ssl termination so easily is that it creates its own certificates on the fly. The way it does this is by installing it’s own root cert on your machine. You’ll have to give it permission by putting in your laptop password the first time you run caddy. If that makes you nervous, feel free to skip this and use whatever solution you’re comfortable with for SSL termination. But as far as I know, you need this part. Choose a domain for your local instance. For me it was polotek-social.local. Something that mkes it obvious that this is not a real online instance. Add an entry to your /etc/hosts and point this to localhost. Or whatever people have to do on Windows these days. Let’s run a mastodon I put all of my changes in my fork of the official mastodon repo. You can clone this branch and follow along. All of the commands assume you are in the root directory of the cloned repo. https://github.com/polotek/mastodon/tree/polotek-docker-build > git clone [email protected]:polotek/mastodon.git > cd mastodon > git co -b polotek-docker-build I rewrote the docker section of the README.md to outline the new instructions. I’m going to walk through my explanation of the changes. Pull docker images This is the easiest part. All of the docker images are prepackaged. Even the rails app. You can use the docker compose command to pull them all. It’ll take a minute or 2. > docker compose pull Setup config files We’re using a couple of config files. The repo comes with .env.production.sample. This is a nice way to outline the minimum configuration that is required. You can copy that to .env.production and everything is already set up to look for that file. The only thing you have to do here is update the LOCAL_DOMAIN field. This should be the same as the domain you chose and put in your /etc/hosts. You can put all of your configuration in this file. But I found it more convenient to separate out the various secrets. These often need to be changed or regenerated. I wrote a script to make that repeatable. Any secrets go in .env.secrets. We’ll come back to how you get those values in a bit. I had to make some other fixes here. Because we’re using docker, we need to update how the rails app finds the other dependencies. The default values seem to assume that redis and postgres are reachable locally on the same machine. I had to change those values to match the docker setup. The REDIS_HOST is redis, and the DB_HOST is db. Because that’s what they are named in the docker-compose file. Diff of config file on github The rest of the changes are just disabling non-essential services like elastic search and s3 storage. Generate secrets We need just a handful of config fields that are randomly generated and considered sensitive. Rails makes it easy to generate secrets. But run the required commands through docker and getting them in the right place is left as an exercise for the reader. I added a small script that runs these commands and outputs the right fields. Rather than try to edit the .env.production file in the right places everytime secrets get regenerated, I think it’s much easier to have them in a separate file. Fortunately, docker-compose allows us to specify multiple files to fill out the environment variables. Diff of config file on github This was a nice quality of life change. And now regenerated secrets and making them available is just one command. > bin/gen_secrets > .env.secrets Any additional secrets can be added by just updating this script. For example, I use 1password to store lots of things, even for development. And I can pull things out using their cli named op. Here’s how I configured the email secrets with the credentials from my mailgun account. # Email echo SMTP_LOGIN=$(op read "op://Dev/Mailgun SMTP/username") echo SMTP_PASSWORD=$(op read "op://Dev/Mailgun SMTP/password") Run the database Running the database is easy. > docker compose up db -d You’ll need to have your database running while you run these next steps. The -d flag will run it in the background so you can get your terminal back. I often prefer to skip the -d and run multiple terminal windows. That way I can know at a glance if something is running or not. But do whatever feels good. The only note here is to explain another small change to docker-compose to get this running. We’re using a docker image that comes ready to run postgres. This is great because it removes a lot of the fuss of running a database. The image also provides some convenient ways to configure the name of the database and the primary user account. This becomes important because maston preconfigures these values for rails. We can see this in the default .env.production values. DB_USER=mastodon DB_NAME=mastodon_production The database name is not a big issue. Rails will create a database with that name if it doesn’t exist. But it will not create the user (maybe there’s a non-standard flag you can set?). We have to make sure postgres already recognizes a user with the name mastodon. That’s easy enough to do by passing these as environment variables to the database container only. Diff of config file on github Load the database schema One thing that’s always a pain when running rails in docker. Rails won’t start successfully until you load the schema into the database and seed it with the minimal data. This is easy to do if you can run the rake tasks locally. You can’t run the rake tasks until you have a properly configured rails. And it’s hard to figure out if your rails is configured properly because it won’t run without the database. I don’t know what this is supposed to look like to a seasoned rails expert. But for me it’s always a matter of getting the db:setup rake task to run successfully at least once. After that, everything else starts making sense. However, how do you get this to work in our docker setup? We can’t just do docker compose up, because the rails container will fail. We can’t use docker compose exec because that expects to attach to an existing instance. So the best thing to do is run a one-off container that only runs the rake task. The way to achieve that with docker compose is docker compose run --rm. The rm flags just makes sure the container gets trashed afterwards. Because we’re running our own command instead of the default one, we don’t want it hanging around and potentially muddying the waters. Once we know the magic incantation, we can setup the database. > docker compose run --rm web bundle exec rails db:setup Note: Usually you don’t put quotes around the whole command. For some reason, this can cause problems in certain cases. You can put quotes around any individual arguments if you need to. Run rails and sidekiq If you’ve gotten through all of the steps above, you’re ready to run the whole shebang. > docker compose up This will start all of the other necessary containers, including rails and sidekiq. Everything should be able to recognize and connect to postgres and redis. We’re in the home stretch. But if you try to reach rails directly in your browser by going to https://localhost:3000, you’ll get this cryptic error. ERROR -- : [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: localhost:3000 It took me a while to track this down. It’s a nice security feature built into rails. When running in production, you need to configure a whitelist of domains that rails will run under. If it receives request headers that don’t match those domains, it produces this error. This prevents certain attacks like dns rebinding. (Which I also learned about at the same time) If you set RAILS_ENV=development, then localhost is added to the whitelist by default. That’s convenient, and what we would expect from dev mode. But remember we’re not running in development mode quite yet. So this is a problem for us. The nice thing is that mastodon has added a domain to the whitelist already. Whatever value you put in the LOCAL_DOMAIN field is recognized by rails. (In fact, if you just set this to localhost you might be good to go. Shoutout to Ben.) However, when you use an actual domain, then most modern web browsers force you to use HTTPS. This is another generally nice security feature that is getting in our way right now. So we need a way to use our LOCAL_DOMAIN, terminate SSL, and then proxy the request to the rails server running inside docker. That brings us to the last piece of the puzzle. Running caddy outside of docker. Run a reverse proxy The configuration for caddy is very basic. We put in our domain, we put in two reverse proxy entries. One for rails and one for the streaming server provided by node.js. Assuming you don’t need anything fancy, caddy provides SSL termination out of the box with no additional configuration. # Caddyfile polotek-social.local reverse_proxy :3000 reverse_proxy /api/v1/streaming/* :4000 We put this in a file named Caddyfile in the root of our mastodon project, then in a new terminal window, start caddy. > caddy run Success? If everything has gone as planned, you should be able to put your local mastodon domain in your browser and see the frontpage of mastodon! Mastodon frontpage running under local domain! In the future, I’ll be looking at how to get actual accounts set up and how to see what we can see under the hood of mastodon. I’m sure I’ll work to make all of this more developement friendly to work with. But I learned a lot about mastodon just by getting this to run. I hope some of these changes can be contributed back to the main project in the future. Or at least serve as lessons that can be incorporated. I’d like to see it be easier for more people to get mastodon set up and start poking around.

2nd Jun 2024 • 14 votes
How to Build Fly Locally With Docker for Mac

In my recent side projet, I’ve been deploying to fly.io and really enjoying it. It’s fairly easy to get setup. And it supports my preferred workflow of deploying my changes early and often. I have run into a few snags though. Fly.io builds your project into a docker image and deploys containers for you. That process is mostly seamless when it works. But sometimes it fails, and you need to debug. By default, fly builds your docker images in the cloud. This is convient and preferred most of the time. But when I wanted to test some changes to my build, I wanted to try building locally using Docker Desktop. This should be easy. The fly cli is quite nice. And there is a flag to build locally. fly deploy --build-only --local-only This failed saying it couldn’t find Docker. > fly deploy --build-only --local-only ==> Verifying app config Validating /Users/polotek/src/harembase/fly.toml Platform: machines ✓ Configuration is valid --> Verified app config ==> Building image Error: failed to fetch an image or build from source: docker is unavailable to build the deployment image I spent quite a bit of time googling for the problem here. You can also run fly doctor --verbose to get some info. (If you run this in your fly app folder, it will show more info not relevant to this topic.) > fly doctor --verbose Testing authentication token... PASSED Testing flyctl agent... PASSED Testing local Docker instance... Nope (We got: failed pinging docker instance: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?) This is fine, we'll use a remote builder. Pinging WireGuard gateway (give us a sec)... PASSED No app provided; skipping app specific checks I found various forum posts discussing this problem. The folks at fly have spent a lot of time investigating some deep technical issues. I appreciate that work, but ultimately none of it seems to reflect my problem. And the issue felt simpler to me. Fly couldn’t find docker. Why not? Where is it looking? Eventually I found the answer on stackoverflow. It turns out that things have settled pretty recently to a basic config setting. By default, Docker Desktop installs the socket for the daemon in a non-global space. Usually in your personal user folder, e.g. ~/.docker/run/docker.sock. But other tools expect the docker daemon socket to be available in a standard location, e.g. /var/run/docker.sock As of this writing, Docker Deskstop has added a recommended way to enable the standard location. In the Docker Desktop dashboard, got to Settings > Advanced and enable “Allow the default Docker socket to be used”. Docker for Mac settings screen This will require your system password and restart. Then you should be able to see the docker socket in the standard place. And fly will be able to see it! Hopefully the next person who’s banging their head against this will have an easier time.

2nd Oct 2023 • 18 votes

More in programming

AI Homelab Part 4: Lowering the floor for Scout

Helping Scout focus and remember

21 hours ago • 2 votes
The Explorer Pattern for product invention and development.

tl;dr: the Explorer Pattern is one individual or team building a functional implementation that minimizes cross-team and cross-functional impacts as much as humanly possible, and then reconciling the details with other teams after that prototype is completed, accepting that this approach can require very substantial rework to that initial development. Why write this? Imprint previously called this the “Tech-led” or “Engineering-led” development model, but it’s pretty unclear to folks what that means. This writeup provides a more defined term, the “Explorer Pattern”, and explains in more detail how this works. Why do we need to adjust the way we work? Before 2026: Code is expensive, and we design process to minimize implementation and rework In and after 2026: Writing software is extremely cheap, but it’s a wider phenomenon: generating output is effectively-free across many dimensions. Code is nearly free — if we have appropriate constraints. Designs are nearly free — if they follow patterns. Analysis is nearly free — if we’ve instrumented things properly Reworking software is extremely cheap. Historically industry has optimized around avoiding engineering rework because engineering was the highest cost / narrowest constraint. In many companies this was interpreted as writing code being the constraint, which was never true, but many common practices were designed as if that were true (e.g. bringing engineering into problem-solving late “to protect their time”) Working on the system is much more effective than historically if we get automated integration and feedback loops right, e.g. not improving a one-off design but improving the design system. Any place where improving the higher-leverage process is slower than solving the one-off is a bug in our process Explorer Pattern: primary-and-replicas metaphor The easiest metaphor to think about the Explorer Pattern is database replication. Databases like MySQL and PostgreSQL have a primary database, where writes are performed. Every write is verified at that point, and once it’s verified, you know that the writes will flow downstream to various secondary replicas. Those downstream replicas can be delayed but they can’t be wrong because writes are verified upstream. (I’ve resisted the desire to talk about the CAP theorem.) A different model for database design is that used by Dynamo or Cassandra (or MongoDB to a certain extent), where you have a replica set, and have to get an acknowledgement from the majority of the replica set for a write to be strongly consistent. This is a very reliable setup, because it means you can tolerate losing a node, but it also introduces inconsistencies, and on a fixed amount of hardware they tend to be much, much slower than running a primary-secondary setup (or rely on bad performance metrics that don’t fsync to disk to create the illusion of throughput). The Explorer Pattern is the former primary-secondary model, where the vast majority of the work is done by the working individual or team. This makes all work internally consistent because it’s passed through the same small group. It doesn’t, however, mean that the work is perfect. For example, you can always insert data that meets the table’s requirements but is still inherently wrong (e.g. 200 is a valid integer, but it doesn’t mean that my credit limit was supposed to be 200, that depends on the surrounding application logic). Other teams become aware of the changes downstream, not synchronously upfront, and it’s instead the working group’s responsibility to both rationalize approach and create visibility into their work for other teams to engage in a non-blocking fashion. Example: Support for Financial Aggregators A concrete example of this practice at Imprint was rolling out financial aggregator support, e.g. allowing folks to connect to their Imprint powered accounts via Plaid. This has always been something folks want, but it’s particularly important in the realm of agentic commerce where more folks’ agents are analyzing their financial state for them. For this project, one of our engineering leaders (Ron), built the first pass of the feature at a low-burn while understanding the shape of the problem in a private alpha. He chatted through some of the details periodically to keep other teams aware, but it was a bit of a side project. However, once he found the full shape of the problem and had implemented a full solution, we cut the replica-lag across teams down to near-zero, identified and ran through the small number of remaining issues, and shipped it. Like many of our projects this year, it was never officially on a roadmap, and shipped a quality feature that our members have wanted for some time. How to follow the Explorer pattern The rough rules for following the Explorer pattern: Know your goal and sponsor: every initiative ultimately has to have a specific outcome it’s pursuing and a sponsor who’s bankrolling that pursuit. This doesn’t always need to be explicit, especially for smaller projects, but once a project goes beyond ~3 weeks of effort, if you can’t very explicitly identify who the stakeholder is sponsoring your project and they would not enthusiastically agree, then you are likely closer to “lost” than “exploring” Small working group is small: at most 5 people, and preferably smaller. Everyone in the group has to keep the entire project in their head for as long as possible to detect problems and opportunities Small working group is flexible: folks in small working groups have to “find a way to yes” rather than “find a way to no”. Exploring is about uncovering the fundamental shape of a problem rather than pretending we know the fundamental shape Publish progress in a public channel: explorers have to report home with the progress they are making so others are aware. This is both a Linear Project and a Slack channel. Make the narrowest viable changes to what already exists: when we introduce unnecessary changes, we invalidate others’ mental models about how the world works. When we haven’t made those changes with care, we’re not only invalidating others’ mental model, we are creating the expectation that we’ve done it very intentionally for a good reason, which has a chilling effect on others improving them later, which is very expensive, whereas the Explorer pattern depends on being as inexpensive as possible Exercise good judgment to identify reasonable hypotheses: discard bad ideas immediately, and then filter the rest by trying to implement them as quickly as possible. Once you see the shape of the implementation, you can figure out which of the remainders are interesting Hold assumptions constant to avoid getting stuck: the only way to navigate complex ambiguous problems is to hold things constant, even when you know they are imperfect. For example, to do headcount planning we have to know our revenue, to know our revenue we have to forecast it, to forecast it we have to know the day that partners will go live next year. We don’t truly know any of those things, so just anchoring on a plausible guess to unblock us is much more valuable than trying to get them accurate. A little prework is useful, but a lot is just artificial precision that’s a distraction from progress Hold space for being totally wrong: when we move fast, with low information, we always find the way out, but often it’s not the way out we anticipated when we got started. Being open to throwing it all away, and starting a second, third, fourth time Ship all the way to production behind a feature flag: features and products move fast when we get the pieces working every step from frontend to backend to data pulls. This is the first step of exploration. Just make sure it’s behind a feature flag that’s only enabled for internal folks!

yesterday • 2 votes
Value Creation vs Job Preservation

Comparing two different ways of work and working environments.

yesterday • 2 votes
Making a dumb TV just smart enough

I own an old, cheap TV. It has no Internet connection, no fancy smart features that make you want to install a Pi-hole, and no ask-me-again-later updates. However, it is connected to my own desktop computer through an HDMI cable, so I can watch movies and TV series from the couch. Setting it up used […] The post Making a dumb TV just smart enough appeared first on Quentin Santos.

3 days ago • 1 votes
Bikepacking across Kyushu

Five days and 540km on a bike across Kyushu, without a laptop. Notes on the peace of riding all day, convenience store dinners, and Japan's emptying countryside.

5 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in