A Niri workspace with 7 visible columns. After having used practically the same xmonad configuration for a decade and a half I’ve now modernized my setup with the scrollable-tiling Wayland compositor Niri. It’s been a bit of a struggle to unlearn my old workflow but I’m really growing to love Niri’s scrollable workflow, especially on my new super ultrawide display. Samsung Odyssey Neo G9 G95NC 57” My new 57” single monitor setup. What kicked off my Niri journey was the purchase of a new super ultrawide monitor. I bought the 57” Odyssey Neo G9 as it was the largest monitor I could find. (It’s marketed as a “gaming” display but it’s really an amazing productivity display.) It replaced my old 3-monitor setup: My old 3-monitor setup. The new display is wider so I had to move the speakers around 10–15cm further apart. I was debating whether to replace the center 31.5” monitor or replace all monitors with a single one but I think I made the right choice with the ultrawide. The curvature wasn’t an issue (I’ve come to prefer it) and the extra vertical space the portrait side monitors provided wasn’t as crucial as I thought. I think an ultrawide is worth it just to get rid of the annoying bezels. Small things can be a big thing sometimes. A more dynamic workflow xmonad and Niri are similar yet different. Both automatically lay out windows as you spawn them but xmonad (at least the way I used it) follows a layout algorithm that re-flows using a “master” window and combines the rest of the windows into one space, while Niri lays out windows in columns. The change is subtle but it implies that a new window won’t change the size of other windows. This is very nice if you spawn a lot of short-lived terminals or web browsers like I do and it reduces the amount of manual reshuffling I spend time on. My xmonad workflow was more static than my Niri one. In xmonad I made heavy use of workspaces, mapping ten workspaces mentally to different programs, such as 0 Firefox and 1 terminal logs on the left monitor; 3, 4 and 5 for different Neovim instances on the center monitor; 8 as chat and 9 for music or video on the right monitor. I had no rules to enforce this; it’s an emergent behaviour that served me well for years. With Niri it’s more dynamic. I still use workspaces but they no longer have direct shortcuts, I simply go up/down in the workspace list. Maybe I’ll add them in the future but with 3–4 workspaces that’s not as necessary. I spawn workspaces/windows when I need them and remove them when I’m done. Usually it’s one workspace per project (yes, I’m now one of those who have multiple up at once) with all the related things such as editor, terminals, and browser with docs. I don’t typically utilize the full screen width and I try to keep the things I’m working on in the center, often leaving 10–30% gaps on the sides. Even though I don’t normally use the “endless scrolling” feature of Niri I re-center selected windows all the time so I can look straight ahead as much as possible. Keyboard shortcuts As a fan of keyboard layouts of course I have to spend some time tinkering with good keyboard shortcuts (especially as Niri’s recommended keybinds don’t map well with my custom keyboard or custom layout). Navigation layer What I did was add a new navigation layer that’s enabled by holding Tab (ring + middle + index on the left-hand side) with all Niri related movement and layout keybinds. In the graphics above, all green-colored keys emit Gui (which gates all window manager commands) and you can see: Long press on Close Window to close a window. The long press requirement prevents accidentally closing windows. Arrows move through columns/windows. Long press resizes them. Workspace Up/Down focuses a different workspace. Center a column. Consume/Expel to combine windows into one column. (consume-or-expel-window-left/consume-or-expel-window-right) Expand Column makes a column take up all remaining space. (expand-column-to-available-width) Audio controls. To press them I release the index finger (keeping the ring and middle finger pressed to keep the layer active) and use the index to press the audio buttons. Mouse buttons. In Niri you can move floating windows with Gui + Left Mouse and Gui + Right Mouse to resize them. As my main mouse is a trackball integrated into the keyboard I had to add them to the left-hand side. I ended up using QMK’s customizable key repress feature that allows me to: Tab combo with my three fingers (layer is active) Release only the index (layer is still active, same as with the audio controls) Press the index again (now detects the press Gui + Left Mouse key down) Use the trackball to move the window And similarly for the right mouse button to resize with the middle finger. Works great! Because there are so many commands I want to send I placed Ctrl on the thumb that provides movement-related commands like so: For example: Arrows move columns/windows in the four directions. Move columns to the neighboring workspaces. Center visible columns. (center-visible-columns) Slightly different consume/expel semantics. (consume-window-into-column/expel-window-from-column) Regular keymaps These are triggered in the “normal” way by first pressing the Super combo and then another key on the base layer (I use autoshift so I shift with a long press). Window management Super + F toggle windowed fullscreen (keep column width) Super + Shift + F fullscreen window (over the entire display) Super + M maximize column (moves other columns) Run stuff Super + Enter terminal Super + E Noctalia’s launcher (also exists on the navigation layer as Launch) Super + S show Noctalia control center Super + Shift + S show Noctalia settings Super + Q power off monitors (they wake on input) Super + Shift + Q show Noctalia session menu (reboot etc) Super + Shift + L lock screen Misc Super + H show hotkey overlay Super + P interactive screenshot Super + Shift + P screenshot selected window Tweaks to the standard CachyOS setup In the process of moving from xmonad to Niri I also moved from Void Linux to CachyOS and I let the installer install Niri and give me a basic configuration together with Noctalia (that provides a statusbar, notifications, and a bunch of things you apparently need). Center the status bar and other Noctalia windows My centered Noctalia status bar. Feels absolutely required on this screen otherwise things end up in the corners. Firefox on XWayland Force Firefox onto XWayland as the Wayland popup manager is broken: environment { MOZ_ENABLE_WAYLAND "0" } Dead keys for Ghostty For some reason dead keys were broken in Ghostty. This is bad for me as the OS keyboard is set to Swedish and it uses them to type ~ (quite a crucial character for a programmer). The fix: environment { GTK_IM_MODULE "ibus" QT_IM_MODULE "ibus" XMODIFIERS "@im=ibus" } This needs ibus installed and running. Melange colorscheme Noctalia discovers custom color schemes under ~/.config/noctalia/colorschemes/<Name>/<Name>.json, so I dropped in my trusty Melange colorscheme there: { "dark": { "mPrimary": "#EBC06D", "mOnPrimary": "#292522", "mSecondary": "#A3A9CE", "mOnSecondary": "#292522", "mTertiary": "#85B695", "mOnTertiary": "#292522", "mError": "#D47766", "mOnError": "#292522", "mSurface": "#292522", "mOnSurface": "#ECE1D7", "mSurfaceVariant": "#34302C", "mOnSurfaceVariant": "#C1A78E", "mOutline": "#867462", "mShadow": "#1a1816", "mHover": "#E49B5D", "mOnHover": "#292522", "terminal": { "normal": { "black": "#867462", "red": "#D47766", "green": "#85B695", "yellow": "#EBC06D", "blue": "#A3A9CE", "magenta": "#CF9BC2", "cyan": "#89B3B6", "white": "#ECE1D7" }, "bright": { "black": "#34302C", "red": "#BD8183", "green": "#78997A", "yellow": "#E49B5D", "blue": "#7F91B2", "magenta": "#B380B0", "cyan": "#7B9695", "white": "#C1A78E" }, "foreground": "#ECE1D7", "background": "#292522", "selectionFg": "#C1A78E", "selectionBg": "#403A36", "cursorText": "#292522", "cursor": "#EBC06D" } } } Then pick the colorscheme: "colorSchemes": { "darkMode": true, "predefinedScheme": "Melange", "useWallpaperColors": false } Layout appearance The default appearance was pretty I admit but way too much blank space and weirdness. Some tweaks: layout { // Required for noctalia-shell to set wallpaper background-color "transparent" // Never auto-center focused columns (too much movement) center-focused-column "never" // But do center a single window always-center-single-column // No extra space around it all struts {} // No gaps between windows gaps 0 // The focus ring was annoying focus-ring { off } // Use a border with consistent width for all windows instead border { on width 2 active-color "#ebc06d" inactive-color "#403a36" } // Setting widths is important with such a large screen preset-column-widths { proportion 0.15 proportion 0.3 proportion 0.4 } default-column-width { proportion 0.15; } // Heights too, why not? preset-window-heights { proportion 0.15 proportion 0.5 proportion 1.0 } } // Prevent the mouse from opening the overview in the corners gestures { hot-corners { off } } Keep windows centered Niri has the always-center-single-column option, which is nice as I want to keep as much as possible in the center of the monitor when I’m working. But I very frequently use 2–3 smaller windows and with my frequent opening and closing I’d like them centered too. Luckily, Niri has an IPC you can use to make a small program that reacts to events and does this for you. I made a small rust project using the niri-ipc crate that does this for me: The autocenter implementation [dependencies] niri-ipc = "26.4.0" use std::collections::HashMap; use std::io; use niri_ipc::socket::Socket; use niri_ipc::{Action, Event, Request, Response, Window, Workspace}; #[derive(Clone, Copy, PartialEq, Eq, Hash)] struct WindowId(u64); #[derive(Clone, Copy, PartialEq, Eq)] struct WorkspaceId(u64); struct OutputName<'a>(&'a str); struct WindowState { workspace: Option<WorkspaceId>, width: f64, } fn main() -> io::Result<()> { let mut socket = Socket::connect()?; if !matches!(socket.send(Request::EventStream)?, Ok(Response::Handled)) { eprintln!("niri rejected event stream"); std::process::exit(1); } let mut known: HashMap<WindowId, WindowState> = HashMap::new(); let mut read_event = socket.read_events(); loop { let result = match read_event()? { // A full snapshot of the current state. Just refresh our state. Event::WindowsChanged { windows } => { known = windows .into_iter() .map(|w| { ( WindowId(w.id), WindowState { workspace: w.workspace_id.map(WorkspaceId), width: w.layout.tile_size.0, }, ) }) .collect(); Ok(()) } Event::WindowOpenedOrChanged { window } => { let workspace = window.workspace_id.map(WorkspaceId); let entry = WindowState { workspace, width: window.layout.tile_size.0, }; let prev = known.insert(WindowId(window.id), entry); match prev { // Don't center floats. _ if window.is_floating => Ok(()), // New window, try to re-center. None => maybe_center_new(&window), // Window changed workspace, try to re-center. Some(state) if state.workspace != workspace => center_focused_if_fits(), // Skip other things. Some(_) => Ok(()), } } Event::WindowClosed { id } => { if known.remove(&WindowId(id)).is_some() { center_focused_if_fits() } else { Ok(()) } } Event::WindowLayoutsChanged { changes } => { // Only re-center if the width was changed, otherwise our re-center will // loop back indefinitely. let mut resized = false; for (id, layout) in changes { if let Some(state) = known.get_mut(&WindowId(id)) { if (state.width - layout.tile_size.0).abs() > 0.5 { state.width = layout.tile_size.0; resized = true; } } } if resized { center_focused_if_fits() } else { Ok(()) } } _ => Ok(()), }; if let Err(e) = result { eprintln!("autocenter: {e}"); } } } /// Center a newly created window if the workspace is focused and if there's surrounding free space left. fn maybe_center_new(window: &Window) -> io::Result<()> { let Some(workspace_id) = window.workspace_id.map(WorkspaceId) else { return Ok(()); }; let Some(focused) = focused_workspace()? else { return Ok(()); }; if WorkspaceId(focused.id) == workspace_id { center_if_fits(&focused)?; } Ok(()) } /// Center windows in the focused workspace if there's surrounding free space left. fn center_focused_if_fits() -> io::Result<()> { if let Some(focused) = focused_workspace()? { center_if_fits(&focused)?; } Ok(()) } /// Center windows in the workspace if there's surrounding free space left. fn center_if_fits(workspace: &Workspace) -> io::Result<()> { let Some(output) = workspace.output.as_deref().map(OutputName) else { return Ok(()); }; let Some(width) = output_width(output)? else { return Ok(()); }; if workspace_width(WorkspaceId(workspace.id))? < f64::from(width) { center_visible_columns()?; } Ok(()) } /// Issue a one-shot query to Niri, wait, and return the response. fn query(request: Request) -> io::Result<Response> { match Socket::connect()?.send(request)? { Ok(response) => Ok(response), Err(msg) => Err(io::Error::other(msg)), } } /// Get the focused workspace. fn focused_workspace() -> io::Result<Option<Workspace>> { match query(Request::Workspaces)? { Response::Workspaces(ws) => Ok(ws.into_iter().find(|w| w.is_focused)), _ => Ok(None), } } /// Get the width of an output (monitor). fn output_width(name: OutputName<'_>) -> io::Result<Option<u32>> { match query(Request::Outputs)? { Response::Outputs(outputs) => Ok(outputs .get(name.0) .and_then(|o| o.logical.as_ref()) .map(|l| l.width)), _ => Ok(None), } } /// Calculates the width of all columns in the workspace. fn workspace_width(workspace_id: WorkspaceId) -> io::Result<f64> { let Response::Windows(windows) = query(Request::Windows)? else { return Ok(0.0); }; let mut columns: HashMap<usize, f64> = HashMap::new(); for w in windows { if w.workspace_id.map(WorkspaceId) != Some(workspace_id) { continue; } if let Some((col, _)) = w.layout.pos_in_scrolling_layout { let width = columns.entry(col).or_insert(0.0); *width = width.max(w.layout.tile_size.0); } } Ok(columns.values().sum()) } /// Send a command to center the visible columns. fn center_visible_columns() -> io::Result<()> { if let Err(msg) = Socket::connect()?.send(Request::Action(Action::CenterVisibleColumns {}))? { eprintln!("center-visible-columns rejected: {msg}"); } Ok(()) } One catch is that if a new window overflows the monitor width, the script won’t center the columns even if there would be free space left afterwards. This is a little weird but it’s consistent with Niri’s center-visible-columns command. I had a small itch to try to hack around it but in the end I left it alone… Is Niri worth it? Yes, absolutely. Niri has been a huge upgrade for me in combination with a single wide screen. My xmonad setup worked really well with three monitors—arguably a better fit in that context than Niri—but for the big-screen use-case Niri is superior. I’m curious how it holds up on my laptop, once I gather enough energy to install CachyOS on it… But that’s a side quest. The big-screen setup I spend most of my days in is the best I’ve ever had, and I have no desire to go back.

Accessing services via raw IP addresses isn’t that swell; I’m no Rain Man. It’s time to set up subdomains for my hietala.xyz domain for internal use. We’ll use the Gateway API to set up routes, cert-manager to give us https without self-signed browser warnings, and ExternalDNS to set up DNS overrides. HTTP routing I believe the flow of resolving http://argocd.hietala.xyz to a service looks something like this: digraph { rankdir=LR "Browser" -> "OPNsense DNS" [label="resolve host"] "OPNsense DNS" -> "Browser" [label="Gateway IP"] "Browser" -> "Cilium Gateway" [label="HTTP request" class="label-below"] "Cilium Gateway" -> "HTTPRoute" [label="match hostname" class="label-below"] "HTTPRoute" -> "Service" [label="backendRef" class="label-below"] "Service" -> "Pod" [label="EndpointSlice" class="label-below"] } First the browser asks my router running OPNsense about argocd.hietala.xyz and gets the Gateway IP address. The request then flows through a route (http or https), to a Service, and eventually a Pod where ArgoCD is running. We’ll handle these one at a time but let’s start by creating the Gateway and giving it an IP (we’ll use 10.1.4.101): apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: main namespace: kube-system annotations: io.cilium/lb-ipam-ips: "10.1.4.101" spec: gatewayClassName: cilium We installed the prerequisites for Gateway when we configured Cilium during our initial cluster bootstrap. Then we need to tell the Gateway to manage all http routes for all namespaces (we’ll get back to https): # Continued from the above Gateway manifest spec: gatewayClassName: cilium listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All Then we can add an HTTPRoute for our ArgoCD application that targets argocd-server at port 80: # Below the ArgoCD Application setup # Resources in the same file are separated by `---` --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: argocd namespace: argocd spec: parentRefs: - name: main namespace: kube-system sectionName: http hostnames: - argocd.hietala.xyz rules: - backendRefs: - name: argocd-server port: 80 If we then add a DNS override from http://argocd.hietala.xyz to 10.1.4.101 (/etc/hosts or OPNsense or similar) then we should be able to reach http://argocd.hietala.xyz. Enabling SSL I want https://argocd.hietala.xyz to “just work” and for that we need to tell Gateway to manage https routes: # ... spec: gatewayClassName: cilium listeners: - name: https port: 443 protocol: HTTPS tls: mode: Terminate certificateRefs: - name: hietala-xyz-tls allowedRoutes: namespaces: from: All - name: http # Http definition from before (Note the tls addition that terminates using a not-yet-defined certificate.) And add the https route itself: apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: argocd namespace: argocd spec: parentRefs: - name: main namespace: kube-system sectionName: https hostnames: - argocd.hietala.xyz rules: - backendRefs: - name: argocd-server port: 80 This doesn’t work just yet as we need to create the hietala-xyz-tls cert. Cert-manager cert-manager seems like the standard way to manage certificates for Kubernetes. I don’t want to expose my services to the internet which means I need a DNS01 challenge. cert-manager doesn’t natively support Namecheap (bummer) but there’s an open source webhook Namecheap provider out there. It hasn’t been updated in a couple of years but I couldn’t find an alternative… YOLO I guess? The cert-manager manifest: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://charts.jetstack.io chart: cert-manager targetRevision: v1.21.0 helm: values: | installCRDs: true destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true Note the installCRDs: true that makes things easier for us, and CreateNamespace=true which will create the cert-manager namespace for us too. Saves some typing. We also set sync-wave to -1 as it needs to sync before the cluster issuer and certificate that we’ll define later. Then the namecheap webhook: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager-webhook-namecheap namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/kelvie/cert-manager-webhook-namecheap.git targetRevision: HEAD path: deploy/cert-manager-webhook-namecheap helm: # Identifier that our issuer will use values: | groupName: acme.namecheap.com destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: syncOptions: - CreateNamespace=false automated: prune: true selfHeal: true Then a ClusterIssuer that uses the namecheap webhook: apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging annotations: # Must come after the cert-manager applications. # It defaults to 0 anyway but this is more explicit. argocd.argoproj.io/sync-wave: "0" # If resources doesn't exist ArgoCD may complain. argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: acme: # Use the staging endpoint during testing! server: https://acme-staging-v02.api.letsencrypt.org/directory email: [email protected] privateKeySecretRef: name: letsencrypt-staging solvers: - dns01: # Use the namecheap webhook webhook: # Same identifier the webhook defined above. groupName: acme.namecheap.com solverName: namecheap config: # These are very sensitive! # Store them in a Sealed Secret apiKeySecretRef: name: namecheap-credentials key: apiKey apiUserSecretRef: name: namecheap-credentials key: apiUser Make sure to use the staging issuer during testing to avoid rate limits. When you’re done playing around you can switch to the production server at https://acme-v02.api.letsencrypt.org/directory. I created a new ClusterIssuer called letsencrypt-prod instead of replacing the url so I can easily change between them if I need to. Create the sealed secret: kubectl create secret generic namecheap-credentials \ --namespace cert-manager \ --from-literal=apiKey="key" \ --from-literal=apiUser="user" \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem \ --format yaml \ > gitops/apps/cert-manager/namecheap-secret.yaml Finally we need to create the hietala-xyz-tls certificate that uses the issuer: apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: hietala-xyz # Must match the namespace of the Gateway namespace: kube-system annotations: # References the issuer so must be synced after. argocd.argoproj.io/sync-wave: "2" argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: # Gateway references this certificate using this name. secretName: hietala-xyz-tls dnsNames: - "*.hietala.xyz" - "hietala.xyz" issuerRef: # Switch to `letsencrypt-prod` later. name: letsencrypt-staging kind: ClusterIssuer When all this has been synced we should be able to see that the certificate is created: $ kubectl get certificate -n kube-system NAME READY SECRET AGE hietala-xyz True hietala-xyz-tls 4d20h And that we can visit https://argocd.hietala.xyz (browser will warn while we use letsencrypt-staging, on prod it should be without errors). (If not, then you have a bunch of debugging to do. Have fun!) Want to redirect http to https? Add this route: apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: argocd-redirect namespace: argocd spec: parentRefs: - name: main namespace: kube-system sectionName: http hostnames: - argocd.hietala.xyz rules: - filters: - type: RequestRedirect requestRedirect: scheme: https statusCode: 301 In general though I skip the http route as I tell Firefox to always use https. ArgoCD is still accessible via its external IP I defined in a previous post but you can also port-forward or proxy to it: # http://localhost:8001/api/v1/namespaces/argocd/services/argocd-server:80/proxy/ kubectl proxy # localhost:8123 kubectl port-forward svc/argocd-server -n argocd 8123:80 Automating DNS overrides By now most of the things are set up in proper GitOps fashion but there’s still one thing I have to do manually: I have to add a DNS override to unbound (it’s on my OPNsense router). Doing it once is fine but it gets old fast. Some of my overrides in unbound. Can you see a pattern? A wildcard domain could work but I have other services running outside of Kubernetes, so I’d like a cleaner solution. That solution is ExternalDNS, which automatically adds overrides for any existing Gateway HTTPRoute. There’s a webhook provider for OPNsense that we’ll use. The manifest: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: external-dns namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default sources: - repoURL: https://kubernetes-sigs.github.io/external-dns/ chart: external-dns targetRevision: 1.21.1 helm: valueFiles: - $values/gitops/apps/external-dns/values.yaml - repoURL: https://git.hietala.xyz/tree/home-ops.git targetRevision: HEAD ref: values destination: server: https://kubernetes.default.svc namespace: external-dns syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true It loads values.yaml from the repo: provider: name: webhook webhook: image: repository: ghcr.io/crutonjohn/external-dns-opnsense-webhook tag: v1.0.0 env: - name: OPNSENSE_HOST value: "https://router.hietala.xyz" # Remember to create the `external-dns` sealed secret. - name: OPNSENSE_API_KEY valueFrom: secretKeyRef: name: external-dns key: api-key - name: OPNSENSE_API_SECRET valueFrom: secretKeyRef: name: external-dns key: api-secret sources: - gateway-httproute policy: sync domainFilters: - hietala.xyz txtOwnerId: talos-dorne This connects to OPNsense, sources routes from the Gateway, targets the hietala.xyz domain, and uses a new external-dns secret: kubectl create secret generic external-dns \ --namespace external-dns \ --from-literal=api-key="key" \ --from-literal=api-secret="secret" \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem \ --format yaml \ > gitops/apps/external-dns/external-dns-secret.yaml With this I don’t have to add manual overrides anymore. Nothing like spending hours to automate a few minutes of work! If you want to extend support to other network things, say a Minecraft server talking TCP over 25565 with a fixed IP, we can add service support to ExternalDNS: sources: - gateway-httproute - service And declare the Minecraft Service like so: apiVersion: v1 kind: Service metadata: name: minecraft annotations: external-dns.alpha.kubernetes.io/hostname: mc.hietala.xyz spec: type: ClusterIP externalIPs: - 10.1.4.53 selector: app: minecraft ports: - port: 25565 targetPort: 25565 protocol: TCP name: minecraft Many steps but the end is nice There’s a few moving parts but once done supporting new apps is satisfyingly easy. For example, to expose the homeassistant service under https://ha.hietala.xyz this is enough: apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: homeassistant spec: parentRefs: - name: main namespace: kube-system sectionName: https hostnames: - ha.hietala.xyz rules: - backendRefs: - name: homeassistant port: 8123 Commit and push, and https://ha.hietala.xyz is ready in a jiffy, certificates and DNS overrides included. I encountered a bug in ArgoCD v3.3.5 where it’ll get stuck syncing the routes with extra lines in the diff (or mark them as out of sync), for example here: - backendRefs: - name: my-app port: 80 group: "" kind: Service weight: 1 matches: - path: type: PathPrefix value: / You can add the lines to each route to fix it but you can also tell ArgoCD to ignore them with these lines in the Application template: apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet spec: template: spec: ignoreDifferences: - group: gateway.networking.k8s.io kind: HTTPRoute jqPathExpressions: - .spec.parentRefs[].group - .spec.parentRefs[].kind - .spec.rules[].backendRefs[].group - .spec.rules[].backendRefs[].kind - .spec.rules[].backendRefs[].weight - .spec.rules[].matches syncPolicy: syncOptions: - ServerSideApply=true - RespectIgnoreDifferences=true

Now we’re getting to the fun stuff: GitOps. The act of pushing beautifully crafted .yaml files and seeing your Kubernetes cluster get red and stall out is surely what life’s all about. Install ArgoCD We have our Kubernetes cluster and now we’re going to bootstrap ArgoCD so it starts syncing from our git repository. Using helm: helm install argocd \ --repo https://argoproj.github.io/argo-helm \ argo-cd \ --namespace argocd \ --create-namespace \ --version 10.1.4 ArgoCD will by default generate an admin password and you can get it with the argocd tool: # You can use the password to login to the web. argocd admin initial-password -n argocd # Make ArgoCD available at localhost:8123 kubectl port-forward -n argocd svc/argocd-server 8123:80 You can use the argocd CLI to manage ArgoCD but I’m going to prefer the declarative way. Fixed IP You can use the port forward method above to access ArgoCD but having a fixed IP is nicer, especially if you want to use the argocd CLI (I only used it when trying to get it all working). I use a separate service for the IP, which allows me to assign an IP outside of the Cilium load balance pool we setup previously (10.1.4.101–10.1.4.255). It looks like this: apiVersion: v1 kind: Service metadata: name: argocd-server-ip namespace: argocd spec: type: ClusterIP externalIPs: - 10.1.4.51 selector: app.kubernetes.io/name: argocd-server ports: - name: http port: 80 targetPort: 8080 It simply routes 10.1.4.51:80 to the IP of argocd-server and port 8080. We need to apply it: kubectl apply -f gitops/bootstrap/argocd.yaml And you should see it assigned and able to visit it on the web: $ kubectl get svc -n argocd argocd-server-ip NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE argocd-server-ip ClusterIP 10.105.209.116 10.1.4.51 80/TCP 3d18h Repository I use a self-hosted Forgejo instance (hosted in a Proxmox LXC) but you can use whatever you want. You need an API token with read permissions. ArgoCD documents a declarative setup that we’ll use. For repositories we need to create a secret: kubectl create secret generic home-ops-repo \ --namespace argocd \ --from-literal=type=git \ --from-literal=url=<repo-url> \ --from-literal=username=<username> \ --from-literal=password=<token> \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \ > gitops/bootstrap/argocd-repo-secret.yaml We also need to add the repository label to the generated file so it looks something like this: apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: home-ops-repo namespace: argocd spec: encryptedData: password: ... type: ... url: ... username: ... template: metadata: name: home-ops-repo namespace: argocd labels: argocd.argoproj.io/secret-type: repository Apply it: kubectl apply -f gitops/bootstrap/argocd-repo-secret.yaml And the repo should show up in the UI or argocd repo list. Initial admin password Instead of letting ArgoCD generate a random password we can set the initial password explicitly. It’s possible to do it either via the Helm chart or the declarative way, which I’ll continue with. After generating it with kubeseal the secret should look like this: apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: argocd-secret namespace: argocd spec: encryptedData: admin.password: ... admin.passwordMtime: ... server.secretkey: ... template: metadata: labels: app.kubernetes.io/name: argocd-secret app.kubernetes.io/part-of: argocd name: argocd-secret namespace: argocd With three keys: password is a bcrypt hash: argocd account bcrypt --password 'supersecret' passwordMtime is the plaintext modification time: date -u +"%Y-%m-%dT%H:%M:%SZ" secretkey is a random value: openssl rand -base64 48 ArgoCD uses server.secretkey to sign session and API tokens, and it will regenerate the key if restarted. I kept getting logged out when I was rebuilding things all over the place and it was annoying so I added this to make it stable. A file structure ArgoCD is now installed but at this point I think it’s good to take a step back before we steam ahead. One of the benefits of ArgoCD is that you can create any kind of file structure and organize the repository any way you want. That’s also one of its problems: if you can do anything it’s hard to decide on a plan forward. I found the blog post How to Structure Your Argo CD Repositories Using Application Sets illuminating and even though their setup is overkill for my homelab, I simplified their ideas into something that works for me. The basic idea is to have all applications in their own folders under gitops/apps, and place all bootstrap stuff under gitops/bootstrap. Borrowing the “level” terminology from the above post here’s the basic idea: digraph { rankdir=TB "root.yaml" [class="loader"] "appset.yaml" [class="loader"] bootstrap_more [label="…"] apps_more [label="…"] "root.yaml" -> "cilium_config.yaml" "root.yaml" -> "argocd.yaml" "root.yaml" -> "appset.yaml" "root.yaml" -> bootstrap_more "appset.yaml" -> "authentik" "appset.yaml" -> "actualbudget" "appset.yaml" -> "miniflux" "appset.yaml" -> "homeassistant" "appset.yaml" -> apps_more // Keep the "…" boxes on the far right of each level. { rank=same; "cilium_config.yaml" -> "argocd.yaml" -> "appset.yaml" -> bootstrap_more [style=invis] } { rank=same; "authentik" -> "actualbudget" -> "miniflux" -> "homeassistant" -> apps_more [style=invis] } } Which loads in three phases: root.yaml Everything under bootstrap/ gets loaded bootstrap/appset.yaml in turn loads all folders under apps/ This is what the directory structure looks like: gitops ├── root.yaml # Level 1 ├── bootstrap # Level 2 │ ├── cilium_config.yaml │ ├── argocd.yaml │ ├── appset.yaml │ └── ... └── apps # Level 3 ├── authentik │ └── ... ├── miniflux │ └── ... └── ... I’m considering separating infrastructure and user workloads, but for now I’m fine with having everything in the apps/ folder. App-of-apps root.yaml will be a master “app-of-apps” as it will be an Application that spawns other applications. It looks like this: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: all-apps namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-100" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://git.hietala.xyz/tree/home-ops.git targetRevision: master path: gitops/bootstrap destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true (You’re going to see lots of yaml files like this, better get used to it.) The important bits are The name all-apps in the argocd namespace It should be synced before everyone else (sync-wave: -100) Should load everything under gitops/bootstrap from my home-ops repo I’ll be using pruning and self-healing to tell ArgoCD to keep everything in sync. With selfHeal: true ArgoCD will revert any manual kubectl changes to a managed resource. This is the “proper GitOps” way as you must go through git but in practice it might be annoying. If you turn it off the ArgoCD will track the difference but won’t fight you. This is the entrypoint and it’s the last manual apply command we’ll use, as it’ll load everything else: kubectl apply -f gitops/root.yaml Application sets Inside the bootstrap/ folder lives, among other things, bootstrap/appset.yaml which is responsible for turning subfolders under apps/ into applications and loading them. This is what I use and it’s an adaption of the code from the aforementioned blog post: apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: appset namespace: argocd spec: # Use Go's engine so we can use basename/path goTemplate: true # The engine silently fails on missing keys by default goTemplateOptions: ["missingkey=error"] # Generate from all subfolders from my repo under apps/ generators: - git: repoURL: https://git.hietala.xyz/tree/home-ops.git revision: master directories: - path: gitops/apps/* template: # The Application name will be folder + "-app" # Creative, I know. metadata: name: "{{.path.basename}}-app" spec: project: default # Describes the manifests of the application, # which is everything under apps/<app-folder>/ source: repoURL: https://git.hietala.xyz/tree/home-ops.git targetRevision: master path: "{{.path.path}}" # Give the application its own namespace destination: server: https://kubernetes.default.svc namespace: "{{.path.basename}}" syncPolicy: syncOptions: - CreateNamespace=true # Needed for larger manifests - ServerSideApply=true # ArgoCD does a dry run to check for resource kinds # it doesn't yet recognize. This breaks some things # so turn it off. - SkipDryRunOnMissingResource=true automated: prune: true selfHeal: true I hope the comments make it clear how the above works. The end result is that everything under apps/ will be loaded and organized under its own application. Other bootstrap things Manifests under bootstrap/ Even though we manually apply some manifests during the bootstrap sequence, they can still be managed by ArgoCD after the fact. This is solved by simply placing the relevant files in bootstrap/, and they’ll be managed going forward. These are what we’ve collected so far: Cilium We installed Cilium using a helm install command. Here’s a (hopefully) corresponding example of a manifest that ArgoCD can manage: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cilium namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://helm.cilium.io chart: cilium targetRevision: 1.19.2 helm: values: | kubeProxyReplacement: true k8sServiceHost: 10.1.4.10 k8sServicePort: 6443 l2announcements: enabled: true externalIPs: enabled: true gatewayAPI: enabled: true ipam: mode: kubernetes operator: replicas: 1 securityContext: privileged: true destination: server: https://kubernetes.default.svc namespace: kube-system syncPolicy: syncOptions: - CreateNamespace=false - ServerSideApply=true automated: prune: true selfHeal: true The duplication here is unfortunate but I couldn’t find a way around it. At least we can reuse cilium_config.yaml with the load balancer IP specifications that we created during cluster setup. Just plop it under bootstrap/ and we can move on. Gateway API Before installing Cilium we also applied the Gateway CRDs using this command: kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml This is the corresponding YAML manifest (smaller as we don’t pass dozens of options): apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: gateway-api namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-2" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/kubernetes-sigs/gateway-api.git targetRevision: v1.2.1 path: config/crd/standard destination: server: https://kubernetes.default.svc namespace: kube-system syncPolicy: automated: prune: true selfHeal: true Note that I specify sync-wave -2 for the Gateway and -1 for Cilium because the Gateway should sync before Cilium. Sealed Secrets We continue in the same manner of converting Helm installations to manifests: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: sealed-secrets namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://bitnami-labs.github.io/sealed-secrets chart: sealed-secrets targetRevision: 2.16.2 helm: values: | fullnameOverride: sealed-secrets-controller destination: server: https://kubernetes.default.svc namespace: sealed-secrets syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true ArgoCD In addition to the secrets used for the declarative setup (repository, admin password) ArgoCD can also manage itself: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: argocd namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://argoproj.github.io/argo-helm chart: argo-cd targetRevision: 10.1.4 helm: values: | configs: secret: createSecret: false destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: syncOptions: - CreateNamespace=false automated: prune: true selfHeal: true --- # Static IP as shown previously, `---` separates different kinds Updated bootstrap sequence This is the updated bootstrap recipe that I manage with Just: [doc("Bootstrap everything from zero")] full: just bootstrap::cluster just bootstrap::cilium just secrets::restore_sealed_secrets_private_key just bootstrap::sealed_secrets just bootstrap::argocd [doc("Bootstrap ArgoCD")] [working-directory('../gitops')] argocd: helm install argocd \ --repo https://argoproj.github.io/argo-helm \ argo-cd \ --namespace argocd \ --create-namespace \ --version 10.1.4 \ --set configs.secret.createSecret=false # We don't let ArgoCD generate the secret, we supply one: kubectl apply -f bootstrap/argocd-secret.yaml # Needed to be able to sync from repo. kubectl apply -f bootstrap/argocd-repo-secret.yaml # Wait for ArgoCD to deploy. kubectl rollout status deployment/argocd-server -n argocd --timeout=300s # Technically not needed as it'll get loaded on sync, this just speeds it up a little. kubectl apply -f bootstrap/argocd.yaml --force-conflicts --server-side # Sync everything else from the repo. kubectl apply -f root.yaml Renovate A pull request generated by the Renovate bot (upping its own dependency). To complete our GitOps setup and to exercise our newfound capabilities let’s setup the Renovate service. Its purpose is to monitor git repositories and to create pull requests where it updates dependencies, see the above image for how it’s created a PR to update itself. The singularity is here any second now. To get Renovate setup we need to create three things: The secret Renovate will pick up RENOVATE_TOKEN in secrets, generated like so: kubectl create secret generic renovate-token -n renovate \ --from-literal=RENOVATE_TOKEN='<token>' \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \ > gitops/apps/renovate/renovate-secret.yaml We’ll later add the name renovate-token to the main manifest. The configuration file Renovate has a bunch of configurations you can add. I’m good with the defaults except to modify the ArgoCD settings to support the custom directory structure I’m using: { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": ["config:recommended"], "argocd": { "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"] }, "kubernetes": { "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"] } } This activates the argocd and kubernetes managers, where argocd watches ArgoCD specific manifests (Application/ApplicationSet) and kubernetes watches plain manifests (such as container images in Deployment). The manifest The Application manifest is similar to the ones we’ve seen before: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: renovate namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://docs.renovatebot.com/helm-charts chart: renovate targetRevision: 46.236.3 helm: values: | existingSecret: renovate-token cronjob: schedule: "0 1 * * *" renovate: config: | { "platform": "forgejo", "endpoint": "https://git.hietala.xyz", "repositories": ["tree/home-ops"], "gitAuthor": "Renovate Bot <[email protected]>", "automerge": false } destination: server: https://kubernetes.default.svc namespace: renovate syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true Note that we configure the secret (renovate-token), how often it should run, and the repository information. Commit and push and it should be ready to go. ArgoCD has a really good UI where you can see what’s happening, if you made a mistake somewhere, and also trigger the cron job manually. Or you can use kubectl, for example: # See the health status of all applications kubectl get applications -A # Trigger the cron job kubectl create job -n renovate --from=cronjob/renovate renovate-manual # Watch the logs kubectl logs -n renovate -l job-name=renovate-manual -f With this we’ve got our first example of managing a Kubernetes service via ArgoCD. We’ll continue to see more examples of this as the series continues.

The Kubernetes iceberg. If I’d have to describe my homelab setup via analogy I guess it would be similar to me on a unicycle carrying plates with both of my hands, or maybe a leaking barrel with water that I try to patch up with silver tape. I’ve also been Kubernetes-curious so I decided to completely redesign my homelab, centered around Kubernetes. It was a bit painful but at least it fulfilled my need for procrastination very well. Overarching goals I’ve got three goals with the setup: Declarative, reproducible, and automated The big goal is to have everything declarative in a single git repository and to easily be able to bootstrap from nothing to a fully working setup. I want to use Infrastructure as Code to create the Kubernetes cluster and GitOps to populate it with all my services automatically from the repo. It should be really easy to make a change; I want to move away from having to ssh into the correct repo and manually do stuff. Backups, backups, backups While a proper GitOps setup means that infrastructure and configuration files are inherently backed up, a proper backup setup is still crucial. Ask me how I know. I haven’t had a proper (as in working) backup solution for years and this time I should have it from the start. Documentation What if I could document my setup, so future me has a chance to understand what’s happening? Writing documentation is boring, so I’ll write some blog posts instead. I’m a bit skeptical that I can fulfill all three goals, but if I manage 2/3 or even 1/3 it’s still a big win compared to my old setup. Kubernetes, too complicated? It’s a fair question and the most common critique towards Kubernetes is that’s just too complicated (especially for a homelab). Discussions online are filled with comments such as: Kubernetes has to be most complex software I’ve ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. trinovantes “Let’s use Kubernetes!” Itamar Turner-Trauring So why would I choose Kubernetes? Because, for whatever reason, Kubernetes is very popular and for every comment complaining about complexity you have comments extolling it’s virtues: I was skeptical about Kubernetes but I now understand why it’s popular. The alternatives are all based on kludgy shell/Python scripts or proprietary cloud products. zelly Kubernetes is the biggest quality-of-life improvement I’ve experienced in my career anchochilis Having experienced the single machine docker-compose deployments, kludgy shell scripts, and proprietary cloud products; I think I need to use Kubernetes myself to be able to form an opinion on it. And in some ways, isn’t experimentation a core part homelabbing? Tech stack There are many valid tech choices for this kind of setup and many of them are reasonable. I don’t know if my choices are reasonable—most were chosen because they sounded cool, others because I just picked one. Here’s list of some of the choices I made, which we’ll setup in this series: Talos Linux for Kubernetes nodes. The coolest way to run Kubernetes. Lightweight and secure, what’s not to like? Terraform to provision VMs on Proxmox and to initialize Talos Linux. Cilium for proxying, CNI, load balancer, and Gateway API provider. I opted for Cilium as it’s one dependency replacing several alternatives (such as kube-proxy, Metallb, and Traefik, which I was leaning towards at first). Gateway API is the new thing you “should” use instead of ingress, and I wanted to try it out. ArgoCD for GitOps. If it was purely for myself FluxCD might have been the better, simpler, choice but we might use ArgoCD at work and I don’t want to deal with two separate systems at the moment. Renovate to keep dependencies up-to-date. CloudNativePG for Postgres on Kubernetes. I’ll also setup timescaledb, although we won’t use it in this series. It’s just to prepare for the future migration of long-term statistics from Home Assistant. Longhorn on NVMEs for persistent storage. Data is backed up using VolSync and Restic. Sanoid, Syncoid and Kopia for backup archive management. Backups are snapshotted and stored in ZFS, which are also encrypted and shipped off-site to Backblaze for storage in the cloud. Backups from Longhorn and Postgres arrives to ZFS via Garage, a self-hosted S3 service. Authentik as an identity provider and single-sign-on platform. It’s nice to not have to login manually everywhere. Huh. Displayed like this it looks like a lot, but fear not! It’ll be worth it in the end. In the next part we’ll start by creating VMs and getting a Kubernetes cluster up and running.