A Niri workspace with 7 visible columns. After having used practically the same xmonad configuration for a decade and a half I’ve now modernized my setup with the scrollable-tiling Wayland compositor Niri. It’s been a bit of a struggle to unlearn my old workflow but I’m really growing to love Niri’s scrollable workflow, especially on my new super ultrawide display. Samsung Odyssey Neo G9 G95NC 57” My new 57” single monitor setup. What kicked off my Niri journey was the purchase of a new super ultrawide monitor. I bought the 57” Odyssey Neo G9 as it was the largest monitor I could find. (It’s marketed as a “gaming” display but it’s really an amazing productivity display.) It replaced my old 3-monitor setup: My old 3-monitor setup. The new display is wider so I had to move the speakers around 10–15cm further apart. I was debating whether to replace the center 31.5” monitor or replace all monitors with a single one but I think I made the right choice with the ultrawide. The curvature wasn’t an issue (I’ve come to prefer it) and the extra vertical space the portrait side monitors provided wasn’t as crucial as I thought. I think an ultrawide is worth it just to get rid of the annoying bezels. Small things can be a big thing sometimes. A more dynamic workflow xmonad and Niri are similar yet different. Both automatically lay out windows as you spawn them but xmonad (at least the way I used it) follows a layout algorithm that re-flows using a “master” window and combines the rest of the windows into one space, while Niri lays out windows in columns. The change is subtle but it implies that a new window won’t change the size of other windows. This is very nice if you spawn a lot of short-lived terminals or web browsers like I do and it reduces the amount of manual reshuffling I spend time on. My xmonad workflow was more static than my Niri one. In xmonad I made heavy use of workspaces, mapping ten workspaces mentally to different programs, such as 0 Firefox and 1 terminal logs on the left monitor; 3, 4 and 5 for different Neovim instances on the center monitor; 8 as chat and 9 for music or video on the right monitor. I had no rules to enforce this; it’s an emergent behaviour that served me well for years. With Niri it’s more dynamic. I still use workspaces but they no longer have direct shortcuts, I simply go up/down in the workspace list. Maybe I’ll add them in the future but with 3–4 workspaces that’s not as necessary. I spawn workspaces/windows when I need them and remove them when I’m done. Usually it’s one workspace per project (yes, I’m now one of those who have multiple up at once) with all the related things such as editor, terminals, and browser with docs. I don’t typically utilize the full screen width and I try to keep the things I’m working on in the center, often leaving 10–30% gaps on the sides. Even though I don’t normally use the “endless scrolling” feature of Niri I re-center selected windows all the time so I can look straight ahead as much as possible. Keyboard shortcuts As a fan of keyboard layouts of course I have to spend some time tinkering with good keyboard shortcuts (especially as Niri’s recommended keybinds don’t map well with my custom keyboard or custom layout). Navigation layer What I did was add a new navigation layer that’s enabled by holding Tab (ring + middle + index on the left-hand side) with all Niri related movement and layout keybinds. In the graphics above, all green-colored keys emit Gui (which gates all window manager commands) and you can see: Long press on Close Window to close a window. The long press requirement prevents accidentally closing windows. Arrows move through columns/windows. Long press resizes them. Workspace Up/Down focuses a different workspace. Center a column. Consume/Expel to combine windows into one column. (consume-or-expel-window-left/consume-or-expel-window-right) Expand Column makes a column take up all remaining space. (expand-column-to-available-width) Audio controls. To press them I release the index finger (keeping the ring and middle finger pressed to keep the layer active) and use the index to press the audio buttons. Mouse buttons. In Niri you can move floating windows with Gui + Left Mouse and Gui + Right Mouse to resize them. As my main mouse is a trackball integrated into the keyboard I had to add them to the left-hand side. I ended up using QMK’s customizable key repress feature that allows me to: Tab combo with my three fingers (layer is active) Release only the index (layer is still active, same as with the audio controls) Press the index again (now detects the press Gui + Left Mouse key down) Use the trackball to move the window And similarly for the right mouse button to resize with the middle finger. Works great! Because there are so many commands I want to send I placed Ctrl on the thumb that provides movement-related commands like so: For example: Arrows move columns/windows in the four directions. Move columns to the neighboring workspaces. Center visible columns. (center-visible-columns) Slightly different consume/expel semantics. (consume-window-into-column/expel-window-from-column) Regular keymaps These are triggered in the “normal” way by first pressing the Super combo and then another key on the base layer (I use autoshift so I shift with a long press). Window management Super + F toggle windowed fullscreen (keep column width) Super + Shift + F fullscreen window (over the entire display) Super + M maximize column (moves other columns) Run stuff Super + Enter terminal Super + E Noctalia’s launcher (also exists on the navigation layer as Launch) Super + S show Noctalia control center Super + Shift + S show Noctalia settings Super + Q power off monitors (they wake on input) Super + Shift + Q show Noctalia session menu (reboot etc) Super + Shift + L lock screen Misc Super + H show hotkey overlay Super + P interactive screenshot Super + Shift + P screenshot selected window Tweaks to the standard CachyOS setup In the process of moving from xmonad to Niri I also moved from Void Linux to CachyOS and I let the installer install Niri and give me a basic configuration together with Noctalia (that provides a statusbar, notifications, and a bunch of things you apparently need). Center the status bar and other Noctalia windows My centered Noctalia status bar. Feels absolutely required on this screen otherwise things end up in the corners. Firefox on XWayland Force Firefox onto XWayland as the Wayland popup manager is broken: environment { MOZ_ENABLE_WAYLAND "0" } Dead keys for Ghostty For some reason dead keys were broken in Ghostty. This is bad for me as the OS keyboard is set to Swedish and it uses them to type ~ (quite a crucial character for a programmer). The fix: environment { GTK_IM_MODULE "ibus" QT_IM_MODULE "ibus" XMODIFIERS "@im=ibus" } This needs ibus installed and running. Melange colorscheme Noctalia discovers custom color schemes under ~/.config/noctalia/colorschemes/<Name>/<Name>.json, so I dropped in my trusty Melange colorscheme there: { "dark": { "mPrimary": "#EBC06D", "mOnPrimary": "#292522", "mSecondary": "#A3A9CE", "mOnSecondary": "#292522", "mTertiary": "#85B695", "mOnTertiary": "#292522", "mError": "#D47766", "mOnError": "#292522", "mSurface": "#292522", "mOnSurface": "#ECE1D7", "mSurfaceVariant": "#34302C", "mOnSurfaceVariant": "#C1A78E", "mOutline": "#867462", "mShadow": "#1a1816", "mHover": "#E49B5D", "mOnHover": "#292522", "terminal": { "normal": { "black": "#867462", "red": "#D47766", "green": "#85B695", "yellow": "#EBC06D", "blue": "#A3A9CE", "magenta": "#CF9BC2", "cyan": "#89B3B6", "white": "#ECE1D7" }, "bright": { "black": "#34302C", "red": "#BD8183", "green": "#78997A", "yellow": "#E49B5D", "blue": "#7F91B2", "magenta": "#B380B0", "cyan": "#7B9695", "white": "#C1A78E" }, "foreground": "#ECE1D7", "background": "#292522", "selectionFg": "#C1A78E", "selectionBg": "#403A36", "cursorText": "#292522", "cursor": "#EBC06D" } } } Then pick the colorscheme: "colorSchemes": { "darkMode": true, "predefinedScheme": "Melange", "useWallpaperColors": false } Layout appearance The default appearance was pretty I admit but way too much blank space and weirdness. Some tweaks: layout { // Required for noctalia-shell to set wallpaper background-color "transparent" // Never auto-center focused columns (too much movement) center-focused-column "never" // But do center a single window always-center-single-column // No extra space around it all struts {} // No gaps between windows gaps 0 // The focus ring was annoying focus-ring { off } // Use a border with consistent width for all windows instead border { on width 2 active-color "#ebc06d" inactive-color "#403a36" } // Setting widths is important with such a large screen preset-column-widths { proportion 0.15 proportion 0.3 proportion 0.4 } default-column-width { proportion 0.15; } // Heights too, why not? preset-window-heights { proportion 0.15 proportion 0.5 proportion 1.0 } } // Prevent the mouse from opening the overview in the corners gestures { hot-corners { off } } Keep windows centered Niri has the always-center-single-column option, which is nice as I want to keep as much as possible in the center of the monitor when I’m working. But I very frequently use 2–3 smaller windows and with my frequent opening and closing I’d like them centered too. Luckily, Niri has an IPC you can use to make a small program that reacts to events and does this for you. I made a small rust project using the niri-ipc crate that does this for me: The autocenter implementation [dependencies] niri-ipc = "26.4.0" use std::collections::HashMap; use std::io; use niri_ipc::socket::Socket; use niri_ipc::{Action, Event, Request, Response, Window, Workspace}; #[derive(Clone, Copy, PartialEq, Eq, Hash)] struct WindowId(u64); #[derive(Clone, Copy, PartialEq, Eq)] struct WorkspaceId(u64); struct OutputName<'a>(&'a str); struct WindowState { workspace: Option<WorkspaceId>, width: f64, } fn main() -> io::Result<()> { let mut socket = Socket::connect()?; if !matches!(socket.send(Request::EventStream)?, Ok(Response::Handled)) { eprintln!("niri rejected event stream"); std::process::exit(1); } let mut known: HashMap<WindowId, WindowState> = HashMap::new(); let mut read_event = socket.read_events(); loop { let result = match read_event()? { // A full snapshot of the current state. Just refresh our state. Event::WindowsChanged { windows } => { known = windows .into_iter() .map(|w| { ( WindowId(w.id), WindowState { workspace: w.workspace_id.map(WorkspaceId), width: w.layout.tile_size.0, }, ) }) .collect(); Ok(()) } Event::WindowOpenedOrChanged { window } => { let workspace = window.workspace_id.map(WorkspaceId); let entry = WindowState { workspace, width: window.layout.tile_size.0, }; let prev = known.insert(WindowId(window.id), entry); match prev { // Don't center floats. _ if window.is_floating => Ok(()), // New window, try to re-center. None => maybe_center_new(&window), // Window changed workspace, try to re-center. Some(state) if state.workspace != workspace => center_focused_if_fits(), // Skip other things. Some(_) => Ok(()), } } Event::WindowClosed { id } => { if known.remove(&WindowId(id)).is_some() { center_focused_if_fits() } else { Ok(()) } } Event::WindowLayoutsChanged { changes } => { // Only re-center if the width was changed, otherwise our re-center will // loop back indefinitely. let mut resized = false; for (id, layout) in changes { if let Some(state) = known.get_mut(&WindowId(id)) { if (state.width - layout.tile_size.0).abs() > 0.5 { state.width = layout.tile_size.0; resized = true; } } } if resized { center_focused_if_fits() } else { Ok(()) } } _ => Ok(()), }; if let Err(e) = result { eprintln!("autocenter: {e}"); } } } /// Center a newly created window if the workspace is focused and if there's surrounding free space left. fn maybe_center_new(window: &Window) -> io::Result<()> { let Some(workspace_id) = window.workspace_id.map(WorkspaceId) else { return Ok(()); }; let Some(focused) = focused_workspace()? else { return Ok(()); }; if WorkspaceId(focused.id) == workspace_id { center_if_fits(&focused)?; } Ok(()) } /// Center windows in the focused workspace if there's surrounding free space left. fn center_focused_if_fits() -> io::Result<()> { if let Some(focused) = focused_workspace()? { center_if_fits(&focused)?; } Ok(()) } /// Center windows in the workspace if there's surrounding free space left. fn center_if_fits(workspace: &Workspace) -> io::Result<()> { let Some(output) = workspace.output.as_deref().map(OutputName) else { return Ok(()); }; let Some(width) = output_width(output)? else { return Ok(()); }; if workspace_width(WorkspaceId(workspace.id))? < f64::from(width) { center_visible_columns()?; } Ok(()) } /// Issue a one-shot query to Niri, wait, and return the response. fn query(request: Request) -> io::Result<Response> { match Socket::connect()?.send(request)? { Ok(response) => Ok(response), Err(msg) => Err(io::Error::other(msg)), } } /// Get the focused workspace. fn focused_workspace() -> io::Result<Option<Workspace>> { match query(Request::Workspaces)? { Response::Workspaces(ws) => Ok(ws.into_iter().find(|w| w.is_focused)), _ => Ok(None), } } /// Get the width of an output (monitor). fn output_width(name: OutputName<'_>) -> io::Result<Option<u32>> { match query(Request::Outputs)? { Response::Outputs(outputs) => Ok(outputs .get(name.0) .and_then(|o| o.logical.as_ref()) .map(|l| l.width)), _ => Ok(None), } } /// Calculates the width of all columns in the workspace. fn workspace_width(workspace_id: WorkspaceId) -> io::Result<f64> { let Response::Windows(windows) = query(Request::Windows)? else { return Ok(0.0); }; let mut columns: HashMap<usize, f64> = HashMap::new(); for w in windows { if w.workspace_id.map(WorkspaceId) != Some(workspace_id) { continue; } if let Some((col, _)) = w.layout.pos_in_scrolling_layout { let width = columns.entry(col).or_insert(0.0); *width = width.max(w.layout.tile_size.0); } } Ok(columns.values().sum()) } /// Send a command to center the visible columns. fn center_visible_columns() -> io::Result<()> { if let Err(msg) = Socket::connect()?.send(Request::Action(Action::CenterVisibleColumns {}))? { eprintln!("center-visible-columns rejected: {msg}"); } Ok(()) } One catch is that if a new window overflows the monitor width, the script won’t center the columns even if there would be free space left afterwards. This is a little weird but it’s consistent with Niri’s center-visible-columns command. I had a small itch to try to hack around it but in the end I left it alone… Is Niri worth it? Yes, absolutely. Niri has been a huge upgrade for me in combination with a single wide screen. My xmonad setup worked really well with three monitors—arguably a better fit in that context than Niri—but for the big-screen use-case Niri is superior. I’m curious how it holds up on my laptop, once I gather enough energy to install CachyOS on it… But that’s a side quest. The big-screen setup I spend most of my days in is the best I’ve ever had, and I have no desire to go back.

Accessing services via raw IP addresses isn’t that swell; I’m no Rain Man. It’s time to set up subdomains for my hietala.xyz domain for internal use. We’ll use the Gateway API to set up routes, cert-manager to give us https without self-signed browser warnings, and ExternalDNS to set up DNS overrides. HTTP routing I believe the flow of resolving http://argocd.hietala.xyz to a service looks something like this: digraph { rankdir=LR "Browser" -> "OPNsense DNS" [label="resolve host"] "OPNsense DNS" -> "Browser" [label="Gateway IP"] "Browser" -> "Cilium Gateway" [label="HTTP request" class="label-below"] "Cilium Gateway" -> "HTTPRoute" [label="match hostname" class="label-below"] "HTTPRoute" -> "Service" [label="backendRef" class="label-below"] "Service" -> "Pod" [label="EndpointSlice" class="label-below"] } First the browser asks my router running OPNsense about argocd.hietala.xyz and gets the Gateway IP address. The request then flows through a route (http or https), to a Service, and eventually a Pod where ArgoCD is running. We’ll handle these one at a time but let’s start by creating the Gateway and giving it an IP (we’ll use 10.1.4.101): apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: main namespace: kube-system annotations: io.cilium/lb-ipam-ips: "10.1.4.101" spec: gatewayClassName: cilium We installed the prerequisites for Gateway when we configured Cilium during our initial cluster bootstrap. Then we need to tell the Gateway to manage all http routes for all namespaces (we’ll get back to https): # Continued from the above Gateway manifest spec: gatewayClassName: cilium listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All Then we can add an HTTPRoute for our ArgoCD application that targets argocd-server at port 80: # Below the ArgoCD Application setup # Resources in the same file are separated by `---` --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: argocd namespace: argocd spec: parentRefs: - name: main namespace: kube-system sectionName: http hostnames: - argocd.hietala.xyz rules: - backendRefs: - name: argocd-server port: 80 If we then add a DNS override from http://argocd.hietala.xyz to 10.1.4.101 (/etc/hosts or OPNsense or similar) then we should be able to reach http://argocd.hietala.xyz. Enabling SSL I want https://argocd.hietala.xyz to “just work” and for that we need to tell Gateway to manage https routes: # ... spec: gatewayClassName: cilium listeners: - name: https port: 443 protocol: HTTPS tls: mode: Terminate certificateRefs: - name: hietala-xyz-tls allowedRoutes: namespaces: from: All - name: http # Http definition from before (Note the tls addition that terminates using a not-yet-defined certificate.) And add the https route itself: apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: argocd namespace: argocd spec: parentRefs: - name: main namespace: kube-system sectionName: https hostnames: - argocd.hietala.xyz rules: - backendRefs: - name: argocd-server port: 80 This doesn’t work just yet as we need to create the hietala-xyz-tls cert. Cert-manager cert-manager seems like the standard way to manage certificates for Kubernetes. I don’t want to expose my services to the internet which means I need a DNS01 challenge. cert-manager doesn’t natively support Namecheap (bummer) but there’s an open source webhook Namecheap provider out there. It hasn’t been updated in a couple of years but I couldn’t find an alternative… YOLO I guess? The cert-manager manifest: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://charts.jetstack.io chart: cert-manager targetRevision: v1.21.0 helm: values: | installCRDs: true destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true Note the installCRDs: true that makes things easier for us, and CreateNamespace=true which will create the cert-manager namespace for us too. Saves some typing. We also set sync-wave to -1 as it needs to sync before the cluster issuer and certificate that we’ll define later. Then the namecheap webhook: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager-webhook-namecheap namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/kelvie/cert-manager-webhook-namecheap.git targetRevision: HEAD path: deploy/cert-manager-webhook-namecheap helm: # Identifier that our issuer will use values: | groupName: acme.namecheap.com destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: syncOptions: - CreateNamespace=false automated: prune: true selfHeal: true Then a ClusterIssuer that uses the namecheap webhook: apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging annotations: # Must come after the cert-manager applications. # It defaults to 0 anyway but this is more explicit. argocd.argoproj.io/sync-wave: "0" # If resources doesn't exist ArgoCD may complain. argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: acme: # Use the staging endpoint during testing! server: https://acme-staging-v02.api.letsencrypt.org/directory email: [email protected] privateKeySecretRef: name: letsencrypt-staging solvers: - dns01: # Use the namecheap webhook webhook: # Same identifier the webhook defined above. groupName: acme.namecheap.com solverName: namecheap config: # These are very sensitive! # Store them in a Sealed Secret apiKeySecretRef: name: namecheap-credentials key: apiKey apiUserSecretRef: name: namecheap-credentials key: apiUser Make sure to use the staging issuer during testing to avoid rate limits. When you’re done playing around you can switch to the production server at https://acme-v02.api.letsencrypt.org/directory. I created a new ClusterIssuer called letsencrypt-prod instead of replacing the url so I can easily change between them if I need to. Create the sealed secret: kubectl create secret generic namecheap-credentials \ --namespace cert-manager \ --from-literal=apiKey="key" \ --from-literal=apiUser="user" \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem \ --format yaml \ > gitops/apps/cert-manager/namecheap-secret.yaml Finally we need to create the hietala-xyz-tls certificate that uses the issuer: apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: hietala-xyz # Must match the namespace of the Gateway namespace: kube-system annotations: # References the issuer so must be synced after. argocd.argoproj.io/sync-wave: "2" argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: # Gateway references this certificate using this name. secretName: hietala-xyz-tls dnsNames: - "*.hietala.xyz" - "hietala.xyz" issuerRef: # Switch to `letsencrypt-prod` later. name: letsencrypt-staging kind: ClusterIssuer When all this has been synced we should be able to see that the certificate is created: $ kubectl get certificate -n kube-system NAME READY SECRET AGE hietala-xyz True hietala-xyz-tls 4d20h And that we can visit https://argocd.hietala.xyz (browser will warn while we use letsencrypt-staging, on prod it should be without errors). (If not, then you have a bunch of debugging to do. Have fun!) Want to redirect http to https? Add this route: apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: argocd-redirect namespace: argocd spec: parentRefs: - name: main namespace: kube-system sectionName: http hostnames: - argocd.hietala.xyz rules: - filters: - type: RequestRedirect requestRedirect: scheme: https statusCode: 301 In general though I skip the http route as I tell Firefox to always use https. ArgoCD is still accessible via its external IP I defined in a previous post but you can also port-forward or proxy to it: # http://localhost:8001/api/v1/namespaces/argocd/services/argocd-server:80/proxy/ kubectl proxy # localhost:8123 kubectl port-forward svc/argocd-server -n argocd 8123:80 Automating DNS overrides By now most of the things are set up in proper GitOps fashion but there’s still one thing I have to do manually: I have to add a DNS override to unbound (it’s on my OPNsense router). Doing it once is fine but it gets old fast. Some of my overrides in unbound. Can you see a pattern? A wildcard domain could work but I have other services running outside of Kubernetes, so I’d like a cleaner solution. That solution is ExternalDNS, which automatically adds overrides for any existing Gateway HTTPRoute. There’s a webhook provider for OPNsense that we’ll use. The manifest: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: external-dns namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default sources: - repoURL: https://kubernetes-sigs.github.io/external-dns/ chart: external-dns targetRevision: 1.21.1 helm: valueFiles: - $values/gitops/apps/external-dns/values.yaml - repoURL: https://git.hietala.xyz/tree/home-ops.git targetRevision: HEAD ref: values destination: server: https://kubernetes.default.svc namespace: external-dns syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true It loads values.yaml from the repo: provider: name: webhook webhook: image: repository: ghcr.io/crutonjohn/external-dns-opnsense-webhook tag: v1.0.0 env: - name: OPNSENSE_HOST value: "https://router.hietala.xyz" # Remember to create the `external-dns` sealed secret. - name: OPNSENSE_API_KEY valueFrom: secretKeyRef: name: external-dns key: api-key - name: OPNSENSE_API_SECRET valueFrom: secretKeyRef: name: external-dns key: api-secret sources: - gateway-httproute policy: sync domainFilters: - hietala.xyz txtOwnerId: talos-dorne This connects to OPNsense, sources routes from the Gateway, targets the hietala.xyz domain, and uses a new external-dns secret: kubectl create secret generic external-dns \ --namespace external-dns \ --from-literal=api-key="key" \ --from-literal=api-secret="secret" \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem \ --format yaml \ > gitops/apps/external-dns/external-dns-secret.yaml With this I don’t have to add manual overrides anymore. Nothing like spending hours to automate a few minutes of work! If you want to extend support to other network things, say a Minecraft server talking TCP over 25565 with a fixed IP, we can add service support to ExternalDNS: sources: - gateway-httproute - service And declare the Minecraft Service like so: apiVersion: v1 kind: Service metadata: name: minecraft annotations: external-dns.alpha.kubernetes.io/hostname: mc.hietala.xyz spec: type: ClusterIP externalIPs: - 10.1.4.53 selector: app: minecraft ports: - port: 25565 targetPort: 25565 protocol: TCP name: minecraft Many steps but the end is nice There’s a few moving parts but once done supporting new apps is satisfyingly easy. For example, to expose the homeassistant service under https://ha.hietala.xyz this is enough: apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: homeassistant spec: parentRefs: - name: main namespace: kube-system sectionName: https hostnames: - ha.hietala.xyz rules: - backendRefs: - name: homeassistant port: 8123 Commit and push, and https://ha.hietala.xyz is ready in a jiffy, certificates and DNS overrides included. I encountered a bug in ArgoCD v3.3.5 where it’ll get stuck syncing the routes with extra lines in the diff (or mark them as out of sync), for example here: - backendRefs: - name: my-app port: 80 group: "" kind: Service weight: 1 matches: - path: type: PathPrefix value: / You can add the lines to each route to fix it but you can also tell ArgoCD to ignore them with these lines in the Application template: apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet spec: template: spec: ignoreDifferences: - group: gateway.networking.k8s.io kind: HTTPRoute jqPathExpressions: - .spec.parentRefs[].group - .spec.parentRefs[].kind - .spec.rules[].backendRefs[].group - .spec.rules[].backendRefs[].kind - .spec.rules[].backendRefs[].weight - .spec.rules[].matches syncPolicy: syncOptions: - ServerSideApply=true - RespectIgnoreDifferences=true

Now we’re getting to the fun stuff: GitOps. The act of pushing beautifully crafted .yaml files and seeing your Kubernetes cluster get red and stall out is surely what life’s all about. Install ArgoCD We have our Kubernetes cluster and now we’re going to bootstrap ArgoCD so it starts syncing from our git repository. Using helm: helm install argocd \ --repo https://argoproj.github.io/argo-helm \ argo-cd \ --namespace argocd \ --create-namespace \ --version 10.1.4 ArgoCD will by default generate an admin password and you can get it with the argocd tool: # You can use the password to login to the web. argocd admin initial-password -n argocd # Make ArgoCD available at localhost:8123 kubectl port-forward -n argocd svc/argocd-server 8123:80 You can use the argocd CLI to manage ArgoCD but I’m going to prefer the declarative way. Fixed IP You can use the port forward method above to access ArgoCD but having a fixed IP is nicer, especially if you want to use the argocd CLI (I only used it when trying to get it all working). I use a separate service for the IP, which allows me to assign an IP outside of the Cilium load balance pool we setup previously (10.1.4.101–10.1.4.255). It looks like this: apiVersion: v1 kind: Service metadata: name: argocd-server-ip namespace: argocd spec: type: ClusterIP externalIPs: - 10.1.4.51 selector: app.kubernetes.io/name: argocd-server ports: - name: http port: 80 targetPort: 8080 It simply routes 10.1.4.51:80 to the IP of argocd-server and port 8080. We need to apply it: kubectl apply -f gitops/bootstrap/argocd.yaml And you should see it assigned and able to visit it on the web: $ kubectl get svc -n argocd argocd-server-ip NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE argocd-server-ip ClusterIP 10.105.209.116 10.1.4.51 80/TCP 3d18h Repository I use a self-hosted Forgejo instance (hosted in a Proxmox LXC) but you can use whatever you want. You need an API token with read permissions. ArgoCD documents a declarative setup that we’ll use. For repositories we need to create a secret: kubectl create secret generic home-ops-repo \ --namespace argocd \ --from-literal=type=git \ --from-literal=url=<repo-url> \ --from-literal=username=<username> \ --from-literal=password=<token> \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \ > gitops/bootstrap/argocd-repo-secret.yaml We also need to add the repository label to the generated file so it looks something like this: apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: home-ops-repo namespace: argocd spec: encryptedData: password: ... type: ... url: ... username: ... template: metadata: name: home-ops-repo namespace: argocd labels: argocd.argoproj.io/secret-type: repository Apply it: kubectl apply -f gitops/bootstrap/argocd-repo-secret.yaml And the repo should show up in the UI or argocd repo list. Initial admin password Instead of letting ArgoCD generate a random password we can set the initial password explicitly. It’s possible to do it either via the Helm chart or the declarative way, which I’ll continue with. After generating it with kubeseal the secret should look like this: apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: argocd-secret namespace: argocd spec: encryptedData: admin.password: ... admin.passwordMtime: ... server.secretkey: ... template: metadata: labels: app.kubernetes.io/name: argocd-secret app.kubernetes.io/part-of: argocd name: argocd-secret namespace: argocd With three keys: password is a bcrypt hash: argocd account bcrypt --password 'supersecret' passwordMtime is the plaintext modification time: date -u +"%Y-%m-%dT%H:%M:%SZ" secretkey is a random value: openssl rand -base64 48 ArgoCD uses server.secretkey to sign session and API tokens, and it will regenerate the key if restarted. I kept getting logged out when I was rebuilding things all over the place and it was annoying so I added this to make it stable. A file structure ArgoCD is now installed but at this point I think it’s good to take a step back before we steam ahead. One of the benefits of ArgoCD is that you can create any kind of file structure and organize the repository any way you want. That’s also one of its problems: if you can do anything it’s hard to decide on a plan forward. I found the blog post How to Structure Your Argo CD Repositories Using Application Sets illuminating and even though their setup is overkill for my homelab, I simplified their ideas into something that works for me. The basic idea is to have all applications in their own folders under gitops/apps, and place all bootstrap stuff under gitops/bootstrap. Borrowing the “level” terminology from the above post here’s the basic idea: digraph { rankdir=TB "root.yaml" [class="loader"] "appset.yaml" [class="loader"] bootstrap_more [label="…"] apps_more [label="…"] "root.yaml" -> "cilium_config.yaml" "root.yaml" -> "argocd.yaml" "root.yaml" -> "appset.yaml" "root.yaml" -> bootstrap_more "appset.yaml" -> "authentik" "appset.yaml" -> "actualbudget" "appset.yaml" -> "miniflux" "appset.yaml" -> "homeassistant" "appset.yaml" -> apps_more // Keep the "…" boxes on the far right of each level. { rank=same; "cilium_config.yaml" -> "argocd.yaml" -> "appset.yaml" -> bootstrap_more [style=invis] } { rank=same; "authentik" -> "actualbudget" -> "miniflux" -> "homeassistant" -> apps_more [style=invis] } } Which loads in three phases: root.yaml Everything under bootstrap/ gets loaded bootstrap/appset.yaml in turn loads all folders under apps/ This is what the directory structure looks like: gitops ├── root.yaml # Level 1 ├── bootstrap # Level 2 │ ├── cilium_config.yaml │ ├── argocd.yaml │ ├── appset.yaml │ └── ... └── apps # Level 3 ├── authentik │ └── ... ├── miniflux │ └── ... └── ... I’m considering separating infrastructure and user workloads, but for now I’m fine with having everything in the apps/ folder. App-of-apps root.yaml will be a master “app-of-apps” as it will be an Application that spawns other applications. It looks like this: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: all-apps namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-100" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://git.hietala.xyz/tree/home-ops.git targetRevision: master path: gitops/bootstrap destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true (You’re going to see lots of yaml files like this, better get used to it.) The important bits are The name all-apps in the argocd namespace It should be synced before everyone else (sync-wave: -100) Should load everything under gitops/bootstrap from my home-ops repo I’ll be using pruning and self-healing to tell ArgoCD to keep everything in sync. With selfHeal: true ArgoCD will revert any manual kubectl changes to a managed resource. This is the “proper GitOps” way as you must go through git but in practice it might be annoying. If you turn it off the ArgoCD will track the difference but won’t fight you. This is the entrypoint and it’s the last manual apply command we’ll use, as it’ll load everything else: kubectl apply -f gitops/root.yaml Application sets Inside the bootstrap/ folder lives, among other things, bootstrap/appset.yaml which is responsible for turning subfolders under apps/ into applications and loading them. This is what I use and it’s an adaption of the code from the aforementioned blog post: apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: appset namespace: argocd spec: # Use Go's engine so we can use basename/path goTemplate: true # The engine silently fails on missing keys by default goTemplateOptions: ["missingkey=error"] # Generate from all subfolders from my repo under apps/ generators: - git: repoURL: https://git.hietala.xyz/tree/home-ops.git revision: master directories: - path: gitops/apps/* template: # The Application name will be folder + "-app" # Creative, I know. metadata: name: "{{.path.basename}}-app" spec: project: default # Describes the manifests of the application, # which is everything under apps/<app-folder>/ source: repoURL: https://git.hietala.xyz/tree/home-ops.git targetRevision: master path: "{{.path.path}}" # Give the application its own namespace destination: server: https://kubernetes.default.svc namespace: "{{.path.basename}}" syncPolicy: syncOptions: - CreateNamespace=true # Needed for larger manifests - ServerSideApply=true # ArgoCD does a dry run to check for resource kinds # it doesn't yet recognize. This breaks some things # so turn it off. - SkipDryRunOnMissingResource=true automated: prune: true selfHeal: true I hope the comments make it clear how the above works. The end result is that everything under apps/ will be loaded and organized under its own application. Other bootstrap things Manifests under bootstrap/ Even though we manually apply some manifests during the bootstrap sequence, they can still be managed by ArgoCD after the fact. This is solved by simply placing the relevant files in bootstrap/, and they’ll be managed going forward. These are what we’ve collected so far: Cilium We installed Cilium using a helm install command. Here’s a (hopefully) corresponding example of a manifest that ArgoCD can manage: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cilium namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://helm.cilium.io chart: cilium targetRevision: 1.19.2 helm: values: | kubeProxyReplacement: true k8sServiceHost: 10.1.4.10 k8sServicePort: 6443 l2announcements: enabled: true externalIPs: enabled: true gatewayAPI: enabled: true ipam: mode: kubernetes operator: replicas: 1 securityContext: privileged: true destination: server: https://kubernetes.default.svc namespace: kube-system syncPolicy: syncOptions: - CreateNamespace=false - ServerSideApply=true automated: prune: true selfHeal: true The duplication here is unfortunate but I couldn’t find a way around it. At least we can reuse cilium_config.yaml with the load balancer IP specifications that we created during cluster setup. Just plop it under bootstrap/ and we can move on. Gateway API Before installing Cilium we also applied the Gateway CRDs using this command: kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml This is the corresponding YAML manifest (smaller as we don’t pass dozens of options): apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: gateway-api namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-2" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/kubernetes-sigs/gateway-api.git targetRevision: v1.2.1 path: config/crd/standard destination: server: https://kubernetes.default.svc namespace: kube-system syncPolicy: automated: prune: true selfHeal: true Note that I specify sync-wave -2 for the Gateway and -1 for Cilium because the Gateway should sync before Cilium. Sealed Secrets We continue in the same manner of converting Helm installations to manifests: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: sealed-secrets namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-1" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://bitnami-labs.github.io/sealed-secrets chart: sealed-secrets targetRevision: 2.16.2 helm: values: | fullnameOverride: sealed-secrets-controller destination: server: https://kubernetes.default.svc namespace: sealed-secrets syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true ArgoCD In addition to the secrets used for the declarative setup (repository, admin password) ArgoCD can also manage itself: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: argocd namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://argoproj.github.io/argo-helm chart: argo-cd targetRevision: 10.1.4 helm: values: | configs: secret: createSecret: false destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: syncOptions: - CreateNamespace=false automated: prune: true selfHeal: true --- # Static IP as shown previously, `---` separates different kinds Updated bootstrap sequence This is the updated bootstrap recipe that I manage with Just: [doc("Bootstrap everything from zero")] full: just bootstrap::cluster just bootstrap::cilium just secrets::restore_sealed_secrets_private_key just bootstrap::sealed_secrets just bootstrap::argocd [doc("Bootstrap ArgoCD")] [working-directory('../gitops')] argocd: helm install argocd \ --repo https://argoproj.github.io/argo-helm \ argo-cd \ --namespace argocd \ --create-namespace \ --version 10.1.4 \ --set configs.secret.createSecret=false # We don't let ArgoCD generate the secret, we supply one: kubectl apply -f bootstrap/argocd-secret.yaml # Needed to be able to sync from repo. kubectl apply -f bootstrap/argocd-repo-secret.yaml # Wait for ArgoCD to deploy. kubectl rollout status deployment/argocd-server -n argocd --timeout=300s # Technically not needed as it'll get loaded on sync, this just speeds it up a little. kubectl apply -f bootstrap/argocd.yaml --force-conflicts --server-side # Sync everything else from the repo. kubectl apply -f root.yaml Renovate A pull request generated by the Renovate bot (upping its own dependency). To complete our GitOps setup and to exercise our newfound capabilities let’s setup the Renovate service. Its purpose is to monitor git repositories and to create pull requests where it updates dependencies, see the above image for how it’s created a PR to update itself. The singularity is here any second now. To get Renovate setup we need to create three things: The secret Renovate will pick up RENOVATE_TOKEN in secrets, generated like so: kubectl create secret generic renovate-token -n renovate \ --from-literal=RENOVATE_TOKEN='<token>' \ --dry-run=client -o yaml \ | kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \ > gitops/apps/renovate/renovate-secret.yaml We’ll later add the name renovate-token to the main manifest. The configuration file Renovate has a bunch of configurations you can add. I’m good with the defaults except to modify the ArgoCD settings to support the custom directory structure I’m using: { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": ["config:recommended"], "argocd": { "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"] }, "kubernetes": { "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"] } } This activates the argocd and kubernetes managers, where argocd watches ArgoCD specific manifests (Application/ApplicationSet) and kubernetes watches plain manifests (such as container images in Deployment). The manifest The Application manifest is similar to the ones we’ve seen before: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: renovate namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://docs.renovatebot.com/helm-charts chart: renovate targetRevision: 46.236.3 helm: values: | existingSecret: renovate-token cronjob: schedule: "0 1 * * *" renovate: config: | { "platform": "forgejo", "endpoint": "https://git.hietala.xyz", "repositories": ["tree/home-ops"], "gitAuthor": "Renovate Bot <[email protected]>", "automerge": false } destination: server: https://kubernetes.default.svc namespace: renovate syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true Note that we configure the secret (renovate-token), how often it should run, and the repository information. Commit and push and it should be ready to go. ArgoCD has a really good UI where you can see what’s happening, if you made a mistake somewhere, and also trigger the cron job manually. Or you can use kubectl, for example: # See the health status of all applications kubectl get applications -A # Trigger the cron job kubectl create job -n renovate --from=cronjob/renovate renovate-manual # Watch the logs kubectl logs -n renovate -l job-name=renovate-manual -f With this we’ve got our first example of managing a Kubernetes service via ArgoCD. We’ll continue to see more examples of this as the series continues.

I recently got a Pebble Time 2 as it seemed like a fun smartwatch away from Google/Apple/Samsung with a good 4 weeks of battery life. One thing I wanted to do is to create a custom watchface for my specific problems. It took more effort to design the watch than I had anticipated and there’s a deceptively large amount of thought that has gone into some of the features here, so I thought it’d be interesting to write a little about it. I don’t get 30 days of battery life with the watch; in practice it’s closer to ~2 weeks, with heart rate monitoring disabled. Issues I want the watch to help me with It’s difficult to describe what I’ve been going through the last ~6 months. It started with a normal depression (yeah, a “normal depression” says a lot already) but it was followed by what felt like a big increase in volume; manageable issues I’ve had forever were suddenly overpowering. I’ve always found it difficult to start chores and boring tasks, yet now it felt like trying to swim through quicksand. At the same time I had huge problems with hyper focus. I could start working on something and the entire day would just disappear; I would sit in front of the computer for 8 hours without any break, interrupted by having to go get my kids from school and I realize that my bladder is exploding and I hadn’t eaten lunch yet. Then I would spend the rest of the day thinking about it, even having difficulty falling asleep because I’m still thinking about it. Needless to say, my time management has gone out the window, even missing appointments for the first time ever. Of course, a watchface won’t solve any of these but I’m desperate and nothing I’m trying is helping. Concretely I’d like the watch to help me with these things: I was going to name the watchface “ADHD hero” but that was a little on the nose as I don’t have an ADHD diagnosis (yet… The investigation is under way but it’s not a quick process). It’s common for people to claim they have ADHD, like the cool kids. The issues I’m having are quite severe and it’s not something “cool to have”. A watchface only for me I like open source but I don’t have any plans of publishing the watchface. The entire point of the watchface is to remove friction in my life, not create another source of anxiety. This way I’m free to change it, rewrite it, pollute it with vibes, and I can focus on making it just good enough for me without having to worry about others. I encourage you to steal anything you like and create your own watchface; it’s straightforward with Pebble. Prototyping with Claude I’m a big believer in being able to quickly iterate through different prototypes. I’ve had success with using Claude to generate HTML prototypes and it can spit out images like this: Design-wise it’s hit-and-miss and Claude seems to lack creativity and taste (but sometimes it comes up with great ideas). You need to guide it properly and I’ve gotten good usage from telling it to generate dozens of variations of a concept and iterating from there. All images in this post are generated in this way at various points of the development. It can also quickly generate interactive prototypes, like this embedded widget that simulates (almost) the entire watchface and its features: A simulation of the watchface, generated by Claude. This is super useful and I’ve used it to fine tune the wedge geometry, color scheme, a chiptune library, and more. These are rough prototypes and they serve to give a direction, not to be a perfect representation of the end result. The interactive widget above for example has a bunch of rendering artifacts that don’t exist in the real watchface. Finding a style The design of the watchface started before I had the watch and before I wrote a line of code. Initially the feel was very different; it was a cleaner, more uniform, design without as much clutter and everything followed a consistent design language. I just knew that it was perfect. The original watchface. Tracking tasks. When the watch arrived I built it and it sucked. It was so boring… I saw the Comic Drop watchface and its comic book panel layout is really interesting. Wouldn’t it be cool to have a similar comic book panel layout that would show the events during the day? Maybe they could be dynamic and move around during the day? A comic book panel subdivision. It would be really neat to have some pixel graphics of a computer during the morning when I should work, a picture of a barbell when I should exercise, etc. Unfortunately that’s beyond my skill level at the moment so I had to give up that idea. So I went back to the original design but iterated on a “comic book” style with thicker lines, brighter colors, and more irregularities: The new comic book style, with elements I’ll discuss later. This is a more interesting watchface and it looks much better on the wrist than the first iteration. Not perfect of course but it catches my interest, which is the point. Calendar overview I want to be able to glance at the watch to see my calendar. Watchfaces such as the Sectograph add event information inside the clock (so an event between 1 and 2 would colorize the section of the analog clock) and I wanted something similar. Here are some prototypes to give you an idea of what I’m talking about: Single event in the day. Four events the next 6 hours. Fully loaded day. This should hopefully make it easier for me to be able to plan my day a little better. Pebble already has built-in notifications for events that work well. This is purely for me to visualize the day. A notable omission in my design is the lack of a text description of the event. That’s nice but it also clutters the design I’m going for. The important thing for me isn’t what the event is, it’s “is there an event at all”. Non-uniformity My brain is very good at filtering away static and uninteresting information. For example, I’ve tried to have Calendar, Habitica, and Todoist widgets on my phone to help me keep track of my events/habits/tasks… But it didn’t take long for my brain to start completely ignoring them despite them taking up 80% of the screen. When I say ignore I mean that literally: I completely stopped noticing them. So what I wanted to try was to spice it up a little by randomizing the event geometry. See these two examples: Uniform events. Non-uniform events. To me the non-uniform prototype is more visually interesting and the changing geometry should hopefully prevent me from filtering them out. The goal is to give a general overview of the day, not to display a precise calendar. I tried to have each event size correspond to the event length but I abandoned that for more randomization. To make generation simpler (and to save on battery) I snap events into 15-minute intervals. I’m not sure yet if this is a good call but so far it’s fine. You can argue that the rounder uniform version would look better on the Pebble Round 2, which is fair. Here’s a similar mockup but with the round’s geometry: Uniform events on the Round 2. Non-uniform events on the Round 2. I still think the non-uniform variant catches the eye more and solves my specific problem a bit better but there’s no denying that a round layout on a round watch fits very well. Moving towards 12 o’clock Another idea I had was instead of statically fixing events to their times I wanted the events to move towards “now” at 12 o’clock. So an event that renders at 3 o’clock is 3 hours away and an event that overlaps 12 is ongoing. While this prevents me from easily looking back at my day (past events disappear) it makes it easier to feel the urgency as events draw closer. A bit weird perhaps but in practice it feels great. Here’s the same event through its lifecycle: drifting in from 3 o’clock, ongoing as it crosses 12 (with the countdown band draining in the clock), then gone once it’s in the past. Something like this: Event is a few hours away. Event is ongoing. Event is in the past and has disappeared. Completion countdown This works pretty well but there’s a problem: how to differentiate between an event that will start in one minute and that has been going on for a while? Because the event gets cut off at 12 there’s no way to tell at a glance. At first I tried to enlarge the event wedge so it touches the clock if it’s ongoing. That was a bit weird so instead I tried to colorize the clock outline with the event that’s ongoing. Then I realized, why not have the outline count down the remaining time of the event, similar to a Time Timer? The countdown band draining as the event completes. In practice this works out wonderfully well. I had dozens of prototypes for line weights, fill direction, the clock itself, etc but I won’t bore you with them. I tell myself that real design should appear to be effortless. Work timer I quite like the completion countdown for calendar events. This kind of design is often associated with tracking tasks (such as in Pomodoro) so why not add it to the watch too? For example, I might want to configure a stretch of 60 min work / 15 min break / 60 min work before I should stop working. Pretty standard stuff. As I’m struggling to get started I’ll add a twist: a working block should have a 5-minute warm-up period. So the timeline looks like this: warmupwork break warmupwork 60 min 15 min 60 min I chose to include the 5 minute warm-up into the work length, so they’re both 60 minutes long instead of 65 minutes. I think it makes sense to have that one count down (from filled to empty) while the working block counts up (from empty to filled), like so: Task 5-min warm-up followed by two 1 hour rounds, with a 15-min break between. Starting/stopping timers on a Pebble watchface This is a post about design, not about implementation, but I still need to address how we can implement a work timer toggle. The issue is that a Pebble watchface is a background process that cannot interact with Pebble’s physical buttons. You can bind the buttons to quick launch Pebble apps but not watchfaces. A watchface can detect steps and taps on the watch but it cannot communicate with other apps. There’s a few ways to get around this: Toggle using taps only. Implement it as a Pebble app, not a watchface. Add a new Pebble app (tied to a button) that round trips commands via the companion app back to the watchface. digraph { rankdir=TB button [label="Quick launch\nbutton"] app [label="Pebble app"] phone [label="Companion app\non phone" class="external"] face [label="Watchface"] button -> app [label="press"] app -> phone [label="toggle"] phone -> face [label="toggle"] { rank=same; app; face } app -> face [style=invis] } I first tried to implement toggling using taps but it was too unreliable. If I went the app route I’d have to relaunch it all the time, which represents unacceptable friction. What’s left is the janky phone round trip solution. I already need a companion app for the calendar sync but it still makes me queasy. Crap like this is why I don’t feel like publishing the watch. *shudder* Clearing alarms and forcing breaks I’ve tried alarms, Pomodoro, and similar before but they all share the same failure mode: I simply ignore the alarm and continue with what I was doing. A Pomodoro can’t be interrupted; it marks 25 minutes of pure work. Francesco Cirillo, The Pomodoro Technique I don’t know, it could be 2 minutes just as likely as 200 minutes. So I figured I’ll try to force myself to take breaks by making it annoying to clear (a bit like the Hand Grenade Alarm clock but in wrist form). The watchface can vibrate, play sound, react to steps, and detect taps and this is the flow I came up with: The alarm transforms the screen and vibrates and plays a tune until it’s dismissed (repeats every 30 seconds). I had to silence it during calendar events to not mess up meetings too much. Visually the required steps are drawn at the border (no events are shown during an alarm) and the alarm is drawn as a comic-style blast (tapping it shrinks the blast). Like this: Dismissing the end of day alarm. How well does it actually work? It’s a pretty cool setup and it is helping but it still fails quite often. Most of the time I forget to start the work timer and other times I’m able to add enough steps by waving my hand around like a crazy person, and then continue working. (If I raise the step count then it gets annoying to dismiss when I’m actually walking.) Could be tuned better I suppose. Other types of alarms Time for a break Stop working End of day Manual alarm Visually there are some distinct alarms I use: I’ve separated them visually and they each use unique vibrations and tunes. An implementation detail: Pebble doesn’t natively render text at an angle. I worked around it by rendering the text at an angle to a bitmap, and drawing that. Other information I’ve previously used watchfaces overloaded with all kinds of information and graphs that were cool for a few minutes and then I never looked at them again. But there are some things I regularly glance at such as the date or if the watch has lost its phone connection. Date and week number Here are some prototypes for dates that I liked but ultimately discarded: The difficulty is to not crowd the watchface too much with the date and the other elements I’ll get to later. In the end I ended up with a simple 31/10 label and a small weekly text without background as I don’t look at the week number that often: The date label together with the weekly number floating freely. I’m not completely sold on the week number display (yellow labels were a lot more fun) but it’s good enough I guess. Step counter It was really difficult for me to find a step counter design I liked. Everything I tried just felt off. Here are three of my attempts: Steps are tracked at the edges. A health bar of some sort. Steps in a separate column. I used to track steps at the edges for most of the design (you can see the step elements in many of the prototype images in this post) but in the end it introduced a bit too much clutter for my taste. While the other two attempts above are crude and could be polished more I didn’t see potential in them. (Examining and cutting away branches is important during design I think.) And then the revelation: I don’t care about the exact step count. The only purpose of showing steps is to help me keep the step count up, so I’m not sitting in front of the computer all day, and having badges for tiers solves that problem even better. Here are the badges I use: Bronze Silver Gold Diamond Bronze is achieved at 33% of my daily goal, silver at 66%, gold at 100%, and diamond is for over-achieving with 150%. I feel that having an easy to reach tier helps me do something on the lazy days and the hard-to-reach tier helps give me an extra kick the days I’m out and about. Yes, not knowing exactly how much is left to the next tier is slightly annoying, but for those cases opening up the Health app on the watch to view the step count is good enough. Battery states High after the rush of figuring out how to display steps let’s use the same approach for battery warnings: Ok, above 15% Low, below 15% Critical, under 5% Yes, I probably should make the little battery interactive or something but this is a 80:20 situation: I get 80% of the benefit from 20% of the effort. It’s good enough. Bluetooth & Sleep / DND More badges! Bluetooth disconnected. Phone in DND or Pebble in Quiet Time. Pebble has a “Quiet Time” mode that silences notifications. This is a separate thing from Android’s sleep mode or do-not-disturb mode. As I have a companion app I can detect the Android state and the watchface combines the two (and silences task alarms for instance) while also showing a “sleep” bubble so I can see these invisible states. The next task I’m a big fan of some of the ideas of Getting Things Done such as the idea of the “next task”: the one actionable thing that you should do next. I’ve used the Todoist Android widget on my phone to display my tasks but I wanted to incorporate it into my watch too. The idea is to only display the next task, something like this: While I think all three prototypes above are visually pleasing I went with the wider display that takes up less vertical space: The next task displayed at the bottom. As a bonus I don’t have to mess around with rendering angled text. Completing/rescheduling the task It works fairly well but there’s a wrinkle: when I’ve done the task or realize I can’t do it now I have to pull up my phone and that’s annoying so in practice I’m leaving a stale task alone forever, priming my brain to ignore it. I’ve tried the Todoist Mini app on the Pebble but there’s so many clicks to find the task and complete or reschedule it. What I did was add it to the app I’m already using to toggle the work timer and give it a menu of options: Completing the next task via the watch. I can still toggle the work timer, complete the next task, or reschedule the task (it’s rescheduled for tomorrow). I’ve bound it to the UP button via “quick launch”. To keep toggling the work timer as simple as possible pressing UP again toggles the timer (so double clicking UP when the watchface is visible toggles the timer). Otherwise it uses the standard Pebble control scheme (UP/DOWN to select and SEL to execute) and the app closes and returns to the watch when an item is selected. So far it’s been working really well but as has always been the case before with these productivity “fixes”; I always tire of them eventually. There’s some companion app interactions I gloss over here (such as latency and if the phone isn’t reachable) but it’s fine, don’t worry about it. I did add a vibration + chime confirmation on the watch as a confirmation that the round trip was completed and it gives a small dopamine hit. Edge markers As a little extra I added some markers to track other types of things along the edges: Markers moving on the edge for alarms, calendar reminders, end of day, and sunrise/sunset. The colored dots for calendar events are genuinely useful as many events require some preparation/travel time. In these cases I add a reminder for the event at, say, 15 minutes before. With the dots this preparation time can now also be seen by glancing at the watch (Pebble natively shows reminder alerts anyway.) The end of day marker is also useful for when I’m planning my working day but sunrise / sunset is mostly a gimmick, and I don’t use the manual alarms . The end result When it’s clear. When it’s (very) busy. At the end I got an interesting watchface I think looks fun and that helps me go about my day. It’s pretty easy to make one, and if you don’t know where to begin LLMs can shorten the time to get up and running.

The Kubernetes iceberg. If I’d have to describe my homelab setup via analogy I guess it would be similar to me on a unicycle carrying plates with both of my hands, or maybe a leaking barrel with water that I try to patch up with silver tape. I’ve also been Kubernetes-curious so I decided to completely redesign my homelab, centered around Kubernetes. It was a bit painful but at least it fulfilled my need for procrastination very well. Overarching goals I’ve got three goals with the setup: Declarative, reproducible, and automated The big goal is to have everything declarative in a single git repository and to easily be able to bootstrap from nothing to a fully working setup. I want to use Infrastructure as Code to create the Kubernetes cluster and GitOps to populate it with all my services automatically from the repo. It should be really easy to make a change; I want to move away from having to ssh into the correct repo and manually do stuff. Backups, backups, backups While a proper GitOps setup means that infrastructure and configuration files are inherently backed up, a proper backup setup is still crucial. Ask me how I know. I haven’t had a proper (as in working) backup solution for years and this time I should have it from the start. Documentation What if I could document my setup, so future me has a chance to understand what’s happening? Writing documentation is boring, so I’ll write some blog posts instead. I’m a bit skeptical that I can fulfill all three goals, but if I manage 2/3 or even 1/3 it’s still a big win compared to my old setup. Kubernetes, too complicated? It’s a fair question and the most common critique towards Kubernetes is that’s just too complicated (especially for a homelab). Discussions online are filled with comments such as: Kubernetes has to be most complex software I’ve ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. trinovantes “Let’s use Kubernetes!” Itamar Turner-Trauring So why would I choose Kubernetes? Because, for whatever reason, Kubernetes is very popular and for every comment complaining about complexity you have comments extolling it’s virtues: I was skeptical about Kubernetes but I now understand why it’s popular. The alternatives are all based on kludgy shell/Python scripts or proprietary cloud products. zelly Kubernetes is the biggest quality-of-life improvement I’ve experienced in my career anchochilis Having experienced the single machine docker-compose deployments, kludgy shell scripts, and proprietary cloud products; I think I need to use Kubernetes myself to be able to form an opinion on it. And in some ways, isn’t experimentation a core part homelabbing? Tech stack There are many valid tech choices for this kind of setup and many of them are reasonable. I don’t know if my choices are reasonable—most were chosen because they sounded cool, others because I just picked one. Here’s list of some of the choices I made, which we’ll setup in this series: Talos Linux for Kubernetes nodes. The coolest way to run Kubernetes. Lightweight and secure, what’s not to like? Terraform to provision VMs on Proxmox and to initialize Talos Linux. Cilium for proxying, CNI, load balancer, and Gateway API provider. I opted for Cilium as it’s one dependency replacing several alternatives (such as kube-proxy, Metallb, and Traefik, which I was leaning towards at first). Gateway API is the new thing you “should” use instead of ingress, and I wanted to try it out. ArgoCD for GitOps. If it was purely for myself FluxCD might have been the better, simpler, choice but we might use ArgoCD at work and I don’t want to deal with two separate systems at the moment. Renovate to keep dependencies up-to-date. CloudNativePG for Postgres on Kubernetes. I’ll also setup timescaledb, although we won’t use it in this series. It’s just to prepare for the future migration of long-term statistics from Home Assistant. Longhorn on NVMEs for persistent storage. Data is backed up using VolSync and Restic. Sanoid, Syncoid and Kopia for backup archive management. Backups are snapshotted and stored in ZFS, which are also encrypted and shipped off-site to Backblaze for storage in the cloud. Backups from Longhorn and Postgres arrives to ZFS via Garage, a self-hosted S3 service. Authentik as an identity provider and single-sign-on platform. It’s nice to not have to login manually everywhere. Huh. Displayed like this it looks like a lot, but fear not! It’ll be worth it in the end. In the next part we’ll start by creating VMs and getting a Kubernetes cluster up and running.