More from macwright.com
Google published Zanzibar: Google’s Consistent, Global Authorization System in 2019. It describes a system for authorization – enforcing who can do what – which maxes out both flexibility and scalability. Google has lots of different apps that rely on Zanzibar, and bigger scale than practically any other company, so it needed Zanzibar. The Zanzibar paper made quite a stir. There are at least four companies that advertise products as being inspired by or based on Zanzibar. It says a lot for everyone to loudly reference this paper on homepages and marketing materials: companies aren’t advertising their own innovation as much as simply saying they’re following the gospel. A short list of companies & OSS products I found: Companies WorkOS FGA Authzed auth0 FGA Ory Permify Open source Ory Keto (Go) Warrant (Go) probably the basis for WorkOS FGA, since WorkOS acquired Warrant. SpiceDB (Go) the basis for Authzed. Permify (Go) OpenFGA (Go) the basis of auth0 FGA. I read the paper, and have a few notes, but the Google Zanzibar Paper, annotated by AuthZed is the same thing from a real domain expert (albeit one who works for one of these companies), so read that too, or instead. Features My brief summary is that the Zanzibar paper describes the features of the system succinctly, and those features are really appealing. They’ve figured out a few primitives from which developers can build really flexible authorization rules for almost any kind of application. They avoid making assumptions about ID formats, or any particular relations, or how groups are set up. It’s abstract and beautiful. The gist of the system is: Objects: things in your data model, like documents Users: needs no explanation Namespaces: for isolating applications Usersets: groups of users Userset rewrite rules: allow usersets to inherit from each other or have other kinds of set relationships Tuples, which are like (object)#(relation)@(user), and are sort of the core ‘rule’ construct for saying who can access what There’s then a neat configuration language which looks like this in an example: name: "doc" relation { name: "owner"} relation { name: "editor" userset_rewrite { union { child { _this f } } child { computed_userset { relation: "owner" } } relation { name: "viewer" userset_rewrite { union { child {_this f} } child { computed_userset & relation: "editor" 3 } child { tuple_to_userset { tupleset { relation: "parent" } computed_userset { object: $TUPLE_USERSET_OBJECT # parent folder relation: "viewer" } } } } } } It’s pretty neat. At this point in the paper I was sold on Zanzibar: I could see this as being a much nicer way to represent authorization than burying it in a bunch of queries. Specifications & Implementation details And then the paper discusses specifications: how much scale it can handle, and how it manages consistency. This is where it becomes much more noticeably Googley. So, with Google’s scale and international footprint, all of their services need to be globally distributed. So Zanzibar is a distributed system, and it is also a system that needs good consistency guarantees so that it avoid the “new enemy” problem, nobody is able to access resources that they shouldn’t, and applications that are relying on Zanzibar can get a consistent view of its data. Pages 5-11 are about this challenge, and it is a big one with a complex, high-end solution, and a lot of details that are very specific to Google. Most noticeably, Zanzibar is built with Spanner Google’s distributed database, and Spanner has the ability to order timestamps using TrueTime, which relies on atomic clocks and GPS antennae: this is not standard equipment for a server. Even CockroachDB, which is explicitly modeled off of Spanner, can’t rely on having GPS & atomic clocks around so it has to take a very different approach. But this time accuracy idea is pretty central to Zanzibar’s idea of zookies, which are sort of like tokens that get sent around in its API and indicate what time reference the client expects so that a follow-up response doesn’t accidentally include stale data. To achieve scalability, Zanzibar is also a multi-server architecture: there are aclservers, watchservers, a Leopard indexing system that creates compressed skip list-based representations of usersets. There’s also a clever solution to the caching & hot-spot problem, in which certain objects or tuples will get lots of requests all at once so their database shard gets overwhelmed. Conclusions Zanzibar is two things: A flexible, relationship-based access control model A system to provide that model to applications at enormous scale and with consistency guarantees My impressions of these things match with AuthZed’s writeup so I’ll just quote & link them: There seems to be a lot of confusion about Zanzibar. Some people think all relationship-based access control is “Zanzibar”. This section really brings to light that the ReBAC concepts have already been explored in depth, and that Zanzibar is really the scaling achievement of bringing those concepts to Google’s scale needs. link And Zookies are very clearly important to Google. They get a significant amount of attention in the paper and are called out as a critical component in the conclusion. Why then do so many of the Zanzibar-like solutions that are cropping up give them essentially no thought? link I finished the paper having absorbed a lot of tricky ideas about how to solve the distributed-consistency problems, and if I were to describe Zanzibar, those would be a big part of the story. But maybe that’s not what people mean when they say Zanzibar, and it’s more a description of features? I did find that Permify has a zookie-like Snap Token, AuthZed/SpiceDB has ZedTokens, and Warrant has Warrant-Tokens. Whereas OpenFGA doesn’t have anything like zookies and neither does Ory Keto. So it’s kind of mixed on whether these Zanzibar-inspired products have Zanzibar-inspired implementations, or focus more on exposing the same API surface. For my own needs, zookies and distributed consistency to the degree described in the Zanzibar paper are overkill. There’s no way that we’d deploy a sharded five-server system for authorization when the main application is doing just fine with single-instance Postgres. I want the API surface that Zanzibar describes, but would trade some scalability for simplicity. Or use a third-party service for authorization. Ideally, I wish there was something like these products but smaller, or delivered as a library rather than a server.
I watched a large part of All Watched Over By Machines of Loving Grace this month. This also counts as a “listening” item, because the theme song, “Baby Love Child” by Pizzicato Five, is also spectacular. Guitar Moves is a good series of interviews by Matt Sweeney, who I mostly know via his involvement in Bonnie Prince Billy. It’s a really cool format. I like how he interviews guitarists with recognizable sounds, and you get to see how little they need to play to sound just like themselves. The episode with St. Vincent is excellent too: she’s one of my guitar heroes: check out the guitar solo in Just The Same But Brand New, or her version of Dig a Pony. I also watched No Other Land. Everyone should watch No Other Land. AI thoughts roundup I don’t have a conclusion. Really, that’s my current state: ambivalence. I acknowledge that these tools are incredibly powerful, I’ve even started incorporating them into my work in certain limited ways (low-stakes code like POCs and unit tests seem like an ideal use case), but I absolutely hate them. I hate the way they’ve taken over the software industry, I hate how they make me feel while I’m using them, and I hate the human-intelligence-insulting postulation that a glorified Excel spreadsheet can do what I can but better. Nolan Lawson: AI ambivalence As I always say, the purpose of the system is what it does. Or, in this case, how I think about AI stuff is mostly affected by how people use AI stuff, and how people use AI stuff is a real mixed bag. There’s the tidal wave of spam, the aesthetic of fascism, the low-effort marketing materials with nonsense images, the non-consensual AI porn. I see all of the bad stuff every day both online and in the odd subway ad. The good stuff seems pretty theoretical, though: the press releases about AI-driven medical advances never seem to break into the real world. The stories about engineers 10x’ing their ability seem pretty mixed: we’re already at the hangover-and-regret phase with programmers bemoaning how they’ve generated so much slop and lost so much knowledge. Anyway, I’m mildly optimistic about the potential! But it’s a lot like crypto in that you could theoretically use the technology for something good but most people loudly used it for bad stuff, and people including me judged it based on what it did. AI has to start doing some good stuff soon. Potential isn’t enough. I think one thing chatGPT’s invention has revealed is how many people - including some very important people in society - find just basic reading and writing to be laborious and cumbersome to perform, and how oddly closely that type of strained literacy correlates with having other shitty opinions. From mtsw on Bluesky, about this story about Andrew Cuomo using ChatGPT to half-assedly write a policy platform. Right off the bat I should say that judging people for their level of literacy reeks of classism and so on. My own ability to read & write has a lot to do with my place in society: I went to good schools, had a stable home life, and smart parents. However, “the way that society was set up” kind of evened this out. Extremely social people with cultural capital and chiseled jawlines and biceps would get their rewards, and people like… myself, we would get rewarded for literacy and critical thinking. When one group needed the other, it was usually some kind of payment or partnership: Cuomo pays his scriptwriter, the TV show creator pays the actors. And some people can do both sides of the equation. But LLMs definitely indicate that people do not like this deal. Whew, they don’t like writing, but they also don’t like paying the writers or reading what they write. Maybe they could rejigger the system so that they could do it all. They have ideas for music and art but no interest in learning about music, practicing instruments, going to art school, or concentrating on a task for a long time, so why not generate it all? Why not, well - there are reasons, those reasons being that the generated output usually passes their own vibe check but once someone who looks closely at things or reads all the words encounters it, everyone points at the slop and it’s embarrassing. (Cuomo will never be embarrassed) Plus, you’re always going to get average results by asking a device that is incapable of creativity or thought. Also, you’ll miss out on the human experience of creating. And you’ll be indirectly feeding output data into training data for future LLMs, consequentially making their output worse. (Cuomo does not care about consequences) Colophon update I’ve moved the images for this website to Bunny (that’s an affiliate link, here’s a non-affiliate link if that’s what you prefer). When I initially moved my photos to this website, I set them up with Amazon S3 for storage and CloudFront to serve them with a CDN. Using AWS is painful for me, so I moved them to Cloudflare R2, which is Cloudflare’s equivalent to S3, and Cloudflare as a CDN. Thanks to owning my own domains, swapping out image hosts is pretty quick: switching to Bunny took all of five minutes. So what’s the deal with Bunny? Partly I’ve become a little more negative on Cloudflare and R2: I think Cloudflare’s technology is neat, but R2 has iffy reliability and Cloudflare has iffy politics. I’m also intrigued by diversifying my dependencies geographically. Bunny is a Slovenian company, and my email is from an Australian company. This probably won’t have any practical effect, but it feels kind of good for obvious reasons to even minutely hedge my bets here. So far Bunny has been great. They don’t support the S3 protocol but they do support SFTP, which works just as well for my purposes and works great with the beautiful Transmit app. Before, with R2, I was using the significantly less beautiful Cyberduck application because Cloudflare R2 doesn’t support all of the S3 protocol. It seems to be just as fast as Cloudflare was, too. And I’m somewhat reassured by the prospect of paying Bunny. I don’t like the feeling of getting “free” services like I can from Cloudflare. I want that customer relationship. Reading Then again, pop culture is powerful, and even the dumbest marketing both affects and reflects it. Busch Light’s can holder shaped like a cup that holds beer is dumb, which is fine, because most beer promos are. But the fact that the brand frames it as a functional, masculine alternative to Stanley’s H2.0 Flowstate affirms a similarly retrograde outlook on gender roles to the one that young American men are seeking out on the political right. From my friend Dave’s article about Busch Light’s weird attempt to riff on the Stanley Tumbler trend. I was once a loyal listener of the Chapo Trap House podcast, but fell off of it in 2020 when their support of Bernie Sanders led them to be jerks about Elizabeth Warren. But reading this Vanity Fair article about the cohosts of the podcast endeared them to me a bit. “Like to thank” is linguistic phlegm. “I’d like to thank the Academy.” They’d “like to thank” me. Well I’d like to be 6’3” and drive a G Wagon, thanks. I’d like you to accept my novella. I’d like to quit paying three dollars to Submittable every time I want to send a story out. The world is full of actions I would like to do. The most direct way to say thank you is just to say it: “thank you, name, for doing X.” “I’d like to thank” is a performative thanks, a thanks with a smirk and a blink, eyeing for extra credit. Just because people say it in their award show acceptance speeches doesn’t mean you should say it, too. In fact, that’s the reason you shouldn’t say it. Loved this article “Close reading my rejections” from friend of the blog Barrett Hathcock.
(async () => { const colors = ['fb6b1d','e83b3b','831c5d','c32454','f04f78','f68181','fca790','e3c896','ab947a','966c6c','625565','3e3546','0b5e65','0b8a8f','1ebc73','91db69','fbff86','fbb954','cd683d','9e4539','7a3045','6b3e75','905ea9','a884f3','eaaded', '8fd3ff', '4d9be6', '4d65b4', '484a77', '30e1b9', '8ff8e2'].map(c => `#${c}`); const mask = document.querySelector('#mask'); const replacement = await fetch('/images/2025-04-12-tidbyt-second-life-tidbyt-mask.svg').then(r => r.text()); mask.style = ''; mask.innerHTML = replacement; let i = 0; let delay = 10; const svg = mask.querySelector('svg'); svg.removeAttribute('width'); svg.removeAttribute('height'); svg.setAttribute('style', 'width:auto;height:auto;position:absolute;top:0;right:0;bottom:0;left:0;opacity:0.4;'); for (const path of svg.querySelectorAll('path')) { delay += 20; delay *= 1.02; setTimeout(() => { path.setAttribute('fill', colors[i++ % colors.length]); }, delay) path.addEventListener('mouseover', () => { path.setAttribute('fill', colors[i++ % colors.length]); }); } })() Remember the Tidbyt? It’s a super low-resolution, internet-connected, wood-paneled display that I wrote a review of it back in 2022. It’s been on my shelf for years now, showing the time, weather, warning me when the UV is going to be high. In 2023 I used it as an excuse to learn some Rust, to render custom graphics. It’s a toy, a distraction, a worry stone for me to work on when I need something open-ended and low-stakes. Anyway, the company that made the Tidbyt is no more. They got acquihired by Modal, a company that makes serverless AI compute hosting. So, they aren’t making devices right now, and the blog post promises that their cloud services will keep working. I don’t hold anything against the Tidbyt team: in fact, our Val Town office was coincidentally right next to theirs in a WeWork, and we met in real life! They’re very nice folks, and were doing so much with a small team. Lots of respect to them. Modal made a smart choice acquiring Tidbyt. But realistically, it’s time to make sure my device doesn’t become e-waste. The Tidbyt is ready for this One of the biggest critiques of the Tidbyt was that it was just an LED matrix and an ESP chip. You could buy an LED matrix on Sparkfun, the ESP, a power supply, some wood for the enclosure, and you’d have your own DIY Tidbyt. Maybe you could do it for half the price! But that’s also a strength. The Tidbyt is not some custom SoC with an exotic custom software stack and boutique hardware. It is what it looks like: a neat combination of commonplace parts. That makes it kind of future-proof and flexible. The first step is to replace the firmware. Tidbyt’s stock firmware routes all of its requests through the Tidbyt company’s servers. I want to eliminate that hop. Replacing the firmware Thankfully, Tidbyt published their ‘HDK’, which is an open source version of their stock firmware. It’s remarkably simple: It connects to Wifi It downloads a WebP image from a URL It displays that WebP image The HDK contains the code to do this stuff. There’s very little code required, but it does drag in a WebP decoder, Wifi library, and a library for running the LED matrix. But, setting up the HDK I ran into issues both small and large: it had issues with HTTPS URLs and Wifi passwords that contain spaces. Plus nobody has been added as a contributor to the HDK repository, so Pull Requests aren’t being accepted and it hasn’t had a change in 7 months. But the community came to the rescue with tronbyt’s firmware-http, a fork of the HDK that fixes every issue I experienced. Open source works! So back in 2022 I included this chart of the Tidbyt network: With an updated HDK, this workflow is a lot simpler. Instead of sending images to the Tidbyt servers and those Tidbyt servers delivering them to my device, the device makes requests directly of the server that generates the images. Replacing pixlet The Tidbyt team wrote pixlet, a little framework for generating pixel graphics that the Tidbyt displays. It lets you define a React-like tree of components - some text in a stack, a rectangle, images, and so on - and does all of the layout and rendering. The tronbyt community also forked pixlet and are actively developing it, which is fantastic. But this part of the stack I really never liked. That’s why I spent so much time reimplementing it in Rust and JavaScript. Partly it’s the language - pixlet apps are written in starlark, which is kind of an outgrowth of the Bazel build system from Google. Starlark is sort of like Python, but isn’t actually compatible with anything in the Python ecosystem. It’s very niche, limited, and overall just weird. I think I understand why Tidbyt would choose Starlark - it’s fast and has hermetic execution - making it safe to run untrusted Starlark programs because they can’t access the filesystem, network, or even the system clock without being given explicit controlled APIs to do those things. If you’re building a cloud service that runs a lot of untrusted user code, dictating that code is all Starlark is a really good cheat code - I know firsthand how hard it is to run untrusted JavaScript. But I’m not building a cloud service full of untrusted code. People who are self-hosting their Tidbyt devices (dozens of us!) don’t benefit from the tradeoffs of the Starlark language. They’d be better off with something normal. I rewrote pixlet again It’s called indiepixel and it’s a Python reimplementation of pixlet. It supports almost the entire pixlet API, and comes with the added benefit of being Python. You can use Python modules! You can read from the filesystem, parse CSVs, do all of your usual Python stuff. You can embed it in a Python application to render some graphics. What does indiepixel do currently? Renders text in the glorious retro BDF pixel font format. Renders pixelated pie charts, rectangles, and boxes. Supports animation for its WebP outputs. Provides a nice UI for browsing your selection of screens. It’ll probably never be finished, but it works well enough to power my Tidbyt. I’m running indiepixel on a free Render server instance, but it should run pretty much the same on any Python-compatible hosting: the only tricky dependency is Pillow, which it uses for image parsing and rendering. My free time for computer-oriented side projects has been limited, due to other commitments and an intention to get offline on the weekends. I’ve been sewing, biking, and running more. So I really want a side project I can enjoy, and indiepixel has fit the bill. It’s really satisfying to implement a new widget and see it rendered in blocky 64x32 pixels. The Pillow image rendering library for Python is mostly wonderful and very powerful. Why Python? Why is indiepixel written in Python? Well - I learned from tidbyt-rs that Rust would be an awkward fit as a scripting language for rendering graphics. The well-known Rust complexities around memory management made simple things difficult for me, which would make them totally unacceptable for others. Besides the attraction of being able to compile a small binary that might be able to run on the Tidbyt itself, Rust didn’t have many other advantages. The Pillow module really is such an advantage for Python. JavaScript doesn’t have a real alternative: there’s sharp, a great module for image conversion, but nothing that has such a great canvas interface. node-canvas is fine, but it doesn’t support WebP or animation, which are critical features for this project. I also wanted a test out the amazing new Python tooling that Astral is cooking up, like uv. I now have a better grasp of the Python ecosystem than I did a few months ago, and it’s optimistic but mixed. uv is amazing, but Python has a lot of legacy cruft around packaging. People are critical of NPM, but I think it did benefit from being established after PyPI and learning from its lessons. Thank you Steven Loria for a PR that fixed everything and made it all work and saved me months of tweaking settings. The graphic I watercolored that Tidbyt a while while ago and have been seriously dragging my feet on finishing this blog post. Sometimes the watercolor-illustration wags the technical-blog-post dog’s tail? Anyway, it’s a callback to that little world, with some small tweaks: this time I thought it’d be nice to have it be both watercolored and interactive. That ‘cybernetic’ feel. The secret recipe: a nice palette from lospec, creating a black & white mask of areas in Affinity Photo and vectorizing it with potrace, and then just some JavaScript that recolors based on hover handling. If you’re using the Tidbyt or some similar pixel-displaying device, try out indiepixel! It’s niche and has required a silly amount of effort to generate a glorified weather clock in my apartment, but it was a fun time chasing another interest.
Reading Whether it’s cryptocurrency scammers mining with FOSS compute resources or Google engineers too lazy to design their software properly or Silicon Valley ripping off all the data they can get their hands on at everyone else’s expense… I am sick and tired of having all of these costs externalized directly into my fucking face. Drew DeVault on the annoyance and cost of AI scrapers. I share some of that pain: Val Town is routinely hammered by some AI company’s poorly-coded scraping bot. I think it’s like this for everyone, and it’s hard to tell if AI companies even care that everyone hates them. And perhaps most recently, when a person who publishes their work under a free license discovers that work has been used by tech mega-giants to train extractive, exploitative large language models? Wait, no, not like that. Molly White wrote a more positive article about the LLM scraping problem, but I have my doubts about its positivity. For example, she suggests that Wikimedia’s approach with “Wikimedia Enterprise” gives LLM companies a way to scrape the site without creating too much cost. But that doesn’t seem like it’s working. The problem is that these companies really truly do not care. Harberger taxes represent an elegant theoretical solution that fails in practice for immobile property. Just as mobile home residents face exploitation through sudden ground rent increases, property owners under a Harberger system would face similar hold-up problems. This creates an impossible dilemma: pay increasingly burdensome taxes or surrender investments at below-market values. Progress and Poverty, a blog about Georgism, has this post about Herberger taxes, which are a super neat idea. The gist is that you would be in charge of saying how much your house is worth, but the added wrinkle is that by saying a price you are bound to be open to selling your house at that price. So if you go too low, someone will buy it, or too high, and you’re paying too much in taxes. It’s clever but doesn’t work, and the analysis points to the vital difference between housing and other goods: that buying, selling, and moving between houses is anything but simple. I’ve always been a little skeptical of the line that the AI crowd feels contempt for artists, or that such a sense is particularly widespread—because certainly they all do not!—but it’s hard to take away any other impression from a trend so widely cheered in its halls as AI Ghiblification. Brian Merchant on the OpenAI Studio Ghibli ‘trend’ is a good read. I can’t stop thinking that AI is in danger of being right-wing coded, the examples of this, like the horrifying White House tweet mentioned in that article, are multiplying. I feel bad when I recoil to innocent usage of the tool by good people who just want something cute. It is kind of fine, on the micro level. But with context, it’s so bad in so many ways. Already the joy and attachment I’ve felt to the graphic style is fading as more shitty Studio Ghibli knockoffs have been created in the last month than in all of the studio’s work. Two days later, at a state dinner in the White House, Mark gets another chance to speak with Xi. In Mandarin, he asks Xi if he’ll do him the honor of naming his unborn child. Xi refuses. Careless People was a good read. It’s devastating for Zuckerberg, Joel Kaplan, and Sheryl Sandberg, as well as a bunch of global leaders who are eager to provide tax loopholes for Facebook. Perhaps the only person who ends the book as a hero is President Obama, who sees through it all. In a March 26 Slack message, Lavingia also suggested that the agency should do away with paper forms entirely, aiming for “full digitization.” “There are over 400 vet-facing forms that the VA supports, and only about 10 percent of those are digitized,” says a VA worker, noting that digitizing forms “can take years because of the sensitivity of the data” they contain. Additionally, many veterans are elderly and prefer using paper forms because they lack the technical skills to navigate digital platforms. “Many vets don’t have computers or can’t see at all,” they say. “My skin is crawling thinking about the nonchalantness of this guy.” Perhaps because of proximity, the story that Sahil Lavingia has been working for DOGE seems important. It was a relief when a few other people noticed it and started retelling the story to the tech sphere, like Dan Brown’s “Gumroad is not open source” and Ernie Smith’s “Gunkroad”, but I have to nitpick on the structure here: using a non-compliant open source license is not the headline, collaborating with fascists and carelessly endangering disabled veterans is. Listening Septet by John Carroll Kirby I saw John Carroll Kirby play at Public Records and have been listening to them constantly ever since. The music is such a paradox: the components sound like elevator music or incredibly cheesy jazz if you listen to a few seconds, but if you keep listening it’s a unique, deep sound. Sierra Tracks by Vega Trails More new jazz! Mammoth Hands and Portico Quartet overlap with Vega Trails, which is a beautiful minimalist band. Watching This short video with John Wilson was great. He says a bit about having a real physical video camera, not just a phone, which reminded me of an old post of mine, Carrying a Camera.
I used to make little applications just for myself. Sixteen years ago (oof) I wrote a habit tracking application, and a keylogger that let me keep track of when I was using a computer, and generate some pretty charts. I’ve taken a long break from those kinds of things. I love my hobbies, but they’ve drifted toward the non-technical, and the idea of keeping a server online for a fun project is unappealing (which is something that I hope Val Town, where I work, fixes). Some folks maintain whole ‘homelab’ setups and run Kubernetes in their basement. Not me, at least for now. But I have been tiptoeing back into some little custom tools that only I use, with a focus on just my own computing experience. Here’s a quick tour. Hammerspoon Hammerspoon is an extremely powerful scripting tool for macOS that lets you write custom keyboard shortcuts, UIs, and more with the very friendly little language Lua. Right now my Hammerspoon configuration is very simple, but I think I’ll use it for a lot more as time progresses. Here it is: hs.hotkey.bind({"cmd", "shift"}, "return", function() local frontmost = hs.application.frontmostApplication() if frontmost:name() == "Ghostty" then frontmost:hide() else hs.application.launchOrFocus("Ghostty") end end) Not much! But I recently switched to Ghostty as my terminal, and I heavily relied on iTerm2’s global show/hide shortcut. Ghostty doesn’t have an equivalent, and Mikael Henriksson suggested a script like this in GitHub discussions, so I ran with it. Hammerspoon can do practically anything, so it’ll probably be useful for other stuff too. SwiftBar I review a lot of PRs these days. I wanted an easy way to see how many were in my review queue and go to them quickly. So, this script runs with SwiftBar, which is a flexible way to put any script’s output into your menu bar. It uses the GitHub CLI to list the issues, and jq to massage that output into a friendly list of issues, which I can click on to go directly to the issue on GitHub. #!/bin/bash # <xbar.title>GitHub PR Reviews</xbar.title> # <xbar.version>v0.0</xbar.version> # <xbar.author>Tom MacWright</xbar.author> # <xbar.author.github>tmcw</xbar.author.github> # <xbar.desc>Displays PRs that you need to review</xbar.desc> # <xbar.image></xbar.image> # <xbar.dependencies>Bash GNU AWK</xbar.dependencies> # <xbar.abouturl></xbar.abouturl> DATA=$(gh search prs --state=open -R val-town/val.town --review-requested=@me --json url,title,number,author) echo "$(echo "$DATA" | jq 'length') PR" echo '---' echo "$DATA" | jq -c '.[]' | while IFS= read -r pr; do TITLE=$(echo "$pr" | jq -r '.title') AUTHOR=$(echo "$pr" | jq -r '.author.login') URL=$(echo "$pr" | jq -r '.url') echo "$TITLE ($AUTHOR) | href=$URL" done Tampermonkey Tampermonkey is essentially a twist on Greasemonkey: both let you run your own JavaScript on anybody’s webpage. Sidenote: Greasemonkey was created by Aaron Boodman, who went on to write Replicache, which I used in Placemark, and is now working on Zero, the successor to Replicache. Anyway, I have a few fancy credit cards which have ‘offers’ which only work if you ‘activate’ them. This is an annoying dark pattern! And there’s a solution to it - CardPointers - but I neither spend enough nor care enough about points hacking to justify the cost. Plus, I’d like to know what code is running on my bank website. So, Tampermonkey to the rescue! I wrote userscripts for Chase, American Express, and Citi. You can check them out on this Gist but I strongly recommend to read through all the code because of the afore-mentioned risks around running untrusted code on your bank account’s website! Obsidian Freeform This is a plugin for Obsidian, the notetaking tool that I use every day. Freeform is pretty cool, if I can say so myself (I wrote it), but could be much better. The development experience is lackluster because you can’t preview output at the same time as writing code: you have to toggle between the two states. I’ll fix that eventually, or perhaps Obsidian will add new API that makes it all work. I use Freeform for a lot of private health & financial data, almost always with an Observable Plot visualization as an eventual output. For example, when I was switching banks and one of the considerations was mortgage discounts in case I ever buy a house (ha 😢), it was fun to chart out the % discounts versus the required AUM. It’s been really nice to have this kind of visualization as ‘just another document’ in my notetaking app. Doesn’t need another server, and Obsidian is pretty secure and private.
More in programming
I’ve written about how I don’t love the idea of overriding basic computing controls. Instead, I generally favor opting to respect user choice and provide the controls their platform does. Of course, this means platforms need to surface better primitives rather than supplying basic ones with an ability to opt out. What am I even talking about? Let me give an example. The Webkit team just shipped a new API for <input type=color> which provides users the ability to pick colors with wide gamut P3 and alpha transparency. The entire API is just a little bit of declarative HTML: <label> Select a color: <input type="color" colorspace="display-p3" alpha> </label> From that simple markup (on iOS) you get this beautiful, robust color picker. That’s a great color picker, and if you’re choosing colors a lot on iOS respectively and encountering this particular UI a lot, that’s even better — like, “Oh hey, I know how to use this thing!” With a picker like that, how many folks really want additional APIs to override that interface and style it themselves? This is the kind of better platform defaults I’m talking about. A little bit of HTML markup, and boom, a great interface to a common computing task that’s tailored to my device and uniform in appearance and functionality across the websites and applications I use. What more could I want? You might want more, like shoving your brand down my throat, but I really don’t need to see BigFinanceCorp Green™️ as a themed element in my color or date picker. If I could give HTML an aspirational slogan, it would be something along the lines of Mastercard’s old one: There are a few use cases platform defaults can’t solve, for everything else there’s HTML. Email · Mastodon · Bluesky
Today’s my last day at Carta, where I got the chance to serve as their CTO for the past two years. I’ve learned so much working there, and I wanted to end my chapter there by collecting my thoughts on what I learned. (I am heading somewhere, and will share news in a week or two after firming up the communication plan with my new team there.) The most important things I learned at Carta were: Working in the details – if you took a critical lens towards my historical leadership style, I think the biggest issue you’d point at is my being too comfortable operating at a high level of abstraction. Utilizing the expertise of others to fill in your gaps is a valuable skill, but–like any single approach–it’s limiting when utilized too frequently. One of the strengths of Carta’s “house leadership style” is expecting leaders to go deep into the details to get informed and push pace. What I practiced there turned into the pieces on strategy testing and developing domain expertise. Refining my approach to engineering strategy – over the past 18 months, I’ve written a book on engineering strategy (posts are all in #eng-strategy-book), with initial chapters coming available for early release with O’Reilly next month. Fingers crossed, the book will be released in approximately October. Coming into Carta, I already had much of my core thesis about how to do engineering strategy, but Carta gave me a number of complex projects to practice on, and excellent people to practice with: thank you to Dan, Shawna and Vogl in particular! More on this project in the next few weeks. Extract the kernel – everywhere I’ve ever worked, teams have struggled understanding executives. In every case, the executives could be clearer, but it’s not particularly interesting to frame these problems as something the executives need to fix. Sure, that’s true they could communicate better, but that framing makes you powerless, when you have a great deal of power to understand confusing communication. After all, even good communicators communicate poorly sometimes. Meaningfully adopting LLMs – a year ago I wrote up notes on adopting LLMs in your products, based on what we’d learned so far. Since then, we’ve learned a lot more, and LLMs themselves have significantly improved. Carta has been using LLMs in real, business-impacting workflows for over a year. That’s continuing to expand into solving more complex internal workflows, and even more interestingly into creating net-new product capabilities that ought to roll out more widely in the next few months (currently released to small beta groups). This is the first major technology transition that I’ve experienced in a senior leadership role (since I was earlier in my career when mobile internet transitioned from novelty to commodity). The immense pressure to adopt faster, combined with the immense uncertainty if it’s a meaningful change or a brief blip was a lot of fun, and was the inspiration for this strategy document around LLM adoption. Multi-dimensional tradeoffs – a phrase that Henry Ward uses frequent is that “everyone’s right, just at a different altitude.” That idea resonates with me, and meshes well with the ideas of multi-dimensional tradeoffs and layers of context that I find improve decision making for folks in roles that require making numerous, complex decisions. Working at Carta, these ideas formalized from something I intuited into something I could explain clearly. Navigators – I think our most successful engineering strategy at Carta was rolling out the Navigator program, which ensured senior-most engineers had context and direct representation, rather than relying exclusively on indirect representation via engineering management. Carta’s engineering managers are excellent, but there’s always something lost as discussions extend across layers. The Navigator program probably isn’t a perfect fit for particularly small companies, but I think any company with more than 100-150 engineers would benefit from something along these lines. How to create software quality – I’ve evolved my thinking about software quality quite a bit over time, but Carta was particularly helpful in distinguishing why some pieces of software are so hard to build despite having little-to-no scale from a data or concurrency perspective. These systems, which I label as “high essential complexity”, deserve more credit for their complexity, even if they have little in the way of complexity from infrastructure scaling. Shaping eng org costs – a few years ago, I wrote about my mental model for managing infrastructure costs. At Carta, I got to refine my thinking about engineering salary costs, with most of those ideas getting incorporated in the Navigating Private Equity ownership strategy, and the eng org seniority mix model. The three biggest levers are (1) “N-1 backfills”, (2) requiring a business rationale for promotions into senior-most levels, and (3) shifting hiring into cost efficient hiring regions. None of these are the sort of inspiring topics that excite folks, but they are all essential to the long term stability of your organization. Explaining engineering costs to boards/execs – Similarly, I finally have a clear perspective on how to represent R&D investment to boards in the same language that they speak in, which I wrote up here, and know how to do it quickly without relying on any manually curated internal datasets. Lots of smaller stuff, like the no wrong doors policy for routing colleagues to appropriate channels, how to request headcount in a way that is convincing to executives, Act Two rationales for how people’s motivations evolve over the course of long careers (and my own personal career mission to advance the industry, why friction isn’t velocity even though many folks act like it is. I’ve also learned quite a bit about venture capital, fund administration, cap tables, non-social network products, operating a multi-business line company, and various operating models. Figuring out how to sanitize those learnings to share the interesting tidbits without leaking internal details is a bit too painful, so I’m omitting them for now. Maybe some will be shareable in four or five years after my context goes sufficiently stale. As a closing thought, I just want to say how much I’ve appreciated the folks I’ve gotten to work with at Carta. From the executive team (Ali, April, Charly, Davis, Henry, Jeff, Nicole, Vrushali) to my directs (Adi, Ciera, Dan, Dave, Jasmine, Javier, Jayesh, Karen, Madhuri, Sam, Shawna) to the navigators (there’s a bunch of y’all). The people truly are always the best part, and that was certainly true at Carta.
Some major updates to our open-source Automerge library, an introduction to Sketchy Calendars, and a peek at our work on collaborative game development. Also some meta content—a refreshed website, and a talk about how we work.
Test UI outcomes, not API requests. Mock network calls in setup, but assert on what users actually see and experience, not implementation details.
Do you feel that the number of applications needed to land a role has skyrocketed? If so, your instincts are correct. According to a Workday Global Workforce Report in September 2024, job applications are growing at a rate four times faster than job openings. This growth is fuelled by a tight job market as well as the new availability of remote work and online job boards. It’s also one of the results of improved generative AI. Around half of all job seekers use AI tools to create their resumes or fill out applications. More than that, a 2024 survey found that 29 percent of applicants were using AI tools to complete skills tests, while 26 percent employed AI tools to mass apply to positions, regardless of fit or qualifications. This never-before-seen flood of applications poses new hardships for both job candidates and recruiters. Candidates must ensure that their applications stand out enough from the pile to receive a recruiter’s attention. Recruiters, meanwhile, are struggling to manage the sheer number of resumes they receive, and winnow through heaps of irrelevant or unqualified applicants to find the ones they need. These problems worsen if you’re an overseas candidate hoping to find a role in Japan. Japan is a popular country for migrants, thereby increasing the competition for each open position. In addition, recruiters here have set expectations and criteria, some of which can be triggered unknowingly by candidates unfamiliar with the Japanese market. With all this in mind, how can you ensure your resume stands out from the crowd—and is there anything else you can do to pass the screening stage? I interviewed nine recruiters, both external and in-house, to learn how applicants can increase their chances of success. Below are their detailed suggestions on improving your resume, avoiding Japan-specific red flags, and persisting even in the face of rejection. The competition The first questions I asked each recruiter were: How many resumes do you review in a month? How long does it take you to review a resume? Some interviewees work for agencies or independently, while others are employed by the companies they screen applicants for. Surprisingly, where they work doesn’t consistently affect how many resumes they receive. What does affect their numbers is whether they accept candidates from overseas. One anonymous contributor stated the case plainly: “The volume of applications depends on whether the job posting targets candidates in Japan or internationally.” In Japan: we receive around 20–100+ applications within the first three days. Outside of Japan: a single job posting can attract 200–1,000 applications within three days. ”[Because] we are generally only open to current residents of Japan, our total applicant count is around 100 or so in a month,” said Caleb McClain, who is both a Senior Software Engineer and a hiring manager at Lunaris. “In the past, when we accepted applications from abroad it was much higher, though I unfortunately don’t have stats for that period. It was unmanageable for a single person (me) reviewing the applications, though! “Given that I deal with 100 or so per month, I probably spend a bit more time than others screening applications, but it depends. I’ll give every candidate a quick read through within a minute or so and, if I didn’t find a reason to immediately reject them, I’ll spend a few more minutes reading about their experience more deeply. I’ll check out the companies they have listed for their experience if I’m not familiar with them and, if they have a Github or personal projects listed, I’ll also spend a few minutes checking those out.” For companies that accept overseas candidates, the workload is greater. Laine Takahashi, a Talent Acquisition employee at HENNGE, estimated that every month they receive around 200 completed applications for engineering mid-career roles and 270 applications for their Global Internship program. Since their application process starts with a coding test as well as a resume and cover letter, it can take up to two weeks to review, score, and respond to each application. Clement Chidiac, Senior Technical Recruiter at Mercari, explained that the number of resumes he reviews monthly varies widely. “As an example, one of the current roles I am working on received 250+ applications in three weeks. Typically a recruiter at Mercari can work from 5–20 positions at a time, so this gives you an idea.” He also said that his initial quick scan of each resume might take between 5–30 seconds. External recruiters process resumes at a similar rate. Edmund Ho, Principal Consultant for Talisman Corporation, works with around 15 clients a month. To find them, he looks at 20–30 resumes a day, or 600–700 a month, and can only spend 30 seconds to 2 minutes on each one before coming to a decision. Axel Algoet, founder and CEO of InnoHyve, only reviews 200 resumes a month—but “if you count LinkedIn profiles, it’s probably around 1,000.” Why LinkedIn? “I usually start by looking at LinkedIn—the companies they’ve worked at and the roles they’ve had,” Algoet explained. “From there, I can quickly tell whether I’m open to talking with them or not. Since I focus on a very specific segment of roles, I can rapidly identify if a candidate might be a fit for my clients.” Applicant Tracking Systems (ATS) Given the sheer volume of resumes to review and respond to, it’s not surprising that companies are using Applicant Tracking Systems. What’s more unexpected is how few recruiters personally use an ATS or AI when evaluating candidates. Both Ho and Algoet reported that though a high percentage of their clients use an ATS—as many as 90 percent, according to Ho—they themselves don’t use one. Ho in particular emphasized that he manually reads every resume he receives. Lunaris doesn’t use an ATS, “unless you count Notion,” joked McClain. “Open to recommendations!” Koji Hamane, Vice President of Human Resources at KOMOJU, said, “Up to 2023, we were managing the pipeline on a spreadsheet basis, and you cannot do it anymore with 3,000 applications [a year]. So it’s more effective and efficient in terms of tracking where each applicant sits in the recruiting process, but it also facilitates communication among [the members of] the interview panel.” The ATS KOMOJU uses is Workable. “Workable, I mean, you know, it works,” Hamane joked. “It’s much better than nothing. . . . Workable actually shows the valid points of the candidates, highlights characteristics, and evaluates the fit for the required positions, like from a 0 to 100 point basis. It helps, but actually you need to go through the details anyway, to properly assess the candidates.” Chidiac explained that Mercari also uses Workable, which has a feature that matches keywords from the job description to the resume, giving the resume a score. “I’ve never made a decision based on that,” said Chidiac. “It’s an indicator, but it’s not accurate enough yet to use it as a decision-making tool.” For example, it doesn’t screen out non-Japanese speakers when Japanese is a requirement for the role. I think these [ATS] tools are going to be better, and they’re going to work. I think it’s a good idea to help junior recruiters. But I think it has to be used as a ‘decision helper,’ not a decision-making tool. There’s also an element of ethics—do you want to be screened out by a robot? HENNGE uses a different ATS, Greenhouse, mostly to communicate with candidates and send them the results of their application. “ Everything they submit,” said Sonam Choden, HENNGE’s Software Engineer Recruiter, “is actually manually checked by somebody in our team. It’s not that everything is automated for the coding test—the bot only checks if they meet the minimum score. Then there is another [human] screener that will actually look over the test itself. If they pass the coding test, then we have another [human] screener looking through each and every document, both the resume and the cover letter.” How to format your resume The good news is that, according to our interviewees, passing the resume screening doesn’t involve trying to master ATS algorithms. However, since many recruiters are manually evaluating a high number of resume every day, they can spend at most only a few minutes on each one. That’s why it’s critical to make your resume stand out positively from the rest. You can see tips on formatting and good practices in our article on the subject, but below recruiters offer detailed explanations of exactly what they’re looking for—and, importantly, what red flags lead to rejection. Red flags The biggest red flags called out by recruiters are frequent job changes, not having skills required by the position, applications from abroad when no visa support is available, mismatches in salary expectations, and lack of required Japanese language ability. Frequent job changes Jumpiness. Job-hopping. Career-switching. Although they had different names for it, nearly everyone listed frequent job changes as the number one red flag on a candidate’s resume—at least, when applying to jobs in Japan. “There’s a term HR in Japan uses: ‘Oh, this guy is jumpy,’” Clement Chidiac told me. When he asked what they meant by that, they told him it referred to a candidate who had only been in their last job for two years or less. “And my first reaction was like, ‘Is that a bad thing?’ I think in the US, and in most tech companies, people change over every two to three years. I remember at my university in France, I was told you need to change your job externally or internally every three years to grow. But in Japan, there’s still the element of loyalty, right?” It’s changing a little bit, but when I have a candidate, a good candidate, that has had four jobs in the past ten years, I know I’m going to get questioned. . . . If I get a candidate that’s changed jobs three times in the past three years, they’re not likely to pass the screening, especially if they’re overseas. “Which is fair, right?” he added. “Because it’s a bit expensive, it’s a bit of a risk, and [it takes] a bit of time.” Why do Japanese companies feel so strongly on this issue? Some of it is simply history—lifetime employment at a single company was the Japanese ideal until quite recently. But as Chidiac pointed out, hiring overseas candidates represents additional investments in both money and time spent navigating the visa system, so it makes sense for Japanese companies to move more cautiously when doing so. Sayaka Sasaki, who was previously employed as a Sourcing Specialist by Tech Japan Inc., told me that recruiters attempt to use past job history to foresee the future. “A lack of consistency in career history can also lead to rejection,” she said. “Recruiters can often predict a candidate’s future career plans and job-switching tendencies based on their past job-change patterns.” Koji Hamane has another reason for considering job tenure. “When you try to leave some achievement or visible impact, [you have to] take some time in the same job, in the same company. So from that perspective, the tenure of each position on a resume really matters. Even though you say, ‘I have this capability and I have this strength,’ your tenure at each company is very short, and [you] don’t leave an impact on those workplaces.” In this sense, Hamane is not evaluating loyalty for its own sake, but considering tenure as a variable to assess the reproducibility of meaningful achievement. For him, achievement and impact—rather than tenure length itself—are the true signals of qualities such as leadership and resilience. Long-time or regular freelancers may face similar scrutiny. Though Chidiac is reluctant to call freelancing a red flag, he acknowledged that it can cause problems. “[With] an engineer that’s been doing freelance for the past three or four years, I know I’m going to get pushback from the hiring team, because they might have worked on three-, four-, five-month projects. They might not have the depth of knowledge that companies on a large scale might want to hire.” Also my question is, if that person has been working on their own for three or four years, how are they going to work in the team? How long are they going to stay with us? Are they going to be happy being part of a company and then maybe having to come to the office, that kind of thing? He gave an example: “If you get 100 applicants for backend engineer roles, it’s sad, but you’re going to go with the ones that fit the most traditional background. If I’m hiring and I’m getting five candidates from PayPay . . . I might prioritize these people as opposed to a freelancer that’s based out of Spain and wants to relocate to Japan, because there are a lot of question marks. That’s the reality of the candidate pool. “Now, if the freelancer in Spain has the exact experience that I want, and I don’t have other applicants, then yeah, of course I’ll talk to that person. I’ll take time to understand [their reasons].” How to “fix” job-hopping on your resume If you have changed jobs frequently, is rejection guaranteed? Not necessarily. These recruiters also offered a host of tips to compensate for job-hopping, freelancing stints, or gaps in your work history. The biggest tip: include an explanation on your resume. Edmund Ho advises offering a “reason for leaving” for short-term jobs, defining short-term as “less than three years.” For example, if the job was a limited contract role, then labelling it as such will prevent Japanese companies from drawing the conclusion that you left prematurely. Lay-offs and failed start-ups will also be looked upon more benevolently than simply quitting. In addition, Ho suggested that those with difficult resumes avail themselves of an agent or recruiter. Since the recruiter will contact the company directly, they have the chance to advocate and explain your job history better than the resume alone can. Sasaki also feels that explanations can help, but added a caveat: “Being honest about what you did during a gap period is not a bad thing. However, it is important to present it in a positive light. For example, if you traveled abroad or spent time at your family home during the gap period, you could write something like this: ‘Once I start a new job, it will be difficult to take a long vacation. So, I took advantage of this break to visit [destination], which I had always dreamed of seeing. Experiencing [specific highlight] was a lifelong goal, and it helped me refresh myself while boosting my motivation for work.’ “If the gap period lasted for more than a year, it is necessary to provide a convincing explanation for the hiring manager. For instance, you could write, ‘I used this time to enhance my skills by studying [specific subject] and preparing for [certification].’ If you have actually obtained a qualification, that would be a perfect way to present your time productively.” Hamane answered the question quite differently. “Do you gamble?” he asked me. He went on: “ When I say ‘gamble,’ ultimately recruiting is decision-making under uncertainty, right? It comes with risks. But the most important question is, what are the downside risks and upside risks?” “In the game of hiring,” Hamane explained, “employers are looking for indicators of future performance. Tenure, to me, is not inherently valuable, but serves as a variable to assess whether a candidate had the opportunity to leave a meaningful impact. It’s not about loyalty or raw length of time, but about whether qualities like resilience or leadership had the chance to emerge. Those qualities often require time. However, I don’t judge the number of years on its own—what matters is whether there is evidence of real contributions.” A shorter tenure with clear impact can be just as strong a signal as longer service. That’s why I view tenure not categorically, but contextually—as one indicator among others. If possible, then, a candidate should focus on highlighting their work contributions and unique strengths in their resume, which can counterbalance the perceived “downside risk” of job-hopping. Incompatibility with the job description Most other red flags can be categorized as “incompatible with the job description.” This includes: Not possessing the required skills Applying from abroad when the position doesn’t offer visa support Mismatch in salary expectations Not speaking Japanese Many of the resumes recruiters receive are wholly unsuited for the position. Hamane estimated that 70 percent of the resumes his department reviews are essentially “random applications.” Almost all the applications are basically not qualified. One of the major reasons why is the Internet. The Internet enables us to apply for any job from anywhere, right? So there are so many applications with no required skills. . . . From my perspective, they are applying on a batch basis, like mass applications. Even if the candidate has the required job skills, if they’re overseas and the position doesn’t offer visa support, their resume almost certainly won’t pass. Caleb McClain, whose company is currently hiring only domestically, said, “The most common reason [for rejection] is the person is applying from abroad. . . . After that, if there’s just a clear skills mismatch, we won’t move forward with them.” Axel Algoet pointed out that nationality can be a problem even if the company is open to hiring from overseas. “I support many companies in the space, aerospace, and defense industries,” he said, “and they are not allowed to hire candidates from certain countries.” It’s important to comprehend any legal issues surrounding sensitive industries before applying, to save both your own and the company’s time. He also mentioned that, while companies do look for candidates with experience at top enterprises, a prestigious background can actually be a red flag—-mostly in terms of compensation. Japanese tech companies on average pay lower wages than American businesses, and a mismatch in expectations can become a major stumbling block in the application process overall. “Especially [for] candidates coming from companies like Indeed or some foreign firms,” Algoet said, “if I know I won’t be able to match or beat their current salary, I tell them upfront.” Not speaking Japanese is another common stumbling block. Companies have different expectations of candidates when it comes to Japanese language ability. Algoet said that, although in his own niche Japanese often isn’t required at all, a Japanese level below JLPT N2 can be a problem for other roles. Sasaki agreed that speaking Japanese to at least the JLPT N3 level would open more doors. Anticipating potential rejection points If you can anticipate why recruiters might reject you, you can structure your resume accordingly, highlighting your strengths while deemphasizing any weak points. For example, if you don’t live in Japan but do speak Japanese, it’s important to bring attention to that fact. “Something that’s annoying,” said Chidiac, “that I’m seeing a lot from a hiring manager point of view, is that they sort of anticipate or presume things. . . . ‘That person has only been in Japan for a year, they can’t speak Japanese.’ But there are some people that have been [going to] Japanese school back home.” That’s why he urges candidates to clearly state both their language ability and their connections to Japan in their resume whenever possible. Chidiac also mentioned seniority issues. “It’s important that you highlight any elements of seniority.” However, he added, “Seniority means different things depending on the environment.” That’s why context is critical in your resume. If you’ve worked for a company in another country or another industry, the recruiter may not intuitively know much about the scale or complexity of the projects you’ve worked on. Without offering some context—the size of the project, the size of the team, the technologies involved, etc.—it’s difficult for recruiters to judge. If you contextualize your projects properly, though, Chidiac believes that even someone with relatively few years of experience may still be viewed favorably for higher roles. If you’ve led a very strong project, you might have the seniority we want. Finally, Edmund Ho suggested an easy trick for those without a STEM degree: just put down the university you graduated from, and not your major. “It’s cheating!” he said with a chuckle. Green flags Creating a great resume isn’t just about avoiding pitfalls. Your resume may also be missing some of the green flags recruiters get excited to see, which can open doors or lead to unexpected offers. Niche skills Niche skills were cited by several as not only being valuable in and of themselves, but also being a great way to open otherwise closed doors. Even when the job description doesn’t call for your unusual ability or experience, it’s probably worth including them in your resume. “I’ll of course take into consideration the requirements as written in our current open listings,” said McClain, “as that represents the core of what we are looking for at any given time. However, I also try to keep an eye out for interesting individuals with skills or experience that may benefit us in ways we haven’t considered yet, or match well with projects that aren’t formally planned but we are excited about starting when we have the time or the right people.” Chidiac agrees that he takes special note of rare skills or very senior candidates on a resume. “We might be able to create an unseeable headcount to secure a rare talent. . . . I think it’s important to have that mindset, especially for niche areas. Machine learning is one that comes to mind, but it could also be very senior [candidates], like staff level or principal level engineers, or people coming from very strong companies, or people that solve problems that we want to solve at the moment, that kind of thing.” I call it the opportunistic approach, like the unusual path, but it’s important to have that in mind when you apply for a company, because you might not be a fit for a role now, but you might not be aware that a role is going to open soon. Sasaki pointed out that niche skills can compensate for an otherwise relatively weak resume, or one that would be bypassed by more traditional Japanese companies. “If the company you are applying to is looking for a niche skill set that only you possess, they will want to speak with you in an interview. So don’t lose hope!” Tailoring to the job description “I don’t think there’s a secret recipe to automatically pass the resume screening, because at the end of the day, you need to match the job, right?” said Chidiac. “But I’ve seen people that use the same resume for different roles, and sometimes it’s missing [relevant] experience or specific keywords. So I think it’s important to really read the job description and think about, ‘Okay, these are all the main skills they want. Let me highlight these in some way.’” If you’re a cloud infrastructure engineer, but you’ve done a lot of coding in the past, or you use a specific technology but it doesn’t show on your CV, you may be automatically rejected either by the recruiter or by the [ATS]. But if you make sure that, ‘Oh yeah, I’ve seen the need for coding skill. I’m going to add that I was a software engineer when I started and I’m doing coding on my side project,’ that will help you with the screening. It’s not necessary to entirely remake your resume each time, Chidiac believes, but you should at least ensure that at the top of the resume you highlight the skills that match the job description. Connections to Japan While most of this advice would be relevant anywhere in the world, recruiters did offer one additional tip for applying in Japan—emphasizing your connection to the country. “Whenever a candidate overseas writes a little thing about any ties to Japan, it usually helps,” said Chidiac. For example, he believes that it helps to highlight your Japanese language ability at the top of your resume. [If] someone writes like, ‘I want to come to Japan,’ ‘I’ve been going to Japanese school for the last five years,’ ‘I’ve got family in Japan,’ . . . that kind of stuff usually helps. Laine Takahashi confirmed that HENNGE shows extra interest in those kinds of candidates. “Either in the cover letter or the CV,” she said, “if they’re not living in Japan, we want them to write about their passion for coming to Japan.” Ho went so far as to state that every overseas candidate he’d helped land a job in Japan had either already learned some Japanese, or had an interest in Japanese culture. Tourists who’d just enjoyed traveling in Japan were less successful, he’d found. How important is a cover letter? Most recruiters had similar advice for candidates, but one serious point of contention arose: cover letters. Depending on their company and hiring style, interviewees’ opinions ranged widely on whether cover letters were necessary or helpful. Cover letters aren’t important “I was trying to remember the last time I read a cover letter,” said Clement Chidiac, “and I honestly don’t think I’ve ever screened an application based on the cover letter.” Instead, Mercari typically requests a resume and poses some screening questions. Chidiac thought this might be a controversial opinion to take, but it was echoed strongly by around half of the other interviewees. When applying to jobs in Japan, there’s no need to write a cover letter, Edmund Ho told me. “Companies in Japan don’t care!” He then added, “One company, HENNGE, uses cover letters. But you don’t need,” he advised, “to write a fancy cover letter.” “I never ask for cover letters,” said Axel Algoet. “Instead, I usually set up a casual twenty-minute call between the hiring manager and the candidate, as a quick intro to decide if it’s worth moving forward with the interview process.” Getting to skip the cover letter and go straight to an early-stage interview is a major advantage Algoet is able to offer his candidates. “That said,” he added, “if a candidate is rejected at the screening stage and I feel the client is making a mistake, we sometimes work on a cover letter together to give it another shot.” Cover letters are extremely important According to Sayaka Sasaki, though, Japanese companies don’t just expect cover letters—they read them quite closely. “Some people may find this hard to believe,” said Sasaki, “but many Japanese companies carefully analyze aspects of a candidate’s personality that cannot be directly read from the text of a cover letter. They expect to see respect, humility, enthusiasm, and sincerity reflected in the writing.” Such companies also expect, or at least hope for, brevity and clarity. “Long cover letters are not a good sign,” said Koji Hamane. “You need to be clear and concise.” He does appreciate cover letters, though, especially for junior candidates, who have less information on their resume. “It supplements [our knowledge of] the candidate’s objectives, and helps us to verify the fit between the candidate’s motivation and the job and the company.” Caleb McClain feels strongly that a good cover letter is the best way for a candidate to stand out from a crowd. “After looking at enough resumes,” he said, “you start to notice similarities and patterns, and as the resume screener I feel a bit of exhaustion over trying to pick out what makes a person unique or better-suited for the position than another.” A well-written and personal cover letter that expresses genuine interest in joining ‘our’ team and company and working on ‘our’ projects will make you stand out and, assuming you meet the requirements otherwise, I will take that interest into serious consideration. “For example,” McClain continued, “we had an applicant in the past who wrote about his experience using our e-commerce site, SolarisJapan, many years ago, and his positive impressions of shopping there. Others wrote about their interests which clearly align with our businesses, or about details from our TokyoDev company profile that appealed to them.” McClain urged candidates to “really tie your experience and interests into what the company does, show us why you’re the best fit! Use the cover letter to stand out in the crowd and show us who you are in ways that a standard resume cannot. If you have interesting projects on Github or blogs on technical topics, share them! But of course,” he added, “make sure they are in a state where you’d want others to read them.” What to avoid in your cover letter “However,” McClain also cautioned, “[cover letters are] a double-edged sword, and for as many times as they’ve caused an application to rise to the top, they’ve also sunk that many.” For this reason, it’s best not to attach a cover letter unless one is specifically requested. Since cover letters are extremely important to some recruiters, however, you should have a good one prepared in advance—and not one authored by an AI tool. “I sometimes receive cover letters,” McClain told me, “that are very clearly written by AI, even going so far as to leave the prompt in the cover letter. Others simply rehash points from their resume, which is a shame and feels like a waste. This is your chance to really sell yourself!” He wasn’t the only recruiter who frowned on using AI. “Avoid simply copying and pasting AI-generated content into your cover letter,” Sasaki advised. “At the very least, you should write the base structure yourself. Using AI to refine your writing is acceptable, but hiring managers tend to dislike cover letters that clearly appear to be AI-written.” Laine Takahashi and Sonam Choden at HENNGE have also received their share of AI-generated letters. Sometimes, Choden explained, the use of AI is blatantly obvious, because the places where the company or applicant’s name should be written aren’t filled out. That doesn’t mean they’re opposed to all use of AI, though. “[The screeners] do not have a problem with the usage of AI technology. It’s just that [you should] show a bit more of your personality,” Takahashi said. She thinks it’s acceptable to use AI “just for making the sentences a bit more pretty, for example, but the story itself is still yours.” A bigger mistake would be not writing a cover letter at all. “There are cases,” Takahashi explained, “where perhaps the candidate thought that we actually don’t look at or read the cover letter.” They sent the CV, and then the cover letter was like, ‘Whatever, you’re not going to read this anyway.’ That’s an automatic fail from our side. “We do understand,” said Choden, “that most developers now think cover letters are an outdated type of process. But for us, there is a lot of benefit in actually going through with the cover letter, because it’s really hard to judge someone by one piece like a resume, right? So the cover letter is perfect to supplement with things that you might not be able to express in a one-page CV.” Other tips for success The interviewees offered a host of other tips to help candidates advance in the application process. Recruiters vs job boards There are pros and cons to working with a recruiter as opposed to applying directly. Partnering with a recruiter can be a complex process in its own right, and candidates should not expect recruiters to guarantee a specific placement or job. Edmund Ho pointed out some of the advantages of working with a recruiter from the start of your job search. Not only can they help fix your resume, or call a company’s HR directly if you’re rejected, but these services are free. After all, external recruiters are paid only if they successfully place you with a company. Axel Algoet also recommended candidates find a recruiter, but he offered a few caveats to this general advice. “Many candidates are unaware of the candidate ownership rule—which means that when a recruiter submits your application, they ‘own’ it for the next 12–18 months. There’s nothing you can do about it after that point.” By that, he means that the agency you work with will be eligible for a fee if you are hired within that timeframe. Other agencies typically won’t submit your application if it is currently “owned” by another. This affects TokyoDev as well: if you apply to a company with a recruiter, and then later apply to another role at that company via TokyoDev within 12 months of the original application, the recruiter receives the hiring fee rather than TokyoDev. That’s why, Algoet said, you should make sure your recruiter is a good fit and can represent you properly. “If you feel they can’t,” he suggested, “walk away.” And if you have less than three years of experience, he suggests skipping a recruiter entirely. “Many companies don’t want to pay recruitment fees for junior candidates,” he added, “but that doesn’t mean they won’t hire you. Reach out to hiring managers directly.” From the internal recruiter’s perspective, Sonam Choden is in favor of candidates who come through job boards. “I think we definitely have more success with job boards where people are actively directly applying, rather than candidates from agents. In terms of the requirements, the candidates introduced by agents have the experience and what we’re looking for, but those candidates introduced by agents might not necessarily be looking for work, or even if they are . . . [HENNGE] might not be their first choice.” Laine Takahashi agreed and cited TokyoDev as one of HENNGE’s best sources for candidates. We’ve been using TokyoDev for the longest time . . . before the [other] job boards that we’re using now. I think TokyoDev was the one that gave us a good head start for hiring inside Japan. “And now we’re expanding to other job boards as well,” she said, “but still, TokyoDev is [at] the top, definitely.” Follow up Ho casually nailed the dilemma around sending a message or email to follow up on your application. “It’s always best to follow up if you don’t hear back,” he said, “but if you follow up too much, it’s irritating.” The question is, how much is too much? When is it too soon to message a recruiter or hiring manager? Ho gave a concrete suggestion: “Send a message after three days to one week.” For Chidiac, following up is a strategy he’s used himself to great effect. “Something that I’ve always done when I look for a job is ping people on LinkedIn, trying to anticipate who is the hiring manager for that role, or who’s the recruiter for that role, and say ‘Hey, I want to apply,’ or ‘I’ve applied.’” [I’ve said] ‘I know I might not be able to do this and this and that, but I’ve done this and this and this. Can we have a quick chat? Do you need me to tailor my CV differently? Do you have any other roles that you think would be a good fit?’ And then, follow up frequently. “This is something that’s important,” he added, “showing that you’ve researched about the company, showing that you’ve attended meetups from time to time, checking the [company] blogs as well. I’ve had people that just said, ‘Hey, I’ve seen on the blogs that you’re working on this. This is what I’ve done in my company. If you’re hiring [for] this team, let me know, right?’ So that could be a good tip to stand out from other applicants. [But] I think there’s no rule. It’s just going to be down to individuals.” “You might,” he continued, “end up talking to someone who’s like, ‘Hey, don’t ever contact me again.’ As an agency recruiter that happened to me, someone said, ‘How did you get my phone [number]? Don’t ever call me again.’ . . . [But] then a lot of the time it’s like, ‘Oh, we’re both French, let’s help each other out,’ or, ‘Oh, yeah, we were at the same university,’ or ‘Hey, I know you know that person.’” Chidiac gave a recent example of a highly-effective follow-up message. “He used to work in top US tech companies for the past 25 years. [After he applied to Mercari], the person messaged me out of the blue: ‘I’m in Japan, I’m semi-retired, I don’t care about money. I really like what Mercari is doing. I’ve done X and Y at these companies.’ . . . So yeah, I was like, I don’t have a role, but this is an exceptional CV. I’ll show it to the hiring team.” There are a few caveats to this advice, however. First, a well-researched, well-crafted follow-up message is necessary to stand out from the crowd—and these days, there is quite a crowd. “Oh my goodness,” Choden exclaimed when I brought up the subject. “I actually wanted to write a post on LinkedIn, apologizing to people for not being able to get back to them, because of the amount of requests to connect and all related to the positions that we have at HENNGE.” Takahashi and Choden explained that many of these messages are attempts to get around the actual hiring process. “Sometimes,” Choden said, “when I do have the time, I try to redirect them. ‘Oh, please, apply here, or go directly to the site,’ because we can’t really do anything, they have to start with the coding test itself. . . . I do look at them,” Choden went on, “and if they’re actually asking a question that I can help with, then I’m more than happy to reply.” Nonetheless, a few candidates have attempted to go over their heads. Sometimes we have some candidates who are asking for updates on their application directly from our CEO. It’s quite shocking, because they send it to his work email as well. “And then he’s like, ‘Is anybody handling this? Why am I getting this email?’,” Choden related. Other applicants have emailed random HENNGE employees, or even members of the overseas branch in Taiwan. Needless to say, such candidates don’t endear themselves to anyone on the hiring team. Be persistent “I know a bunch of people,” Chidiac told me, “that managed to land a job because they’ve tried harder going to meetups, reaching out to people, networking, that kind of thing.” One of those people was Chidiac himself, who in 2021 was searching for an in-house recruiter position in Japan, while not speaking Japanese. In his job hunt, Chidiac was well aware that he faced some major disadvantages. “So I went the extra mile by contacting the company directly and being like, ‘This is what I’ve done, I’ve solved these problems, I’ve done this, I’ve done that, I know the Japanese market . . . [but] I don’t speak Japanese.’” There’s a bit of a reality check that everyone has to have on what they can bring to the table and how much effort they need to [put forth]. You’re going to have to sell yourself and reach out and find your people. “Does it always work? No. Does it often work? No. But it works, right?” said Chidiac with a laugh. “Like five percent of the time it works every time. But you need to understand that there are some markets that are tougher than others.” Ho agreed that job-hunters, particularly candidates who are overseas hoping to work in Japan for the first time, face a tough road. He recommended applying to as many jobs as possible, but in a strictly organized way. “Make an Excel sheet for your applications,” he urged. Such a spreadsheet should track your applications, when you followed up on those applications, and the probation period for reapplying to that company when you receive a rejection. Most importantly, Ho believes candidates should maintain a realistic, but optimistic, view of the process. “Keep a longer mindset,” he suggested. “Maybe you don’t get an offer the first year, but you do the second year.” Conclusion Given the staggering number of applications recruiters must process, and the increasing competition for good roles—especially those open to candidates overseas—it’s easy to become discouraged. Nonetheless, Japan needs international developers. Given Japan’s demographics, as well as the government’s interest in implementing AI and digital transformation (DX) solutions for social problems, that fact won’t change anytime soon. We at TokyoDev suggest that candidates interested in working in Japan adopt two basic approaches. First, follow the advice in this article and also in our resume-writing guide to prevent your resume from being rejected for common flaws. You can highlight niche skills, write an original cover letter, and send appropriate follow-up messages to the recruiters and hiring managers you hope to impress. Second, persistence is key. The work culture in Japan is evolving and there are more openings for new candidates. Japan’s startup scene is also burgeoning, and modern tech companies—such as Mercari—continue to grow and hire. If your long-term goal is to work in Japan, then it’s worth investing the time to keep applying. That said, hopefully the suggestions offered above will help turn what might have been a lengthy job-hunt into a quicker and more successful search. To apply to open positions right now, see our job board. If you want to hear more tips from other international developers in Japan, check out the TokyoDev Discord. We also have articles with more advice on job hunting, relocating to Japan, and life in Japan.