Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
15
I'm writing this from the Starbucks in the Sapporo Grand Hotel. When I arrived, there was already [another Rubyist here](https://twitter.com/igaiga555/status/247511186394972160). Five minutes before, I met [another one](https://twitter.com/yotii23) walking down the street. Last night, I went into a Sapporo bar, and [a speaker was there](http://dane.heroku.com/), and soon after, I saw [yet another Rubyist](https://path.com/p/ToIzU) passing by, and called him to join us. This kind of serendipity is what made [Sapporo Ruby Kaigi 2012](http://sapporo.rubykaigi.org/2012/) such a wonderful conference. My first Sapporo Ruby Kaigi was in [2010](http://regional.rubykaigi.org/sapporo03/). With only about 120 participants, it was much smaller in size than Ruby Kaigi's 700+ people. Despite its relatively small size, the organizers put together a top-notch programme. I felt very fortunate that I could participate in it, as I thoroughly enjoyed being part of a community of such passionate...
over a year ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from TokyoDev

TokyoDev’s 2024 Recap: Challenges, Milestones, and the Road Ahead

In 2023, I scaled TokyoDev from a one-man operation to a team. The idea was to get some tasks off my plate, but while I’ve succeeded at passing off responsibilities to others, I somehow didn’t gain any more free time. This is because working with new people also created new ideas and opportunities, which I haven’t been able to pass up. In 2024, we saw the first fruits of this collaboration, achieving things I never would have been able to pull off by myself. For instance, we started producing Japanese-language content teaching employers how to build international teams, had a sponsor booth at Japan’s largest Ruby conference, brought the developer community and our clients together through events, and built an editing process that increased the overall quality of our content, and found new contributors who have written some extremely popular articles. As the year winds to a close, I’ve been reflecting on both these accomplishments and the challenges we’ve faced, and how they’re paving the way for what is to come. 65 developers got a job via TokyoDev In 2024, we tracked 65 developers who were successfully hired after applying for a position on TokyoDev. This number was down from last year’s 71 developers. Interestingly, while the total number of hires decreased, the number of companies that hired successfully went up, from 29 to 31. One reason for fewer hires was that several of our most successful clients shifted their focus away from non-Japanese speaking engineers in favour of fluent Japanese speakers. TokyoDev has always been most successful at helping companies with hiring talented engineers with little-to-no Japanese skills, and so with their change in focus, we haven’t been able to help them to the same degree as last year. However, another factor was simply timing. We count successful hires based upon when we receive a fee for them. The time between when a company posts a job to when we receive the fee is typically 3–6 months, as it takes a while for a company to interview candidates, make offers, secure visas, and so on. This means that, even though we currently have a lot of successful hires in the pipeline, they won’t be reflected in this year’s stats. For instance, while we had six successful hires per month in January and February 2024, we have nine projected successful hires in both January and February 2025 I’m optimistic about how we’re going to do next year. 60 articles written by 19 authors One of our greatest accomplishments in 2024 was establishing a repeatable editing process that has allowed us to create extremely high-quality articles. The top five articles by number of visitors were: How I Got a Digital Nomad Visa for Japan by Christian Mack The English Paradox: Four Decades of Life and Language in Japan by Tom Gally The rise and fall of D&D in Japan by Masaki Yanagida How I obtained a J-FIND visa in Japan by Oguzhan Karagözoglu Japan Needs International Developers by Rebecca Callahan The cool thing is that four of the five articles draw upon external contributors’ unique personal experiences, which allowed them to share information with our community that no one else could. Besides our English articles, we also launched a new sub-site that’s helping Japanese companies build global engineering teams. It’s still in its infancy, but we already have 18 articles for it, and we have some other great ideas for the coming year. 18 developer stories published We write developer stories to highlight the experiences of employees at our client companies to give candidates a better understanding of what it’s actually like to work there. This year, we released 18 developer stories. The top five stories by number of visitors were: Realising Dreams of AI and Japan at Recursive “We’re the first global team in Fukuoka”: English Evolution at Money Forward Bringing AI to the Construction Industry with EARTHBRAIN Becoming a Tech Lead at KOMOJU Succeeding as a Senior Engineer at Kraken 814 developers answered our survey Since 2019, TokyoDev has conducted an annual survey of international software developers living in Japan. The 2024 edition was the biggest yet, with 814 developers sharing details on their salaries, working conditions, and the technologies they use. I’ve had people tell me how useful our survey is—some even used it to negotiate better salaries when applying for jobs—so I’m glad it has continued to grow and add to the community. 2,800+ people joined our Discord server In 2024, over 2,800 people joined TokyoDev’s Discord server. This community has proved incredibly valuable. Not only has it helped people get their lives up and running in Japan, but it also has been a great source of inspiration for article topics, and a way to find potential contributors to the site. 7 events hosted In 2024, we continued to expand the in-person meetups we held. Highlights included a pair of events in Okinawa during Ruby Kaigi, an excellent beer garden in collaboration with WAY equity partners (at least I hear it was good, I got COVID the day before), and the launch of our TokyoDev Talks. 5 organizations sponsored TokyoDev owes its origins to the developer community in Japan, so it’s important to me that we use our success to give back to it. We have continued to do this through supporting the following organizations: RubyKaigi: The main Ruby conference in Japan. Attending the 2010 edition was what inspired me to start blogging on this site. Rails Girls Japan: Holds free workshops to help women pick up Ruby on Rails. Tokyo Test Fest: The first edition of an international software quality conference in Japan. Women In Technology Japan: A community that bridges the gender gap in tech and promotes diversity and inclusion in Japan. Women Who Code: This one was heartbreaking for me, as they went bankrupt almost immediately after our sponsorship. That community was succeeded by Women in Software Engineering Japan, where I’m serving as an advisor. 9 people contributed to our team Besides the contributors who wrote articles for the site or made illustrations for them, we have a number of people doing work for us on an ongoing basis. Daniel López Prat improved the infrastructure that runs the TokyoDev site, including adding additional monitoring and keeping our libraries up to date. Keiko Kimoto helped with translation and other administrative tasks, including helping us obtain a trademark for TokyoDev. Mathieu Mayer helped with product design, UX, and frontend development, such as refreshing our articles index. Michelle Tan conducted developer story interviews and helped with full stack development by building things like an admin interface for our clients to use. Rebecca Callahan interviewed contributors, wrote articles for us, and led our editorial team and process. Sacha Greif continued to refine the software that runs our survey, and helped with creating this year’s survey. Sayana Takagi acted as our client representative and led the creation of our sub-site aimed at Japanese companies. Scott Rothrock moderated our Discord community, wrote articles for us, and contributed to our editorial team and process. Looking ahead to 2025 We have a number of exciting things in the works. We’re in the planning phase of several in-person events and have plans to sponsor more communities, and are also working on a new way of connecting international developers with Japanese companies that I hope to be able to talk about soon. Thanks to everyone who has supported us this year. I’m looking forward to continuing to grow together through the next!

a month ago 18 votes
So You Want to Be a Game Dev in Japan

Given how many of us grew up playing classic Japanese games, it’s no surprise that people are keen to work on games in Japan. But what’s the reality on the ground? What skills do you need to succeed in the Japanese game industry, and what challenges can you expect to encounter? To find out, I interviewed a number of current and former members of the game industry here in Japan, for their thoughts on: What makes Japanese game development different? The upside of working on games in Japan The downside How they got started in the Japanese game industry Their top tips for success Conclusion Our interviewees hail from five different countries, work in roles ranging from studio head to entry-level programmer, and have developed everything from Nintendo games to 18+ eroge. Marc Trudel is the Studio Head at Wizcorp, a studio specializing in visual effects, porting games, and developing custom tools and engines for the Japanese game industry. He’s Canadian and has lived in Japan since 2009. Mathieu Siboulotte is a French developer hired as a game designer at the international creative company Studio No Border in 2020. Tristan Metz came from the Netherlands in 2024 to work as a game programmer at AVR Japan, an XR solution company. Minh Nguyen is a Vietnamese developer who worked for DMM Games, which specializes in erotic games (eroge), from 2017–2020 before switching industries. Jared Hays is an American who was employed by the Nintendo-affiliated company Good Feel between 2011 and 2017. He now works for a gaming company in the US. What makes Japanese game development different? Tristan Metz pointed out the most obvious difference: “In the Netherlands we tend to stick to only English, whereas in Japan both Japanese and English often tend to be required.” I also feel that in Japan there is much more of a hierarchy and emphasis on your position within the company. . . . I do think that even the Japanese language itself inherently makes interacting with people feel more formal. “This might also have to do with the fact that the Netherlands tends to have a very flat hierarchy for many organizations,” Metz concluded. Minh Nguyen doesn’t see as much difference in company structure. “I think in terms of team composition it won’t differ that much compared to other countries. We would have a product owner (the term is “director”) who makes initiatives for overall game direction; planners, who design game specs, feature, and gameplay-balancing; designers, who create art and models for the characters and environments and UIs in-game; and developers, who implement back-end or front-end. Testers are usually from outsourcing companies who do testing and verify behavior, and planners would also have to test new releases themselves.” According to Nguyen, the development process is about the same as well. “We would also use Scrum/Kanban to iterate development cycles, as seen in other software companies,” he said. What he has noticed is that Japan focuses on a different category of games. Games in Japan are mostly RPG-flavored [social] games that have characters and a lottery system called gacha to make money. If you are more into MOBA/FPS/strategy then unfortunately those are nonexistent here. For Marc Trudel, the real appeal of developing in Japan is the intense passion of those in the industry. “I had one client that’s head of R&D,” Trudel told me, “and we’re having dinner together, and we’re talking about what games do we play when we have time off, and he says, ‘I don’t play games.’ I was like, ‘Okay, so what do you do?’ ‘Well, I read about neurology in infants and children, to try to understand how the brain works in the context of gaming.’” “He’s not leading game projects,” Trudel added, “and yet he clearly has an interest in how those game projects can have a beneficial impact on younger people. . . . It impresses me.” That dedication, and that passion [Japanese developers] have . . . I don’t want to say it’s not found overseas, but here it seems to me like it’s so consistent. Everyone I talk to is going to have a story of their own, a focus of their own. But Jared Hays thinks that trademark passion can easily manifest as stubborn single-mindedness. “There’s a certain amount of, ‘If it’s not working, just try harder.’ “One of the biggest differences, and certainly the biggest in work culture, was that in Japan there was very little interest in improving processes. So it was, ‘Well, we made the last game this way and it’s shipped, and it sold, and we made money, so we’re just gonna make the next game the same way.’” And there was really no room to say, ‘Have you tried doing X?’ Or, ‘Yes, we made the game, but we did this thing that drove people crazy, people quit, everyone got burned out.’ It was not a good way to make the game. [They] just said, ‘Yes, but it worked.’ Hays offered an example: “Yoshi’s Woolly World was really built on top of Kirby’s Epic Yarn, on top of the tech stack that they had built, which was in turn built on top of a WarioWare game they had made. So those first two were for the Wii, and then Woolly World was for the Wii U. And as they moved onto the next game, the scale of the game and the things they tried to do got bigger, because players expected more. . . . So, Woolly World had no project management, no production, zero. There was no issue tracking. There was no schedule planning, nothing.” I went to school for both computer science and game development, and one of the courses that I found incredibly valuable was a course, not in the nuts and bolts of programming, but in software engineering that really taught working in a team and coordinating with people, communication, delivering milestones, all of the things that most modern devs consider the other 80 percent of the job. And they just had none of that. “If we could align our schedules so that people aren’t sitting around waiting for other people half the week, maybe people wouldn’t need to work overtime,” Hays said. “Maybe the director wouldn’t need to keep a sleeping bag under his desk.” A lack of project management was the biggest difference Hays found, but he also pointed out that Japanese developers are working with more limited resources than their counterparts overseas. “Japanese devs are really harmed by the lack of a Japanese Stack Overflow.” Because in English, if you Google a programming problem, there is an answer. And in Japanese, you Google and it’s just some random guy’s blog where he’s like, ‘Hey, I tried using Unity for the weekend and here’s what I found out.’ So there’s much less centralized information and information exchange between developers. The upside of working on games in Japan For some developers, the best part about working in the Japanese game industry is the games themselves. “My favourite thing about game development in Japan,” said Metz, “are the cool projects and opportunities it brings, which was also the biggest reason for me to move here. Japan is well-known for its famous games from Nintendo, Sega, Square Enix, and countless more. There are a plethora of opportunities in this country and that really excites me.” Hays concurred. I literally made a game with Yoshi. I would never, ever get that here [in the US]. Hays and Metz agreed that collaborating with their coworkers proved an incredible perk. “Like I said,” Hays told me, “some of the people who started the company were super veterans and were incredibly knowledgeable. And it was really awesome to get to work with most of the people, [though] not all of the people. So experientially, it was great.” Trudel has also enjoyed working with Japanese game developers. They know all the games through and through. They have really pointed opinions on what they think is good. . . . It’s a craft for them. It’s something they dedicate their life to. Sometimes, that dedication proves almost uncomfortable, at least for Nguyen. “The games I had worked on were all 18+. . . . During the title alpha/beta release they made the whole company playtest it. I had to play through explicit content along with the surrounding people.” Still, he was impressed by how wholeheartedly they tackled each project. “For an adult game title, the people around me were extremely serious and very committed to making the game become a hit. That was a unique feeling and experience for me.” It’s a bit different for Mathieu Siboulotte, working at a small studio with an international team. “So far, the working environment in my studio is kind of unique. We almost never do any overtime, we have some flexible hours and some remote work days. We are a very small structure of only four people, so we kind of come when we want and leave when we want, as long as we do our hours! For my project, my team is split between France and Japan, so my hours are mostly in the late morning until the evening, so we can share many hours together!” But like the other designers, he draws inspiration from his colleagues. “I regularly join a meetup of French game devs in Japan or the Tokyo Indie Game Show in Akihabara. It is great to test new prototypes and connect with people there!” The downside As for the downside of game development in Japan, Trudel mourns the vanishing culture of mentorship. “I feel like this is kind of getting lost in Japan,” he explained, “that senpai [older mentor] that’s going to take you under their wing.” Maybe it’s specific to the game industry, but I’m starting to see there are not a lot of people that even want to take those responsibilities—or for those who do, sometimes it’s going to be a bit more of a power trip. In general, he explained, game projects in Japan tend to lack both money and expertise. “They’re all built on custom engines, yet every R&D project is underfunded. And not only is it underfunded, but they just can’t find the resources to really do the job to bring their technological assets to the next level, right? “It’s true in R&D, and to some degree it’s true for game-making proper as well, where they don’t quite have the technical abilities to really fully [realize] their creative vision. So that’s really where we [Wizcorp] come in . . . to try to fill in the gaps.” Hays also noted the lack of good leadership. The mentorship wasn’t great, and people I was supposed to be seeking advice from as more senior engineers often were senior because they’d been there a while, and not necessarily because they were incredibly knowledgeable or good at passing that knowledge along. He explained that “Right after I worked on Woolly World, I worked on a game that was on mobile and on Facebook, and that was part of our foray into self-published first party IP. “It was me and one senior dev. He was doing all of the client stuff, I was doing all of the server stuff. It was in Unity, which I had used in school, and the backend was on Google Cloud Platform, which I had also used in school. And we approached a milestone that was like, it should be playable by now. We got to a point where the server was stood up enough and the client was stood up enough that they should connect and you could actually experience gameplay. “And we turned it on and the client performance was so bad, it was unplayable. And I turned to the senior dev, who, again, has been in the industry probably since I was born. . . . I was like, ‘Did you run the profiler? I looked at the profiler. . . . ‘It’s going crazy doing all of this sprite rendering, but it looks like you wrote this rendering code.’ “He’s like, ‘Well, yeah, that’s how you have to render the sprites.’ No, it’s a game engine. It does that. You didn’t need to do this. And so again, as the only server engineer, I had to take several weeks to rewrite the client because he had no idea what he was doing and didn’t attempt to find any of this information.” In general, Hays found, the emphasis on appearances undermined genuine efficacy. “Shortly after I started, the head of the programming department took me aside. [He said], “People don’t like that you’re leaving on time.’ ‘Am I behind on any of my work?’ ‘No, it just looks bad.’” This was the same head of the programming department who told me that the company wouldn’t hire female programmers because they would distract the male programmers. “Never mind the fact that probably 30 to 40 percent of the company is women,” Hays said. “The art department had no problem hiring them. The design department was fine. It was just the programming department.” Nguyen confirmed that most game developers in Japan fit a narrow profile: young, unmarried, and willing to put in any amount of overtime. “Once you have a family, things change,” he said. “Most game companies’ demographics are single and young people, often with little to no responsibilities outside work, and once your priorities shift from games to family you start feeling like you are an outsider.” The long hours in particular are difficult to manage, according to Nguyen. “There are crunch times before each release, and this can be stressful depending on how the project is managed.” My friend and I used to have to work long hours of overtime during the pre-release period. It can suck when people around you have no responsibilities outside work and happily put in the hours while you can’t. It was these pressures that led Nguyen to switch to a different industry. He also agreed with Trudel that the tech at many game companies is outdated. “There is an inertia to upgrade,” he said. “The most important thing in a game is not tech, it is gameplay and art.” As for Metz, he mostly would like to be paid more. My least favourite thing about game development in Japan is probably the low wages. Wages in Japan are a lot lower than in the Netherlands, but I have the impression that programmers have better financial opportunities in other programming industries outside of gaming. “I have the impression this difference is also not as big in the Netherlands,” he added, “but I could be completely wrong on that.” Hays encountered the same pay problem. “The pay is terrible. Just for reference, I looked up one of my old withholding slips from 2013, and I was taking home less than 4 million. That is bad. And I tripled my salary by moving back to the US.” How they got started in the Japanese game industry Some of our interviewees landed in the Japanese game industry mostly by accident. Trudel, for instance, originally came to Japan for the martial arts. Between 2007 and 2009 he visited Japan whenever he could, for anywhere between a weekend to several months at a time. “During [one of my longer trips], I would basically be training three times a day, five or six days a week,” he said. “I would just be dojo-hopping basically, to kind of get my bearings in terms of figuring out if I saw something in it. But what I ended up sticking to was basically the classical martial arts of Japan.” He finally moved officially to Japan on October 20th in 2009, on a working holiday visa. “I didn’t actually have the job lined up when I came in,” he explained. I had a little bit of money set aside. I figured okay, well, let’s just try to find a job. Now that I’m here [in Japan], I can visit people in person. You didn’t really have Zoom at the time. . . . When I started at Wizcorp, I was hired at the same time as one other person, but I was essentially employee number three.” “At the time I was hired as an engineer,” Trudel said, “and ended up doing all the IT stuff.” His official title was System Architect and Network Administrator. “And from there, once we started to move gradually into games from 2010, I started to take [on a] technical leadership position.” Having also served as CTO and COO of Wizcorp, Trudel became the Studio Head in February of 2023. As of now, he’s worked at Wizcorp for over 15 years. Nguyen also was always interested in coming to Japan. Our uni curriculum offered Japanese lessons and allowed the credits to accumulate as well, so I had studied Japanese and gotten my N1 even before my very first trip to Japan. A big thanks to my uni, which made this miracle happen! By contrast, Siboulotte chose the game industry, but not Japan specifically. He started learning game development with RPG Maker when he was 16, but didn’t study game development in university. “I thought joining this industry was impossible,” he said. Instead, he majored in international trade, while continuing game projects on the side. He then received a bachelor’s degree in cultural product marketing, which gave him the opportunity to join an animation studio as a producer’s assistant. “However, game development was still on my mind,” Siboulotte said, “so I decided to leave and start university again from scratch, to study game design, in 2017.” He got his lucky break with Studio No Border, an international creative studio that’s affiliated with the French entertainment group Ankama. “I joined this project in 2020,” Siboulotte said, “right before Japan closed in a lockdown. It was also my first ‘real’ position in the game industry.” I honestly absolutely arrived here by chance. I was just looking for my first gig after my graduation and internship, and this job happened to be the first one to give me a reply! “I applied on the website AFJV, a French website listing games positions in France or with French language involved, and after around four months of tests and interviews, I finally got a green light to come to Japan!” Hays and Metz were both specifically interested in working on Japanese games. While still a college student, Hays spent time in Osaka and loved it. After graduating in 2011, he returned to Japan and was quickly introduced to Good Feel, where he got to pursue his dream of working on Nintendo games. Metz told me there was a specific moment that clinched his desire to come to Japan. The final push to want to commit to the game dev industry in Japan was a 2017 GDC talk by Nintendo about The Legend of Zelda: Breath of the Wild. However, Metz was also well-prepared for the move. “I have always been fascinated with Japanese culture from a young age,” he said, “and have been self-studying the language off and on for around 9 years.” Their top tips for success The most consistent advice interviewees offered: learn Japanese. “It might be obvious for those wanting to work in Japan,” said Metz, “but I think that game development in Japan is much more reclusive than IT companies in Japan and will require a very high level of language fluency.” Nguyen agreed. You need to speak a very high level of Japanese. N3 or even N2 might not be enough to collaborate effectively. From there, however, the advice began to differ, depending on whether interviewees thought that game development in Japan was a good long-term career goal, or whether it should be for the short-term only. “Be sure to get out of the industry while you are still young and you want to advance your career more in tech,” Nguyen stressed. “The game industry doesn’t usually put as much emphasis on tech as others and it is certainly not a tech-driven industry, so staying there a long time can be detrimental to your career.” If you value experience working with tech, plan your departure even before entering the game industry. My friend and I struggled a lot when we were trying to jump ship. Hays concurred with Nguyen’s advice. “Treat it like a gap year,” he said. “Spend three, four, five years doing this because it’s what you really want to do. And then probably take that and go home.” “I’m not going to say, ‘No, never do it, stay away, don’t touch it with a 30 foot pole’,” Hays also said. “But make sure you understand what you’re getting yourself into and that in all likelihood, you’re probably sacrificing some career progression and certainly income for the sake of this opportunity.” Above all, Hays believes it’s important to do your research before going in. I will say as someone who loves video games, loves Japan, loves Japanese video games, if you’re doing it because you have an idealized vision of what working on your favorite games must be like, then you should spend time looking at real-world information. “Look for testimonials of people who worked at [those] companies, look for information about the pay and the working conditions. Just because it’s your dream doesn’t mean you shouldn’t research the company like you would a company in your home country.” Trudel also emphasized the need for research beforehand. “Do visit first. Being a tourist is not the same thing as living here, but it’ll give you some idea.” From there, Trudel’s advice differs, because he’s more optimistic about game development prospects in Japan. It does require, he believes, a great deal of commitment and the willingness to adapt. Japanese society works very differently. The game industry works very differently. Every client is going to work very differently, culturally speaking, and you need to find a way to acclimate to that, and blend in to some degree. “You need to be able to find a way to communicate,” Trudel went on, “where it’s going to make things move in the right direction.” It can be tricky, he told me, but “it’s not impossible.” Japanese ability of course helps immensely: “I mean, the one big mistake that I made was not learning the language enough before coming here,” Trudel admitted. “This set me back some years.” But language skills aren’t the whole picture. “I know that for me, as much as I struggled with language early on because I couldn’t understand what people were saying. . . . Because I couldn’t understand, I had to pick up on nonverbal cues more and on all etiquette stuff a lot more quickly, just to not get into trouble. Basically, it made me pay more attention to things.” Language is not enough. Language is just the gateway to the culture, really. From there, you have to walk through it. “I think a good way to do that,” Trudel advised, “is to engage in cultural activities. It could be sports, I mean, you can go and join a volleyball team for all I care, but having these kinds of activities where you need some form of interaction, some form of communication, verbal and nonverbal, to be able to engage in the activity, will make a big difference. “Plus it’s going to give you a bit of a network, beyond just having colleagues at work.” Trudel is strongly in favor of all forms of networking, even before you come to Japan. “Reach out,” he suggested. “I mean, you have LinkedIn, you have Facebook, you have all these social networks where there are some groups for the Japanese gaming industry. Talk to people, ask questions, see what they’re about.” He clarified, though, that you’ll get better results if you focus on gathering information over clinching a job. “[Those who message me about jobs], every time I’m going to tell them there’s an application process and a candidate selection process, and I’m out of the loop there. “But if you were just asking about whether you might find something to your liking [in Japan], I’m happy to jump on a 30 minute call with you and try to figure it out and just have that discussion. I’m assuming that not everyone is going to be necessarily as willing to engage like that, but it’s just a numbers game, right? “You know, the more people you reach out to, the more people are going to answer back, and then you’re going to be better informed.” Conclusion While the experiences shared here have been subjective, two of the major points—the low wages and the need for Japanese ability—were confirmed in TokyoDev’s 2024 survey. The gaming and education industries had the lowest compensation, with respondents making a median of ¥8 million a year. 37% of respondents in the gaming industry always used Japanese with their colleagues, the highest percentage of any industry. So if you’re interested in being a game developer in Japan, it’s best to start studying Japanese as soon as possible (and perhaps to be independently wealthy). But if a conventional Japanese game company isn’t for you, there are also a number of international game studios that can offer a more flexible and English-speaking environment. If you’re both ambitious and determined, like Siboulotte, you might aim to strike out on your own. The [Japanese] indie scene looks huge, and it gave me the will to make my own indie game besides work. I am really looking forward to BitSummit in Kyoto to try and show my prototype at a big fair again! Wizcorp and other game companies in Japan are hiring now, so check out our game development jobs page. Want to hear more experiences? Continue the discussion in our Discord community.

a month ago 23 votes
Remote Worker Rights In Japan

Are you working remotely for a Japanese company? What happens if your company suddenly issues a return-to-office mandate? Will you have to move back to Tokyo? What if remote work is in your contract—do you have the right to refuse to return to the office? What standing do you have to negotiate with your company? What are your chances of persuading management to change their minds? These are the questions TokyoDev set out to answer, because return-to-office mandates are on the rise in Japan. Return-to-office mandates: the numbers The TokyoDev 2024 survey showed that, among respondents, only 9% worked five days a week in the office. However, companies are increasingly switching from fully-remote to a hybrid working pattern. In 2023, 43% of survey-takers could work fully remotely if they wished, but in 2024, only 38% could say the same. This trend is not confined to Japan. According to Morgan McKinley, in Hong Kong 91% of companies insisted their employees return to the office, while only around 40% of companies in the UK and Canada are asking the same. Japanese companies come in around the middle, with 62% requesting that their employees come back into the office at least some days. Return-to-office orders are having a direct impact on employee attrition. While only 10% of TokyoDev survey respondents who could choose whether or not to work remotely were interested in changing jobs, 18% of those in a hybrid environment were job-hunting, and 39% of those required to work full-time at the office were actively searching for new roles. 49% of survey-takers valued the ability to work remotely over everything else. Meanwhile, those who had to attend an office were more negative about their workplace in every aspect except job security than those who could work fully remotely. In short, tech companies in Japan should be advised that insisting on in-office work, or even hybrid work, could strongly affect their recruitment and retention of employees. By contrast, those who allow fully-remote work can expect to see a rise in applications. What the Tokyo Labor Bureau has to say For those developers whose companies issue a return-to-office mandate, what are their rights under Japanese law? And what is the experience like for those developers who try to enlist the help of Japan’s foreign worker resource centers? To find out, I called them myself. Each time I represented myself as a developer dealing with an increasingly typical situation: though I had always worked remotely, and had moved to a distant prefecture while doing so, I was suddenly being ordered to report to the Tokyo office once a week. What could I do about it? Labor Standards Advice Hotline I started by calling this hotline, which quickly set me up with an interpreter. With each question she listened to me in English, then spoke to her superiors in Japanese and translated their reply back to me. The upshot of their advice was that I should contact either my local prefectural labor bureau, or (since my hypothetical employer was based in Tokyo) the Tokyo Labor Bureau. There wasn’t much legal advice or support that they could offer other than directing me to the appropriate resources. However, they did tell me that most of my case depended on the exact wording of my contract. If it was specified in my contract that I could work fully remotely, regardless of changes in the company’s work plans, then I had a good chance of insisting on continuing to do so—or, at the very least, negotiating from a position of strength. If, however, my contract said that I could work remotely with the company’s permission, or contingent upon company circumstances, then my only hope was to ask for some kind of compromise. If remote work was specified in my contract, I was told, and the company continued to insist that I come to the Tokyo office when I already lived in another prefecture, then I could be eligible for leave allowance, or a payment of 60% of my salary. They weren’t able to give further details on the subject, however, and again directed me to one of the labor bureaus for more details. The Tokyo Labor Bureau When I rang the Tokyo Labor Bureau and presented them with the same dilemma, it was easy to locate someone who spoke English, but their answers were less optimistic. Essentially, the woman on the phone said, there was no provision for return-to-office mandates—-or indeed, anything about remote work—in Japanese law. This left me with limited options. The Tokyo Labor Bureau has a “resolution system” designed to help employees and companies mediate conflicts. This is available only in Japanese, so I would need to bring a friend or translator; however, it is free of charge. In general, while certainly willing to help, she didn’t seem too optimistic about my chances of pushing back against the company’s order. It would be “kind of hard,” she admitted. Since this was more about my particular contract than general labor law, she also suggested that this was really a matter for the lawyers, and gave me the number for the Foreign Residents Support Center. The Foreign Residents Support Center When I first called the center, their lawyer was busy, but the woman on the phone apologized profusely and asked if they could call me back, which they did within a few hours. The lawyer I spoke to had yet another set of suggestions for my return-to-office scenario, but of the people I’d asked so far, he seemed the most optimistic regarding my chances. Unfortunately, it seems that hard-and-fast answers are difficult to come by. In principle, he said, I should not have to obey the mandate if my contract states that fully-remote work is allowed. In fact, I might be able to do so even if fully-remote work wasn’t specified in my contract. If there was correspondence exchanged when I signed the contract that promised fully-remote work, or possibly even verbal statements (though this would naturally be harder to prove), I could argue that fully-remote work was a “specific condition” of my employment. If there is some rational reason for the once-a-week visit to the office, he went on, then the company would be obligated to pay my commuting costs from the distant prefecture to the Tokyo office. However, like me, he considered it unlikely that the weekly meeting was really all that necessary. Instead, he thought I could press for doing the meeting via video call, and that this would fulfill my obligation to the company without incurring additional hassle or expense for either side. The tricky part was that all of this was speculative, and a lot would depend on specific qualifications. For example, when discussing whether the return-to-office order was actually illegal, he said it depended on several different factors: Was fully-remote work promised in my contract? Could my job be done fully remotely? Does the company have an important reason for this order? Of course, the last question is the hardest to nail down. The company has to compare the necessity of the order to the disadvantage of the employee, I was told, which appears to leave a lot of legal wiggle room for a strict or unscrupulous company. And contrary to what the Labor Standards Advice Hotline had suggested, he did not think I would be eligible for leave allowance. If the company refused to budge, he said I should contact my local bar association or city hall to find representation. If I was unable to locate a lawyer on my own, I was free to call back and they would assist me again. However, like everyone else I spoke to, the lawyer strongly suggested that I attempt to negotiate with my company instead. Given my specific circumstances, he suggested that if the company covered the commuting costs, I could perhaps offer to return to the office once every two weeks. In general, he assured me that I shouldn’t be afraid to bargain in this way, particularly if I worked for a small company that might find me difficult to replace. A less positive experience Sadly, sometimes neither negotiation nor legal action are possible. Several TokyoDev members spoke anonymously on their companies’ return-to-office mandates, and one of them described his own experience in consulting a lawyer. It was a lawyer [where] you get 30 minutes of pre-consultation. I sent him my job description, my contract and stuff, and then he looked it up. He said that even though it’s written in the contract that remote work is possible, there’s no precedent in the Supreme Court. . . . He said that if you want to fight, of course I can help you fight it. But in the end, if you lose or if the company dismisses you in the middle of it, then you have bigger problems. Although the opportunity for remote work had been promised in his job description, the actual employment contracts were more vague in their terms. Technically, the company wasn’t violating the contracts. Employees suspected that the company was using this return-to-office mandate to reduce their workforce without violating Japanese employment laws, but such an assertion would be difficult to prove. In the end, the developer decided against legal action. “I did not try to lawyer my way through because I know, once I file a lawsuit or something like that, then it’s going to be big trouble for me.” He is, however, actively searching for a new role, as were other developers we spoke to who had been ordered back to the office. To be clear about the prospect of retaliation, Japan law is strict about the circumstances under which an employee can be terminated. An employee negotiating in good faith around remote work isn’t an acceptable reason, and would run afoul of Japanese law: An employer is only allowed to dismiss an employee if there are objectively reasonable grounds for dismissal, and dismissal is deemed to be appropriate in light of socially accepted ideas. Furthermore, all possible grounds for dismissal must be clearly stated in the work rules if the dismissal of an employee is to be valid. The union option To Dennis Tesolat, General Secretary at the General Union, the solution to these return-to-office mandates is obvious. He calls it “union math.” If tech workers were to get together, they could command a lot at the negotiation table. I met with both Tesolat and Sonomi Terao, the Executive Officer at the General Union. They believe most developers don’t consider unionizing because they’re office workers rather than in the trades, but they are in a great situation to do so. “There’s power in numbers,” Tesolat said, “but [also] just one person joining can be effective.” The General Union, which is headquartered in Osaka but accepts members from all over Japan, already has at least one worker dealing with an unwanted return-to-office mandate. They wouldn’t mind taking on more such cases. Companies don’t want to fight, they want to make money. But we’re a union, it’s our job, so we don’t mind. In fact, Tesolat said, sometimes zero confrontation is required. Just sending in the notification of an employee’s General Union membership often causes management to back off their demands. “At least somebody else now is watching you,” said Tesolat. “Is it a big help, is it going to change your whole situation? No, but they might leave you alone.” And if they don’t, “You just have more options [with a union]. The chance to negotiate, to be supported by colleagues, the right to dispute. The option of court is always there, but it’s not the first option. Nine out of ten times we solve things without using that court option.” What’s key, he said, is not approaching the negotiating table alone. “Dispute and negotiating—that’s our job. . . . And once you [mess] it up, we can’t help you at that point.” This is especially true if you’re an international developer working for a Japanese company, “because the whole manner of negotiation is different. . . . The chance for a lot of misunderstanding is there.” That’s the thing about negotiating on your own. It’s hard, you don’t always know what to do it . . . and if there’s retaliation from that, ‘So what?’ But if the union does it, and there’s retaliation, there’s trade union law that says you can’t do that. What about retaliation for joining a union? Tesolat laughed and said that in his thirty years of experience, he’s seen fewer than ten straightforward retaliation cases. That leaves open the possibility of indirect retaliation, but Tesolat again pointed out that the union exists to deal with precisely that sort of issue. In short, “I would worry about a lot of other things before I’d worry about joining a union.” Two years ago, the General Union didn’t even have an IT branch. During the pandemic, however, the General Union—which had initially confined its membership to the Kansai area—began accepting applications from all over Japan, and from a greater variety of professions. “People were getting fired, they weren’t getting paid, and we couldn’t say no,” Tesolat said, “so we opened the door.” As a result, membership shot up by 35%. Recently, they’ve seen another surge in tech worker applications: “A lot of people started getting scared after the layoffs in America.” With return-to-office mandates increasing, the General Union may see their numbers continue to rise—and that’s good for “union math.” How to reverse your company’s return-to-office policy One anonymous developer we spoke to successfully reversed the return-to-office policy for his entire team. During Covid, he told us, the team worked fully remotely, but after the pandemic was over the management team insisted that developers return to the office five days a week. Eventually our interviewee was able to persuade them to restore remote work, first on a hybrid basis, and later full-time. “This was at a very small company though,” he explained, “where we had more leverage than what you would normally expect in a midsize or big company. Since the push also came from me as the lead developer, management eventually accepted it.” “I was the first developer in the company,” he added, “and I was often asked about what we needed to do to get a dev department running. “One of the things I mentioned is that it is hard to keep developers for a long time, so you need incentives. You either give them a raise, or benefits, or both. Since it was a small company it had no way to compete with bigger ones when it came to benefits and salary. So, what else can you do? If you are small and agile, you can afford to give remote work benefits, as it will cost you little or nothing to do so.” From the company’s perspective, my point was more about being able to retain people and also have an easier time finding new ones. The cost of hiring and onboarding a new developer is quite high in my experience. If you have a good worker and you are not in a position to be giving raises to everyone, remote work is an easy way to keep your devs happy. The fact that other companies do not offer it also means that it is harder for them to be poached. I asked if he had any advice for other developers who wanted remote work. “If I were to give any tips to other developers that are unhappy with their situation,” he said, “it would be to let their company know about it. “For example, they probably have a one-on-one discussion with their manager every so often. This is a good opportunity to ask if the company is considering remote work, [explain] why they want it, and so on. I would not expect a change immediately, especially if they are in a more traditional Japanese company. But consistently asking about it and showing that it really matters is what made the devs here get remote work. ”[Ask the company to] try running a trial with just one developer or two, and evaluate the pros and cons. If the company outright states that it will never allow it no matter what, or it becomes clear that they will not do it, I would start looking for new opportunities that provide the benefit.” So, it is always about leverage. If the company does not think you are worth what you are asking, your only choice is to go to a place that thinks you are worth it. And, of course, keep studying and learning to improve the chances of someone thinking that you are. Conclusion In the TokyoDev 2024 survey, there’s a clear correlation between in-office work and job-hunting. Full-time office workers are looking for new opportunities at the highest rate, followed by hybrid workers, whereas only 10% of fully-remote workers are looking for new roles. As more companies become aware of how highly their employees prioritize remote work, we should expect to see a decline in return-to-office mandates. Even those who don’t wish to change jobs may be able to use this trend to negotiate with their companies. Of course, not all of those negotiations will be successful, and the advice offered by Japanese labor bureaus and legal support centers can be highly variable. However, most of the people I contacted were supportive and helpful. Perhaps, if you encounter negativity or opposition from government workers, you should avail yourself of the old immigration tactic: if you don’t like the answer you got, ask someone else. Has your company asked you to return to the office? We have a list of fully remote developer jobs for you. If you want to continue the conversation, join our Discord community.

2 months ago 19 votes
TokyoDev supports the legalization of same-sex marriage in Japan

Japan’s lack of recognition for same-sex marriage doesn’t affect me personally, but it does affect people I care about. So when I saw Business for Marriage Equality, a campaign that highlights 500+ companies and organizations that support legalization of same-sex marriage, I immediately had TokyoDev join it. TokyoDev unequivocally supports LGBT+ rights, including same-sex marriage. Japan does not currently recognize same-sex marriages or civil unions. While the same-sex partnership systems offered by municipalities and prefectures cover 85% of the population, these provide limited benefits, and are not equivalent to legal recognition. Several courts have ruled that the lack of legal recognition is unconstitutional, most recently the Tokyo High Court. But the fact remains that same-sex couples in Japan lack the fundamental rights granted to heterosexual ones. Beyond the moral issues this inequality raises, it also causes practical ones. Globally, same-sex marriage is recognized in 36 countries, whose citizens represent 20% of the world’s population. Lack of a clear legal framework for acknowledging same-sex unions makes the relocation of foreign same-sex couples to Japan fraught with difficulty. For instance, if both partners are non-Japanese and one of them relocates to Japan on a working visa, the other may be able to come on a designated activities visa, though those visas can be difficult to acquire. However, a non-Japanese person married to a Japanese citizen is ineligible for such a visa. This uncertainty about whether a couple can relocate together—not to mention the questions it raises about what discrimination they may face—means that talented individuals who would otherwise enthusiastically relocate to Japan are not doing so. I’ve observed their reluctance firsthand. As Japan needs international developers, this lack of legal equality for same-sex relationships is holding the country back by damaging its impression among skilled talent overseas. I’m optimistic about the future, though. Momentum continues to build around the legalization of same-sex marriage here, and I think it is inevitable that it will eventually be achieved. I hope that our support contributes in a small way to help those affected, until the day of legalization arrives.

2 months ago 16 votes
AI Disrupts The Traditional Hiring Process

In the face of AI, there are two strategies: embrace the algorithm, or embrace being human. As a job board operator, this has been weighing on me. I see hints that AI will disrupt the industry and transform the hiring process, perhaps not for the better. Ultimately, I’ve decided that TokyoDev will not use AI to influence the hiring process. We will instead focus on doing what AI cannot: building a supportive community, crafting high-quality resources to help individuals become stronger candidates, and leveraging our deep expertise to help companies showcase what makes them unique, attracting the talent they seek. Challenges posed by AI Since AI has entered common usage, it has created a number of issues in the hiring industry, both for candidates and for companies. AI makes it incredibly easy to send customized job applications Since online job applications became common, candidates have been able to apply for countless positions in a short period of time. Anecodetally, I’ve heard of people submitting 1,000+ job applications. The initial downside to this approach was that candidates couldn’t spend a lot of time customizing those applications, so they tended to be low quality. Now AI tools have emerged that not only allow candidates to automate their job applications, but to submit a fully-customized application based on the job description and their resume. While I believe these tools won’t yield the best results, for many, the temptation to use them is just too high, particularly for candidates who aren’t getting job offers. This means that positions are receiving an increasing number of applications that seem to be tailored to them and are superficially passable. The more people use such tools, the worse the problem becomes. This is particularly true for positions that support remote work globally. For example, by some estimates, there are 28 million software developers worldwide. If even only 0.01% of all developers were to use AI tooling, a position could potentially receive 2,800 such unthoughtful, AI-generated applications. Only after human reviewers have invested some time in reading the AI-generated application will they see the cracks in the facade. Countless human hours will therefore be spent filtering out AI-generated applications, unless recruiters take the next logical step. Large application volume makes AI screening attractive This can lead to a technological arms race: as candidates seek to lower the effort spent on job applications, those screening the applications will seek to do the same. Recruiters have always spent only a short amount of time reviewing a given application: some estimates put it at seven seconds. In our previous hypothetical of 2,800 applications, that adds up to almost 5 ½ hours of review. This is a time sink that recruiters will jump at the chance to cut down. Now, AI-based screening tools are starting to be built into Applicant Tracking Systems (ATS). These tools don’t automatically accept or reject candidates. Rather, they opt to assist in evaluation, usually by assigning scores based on how well candidates match the job description. You can see an example of how this works with Workable’s overview of their AI screening assistant. Even though these tools do not explicitly reject candidates, they implicitly do so. If a recruiter sees a candidate has a low match score, they’re likely to just click the button to reject the candidate themselves. If they weren’t going to trust the evaluation and were going to read the application themselves, why would they bother to use the AI evaluation to begin with? This will lead to qualified candidates getting rejected—but most recruiters are okay with this, as they fundamentally see the job of the initial screening as rejecting unqualified candidates, rather than identifying qualified ones. AI is an existential threat to job boards Before TokyoDev was a job board, it was my personal blog with a mailing list. Despite the mailing list only having a couple hundred subscribers on it, and me only posting opportunities every once in a blue moon, several people got jobs through it. This meant we had an incredibly high signal to noise ratio. Since turning the mailing list into Japan’s first job board for international software developers, the number of applications we process has skyrocketed. On our job board, positions that accept overseas candidates will often receive hundreds of applicants from us alone. While our clients have compared the quality of our candidates favourably to competitors, the fact remains that our signal to noise ratio has worsened. I can see that, as AI-created applications become more common, they could grow to be an existential threat to our business. Our value lies in the superior candidates that we provide to our clients; if all of the applications through us are the same as other job boards’, why would clients choose us? Furthermore, these automated tools make it likely that a company’s own ATS will be getting ever-increasing volumes of applications. If their ATS does a good enough job of highlighting qualified candidates, there will be no need to bother with third party sites such as ours. Why TokyoDev does not use AI screening In the face of these challenges, I’ve seen other job boards begin to implement their own AI-based screening, scoring candidates in a similar fashion to ATS. After all, it’s incredibly easy to use services like ChatGPT to do this: any AI will happily give you back such a score, whether or not it’s meaningful or would correlate with what a skilled human evaluator would give. TokyoDev, however, does not plan to implement any AI-based screening, both for philosophical and practical reasons. Screening candidates would be illegal In Japan, recruiting is a regulated industry. We’re regulated as a job board, not a recruitment agency. That means we’re prohibited from screening candidates or modifying the applications they make. Some job boards in Japan have registered themselves as recruitment agencies, allowing them to do such screening. While TokyoDev could do this too, I’ve intentionally not pursued this option, as getting involved with screening would take away from our primary focus: sharing opportunities in Japan with international developers. Transparency is a guiding principle Transparency is incredibly important to me. My belief in the importance of transparency is why we’ve been explicit about how we make money, and how that influences our operations. An AI-based screening system is fundamentally not transparent. Such a system would mean putting candidates at the whim of an AI, the processes of which cannot be deeply or thoroughly inspected. Those job boards utilizing AI screening could attempt to frame this as a good thing; they could say that “AI highlights your strengths,” or that it allows exceptional candidates to shine. But even so, it wouldn’t make sense to ever show how well a candidate was scored by an AI, as revealing that information to applicants would encourage and allow them to game the system, reducing the quality of applications that the client eventually receives. The process would need to be kept mysterious, which is antithetical to the way I’ve always run TokyoDev. AI tools have been shown to be biased Amazon used AI to screen candidates, but scrapped it after they uncovered that it preferred male candidates, and was doing things like penalizing resumes that included the word “women’s” in it. While AI has progressed a lot in the last 10 years, and it may theoretically be possible to create an unbiased tool, I think it is quite challenging to actually do so, let alone definitively prove that it doesn’t have unacceptable bias. Tackling the gender divide in our industry is an important issue to me. TokyoDev supports communities that empower women in technology, and provides companies with advice on how to improve gender diversity. It would be counterproductive for us to inject additional bias into the hiring process. Companies developing ATS have more resources to invest in AI TokyoDev is a niche job board with finite resources. We’re not going to be able to build a better AI-powered screening system than companies with development teams producing ATS products. I wouldn’t want to come up with some haphazard solution myself, either. Helping companies decide who is a good fit is a huge responsibility that can change lives. If I were to implement some AI screening system, and it failed to identify even a single qualified candidate who might have gotten the job otherwise, that would weigh too heavily on my shoulders. How TokyoDev works to improve the process This is not to say that we’re doing nothing in the face of AI. Rather than leveraging AI to reject poor candidates, TokyoDev is working to increase the number of high-quality candidates. For example, when candidates are troubled by an unclear job description or application process, they can reach out to me directly, and I am usually able to cut through the bureaucracy and find a truthful answer. This is something AI cannot do. On a broader scale, we invest in the hiring ecosystem by building our community, educating applicants, and helping companies highlight their best qualities to attract top talent. We also work to better the software industry as a whole in Japan; these improved work conditions will then attract more qualified candidates. Community building I’ve been volunteering at and organizing tech events since 2010. As we’ve seen success with TokyoDev, I’ve worked to reinvest back into the local developer community, through sponsorship and more. We’ve also started our own community, both online through a Discord server, and through hosting in-person events. Our primary motivation is the value these create for our members, but they also have a positive impact on our business. Because a community promotes peer-to-peer knowledge sharing, it’s helped our members become better candidates. A prime example is the “resume review” channel in our Discord, where members can post their resume and get feedback from others. I’ve seen firsthand how members have used the channel to develop an insta-reject resume into one that successfully passes screenings. We also educate our members on immigration-related processes, settling into life in Japan, and successfully dealing with diverse personal and professional stresses. Besides that, many talented developers seek out our events, to hone their skills and make connections. By having these people in our audience, it’s more likely that they’ll choose to use our job board over others, which means we have higher-caliber applicants. Informing candidates Alongside the peer-to-peer education members of our community get, we also invest in creating high-quality resources that can help people become better candidates. Some examples include our articles on writing resumes for jobs in Japan, using recruitment agencies in Japan, and a guide to software engineer visas in Japan. In addition, we write articles about Japanese work culture, the process of moving to Japan, and how to live here comfortably with pets, children, and more. This is because Japan is in need of talented and senior developers, and we want to do everything we can to attract those kinds of people to the Japanese companies we work with. Good candidates are hungry for information so that they can make informed decisions; the smoother and more appealing we can make the process of immigration, the more top-notch international candidates will be interested in applying. Besides our articles, we also conduct an annual survey of international developers living in Japan, which enables people to better understand the market. These resources help all potential candidates, regardless of where they ultimately apply. Helping companies highlight themselves In-demand candidates have the choice of where they want to work, so they won’t bother applying to a company that doesn’t look attractive to them. At TokyoDev, we manually review every job description that is posted on the platform. Using our industry experience, we work with the client to make more informative, appealing listings that are standardized and easy to understand. This is especially important in Japan, where English-speaking postings are often created by using machine translation, and can be missing crucial information for international engineers or contain confusing, ambiguous English. We also write developer stories to help companies demonstrate that their own employees have a positive view of their work environment. These stories highlight the culture of the company in a genuine way, and provide an “insider” perspective for would-be candidates. Conclusion TokyoDev has grown to be a much greater endeavour than my personal blog ever set out to be. Through it, I have been able to help hundreds of developers find their paths to Japan. We have made measurable change in our industry by educating people about their rights, supporting diversity, and assisting companies in understanding how to attract and retain international engineers. AI screening contains inscrutable processes that are biased against certain classes of applicants and prevent communication on a meaningful, human level. Building it into TokyoDev would turn it into a cold, sterile service that runs counter to the warm and welcoming place we currently are. Instead of turning to a machine to overcome modern hiring challenges, we will continue to embrace our humanity, by connecting with people on a personal level to grow and promote high-quality candidates. We will do this because we believe it’s not only the right way to hire, it’s genuinely the most effective, transparent, and helpful method.

2 months ago 16 votes

More in programming

The Exodus Curve

The concept of Product-Market Fit (PMF) collapse has gained renewed attention with the rise of large language models (LLMs), as highlighted in a recent Reforge article. The article argues we’re witnessing unprecedented market disruption, in this post, I propose we’re experiencing an acceleration of a familiar pattern rather than a fundamentally new phenomenon. Adoption Curves […] The post The Exodus Curve appeared first on Marc Astbury.

8 hours ago 2 votes
Serving the country

In 1940, President Roosevelt tapped William S. Knudsen to run the government's production of military equipment. Knudsen had spent a pivotal decade at Ford during the mass-production revolution, and was president of General Motors, when he was drafted as a civilian into service as a three-star general. Not bad for a Dane, born just ten minutes on bike from where I'm writing this in Copenhagen! Knudsen's leadership raised the productive capacity of the US war machine by a 100x in areas like plane production, where it went from producing 3,000 planes in 1939 to over 300,000 by 1945. He was quoted on his achievement: "We won because we smothered the enemy in an avalanche of production, the like of which he had never seen, nor dreamed possible". Knudsen wasn't an elected politician. He wasn't even a military man. But Roosevelt saw that this remarkable Dane had the skills needed to reform a puny war effort into one capable of winning the Second World War. Do you see where I'm going with this? Elon Musk is a modern day William S. Knudsen. Only even more accomplished in efficiency management, factory optimization, and first-order systems thinking. No, America isn't in a hot war with the Axis powers, but for the sake of the West, it damn well better be prepared for one in the future. Or better still, be so formidable that no other country or alliance would even think to start one. And this requires a strong, confident, and sound state with its affairs in order. If you look at the government budget alone, this is direly not so. The US was knocking on a two-trillion-dollar budget deficit in 2024! Adding to a towering debt that's now north of 36 trillion. A burden that's already consuming $881 billion in yearly interest payments. More than what's spent on the military or Medicare. Second to only Social Security on the list of line items. Clearly, this is not sustainable. This is the context of DOGE. The program, lead by Musk, that's been deputized by Trump to turn the ship around. History doesn't repeat, but it rhymes, and Musk is dropping beats that Knudsen would have surely been tapping his foot to. And just like Knudsen in his time, it's hard to think of any other American entrepreneur more qualified to tackle exactly this two-trillion dollar problem.  It is through The Musk Algorithm that SpaceX lowered the cost of sending a kilo of goods into lower orbit from the US by well over a magnitude. And now America's share of worldwide space transit has risen from less than 30% in 2010 to about 85%. Thanks to reusable rockets and chopstick-catching landing towers. Thanks to Musk. Or to take a more earthly example with Twitter. Before Musk took over, Twitter had revenues of $5 billion and earned $682 million. After the take over, X has managed to earn $1.25 billion on $2.7 billion in revenue. Mostly thank to the fact that Musk cut 80% of the staff out of the operation, and savaged the cloud costs of running the service. This is not what people expected at the time of the take over! Not only did many commentators believe that Twitter was going to collapse from the drastic costs in staff, they also thought that the financing for the deal would implode. Chiefly as a result of advertisers withdrawing from the platform under intense media pressure. But that just didn't happen. Today, the debt used to take over Twitter and turn it into X is trading at 97 cents on the dollar. The business is twice as profitable as it was before, and arguably as influential as ever. All with just a fifth of the staff required to run it. Whatever you think of Musk and his personal tweets, it's impossible to deny what an insane achievement of efficiency this has been! These are just two examples of Musk's incredible ability to defy the odds and deliver the most unbelievable efficiency gains known to modern business records. And we haven't even talked about taking Tesla from producing 35,000 cars in 2014 to making 1.7 million in 2024. Or turning xAI into a major force in AI by assembling a 100,000 H100 cluster at "superhuman" pace.  Who wouldn't want such a capacity involved in finding the waste, sloth, and squander in the US budget? Well, his political enemies, of course! And I get it. Musk's magic is balanced with mania and even a dash of madness. This is usually the case with truly extraordinary humans. The taller they stand, the longer the shadow. Expecting Musk to do what he does and then also be a "normal, chill dude" is delusional. But even so, I think it's completely fair to be put off by his tendency to fire tweets from the hip, opine on world affairs during all hours of the day, and offer his support to fringe characters in politics, business, and technology. I'd be surprised if even the most ardent Musk super fans don't wince a little every now and then at some of the antics. And yet, I don't have any trouble weighing those antics against the contributions he's made to mankind, and finding an easy and overwhelming balance in favor of his positive achievements. Musk is exactly the kind of formidable player you want on your team when you're down two trillion to nothing, needing a Hail Mary pass for the destiny of America, and eager to see the West win the future. He's a modern-day Knudsen on steroids (or Ketamine?). Let him cook.

5 hours ago 2 votes
Unexpected errors in the BagIt area

Last week, James Truitt asked a question on Mastodon: James Truitt (he/him) @linguistory@code4lib.social Mastodon #digipres folks happen to have a handy repo of small invalid bags for testing purposes? I'm trying to automate our ingest process, and want to make sure I'm accounting for as many broken expectations as possible. Jan 31, 2025 at 07:49 PM The “bags” he’s referring to are BagIt bags. BagIt is an open format developed by the Library of Congress for packaging digital files. Bags include manifests and checksums that describe their contents, and they’re often used by libraries and archives to organise files before transfering them to permanent storage. Although I don’t use BagIt any more, I spent a lot of time working with it when I was a software developer at Wellcome Collection. We used BagIt as the packaging format for files saved to our cloud storage service, and we built a microservice very similar to what James is describing. The “bag verifier” would look for broken bags, and reject them before they were copied to long-term storage. I wrote a lot of bag verifier test cases to confirm that it would spot invalid or broken bags, and that it would give a useful error message when it did. All of the code for Wellcome’s storage service is shared on GitHub under an MIT license, including the bag verifier tests. They’re wrapped in a Scala test framework that might not be the easiest thing to read, so I’m going to describe the test cases in a more human-friendly way. Before diving into specific examples, it’s worth remembering: context is king. BagIt is described by RFC 8493, and you could create invalid bags by doing a line-by-line reading and deliberately ignoring every “MUST” or “SHOULD” but I wouldn’t recommend this aproach. You’d get a long list of test cases, but you’d be overwhelmed by examples, and you might miss specific requirements for your system. The BagIt RFC is written for the most general case, but if you’re actually building a storage service, you’ll have more concrete requirements and context. It’s helpful to look at that context, and how it affects the data you want to store. Who’s creating the bags? How will they name files? Where are you going to store bags? How do bags fit into your wider systems? And so on. Understanding your context will allow you to skip verification steps that you don’t need, and to add verification steps that are important to you. I doubt any two systems implement the exact same set of checks, because every system has different context. Here are examples of potential validation issues drawn from the BagIt specification and my real-world experience. You won’t need to check for everything on this list, and this list isn’t exhaustive – but it should help you think about bag validation in your own context. The Bag Declaration bagit.txt This file declares that this is a BagIt bag, and the version of BagIt you’re using (RFC 8493 §2.1.1). It looks the same in almost every bag, for example: BagIt-Version: 1.0 Tag-File-Character-Encoding: UTF-8 This tightly prescribed format means it can only be invalid in a few ways: What if the bag doesn’t have a bag declaration? It’s a required element of every BagIt bag; it has to be there. What if the bag declaration is the wrong format? It should contain exactly two lines: a version number and a character encoding, in that order. What if the bag declaration has an unexpected version number? If you see a BagIt version that you’ve not seen before, the bag might have a different structure than what you expect. The Payload Files and Payload Manifest The payload files are the actual content you want to save and preserve. They get saved in the payload directory data/ (RFC 8493 §2.1.2), and there’s a payload manifest manifest-algorithm.txt that lists them, along with their checksums (RFC 8493 §2.1.3). Here’s an example of a payload manifest with MD5 checksums: 37d0b74d5300cf839f706f70590194c3 data/waterfall.jpg This tells us that the bag contains a single file data/waterfall.jpg, and it has the MD5 checksum 37d0…. These checksums can be used to verify that the files have transferred correctly, and haven’t been corrupted in the process. There are lots of ways a payload manifest could be invalid: What if the bag doesn’t have a payload manifest? Every BagIt bag must have at least one Payload Manifest file. What if the payload manifest is the wrong format? These files have a prescribed format – one file per line, with a checksum and file path. What if the payload manifest refers to a file that isn’t in the bag? Either one of the files in the bag has been deleted, or the manifest has an erroneous entry. What if the bag has a file that isn’t listed in the payload manifest? The manifest should be a complete listing of all the payload files in the bag. If the bag has a file which isn’t in the payload manifest, either that file isn’t meant to be there, or the manifest is missing an entry. Checking for unlisted files is how I spotted unwanted .DS_Store and Thumbs.db files. What if the checksum in the payload manifest doesn’t match the checksum of the file? Either the file has been corrupted, or the checksum is incorrect. What if there are payload files outside the data/ directory? All the payload files should be stored in data/. Anything outside that is an error. What if there are duplicate entries in the payload manifest? Every payload file must be listed exactly once in the manifest. This avoids ambiguity – suppose a file is listed twice, with two different checksums. Is the bag valid if one of those checksums is correct? Requiring unique entries avoids this sort of issue. What if the payload directory is empty? This is perfectly acceptable in the BagIt RFC, but it may not be what you want. If you know that you will always be sending bags that contain files, you should flag empty payload directories as an error. What if the payload manifest contains paths outside data/, or relative paths that try to escape the bag? (e.g. ../file.txt) Now we’re into “malicious bag” territory – a bag uploaded by somebody who’s trying to compromise your ingest pipeline. Any such bags should be treated with suspicion and rejected. If you’re concerned about malicious bags, you need a more thorough test suite to catch other shenanigans. We never went this far at Wellcome Collection, because we didn’t ingest bags from arbitrary sources. The bags only came from internal systems, and our verification was mainly about spotting bugs in those systems, not defending against malicious actors. A bag can contain multiple payload manifests – for example, it might contain both MD5 and SHA1 checksums. Every payload manifest must be valid for the overall bag to be valid. Payload filenames There are lots of gotchas around filenames and paths. It’s a complicated problem, and I definitely don’t understand all of it. It’s worth understanding the filename rules of any filesystem where you will be storing bags. For example, Azure Blob Storage has a number of rules around how you can name files, and Amazon S3 has different rules. We stored files in both at Wellcome Collection, and so the storage service had to enforce the superset of these rules. I’ve listed some edge cases of filenames you might want to consider, but it’s not a comlpete list. There are lots of ways that unexpected filenames could cause you issues, but whether you care depends on the source of your bags. If you control the bags and you know you’re not going to include any weird filenames, you can probably skip most of these. We only checked for one of these conditions at Wellcome Collection, because we had a pre-ingest step that normalised filenames. It converted filenames to ASCII, and saved a mapping between original and normalised filename in the bag. However, the normalisation was only designed for one filesystem, and produced filenames with trailing dots that were still disallowed in Azure Blob. What if a filename is too long? Some systems have a maximum path length, and an excessively deep directory structure or long filename could cause issues. What if a filename contains special characters? Spaces, emoji, or special characters (\, :, *, etc.) can cause problems for some tools. You should also think about characters that need to be URL-encoded. What if a filename has trailing spaces or dots? Some filesystems can’t support filenames ending in a dot or a space. What happens if your bag contains such a file, and you try to save it to the filesystem? This caused us issues at Wellcome Collection. We initially stored bags just in Amazon S3, which is happy to take filenames with a trailing dot – then we added backups to Azure Blob, which doesn’t. One of the bags we’d stored in Amazon S3 had a trailing dot in the filename, and caused us headaches when we tried to copy it to Azure. What if a filename contains a mix of path separators? The payload manifest uses a forward slash (/) as a path separator. If you have a filename with an alternative path separator, it might behave differently on different systems. For example, consider the payload file a\b\c. This would be a single file on macOS or Linux, but it would be nested inside two folders on Windows. What if the filenames are a mix of uppercase and lowercase characters? Some fileystems are case-sensitive, others aren’t. This can cause issues when you move bags between systems. For example, suppose a bag contains two different files Macrodata.txt and macrodata.txt. When you save that bag on a case-insensitive filesystem, only one file will be saved. What if the same filename appears twice with different Unicode normalisations? This is similar to filenames which only differ in upper/lowercase. They might be treated as two files on one filesystem, but collapsed into one file on another. The classic example is the word “café”: this can be encoded as caf\xc3\xa9 (UTF-8 encoded é) or cafe\xcc\x81 (e + combining acute accent). What if a filename contains a directory reference? A directory reference is /./ (current directory) or /../ (parent directory). It’s used on both Unix and Windows-like systems, and it’s another case of two filenames that look different but can resolve to the same path. For example: a/b, a/./b and a/subdir/../b all resolve to the same path under these rules. This can cause particular issues if you’re moving between local filesystems and cloud storage. Local filesystems treat filenames as hierarchical paths, where cloud storage like Amazon S3 often treats them as opaque strings. This can cause issues if you try to copy files from cloud storage to a local system – if you’re not careful, you could lose files in the process. The Tag Manifest tagmanifest-algorithm.txt Similar to the payload manifest, the tag manifest lists the tag files and their checksums. A “tag file” is the BagIt term for any metadata file that isn’t part of the payload (RFC 8493 §2.2.1). Unlike the payload manifest, the tag manifest is optional. A bag without a tag manifest can still be a valid bag. If the tag manifest is present, then many of the ways that a payload manifest can invalidate a bag – malformed contents, unreferenced files, or incorrect checksums – can also apply to tag manifests. There are some additional things to consider: What if a tag manifest lists payload files? The tag manifest lists tag files; the payload manifest lists payload files in the data/ directory. A tag manifest that lists files in the data/ directory is incorrect. What if the bag has a file that isn’t listed in either manifest? Every file in a bag (except the tag manifests) should be listed in either a payload or a tag manifest. A file that appears in neither could mean an unexpected file, or a missing manifest entry. Although the tag manifest is optional in the BagIt spec, at Wellcome Collection we made it a required file. Every bag had to have at least one tag manifest file, or our storage service would refuse to ingest it. The Bag Metadata bag-info.txt This is an optional metadata file that describes the bag and its contents (RFC 8493 §2.2.2). It’s a list of metadata elements, as simple label-value pairs, one per line. Here’s an example of a bag metadata file: Source-Organization: Lumon Industries Organization-Address: 100 Main Street, Kier, PE, 07043 Contact-Name: Harmony Cobel Unlike the manifest files, this is primarily intended for human readers. You can put arbitrary metadata in here, so you can add fields specific to your organisation. Although this file is more flexible, there are still ways it can be invalid: What if the bag metadata is the wrong format? It should have one metadata entry per line, with a label-value pair that’s separated by a colon. What if the Payload-Oxum is incorrect? The Payload-Oxum contains some concise statistics about the payload files: their total size in bytes, and how many there are. For example: Payload-Oxum: 517114.42 This tells us that the bag contains 42 payload files, and their total size is 517,114 bytes. If these stats don’t match the rest of the bag, something is wrong. What if non-repeatable metadata element names are repeated? The BagIt RFC defines a small number of reserved metadata element names which have a standard meaning. Although most metadata element names can be repeated, there are some which can’t, because they can only have one value. In particular: Bagging-Date, Bag-Size, Payload-Oxum and Bag-Group-Identifier. Although the bag metadata file is optional in a general BagIt bag, you may want to add your own rules based on how you use it. For example, at Wellcome Collection, we required all bags to have an External-Identifier value, that matched a specific schema. This allowed us to link bags to records in other databases, and our bag verifier would reject bags that didn’t include it. The Fetch File fetch.txt This is an optional element that allows you to reference files stored elsewhere (RFC 8493 §2.2.3). It tells the person reading the bag that a file hasn’t been included in this copy of the bag; they have to go and fetch it from somewhere else. The file is still recorded in the payload manifest (with a checksum you can verify), but you don’t have a complete bag until you’ve downloaded all the files. Here’s an example of a fetch.txt: https://topekastar.com/~daria/article.txt 1841 data/article.txt This tells us that data/article.txt isn’t included in this copy of the bag, but we we can download it from https://topekastar.com/~daria/article.txt. (The number 1841 is the size of the file in bytes. It’s optional.) Using fetch.txt allows you to send a bag with “holes”, which saves disk space and network bandwidth, but at a cost – we’re now relying on the remote location to remain available. From a preservation standpoint, this is scary! If topekastar.com goes away, this bag will be broken. I know some people don’t use fetch.txt for precisely this reason. If you do use fetch.txt, here are some things to consider: What if the fetch file is the wrong format? There’s a prescribed format – one file per line, with a URL, optional file size, and file path. What if the fetch file lists a file which isn’t in the payload manifest? The fetch.txt should only tell us that a file is stored elsewhere, and shouldn’t be introducing otherwise unreferenced files. If a file appears in fetch.txt but not the payload manifest, then we can’t verify the remote file because we don’t have a checksum for it. There’s either an erroneous fetch file entry or a missing manifest entry. What if the fetch file points to a file at an unusable URL? The URL is only useful if the person who receives the bag can use it to download the file. If they can’t, the bag might technically be valid, but it’s functionally broken. For example, you might reject URLs that don’t start with http:// or https://. What if the fetch file points to a file with the wrong length? The fetch.txt can optionally specify the size of a file, so you know how much storage you need to download it. If you download the file, the actual size should match the stated size. What if the fetch files points to a file that’s already included in the bag? Now you have two ways to get this file: you can read it from the bag, or from the remote URL. If a file is listed in both fetch.txt and included in the bag, either that file isn’t meant to be in the bag, or the fetch file has an erroneous entry. We used fetch files at Wellcome Collection to implement versioning, and we added extra rules about what remote URLs were allowed. In particular, we didn’t allow fetching a file from just anywhere – you could fetch from our S3 buckets, but not the general Internet. The bag verifier would reject a fetch file entry that pointed elsewhere. These examples illustrate just how many ways a BagIt bag can be invalid, from simple structural issues to complex edge cases. Remember: the key is to understand your specific needs and requirements. By considering your context – who creates your bags, where they’ll be stored, and how they fit into your wider systems – you can build a validation process to catch the issues that matter to you, while avoiding unnecessary complexity. I can give you my ideas, but only you can build your system. [If the formatting of this post looks odd in your feed reader, visit the original article]

6 hours ago 1 votes
Servers can last a long time

We bought sixty-one servers for the launch of Basecamp 3 back in 2015. Dell R430s and R630s, packing thousands of cores and terabytes of RAM. Enough to fill all the app, job, cache, and database duties we needed. The entire outlay for this fleet was about half a million dollars, and it's only now, almost a decade later, that we're finally retiring the bulk of them for a full hardware refresh. What a bargain! That's over 3,500 days of service from this fleet, at a fully amortized cost of just $142/day. For everything needed to run Basecamp. A software service that has grossed hundreds of millions of dollars in that decade. We've of course had other expenses beyond hardware from operating Basecamp over the past decade. The ops team, the bandwidth, the power, and the cabinet rental across both our data centers. But none the less, owning our own iron has been a fantastically profitable proposition. Millions of dollars saved over renting in the cloud. And we aren't even done deriving value from this venerable fleet! The database servers, Dell R630s w/ Xeon E5-2699 CPUs and 768G of RAM, are getting handed down to some of our heritage apps. They will keep on trucking until they give up the ghost. When we did the public accounting for our cloud exit, it was based on five years of useful life from the hardware. But as this example shows, that's pretty conservative. Most servers can easily power your applications much longer than that. Owning your own servers has easily been one of our most effective cost advantages. Together with running a lean team. And managing our costs remains key to reaping the profitable fruit from the business. The dollar you keep at the end of the year is just as real whether you earn it or save it. So you just might want to run those cloud-exit numbers once more with a longer server lifetime value. It might just tip the equation, and motivate you to become a server owner rather than a renter.

yesterday 4 votes
How should we control access to user data?

At some point in a startup’s lifecycle, they decide that they need to be ready to go public in 18 months, and a flurry of IPO-readiness activity kicks off. This strategy focuses on a company working on IPO readiness, which has identified a gap in their internal controls for managing access to their users’ data. It’s a company that wants to meaningfully improve their security posture around user data access, but which has had a number of failed security initiatives over the years. Most of those initiatives have failed because they significantly degraded internal workflows for teams like customer support, such that the initial progress was reverted and subverted over time, to little long-term effect. This strategy represents the Chief Information Security Officer’s (CISO) attempt to acknowledge and overcome those historical challenges while meeting their IPO readiness obligations, and–most importantly–doing right by their users. This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts. Reading this document To apply this strategy, start at the top with Policy. To understand the thinking behind this strategy, read sections in reverse order, starting with Explore, then Diagnose and so on. Relative to the default structure, this document has been refactored in two ways to improve readability: first, Operation has been folded into Policy; second, Refine has been embedded in Diagnose. More detail on this structure in Making a readable Engineering Strategy document. Policy & Operations Our new policies, and the mechanisms to operate them are: Controls for accessing user data must be significantly stronger prior to our IPO. Senior leadership, legal, compliance and security have decided that we are not comfortable accepting the status quo of our user data access controls as a public company, and must meaningfully improve the quality of resource-level access controls as part of our pre-IPO readiness efforts. Our Security team is accountable for the exact mechanisms and approach to addressing this risk. We will continue to prioritize a hybrid solution to resource-access controls. This has been our approach thus far, and the fastest available option. Directly expose the log of our resource-level accesses to our users. We will build towards a user-accessible log of all company accesses of user data, and ensure we are comfortable explaining each and every access. In addition, it means that each rationale for access must be comprehensible and reasonable from a user perspective. This is important because it aligns our approach with our users’ perspectives. They will be able to evaluate how we access their data, and make decisions about continuing to use our product based on whether they agree with our use. Good security discussions don’t frame decisions as a compromise between security and usability. We will pursue multi-dimensional tradeoffs to simultaneously improve security and efficiency. Whenever we frame a discussion on trading off between security and utility, it’s a sign that we are having the wrong discussion, and that we should rethink our approach. We will prioritize mechanisms that can both automatically authorize and automatically document the rationale for accesses to customer data. The most obvious example of this is automatically granting access to a customer support agent for users who have an open support ticket assigned to that agent. (And removing that access when that ticket is reassigned or resolved.) Measure progress on percentage of customer data access requests justified by a user-comprehensible, automated rationale. This will anchor our approach on simultaneously improving the security of user data and the usability of our colleagues’ internal tools. If we only expand requirements for accessing customer data, we won’t view this as progress because it’s not automated (and consequently is likely to encourage workarounds as teams try to solve problems quickly). Similarly, if we only improve usability, charts won’t represent this as progress, because we won’t have increased the number of supported requests. As part of this effort, we will create a private channel where the security and compliance team has visibility into all manual rationales for user-data access, and will directly message the manager of any individual who relies on a manual justification for accessing user data. Expire unused roles to move towards principle of least privilege. Today we have a number of roles granted in our role-based access control (RBAC) system to users who do not use the granted permissions. To address that issue, we will automatically remove roles from colleagues after 90 days of not using the role’s permissions. Engineers in an active on-call rotation are the exception to this automated permission pruning. Weekly reviews until we see progress; monthly access reviews in perpetuity. Starting now, there will be a weekly sync between the security engineering team, teams working on customer data access initiatives, and the CISO. This meeting will focus on rapid iteration and problem solving. This is explicitly a forum for ongoing strategy testing, with CISO serving as the meeting’s sponsor, and their Principal Security Engineer serving as the meeting’s guide. It will continue until we have clarity on the path to 100% coverage of user-comprehensible, automated rationales for access to customer data. Separately, we are also starting a monthly review of sampled accesses to customer data to ensure the proper usage and function of the rationale-creation mechanisms we build. This meeting’s goal is to review access rationales for quality and appropriateness, both by reviewing sampled rationales in the short-term, and identifying more automated mechanisms for identifying high-risk accesses to review in the future. Exceptions must be granted in writing by CISO. While our overarching Engineering Strategy states that we follow an advisory architecture process as described in Facilitating Software Architecture, the customer data access policy is an exception and must be explicitly approved, with documentation, by the CISO. Start that process in the #ciso channel. Diagnose We have a strong baseline of role-based access controls (RBAC) and audit logging. However, we have limited mechanisms for ensuring assigned roles follow the principle of least privilege. This is particularly true in cases where individuals change teams or roles over the course of their tenure at the company: some individuals have collected numerous unused roles over five-plus years at the company. Similarly, our audit logs are durable and pervasive, but we have limited proactive mechanisms for identifying anomalous usage. Instead they are typically used to understand what occurred after an incident is identified by other mechanisms. For resource-level access controls, we rely on a hybrid approach between a 3rd-party platform for incoming user requests, and approval mechanisms within our own product. Providing a rationale for access across these two systems requires manual work, and those rationales are later manually reviewed for appropriateness in a batch fashion. There are two major ongoing problems with our current approach to resource-level access controls. First, the teams making requests view them as a burdensome obligation without much benefit to them or on behalf of the user. Second, because the rationale review steps are manual, there is no verifiable evidence of the quality of the review. We’ve found no evidence of misuse of user data. When colleagues do access user data, we have uniformly and consistently found that there is a clear, and reasonable rationale for that access. For example, a ticket in the user support system where the user has raised an issue. However, the quality of our documented rationales is consistently low because it depends on busy people manually copying over significant information many times a day. Because the rationales are of low quality, the verification of these rationales is somewhat arbitrary. From a literal compliance perspective, we do provide rationales and auditing of these rationales, but it’s unclear if the majority of these audits increase the security of our users’ data. Historically, we’ve made significant security investments that caused temporary spikes in our security posture. However, looking at those initiatives a year later, in many cases we see a pattern of increased scrutiny, followed by a gradual repeal or avoidance of the new mechanisms. We have found that most of them involved increased friction for essential work performed by other internal teams. In the natural order of performing work, those teams would subtly subvert the improvements because it interfered with their immediate goals (e.g. supporting customer requests). As such, we have high conviction from our track record that our historical approach can create optical wins internally. We have limited conviction that it can create long-term improvements outside of significant, unlikely internal changes (e.g. colleagues are markedly less busy a year from now than they are today). It seems likely we need a new approach to meaningfully shift our stance on these kinds of problems. Explore Our experience is that best practices around managing internal access to user data are widely available through our networks, and otherwise hard to find. The exact rationale for this is hard to determine, but it seems possible that it’s a topic that folks are generally uncomfortable discussing in public on account of potential future liability and compliance issues. In our exploration, we found two standardized dimensions (role-based access controls, audit logs), and one highly divergent dimension (resource-specific access controls): Role-based access controls (RBAC) are a highly standardized approach at this point. The core premise is that users are mapped to one or more roles, and each role is granted a certain set of permissions. For example, a role representing the customer support agent might be granted permission to deactivate an account, whereas a role representing the sales engineer might be able to configure a new account. Audit logs are similarly standardized. All access and mutation of resources should be tied in a durable log to the human who performed the action. These logs should be accumulated in a centralized, queryable solution. One of the core challenges is determining how to utilize these logs proactively to detect issues rather than reactively when an issue has already been flagged. Resource-level access controls are significantly less standardized than RBAC or audit logs. We found three distinct patterns adopted by companies, with little consistency across companies on which is adopted. Those three patterns for resource-level access control were: 3rd-party enrichment where access to resources is managed in a 3rd-party system such as Zendesk. This requires enriching objects within those systems with data and metadata from the product(s) where those objects live. It also requires implementing actions on the platform, such as archiving or configuration, allowing them to live entirely in that platform’s permission structure. The downside of this approach is tight coupling with the platform vendor, any limitations inherent to that platform, and the overhead of maintaining engineering teams familiar with both your internal technology stack and the platform vendor’s technology stack. 1st-party tool implementation where all activity, including creation and management of user issues, is managed within the core product itself. This pattern is most common in earlier stage companies or companies whose customer support leadership “grew up” within the organization without much exposure to the approach taken by peer companies. The advantage of this approach is that there is a single, tightly integrated and infinitely extensible platform for managing interactions. The downside is that you have to build and maintain all of that work internally rather than pushing it to a vendor that ought to be able to invest more heavily into their tooling. Hybrid solutions where a 3rd-party platform is used for most actions, and is further used to permit resource-level access within the 1st-party system. For example, you might be able to access a user’s data only while there is an open ticket created by that user, and assigned to you, in the 3rd-party platform. The advantage of this approach is that it allows supporting complex workflows that don’t fit within the platform’s limitations, and allows you to avoid complex coupling between your product and the vendor platform. Generally, our experience is that all companies implement RBAC, audit logs, and one of the resource-level access control mechanisms. Most companies pursue either 3rd-party enrichment with a sizable, long-standing team owning the platform implementation, or rely on a hybrid solution where they are able to avoid a long-standing dedicated team by lumping that work into existing teams.

yesterday 2 votes