Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
13
TokyoDev has already reported that Japan really needs international developers. But the more Japanese companies we’ve interviewed, the more we’ve realized that a talent shortage is not the only reason for Japanese companies to hire from overseas. There are a host of other advantages to recruiting internationally, and a growing number of managers are beginning to recognize the benefits. To gain more perspective on how multinational teams enhance their Japanese companies, we conducted interviews with the following businesses: Autify, which provides an AI-based software test automation platform Beatrust, which provides solutions to help companies maximize their human capital Cybozu, Japan’s leading groupware provider DeepX, which automates heavy equipment machinery Givery, which scouts, hires, and trains world-class engineers KOMOJU by Degica, a payment processor MODE, which pioneers innovative solutions to connect the digital and physical worlds Yaraku, which offers web-based translation...
a week ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from TokyoDev

The Challenges Faced by Multinational Teams and Japanese Companies

It’s a fact that Japan needs more international developers. That doesn’t mean integrating those developers into Japanese companies, as well as Japanese society, is a simple process. But what are the most common challenges encountered by these companies with multinational teams? To find out, TokyoDev interviewed a number of Japanese companies with international employees. In addition to discussing the benefits of hiring overseas, we also wanted to learn more about what challenges they had faced, and how they had overcome them. The companies interviewed included: Autify, which provides an AI-based software test automation platform Beatrust, which has created a search platform that automatically structures profile information Cybozu, Japan’s leading groupware provider DeepX, which automates heavy equipment machinery Givery, which scouts, hires, and trains world-class engineers Shippio, which digitalizes international trading and is Japan’s first digital forwarding company Yaraku, which offers web-based translation management According to those companies, the issues they experienced fell into two categories: addressing the language barrier, and helping new hires come to Japan. The language barrier Language issues are by far the most universal problem faced by Japanese companies with multinational teams. As a result, all of the companies we spoke to have evolved their own unique solutions. AI translation To help improve English-Japanese communication, Yaraku has turned to AI and its own translation tool, YarakuZen. With these they’ve reduced comprehension issues down to verbal communication alone. Since their engineering teams primarily communicate via text anyway, this has solved the majority of their language barrier issue, and engineers feel that they can now work smoothly together. Calling on bilinguals While DeepX employs engineers from over 20 countries, English is the common language between them. Documentation is written in English, and even Japanese departments still write minutes in English so colleagues can check them later. Likewise, explanations of company-wide meetings are delivered in both Japanese and English. Still, a communication gap exists. To overcome it, DeepX assigns Japanese project managers who can also speak English well. English skills weren’t previously a requirement, but once English became the official language of the engineering team, bilingualism was an essential part of the role. These project managers are responsible for taking requests from clients and communicating them accurately to the English-only engineers. In addition, DeepX is producing more bilingual employees by offering online training in both Japanese and English. The online lessons have proven particularly popular with international employees who have just arrived in Japan. Beatrust has pursued a similar policy of encouraging employees to learn and speak both languages. Dr. Andreas Dippon, the Vice President of Engineering at Beatrust, feels that bilingual colleagues are absolutely necessary to business. I think the biggest mistake you can make is just hiring foreigners who speak only English and assuming all the communication inside engineering is just English and that’s fine. You need to understand that business communication with [those] engineers will be immensely difficult . . . You need some almost bilingual people in between the business side and the engineering side to make it work. Similar to DeepX, Beatrust offers its employees a stipend for language learning. “So nowadays, it’s almost like 80 percent of both sides can speak English and Japanese to some extent, and then there are like two or three people on each side who cannot speak the other language,” Dippon said. “So we have like two or three engineers who cannot speak Japanese at all, and we have two or three business members who cannot speak English at all.” But in the engineering team itself “is 100 percent English. And the business team is almost 100 percent Japanese.” “ Of course the leaders try to bridge the gap,” Dippon explained. “So I’m now joining the business meetings that are in Japanese and trying to follow up on that and then share the information with the engineering team, and [it’s] also the same for the business lead, who is joining some engineering meetings and trying to update the business team on what’s happening inside engineering.” “Mixed language” Shippio, on the other hand, encountered negative results when they leaned too hard on their bilingual employees. Initially they asked bilinguals to provide simultaneous interpretation at meetings, but quickly decided that the burden on them was too great and not sustainable in the long term. Instead, Shippio has adopted a policy of “mixed language,” or combining Japanese and English together. The goal of mixed language is simple: to “understand each other.” Many employees who speak one language also know a bit of the other, and Shippio has found that by fostering a culture of flexible communication, employees can overcome the language barrier themselves. For example, a Japanese engineer might forget an English word, in which case he’ll do his best to explain the meaning in Japanese. If the international engineer can understand a bit of Japanese, he’ll be able to figure out what his coworker intended to say, at which point they will switch back to English. This method, while idiosyncratic to every conversation, strikes a balance between the stress of speaking another language and consideration for the other person. The most important thing, according to Shippio, is that the message is conveyed in any language. Meeting more often Another method these companies use is creating structured meeting schedules designed to improve cross-team communication. Givery teams hold what they call “win sessions” and “sync-up meetings” once or twice a month, to ensure thorough information-sharing within and between departments. These two types of meetings have different goals: A “win session” reviews business or project successes, with the aim of analyzing and then repeating that success in future. A “sync-up meeting” helps teams coordinate project deadlines. They report on their progress, discuss any obstacles that have arisen, and plan future tasks. In these meetings employees often speak Japanese, but the meetings are translated into English, and sometimes supplemented with additional English messages and explanations. By building these sort of regular, focused meetings into the company’s schedule, Givery aims to overcome language difficulties with extra personal contact. Beatrust takes a similarly structured, if somewhat more casual, approach. They tend to schedule most meetings on Friday, when engineers are likely to come to the office. However, in addition to the regular meetings, they also hold the “no meeting hour” for everyone, including the business team. “One of the reasons is to just let people talk to each other,” Dippon explained. “Let the engineers talk to business people and to each other.” This kind of interaction, we don’t really care if it’s personal stuff or work stuff that they talk about. Just to be there, talking to each other, and getting this feeling of a team [is what’s important]. . . . This is hugely beneficial, I think. Building Bonds Beatrust also believes in building team relationships through regular off-site events. “Last time we went to Takaosan, the mountain area,” said Dippon. “It was nice, we did udon-making. . . . This was kind of a workshop for QRs, and this was really fun, because even the Japanese people had never done it before by themselves. So it was a really great experience. After we did that, we had a half-day workshop about team culture, company culture, our next goals, and so on.” Dippon in particular appreciates any chance to learn more about his fellow employees. Like, ‘Why did you leave your country? Why did you come to Japan? What are the problems in your country? What’s good in your country?’ You hear a lot of very different stories. DeepX also hopes to deepen the bonds between employees with different cultures and backgrounds via family parties, barbecues, and other fun, relaxing events. This policy intensified after the COVID-19 pandemic, during which Japan’s borders were closed and international engineers weren’t able to immigrate. When the borders opened and those engineers finally did arrive, DeepX organized in-house get-togethers every two weeks, to fortify the newcomers’ relationships with other members of the company. Sponsoring visas Not every company that hires international developers actually brings them to Japan—-quite a few prefer to hire foreign employees who are already in-country. However, for those willing to sponsor new work visas, there is universal consensus on how best to do it: hire a professional. Cybozu has gone to the extent of bringing those professionals in-house. The first international member they hired was an engineer living in the United States. Though he wanted to work in Japan, at that time they didn’t have any experience in acquiring a work visa or relocating an employee, so they asked him to work for their US subsidiary instead. But as they continued hiring internationally, Cybozu realized that quite a few engineers were interested in physically relocating to Japan. To facilitate this, the company set up a new support system for their multinational team, for the purpose of providing their employees with work visas. Other companies prefer to outsource the visa process. DeepX, for example, has hired a certified administrative scrivener corporation to handle visa applications on behalf of the company. Autify also goes to a “dedicated, specialized” lawyer for immigration procedures. Thomas Santonja, VP of Engineering at Autify, feels that sponsoring visas is a necessary cost of business and that the advantages far outweigh the price. We used to have fully remote, long-term employees outside of Japan, but we stopped after we noticed that there is a lot of value in being able to meet in person and join in increased collaboration, especially with Japanese-speaking employees that are less inclined to make an effort when they don’t know the people individually. “It’s kind of become a requirement, in the last two years,” he concluded, “to at least be capable of being physically here.” However, Autify does prevent unnecessary expenses by having a new employee work remotely from their home country for a one month trial period before starting the visa process in earnest. So far, the only serious issues they encountered were with an employee based in Egypt; the visa process became so complicated, Autify eventually had to give up. But Autify also employs engineers from France, the Philippines, and Canada, among other countries, and has successfully brought their workers over many times. Helping employees adjust Sponsoring a visa is only the beginning of bringing an employee to Japan. The next step is providing special support for international employees, although this can look quite different from company to company. DeepX points out that just working at a new company is difficult enough; also beginning a new life in a new country, particularly when one doesn’t speak the language, can be incredibly challenging. That’s why DeepX not only covers the cost of international flights, but also implemented other support systems for new arrivals. To help them get started in Japan, DeepX provides a hired car to transport them from the airport, and a furnished monthly apartment for one month. Then they offer four days of special paid leave to complete necessary procedures: opening a bank account, signing a mobile phone contract, finding housing, etc. The company also introduces real estate companies that specialize in helping foreigners find housing, since that can sometimes be a difficult process on its own. Dippon at Beatrust believes that international employees need ongoing support, not just at the point of entry, and that it’s best to have at least one person in-house who is prepared to assist them. I think that one trap many companies run into is that they know all about Japanese laws and taxes and so on, and everybody grew up with that, so they are all familiar. But suddenly you have foreigners who have basically no idea about the systems, and they need a lot of support, because it can be quite different. Santonja at Autify, by contrast, has had a different experience helping employees get settled. “I am extremely tempted to say that I don’t have any challenges. I would be extremely hard pressed to tell you anything that could be remotely considered difficult or, you know, require some organization or even extra work or thinking.” Most people we hire look for us, right? So they are looking for an opportunity to move to Japan and be supported with a visa, which is again a very rare occurrence. They tend to be extremely motivated to live and make it work here. So I don’t think that integration in Japan is such a challenge. Conclusion To companies unfamiliar with the process, the barriers to hiring internationally may seem high. However, there are typically only two major challenges when integrating developers from other countries. The first, language issues, has a variety of solutions ranging from the technical to the cultural. The second, attaining the correct work visa, is best handled by trained professionals, whether in-house or through contractors. Neither of these difficulties is insurmountable, particularly with expert assistance. In addition, Givery in particular has stressed that it’s not necessary to figure out all the details in advance of hiring: in fact, it can benefit a company to introduce international workers early on, before its own internal policies are overly fixed. This information should also benefit international developers hoping to work in Japan. Since this article reflects the top concerns of Japanese companies, developers can work to proactively relieve those worries. Learning even basic Japanese helps reduce the language barrier, while becoming preemptively familiar with Japanese society reassures employers that you’re capable of taking care of yourself here. If you’d like to learn more about the benefits these companies enjoy from hiring international developers, see part one of this article series here. Want to find a job in Japan? Check out the TokyoDev job board. If you want to know more about multinational teams, moving to Japan, or Japanese work life in general, see our extensive library of articles. If you’d like to continue the conversation, please join the TokyoDev Discord.

3 days ago 5 votes
TokyoDev’s 2024 Recap: Challenges, Milestones, and the Road Ahead

In 2023, I scaled TokyoDev from a one-man operation to a team. The idea was to get some tasks off my plate, but while I’ve succeeded at passing off responsibilities to others, I somehow didn’t gain any more free time. This is because working with new people also created new ideas and opportunities, which I haven’t been able to pass up. In 2024, we saw the first fruits of this collaboration, achieving things I never would have been able to pull off by myself. For instance, we started producing Japanese-language content teaching employers how to build international teams, had a sponsor booth at Japan’s largest Ruby conference, brought the developer community and our clients together through events, and built an editing process that increased the overall quality of our content, and found new contributors who have written some extremely popular articles. As the year winds to a close, I’ve been reflecting on both these accomplishments and the challenges we’ve faced, and how they’re paving the way for what is to come. 65 developers got a job via TokyoDev In 2024, we tracked 65 developers who were successfully hired after applying for a position on TokyoDev. This number was down from last year’s 71 developers. Interestingly, while the total number of hires decreased, the number of companies that hired successfully went up, from 29 to 31. One reason for fewer hires was that several of our most successful clients shifted their focus away from non-Japanese speaking engineers in favour of fluent Japanese speakers. TokyoDev has always been most successful at helping companies with hiring talented engineers with little-to-no Japanese skills, and so with their change in focus, we haven’t been able to help them to the same degree as last year. However, another factor was simply timing. We count successful hires based upon when we receive a fee for them. The time between when a company posts a job to when we receive the fee is typically 3–6 months, as it takes a while for a company to interview candidates, make offers, secure visas, and so on. This means that, even though we currently have a lot of successful hires in the pipeline, they won’t be reflected in this year’s stats. For instance, while we had six successful hires per month in January and February 2024, we have nine projected successful hires in both January and February 2025 I’m optimistic about how we’re going to do next year. 60 articles written by 19 authors One of our greatest accomplishments in 2024 was establishing a repeatable editing process that has allowed us to create extremely high-quality articles. The top five articles by number of visitors were: How I Got a Digital Nomad Visa for Japan by Christian Mack The English Paradox: Four Decades of Life and Language in Japan by Tom Gally The rise and fall of D&D in Japan by Masaki Yanagida How I obtained a J-FIND visa in Japan by Oguzhan Karagözoglu Japan Needs International Developers by Rebecca Callahan The cool thing is that four of the five articles draw upon external contributors’ unique personal experiences, which allowed them to share information with our community that no one else could. Besides our English articles, we also launched a new sub-site that’s helping Japanese companies build global engineering teams. It’s still in its infancy, but we already have 18 articles for it, and we have some other great ideas for the coming year. 18 developer stories published We write developer stories to highlight the experiences of employees at our client companies to give candidates a better understanding of what it’s actually like to work there. This year, we released 18 developer stories. The top five stories by number of visitors were: Realising Dreams of AI and Japan at Recursive “We’re the first global team in Fukuoka”: English Evolution at Money Forward Bringing AI to the Construction Industry with EARTHBRAIN Becoming a Tech Lead at KOMOJU Succeeding as a Senior Engineer at Kraken 814 developers answered our survey Since 2019, TokyoDev has conducted an annual survey of international software developers living in Japan. The 2024 edition was the biggest yet, with 814 developers sharing details on their salaries, working conditions, and the technologies they use. I’ve had people tell me how useful our survey is—some even used it to negotiate better salaries when applying for jobs—so I’m glad it has continued to grow and add to the community. 2,800+ people joined our Discord server In 2024, over 2,800 people joined TokyoDev’s Discord server. This community has proved incredibly valuable. Not only has it helped people get their lives up and running in Japan, but it also has been a great source of inspiration for article topics, and a way to find potential contributors to the site. 7 events hosted In 2024, we continued to expand the in-person meetups we held. Highlights included a pair of events in Okinawa during Ruby Kaigi, an excellent beer garden in collaboration with WAY equity partners (at least I hear it was good, I got COVID the day before), and the launch of our TokyoDev Talks. 5 organizations sponsored TokyoDev owes its origins to the developer community in Japan, so it’s important to me that we use our success to give back to it. We have continued to do this through supporting the following organizations: RubyKaigi: The main Ruby conference in Japan. Attending the 2010 edition was what inspired me to start blogging on this site. Rails Girls Japan: Holds free workshops to help women pick up Ruby on Rails. Tokyo Test Fest: The first edition of an international software quality conference in Japan. Women In Technology Japan: A community that bridges the gender gap in tech and promotes diversity and inclusion in Japan. Women Who Code: This one was heartbreaking for me, as they went bankrupt almost immediately after our sponsorship. That community was succeeded by Women in Software Engineering Japan, where I’m serving as an advisor. 9 people contributed to our team Besides the contributors who wrote articles for the site or made illustrations for them, we have a number of people doing work for us on an ongoing basis. Daniel López Prat improved the infrastructure that runs the TokyoDev site, including adding additional monitoring and keeping our libraries up to date. Keiko Kimoto helped with translation and other administrative tasks, including helping us obtain a trademark for TokyoDev. Mathieu Mayer helped with product design, UX, and frontend development, such as refreshing our articles index. Michelle Tan conducted developer story interviews and helped with full stack development by building things like an admin interface for our clients to use. Rebecca Callahan interviewed contributors, wrote articles for us, and led our editorial team and process. Sacha Greif continued to refine the software that runs our survey, and helped with creating this year’s survey. Sayana Takagi acted as our client representative and led the creation of our sub-site aimed at Japanese companies. Scott Rothrock moderated our Discord community, wrote articles for us, and contributed to our editorial team and process. Looking ahead to 2025 We have a number of exciting things in the works. We’re in the planning phase of several in-person events and have plans to sponsor more communities, and are also working on a new way of connecting international developers with Japanese companies that I hope to be able to talk about soon. Thanks to everyone who has supported us this year. I’m looking forward to continuing to grow together through the next!

a month ago 22 votes
So You Want to Be a Game Dev in Japan

Given how many of us grew up playing classic Japanese games, it’s no surprise that people are keen to work on games in Japan. But what’s the reality on the ground? What skills do you need to succeed in the Japanese game industry, and what challenges can you expect to encounter? To find out, I interviewed a number of current and former members of the game industry here in Japan, for their thoughts on: What makes Japanese game development different? The upside of working on games in Japan The downside How they got started in the Japanese game industry Their top tips for success Conclusion Our interviewees hail from five different countries, work in roles ranging from studio head to entry-level programmer, and have developed everything from Nintendo games to 18+ eroge. Marc Trudel is the Studio Head at Wizcorp, a studio specializing in visual effects, porting games, and developing custom tools and engines for the Japanese game industry. He’s Canadian and has lived in Japan since 2009. Mathieu Siboulotte is a French developer hired as a game designer at the international creative company Studio No Border in 2020. Tristan Metz came from the Netherlands in 2024 to work as a game programmer at AVR Japan, an XR solution company. Minh Nguyen is a Vietnamese developer who worked for DMM Games, which specializes in erotic games (eroge), from 2017–2020 before switching industries. Jared Hays is an American who was employed by the Nintendo-affiliated company Good Feel between 2011 and 2017. He now works for a gaming company in the US. What makes Japanese game development different? Tristan Metz pointed out the most obvious difference: “In the Netherlands we tend to stick to only English, whereas in Japan both Japanese and English often tend to be required.” I also feel that in Japan there is much more of a hierarchy and emphasis on your position within the company. . . . I do think that even the Japanese language itself inherently makes interacting with people feel more formal. “This might also have to do with the fact that the Netherlands tends to have a very flat hierarchy for many organizations,” Metz concluded. Minh Nguyen doesn’t see as much difference in company structure. “I think in terms of team composition it won’t differ that much compared to other countries. We would have a product owner (the term is “director”) who makes initiatives for overall game direction; planners, who design game specs, feature, and gameplay-balancing; designers, who create art and models for the characters and environments and UIs in-game; and developers, who implement back-end or front-end. Testers are usually from outsourcing companies who do testing and verify behavior, and planners would also have to test new releases themselves.” According to Nguyen, the development process is about the same as well. “We would also use Scrum/Kanban to iterate development cycles, as seen in other software companies,” he said. What he has noticed is that Japan focuses on a different category of games. Games in Japan are mostly RPG-flavored [social] games that have characters and a lottery system called gacha to make money. If you are more into MOBA/FPS/strategy then unfortunately those are nonexistent here. For Marc Trudel, the real appeal of developing in Japan is the intense passion of those in the industry. “I had one client that’s head of R&D,” Trudel told me, “and we’re having dinner together, and we’re talking about what games do we play when we have time off, and he says, ‘I don’t play games.’ I was like, ‘Okay, so what do you do?’ ‘Well, I read about neurology in infants and children, to try to understand how the brain works in the context of gaming.’” “He’s not leading game projects,” Trudel added, “and yet he clearly has an interest in how those game projects can have a beneficial impact on younger people. . . . It impresses me.” That dedication, and that passion [Japanese developers] have . . . I don’t want to say it’s not found overseas, but here it seems to me like it’s so consistent. Everyone I talk to is going to have a story of their own, a focus of their own. But Jared Hays thinks that trademark passion can easily manifest as stubborn single-mindedness. “There’s a certain amount of, ‘If it’s not working, just try harder.’ “One of the biggest differences, and certainly the biggest in work culture, was that in Japan there was very little interest in improving processes. So it was, ‘Well, we made the last game this way and it’s shipped, and it sold, and we made money, so we’re just gonna make the next game the same way.’” And there was really no room to say, ‘Have you tried doing X?’ Or, ‘Yes, we made the game, but we did this thing that drove people crazy, people quit, everyone got burned out.’ It was not a good way to make the game. [They] just said, ‘Yes, but it worked.’ Hays offered an example: “Yoshi’s Woolly World was really built on top of Kirby’s Epic Yarn, on top of the tech stack that they had built, which was in turn built on top of a WarioWare game they had made. So those first two were for the Wii, and then Woolly World was for the Wii U. And as they moved onto the next game, the scale of the game and the things they tried to do got bigger, because players expected more. . . . So, Woolly World had no project management, no production, zero. There was no issue tracking. There was no schedule planning, nothing.” I went to school for both computer science and game development, and one of the courses that I found incredibly valuable was a course, not in the nuts and bolts of programming, but in software engineering that really taught working in a team and coordinating with people, communication, delivering milestones, all of the things that most modern devs consider the other 80 percent of the job. And they just had none of that. “If we could align our schedules so that people aren’t sitting around waiting for other people half the week, maybe people wouldn’t need to work overtime,” Hays said. “Maybe the director wouldn’t need to keep a sleeping bag under his desk.” A lack of project management was the biggest difference Hays found, but he also pointed out that Japanese developers are working with more limited resources than their counterparts overseas. “Japanese devs are really harmed by the lack of a Japanese Stack Overflow.” Because in English, if you Google a programming problem, there is an answer. And in Japanese, you Google and it’s just some random guy’s blog where he’s like, ‘Hey, I tried using Unity for the weekend and here’s what I found out.’ So there’s much less centralized information and information exchange between developers. The upside of working on games in Japan For some developers, the best part about working in the Japanese game industry is the games themselves. “My favourite thing about game development in Japan,” said Metz, “are the cool projects and opportunities it brings, which was also the biggest reason for me to move here. Japan is well-known for its famous games from Nintendo, Sega, Square Enix, and countless more. There are a plethora of opportunities in this country and that really excites me.” Hays concurred. I literally made a game with Yoshi. I would never, ever get that here [in the US]. Hays and Metz agreed that collaborating with their coworkers proved an incredible perk. “Like I said,” Hays told me, “some of the people who started the company were super veterans and were incredibly knowledgeable. And it was really awesome to get to work with most of the people, [though] not all of the people. So experientially, it was great.” Trudel has also enjoyed working with Japanese game developers. They know all the games through and through. They have really pointed opinions on what they think is good. . . . It’s a craft for them. It’s something they dedicate their life to. Sometimes, that dedication proves almost uncomfortable, at least for Nguyen. “The games I had worked on were all 18+. . . . During the title alpha/beta release they made the whole company playtest it. I had to play through explicit content along with the surrounding people.” Still, he was impressed by how wholeheartedly they tackled each project. “For an adult game title, the people around me were extremely serious and very committed to making the game become a hit. That was a unique feeling and experience for me.” It’s a bit different for Mathieu Siboulotte, working at a small studio with an international team. “So far, the working environment in my studio is kind of unique. We almost never do any overtime, we have some flexible hours and some remote work days. We are a very small structure of only four people, so we kind of come when we want and leave when we want, as long as we do our hours! For my project, my team is split between France and Japan, so my hours are mostly in the late morning until the evening, so we can share many hours together!” But like the other designers, he draws inspiration from his colleagues. “I regularly join a meetup of French game devs in Japan or the Tokyo Indie Game Show in Akihabara. It is great to test new prototypes and connect with people there!” The downside As for the downside of game development in Japan, Trudel mourns the vanishing culture of mentorship. “I feel like this is kind of getting lost in Japan,” he explained, “that senpai [older mentor] that’s going to take you under their wing.” Maybe it’s specific to the game industry, but I’m starting to see there are not a lot of people that even want to take those responsibilities—or for those who do, sometimes it’s going to be a bit more of a power trip. In general, he explained, game projects in Japan tend to lack both money and expertise. “They’re all built on custom engines, yet every R&D project is underfunded. And not only is it underfunded, but they just can’t find the resources to really do the job to bring their technological assets to the next level, right? “It’s true in R&D, and to some degree it’s true for game-making proper as well, where they don’t quite have the technical abilities to really fully [realize] their creative vision. So that’s really where we [Wizcorp] come in . . . to try to fill in the gaps.” Hays also noted the lack of good leadership. The mentorship wasn’t great, and people I was supposed to be seeking advice from as more senior engineers often were senior because they’d been there a while, and not necessarily because they were incredibly knowledgeable or good at passing that knowledge along. He explained that “Right after I worked on Woolly World, I worked on a game that was on mobile and on Facebook, and that was part of our foray into self-published first party IP. “It was me and one senior dev. He was doing all of the client stuff, I was doing all of the server stuff. It was in Unity, which I had used in school, and the backend was on Google Cloud Platform, which I had also used in school. And we approached a milestone that was like, it should be playable by now. We got to a point where the server was stood up enough and the client was stood up enough that they should connect and you could actually experience gameplay. “And we turned it on and the client performance was so bad, it was unplayable. And I turned to the senior dev, who, again, has been in the industry probably since I was born. . . . I was like, ‘Did you run the profiler? I looked at the profiler. . . . ‘It’s going crazy doing all of this sprite rendering, but it looks like you wrote this rendering code.’ “He’s like, ‘Well, yeah, that’s how you have to render the sprites.’ No, it’s a game engine. It does that. You didn’t need to do this. And so again, as the only server engineer, I had to take several weeks to rewrite the client because he had no idea what he was doing and didn’t attempt to find any of this information.” In general, Hays found, the emphasis on appearances undermined genuine efficacy. “Shortly after I started, the head of the programming department took me aside. [He said], “People don’t like that you’re leaving on time.’ ‘Am I behind on any of my work?’ ‘No, it just looks bad.’” This was the same head of the programming department who told me that the company wouldn’t hire female programmers because they would distract the male programmers. “Never mind the fact that probably 30 to 40 percent of the company is women,” Hays said. “The art department had no problem hiring them. The design department was fine. It was just the programming department.” Nguyen confirmed that most game developers in Japan fit a narrow profile: young, unmarried, and willing to put in any amount of overtime. “Once you have a family, things change,” he said. “Most game companies’ demographics are single and young people, often with little to no responsibilities outside work, and once your priorities shift from games to family you start feeling like you are an outsider.” The long hours in particular are difficult to manage, according to Nguyen. “There are crunch times before each release, and this can be stressful depending on how the project is managed.” My friend and I used to have to work long hours of overtime during the pre-release period. It can suck when people around you have no responsibilities outside work and happily put in the hours while you can’t. It was these pressures that led Nguyen to switch to a different industry. He also agreed with Trudel that the tech at many game companies is outdated. “There is an inertia to upgrade,” he said. “The most important thing in a game is not tech, it is gameplay and art.” As for Metz, he mostly would like to be paid more. My least favourite thing about game development in Japan is probably the low wages. Wages in Japan are a lot lower than in the Netherlands, but I have the impression that programmers have better financial opportunities in other programming industries outside of gaming. “I have the impression this difference is also not as big in the Netherlands,” he added, “but I could be completely wrong on that.” Hays encountered the same pay problem. “The pay is terrible. Just for reference, I looked up one of my old withholding slips from 2013, and I was taking home less than 4 million. That is bad. And I tripled my salary by moving back to the US.” How they got started in the Japanese game industry Some of our interviewees landed in the Japanese game industry mostly by accident. Trudel, for instance, originally came to Japan for the martial arts. Between 2007 and 2009 he visited Japan whenever he could, for anywhere between a weekend to several months at a time. “During [one of my longer trips], I would basically be training three times a day, five or six days a week,” he said. “I would just be dojo-hopping basically, to kind of get my bearings in terms of figuring out if I saw something in it. But what I ended up sticking to was basically the classical martial arts of Japan.” He finally moved officially to Japan on October 20th in 2009, on a working holiday visa. “I didn’t actually have the job lined up when I came in,” he explained. I had a little bit of money set aside. I figured okay, well, let’s just try to find a job. Now that I’m here [in Japan], I can visit people in person. You didn’t really have Zoom at the time. . . . When I started at Wizcorp, I was hired at the same time as one other person, but I was essentially employee number three.” “At the time I was hired as an engineer,” Trudel said, “and ended up doing all the IT stuff.” His official title was System Architect and Network Administrator. “And from there, once we started to move gradually into games from 2010, I started to take [on a] technical leadership position.” Having also served as CTO and COO of Wizcorp, Trudel became the Studio Head in February of 2023. As of now, he’s worked at Wizcorp for over 15 years. Nguyen also was always interested in coming to Japan. Our uni curriculum offered Japanese lessons and allowed the credits to accumulate as well, so I had studied Japanese and gotten my N1 even before my very first trip to Japan. A big thanks to my uni, which made this miracle happen! By contrast, Siboulotte chose the game industry, but not Japan specifically. He started learning game development with RPG Maker when he was 16, but didn’t study game development in university. “I thought joining this industry was impossible,” he said. Instead, he majored in international trade, while continuing game projects on the side. He then received a bachelor’s degree in cultural product marketing, which gave him the opportunity to join an animation studio as a producer’s assistant. “However, game development was still on my mind,” Siboulotte said, “so I decided to leave and start university again from scratch, to study game design, in 2017.” He got his lucky break with Studio No Border, an international creative studio that’s affiliated with the French entertainment group Ankama. “I joined this project in 2020,” Siboulotte said, “right before Japan closed in a lockdown. It was also my first ‘real’ position in the game industry.” I honestly absolutely arrived here by chance. I was just looking for my first gig after my graduation and internship, and this job happened to be the first one to give me a reply! “I applied on the website AFJV, a French website listing games positions in France or with French language involved, and after around four months of tests and interviews, I finally got a green light to come to Japan!” Hays and Metz were both specifically interested in working on Japanese games. While still a college student, Hays spent time in Osaka and loved it. After graduating in 2011, he returned to Japan and was quickly introduced to Good Feel, where he got to pursue his dream of working on Nintendo games. Metz told me there was a specific moment that clinched his desire to come to Japan. The final push to want to commit to the game dev industry in Japan was a 2017 GDC talk by Nintendo about The Legend of Zelda: Breath of the Wild. However, Metz was also well-prepared for the move. “I have always been fascinated with Japanese culture from a young age,” he said, “and have been self-studying the language off and on for around 9 years.” Their top tips for success The most consistent advice interviewees offered: learn Japanese. “It might be obvious for those wanting to work in Japan,” said Metz, “but I think that game development in Japan is much more reclusive than IT companies in Japan and will require a very high level of language fluency.” Nguyen agreed. You need to speak a very high level of Japanese. N3 or even N2 might not be enough to collaborate effectively. From there, however, the advice began to differ, depending on whether interviewees thought that game development in Japan was a good long-term career goal, or whether it should be for the short-term only. “Be sure to get out of the industry while you are still young and you want to advance your career more in tech,” Nguyen stressed. “The game industry doesn’t usually put as much emphasis on tech as others and it is certainly not a tech-driven industry, so staying there a long time can be detrimental to your career.” If you value experience working with tech, plan your departure even before entering the game industry. My friend and I struggled a lot when we were trying to jump ship. Hays concurred with Nguyen’s advice. “Treat it like a gap year,” he said. “Spend three, four, five years doing this because it’s what you really want to do. And then probably take that and go home.” “I’m not going to say, ‘No, never do it, stay away, don’t touch it with a 30 foot pole’,” Hays also said. “But make sure you understand what you’re getting yourself into and that in all likelihood, you’re probably sacrificing some career progression and certainly income for the sake of this opportunity.” Above all, Hays believes it’s important to do your research before going in. I will say as someone who loves video games, loves Japan, loves Japanese video games, if you’re doing it because you have an idealized vision of what working on your favorite games must be like, then you should spend time looking at real-world information. “Look for testimonials of people who worked at [those] companies, look for information about the pay and the working conditions. Just because it’s your dream doesn’t mean you shouldn’t research the company like you would a company in your home country.” Trudel also emphasized the need for research beforehand. “Do visit first. Being a tourist is not the same thing as living here, but it’ll give you some idea.” From there, Trudel’s advice differs, because he’s more optimistic about game development prospects in Japan. It does require, he believes, a great deal of commitment and the willingness to adapt. Japanese society works very differently. The game industry works very differently. Every client is going to work very differently, culturally speaking, and you need to find a way to acclimate to that, and blend in to some degree. “You need to be able to find a way to communicate,” Trudel went on, “where it’s going to make things move in the right direction.” It can be tricky, he told me, but “it’s not impossible.” Japanese ability of course helps immensely: “I mean, the one big mistake that I made was not learning the language enough before coming here,” Trudel admitted. “This set me back some years.” But language skills aren’t the whole picture. “I know that for me, as much as I struggled with language early on because I couldn’t understand what people were saying. . . . Because I couldn’t understand, I had to pick up on nonverbal cues more and on all etiquette stuff a lot more quickly, just to not get into trouble. Basically, it made me pay more attention to things.” Language is not enough. Language is just the gateway to the culture, really. From there, you have to walk through it. “I think a good way to do that,” Trudel advised, “is to engage in cultural activities. It could be sports, I mean, you can go and join a volleyball team for all I care, but having these kinds of activities where you need some form of interaction, some form of communication, verbal and nonverbal, to be able to engage in the activity, will make a big difference. “Plus it’s going to give you a bit of a network, beyond just having colleagues at work.” Trudel is strongly in favor of all forms of networking, even before you come to Japan. “Reach out,” he suggested. “I mean, you have LinkedIn, you have Facebook, you have all these social networks where there are some groups for the Japanese gaming industry. Talk to people, ask questions, see what they’re about.” He clarified, though, that you’ll get better results if you focus on gathering information over clinching a job. “[Those who message me about jobs], every time I’m going to tell them there’s an application process and a candidate selection process, and I’m out of the loop there. “But if you were just asking about whether you might find something to your liking [in Japan], I’m happy to jump on a 30 minute call with you and try to figure it out and just have that discussion. I’m assuming that not everyone is going to be necessarily as willing to engage like that, but it’s just a numbers game, right? “You know, the more people you reach out to, the more people are going to answer back, and then you’re going to be better informed.” Conclusion While the experiences shared here have been subjective, two of the major points—the low wages and the need for Japanese ability—were confirmed in TokyoDev’s 2024 survey. The gaming and education industries had the lowest compensation, with respondents making a median of ¥8 million a year. 37% of respondents in the gaming industry always used Japanese with their colleagues, the highest percentage of any industry. So if you’re interested in being a game developer in Japan, it’s best to start studying Japanese as soon as possible (and perhaps to be independently wealthy). But if a conventional Japanese game company isn’t for you, there are also a number of international game studios that can offer a more flexible and English-speaking environment. If you’re both ambitious and determined, like Siboulotte, you might aim to strike out on your own. The [Japanese] indie scene looks huge, and it gave me the will to make my own indie game besides work. I am really looking forward to BitSummit in Kyoto to try and show my prototype at a big fair again! Wizcorp and other game companies in Japan are hiring now, so check out our game development jobs page. Want to hear more experiences? Continue the discussion in our Discord community.

a month ago 25 votes
Remote Worker Rights In Japan

Are you working remotely for a Japanese company? What happens if your company suddenly issues a return-to-office mandate? Will you have to move back to Tokyo? What if remote work is in your contract—do you have the right to refuse to return to the office? What standing do you have to negotiate with your company? What are your chances of persuading management to change their minds? These are the questions TokyoDev set out to answer, because return-to-office mandates are on the rise in Japan. Return-to-office mandates: the numbers The TokyoDev 2024 survey showed that, among respondents, only 9% worked five days a week in the office. However, companies are increasingly switching from fully-remote to a hybrid working pattern. In 2023, 43% of survey-takers could work fully remotely if they wished, but in 2024, only 38% could say the same. This trend is not confined to Japan. According to Morgan McKinley, in Hong Kong 91% of companies insisted their employees return to the office, while only around 40% of companies in the UK and Canada are asking the same. Japanese companies come in around the middle, with 62% requesting that their employees come back into the office at least some days. Return-to-office orders are having a direct impact on employee attrition. While only 10% of TokyoDev survey respondents who could choose whether or not to work remotely were interested in changing jobs, 18% of those in a hybrid environment were job-hunting, and 39% of those required to work full-time at the office were actively searching for new roles. 49% of survey-takers valued the ability to work remotely over everything else. Meanwhile, those who had to attend an office were more negative about their workplace in every aspect except job security than those who could work fully remotely. In short, tech companies in Japan should be advised that insisting on in-office work, or even hybrid work, could strongly affect their recruitment and retention of employees. By contrast, those who allow fully-remote work can expect to see a rise in applications. What the Tokyo Labor Bureau has to say For those developers whose companies issue a return-to-office mandate, what are their rights under Japanese law? And what is the experience like for those developers who try to enlist the help of Japan’s foreign worker resource centers? To find out, I called them myself. Each time I represented myself as a developer dealing with an increasingly typical situation: though I had always worked remotely, and had moved to a distant prefecture while doing so, I was suddenly being ordered to report to the Tokyo office once a week. What could I do about it? Labor Standards Advice Hotline I started by calling this hotline, which quickly set me up with an interpreter. With each question she listened to me in English, then spoke to her superiors in Japanese and translated their reply back to me. The upshot of their advice was that I should contact either my local prefectural labor bureau, or (since my hypothetical employer was based in Tokyo) the Tokyo Labor Bureau. There wasn’t much legal advice or support that they could offer other than directing me to the appropriate resources. However, they did tell me that most of my case depended on the exact wording of my contract. If it was specified in my contract that I could work fully remotely, regardless of changes in the company’s work plans, then I had a good chance of insisting on continuing to do so—or, at the very least, negotiating from a position of strength. If, however, my contract said that I could work remotely with the company’s permission, or contingent upon company circumstances, then my only hope was to ask for some kind of compromise. If remote work was specified in my contract, I was told, and the company continued to insist that I come to the Tokyo office when I already lived in another prefecture, then I could be eligible for leave allowance, or a payment of 60% of my salary. They weren’t able to give further details on the subject, however, and again directed me to one of the labor bureaus for more details. The Tokyo Labor Bureau When I rang the Tokyo Labor Bureau and presented them with the same dilemma, it was easy to locate someone who spoke English, but their answers were less optimistic. Essentially, the woman on the phone said, there was no provision for return-to-office mandates—-or indeed, anything about remote work—in Japanese law. This left me with limited options. The Tokyo Labor Bureau has a “resolution system” designed to help employees and companies mediate conflicts. This is available only in Japanese, so I would need to bring a friend or translator; however, it is free of charge. In general, while certainly willing to help, she didn’t seem too optimistic about my chances of pushing back against the company’s order. It would be “kind of hard,” she admitted. Since this was more about my particular contract than general labor law, she also suggested that this was really a matter for the lawyers, and gave me the number for the Foreign Residents Support Center. The Foreign Residents Support Center When I first called the center, their lawyer was busy, but the woman on the phone apologized profusely and asked if they could call me back, which they did within a few hours. The lawyer I spoke to had yet another set of suggestions for my return-to-office scenario, but of the people I’d asked so far, he seemed the most optimistic regarding my chances. Unfortunately, it seems that hard-and-fast answers are difficult to come by. In principle, he said, I should not have to obey the mandate if my contract states that fully-remote work is allowed. In fact, I might be able to do so even if fully-remote work wasn’t specified in my contract. If there was correspondence exchanged when I signed the contract that promised fully-remote work, or possibly even verbal statements (though this would naturally be harder to prove), I could argue that fully-remote work was a “specific condition” of my employment. If there is some rational reason for the once-a-week visit to the office, he went on, then the company would be obligated to pay my commuting costs from the distant prefecture to the Tokyo office. However, like me, he considered it unlikely that the weekly meeting was really all that necessary. Instead, he thought I could press for doing the meeting via video call, and that this would fulfill my obligation to the company without incurring additional hassle or expense for either side. The tricky part was that all of this was speculative, and a lot would depend on specific qualifications. For example, when discussing whether the return-to-office order was actually illegal, he said it depended on several different factors: Was fully-remote work promised in my contract? Could my job be done fully remotely? Does the company have an important reason for this order? Of course, the last question is the hardest to nail down. The company has to compare the necessity of the order to the disadvantage of the employee, I was told, which appears to leave a lot of legal wiggle room for a strict or unscrupulous company. And contrary to what the Labor Standards Advice Hotline had suggested, he did not think I would be eligible for leave allowance. If the company refused to budge, he said I should contact my local bar association or city hall to find representation. If I was unable to locate a lawyer on my own, I was free to call back and they would assist me again. However, like everyone else I spoke to, the lawyer strongly suggested that I attempt to negotiate with my company instead. Given my specific circumstances, he suggested that if the company covered the commuting costs, I could perhaps offer to return to the office once every two weeks. In general, he assured me that I shouldn’t be afraid to bargain in this way, particularly if I worked for a small company that might find me difficult to replace. A less positive experience Sadly, sometimes neither negotiation nor legal action are possible. Several TokyoDev members spoke anonymously on their companies’ return-to-office mandates, and one of them described his own experience in consulting a lawyer. It was a lawyer [where] you get 30 minutes of pre-consultation. I sent him my job description, my contract and stuff, and then he looked it up. He said that even though it’s written in the contract that remote work is possible, there’s no precedent in the Supreme Court. . . . He said that if you want to fight, of course I can help you fight it. But in the end, if you lose or if the company dismisses you in the middle of it, then you have bigger problems. Although the opportunity for remote work had been promised in his job description, the actual employment contracts were more vague in their terms. Technically, the company wasn’t violating the contracts. Employees suspected that the company was using this return-to-office mandate to reduce their workforce without violating Japanese employment laws, but such an assertion would be difficult to prove. In the end, the developer decided against legal action. “I did not try to lawyer my way through because I know, once I file a lawsuit or something like that, then it’s going to be big trouble for me.” He is, however, actively searching for a new role, as were other developers we spoke to who had been ordered back to the office. To be clear about the prospect of retaliation, Japan law is strict about the circumstances under which an employee can be terminated. An employee negotiating in good faith around remote work isn’t an acceptable reason, and would run afoul of Japanese law: An employer is only allowed to dismiss an employee if there are objectively reasonable grounds for dismissal, and dismissal is deemed to be appropriate in light of socially accepted ideas. Furthermore, all possible grounds for dismissal must be clearly stated in the work rules if the dismissal of an employee is to be valid. The union option To Dennis Tesolat, General Secretary at the General Union, the solution to these return-to-office mandates is obvious. He calls it “union math.” If tech workers were to get together, they could command a lot at the negotiation table. I met with both Tesolat and Sonomi Terao, the Executive Officer at the General Union. They believe most developers don’t consider unionizing because they’re office workers rather than in the trades, but they are in a great situation to do so. “There’s power in numbers,” Tesolat said, “but [also] just one person joining can be effective.” The General Union, which is headquartered in Osaka but accepts members from all over Japan, already has at least one worker dealing with an unwanted return-to-office mandate. They wouldn’t mind taking on more such cases. Companies don’t want to fight, they want to make money. But we’re a union, it’s our job, so we don’t mind. In fact, Tesolat said, sometimes zero confrontation is required. Just sending in the notification of an employee’s General Union membership often causes management to back off their demands. “At least somebody else now is watching you,” said Tesolat. “Is it a big help, is it going to change your whole situation? No, but they might leave you alone.” And if they don’t, “You just have more options [with a union]. The chance to negotiate, to be supported by colleagues, the right to dispute. The option of court is always there, but it’s not the first option. Nine out of ten times we solve things without using that court option.” What’s key, he said, is not approaching the negotiating table alone. “Dispute and negotiating—that’s our job. . . . And once you [mess] it up, we can’t help you at that point.” This is especially true if you’re an international developer working for a Japanese company, “because the whole manner of negotiation is different. . . . The chance for a lot of misunderstanding is there.” That’s the thing about negotiating on your own. It’s hard, you don’t always know what to do it . . . and if there’s retaliation from that, ‘So what?’ But if the union does it, and there’s retaliation, there’s trade union law that says you can’t do that. What about retaliation for joining a union? Tesolat laughed and said that in his thirty years of experience, he’s seen fewer than ten straightforward retaliation cases. That leaves open the possibility of indirect retaliation, but Tesolat again pointed out that the union exists to deal with precisely that sort of issue. In short, “I would worry about a lot of other things before I’d worry about joining a union.” Two years ago, the General Union didn’t even have an IT branch. During the pandemic, however, the General Union—which had initially confined its membership to the Kansai area—began accepting applications from all over Japan, and from a greater variety of professions. “People were getting fired, they weren’t getting paid, and we couldn’t say no,” Tesolat said, “so we opened the door.” As a result, membership shot up by 35%. Recently, they’ve seen another surge in tech worker applications: “A lot of people started getting scared after the layoffs in America.” With return-to-office mandates increasing, the General Union may see their numbers continue to rise—and that’s good for “union math.” How to reverse your company’s return-to-office policy One anonymous developer we spoke to successfully reversed the return-to-office policy for his entire team. During Covid, he told us, the team worked fully remotely, but after the pandemic was over the management team insisted that developers return to the office five days a week. Eventually our interviewee was able to persuade them to restore remote work, first on a hybrid basis, and later full-time. “This was at a very small company though,” he explained, “where we had more leverage than what you would normally expect in a midsize or big company. Since the push also came from me as the lead developer, management eventually accepted it.” “I was the first developer in the company,” he added, “and I was often asked about what we needed to do to get a dev department running. “One of the things I mentioned is that it is hard to keep developers for a long time, so you need incentives. You either give them a raise, or benefits, or both. Since it was a small company it had no way to compete with bigger ones when it came to benefits and salary. So, what else can you do? If you are small and agile, you can afford to give remote work benefits, as it will cost you little or nothing to do so.” From the company’s perspective, my point was more about being able to retain people and also have an easier time finding new ones. The cost of hiring and onboarding a new developer is quite high in my experience. If you have a good worker and you are not in a position to be giving raises to everyone, remote work is an easy way to keep your devs happy. The fact that other companies do not offer it also means that it is harder for them to be poached. I asked if he had any advice for other developers who wanted remote work. “If I were to give any tips to other developers that are unhappy with their situation,” he said, “it would be to let their company know about it. “For example, they probably have a one-on-one discussion with their manager every so often. This is a good opportunity to ask if the company is considering remote work, [explain] why they want it, and so on. I would not expect a change immediately, especially if they are in a more traditional Japanese company. But consistently asking about it and showing that it really matters is what made the devs here get remote work. ”[Ask the company to] try running a trial with just one developer or two, and evaluate the pros and cons. If the company outright states that it will never allow it no matter what, or it becomes clear that they will not do it, I would start looking for new opportunities that provide the benefit.” So, it is always about leverage. If the company does not think you are worth what you are asking, your only choice is to go to a place that thinks you are worth it. And, of course, keep studying and learning to improve the chances of someone thinking that you are. Conclusion In the TokyoDev 2024 survey, there’s a clear correlation between in-office work and job-hunting. Full-time office workers are looking for new opportunities at the highest rate, followed by hybrid workers, whereas only 10% of fully-remote workers are looking for new roles. As more companies become aware of how highly their employees prioritize remote work, we should expect to see a decline in return-to-office mandates. Even those who don’t wish to change jobs may be able to use this trend to negotiate with their companies. Of course, not all of those negotiations will be successful, and the advice offered by Japanese labor bureaus and legal support centers can be highly variable. However, most of the people I contacted were supportive and helpful. Perhaps, if you encounter negativity or opposition from government workers, you should avail yourself of the old immigration tactic: if you don’t like the answer you got, ask someone else. Has your company asked you to return to the office? We have a list of fully remote developer jobs for you. If you want to continue the conversation, join our Discord community.

2 months ago 22 votes

More in programming

Five Kinds of Nondeterminism

No newsletter next week, I'm teaching a TLA+ workshop. Speaking of which: I spend a lot of time thinking about formal methods (and TLA+ specifically) because it's where the source of almost all my revenue. But I don't share most of the details because 90% of my readers don't use FM and never will. I think it's more interesting to talk about ideas from FM that would be useful to people outside that field. For example, the idea of "property strength" translates to the idea that some tests are stronger than others. Another possible export is how FM approaches nondeterminism. A nondeterministic algorithm is one that, from the same starting conditions, has multiple possible outputs. This is nondeterministic: # Pseudocode def f() { return rand()+1; } When specifying systems, I may not encounter nondeterminism more often than in real systems, but I am definitely more aware of its presence. Modeling nondeterminism is a core part of formal specification. I mentally categorize nondeterminism into five buckets. Caveat, this is specifically about nondeterminism from the perspective of system modeling, not computer science as a whole. If I tried to include stuff on NFAs and amb operations this would be twice as long.1 1. True Randomness Programs that literally make calls to a random function and then use the results. This the simplest type of nondeterminism and one of the most ubiquitous. Most of the time, random isn't truly nondeterministic. Most of the time computer randomness is actually pseudorandom, meaning we seed a deterministic algorithm that behaves "randomly-enough" for some use. You could "lift" a nondeterministic random function into a deterministic one by adding a fixed seed to the starting state. # Python from random import random, seed def f(x): seed(x) return random() >>> f(3) 0.23796462709189137 >>> f(3) 0.23796462709189137 Often we don't do this because the point of randomness is to provide nondeterminism! We deliberately abstract out the starting state of the seed from our program, because it's easier to think about it as locally nondeterministic. (There's also "true" randomness, like using thermal noise as an entropy source, which I think are mainly used for cryptography and seeding PRNGs.) Most formal specification languages don't deal with randomness (though some deal with probability more broadly). Instead, we treat it as a nondeterministic choice: # software if rand > 0.001 then return a else crash # specification either return a or crash This is because we're looking at worst-case scenarios, so it doesn't matter if crash happens 50% of the time or 0.0001% of the time, it's still possible. 2. Concurrency # Pseudocode global x = 1, y = 0; def thread1() { x++; x++; x++; } def thread2() { y := x; } If thread1() and thread2() run sequentially, then (assuming the sequence is fixed) the final value of y is deterministic. If the two functions are started and run simultaneously, then depending on when thread2 executes y can be 1, 2, 3, or 4. Both functions are locally sequential, but running them concurrently leads to global nondeterminism. Concurrency is arguably the most dramatic source of nondeterminism. Small amounts of concurrency lead to huge explosions in the state space. We have words for the specific kinds of nondeterminism caused by concurrency, like "race condition" and "dirty write". Often we think about it as a separate topic from nondeterminism. To some extent it "overshadows" the other kinds: I have a much easier time teaching students about concurrency in models than nondeterminism in models. Many formal specification languages have special syntax/machinery for the concurrent aspects of a system, and generic syntax for other kinds of nondeterminism. In P that's choose. Others don't special-case concurrency, instead representing as it as nondeterministic choices by a global coordinator. This more flexible but also more inconvenient, as you have to implement process-local sequencing code yourself. 3. User Input One of the most famous and influential programming books is The C Programming Language by Kernighan and Ritchie. The first example of a nondeterministic program appears on page 14: For the newsletter readers who get text only emails,2 here's the program: #include /* copy input to output; 1st version */ main() { int c; c = getchar(); while (c != EOF) { putchar(c); c = getchar(); } } Yup, that's nondeterministic. Because the user can enter any string, any call of main() could have any output, meaning the number of possible outcomes is infinity. Okay that seems a little cheap, and I think it's because we tend to think of determinism in terms of how the user experiences the program. Yes, main() has an infinite number of user inputs, but for each input the user will experience only one possible output. It starts to feel more nondeterministic when modeling a long-standing system that's reacting to user input, for example a server that runs a script whenever the user uploads a file. This can be modeled with nondeterminism and concurrency: We have one execution that's the system, and one nondeterministic execution that represents the effects of our user. (One intrusive thought I sometimes have: any "yes/no" dialogue actually has three outcomes: yes, no, or the user getting up and walking away without picking a choice, permanently stalling the execution.) 4. External forces The more general version of "user input": anything where either 1) some part of the execution outcome depends on retrieving external information, or 2) the external world can change some state outside of your system. I call the distinction between internal and external components of the system the world and the machine. Simple examples: code that at some point reads an external temperature sensor. Unrelated code running on a system which quits programs if it gets too hot. API requests to a third party vendor. Code processing files but users can delete files before the script gets to them. Like with PRNGs, some of these cases don't have to be nondeterministic; we can argue that "the temperature" should be a virtual input into the function. Like with PRNGs, we treat it as nondeterministic because it's useful to think in that way. Also, what if the temperature changes between starting a function and reading it? External forces are also a source of nondeterminism as uncertainty. Measurements in the real world often comes with errors, so repeating a measurement twice can give two different answers. Sometimes operations fail for no discernable reason, or for a non-programmatic reason (like something physically blocks the sensor). All of these situations can be modeled in the same way as user input: a concurrent execution making nondeterministic choices. 5. Abstraction This is where nondeterminism in system models and in "real software" differ the most. I said earlier that pseudorandomness is arguably deterministic, but we abstract it into nondeterminism. More generally, nondeterminism hides implementation details of deterministic processes. In one consulting project, we had a machine that received a message, parsed a lot of data from the message, went into a complicated workflow, and then entered one of three states. The final state was totally deterministic on the content of the message, but the actual process of determining that final state took tons and tons of code. None of that mattered at the scope we were modeling, so we abstracted it all away: "on receiving message, nondeterministically enter state A, B, or C." Doing this makes the system easier to model. It also makes the model more sensitive to possible errors. What if the workflow is bugged and sends us to the wrong state? That's already covered by the nondeterministic choice! Nondeterministic abstraction gives us the potential to pick the worst-case scenario for our system, so we can prove it's robust even under those conditions. I know I beat the "nondeterminism as abstraction" drum a whole lot but that's because it's the insight from formal methods I personally value the most, that nondeterminism is a powerful tool to simplify reasoning about things. You can see the same approach in how I approach modeling users and external forces: complex realities black-boxed and simplified into nondeterministic forces on the system. Anyway, I hope this collection of ideas I got from formal methods are useful to my broader readership. Lemme know if it somehow helps you out! I realized after writing this that I already talked wrote an essay about nondeterminism in formal specification just under a year ago. I hope this one covers enough new ground to be interesting! ↩ There is a surprising number of you. ↩

7 hours ago 3 votes
Europe must become dangerous again

Trump is doing Europe a favor by revealing the true cost of its impotency. Because, in many ways, he has the manners and the honesty of a child. A kid will just blurt out in the supermarket "why is that lady so fat, mommy?". That's not a polite thing to ask within earshot of said lady, but it might well be a fair question and a true observation! Trump is just as blunt when he essentially asks: "Why is Europe so weak?". Because Europe is weak, spiritually and militarily, in the face of Russia. It's that inherent weakness that's breeding the delusion that Russia is at once both on its last legs, about to lose the war against Ukraine any second now, and also the all-potent superpower that could take over all of Europe, if we don't start World Word III to counter it. This is not a coherent position. If you want peace, you must be strong. The big cats in the international jungle don't stick to a rules-based order purely out of higher principles, but out of self-preservation. And they can smell weakness like a tiger smells blood. This goes for Europe too. All too happy to lecture weaker countries they do not fear on high-minded ideals of democracy and free speech, while standing aghast and weeping powerlessly when someone stronger returns the favor. I'm not saying that this is right, in some abstract moral sense. I like the idea of a rules-based order. I like the idea of territorial sovereignty. I even like the idea that the normal exchanges between countries isn't as blunt and honest as those of a child in the supermarket. But what I like and "what is" need separating. Europe simply can't have it both ways. Be weak militarily, utterly dependent on an American security guarantee, and also expect a seat at the big-cat table. These positions are incompatible. You either get your peace dividend -- and the freedom to squander it on net-zero nonsense -- or you get to have a say in how the world around you is organized. Which brings us back to Trump doing Europe a favor. For all his bluster and bullying, America is still a benign force in its relation to Europe. We're being punked by someone from our own alliance. That's a cheap way of learning the lesson that weakness, impotence, and peace-dividend thinking is a short-term strategy. Russia could teach Europe a far more costly lesson. So too China. All that to say is that Europe must heed the rude awakening from our cowboy friends across the Atlantic. They may be crude, they may be curt, but by golly, they do have a point. Get jacked, Europe, and you'll no longer get punked. Stay feeble, Europe, and the indignities won't stop with being snubbed in Saudi Arabia.

16 hours ago 3 votes
How I create static websites for tiny archives

Last year I wrote about using static websites for tiny archives. The idea is that I create tiny websites to store and describe my digital collections. There are several reasons I like this approach: HTML is flexible and lets me display data in a variety of ways; it’s likely to remain readable for a long time; it lets me add more context than a folder full of files. I’m converting more and more of my local data to be stored in static websites – paperwork I’ve scanned, screenshots I’ve taken, and web pages I’ve bookmarked. I really like this approach. I got a lot of positive feedback, but the most common reply was “please share some source code”. People wanted to see examples of the HTML and JavaScript I was using I deliberately omitted any code from the original post, because I wanted to focus on the concept, not the detail. I was trying to persuade you that static websites are a good idea for storing small archives and data sets, and I didn’t want to get distracted by the implementation. There’s also no single code base I could share – every site I build is different, and the code is often scrappy or poorly documented. I’ve built dozens of small sites this way, and there’s no site that serves as a good example of this approach – they’re all built differently, implement a subset of my ideas, or have hard-coded details. Even if I shared some source code, it would be difficult to read or understand what’s going on. However, there’s clearly an appetite for that sort of explanation, so this follow-up post will discuss the “how” rather than the “why”. There’s a lot of code, especially JavaScript, which I’ll explain in small digestible snippets. That’s another reason I didn’t describe this in the original post – I didn’t want anyone to feel overwhelmed or put off. A lot of what I’m describing here is nice-to-have, not essential. You can get started with something pretty simple. I’ll go through a feature at a time, as if we were building a new static site. I’ll use bookmarks as an example, but there’s nothing in this post that’s specific to bookmarking. If you’d like to see everything working together, check out the demo site. It includes the full code for all the sections in this post. Let’s dive in! Start with a hand-written HTML page (demo) Reduce repetition with JavaScript templates (demo) Add filtering to find specific items (demo) Introduce sorting to bring order to your data (demo) Use pagination to break up long lists (demo) Provide feedback with loading states and error handling (demo 1, demo 2) Test the code with QUnit and Playwright Manipulate the metadata with Python Store the website code in Git Closing thoughts demo) A website can be a single HTML file you edit by hand. Open a text editor like TextEdit or Notepad, copy-paste the following text, and save it in a file named bookmarks.html. <h1>Bookmarks</h1> <ul> <li><a href="https://estherschindler.medium.com/the-old-family-photos-project-lessons-in-creating-family-photos-that-people-want-to-keep-ea3909129943">Lessons in creating family photos that people want to keep, by Esther Schindler (2018)</a></li> <li><a href="https://www.theatlantic.com/technology/archive/2015/01/why-i-am-not-a-maker/384767/">Why I Am Not a Maker, by Debbie Chachra (The Atlantic, 2015)</a></li> <li><a href="https://meyerweb.com/eric/thoughts/2014/06/10/so-many-nevers/">So Many Nevers, by Eric Meyer (2014)</a></li> </ul> If you open this file in your web browser, you’ll see a list of three links. You can also check out my demo page to see this in action. This is an excellent way to build a website. If you stop here, you’ve got all the flexibility and portability of HTML, and this file will remain readable for a very long time. I build a lot of sites this way. I like it for small data sets that I know are never going to change, or which change very slowly. It’s simple, future-proof, and easy to edit if I ever need to. demo) As you store more data, it gets a bit tedious to keep copying the HTML markup for each item. Wouldn’t it be useful if we could push it into a reusable template? When a site gets bigger, I convert the metadata into JSON, then I use JavaScript and template literals to render it on the page. Let’s start with a simple example of metadata in JSON. My real data has more fields, like date saved or a list of keyword tags, but this is enough to get the idea: const bookmarks = [ { "url": "https://estherschindler.medium.com/the-old-family-photos-project-lessons-in-creating-family-photos-that-people-want-to-keep-ea3909129943", "title": "Lessons in creating family photos that people want to keep, by Esther Schindler (2018)" }, { "url": "https://www.theatlantic.com/technology/archive/2015/01/why-i-am-not-a-maker/384767/", "title": "Why I Am Not a Maker, by Debbie Chachra (The Atlantic, 2015)" }, { "url": "https://meyerweb.com/eric/thoughts/2014/06/10/so-many-nevers/", "title": "So Many Nevers, by Eric Meyer (2014)" } ]; Then I have a function that renders the data for a single bookmark as HTML: function Bookmark(bookmark) { return ` <li> <a href="${bookmark.url}">${bookmark.title}</a> </li> `; } Having a function that returns HTML is inspired by React and Next.js, where code is split into “components” that each render part of the web app. This function is simpler than what you’d get in React. Part of React’s behaviour is that it will re-render the page if the data changes, but my function won’t do that. That’s okay, because my data isn’t going to change. The HTML gets rendered once when the page loads, and that’s enough. I’m using a template literal because I find it simple and readable. It looks pretty close to the actual HTML, so I have a pretty good idea of what’s going to appear on the page. Template literals are dangerous if you’re getting data from an untrusted source – it could allow somebody to inject arbitrary HTML into your page – but I’m writing all my metadata, so I trust it. I know there are other ways to construct HTML in JavaScript, like document.createElement(), the <template> element, or Web Components – but template literals have always been sufficient for me, and I’ve never had a reason to explore other options. Now we have to call this function when the page loads, and render the list of bookmarks. Here’s the rest of the code: <script> window.addEventListener("DOMContentLoaded", () => { document.querySelector("#listOfBookmarks").innerHTML = bookmarks.map(Bookmark).join(""); }); </script> <h1>Bookmarks</h1> <ul id="listOfBookmarks"></ul> I’m listening for the DOMContentLoaded event, which occurs when the HTML page has been fully parsed. When that event occurs, it looks for <ul id="listOfBookmarks"> in the page, and inserts the HTML for the list of bookmarks. We have to wait for this event so the <ul> actually exists. If we tried to run it immediately, it might run before the <ul> exists, and then it wouldn’t know where to insert the HTML. I’m using querySelector() to find the <ul> I want to modify – this is a newer alternative to functions like getElementById(). It’s quite flexible, because I can target any CSS selector, and I find CSS rules easier to remember than the family of getElementBy* functions. Although it’s slightly slower in benchmarks, the difference is negligible and it’s easier for me to remember. If you want to see this page working, check out the demo page. I use this pattern as a starting point for a lot of my static sites – metadata in JSON, some functions that render HTML, and an event listener that renders the whole page after it loads. Once I have the basic site, I add data, render more HTML, and write CSS styles to make it look pretty. This is where I can have fun, and really customise each site. I keep tweaking until I have something I like. I’m ignoring CSS because that could be a whole other post, and there’s a vintage charm to unstyled HTML – it’s fine for what we’re discussing today. What else can we do? demo) As the list gets even longer, it’s useful to have a way to find specific items in the list – I don’t want to scroll the whole thing every time. I like adding keyword tags to my data, and then filtering for items with particular tags. If I add other metadata fields, I could filter on those too. Here’s a brief sketch of the sort of interface I like: I like to be able to define a series of filters, and apply them to focus on a specific subset of items. I like to combine multiple filters to refine my search, and to see a list of applied filters with a way to remove them, if I’ve filtered too far. I like to apply filters from a global menu, or to use controls on each item to find similar items. I use URL query parameters to store the list of currently-applied filters, for example: bookmarks.html?tag=animals&tag=wtf&publicationYear=2025 This means that any UI element that adds or removes a filter is a link to a new URL, so clicking it loads a new page, which triggers a complete re-render with the new filters. When I write filtering code, I try to make it as easy as possible to define new filters. Every site needs a slightly different set of filters, but the overall principle is always the same: here’s a long list of items, find the items that match these rules. Let’s start by expanding our data model to include a couple of new fields: const bookmarks = [ { "url": "https://estherschindler.medium.com/the-old-family-photos-project-lessons-in-creating-family-photos-that-people-want-to-keep-ea3909129943", "title": "Lessons in creating family photos that people want to keep, by Esther Schindler (2018)", "tags": ["photography", "preservation"], "publicationYear": "2018" }, … ]; Then we can define some filters we might use to narrow the list: const bookmarkFilters = [ { id: 'tag', label: 'tagged with', filterFn: (bookmark, tagName) => bookmark.tags.includes(tagName), }, { id: 'publicationYear', label: 'published in', filterFn: (bookmark, year) => bookmark.publicationYear === year, }, ]; Each filter has three fields: id matches the name of the associated URL query parameter label is how the filter will be described in the list of applied filters filterFn is a function that takes two arguments: a bookmark, and a filter value, and returns true/false depending on whether the bookmark matches this filter This list is the only place where I need to customise the filters for a particular site; the rest of the filtering code is completely generic. This means there’s only one place I need to make changes if I want to add or remove filters. The next piece of the filtering code is a generic function that filters a list of items, and takes the list of filters as an argument: /* * Filter a list of items. * * This function takes the list of items and available filters, and the * URL query parameters passed to the page. * * This function returns a list with the items that match these filters, * and a list of filters that have been applied. */ function filterItems({ items, filters, params }) { // By default, all items match, and no filters are applied. var matchingItems = items; var appliedFilters = []; // Go through the URL query params one by one, and look to // see if there's a matching filter. for (const [key, value] of params) { console.debug(`Checking query parameter ${key}`); const matchingFilter = filters.find(f => f.id === key); if (typeof matchingFilter === 'undefined') { continue; } // There's a matching filter! Go ahead and filter the // list of items to only those that match. console.debug(`Detected filter ${JSON.stringify(matchingFilter)}`); matchingItems = matchingItems.filter( item => matchingFilter.filterFn(item, value) ); // Construct a new query string that doesn't include // this filter. const altQuery = new URLSearchParams(params); altQuery.delete(key, value); const linkToRemove = "?" + altQuery.toString(); appliedFilters.push({ type: matchingFilter.id, label: matchingFilter.label, value, linkToRemove, }) } return { matchingItems, appliedFilters }; } This function doesn’t care what sort of items I’m passing, or what the actual filters are, so I can reuse it between different sites. It returns the list of matching items, and the list of applied filters. The latter allows me to show that list on the page. linkToRemove is a link to the same page with this filter removed, but keeping any other filters. This lets us provide a button that removes the filter. The final step is to wire this filtering into the page render. We need to make sure we only show items that match the filter, and show the user a list of applied filters. Here’s the new code: <script> window.addEventListener("DOMContentLoaded", () => { const params = new URLSearchParams(window.location.search); const { matchingItems: matchingBookmarks, appliedFilters } = filterItems({ items: bookmarks, filters: bookmarkFilters, params: params, }); document.querySelector("#appliedFilters").innerHTML = appliedFilters .map(f => `<li>${f.label}: ${f.value} <a href="${f.linkToRemove}">(remove)</a></li>`) .join(""); document.querySelector("#listOfBookmarks").innerHTML = matchingBookmarks.map(Bookmark).join(""); }); </script> <h1>Bookmarks</h1> <p>Applied filters:</p> <ul id="appliedFilters"></ul> <p>Bookmarks:</p> <ul id="listOfBookmarks"></ul> I stick to simple filters that can be phrased as a yes/no question, and I rely on my past self to have written sufficiently useful metadata. At least in static sites, I’ve never implemented anything like a fuzzy text search, where it’s less obvious whether a particular item should match. You can check out the filtering code on the demo page. demo) The next feature I usually implement is sorting. I build a dropdown menu with all the options, and picking one reloads the page with the new sort order. Here’s a quick design sketch: For example, I often sort by the date I saved an item, so I can find an item I saved recently. Another sort order I often use is “random”, which shuffles the items and is a fun way to explore the data. As with filters, I put the current sort order in a query parameter, for example: bookmarks.html?sortOrder=titleAtoZ As before, I want to write this in a generic way and share code between different sites. Let’s start by defining a list of sort options: const bookmarkSortOptions = [ { id: 'titleAtoZ', label: 'title (A to Z)', compareFn: (a, b) => a.title > b.title ? 1 : -1, }, { id: 'publicationYear', label: 'publication year (newest first)', compareFn: (a, b) => Number(b.publicationYear) - Number(a.publicationYear), }, ]; Each sort option has three fields: id is the value that will appear in the URL query parameter label is the human-readable label that will appear in the dropdown compareFn(a, b) is a function that compares two items, and will be passed directly to the JavaScript sort function. If it returns a negative value, then a sorts before b. If it returns a positve value, then a sorts after b. Next, we can define a function that will sort a list of items: /* * Sort a list of items. * * This function takes the list of items and available options, and the * URL query parameters passed to the page. * * It returns a list with the items in sorted order, and the * sort order that was applied. */ function sortItems({ items, sortOptions, params }) { // Did the user pass a sort order in the query parameters? const sortOrderId = getSortOrder(params); // What sort order are we using? // // Look for a matching sort option, or use the default if the sort // order is null/unrecognised. For now, use the first defined // sort order as the default. const defaultSort = sortOptions[0]; const selectedSort = sortOptions.find(s => s.id === sortOrderId) || defaultSort; console.debug(`Selected sort: ${JSON.stringify(selectedSort)}`); // Now apply the sort to the list of items. const sortedItems = items.sort(selectedSort.compareFn); return { sortedItems, appliedSortOrder: selectedSort }; } /* Get the current sort order from the URL query parameters. */ function getSortOrder(params) { return params.get("sortOrder"); } This function works with any list of items and sort orders, making it easy to reuse across different sites. I only have to define the list of sort orders once. This approach makes it easy to add new sort orders, and to write a component that renders a dropdown menu to pick the sort order: /* * Create a dropdown control to choose the sort order. When you pick * a different value, the page reloads with the new sort. */ function SortOrderDropdown({ sortOptions, appliedSortOrder }) { return ` <select onchange="setSortOrder(this.value)"> ${ sortOptions .map(({ id, label }) => ` <option value="${id}" ${id === appliedSortOrder.id ? 'selected' : ''}> ${label} </option> `) .join("") } </select> `; } function setSortOrder(sortOrderId) { const params = new URLSearchParams(window.location.search); params.set("sortOrder", sortOrderId); window.location.search = params.toString(); } Finally, we can wire the sorting code into the rest of the app. After filtering, we sort the items and then render the sorted list. We also show the sort controls on the page: <script> window.addEventListener("DOMContentLoaded", () => { const params = new URLSearchParams(window.location.search); const { matchingItems: matchingBookmarks, appliedFilters } = filterItems(…); … const { sortedItems: sortedBookmarks, appliedSortOrder } = sortItems({ items: matchingBookmarks, sortOptions: bookmarkSortOptions, params, }); document.querySelector("#sortOrder").innerHTML += SortOrderDropdown({ sortOptions: bookmarkSortOptions, appliedSortOrder }); document.querySelector("#listOfBookmarks").innerHTML = sortedBookmarks.map(Bookmark).join(""); }); </script> <p id="sortOrder">Sort by:</p> You can check out the sorting code on the demo page. demo) If you have a really long list of items, you may want to break them into multiple pages. This isn’t something I do very often. Modern web browsers are very performant, and you can put thousands of elements on the page without breaking a sweat. I’ve only had to add pagination in a couple of very image-heavy sites – if it’s a text-based site, I just show everything. (You may notice that, for example, there are no paginated lists anywhere on this site. By writing lean HTML, I can fit all my lists on a single page.) If I do want pagination, I stick to a classic design: As with other features, I use a URL query parameter to track the current page number: bookmarks.html?pageNumber=2 This code can be written in a completely generic way – it doesn’t have to care what sort of items we’re paginating. First, let’s write a function that will select a page of items for us. If we’re on page N, what items should we be showing? /* * Get a page of items. * * This function will reduce the list of items to the items that should * be shown on this particular page. */ function paginateItems({ items, pageNumber, pageSize }) { // Page numbers are 1-indexed, so page 1 corresponds to // the indices 0…(pageSize - 1). const startOfPage = (pageNumber - 1) * pageSize; const endOfPage = pageNumber * pageSize; const thisPage = items.slice(startOfPage, endOfPage); return { thisPage, totalPages: Math.ceil(items.length / pageSize), }; } In some of my sites, the page size is a suggestion rather than a hard rule. If there are 27 items and the page size is 25, I think it’s nicer to show all the items on one page than push a few items onto a second page which barely has anything on it. But that might reflect my general dislike of pagination, and it’s definitely a nice-to-have rather than a required feature. Once we know what page we’re on and how many pages there are, we can create a component to render some basic pagination controls: /* * Renders a list of pagination controls. * * This includes links to prev/next pages and the current page number. */ function PaginationControls({ pageNumber, totalPages, params }) { // If there are no pages, we don't need pagination controls. if (totalPages === 1) { return ""; } // Do we need a link to the previous page? Only if we're past page 1. if (pageNumber > 1) { const prevPageUrl = setPageNumber({ params, pageNumber: pageNumber - 1 }); prevPageLink = `<a href="${prevPageUrl}">&larr; prev</a>`; } else { prevPageLink = null; } // Do we need a link to the next page? Only if we're before // the last page. if (pageNumber < totalPages) { const nextPageUrl = setPageNumber({ params, pageNumber: pageNumber + 1 }); nextPageLink = `<a href="${nextPageUrl}">next &rarr;</a>`; } else { nextPageLink = null; } const pageText = `Page ${pageNumber} of ${totalPages}`; // Construct the final result. return [prevPageLink, pageText, nextPageLink] .filter(p => p !== null) .join(" / "); } /* Returns a URL that points to the new page number. */ function setPageNumber({ params, pageNumber }) { const updatedParams = new URLSearchParams(params); updatedParams.set("pageNumber", pageNumber); return `?${updatedParams.toString()}`; } Finally, let’s wire this code into the rest of the app. We get the page number from the URL query parameters, paginate the list of filtered and sorted items, and show some pagination controls: <script> /* Get the current page number. */ function getPageNumber(params) { return Number(params.get("pageNumber")) || 1; } window.addEventListener("DOMContentLoaded", () => { const params = new URLSearchParams(window.location.search); const { matchingItems: matchingBookmarks, appliedFilters } = filterItems(…); const { sortedItems: sortedBookmarks, appliedSortOrder } = sortItems(…); const pageNumber = getPageNumber(params); const { thisPage: thisPageOfBookmarks, totalPages } = paginateItems({ items: sortedBookmarks, pageNumber, pageSize: 25, }); document.querySelector("#paginationControls").innerHTML += PaginationControls({ pageNumber, totalPages, params }); document.querySelector("#listOfBookmarks").innerHTML = thisPageOfBookmarks.map(Bookmark).join(""); }); </script> <p id="paginationControls">Pagination controls: </p> One thing that makes pagination a little tricky is that it affects filtering and sorting as well – when you change either of those, you probably want to reset to the first page. For example, if you’re filtering for animals and you’re on page 3, then you add a second filter for giraffes, you should reset to page 1. If you stay on page 3, it might be confusing if there are less than 3 pages of results with the new filter. The key to this is calling params.delete("pageNumber") when you update the URL query parameters. You can play with the pagination on the demo page. demo 1, demo 2) One problem with relying on JavaScript to render the page is that sometimes JavaScript goes wrong. For example, I write a lot of my metadata by hand, and a typo can create invalid JSON and break the page. There are also people who disable JavaScript, or sometimes it just doesn’t work. If I’m using the site, I can open the Developer Tools in my web browser and start debugging there – but that’s not a great experience. If you’re not expecting something to go wrong, it will just look like the page is taking a long time to load. We can do better. To start, we can add a <noscript> element that explains to users that they need to enable JavaScript. This will only be shown if they’ve disabled JavaScript: <noscript> <strong>You need to enable JavaScript to use this site!</strong> </noscript> I have a demo page which disables JavaScript, so you can see how the noscript tag behaves. This won’t help if JavaScript is broken rather than disabled, so we also need to add error handling. We can listen for the error event on the window, and report an error to the user – for example, if a script fails to load. <div id="errors"></div> <script> window.addEventListener("error", function(event) { document .querySelector('#errors') .innerHTML = `<strong>Something went wrong when loading the page!</strong>`; }); </script> We can also attach an onerror handler to specific script tags, which allows us to customise the error message – we can tell the user that a particular file failed to load. <script src="app.js" onerror="alert('Something went wrong while loading app.js')"></script> I have another demo page which has a basic error handler. Finally, I like to include a loading indicator, or some placeholder text that will be replaced when the page will finish loading – this tells the user where they can expect to see something load in. <ul id="listOfBookmarks">Loading…</ul> It’s somewhat rare for me to add a loading indicator or error handling, just because I’m the only user of my static sites, and it’s easier for me to use the developer tools when something breaks. But providing mechanisms for the user to understand what’s going on is crucial if you want to build static sites like this that other people will use. Test the code with QUnit and Playwright If I’m writing a very complicated viewer, it’s helpful to have tests. I’ve found two test frameworks that I particularly like for this purpose. QUnit is a JavaScript library that I use for unit testing – to me, that means testing individual functions and components. For example, QUnit was very helpful when I was writing the early iterations of the sorting and filtering code, and writing tests caught a number of mistakes. You can run QUnit in the browser, and it only requires two files, so I can test a project without creating a whole JavaScript build system or dependency tree. Here’s an example of a QUnit test: QUnit.test("sorts bookmarks by title", function(assert) { // Create three bookmarks with different titles const bookmarkA = { title: "Almanac for apples" }; const bookmarkC = { title: "Compendium of coconuts" }; const bookmarkP = { title: "Page about papayas" }; const params = new URLSearchParams("sortOrder=titleAtoZ"); // Pass the bookmarks in the wrong order, so they can't be sorted // correctly "by accident" const { sortedItems, appliedSortOrder } = sortItems({ items: [bookmarkC, bookmarkA, bookmarkP], sortOptions: bookmarkSortOptions, params, }); // Check the bookmarks have been sorted in the right order assert.deepEqual(sortedItems, [bookmarkA, bookmarkC, bookmarkP]); }); You can see this test running in the browser in my demo page. Playwright is a testing library that can open a web app in a real web browser, interact with the page, and check that the app behaves correctly. It’s often used for dynamic web apps, but it works just as well for static pages. For example, it can test that if you select a new sort order, the page reloads and show results in the correct order. Here’s an example of a simple test written with Playwright in Python: from playwright.sync_api import expect, sync_playwright with sync_playwright() as p: browser = p.webkit.launch() # Open the HTML file in the browser page = browser.new_page() page.goto('file:///Users/alexwlchan/Sites/sorting.html') # Look for an <li> element with one of the bookmarks -- this will # only appear if the page has rendered correctly. expect(page.get_by_text("So Many Nevers")).to_be_visible() browser.close() These tools are a great safety net for catching mistakes, but I don’t always need them. I only write tests for my more complicated sites – when the sorting/filtering code is particularly complex, there’s a lot of rendering code, or I anticipate making major changes in future. I don’t bother with tests when the site is simple and unlikely to change, and I can just do manual checks when I write it the first time. Tests are less useful if I know I’ll never make changes. This is getting away from the idea of a self-contained static website, because now I’m relying on third-party code, and for Playwright I need to maintain a working Python environment. I’m okay with this, because the website is still usable even if I can no longer run the tests. These are useful sidecar tools, but I only need them if I’m making changes. If I finish a site and I know I won’t change it again, I don’t need to worry about whether the tests will still work years later. Manipulate the metadata with Python For small sites, we could write all this JavaScript directly in <script> tags or in a single file. As we get more data, splitting the metadata and application logic makes everything easier to manage. One pattern I’ve adopted is to put all the item metadata into a single, standalone JavaScript file that assigns a single variable: const bookmarks = […]; and then load that file in the HTML page with a <script src="metadata.js"> element. I use JavaScript rather than pure JSON because browsers don’t allow fetching local JSON files via file://. If you open an HTML page without a web server, the browser will block requests to fetch a JSON file because of security restrictions. By storing data in a JavaScript file instead, I can load it with a simple <script> tag. I wrote a small Python library javascript-data-files that lets me interact with JSON stored this way. This allows me to write scripts that add data to the metadata file (like saving a new bookmark) or to verify the existing metadata (like checking that I have an archived copy of every bookmark). I’ll write more about this in future posts, because this one is long enough already. For example, let’s add a new bookmark to the metadata.js file: from javascript_data_files import read_js, write_js bookmarks = read_js("metadata.js", varname="bookmarks") bookmarks.append({ "url": "https://www.theguardian.com/lifeandstyle/2019/jan/13/ella-risbridger-john-underwood-friendship-life-new-family", "title": "When my world fell apart, my friends became my family, by Ella Risbridger (2019)" }) write_js("metadata.js", varname="bookmarks", value=bookmarks) We’re starting to blur the line between a static site and a static site generator. These scripts only work if I have a working Python environment, which is less future-proof than pure HTML. I’m happy with this compromise, because the website is fully functional without them – I only need to run these scripts if I’m modifying the metadata. If I stop making changes and the Python environment breaks, I can still read everything I’ve already saved. Store the website code in Git I create Git repositories for all of my local websites. This allows me to track changes, and it means I can experiment freely – I can always roll back if I break something. These Git repositories only live on my local machine. I run git init . in the folder, I create commits to record any changes, and that’s it. I don’t push the repository to GitHub or another remote Git server. (Although I do have backups of every site, of course.) Git has a lot of features for writing code in a collaborative environment, but I don’t need any of those here – I’m the only person working on these sites. Most of the time, I just use two commands: $ git add bookmarks.html $ git commit -m "Add filtering by author" This creates a labelled snapshot of my latest changes to bookmarks.html. I only track the text files in Git – the HTML, CSS, and JavaScript. I don’t track binary files like images and videos. Git struggles with those larger files, and I don’t edit those as much as the text files, so having them in version control is less useful. I write a gitignore file to ignore all of them. Closing thoughts There are lots of ideas here, but you don’t need to use all of them – most of my sites only use a few. Every site is different, and you can pick what makes most sense for your project. If you’re building a static site for a tiny archive, start with a simple HTML file. Add features like templates, sorting, and filtering incrementally as they become useful. You don’t need to add them all upfront – that can make things more complicated than they need to be. This approach can scale from simple collections to sophisticated archives. A static website built with HTML and JavaScript is easy to maintain and modify, has no external dependencies, and is future-proof against a lot of technological changes. I’ve come to love using static websites to store my local data. They’re flexible, resilient, and surprisingly powerful. I hope you’ll consider it too, and that these ideas help you get started. [If the formatting of this post looks odd in your feed reader, visit the original article]

yesterday 4 votes
Nobody Profits

Intellectual property is a really dumb idea. “But piracy is theft. Clean and simple. It’s smash and grab. It ain’t no different than smashing a window at Tiffany’s and grabbing merchandise.” - Joe Biden, 46th president of the USA Except it isn’t and Joe Biden is a senile moron. Because when you smash the windows and grab the stuff, Tiffany’s no longer has the stuff. With piracy, everyone has the stuff. It’s a lot more like taking a picture, which Tiffany’s probably encourages. Win-win cooperation. Wealth is being increasingly concentrated. What’s shocking to me is how much everyone still cares about money. Even the die-hard complain about capitalism type deeply cares, because the opposite of love isn’t hate, it’s indifference. I hate scammers, but I’m pretty indifferent to money. The best outcome of AI is if it delivers huge amounts of value to society but no profit to anyone. The old days of the Internet were this goldmine. The Internet delivered huge value but no profit, and that’s why it was good. Suddenly we had all these new powers. Then people figured out how to monetize it. It was a race to extract every tiny bit of value, and now we have today’s Internet. Can this play out differently with AI? Let’s build technology and open source software that market breaks everything. Let’s demoralize the scammers so hard that they don’t even try. Every loser and grifter will be gone from technology because there’s nothing to be gained there. They can play golf all day or something. If I ever figure out how to channel power like Elon, I will do this. Spin up open source projects in every sector to eliminate all the capturable value. This is what I’m trying to do with comma.ai and tinygrad. I dream of a day when company valuations halve when I create a GitHub repo. Someday.

2 days ago 4 votes
My Top 15 OS Books: From Theory and Implementation to Systems Programming

A personal guide to the most useful books for understanding operating systems

2 days ago 7 votes