Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Hixie's Natural Log
When you build something, you have to pick some design goals and priorities. Ideally you do so explicitly, but even if you don't, you're still implicitly doing so based on your design choices. These choices are trade-offs. If you want to write a quiet song, it won't be loud. If you are writing a software tool and you want to prioritize speed over simplicity, then it won't be as simple as if you'd prioritized simplicity over speed. There are two main signs that you've succeeded at your goals. The first, and more pleasant, is that you get compliments about how your thing is like you wanted it to be. "I love that song, it's so quiet!" "Your tool is so fast!" Why thank you, that's exactly what I was going for. The second sign, though, is that you will get complaints. Specifically, people will complain that your thing does not achieve the things you didn't set out to achieve. "I wish this song was louder", "this tool is so hard to use". That you are receiving complaints at all means that people are aware of your creation; that they are complaining about what you specifically set out to make a non-goal is a side-effect of the fact that you made that trade-off. The worst thing to do, when you receive such complaints, is take them to heart and try to fix them. This is because by definition you wanted these complaints. They are a sign that the thing you built is built as you wanted to build it. The people complaining want something different, they don't want your thing. It's just that your thing is so good that it's the thing they're compelled towards even though it doesn't prioritize the things they care about most. If you try to fix these complaints, you will, again by definition, be compromising on your goals. If you make the song have a loud part, then it's no longer a quiet song. You wanted a quiet song. Now it's a song that's quiet in parts and loud in parts. It probably still doesn't satisfy the needs of the people who want a loud song, and now it also doesn't satisfy the needs of the people who wanted your original quiet song. If you make your tool easier to use by compromising on the speed, then now you have a tool that's both not as fast as it could be and not as usable as it could have been if you'd started with that as a goal. It's important, therefore, to separate out complaints into those that are complaints you expect based on your design goals (which you should acknowledge but not fix), vs complaints that are either orthogonal to your goals (which you can fix without compromising your goals), or that are in line with your goals (which you should prioritize, since that's what you said to yourself was most important in the first place).
My involvement in web standards started with the CSS working group. One of the things that we struggled with as a working group was that we would specify how the technology should work, but the browser vendors' implementations weren't exactly what we intended, and web authors would then write web pages that worked with those browsers, even though that meant the web pages themselves were also not doing things like the specifications said they should. The folks I worked with at the W3C (especially the academics and people working for organizations that did not themselves implement browsers) would frequently bemoan this state of affairs, expressing surprise at how they, the people in charge of the standards, were not being respected by the people implementing the standards. One of the key insights I had very early on in my work, before working on HTML5, which really influenced the WHATWG and its work, is the realization that the power dynamics at work were not at all the power dynamics that the folks at the W3C described. The reality of the situation was that the power lay entirely in the hands of the users. The users chose browsers. A browser vendor that ignored what the users wanted would lose market share. Market share is everything in this space. Browser vendors want users because they can convert users into dollars (in various ways, but they typically boil down to someone showing them ads and paying the browser vendors for the privilege). In turn, the browser vendors had more power than the specifications. What they implement is, by definition, what the technology is. The specification can say in absolute clarity that the keyword "marigold" should look yellow, but if a browser vendor makes it look red, then no web author is going to use it to mean yellow, and many will use it to mean red. There is a feedback loop here: if one browser implements "marigold" to mean red, and some important web site (or many unimportant web sites) rely on it, and say something like "best viewed in ThisOrThat browser!" because that's the one they use and in that browser it looks red and red is what looks best, then the other browser vendors are incentivised to make sure that the web page looks good in their browser too. Regardless of what the specification says, therefore, they are going to make "marigold" look red and not yellow. When I realized this, I also realized a corollary: if you have two competing specifications that both claim to define the same technology, but one matches what the browsers already do while the other one does not, the browser vendors are going to find it more useful to follow the one that matches what they do. This is because they can trust that implementing that specification will get them more market share. It means they won't have to stop and think at every step, "will following this specification cause me to lose users?". It is easier for them to use a specification that takes into account their needs in this way. We actually tried to explain this to the W3C membership. There was a big meeting in 2004 at Adobe in San Jose, the "W3C Workshop on Web Applications and Compound Documents". We tried to convey the above (I didn't quite understand it in the stark "power dynamic" terms yet, or at least, I didn't really express it in those terms, but if you read our position paper you can see this insight starting to crystalize). At this meeting, we made a pitch for the W3C to continue to maintain HTML and to care about what the browser vendors wanted. Representatives from Microsoft and Sun (in many ways arch enemies at the time) supported us. I seem to recall Apple being more quiet about it at the meeting but also essentially supporting the principles. The W3C membership resoundly rejected this whole concept. One of the W3C staff even explicitly said something along the lines of "if you want to do this you should do it elsewhere". That's what led to the WHATWG being founded a few weeks later. The WHATWG was founded on this core principle — the specifications need to actually specify reality. When the browsers disagree with the spec, the spec is by definition incorrect and needs to change, regardless of how much technically superior the design in the spec is. Naturally, when you provide browser vendors with something that valuable, they will follow. You end up with a weird inverted power dynamic. The spec writer (when they follow this principle) has all the power, but only within the space that the browser vendors are themselves willing to play; and the browser vendors have all the power, but only within the space that the users are willing to put up with. It's very easy to appear to be in control when you tell people to do the thing they were going to do anyway (or at least, one of the things they were willing to do if they were to think about it). There is a (probably apocryphal) quote supposedly by Alexandre Auguste Ledru-Rollin that is often cited in mockery of bad leadership, but that perfectly matches the power dynamic here: "There go my people; I must find out where they are going so I can lead them". (Thanks to Leonard Damhorst for prompting me to write this post.)
I often get asked how many people contribute to Flutter. It's a hard question to answer because "contribute" is a very vague concept. There's tens of thousands of packages on pub.dev, all of which are written by contributors to the community. There's over 100,000 of issues filed in our issue database, filed by more than 35,000 people over the years (the exact number is hard to pin down because people sometimes delete their GitHub accounts; about 700 issues have been filed by people who have since deleted their account). Many more people still have used the "thumbs-up" reaction to indicate that an issue matters to them, with almost 165,000 thumbs up from about 45,000 people. All of these people are valuable contributors to Flutter. Usually, when pressed, people try to clarify by asking about "the core team". Again though it's hard to say exactly what that means, but let's assume they mean "people with commit access". That is, people we trust enough to have added to the GitHub repo as collaborators. This includes people who work on Flutter for companies like Google, Canonical, or Nevercode, and it includes people like me who are self-employed and/or contribute to Flutter on a volunteer basis. Currently that's about 280 people. So is that the answer? Well, no, not really. Some people have commit access but aren't active (maybe they got access because of their employer, but were then reassigned to work on another project, and the bureaucracy hasn't caught up with them yet — we only audit the membership occasionally because it's rather tedious to do). Some people have been very active recently but don't have commit access (e.g. because they were just laid off and a bot automatically removed their access; they might even resume working on Flutter in the future, as a volunteer or funded by another company). So what's the answer? I recently drilled down through our data to see if I could answer this. I will caveat the following numbers by saying that this changes all the time. We added a new team member just today (hi Nate!) who is not counted as a team member in the following numbers because we collected the data a few weeks ago (it takes literally days to scrape all the data from GitHub, and then hours to explore the resulting very large and very slow spreadsheet). Also, some of my definitions are a bit arbitrary, and slightly tweaking the limits would probably change the numbers noticeably. First, I collected a list of everyone who has ever created an issue, commented on an issue, put an emoji reaction on the first comment of an issue, or submitted a PR, excluding bots and people who deleted their GitHub account. (Actually Piinks did the actual data collection. Thanks!) I limited this to a subset of the GitHub repos of the flutter org that is relatively inclusive but does not count everything (we have a lot of historical repositories and so forth). This finds about 94,357 people. (So there you go. The Flutter team is about a hundred thousand people!) To avoid padding the numbers with people who left the project long ago, and to avoid counting "drive-by" contributors who came, did a bunch of work, and then left, I then limited the data set to people who contributed over a period of more than 180 days, and who last contributed sometime in 2024. Because of the definition of "contributed" described above, that means that someone who added a thumbs-up to an issue in December 2020 and then filed an issue in January 2024, and did nothing else, is included, but someone who submitted two PRs in March 2024 is not. Like I said, this is a bit arbitrary. Anyway, that leaves 3,839 people, of which 182 currently have commit access, 27 once had commit access but don't currently (these are mainly people who either got laid off recently and had their commit access revoked by an automated process, or people who were once team members, left, lost access from inactivity long ago, and then later came to comment on issues or file new issues — it's surprisingly common for people who once worked on Flutter full time to stick around even when their employment changes), and about 3,627 people who have never had commit access. Of those who have never had commit access, 2,407 have filed at least one issue or submitted at least one PR (accounting for a total of 12,383 issues and 2,613 PRs). Of those, 341 have filed 5 to 9 issues (2,242 issues total), and 296 have filed 10 or more issues in their lifetime (7,021 total issues). Similarly, of the "never had commit access" cohort, 73 people have sent 5 to 9 pull requests in their lifetime (458 total PRs) and 47 have sent 10 or more (1,321 PRs total). (For context, 4,663 people have ever submitted a pull request, and 429 have ever submitted more than 10 PRs.) Of the people who currently have commit access, 98 people have submitted more than one PR every 3 weeks on average since they first got involved (accounting for 49,173 PRs), 75 people have closed at least one issue every 3 weeks (accounting for 48,490 total issue closures), of which 10 are not in the first group (mostly that's our triage team), and 150 people have commented at least once every 3 weeks. A follow-up question a lot of people ask is "do they all work for Google?". This is a surprisingly hard question to answer. There are a lot of weird edge cases. For example, one person worked on Flutter for a company that Google hired to work on Flutter, but then quit that company, asked for their commit privileges to be removed, but continued to be active in the community. Several people who have quit Google (such as myself), or been laid off by Google, have continued to be active in one sense or another (I think I submit more code to Flutter now than I did in my last year at Google). It's also hard to answer because a lot more people at Google contribute to Flutter than just those on Google's Flutter team, and a lot of people on Google's Flutter team contribute in ways that don't show up on GitHub (e.g. product management, marketing, developer relations, internal tooling). Of the 98 people who have commit access, have been active for more than 180 days, have contributed at least once this year, and have submitted more than one PR every 3 weeks on average for the entire time they've been contributing, I estimate (based on what I know of people's employment and so forth) that about 85% are Googlers or somehow get their funding from Google, and about 15% are currently independent of Google. (This is by no means the entirety of the Google team contributing to Flutter; as I mentioned earlier, many folks at Google working on Flutter don't appear in these statistics.) I'm not sure what conclusion to draw from this; it's both more people than I expected to see funded by Google, which is great, and fewer people that aren't funded by Google, which is less great. On the other hand, it's still a significant number of non-Google-funded people. Is it enough? I think that really depends on what your goals are. I think if your goal is for Flutter to be an order of magnitude better than other UI frameworks, then frankly no, it's not enough. There is a ton of work to be done to get there. We know what it would take, but we don't have the people to do it today. On the other hand if your goal is to be a great framework, on par with others, then it's probably adequate. It would certainly be difficult to continue to be great with fewer people today. Of course, that may change as we complete big efforts, or as we take on new ones, or as the landscape changes, it's all hard to predict. That said, I would love to see more direct contributions from non-Google sources, if for no other reason but to end this silly "will Google cancel Flutter" line of questioning that has followed the project since its inception. It's a dumb question. Flutter's an open source UI framework. It will never die. It will become old and something else will shine brighter one day, just as happens with literally every other UI framework ever. That's just how our industry works. There's no reason to believe that'll happen any time soon though, and certainly no reason for it to happen earlier for Flutter than any other modern UI framework.
Despite my departure from Google, I am not leaving Flutter — the great thing about open source and open standards is that the product and the employer are orthogonal. I've had three employers in my career, and in all three cases when I left my employer I continued my job. With Netscape I was a member of the team before my internship, during my internship, and after my internship. With Opera Software, I joined while working on standards, kept working on standards, and left while working on the same standard that I then continued to work on at Google. So this is not a new thing for me. Flutter is amazingly successful. It's already the leading mobile app development framework, and I think we're close to having the table stakes required to make it the obvious default choice for desktop development as well (it's already there for some use cases). It's increasingly used in embedded scenarios. And Flutter is extremely well positioned to be the first truly usable Wasm framework as the web transitions to the more powerful, lower-level Wasm-based model over the next few years. In the coming month I will prepare our roadmap for 2024 (in consultation with the rest of the team). For me personally, however, my focus will probably be on fixing fun bugs, and on making progress on blankcanvas, my library for making it easy to build custom widget sets. I also expect I will be continuing to work on package:rfw, the UI-push library, as there has been increasing interest from teams using Flutter and wanting ways to present custom interfaces determined by the server at runtime without requiring the user to download an updated app.
I joined Google in October 2005, and handed in my resignation 18 years later. Last week was my last week at Google. I feel very lucky to have experienced the early post-IPO Google; unlike most companies, and contrary to the popular narrative, Googlers, from the junior engineer all the way to the C-suite, were genuinely good people who cared very much about doing the right thing. The oft-mocked "don't be evil" truly was the guiding principle of the company at the time (largely a reaction to contemporaries like Microsoft whose operating procedures put profits far above the best interests of customers and humanity as a whole). Many times I saw Google criticised for actions that were sincerely intended to be good for society. Google Books, for example. Much of the criticism Google received around Chrome and Search, especially around supposed conflicts of interest with Ads, was way off base (it's surprising how often coincidences and mistakes can appear malicious). I often saw privacy advocates argue against Google proposals in ways that were net harmful to users. Some of these fights have had lasting effects on the world at large; one of the most annoying is the prevalence of pointless cookie warnings we have to wade through today. I found it quite frustrating how teams would be legitimately actively pursuing ideas that would be good for the world, without prioritising short-term Google interests, only to be met with cynicism in the court of public opinion. Charlie's patio at Google, 2011. Image has been manipulated to remove individuals. Early Google was also an excellent place to work. Executives gave frank answers on a weekly basis, or were candid about their inability to do so (e.g. for legal reasons or because some topic was too sensitive to discuss broadly). Eric Schmidt regularly walked the whole company through the discussions of the board. The successes and failures of various products were presented more or less objectively, with successes celebrated and failures examined critically with an eye to learning lessons rather than assigning blame. The company had a vision, and deviations from that vision were explained. Having experienced Dilbert-level management during my internship at Netscape five years earlier, the uniform competence of people at Google was very refreshing. For my first nine years at Google I worked on HTML and related standards. My mandate was to do the best thing for the web, as whatever was good for the web would be good for Google (I was explicitly told to ignore Google's interests). This was a continuation of the work I started while at Opera Software. Google was an excellent host for this effort. My team was nominally the open source team at Google, but I was entirely autonomous (for which I owe thanks to Chris DiBona). Most of my work was done on a laptop from random buildings on Google's campus; entire years went by where I didn't use my assigned desk. In time, exceptions to Google's cultural strengths developed. For example, as much as I enjoyed Vic Gundotra's enthusiasm (and his initial vision for Google+, which again was quite well defined and, if not necessarily uniformly appreciated, at least unambiguous), I felt less confident in his ability to give clear answers when things were not going as well as hoped. He also started introducing silos to Google (e.g. locking down certain buildings to just the Google+ team), a distinct departure from the complete internal transparency of early Google. Another example is the Android team (originally an acquisition), who never really fully acclimated to Google's culture. Android's work/life balance was unhealthy, the team was not as transparent as older parts of Google, and the team focused on chasing the competition more than solving real problems for users. My last nine years were spent on Flutter. Some of my fondest memories of my time at Google are of the early days of this effort. Flutter was one of the last projects to come out of the old Google, part of a stable of ambitious experiments started by Larry Page shortly before the creation of Alphabet. We essentially operated like a startup, discovering what we were building more than designing it. The Flutter team was very much built out of the culture of young Google; for example we prioritised internal transparency, work/life balance, and data-driven decision making (greatly helped by Tao Dong and his UXR team). We were radically open from the beginning, which made it easy for us to build a healthy open source project around the effort as well. Flutter was also very lucky to have excellent leadership throughout the years, such as Adam Barth as founding tech lead, Tim Sneath as PM, and Todd Volkert as engineering manager. We also didn't follow engineering best practices for the first few years. For example we wrote no tests and had precious little documentation. This whiteboard is what passed for a design doc for the core Widget, RenderObject, and dart:ui layers. This allowed us to move fast at first, but we paid for it later. Flutter grew in a bubble, largely insulated from the changes Google was experiencing at the same time. Google's culture eroded. Decisions went from being made for the benefit of users, to the benefit of Google, to the benefit of whoever was making the decision. Transparency evaporated. Where previously I would eagerly attend every company-wide meeting to learn what was happening, I found myself now able to predict the answers executives would give word for word. Today, I don't know anyone at Google who could explain what Google's vision is. Morale is at an all-time low. If you talk to therapists in the bay area, they will tell you all their Google clients are unhappy with Google. Then Google had layoffs. The layoffs were an unforced error driven by a short-sighted drive to ensure the stock price would keep growing quarter-to-quarter, instead of following Google's erstwhile strategy of prioritising long-term success even if that led to short-term losses (the very essence of "don't be evil"). The effects of layoffs are insidious. Whereas before people might focus on the user, or at least their company, trusting that doing the right thing will eventually be rewarded even if it's not strictly part of their assigned duties, after a layoff people can no longer trust that their company has their back, and they dramatically dial back any risk-taking. Responsibilities are guarded jealously. Knowledge is hoarded, because making oneself irreplaceable is the only lever one has to protect oneself from future layoffs. I see all of this at Google now. The lack of trust in management is reflected by management no longer showing trust in the employees either, in the form of inane corporate policies. In 2004, Google's founders famously told Wall Street "Google is not a conventional company. We do not intend to become one." but that Google is no more. Much of these problems with Google today stem from a lack of visionary leadership from Sundar Pichai, and his clear lack of interest in maintaining the cultural norms of early Google. A symptom of this is the spreading contingent of inept middle management. Take Jeanine Banks, for example, who manages the department that somewhat arbitrarily contains (among other things) Flutter, Dart, Go, and Firebase. Her department nominally has a strategy, but I couldn't leak it if I wanted to; I literally could never figure out what any part of it meant, even after years of hearing her describe it. Her understanding of what her teams are doing is minimal at best; she frequently makes requests that are completely incoherent and inapplicable. She treats engineers as commodities in a way that is dehumanising, reassigning people against their will in ways that have no relationship to their skill set. She is completely unable to receive constructive feedback (as in, she literally doesn't even acknowledge it). I hear other teams (who have leaders more politically savvy than I) have learned how to "handle" her to keep her off their backs, feeding her just the right information at the right time. Having seen Google at its best, I find this new reality depressing. There are still great people at Google. I've had the privilege to work with amazing people on the Flutter team such as JaYoung Lee, Kate Lovett, Kevin Chisholm, Zoey Fan, Dan Field, and dozens more (sorry folks, I know I should just name all of you but there's too many!). In recent years I started offering career advice to anyone at Google and through that met many great folks from around the company. It's definitely not too late to heal Google. It would require some shake-up at the top of the company, moving the centre of power from the CFO's office back to someone with a clear long-term vision for how to use Google's extensive resources to deliver value to users. I still believe there's lots of mileage to be had from Google's mission statement (to organize the world’s information and make it universally accessible and useful). Someone who wanted to lead Google into the next twenty years, maximising the good to humanity and disregarding the short-term fluctuations in stock price, could channel the skills and passion of Google into truly great achievements. I do think the clock is ticking, though. The deterioration of Google's culture will eventually become irreversible, because the kinds of people whom you need to act as moral compass are the same kinds of people who don't join an organisation without a moral compass.
More in programming
One of the first types we learn about is the boolean. It's pretty natural to use, because boolean logic underpins much of modern computing. And yet, it's one of the types we should probably be using a lot less of. In almost every single instance when you use a boolean, it should be something else. The trick is figuring out what "something else" is. Doing this is worth the effort. It tells you a lot about your system, and it will improve your design (even if you end up using a boolean). There are a few possible types that come up often, hiding as booleans. Let's take a look at each of these, as well as the case where using a boolean does make sense. This isn't exhaustive—[1]there are surely other types that can make sense, too. Datetimes A lot of boolean data is representing a temporal event having happened. For example, websites often have you confirm your email. This may be stored as a boolean column, is_confirmed, in the database. It makes a lot of sense. But, you're throwing away data: when the confirmation happened. You can instead store when the user confirmed their email in a nullable column. You can still get the same information by checking whether the column is null. But you also get richer data for other purposes. Maybe you find out down the road that there was a bug in your confirmation process. You can use these timestamps to check which users would be affected by that, based on when their confirmation was stored. This is the one I've seen discussed the most of all these. We run into it with almost every database we design, after all. You can detect it by asking if an action has to occur for the boolean to change values, and if values can only change one time. If you have both of these, then it really looks like it is a datetime being transformed into a boolean. Store the datetime! Enums Much of the remaining boolean data indicates either what type something is, or its status. Is a user an admin or not? Check the is_admin column! Did that job fail? Check the failed column! Is the user allowed to take this action? Return a boolean for that, yes or no! These usually make more sense as an enum. Consider the admin case: this is really a user role, and you should have an enum for it. If it's a boolean, you're going to eventually need more columns, and you'll keep adding on other statuses. Oh, we had users and admins, but now we also need guest users and we need super-admins. With an enum, you can add those easily. enum UserRole { User, Admin, Guest, SuperAdmin, } And then you can usually use your tooling to make sure that all the new cases are covered in your code. With a boolean, you have to add more booleans, and then you have to make sure you find all the places where the old booleans were used and make sure they handle these new cases, too. Enums help you avoid these bugs. Job status is one that's pretty clearly an enum as well. If you use booleans, you'll have is_failed, is_started, is_queued, and on and on. Or you could just have one single field, status, which is an enum with the various statuses. (Note, though, that you probably do want timestamp fields for each of these events—but you're still best having the status stored explicitly as well.) This begins to resemble a state machine once you store the status, and it means that you can make much cleaner code and analyze things along state transition lines. And it's not just for storing in a database, either. If you're checking a user's permissions, you often return a boolean for that. fn check_permissions(user: User) -> bool { false // no one is allowed to do anything i guess } In this case, true means the user can do it and false means they can't. Usually. I think. But you can really start to have doubts here, and with any boolean, because the application logic meaning of the value cannot be inferred from the type. Instead, this can be represented as an enum, even when there are just two choices. enum PermissionCheck { Allowed, NotPermitted(reason: String), } As a bonus, though, if you use an enum? You can end up with richer information, like returning a reason for a permission check failing. And you are safe for future expansions of the enum, just like with roles. You can detect when something should be an enum a proliferation of booleans which are mutually exclusive or depend on one another. You'll see multiple columns which are all changed at the same time. Or you'll see a boolean which is returned and used for a long time. It's important to use enums here to keep your program maintainable and understandable. Conditionals But when should we use a boolean? I've mainly run into one case where it makes sense: when you're (temporarily) storing the result of a conditional expression for evaluation. This is in some ways an optimization, either for the computer (reuse a variable[2]) or for the programmer (make it more comprehensible by giving a name to a big conditional) by storing an intermediate value. Here's a contrived example where using a boolean as an intermediate value. fn calculate_user_data(user: User, records: RecordStore) { // this would be some nice long conditional, // but I don't have one. So variables it is! let user_can_do_this: bool = (a && b) && (c || !d); if user_can_do_this && records.ready() { // do the thing } else if user_can_do_this && records.in_progress() { // do another thing } else { // and something else! } } But even here in this contrived example, some enums would make more sense. I'd keep the boolean, probably, simply to give a name to what we're calculating. But the rest of it should be a match on an enum! * * * Sure, not every boolean should go away. There's probably no single rule in software design that is always true. But, we should be paying a lot more attention to booleans. They're sneaky. They feel like they make sense for our data, but they make sense for our logic. The data is usually something different underneath. By storing a boolean as our data, we're coupling that data tightly to our application logic. Instead, we should remain critical and ask what data the boolean depends on, and should we maybe store that instead? It comes easier with practice. Really, all good design does. A little thinking up front saves you a lot of time in the long run. I know that using an em-dash is treated as a sign of using LLMs. LLMs are never used for my writing. I just really like em-dashes and have a dedicated key for them on one of my keyboard layers. ↩ This one is probably best left to the compiler. ↩
As I slowly but surely work towards the next release of my setcmd project for the Amiga (see the 68k branch for the gory details and my total noob-like C flailing around), I’ve made heavy use of documentation in the AmigaGuide format. Despite it’s age, it’s a great Amiga-native format and there’s a wealth of great information out there for things like the C API, as well as language guides and tutorials for tools like the Installer utility - and the AmigaGuide markup syntax itself. The only snag is, I had to have access to an Amiga (real or emulated), or install one of the various viewer programs on my laptops. Because like many, I spend a lot of time in a web browser and occasionally want to check something on my mobile phone, this is less than convenient. Fortunately, there’s a great AmigaGuideJS online viewer which renders AmigaGuide format documents using Javascript. I’ve started building up a collection of useful developer guides and other files in my own reference library so that I can access this documentation whenever I’m not at my Amiga or am coding in my “modern” dev environment. It’s really just for my own personal use, but I’ll be adding to it whenever I come across a useful piece of documentation so I hope it’s of some use to others as well! And on a related note, I now have a “unified” code-base so that SetCmd now builds and runs on 68k-based OS 3.x systems as well as OS 4.x PPC systems like my X5000. I need to: Tidy up my code and fix all the “TODO” stuff Update the Installer to run on OS 3.x systems Update the documentation Build a new package and upload to Aminet/OS4Depot Hopefully I’ll get that done in the next month or so. With the pressures of work and family life (and my other hobbies), progress has been a lot slower these last few years but I’m still really enjoying working on Amiga code and it’s great to have a fun personal project that’s there for me whenever I want to hack away at something for the sheer hell of it. I’ve learned a lot along the way and the AmigaOS is still an absolute joy to develop for. I even brought my X5000 to the most recent Kickstart Amiga User Group BBQ/meetup and had a fun day working on the code with fellow Amigans and enjoying some classic gaming & demos - there was also a MorphOS machine there, which I think will be my next target as the codebase is slowly becoming more portable. Just got to find some room in the “retro cave” now… This stuff is addictive :)
A little while back I heard about the White House launching their version of a Drudge Report style website called White House Wire. According to Axios, a White House official said the site’s purpose was to serve as “a place for supporters of the president’s agenda to get the real news all in one place”. So a link blog, if you will. As a self-professed connoisseur of websites and link blogs, this got me thinking: “I wonder what kind of links they’re considering as ‘real news’ and what they’re linking to?” So I decided to do quick analysis using Quadratic, a programmable spreadsheet where you can write code and return values to a 2d interface of rows and columns. I wrote some JavaScript to: Fetch the HTML page at whitehouse.gov/wire Parse it with cheerio Select all the external links on the page Return a list of links and their headline text In a few minutes I had a quick analysis of what kind of links were on the page: This immediately sparked my curiosity to know more about the meta information around the links, like: If you grouped all the links together, which sites get linked to the most? What kind of interesting data could you pull from the headlines they’re writing, like the most frequently used words? What if you did this analysis, but with snapshots of the website over time (rather than just the current moment)? So I got to building. Quadratic today doesn’t yet have the ability for your spreadsheet to run in the background on a schedule and append data. So I had to look elsewhere for a little extra functionality. My mind went to val.town which lets you write little scripts that can 1) run on a schedule (cron), 2) store information (blobs), and 3) retrieve stored information via their API. After a quick read of their docs, I figured out how to write a little script that’ll run once a day, scrape the site, and save the resulting HTML page in their key/value storage. From there, I was back to Quadratic writing code to talk to val.town’s API and retrieve my HTML, parse it, and turn it into good, structured data. There were some things I had to do, like: Fine-tune how I select all the editorial links on the page from the source HTML (I didn’t want, for example, to include external links to the White House’s social pages which appear on every page). This required a little finessing, but I eventually got a collection of links that corresponded to what I was seeing on the page. Parse the links and pull out the top-level domains so I could group links by domain occurrence. Create charts and graphs to visualize the structured data I had created. Selfish plug: Quadratic made this all super easy, as I could program in JavaScript and use third-party tools like tldts to do the analysis, all while visualizing my output on a 2d grid in real-time which made for a super fast feedback loop! Once I got all that done, I just had to sit back and wait for the HTML snapshots to begin accumulating! It’s been about a month and a half since I started this and I have about fifty days worth of data. The results? Here’s the top 10 domains that the White House Wire links to (by occurrence), from May 8 to June 24, 2025: youtube.com (133) foxnews.com (72) thepostmillennial.com (67) foxbusiness.com (66) breitbart.com (64) x.com (63) reuters.com (51) truthsocial.com (48) nypost.com (47) dailywire.com (36) From the links, here’s a word cloud of the most commonly recurring words in the link headlines: “trump” (343) “president” (145) “us” (134) “big” (131) “bill” (127) “beautiful” (113) “trumps” (92) “one” (72) “million” (57) “house” (56) The data and these graphs are all in my spreadsheet, so I can open it up whenever I want to see the latest data and re-run my script to pull the latest from val.town. In response to the new data that comes in, the spreadsheet automatically parses it, turn it into links, and updates the graphs. Cool! If you want to check out the spreadsheet — sorry! My API key for val.town is in it (“secrets management” is on the roadmap). But I created a duplicate where I inlined the data from the API (rather than the code which dynamically pulls it) which you can check out here at your convenience. Email · Mastodon · Bluesky
SumatraPDF is a fast, small, open-source PDF reader for Windows, written in C++. This article describes how I implemented StrVec class for efficiently storing multiple strings. Much ado about the strings Strings are among the most used types in most programs. Arrays of strings are also used often. I count ~80 uses of StrVec in SumatraPDF code. This article describes how I implemented an optimized array of strings in SumatraPDF C++ code . No STL for you Why not use std::vector<std::string>? In SumatraPDF I don’t use STL. I don’t use std::string, I don’t use std::vector. For me it’s a symbol of my individuality, and my belief in personal freedom. As described here, minimum size of std::string on 64-bit machines is 32 bytes for msvc / gcc and 24 bytes for short strings (15 chars for msvc / gcc, 22 chars for clang). For longer strings we have more overhead: 32⁄24 bytes for the header memory allocator overhead allocator metadata padding due to rounding allocations to at least 16 bytes There’s also std::vector overhead: for fast appends (push()) std::vectorimplementations over-allocated space Longer strings are allocated at random addresses so they can be spread out in memory. That is bad for cache locality and that often cause more slowness than executing lots of instructions. Design and implementation of StrVec StrVec (vector of strings) solves all of the above: per-string overhead of only 8 bytes strings are laid out next to each other in memory StrVec High level design of StrVec: backing memory is allocated in singly-linked pages similar to std::vector, we start with small page and increase the size of the page. This strikes a balance between speed of accessing a string at random index and wasted space unlike std::vector we don’t reallocate memory (most of the time). That saves memory copy when re-allocating backing space Here’s all there is to StrVec: struct StrVec { StrVecPage* first = nullptr; int nextPageSize = 256; int size = 0; } size is a cached number of strings. It could be calculated by summing the size in all StrVecPages. nextPageSize is the size of the next StrVecPage. Most array implementation increase the size of next allocation by 1.4x - 2x. I went with the following progression: 256 bytes, 1k, 4k, 16k, 32k and I cap it at 64k. I don’t have data behind those numbers, they feel right. Bigger page wastes more space. Smaller page makes random access slower because to find N-th string we need to traverse linked list of StrVecPage. nextPageSize is exposed to allow the caller to optimize use. E.g. if it expects lots of strings, it could set nextPageSize to a large number. StrVecPage Most of the implementation is in StrVecPage. The big idea here is: we allocate a block of memory strings are allocated from the end of memory block at the beginning of the memory block we build and index of strings. For each string we have: u32 size u32 offset of the string within memory block, counting from the beginning of the block The layout of memory block is: StrVecPage struct { size u32; offset u32 } [] … not yet used space strings This is StrVecPage: struct StrVecPage { struct StrVecPage* next; int pageSize; int nStrings; char* currEnd; } next is for linked list of pages. Since pages can have various sizes we need to record pageSize. nStrings is number of strings in the page and currEnd points to the end of free space within page. Implementing operations Appending a string Appending a string at the end is most common operation. To append a string: we calculate how much memory inside a page it’ll need: str::Len(string) + 1 + sizeof(u32) + sizeof(u32). +1 is for 0-termination for compatibility with C APIs that take char*, and 2xu32 for size and offset. If we have enough space in last page, we add size and offset at the end of index and append a string from the end i.e. `currEnd - (str::Len(string) + 1). If there is not enough space in last page, we allocate new page We can calculate how much space we have left with: int indexEntrySize = sizeof(u32) + sizeof(u32); // size + offset char* indexEnd = (char*)pageStart + sizeof(StrVecPage) + nStrings*indexEntrySize int nBytesFree = (int)(currEnd - indexEnd) Removing a string Removing a string is easy because it doesn’t require moving memory inside StrVecPage. We do nStrings-- and move index values of strings after the removed string. I don’t bother freeing the string memory within a page. It’s possible but complicated enough I decided to skip it. You can compact StrVec to remove all overhead. If you do not care about preserving order of strings after removal, I haveRemoveAtFast() which uses a trick: instead of copying memory of all index values after removed string, I copy a single index from the end into a slot of the string being removed. Replacing a string or inserting in the middle Replacing a string or inserting a string in the middle is more complicated because there might not be enough space in the page for the string. When there is enough space, it’s as simple as append. When there is not enough space, I re-use the compacting capability: I compact all existing pages into a single page with extra space for the string and some extra space as an optimization for multiple inserts. Iteration A random access requires traversing a linked list. I think it’s still fast because typically there aren’t many pages and we only need to look at a single nStrings value. After compaction to a single page, random access is as fast as it could ever be. C++ iterator is optimized for sequential access: struct iterator { const StrVec* v; int idx; // perf: cache page, idxInPage from prev iteration int idxInPage; StrVecPage* page; } We cache the current state of iteration as page and idxInPage. To advance to next string we advance idxInPage. If it exceeds nStrings, we advance to page->next. Optimized search Finding a string is as optimized as it could be without a hash table. Typically to compare char* strings you need to call str::Eq(s, s2) for every string you compare it to. That is a function call and it has to touch s2 memory. That is bad for performance because it blows the cache. In StrVec I calculate length of the string to find once and then traverse the size / offset index. Only when size is different I have to compare the strings. Most of the time we just look at offset / size in L1 cache, which is very fast. Compacting If you know that you’ll not be adding more strings to StrVec you can compact all pages into a single page with no overhead of empty space. It also speeds up random access because we don’t have multiple pages to traverse to find the item and a given index. Representing a nullptr char* Even though I have a string class, I mostly use char* in SumatraPDF code. In that world empty string and nullptr are 2 different things. To allow storing nullptr strings in StrVec (and not turning them into empty strings on the way out) I use a trick: a special u32 value kNullOffset represents nullptr. StrVec is a string pool allocator In C++ you have to track the lifetime of each object: you allocate with malloc() or new when you no longer need to object, you call free() or delete However, the lifetime of allocations is often tied together. For example in SumatraPDF an opened document is represented by a class. Many allocations done to construct that object last exactly as long as the object. The idea of a pool allocator is that instead of tracking the lifetime of each allocation, you have a single allocator. You allocate objects with the same lifetime from that allocator and you free them with a single call. StrVec is a string pool allocator: all strings stored in StrVec have the same lifetime. Testing In general I don’t advocate writing a lot of tests. However, low-level, tricky functionality like StrVec deserves decent test coverage to ensure basic functionality works and to exercise code for corner cases. I have 360 lines of tests for ~700 lines of of implementation. Potential tweaks and optimization When designing and implementing data structures, tradeoffs are aplenty. Interleaving index and strings I’m not sure if it would be faster but instead of storing size and offset at the beginning of the page and strings at the end, we could store size / string sequentially from the beginning. It would remove the need for u32 of offset but would make random access slower. Varint encoding of size and offset Most strings are short, under 127 chars. Most offsets are under 16k. If we stored size and offset as variable length integers, we would probably bring down average per-string overhead from 8 bytes to ~4 bytes. Implicit size When strings are stored sequentially size is implicit as difference between offset of the string and offset of next string. Not storing size would make insert and set operations more complicated and costly: we would have to compact and arrange strings in order every time. Storing index separately We could store index of size / offset in a separate vector and use pages to only allocate string data. This would simplify insert and set operations. With current design if we run out of space inside a page, we have to re-arrange memory. When offset is stored outside of the page, it can refer to any page so insert and set could be as simple as append. The evolution of StrVec The design described here is a second implementation of StrVec. The one before was simply a combination of str::Str (my std::string) for allocating all strings and Vec<u32> (my std::vector) for storing offset index. It had some flaws: appending a string could re-allocate memory within str::Str. The caller couldn’t store returned char* pointer because it could be invalidated. As a result the API was akward and potentially confusing: I was returning offset of the string so the string was str::Str.Data() + offset. The new StrVec doesn’t re-allocate on Append, only (potentially) on InsertAt and SetAt. The most common case is append-only which allows the caller to store the returned char* pointers. Before implementing StrVec I used Vec<char*>. Vec is my version of std::vector and Vec<char*> would just store pointer to individually allocated strings. Cost vs. benefit I’m a pragmatist: I want to achieve the most with the least amount of code, the least amount of time and effort. While it might seem that I’m re-implementing things willy-nilly, I’m actually very mindful of the cost of writing code. Writing software is a balance between effort and resulting quality. One of the biggest reasons SumatraPDF so popular is that it’s fast and small. That’s an important aspect of software quality. When you double click on a PDF file in an explorer, SumatraPDF starts instantly. You can’t say that about many similar programs and about other software in general. Keeping SumatraPDF small and fast is an ongoing focus and it does take effort. StrVec.cpp is only 705 lines of code. It took me several days to complete. Maybe 2 days to write the code and then some time here and there to fix the bugs. That being said, I didn’t start with this StrVec. For many years I used obvious Vec<char*>. Then I implemented somewhat optimized StrVec. And a few years after that I implemented this ultra-optimized version. References SumatraPDF is a small, fast, multi-format (PDF/eBook/Comic Book and more), open-source reader for Windows. The implementation described here: StrVec.cpp, StrVec.h, StrVec_ut.cpp By the time you read this, the implementation could have been improved.
Consent morality is the idea that there are no higher values or virtues than allowing consenting adults to do whatever they please. As long as they're not hurting anyone, it's all good, and whoever might have a problem with that is by definition a bigot. This was the overriding morality I picked up as a child of the 90s. From TV, movies, music, and popular culture. Fly your freak! Whatever feels right is right! It doesn't seem like much has changed since then. What a moral dead end. I first heard the term consent morality as part of Louise Perry's critique of the sexual revolution. That in the context of hook-up culture, situationships, and falling birthrates, we have to wrestle with the fact that the sexual revolution — and it's insistence that, say, a sky-high body count mustn't be taboo — has led society to screwy dating market in the internet age that few people are actually happy with. But the application of consent morality that I actually find even more troubling is towards parenthood. As is widely acknowledged now, we're in a bit of a birthrate crisis all over the world. And I think consent morality can help explain part of it. I was reminded of this when I posted a cute video of a young girl so over-the-moon excited for her dad getting off work to argue that you'd be crazy to trade that for some nebulous concept of "personal freedom". Predictably, consent morality immediately appeared in the comments: Some people just don't want children and that's TOTALLY OKAY and you're actually bad for suggesting they should! No. It's the role of a well-functioning culture to guide people towards The Good Life. Not force, but guide. Nobody wants to be convinced by the morality police at the pointy end of a bayonet, but giving up on the whole idea of objective higher values and virtues is a nihilistic and cowardly alternative. Humans are deeply mimetic creatures. It's imperative that we celebrate what's good, true, and beautiful, such that these ideals become collective markers for morality. Such that they guide behavior. I don't think we've done a good job at doing that with parenthood in the last thirty-plus years. In fact, I'd argue we've done just about everything to undermine the cultural appeal of the simple yet divine satisfaction of child rearing (and by extension maligned the square family unit with mom, dad, and a few kids). Partly out of a coordinated campaign against the family unit as some sort of trad (possibly fascist!) identity marker in a long-waged culture war, but perhaps just as much out of the banal denigration of how boring and limiting it must be to carry such simple burdens as being a father or a mother in modern society. It's no wonder that if you incessantly focus on how expensive it is, how little sleep you get, how terrifying the responsibility is, and how much stress is involved with parenthood that it doesn't seem all that appealing! This is where Jordan Peterson does his best work. In advocating for the deeper meaning of embracing burden and responsibility. In diagnosing that much of our modern malaise does not come from carrying too much, but from carrying too little. That a myopic focus on personal freedom — the nights out, the "me time", the money saved — is a spiritual mirage: You think you want the paradise of nothing ever being asked of you, but it turns out to be the hell of nobody ever needing you. Whatever the cause, I think part of the cure is for our culture to reembrace the virtue and the value of parenthood without reservation. To stop centering the margins and their pathologies. To start centering the overwhelming middle where most people make for good parents, and will come to see that role as the most meaningful part they've played in their time on this planet. But this requires giving up on consent morality as the only way to find our path to The Good Life. It involves taking a moral stance that some ways of living are better than other ways of living for the broad many. That parenthood is good, that we need more children both for the literal survival of civilization, but also for the collective motivation to guard against the bad, the false, and the ugly. There's more to life than what you feel like doing in the moment. The worst thing in the world is not to have others ask more of you. Giving up on the total freedom of the unmoored life is a small price to pay for finding the deeper meaning in a tethered relationship with continuing a bloodline that's been drawn for hundreds of thousands of years before it came to you. You're never going to be "ready" before you take the leap. If you keep waiting, you'll wait until the window has closed, and all you see is regret. Summon a bit of bravery, don't overthink it, and do your part for the future of the world. It's 2.1 or bust, baby!
