Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from On Test Automation
Last weekend, I wrote a more or less casual post on LinkedIn containing the ‘rules’ (it’s more of a list of terms and conditions, really) I set for myself when it comes to using AI. That post received some interesting comments that made me think and refine my thoughts on when (not) to use AI to support me in my work. Thank you to all of you who commented for doing so, and for showing me that there still is value in being active on LinkedIn in between all the AI-generated ‘content’. I really appreciate it. Now, AI and LLMs like ChatGPT or Claude can be very useful, that is, when used prudently. I think it is very important to be conscious and cautious when it comes to using AI, though, which is why I wrote that post. I wrote it mostly for myself, to structure my thoughts around AI, but also because I think it is important that others are at least conscious of what they’re doing and working with. That doesn’t mean you have to adhere to or even agree with my views and the way I use these tools, by the way. Different strokes for different folks. Because of the ephemeral nature of these LinkedIn posts, and the importance of the topic to me, I want to repeat the ‘rules’ (again, more of a T&C list) I wrote down here. This is the original, unchanged list from the post I wrote on February 15: I only use it to support me in completing tasks I understand. I need to be able to scrutinize the output the AI system produces and see if it is both sound and fit for the purpose I want to use it for. I never use it to explain to me something I don’t know yet or don’t understand enough. I have seen and read about too many hallucinations to trust them to teach me what I don’t understand. Instead, I use books, articles, and other content from authors and sources I do trust if I’m looking to learn something new. I never EVER use it for creative work. I don’t use AI-generated images anywhere, and all of my blogs, LinkedIn posts, comments, course material and other written text are 100% my own, warts and all. My views, my ideas, my voice. Interestingly, most of the comments were written in reaction to the first two bullet points at the time I wrote this blog post. I don’t know exactly why this is the case, it might be because the people who read it agree (which I doubt seeing the tsunami of AI-generated content that’s around these days), or maybe because there’s a bit of stigma around admitting to use AI for content generation. I don’t know. What I do know is that it is an important principle to me. I wrote about the reasons for that in an earlier blog post, so I won’t repeat myself here. Like so many terms and conditions, the list I wrote down in this post will probably evolve over time, but what will not change is me remaining very careful around where I use and where I don’t use AI to help me in my work. Especially now that the speed with which new developments in the AI space are presented to us and the claims around what it can and will do only get bigger, I think it is wise to remain cautious and look at these developments with a critical and very much human view.
When I build and release new features or bug fixes for RestAssured.Net, I rely heavily on the acceptance tests that I wrote over time. Next to serving as living documentation for the library, I run these tests both locally and on every push to GitHub to see if I didn’t accidentally break something, for different versions of .NET. But how reliable are these tests really? Can I trust them to pass and fail when they should? Did I cover all the things that are important? I speak, write and teach about the importance of testing your tests on a regular basis, so it makes sense to start walking the talk and get more insight into the quality of the RestAssured.Net test suite. One approach to learning more about the quality of your tests is through a technique called mutation testing. I speak about and demo testing your tests and using mutation testing to do so on a regular basis (you can watch a recent talk here), but until now, I’ve pretty much exclusively used PITest for Java. As RestAssured.Net is a C# library, I can’t use PITest, but I’d heard many good things about Stryker.NET, so this would be a perfect opportunity to finally use it. Adding Stryker.NET to the RestAssured.Net project The first step was to add Stryker.Net to the RestAssured.Net project. Stryker.NET is a dotnet tool, so installing it is straightforward: run dotnet new tool-manifest to create a new, project-specific tool manifest (this was the first local dotnet tool for this project) and then dotnet tool install dotnet-stryker to add Stryker.NET as a dotnet tool to the project. Running mutation tests for the first time Running mutation tests with Stryker.NET is just as straightforward: dotnet stryker --project RestAssured.Net.csproj from the tests project folder is all it takes. Because both my test suite (about 200 tests) and the project itself are relatively small code bases, and because my test suite runs quickly, running mutation tests for my entire project works for me. It still took around five minutes for the process to complete. If you have a larger code base, and longer-running test suites, you’ll see that mutation testing will take much, much longer. In that case, it’s probably best to start on a subset of your code base and a subset of your test suite. After five minutes and change, the results are in: Stryker.NET created 538 mutants from my application code base. Of these: 390 were killed, that is, at least one test failed because of this mutation, 117 survived, that is, the change did not make any of the tests fail, and 31 resulted in a timeout, which I’ll need to investigate further, but I suspect it has something to do with HTTP timeouts (RestAssured.Net is an HTTP API testing library, and all acceptance tests perform actual HTTP requests) This leads to an overall mutation testing score of 59.97%. Is that good? Is that bad? In all honesty, I don’t know, and I don’t care. Just like with code coverage, I am not a fan of setting fixed targets for this type of metric, as these will typically lead to writing tests for the sake of improving a score rather than for actual improvement of the code. What I am much more interested in is the information that Stryker.NET produced during the mutation testing process. Opening the HTML report I was surprised to see that out of the box, Stryker.NET produces a very good-looking and incredibly helpful HTML report. It provides both a high-level overview of the results: as well as in-depth detail for every mutant that was killed or that survived. It offers a breakdown of the results per namespace and per class, and it is the starting point for further drilling down into results for individual mutants. Let’s have a look and see if the report provides some useful, actionable information for us to improve the RestAssured.Net test suite. Missing coverage Like many other mutation testing tools, Stryker.NET provides code coverage information along with mutation coverage information. That is, if there is code in the application code base that was mutated, but that is not covered by any of the tests, Stryker.NET will inform you about it. Here’s an example: Stryker.NET changed the message of an exception thrown when RestAssured.Net is asked to deserialize a response body that is either null or empty. Apparently, there is no test in the test suite that covers this path in the code. As this particular code path deals with exception handling, it’s probably a good idea to add a test for it: [Test] public void EmptyResponseBodyThrowsTheExpectedException() { var de = Assert.Throws<DeserializationException>(() => { Location responseLocation = (Location)Given() .When() .Get($"{MOCK_SERVER_BASE_URL}/empty-response-body") .DeserializeTo(typeof(Location)); }); Assert.That(de?.Message, Is.EqualTo("Response content is null or empty.")); } I added the corresponding test in this commit. Removed code blocks Another type of mutant that Stryker.NET generates is the removal of a code block. Going by the mutation testing report, it seems like there are a few of these mutants that are not detected by any of the tests. Here’s an example: The return statement for the Put() method body, which is used to perform an HTTP PUT operation, is replaced with an empty method body, but this is not picked up by any of the tests. The same applies to the methods for HTTP PATCH, DELETE, HEAD and OPTIONS. Looking at the tests that cover the different HTTP verbs, this makes sense. While I do call each of these HTTP methods in a test, I don’t assert on the result for the aforementioned HTTP verbs. I am basically relying on the fact that no exception is thrown when I call Put() when I say ‘it works’. Let’s change that by at least asserting on a property of the response that is returned when these HTTP verbs are used: [Test] public void HttpPutCanBeUsed() { Given() .When() .Put($"{MOCK_SERVER_BASE_URL}/http-put") .Then() .StatusCode(200); } These assertions were added to the RestAssured.Net test suite in this commit. Improving testability The next signal I received from this initial mutation testing run is an interesting one. It tells me that even though I have acceptance tests that add cookies to the request and that only pass when the request contains the cookies I set, I’m not properly covering some logic that I added: To understand what is going on here, it is useful to know that a Cookie in C# offers a constructor that creates a Cookie specifying only a name and a value, but that a cookie has to have a domain value set. To enforce that, I added the logic you see in the screenshot. However, Stryker.NET tells me I’m not properly testing this logic, because changing its implementation doesn’t cause any tests to fail. Now, I might be able to test this specific logic with a few added acceptance tests, but it really is only a small piece of logic, and I should be able to test that logic in isolation, right? Well, not with the code written in the way it currently is… So, time to extract that piece of logic into a class of its own, which will improve both the modularity of the code and allow me to test it in isolation. First, let’s extract the logic into a CookieUtils class: internal class CookieUtils { internal Cookie SetDomainFor(Cookie cookie, string hostname) { if (string.IsNullOrEmpty(cookie.Domain)) { cookie.Domain = hostname; } return cookie; } } I deliberately made this class internal as I don’t want it to be directly accessible to RestAssured.Net users. However, as I do need to access it in the tests, I have to add this little snippet to the RestAssured.Net.csproj file: <ItemGroup> <InternalsVisibleTo Include="$(MSBuildProjectName).Tests" /> </ItemGroup> Now, I can add unit tests that should cover both paths in the SetDomainFor() logic: [Test] public void CookieDomainIsSetToDefaultValueWhenNotSpecified() { Cookie cookie = new Cookie("cookie_name", "cookie_value"); CookieUtils cookieUtils = new CookieUtils(); cookie = cookieUtils.SetDomainFor(cookie, "localhost"); Assert.That(cookie.Domain, Is.EqualTo("localhost")); } [Test] public void CookieDomainIsUnchangedWhenSpecifiedAlready() { Cookie cookie = new Cookie("cookie_name", "cookie_value", "/my_path", "strawberry.com"); CookieUtils cookieUtils = new CookieUtils(); cookie = cookieUtils.SetDomainFor(cookie, "localhost"); Assert.That(cookie.Domain, Is.EqualTo("strawberry.com")); } These changes were added to the RestAssured.Net source and test code in this commit. An interesting mutation So far, all the signals that appeared in the mutation testing report generated by Stryker.NET have been valuable, as in: they have pointed me at code that isn’t covered by any tests yet, to tests that could be improved, and they have led to code refactoring to improve testability. Using Stryker.NET (and mutation testing in general) does sometimes lead to some, well, interesting mutations, like this one: I’m checking that a certain string is either null or an empty string, and if either condition is true, RestAssured.Net throws an exception. Perfectly valid. However, Stryker.NET changes the logical OR to a logical AND (a common mutation), which makes it impossible for the condition to evaluate to true. Is that even a useful mutation to make? Well, to some extent, it is. Even if the code doesn’t make sense anymore after it has been mutated, it does tell you that your tests for this logical condition probably need some improvement. In this case, I don’t have to add more tests, as we discussed this exact statement earlier (remember that it had no test coverage at all). It did make me look at this statement once again, though, and I only then realized that I could simplify this code snippet to if (string.IsNullOrEmpty(responseBodyAsString)) { throw new DeserializationException("Response content is null or empty."); } Instead of a custom-built logical OR, I am now using a construct built into C#, which is arguably the safer choice. In general, if your mutation testing tool generates several (or even many) mutants for the same code statement or block, it might be a good idea to have another look at that code and see if it can be simplified. This was just a very small example, but I think this observation holds true in general. This change was added to the RestAssured.Net source and test code in this commit. Running mutation tests again and inspecting the results Now that several (supposed) improvements to the tests and the code have been made, let’s run the mutation tests another time to see if the changes improved our score. In short: 397 mutants were killed now, up from 390 (that’s good) 111 mutants survived, down from 117 (that’s also good) there were 32 timeouts, up from 31 (that needs some further investigation) Overall, the mutation testing score went up from 59,97% to 61,11%. This might not seem like much, but it is definitely a step in the right direction. The most important thing for me right now is that my tests for RestAssured.Net have improved, my code has improved and I learned a lot about mutation testing and Stryker.NET in the process. Am I going to run mutation tests every time I make a change? Probably not. There is quite a lot of information to go through, and that takes time, time that I don’t want to spend for every build. For that reason, I’m also not going to make these mutation tests part of the build and test pipeline for RestAssured.Net, at least not any time soon. This was nonetheless both a very valuable and a very enjoyable exercise, and I’ll definitely keep improving the tests and the code for RestAssured.Net using the suggestions that Stryker.NET presents.
This blog post is another one in the ‘writing things down to structure my thinking on where I want my career to go’ series. I will get back to writing technical and automation blog posts soon, but I need to finish my contract testing course first. One of the things I like to do most in life is traveling and seeing new places. Well, seeing new places, mostly, as the novelty of waiting, flying and staying in hotel rooms has definitely worn off by now. I am in the privileged position (really, that is what it is: I’m privileged, and I fully realize that) that I get to scratch this travel itch professionally on a regular basis these days. Over the last few years, I have been invited to contribute to meetups and conferences abroad, and I also get to run in-house training sessions with companies outside the Netherlands a couple of times per year. Most of this traveling takes place within Europe, but for the last three years, I have been able to travel outside of Europe once every year (South Africa in 2022, Canada in 2023 and the United States in 2024), and needless to say I have enjoyed those opportunities very much. To give you an idea of the amount of traveling I do: for 2025, I now have four work-related trips abroad scheduled, and I am pretty sure at least a few more will be added to that before the year ends (it’s only just February…). That might not be much travel by some people’s standards, but for me, it is. And it seems the number of opportunities I get for traveling increase year over year, to the point where I have to say ‘no’ to several of these opportunities. Say no? Why? I thought you just said you loved to travel? Yes, that’s true. I do love to travel. But I also love spending time at home with my family, and that comes first. Always. Now, my sons are getting older, and being away from home for a few days doesn’t put as much pressure on them and on my wife as it did a few years ago. Still, I always need to find a balance between spending time with them and spending time at work. I am away from home for work not just when I’m abroad. I run evening training sessions with clients here in the Netherlands on a regular basis, too, as well as training sessions in my evenings for clients in different time zones, mainly US-based clients. And all that adds up. I try to only be away from home one night per week, but often, it’s two. When I travel abroad, it’s even more than that. Again, I’m not complaining. Not at all. It is an absolute privilege to get to travel for work and get paid to do that, but I cannot do that indefinitely, and that’s why I have made a decision: With a few exceptions (more on those below), I am going to say ‘no’ to conferences abroad from now on. This is a tough decision for me to make, but sometimes that’s exactly what you need to do. Tough, because I have very fond memories of all the conferences and meetups abroad I have contributed to. My first one, Romanian Testing Conference in 2017. My first keynote abroad, UKStar in 2019. My first one outside of Europe, Targeting Quality in 2023. They were all amazing, because of the travel and sightseeing (when time allowed), but also because of all the people I have met at these conferences. Yet, I can meet at least some of these people at conferences here in the Netherlands, too. Test Automation Days, the TestNet events, the Dutch Testing Day and TestMass all provide a great opportunity for me to catch up with my network. Sometimes, international conferences come to the Netherlands, too, like AutomationSTAR this year. And then there are plenty of smaller meetups here in the Netherlands (and Belgium) where I can meet and catch up with people as well. Plus, the money. I am not going to be a hypocrite and say that money doesn’t play into this. For the reasons mentioned above, I have a limited number of opportunities to travel every year, and I prefer to spend those on in-house training sessions with clients abroad, simply because the pay is much better. Even when a conference compensates flights and hotel (as they should) and offer a speaker or workshop facilitator fee (a nice bonus), it will be significantly less of a payday than when I run a training session with a client. That’s not the fault of those conferences, not at all, especially when they’re compensating their speakers fairly, but this is simply a matter of numbers and budgets. At the moment, I have one, maybe two contributions to conferences abroad coming up, and I gave them my word, so I’ll be there. That’s the SAST 30-year anniversary conference in October, plus one other conference that I’m talking to but haven’t received a ‘yes’ or ‘no’ from yet. Other than that, if conferences reach out to me, it’s likely to be a ‘no’ from now on, unless: the event pays a fee comparable to my rate for in-house training I can combine the event with paid in-house training (for example with a sponsor) it is a country or region I really, really want to visit, either for personal reasons or because I want to grow my professional network there I don’t see the first one happening soon, and the list of destinations for the third one is very short (Norway, Canada, New Zealand, that’s pretty much it), so unless we can arrange paid in-house training alongside the conference, the answer will be a ‘no’ from me. Will this reduce the number of travel opportunities for me? Maybe. Maybe not. Again, I see the number of requests I get for in-house training abroad growing, too, and if that dies down, it’ll be a sign for me that I’ll have to work harder to create those opportunities. For 2025, things are looking pretty good, with trips for training to Romania, North Macedonia and Denmark already scheduled, and several leads for more in the pipeline. And if the number of opportunities does go down, that’s fine, too. I’m happy to spend that time with family, working on other things, or riding my bike. And I’m sure there will be a few opportunities to speak at online meetups, events and webinars, too.
As is the case every year, 2025 is starting off relatively slowly. There’s not a lot of training courses to run yet, and since a few of the projects I worked on wrapped up in December, I find myself with a little bit of extra time and headspace on my hands. I actually enjoy these slower moments, because they give me some time to think about where my professional career is going, if I’m still happy with the direction it is going on, and what I would like to see changed. Last year, I quit doing full time projects as an individual contributor to development teams in favour of part-time consultancy work and more focus on my training services. 2024 has been a great year overall, and I would be happy to continue working in this way in 2025. However, as a thought experiment, I took some time to think about what it would take for me to go back to full time roles, or maybe (maybe!) even consider joining a company on a permanent basis. Please note that this post is not intended as an ‘I need a job!’ cry for help. My pipeline for 2025 is slowly but surely filling up, and again, I am very happy with the direction my career is going at the moment. However, I have learned that it never hurts to leave your options open, and even though I love the variety in my working days these days, I think I would enjoy working with one team, on one goal, for an extended amount of time, too, under the right conditions. If nothing else, this post might serve as a reference post to send to people and companies that reach out to me with a full time contract opportunity or even a permanent job opening. This is also not a list of requirements that is set in stone. As my views on what would make a great job change (and they will), I will update this post to reflect those views. So, to even consider joining a company on a full-time contract or even a permanent basis, there are basically three things I will and should consider: What does the job look like? What will I be doing on a day-to-day basis? What are the must-haves regarding terms and conditions? What are the nice to haves that would provide the icing on the cake for me? Let’s take a closer look at each of these things. What I look for in a job As I mentioned before, I am not looking for a job as an individual contributor to a development team. I have done that for many years, and it does not really give me the energy that it used to. On the other hand, I am definitely not looking for a hands-off, managerial kind of role, as I’d like to think I would make an atrocious manager. Plus, I simply enjoy being hands-on and writing code way too much to let that go. I would like to be responsible for designing and implementing the testing and automation strategy for a product I believe in. It would be a lead role, but, as mentioned, with plenty (as in daily) opportunities to get hands-on and contribute to the code. The work would have to be technically and mentally challenging enough to keep me motivated in the long term. Getting bored quickly is something I suffer from, which is the main driver behind only doing part-time projects and working on multiple different things in parallel right now. I don’t want to work for a consultancy and be ‘farmed out’ to their clients. I’ve done that pretty much my entire career, and if that’s what the job will look like, I’d rather keep working the way I’m working now. The must-haves There are (quite) a few things that are non-negotiable for me to even consider joining a company full time, no matter if it’s on a contract or a permanent basis. The pay must be excellent. Let’s not beat around the bush here: people work to make money. I do, too. I’m doing very well right now, and I don’t want that to change. The company should be output-focused, as in they don’t care when I work, how many hours I put in and where I work from, as long as the job gets done. I am sort of spoiled by my current way of working, I fully realise that, but I’ve grown to love the flexibility. By the way, please don’t read ‘flexible’ as ‘working willy-nilly’. Most work is not done in a vacuum, and you will have to coordinate with others. The key word here is ‘balance’. Collaboration should be part of the company culture. I enjoy working in pair programming and pair testing setups. What I do not like are pointless meetings, and that includes having Scrum ceremonies ‘just because’. The company should be a remote-first company. I don’t mind the occasional office day, but I value my time too much to spend hours per week on commuting. I’ve done that for years, and it is time I’ll never get back. The company should actively stimulate me contributing to conferences and meetups. Public speaking is an important part of my career at the moment, and I get a lot of value from it. I don’t want to give that up. There should be plenty of opportunities for teaching others. This is what I do for a living right now, I really enjoy it, and I’d like to think I’m pretty good at it, too. Just like with the public speaking, I don’t want to give that up. This teaching can take many forms, though. Running workshops and regular pairing with others are just two examples. The job should scratch my travel itch. I travel abroad for work on average about 5-6 times per year these days, and I would like to keep doing that, as I get a lot of energy from seeing different places and meeting people. Please note that ‘traveling’ and ‘commuting’ are two completely different things. Yes, I realize this is quite a long list, but I really enjoy my career at the moment, and there are a lot of aspects to it that I’m not ready to give up. The nice to haves There are also some things that are not strictly necessary, but would be very nice to have in a job or full time contract: The opportunity to continue working on side gigs. I have a few returning customers that I’ve been working with for years, and I would really appreciate the opportunity to continue doing that. I realise that I would have to give up some things, but there are a few clients that I would really like to keep working with. By the way, this is only a nice to have for permanent jobs. For contracting gigs, it is a must-have. It would be very nice if the technology stack that the company is using is based on C#. I’ve been doing quite a bit of work in this stack over the years and I would like to go even deeper. If the travel itch I mentioned under the must-haves could be scratched with regular travel to Canada, Norway or South Africa, three of my favourite destinations in the world, that would be a very big plus. I realize that the list of requirements above is a long one. I don’t think there is a single job out there that ticks all the boxes. But, again, I really like what I’m doing at the moment, and most of the boxes are ticked at the moment. I would absolutely consider going full time with a client or even an employer, but I want it to be a step forward, not a step back. After all, this is mostly a thought experiment at the moment, and until that perfect contract or job comes along, I’ll happily continue what I’m doing right now.
More in programming
A little while back I heard about the White House launching their version of a Drudge Report style website called White House Wire. According to Axios, a White House official said the site’s purpose was to serve as “a place for supporters of the president’s agenda to get the real news all in one place”. So a link blog, if you will. As a self-professed connoisseur of websites and link blogs, this got me thinking: “I wonder what kind of links they’re considering as ‘real news’ and what they’re linking to?” So I decided to do quick analysis using Quadratic, a programmable spreadsheet where you can write code and return values to a 2d interface of rows and columns. I wrote some JavaScript to: Fetch the HTML page at whitehouse.gov/wire Parse it with cheerio Select all the external links on the page Return a list of links and their headline text In a few minutes I had a quick analysis of what kind of links were on the page: This immediately sparked my curiosity to know more about the meta information around the links, like: If you grouped all the links together, which sites get linked to the most? What kind of interesting data could you pull from the headlines they’re writing, like the most frequently used words? What if you did this analysis, but with snapshots of the website over time (rather than just the current moment)? So I got to building. Quadratic today doesn’t yet have the ability for your spreadsheet to run in the background on a schedule and append data. So I had to look elsewhere for a little extra functionality. My mind went to val.town which lets you write little scripts that can 1) run on a schedule (cron), 2) store information (blobs), and 3) retrieve stored information via their API. After a quick read of their docs, I figured out how to write a little script that’ll run once a day, scrape the site, and save the resulting HTML page in their key/value storage. From there, I was back to Quadratic writing code to talk to val.town’s API and retrieve my HTML, parse it, and turn it into good, structured data. There were some things I had to do, like: Fine-tune how I select all the editorial links on the page from the source HTML (I didn’t want, for example, to include external links to the White House’s social pages which appear on every page). This required a little finessing, but I eventually got a collection of links that corresponded to what I was seeing on the page. Parse the links and pull out the top-level domains so I could group links by domain occurrence. Create charts and graphs to visualize the structured data I had created. Selfish plug: Quadratic made this all super easy, as I could program in JavaScript and use third-party tools like tldts to do the analysis, all while visualizing my output on a 2d grid in real-time which made for a super fast feedback loop! Once I got all that done, I just had to sit back and wait for the HTML snapshots to begin accumulating! It’s been about a month and a half since I started this and I have about fifty days worth of data. The results? Here’s the top 10 domains that the White House Wire links to (by occurrence), from May 8 to June 24, 2025: youtube.com (133) foxnews.com (72) thepostmillennial.com (67) foxbusiness.com (66) breitbart.com (64) x.com (63) reuters.com (51) truthsocial.com (48) nypost.com (47) dailywire.com (36) From the links, here’s a word cloud of the most commonly recurring words in the link headlines: “trump” (343) “president” (145) “us” (134) “big” (131) “bill” (127) “beautiful” (113) “trumps” (92) “one” (72) “million” (57) “house” (56) The data and these graphs are all in my spreadsheet, so I can open it up whenever I want to see the latest data and re-run my script to pull the latest from val.town. In response to the new data that comes in, the spreadsheet automatically parses it, turn it into links, and updates the graphs. Cool! If you want to check out the spreadsheet — sorry! My API key for val.town is in it (“secrets management” is on the roadmap). But I created a duplicate where I inlined the data from the API (rather than the code which dynamically pulls it) which you can check out here at your convenience. Email · Mastodon · Bluesky
Kubernetes is not exactly the most fun piece of technology around. Learning it isn’t easy, and learning the surrounding ecosystem is even harder. Even those who have managed to tame it are still afraid of getting paged by an ETCD cluster corruption, a Kubelet certificate expiration, or the DNS breaking down (and somehow, it’s always the DNS). Samuel Sianipar If you’re like me, the thought of making your own orchestrator has crossed your mind a few times. The result would, of course, be a magical piece of technology that is both simple to learn and wouldn’t break down every weekend. Sadly, the task seems daunting. Kubernetes is a multi-million lines of code project which has been worked on for more than a decade. The good thing is someone wrote a book that can serve as a good starting point to explore the idea of building our own container orchestrator. This book is named “Build an Orchestrator in Go”, written by Tim Boring, published by Manning. The tasks The basic unit of our container orchestrator is called a “task”. A task represents a single container. It contains configuration data, like the container’s name, image and exposed ports. Most importantly, it indicates the container state, and so acts as a state machine. The state of a task can be Pending, Scheduled, Running, Completed or Failed. Each task will need to interact with a container runtime, through a client. In the book, we use Docker (aka Moby). The client will get its configuration from the task and then proceed to pull the image, create the container and start it. When it is time to finish the task, it will stop the container and remove it. The workers Above the task, we have workers. Each machine in the cluster runs a worker. Workers expose an API through which they receive commands. Those commands are added to a queue to be processed asynchronously. When the queue gets processed, the worker will start or stop tasks using the container client. In addition to exposing the ability to start and stop tasks, the worker must be able to list all the tasks running on it. This demands keeping a task database in the worker’s memory and updating it every time a task change’s state. The worker also needs to be able to provide information about its resources, like the available CPU and memory. The book suggests reading the /proc Linux file system using goprocinfo, but since I use a Mac, I used gopsutil. The manager On top of our cluster of workers, we have the manager. The manager also exposes an API, which allows us to start, stop, and list tasks on the cluster. Every time we want to create a new task, the manager will call a scheduler component. The scheduler has to list the workers that can accept more tasks, assign them a score by suitability and return the best one. When this is done, the manager will send the work to be done using the worker’s API. In the book, the author also suggests that the manager component should keep track of every tasks state by performing regular health checks. Health checks typically consist of querying an HTTP endpoint (i.e. /ready) and checking if it returns 200. In case a health check fails, the manager asks the worker to restart the task. I’m not sure if I agree with this idea. This could lead to the manager and worker having differing opinions about a task state. It will also cause scaling issues: the manager workload will have to grow linearly as we add tasks, and not just when we add workers. As far as I know, in Kubernetes, Kubelet (the equivalent of the worker here) is responsible for performing health checks. The CLI The last part of the project is to create a CLI to make sure our new orchestrator can be used without having to resort to firing up curl. The CLI needs to implement the following features: start a worker start a manager run a task in the cluster stop a task get the task status get the worker node status Using cobra makes this part fairly straightforward. It lets you create very modern feeling command-line apps, with properly formatted help commands and easy argument parsing. Once this is done, we almost have a fully functional orchestrator. We just need to add authentication. And maybe some kind of DaemonSet implementation would be nice. And a way to handle mounting volumes…
As I slowly but surely work towards the next release of my setcmd project for the Amiga (see the 68k branch for the gory details and my total noob-like C flailing around), I’ve made heavy use of documentation in the AmigaGuide format. Despite it’s age, it’s a great Amiga-native format and there’s a wealth of great information out there for things like the C API, as well as language guides and tutorials for tools like the Installer utility - and the AmigaGuide markup syntax itself. The only snag is, I had to have access to an Amiga (real or emulated), or install one of the various viewer programs on my laptops. Because like many, I spend a lot of time in a web browser and occasionally want to check something on my mobile phone, this is less than convenient. Fortunately, there’s a great AmigaGuideJS online viewer which renders AmigaGuide format documents using Javascript. I’ve started building up a collection of useful developer guides and other files in my own reference library so that I can access this documentation whenever I’m not at my Amiga or am coding in my “modern” dev environment. It’s really just for my own personal use, but I’ll be adding to it whenever I come across a useful piece of documentation so I hope it’s of some use to others as well! And on a related note, I now have a “unified” code-base so that SetCmd now builds and runs on 68k-based OS 3.x systems as well as OS 4.x PPC systems like my X5000. I need to: Tidy up my code and fix all the “TODO” stuff Update the Installer to run on OS 3.x systems Update the documentation Build a new package and upload to Aminet/OS4Depot Hopefully I’ll get that done in the next month or so. With the pressures of work and family life (and my other hobbies), progress has been a lot slower these last few years but I’m still really enjoying working on Amiga code and it’s great to have a fun personal project that’s there for me whenever I want to hack away at something for the sheer hell of it. I’ve learned a lot along the way and the AmigaOS is still an absolute joy to develop for. I even brought my X5000 to the most recent Kickstart Amiga User Group BBQ/meetup and had a fun day working on the code with fellow Amigans and enjoying some classic gaming & demos - there was also a MorphOS machine there, which I think will be my next target as the codebase is slowly becoming more portable. Just got to find some room in the “retro cave” now… This stuff is addictive :)
One of the first types we learn about is the boolean. It's pretty natural to use, because boolean logic underpins much of modern computing. And yet, it's one of the types we should probably be using a lot less of. In almost every single instance when you use a boolean, it should be something else. The trick is figuring out what "something else" is. Doing this is worth the effort. It tells you a lot about your system, and it will improve your design (even if you end up using a boolean). There are a few possible types that come up often, hiding as booleans. Let's take a look at each of these, as well as the case where using a boolean does make sense. This isn't exhaustive—[1]there are surely other types that can make sense, too. Datetimes A lot of boolean data is representing a temporal event having happened. For example, websites often have you confirm your email. This may be stored as a boolean column, is_confirmed, in the database. It makes a lot of sense. But, you're throwing away data: when the confirmation happened. You can instead store when the user confirmed their email in a nullable column. You can still get the same information by checking whether the column is null. But you also get richer data for other purposes. Maybe you find out down the road that there was a bug in your confirmation process. You can use these timestamps to check which users would be affected by that, based on when their confirmation was stored. This is the one I've seen discussed the most of all these. We run into it with almost every database we design, after all. You can detect it by asking if an action has to occur for the boolean to change values, and if values can only change one time. If you have both of these, then it really looks like it is a datetime being transformed into a boolean. Store the datetime! Enums Much of the remaining boolean data indicates either what type something is, or its status. Is a user an admin or not? Check the is_admin column! Did that job fail? Check the failed column! Is the user allowed to take this action? Return a boolean for that, yes or no! These usually make more sense as an enum. Consider the admin case: this is really a user role, and you should have an enum for it. If it's a boolean, you're going to eventually need more columns, and you'll keep adding on other statuses. Oh, we had users and admins, but now we also need guest users and we need super-admins. With an enum, you can add those easily. enum UserRole { User, Admin, Guest, SuperAdmin, } And then you can usually use your tooling to make sure that all the new cases are covered in your code. With a boolean, you have to add more booleans, and then you have to make sure you find all the places where the old booleans were used and make sure they handle these new cases, too. Enums help you avoid these bugs. Job status is one that's pretty clearly an enum as well. If you use booleans, you'll have is_failed, is_started, is_queued, and on and on. Or you could just have one single field, status, which is an enum with the various statuses. (Note, though, that you probably do want timestamp fields for each of these events—but you're still best having the status stored explicitly as well.) This begins to resemble a state machine once you store the status, and it means that you can make much cleaner code and analyze things along state transition lines. And it's not just for storing in a database, either. If you're checking a user's permissions, you often return a boolean for that. fn check_permissions(user: User) -> bool { false // no one is allowed to do anything i guess } In this case, true means the user can do it and false means they can't. Usually. I think. But you can really start to have doubts here, and with any boolean, because the application logic meaning of the value cannot be inferred from the type. Instead, this can be represented as an enum, even when there are just two choices. enum PermissionCheck { Allowed, NotPermitted(reason: String), } As a bonus, though, if you use an enum? You can end up with richer information, like returning a reason for a permission check failing. And you are safe for future expansions of the enum, just like with roles. You can detect when something should be an enum a proliferation of booleans which are mutually exclusive or depend on one another. You'll see multiple columns which are all changed at the same time. Or you'll see a boolean which is returned and used for a long time. It's important to use enums here to keep your program maintainable and understandable. Conditionals But when should we use a boolean? I've mainly run into one case where it makes sense: when you're (temporarily) storing the result of a conditional expression for evaluation. This is in some ways an optimization, either for the computer (reuse a variable[2]) or for the programmer (make it more comprehensible by giving a name to a big conditional) by storing an intermediate value. Here's a contrived example where using a boolean as an intermediate value. fn calculate_user_data(user: User, records: RecordStore) { // this would be some nice long conditional, // but I don't have one. So variables it is! let user_can_do_this: bool = (a && b) && (c || !d); if user_can_do_this && records.ready() { // do the thing } else if user_can_do_this && records.in_progress() { // do another thing } else { // and something else! } } But even here in this contrived example, some enums would make more sense. I'd keep the boolean, probably, simply to give a name to what we're calculating. But the rest of it should be a match on an enum! * * * Sure, not every boolean should go away. There's probably no single rule in software design that is always true. But, we should be paying a lot more attention to booleans. They're sneaky. They feel like they make sense for our data, but they make sense for our logic. The data is usually something different underneath. By storing a boolean as our data, we're coupling that data tightly to our application logic. Instead, we should remain critical and ask what data the boolean depends on, and should we maybe store that instead? It comes easier with practice. Really, all good design does. A little thinking up front saves you a lot of time in the long run. I know that using an em-dash is treated as a sign of using LLMs. LLMs are never used for my writing. I just really like em-dashes and have a dedicated key for them on one of my keyboard layers. ↩ This one is probably best left to the compiler. ↩
SumatraPDF is a fast, small, open-source PDF reader for Windows, written in C++. This article describes how I implemented StrVec class for efficiently storing multiple strings. Much ado about the strings Strings are among the most used types in most programs. Arrays of strings are also used often. I count ~80 uses of StrVec in SumatraPDF code. This article describes how I implemented an optimized array of strings in SumatraPDF C++ code . No STL for you Why not use std::vector<std::string>? In SumatraPDF I don’t use STL. I don’t use std::string, I don’t use std::vector. For me it’s a symbol of my individuality, and my belief in personal freedom. As described here, minimum size of std::string on 64-bit machines is 32 bytes for msvc / gcc and 24 bytes for short strings (15 chars for msvc / gcc, 22 chars for clang). For longer strings we have more overhead: 32⁄24 bytes for the header memory allocator overhead allocator metadata padding due to rounding allocations to at least 16 bytes There’s also std::vector overhead: for fast appends (push()) std::vectorimplementations over-allocated space Longer strings are allocated at random addresses so they can be spread out in memory. That is bad for cache locality and that often cause more slowness than executing lots of instructions. Design and implementation of StrVec StrVec (vector of strings) solves all of the above: per-string overhead of only 8 bytes strings are laid out next to each other in memory StrVec High level design of StrVec: backing memory is allocated in singly-linked pages similar to std::vector, we start with small page and increase the size of the page. This strikes a balance between speed of accessing a string at random index and wasted space unlike std::vector we don’t reallocate memory (most of the time). That saves memory copy when re-allocating backing space Here’s all there is to StrVec: struct StrVec { StrVecPage* first = nullptr; int nextPageSize = 256; int size = 0; } size is a cached number of strings. It could be calculated by summing the size in all StrVecPages. nextPageSize is the size of the next StrVecPage. Most array implementation increase the size of next allocation by 1.4x - 2x. I went with the following progression: 256 bytes, 1k, 4k, 16k, 32k and I cap it at 64k. I don’t have data behind those numbers, they feel right. Bigger page wastes more space. Smaller page makes random access slower because to find N-th string we need to traverse linked list of StrVecPage. nextPageSize is exposed to allow the caller to optimize use. E.g. if it expects lots of strings, it could set nextPageSize to a large number. StrVecPage Most of the implementation is in StrVecPage. The big idea here is: we allocate a block of memory strings are allocated from the end of memory block at the beginning of the memory block we build and index of strings. For each string we have: u32 size u32 offset of the string within memory block, counting from the beginning of the block The layout of memory block is: StrVecPage struct { size u32; offset u32 } [] … not yet used space strings This is StrVecPage: struct StrVecPage { struct StrVecPage* next; int pageSize; int nStrings; char* currEnd; } next is for linked list of pages. Since pages can have various sizes we need to record pageSize. nStrings is number of strings in the page and currEnd points to the end of free space within page. Implementing operations Appending a string Appending a string at the end is most common operation. To append a string: we calculate how much memory inside a page it’ll need: str::Len(string) + 1 + sizeof(u32) + sizeof(u32). +1 is for 0-termination for compatibility with C APIs that take char*, and 2xu32 for size and offset. If we have enough space in last page, we add size and offset at the end of index and append a string from the end i.e. `currEnd - (str::Len(string) + 1). If there is not enough space in last page, we allocate new page We can calculate how much space we have left with: int indexEntrySize = sizeof(u32) + sizeof(u32); // size + offset char* indexEnd = (char*)pageStart + sizeof(StrVecPage) + nStrings*indexEntrySize int nBytesFree = (int)(currEnd - indexEnd) Removing a string Removing a string is easy because it doesn’t require moving memory inside StrVecPage. We do nStrings-- and move index values of strings after the removed string. I don’t bother freeing the string memory within a page. It’s possible but complicated enough I decided to skip it. You can compact StrVec to remove all overhead. If you do not care about preserving order of strings after removal, I haveRemoveAtFast() which uses a trick: instead of copying memory of all index values after removed string, I copy a single index from the end into a slot of the string being removed. Replacing a string or inserting in the middle Replacing a string or inserting a string in the middle is more complicated because there might not be enough space in the page for the string. When there is enough space, it’s as simple as append. When there is not enough space, I re-use the compacting capability: I compact all existing pages into a single page with extra space for the string and some extra space as an optimization for multiple inserts. Iteration A random access requires traversing a linked list. I think it’s still fast because typically there aren’t many pages and we only need to look at a single nStrings value. After compaction to a single page, random access is as fast as it could ever be. C++ iterator is optimized for sequential access: struct iterator { const StrVec* v; int idx; // perf: cache page, idxInPage from prev iteration int idxInPage; StrVecPage* page; } We cache the current state of iteration as page and idxInPage. To advance to next string we advance idxInPage. If it exceeds nStrings, we advance to page->next. Optimized search Finding a string is as optimized as it could be without a hash table. Typically to compare char* strings you need to call str::Eq(s, s2) for every string you compare it to. That is a function call and it has to touch s2 memory. That is bad for performance because it blows the cache. In StrVec I calculate length of the string to find once and then traverse the size / offset index. Only when size is different I have to compare the strings. Most of the time we just look at offset / size in L1 cache, which is very fast. Compacting If you know that you’ll not be adding more strings to StrVec you can compact all pages into a single page with no overhead of empty space. It also speeds up random access because we don’t have multiple pages to traverse to find the item and a given index. Representing a nullptr char* Even though I have a string class, I mostly use char* in SumatraPDF code. In that world empty string and nullptr are 2 different things. To allow storing nullptr strings in StrVec (and not turning them into empty strings on the way out) I use a trick: a special u32 value kNullOffset represents nullptr. StrVec is a string pool allocator In C++ you have to track the lifetime of each object: you allocate with malloc() or new when you no longer need to object, you call free() or delete However, the lifetime of allocations is often tied together. For example in SumatraPDF an opened document is represented by a class. Many allocations done to construct that object last exactly as long as the object. The idea of a pool allocator is that instead of tracking the lifetime of each allocation, you have a single allocator. You allocate objects with the same lifetime from that allocator and you free them with a single call. StrVec is a string pool allocator: all strings stored in StrVec have the same lifetime. Testing In general I don’t advocate writing a lot of tests. However, low-level, tricky functionality like StrVec deserves decent test coverage to ensure basic functionality works and to exercise code for corner cases. I have 360 lines of tests for ~700 lines of of implementation. Potential tweaks and optimization When designing and implementing data structures, tradeoffs are aplenty. Interleaving index and strings I’m not sure if it would be faster but instead of storing size and offset at the beginning of the page and strings at the end, we could store size / string sequentially from the beginning. It would remove the need for u32 of offset but would make random access slower. Varint encoding of size and offset Most strings are short, under 127 chars. Most offsets are under 16k. If we stored size and offset as variable length integers, we would probably bring down average per-string overhead from 8 bytes to ~4 bytes. Implicit size When strings are stored sequentially size is implicit as difference between offset of the string and offset of next string. Not storing size would make insert and set operations more complicated and costly: we would have to compact and arrange strings in order every time. Storing index separately We could store index of size / offset in a separate vector and use pages to only allocate string data. This would simplify insert and set operations. With current design if we run out of space inside a page, we have to re-arrange memory. When offset is stored outside of the page, it can refer to any page so insert and set could be as simple as append. The evolution of StrVec The design described here is a second implementation of StrVec. The one before was simply a combination of str::Str (my std::string) for allocating all strings and Vec<u32> (my std::vector) for storing offset index. It had some flaws: appending a string could re-allocate memory within str::Str. The caller couldn’t store returned char* pointer because it could be invalidated. As a result the API was akward and potentially confusing: I was returning offset of the string so the string was str::Str.Data() + offset. The new StrVec doesn’t re-allocate on Append, only (potentially) on InsertAt and SetAt. The most common case is append-only which allows the caller to store the returned char* pointers. Before implementing StrVec I used Vec<char*>. Vec is my version of std::vector and Vec<char*> would just store pointer to individually allocated strings. Cost vs. benefit I’m a pragmatist: I want to achieve the most with the least amount of code, the least amount of time and effort. While it might seem that I’m re-implementing things willy-nilly, I’m actually very mindful of the cost of writing code. Writing software is a balance between effort and resulting quality. One of the biggest reasons SumatraPDF so popular is that it’s fast and small. That’s an important aspect of software quality. When you double click on a PDF file in an explorer, SumatraPDF starts instantly. You can’t say that about many similar programs and about other software in general. Keeping SumatraPDF small and fast is an ongoing focus and it does take effort. StrVec.cpp is only 705 lines of code. It took me several days to complete. Maybe 2 days to write the code and then some time here and there to fix the bugs. That being said, I didn’t start with this StrVec. For many years I used obvious Vec<char*>. Then I implemented somewhat optimized StrVec. And a few years after that I implemented this ultra-optimized version. References SumatraPDF is a small, fast, multi-format (PDF/eBook/Comic Book and more), open-source reader for Windows. The implementation described here: StrVec.cpp, StrVec.h, StrVec_ut.cpp By the time you read this, the implementation could have been improved.
