Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
1

Mutation testing - not just for unit tests

from On Test Automation [alt+shift+b] in programming

I wrote about mutation testing a few times on this blog, and I even have a mutation testing workshop that I run on a pretty frequent basis. One misconception that pops up sometimes, either explicitly in a question or implicitly when I hear others talk about mutation testing, is that mutation testing only works for unit tests. That’s not true. You can use mutation testing to find out more about the quality of other types of tests, too. In this blog post, I’ll show you an example using a code base that I’m using in my mutation testing workshops as well. I’ll link to the codebase at the end of this blog post, so you can have a look and try it out for yourself, too. Keep in mind that while mutation testing isn’t just for unit tests, it works best for tests that run quickly. Your test suite will be run for every mutation in the product code that your mutation testing tool generates, so if you don’t want to wait hours or even days for your mutation testing results, you’d be wise to use it with tests that run fast, that is, with an execution time measured in milliseconds, maybe a second or two tops. Our application under test To demonstrate that mutation testing works for other types of tests, too, I wrote an API in Java using Spring Boot, complete with a controller, service and repository layers, connecting to an H2 in-memory database. The API allows you to create, retrieve and delete bank account instances, as well as perform different types of transactions: deposit into an account, withdraw from an account and add interest to an account balance. Here’s the implementation of the deposit operation in the controller class and the corresponding service layer to give you an idea what that looks like: // From AccountController.java @PostMapping(path = "/{id}/deposit/{amount}", consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE) public ResponseEntity<Account> depositToAccount(@PathVariable("id") Long id, @PathVariable("amount") double...
3rd Sep 2025

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from On Test Automation

Contract testing implementation challenges

This post was previously published through my newsletter on October 19, 2026. From time to time, I will republish newsletter issues on my blog here if I think people (and search engines) might benefit from it. If you want to read everything I’ve written once it is posted, I recommend signing up for the newsletter. For the last 5-6 years or so, I’ve been talking about, teaching and helping teams implement contract testing on a regular basis. To make that more concrete: at the moment, I’m having ongoing conversations with four different potential clients about helping them in some way with their contract testing implementation, and so far, I’ve run my contract testing workshop four times this year. It probably doesn’t come as a surprise to you when I say that I really like the concept of contract testing as a means of addressing integration testing challenges in distributed software systems. However, no matter if I’m speaking, teaching or implementing contract testing, certain challenges almost always come up at some point, and I want to highlight some of those today. Understanding the concept of contract testing First of all, it often takes people a while to understand the concept of contract testing, especially when they’re used to what I call the ‘traditional’ approach to integration testing: deploying the systems and services in scope to a shared testing environment and then testing them ‘as a whole’. Of course, the more distributed your system and your development process is, and the higher the deployment frequency, the more complex and the slower this integration testing approach is going to be. The premise behind contract testing is to break that integration testing puzzle down into small chunks of individual consumer-provider pairs, and then test the integration between those two in an asynchronous manner. By asynchronous, I mean that there is no need to deploy consumer and provider into a shared environment; both parties perform their part of the process in their own, local development environment. This means you can detect potential integration issues before you deploy, instead of stumbling onto them after you do so. This makes contract testing (or contract-based integration testing, to use its proper name) a significantly different approach to integration testing compared to the more ‘traditional’ approach. In my experience, it usually takes people time to wrap their head around how the process actually works, which steps you should take and in which order, and what you (do not yet) know after each step. It often takes going through several contract testing cycles before people start to ‘get it’. That’s not their fault, that is to be expected, since the contract-based approach to integration testing is, again, quite different from what they’ve been doing for so long. To better cater to this, I redesigned my contract testing workshop a year and a half or so ago. It is now a full-day workshop (half a day is simply too short), and I take much smaller steps and repeat them more often. From what I see, this has led to a better understanding of the concepts with my audience, and that’s the whole point. Contract testing === Pact I’ve talked to quite a few teams that had heard of contract testing somewhere, decided that that was the way out of their integration testing problems, and then immediately started writing contract tests with Pact. Understandable, because Pact is a very useful, very powerful and very popular tool, plus, it’s open source, which makes it easy for teams to get started. Often, they skipped a step in the process. Pact is very much a consumer-driven contract testing tool by nature, and while consumer-driven contract testing is the richest and most powerful of the different approaches to contract testing, it is not necessarily the best fit for every team and every context. It’s a classic example of using the wrong tool for the problem at hand, because they didn’t take enough time looking at the problem and the available solutions. That’s not the problem of Pact, even though it is often framed that way (‘Pact didn’t solve my problems, therefore Pact is not a good tool’). That’s a ‘we didn’t fully understand the problem and the context’ problem. Contract testing removes the need for communication Another challenge that I often see when I talk about or help teams with contract testing is the fact that teams don’t talk to each other. This then leads to integration issues, because team A expects something from team B that team B doesn’t supply. Contract testing is not the solution to these problems. It can’t be - tools alone do not solve problems, let alone communication problems. What contract testing does do is make potential integration issues visible early on in the development process, often earlier than (again) more ‘traditional’ approaches to integration testing. That’s where the value of contract testing can be found. It is not a means to replace communication between teams. In fact, and this is especially true for consumer-driven contract testing, contract testing cannot succeed without good communication between consumer and provider teams. Contract testing replaces all other types of integration testing This one comes up less often than it used to, but sometimes, I still hear people saying that when they will have contract-based integration tests in place, they don’t need other types of integration tests anymore - including E2E tests. That’s simply not true. While contract testing is a great way to find out earlier about the impact of a change in a specific consumer or provider further downstream, in many cases, I wouldn’t rely on these tests alone before moving to production. Because of its asynchronous nature, contract-based integration testing only verifies whether a consumer and a provider speak the same language - can a consumer understand the message returned by a provider in a specific situation? It does not verify that the request or the response message reaches the other party in the first place, for example. In other words, it doesn’t test connectivity between a consumer and a provider. And because integration testing is broken down into individual consumer-provider pairs, you don’t see the entire system working ‘as a whole’. This means you might want to keep a few end-to-end tests in place, even when you’ve got good contract testing coverage across the board. The above is not a complete list of challenges with contract testing, but the ones I mentioned here are those that come up most often.

6 days ago • 1 votes
Should testers write unit tests?

This post was previously published through my newsletter on May 25, 2026. From time to time, I will republish newsletter issues on my blog here if I think people (and search engines) might benefit from it. If you want to read everything I’ve written once it is posted, I recommend signing up for the newsletter. A few months ago, I had the pleasure of delivering a keynote at an internal developer conference for one of the largest banks in the Netherlands. While I still don’t really see myself as a keynote speaker - I enjoy doing practical, hands-on sessions much more - I had a great time talking about challenges of E2E testing, breaking down E2E tests and the test automation quadrant model that I use in my thinking, speaking and teaching about test automation these days. However, the keynote or the contents of it are not what I want to talk about this week. Instead, it was a question that came up during the Q&A after the talk that triggered me to write this post. That question was “Do you think that testers should be writing unit tests?” It’s not the first time I heard that question. In fact, I’ve seen and heard whether or not testers should be involved in unit testing being discussed regularly in the past, with arguments for and against the various standpoints. However, I was under the impression that we, collectively, had found some sort of answer to the question and moved past this point by now. Guess I was mistaken. I tried to give the person asking the question an answer as well as I could, but given that there is quite a bit to unpack around the topic, I’m not sure if I gave them the entire story. So, that’s what I’ll try and do here. Who knows they might even read it… So, should testers be writing unit tests? Well, my answer is either a ‘yes’ or a ‘no’, depending on how you interpret the question. Before we explore these various interpretations, what is a unit test anyway? Well, by now, I don’t really know anymore, as there are so many definitions floating around, some of them contradicting each other. This is one of the reasons I came up with the test automation quadrant model as an alternative to the well-known automation pyramid model, but again, that’s not the topic of this post. For the sake of the argument, let’s define a unit test as a test that verifies a very small piece of behaviour of our product, without relying on external interfaces like APIs, databases or file systems, for example. With that definition in place, let’s look at two different interpretations of the question of ‘should testers be writing unit tests?’. Interpretation 1: Testers should be responsible for unit testing Well, no, I don’t think they should, no matter how good of a tester they are, and no matter what their coding skills are. Writing unit tests is an activity that should be performed in support of and lockstep with software development. Often, especially in practices like test-driven development, tests are written first, and they drive the design and development of the product. Leaving the writing of these tests to testers, especially if it is done after the product code itself is written, is both inefficient and a potential source of problems. Inefficient, because the product has already been written, yet we only know whether its behaviour matches expectations once the tests are written and run. Also, because there’s a handoff happening between ‘development’ and ‘testing’, which takes up valuable time as the developer will switch to a different task while the tester writes the unit tests. If there’s a problem with the product that is discovered during unit testing, the developer needs to make a context switch back to the original task, and the more context switching you do during the day, the less efficiently you will work and the less you will get done. A potential source of problems, because when the developer only focuses on shipping a potentially working product, they will likely not spend too much time thinking about what to test for, or how to make the product (the code, in this case) easily testable in the first place. Also, the handoff from ‘development’ to ‘test’ I described before leaves open room for different interpretations of what the software should do, leading to potential bugs slipping through and the resulting back-and-forth discussions after the fact. So, no, I don’t think we should leave unit testing to testers. I still sometimes hear about developer who don’t want to or do not know how to write (decent) unit tests, and I think that’s a problem that needs to be fixed at the source, instead of trying to patch it up by someone else writing the unit tests for the already-created product. Interpretation 2: testers should be involved in unit testing If we interpret the question this way, I think the answer should be a resounding ‘yes’, testers should be involved in unit testing. I mean, there’s the word ‘testing’ in the name, why shouldn’t we involve the people who specialize in testing in the process? What that involvement looks like, exactly, depends on the context, of course. Again, ‘being involved’ and ‘being responsible’ are two entirely different things. While I believe that writing unit tests is a development activity, there are a couple of ways in which testers can add value, too: They can review the unit tests to learn about what has been covered already, so that they do not repeat that testing later on They can review the unit tests to identify what has not yet been covered, and either give that back as feedback to a developer, or add the missing tests themselves They can suggest and use techniques like mutation testing to test the tests and find out whether the tests that were written before are actually able to catch meaningful problems I’m sure there are a few more benefits, but these alone should, in my opinion, be enough for any team to not exclude testers from the process of writing unit tests from here on. And yes, that will require some additional skills from both testers and developers. This is where the power of collaboration comes in. I’m a big fan of pair programming and testing, and I’ve seen a lot of good things come from testers and developers pairing up to write, review and improve unit tests. The developer improves their testing skills, the tester learns more about both the product they’re testing and the development process, and in the end, both the entire team and the product itself reaps the benefits. That’s a long answer to a simple question, and I’m sure there are some nuances that I didn’t yet unpack, but generally speaking, these are my views on the question of whether testers should be writing unit tests. Oh, and before you ask ‘but what about integration / end-to-end / performance / security / … tests’? That’s easy. Simply replace ‘unit tests’ and ‘unit testing’ with ‘integration / end-to-end / performance / security / … tests’ and ‘… testing’, and you’ll have my answer. I’ve always found it a little strange that unit tests have traditionally been seen as ‘different’ from other types of tests, as if they’re some kind of special artefact that only developers know how to write. If that’s what you think, too, let me let you in on a little secret: they aren’t special. Unit tests are simply tests that test a small piece of the behaviour of the product that we write, written against and invoking a specific interface of the product: the source code. Other types of tests do exactly the same thing: verify parts of our product behaviour by invoking one or more specific interfaces (APIs, the UI, a database, a queue, …). They just have a different scope, and with that, they verify behaviour at a different scope. That’s all. This, again, is the reason where I think the traditional test automation pyramid model lacks: I really don’t care that much about what is a unit / integration / end-to-end test. All I really care about is tests that produce valuable information about the state and the behaviour of our product in as efficient a way as possible. Unit tests really aren’t any different.

20th Aug 2026 • 1 votes
I’m (re-)starting a newsletter

Just a quick update to let those of you who bookmarked this blog or who have subscribed to my RSS feed know that I have (re-)started a newsletter. Why a newsletter? As you might know (or not), while I’ve been pretty active on LinkedIn over the years, I do have a love-hate (or rather an appreciate-hate) relationship with that platform. Lately, I’ve been noticing that the pendulum is swinging in the ‘hate’ direction more often, mainly because the ever-changing algorithm used by LinkedIn makes it incredibly hard to predict if people are even going to see what I write. I’d rather publish my thoughts, ideas and other ramblings via a platform that I do control, and that platform will be a newsletter. I’ve had a newsletter in the past, but that only lived for about three months. This time, I intend to keep writing and publishing a new issue every week. The first edition goes out a few hours after I’m writing this, and a new issue will be sent to subscribers every Monday morning around 11 AM CET. But what about the blog? I’ll still publish to the blog, too, but that will be on a much less regular basis. Just like it has been for a while, really. The idea is to post the more ‘technical’ posts, that is, the ones including code, directly to my blog, whereas the ‘text-and-images-only’ posts go through my blog post first. My priority is with the newsletter, though. How to subscribe That’s easy, just go to the subscription page, leave your email address, click the button on the confirmation email and you’re in. I promise I won’t use the newsletter or your email to spam or sell to you. Ever.

11th May 2026 • 1 votes
The ‘valuable’ in valuable feedback, fast

When I talk about the goals and the purpose of test automation, I often use the phrase ‘valuable feedback, fast’: we use tools to support our testing to help us get valuable information about the state of our product in the most efficient manner possible. The ‘fast’ part of ‘valuable feedback, fast’ is pretty self-explanatory for most people: as build and release cycles are becoming shorter, teams want to be informed timely about any unexpected changes in behaviour of their product, often after every change they make to that product. Tools can help them achieve that by running quick, focused tests automatically when a change is made or committed to version control. Of course, it takes plenty of hard work to write those tests to be fast, but that’s not what I wanted to talk about here. The ‘valuable’ in ‘valuable feedback, fast’ is a much more ambiguous term, and one that deserves some more explanation. To me, there are multiple dimensions to what makes a test valuable, and in this post, I want to unpack and address them one by one. Valuable = important to someone who matters Borrowing from the classic definition of ‘quality’ as defined by Jerry Weinberg and further refined by James Bach and Michael Bolton, this is where it all starts. The information presented by a test should be important to someone who matters. That someone could be a member of the development team, a stakeholder such as a product owner or business analyst, the end user of the product, or a combination of those. Without that importance, a test is meaningless, dead weight. It could be the most reliable, best-written test ever, but if the information that is provided by it is not important to someone who matters in the context of the product, why bother writing, running and maintaining the test? Valuable = covering what matters Test coverage is a tricky subject, and I want to steer clear of the discussion on what ‘coverage’ means exactly in this blog post. The only realistic answer is ‘it depends’, anyway, as there are so many ways to define coverage (line, branch, requirements, mutation, …). Having said that, for the information provided by our tests to be valuable, teams should invest time in making sure that the tests cover the parts of the product behaviour that are deemed ‘important enough’ in a sufficient manner. What exactly constitutes ‘sufficient’ here depends on, you guessed it, the context. Some products require deeper, more thorough coverage than others. The same applies to individual parts of the same product. It all depends on the acceptable amount of risk a team is willing to take before putting a product in the hands of their users. Teams would do well to have a continual discussion about these risks and the extent to which they are covered by the tests that accompany and scrutinize the product. Valuable = trustworthy The higher the degree of automation in the build and delivery process of a product, and that includes testing, the more teams will rely (and have to rely) on the results of the execution of that automation. Concerning tests, that means that teams need to be able to rely on the information presented by the tests, because they will make decisions based on that information. The nature of that decision might vary from anywhere between ‘this build seems sufficiently stable to warrant deeper testing’ to ‘this change is ready to be put in the hands of our users’. No matter what the specific decision is, if teams make it based on the results of your test automation, even in part, they can only confidently do so if the information provided by the tests is trustworthy. In practice, that means that when a test emits a signal indicating a problem with the product, the team can safely conclude that there is a problem with the product, not with the test, the data it uses or the environment it runs in (no false positives). It also means that when a test does not emit such a signal, the team can trust that the particular piece of behaviour exercised by the test is working according to expectations expressed in the test (no false negatives). Valuable = actionable Another dimension of the value of the feedback provided by a test is that it should be actionable. This applies specifically to those situations where a test ‘fails’, i.e., it indicates a problem with the product I have put ‘fails’ between quotes here, because the test didn’t fail, the product failed the test. There’s a difference. Anyway, when a test result indicates a (potential) problem with the product, teams need to able to act on that information as soon as possible, spending as little time digging deeper into the product or into the test as possible to identify the root cause of the problem. Some practices that might help here are: Making your test scope as small as possible - the fewer moving parts your test has, the easier it will be to identify which of those parts made a move that was unexpected Have good test names - A descriptive test name that tells you what part of the behaviour your product verifies and what the expected behaviour is helps in finding out where exactly the problem might be found Use custom assertion messages - Many test frameworks allow you to specify custom, descriptive error messages in case of assertion failures (something RestAssured.Net supports as of version 5.0.0, too) So, is this a complete and final definition of what ‘valuable’ means to me when I talk about ‘valuable feedback, fast’ as the goal of test automation? I don’t think so. I don’t know if it is complete, but it definitely is a good reflection of my current thoughts on ‘value’ in test automation right now. Those thoughts are definitely not ‘final’, and I would appreciate your takes on what I wrote here.

1st Apr 2026 • 1 votes
Writing tests with Claude Code - part 1 - initial results

In a recent post, I wrote about how I used Claude Code to analyze the code for RestAssured.Net and then perform a refactoring action, using hand-written tests as the safety net. In that post, I wrote that I didn’t want Claude to touch the tests themselves, and why. I was still curious, though, to find out for myself what Claude was capable of in terms of writing tests. In this blog post, I’ll share with you some first steps in doing exactly that, and you’ll read about my thoughts and my thought process along the way. You’ll see how I create an initial suite of tests for a small Spring Boot-based API that I wrote for use in my workshops, and how I think about and assess the results. In a follow-up blog post, I’ll show you how I improved the test suite based on my findings, again using Claude Code. The starting point As a starting point, I created a new repository containing the code for the API I use in my mutation testing workshop. I removed the existing tests, as we’re going to ask Claude to generate these for us. I also removed the README and the GitHub Actions build pipeline definition, as I want Claude to write tests based only on the product code itself, without being primed by other artifacts in the codebase. The only thing I left in are the dependencies used to write and run the tests, in this case REST Assured and JUnit. After installing and initializing Claude, I gave it a first prompt: “Add acceptance tests for the endpoints exposed by the AccountController to this project. Cover all the logic in the AccountService class. Use REST Assured as the tool to interact with the API. Use JUnit 5 as the test runner. Both libraries are already part of the project, see the pom.xml. Assert status codes and relevant response body elements as part of the tests. Extract common request properties into a RequestSpecification.” After some deliberation, Claude added a new test file to the project, containing 23 tests, all of them passing. You can see these tests here. What you’re seeing in this file is the raw output from the above prompt, I haven’t changed anything in there. It took Claude only a minute or two to write these tests, which definitely is a lot faster than what I could have done myself. But how good are they, really? A first look at the tests Let’s look at the quality of the code first. I’m seeing people argue that code quality is not really all that important anymore once AI will write most of our code, but I beg to differ, especially when it concerns our tests. Tests are documentation of the intended behaviour of our code, and I would say that being able to read that documentation as a human being, without too much effort, remains very important. So, is our code easy to read? There’s a @BeforeEach hook creating the RequestSpecification (an object in REST Assured containing shared HTTP request properties). There’s a helper method to create a new account passing in the AccountType and a predefined balance. There’s the aforementioned 23 tests that, especially at first glance, seem to verify things that are valuable. What Claude did not do, probably because I didn’t explicitly ask for it, is add an abstraction layer to make the code easier to read, such as the one described here. We’ll see how Claude does in this area in the next blog post, as I want to stick to assessing the quality of the initial output from Claude in this one. And I have to say, all in all, for a first try, I’m not unhappy with what I’m seeing. Yes, there’s room for improvement, but I have seen humans do far worse than this. The tests seem to cover all endpoints defined in the API controller, and most paths in the business logic defined in the service layer. I should note here that I was able to fairly quickly come to this conclusion only because: I wrote the code for the API, so I have knowledge of the inner workings and the intent of the API, and I have plenty of experience writing tests for APIs and writing tests in REST Assured, so I’d like to think I know what ‘good’ looks like If you don’t have that prior knowledge and experience, it will be harder to draw meaningful conclusions from just looking at what Claude coughs up. And there’s a significant risk there: the risk of saying ‘looks good to me’ without actually understanding what you’re approving, and then ending up with a safety net of tests that is riddled with holes. Testing the generated tests with mutation testing To further increase our understanding of the value of the tests that were generated for me, let’s see if these tests can fail. If they can’t, the fact that we have generated 23 passing tests in two minutes flat is nothing more than an example of productivity theater. To check if our tests can actually fail, let’s use a mutation testing tool to scrutinize our tests a little more. In this case, because we’re working with Java code, I’ll use PITest as my mutation testing tool of choice. I configured the tool to mutate all the code in the project and run all the tests, to get a complete overview of the quality of the test suite generated. Note that in a real life-sized project, you probably want to start by mutating only part of the code base and run part of the tests to get mutation testing feedback within a reasonable amount of time. After about a minute, PITest reports back that the initial test suite achieves 95% line coverage. This looks impressive, but it doesn’t really tell me anything. The much more valuable metric here is the number of mutants that were killed by the test suite. PITest reports that this is 91%, which, again is pretty good. In absolute numbers, out of 55 mutants generated by PITest, 50 were detected by the initial test suite. Two follow-up questions arise immediately: Which mutants were missed by the tests, and what is the impact of that? Could we have achieved the same amount of (line and mutation) coverage with fewer tests? In other words, do we have tests that are dead weight? Looking at the surviving mutants First, let’s have a look at the mutants that survived, i.e., changes in the API code that were not detected by any of the tests. To start, in the CustomizedResponseEntityExceptionHandler, the HTTP 500 path isn’t covered in any of the tests, and that causes a surviving mutant. By design, the API returns an HTTP 500 when an Exception occurs that isn’t a ResourceNotFoundException (returning an HTTP 404) or a BadRequestException (returning an HTTP 400). This looks like a useful path to cover in a test. Second, the API returns an HTTP 204 in response to a GET call to /accounts when there are no accounts in the database. That path isn’t covered in the tests. This, too, seems like a useful path to test, because it is intentional API behaviour. Finally, the tests that were written do not properly cover some of the boundary values, both in the logic that implements the business rule of ‘you cannot overdraw on a savings account’ and in the interest calculation logic. Once more, I would like to have these situations covered by tests. Coincidentally (or maybe not?), these are all cases that I cover in my mutation testing workshop, too. This, to me, indicates that mutation testing is a powerful way to assess what is tested and what isn’t, no matter if you wrote the tests or you had them write by an LLM. I’m also happy to see that I’m probably covering the right things in my workshop. Note: I can confidently and quickly perform this analysis of the signals produced by PITest, and of the quality of my tests, because I know that mutation testing as a technique exists, and because I know how it works. Most importantly, I’m motivated / I feel like I am morally obliged to do so, because I deeply value writing tests that test meaningful things and that are actually able to detect changes in product behaviour. If all I cared about was having some tests to cover the API and declared, for example, 90% line coverage as ‘good enough’, I would be done by now. However, I don’t. In the next blog post, I want to return this feedback to Claude and see how well it does in updating the existing test suite based on my observations. I also want to see if I can add mutation testing to the test generation loop, and have Claude achieve better mutation coverage without my interfering. For now, I’ll conclude that when I ask Claude to generate tests in the way I have done, it produces pretty good results in terms of both line and mutation coverage, but that it missed certain key paths in my application code. Identifying dead weight in our test suite As a next step, I want to find out if the test suite that was generated by Claude contains dead weight, that is, do we have any tests that do not uniquely contribute to either line or mutation coverage? To do so, I asked PITest to generate a report in XML format next to the HTML report, as (for some reason) only the XML report contains information about which test killed a specific mutant. Performing this analysis required a bit of elbow grease, as I had to manually search the XML test report for occurrences of the test name for every test in the test suite. This, too, is probably a process that can be automated, but for now, I’m OK with doing this the manual way, since there’s only 23 tests in the suite anyway. This search tells me that four tests that were generated by Claude did were not mentioned as a test killing a mutant in the results file. In all four cases, the reason behind this is that the exact same code path is exercised in another test. For example, one of the tests performs a withdrawal on a checking account and verifies that the balance is updated accordingly: @Test void withdraw_positiveAmount_fromCheckingAccount_updatesBalance() { long id = createAccount(AccountType.CHECKING, 500.0); given(requestSpec) .post("/{id}/withdraw/{amount}", id, 200.0) .then() .statusCode(200) .body("balance", equalTo(300.0f)); } The next test in the suite, however, does the exact same thing for a savings account: @Test void withdraw_positiveAmount_fromSavingsAccount_withSufficientFunds_updatesBalance() { long id = createAccount(AccountType.SAVINGS, 500.0); given(requestSpec) .post("/{id}/withdraw/{amount}", id, 200.0) .then() .statusCode(200) .body("balance", equalTo(300.0f)); } After removing these four tests from the suite and running mutation testing again, as expected, I can see that the impact on both line and mutation coverage is 0, meaning that these four tests can indeed be classified as ‘dead weight’. Conclusions So, after completing the analysis of the results of asking Claude Code to generate tests for a new code base, what do I think? Well, while I am impressed, I think a couple of words of warning are in order. I am positively surprised by the quality and the coverage of the initial test suite. 95% line coverage and 91% mutation coverage are good numbers, and all that coverage was generated in a few minutes, definitely a lot less time than it would have taken me to write these tests myself. There is some room for improvement in terms of readability of the tests, but that can probably be resolved by being more specific in my prompt and / or using dedicated Claude Code skills. I’ll explore and write about that soon. While Claude achieved a pretty decent mutation coverage, it did oversee a few critical paths in the code. Maybe I was simply ‘unlucky’, and another attempt with the same prompt would have given better results. I don’t know, but it does tell me not to simply accept what Claude gives me at face value. The same applies to the tests that Claude did generate. 4 out of the 23 tests generated were dead weight, which equates to 17% of the test suite. Now, n = 1, and this is a small codebase and test suite, so the numbers might be skewed, but again, if you want your test suite to be as efficient and effective as possible, these are numbers that you probably don’t want to ignore. Finally, there are of course many things that Claude did not do, mainly because I didn’t ask it to. An example of that would be telling me that since we’re working with a banking API, it probably would be a good idea to add some form of authentication to the endpoints. There’s a lot more to unpack about what Claude does and does not do, and I will probably write about that in more detail in another blog post, but not here. First, in a follow-up blog post, I’ll document the process of improving the existing test suite that Claude generated, both in terms of coverage and of coding style. I will once again be using Claude and mutation testing to do that. The code for the API that was used in this blog post, as well as the initial suite of tests generated by Claude, can be found here.

9th Mar 2026 • 1 votes

More in programming

AI Assisted Sermon

Once a year at my church, I usually give one sermon. I was asked to tie the theme to a young adult retreat we had taken to the Sunshine Coast. My first experience giving a sermon was many years ago at my old home church in El Monte, California. It was a Vietnamese-American church, but…

an hour ago • 1 votes
Must we always use LLMs this way?

As with my last post on AI, let me open with a statement: Just because most people hold AI a certain way, doesn’t mean that we always must, either now or in the future. As I noted in that piece, and my shades of gray post, I am well aware — and agree with — … Continue reading Must we always use LLMs this way? →

yesterday • 1 votes
Revisiting Buttons and Cursors

The CSS Working Group resolved to use cursor: pointer for buttons and other clickable form controls in the new appearance: base styles. That happened on October 21, but I posted about it yesterday and, uh, people have feelings about cursors. I wrote about buttons and cursors back in 2022, when a post titled "Buttons shouldn't have a hand cursor" was making the rounds. I admitted I didn't have a strong opinion either way, then argued that not using the hand cursor had become the new anti-pattern anyway. The web isn't an operating system, and people expect clickable things to show a hand. The issue is Cursors For Base Appearance in CSS Forms Level 1. The group adopted the rules from this comment, and the part that's triggering half the Web is: select, option, input[type=checkbox], input[type=radio], button, input[type=button], input[type=submit], input[type=reset], input[type=color], input[type=image], ::file-selector-button { cursor: pointer; } It seems like we have a tabs vs. spaces sort of debate. Again. Nicole Sullivan pulled cursor data from a bunch of design systems and, in September, 27 of the 33 systems that set a cursor used pointer. After she added more systems, it was closer to a 50/50 split. Elika Etemad made the usability case in the meeting. Per the minutes, she said the hand "makes it clear, as soon as you're hovering an interactive control, you're supposed to click on it," and called it a "usability benefit even if not historically done on some platforms." "But the hand means it's a link" Haz had an interesting reaction: The debate is over, and evil has won. His point is that he can't tell whether ⌘-clicking something will open it in a new window anymore. He feels more confident clicking a button when he knows it won't change the URL, like when it just opens a popover or a panel. He's not wrong about that.* I replied that I've been on team evil since 2022 (albeit without a strong opinion, just following what users seem to expect). Touch screens don't have a cursor at all, and on desktop, browsers show a link's URL in the corner when you hover it. That's some type of hint, at least. Haz pushed back that he'd have to watch the bottom-left corner with his peripheral vision, which is fair. He also said he never cared which cursor buttons got, he just wanted links to have their own. I can appreciate that. "But desktop apps don't do it!" Jacob Paris liked the standard cursor on buttons because it makes web apps feel more like desktop apps than websites. My opinion hasn't changed, though. People have already learned that a hand cursor means clickable on the Web. It's by no means mandatory, but it's familiar and satisfying, so it stuck. Either way, nothing changes today. The resolution only covers appearance: base, which browsers haven't shipped yet. Buttons and selects still use appearance: auto, and their cursor is still default. If your CSS reset, design system, or web components already set cursor: pointer on buttons, keep it. When appearance: base ships, controls that opt in will get the same cursors by default. But remember, these are only defaults! You can still add whatever cursor you want to whatever element you want. Nobody's gonna stop you. In 2022, I said I'd keep changing the cursor on buttons until there's a good reason to stop. In 2026, I still haven't found one…and neither has the CSS Working Group. *Thinking about this more, if 50% of design systems opted to add cursor: pointer to buttons already, isn't this already a moot argument? **In general, I think the Web should be more personal and expressive so who even cares about a default? This is especially true for personal and fun sites. Just don't go too crazy, especially when it affects accessibility and the overall user experience.

yesterday • 1 votes
TokyoDev Welcomes Its Head of Operations

TokyoDev has grown substantially over the past few years. We’ve expanded our community, dramatically increased the scope and frequency of our events, and have curated a library of articles for international developers. Despite all that, the way we run the business hasn’t changed much. I’ve continued to coordinate most of these activities myself, making the final decisions and keeping everything connected. In short, TokyoDev cannot run without my daily input. That approach worked well when TokyoDev was smaller, but it’s increasingly limiting how much we can accomplish. To help the business grow further, I wanted to find someone to take over some of my responsibilities and bring a fresh perspective. This is a significant step for me personally, as while I have worked with many contractors over the years, I have never actually hired a regular employee. Most importantly, I wanted to ensure that this person was someone I could trust to understand what TokyoDev stands for, and to make decisions in line with our philosophy. So I’m happy to announce that Scott Rothrock is joining TokyoDev full time as our Head of Operations. About Scott Scott has lived in Japan for nearly 20 years and has worked as a software engineer in multiple industries and environments, including small Japanese-owned businesses, overseas employers, and international companies based in Japan. Until the end of October 2026, he was the head of the Japan Markets team at Kraken Technologies, where he led an international team of software engineers in delivering complicated, regulated products on strict timelines for Japanese electricity and gas retailers, while also ensuring compliance with evolving organizational and global standards. He’s made a name for himself both inside and outside Japan as a kind, effective mentor who can cultivate great engineers and leaders. Scott’s previous role in TokyoDev Some of you may already know Scott in another capacity, as he has been involved with TokyoDev since 2022. He began as a community volunteer and later became our community manager, which also included responsibilities such as managing our article pipeline from submission to publishing. When I created TokyoDev’s Discord, I had no previous experience running an online community. Scott joined soon after I started and began helping other members, stepping in when people behaved inappropriately, and configuring our roles and permissions and safety features. I was impressed by his initiative and asked him to become an official moderator in January 2023. Outside of his role as a moderator, Scott has always pushed TokyoDev to grow; for example, he arranged internet events such as livestreamed AMAs and interviews. He organized TokyoDev’s first meetup as a volunteer in February 2023, which taught me that there was interest in TokyoDev events outside of the internet. By the end of 2023, I had brought him on as our part-time community manager. What I saw throughout his tenure was that Scott continued to take initiative in expanding TokyoDev. Scott was one of our first external contributors, writing about his experience living with a disability in Japan. He encouraged me to expand and refine both our editing and submission processes so that we could reach out to even more contributors and platform their perspectives. The editorial team now consists of him and Rebecca Callahan, and they’ve had a hand in every article that TokyoDev has published. In other words, since 2022, Scott has shown that he cares about TokyoDev as much as I do. He is invested in what we do and why we do it, and shares my vision for what TokyoDev can become. What changes now Scott’s experiences as an engineer, a mentor, and a leader make me confident that, as our Head of Operations, he will continue to help TokyoDev mature. A big part of what has made TokyoDev successful is that I see the software developers who use our site as my peers and genuinely want to help them. Scott shares that perspective, and will join me in ensuring that TokyoDev remains true to its principles while we grow. So what changes in the short term? Nothing dramatic. I will teach Scott about managing TokyoDev as a business, and he will document our processes and products as a part of that learning. He will also look for areas of improvement while ensuring that we remain compliant with regulations and labor law. Over time, I will hand over some other responsibilities. I will continue to run TokyoDev, but TokyoDev will become less dependent on me for its day-to-day operations. Long term, I’m excited about what this will enable us to do. TokyoDev has always been about helping international software developers build successful careers in Japan, and bringing Scott on full time puts us in a much better position to continue doing that. We look forward to writing the next chapter of TokyoDev together.

2 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in