I wrote a post for the Tailscale blog about a long-running series of corruption incidents, and how they eventually led us to find an SQLite bug that predates my entire programming career. I’m incredibly proud of this, both the work and the blog post. Before Tailscale, I was coming from smaller teams where I didn’t get to tackle problems of this scale or complexity. This was exactly the sort of tricky, deep technical challenge I wanted to be part of (though I’d rather it hadn’t been quite so stressful)! I’m glad I got to play a small part in these incidents, and I learnt so much from the more experienced engineers I worked with. I never want to hear the words “SQLite corruption” again, but if I do, I’d want to have Tailscalars at my side. Writing the blog post has a blast, too. The piece transformed from a rough draft into a solid, engaging piece of writing, thanks to thoughtful feedback from many people at Tailscale. Most of my writing is self-edited, and it’s always a pleasure to work with a dedicated editor. Please check out the blog post if you haven’t read it already – I think it’s a fascinating technical story, and one readers of this site are bound to enjoy. [If the formatting of this post looks odd in your feed reader, visit the original article]

Yesterday at work, a customer spotted a typo in our UI: “you can use the use the Tailscale CLI”. After the typo was fixed, I wanted to find other cases of accidentally repeated words or phrases. I used two regular expressions to search every codebase for unnecessary repetition. The first regex finds repeated words: \b([A-Za-z]+) \1\b Backfill product data from from Stripe Learn more about about inviting users Argument must be be one of host name, IP set name, IP prefix, or IP There’s a capturing group for a single word made up of letters ([A-Za-z]+), a space, then a backreference to the group. That expression is surrounded by word boundary assertions \b, which check that I’m at the start/end of a word – this avoids finding repeated character sequyences that within longer words, like “with the reason”. The second regex finds repeated phrases: \b([A-Za-z]+ [A-Za-z]+) \1\b Follow the steps in the in the "How to" section Log in to in to your account To configure federated identities federated identities using the Go SDK I’ve changed the capturing group, so now it looks for two words separated by a space. Sometimes repetition is useful, like when I really really went to emphasise a point, but often it’s just a typo. Cleaning up these mistakes has been a fun Friday cleanup task. [If the formatting of this post looks odd in your feed reader, visit the original article]

A month ago, I wrote about my Playwright fixture for testing static websites in a browser. I’ve been copying that fixture from project-to-project, but recently I decided to add it to chives, the utility library I use for all my static websites (or tiny archives). One of my rules for chives is that everything in it has to be tested – but how do you test a pytest fixture? Test code is just code, and it isn’t immune to bugs. Who tests the tests? Enter Pytester, a tool designed for testing pytest plugins. Pytester allows you to run isolated test suites, make assertions about the outcomes, and verify the behaviour of custom fixtures. In your top-level test suite, you always want everything to be passing, but with Pytester you can write a mixture of passing and failing tests, and check the results are what you expect. Pytester is disabled by default, so you first enable it in your top-level conftest.py file (the pytest configuration file where you configure plugins and fixtures): # conftest.py pytest_plugins = ["pytester"] Here’s an example of using Pytester where we create a test suite with two tests and check that one passes, one fails: from pytest import Pytester def test_with_pytester(pytester: Pytester): """ Run an isolated test suite with pytester. """ # Make a temporary pytest test file pytester.makepyfile( """ def test_arithmetic(): assert 2 + 2 == 4 def test_list_inclusion(): assert "yellow" in ["red", "green", "blue"] """ ) # Run the isolated test suite with pytest result = pytester.runpytest() # Check that one test passed, one failed result.assert_outcomes(passed=1, failed=1) I can imagine creating something similar with some complicated collection of nested functions, exec() and pytest.raises, but using Pytester is a cleaner interface than what I’d build. Under the hood, Pytester creates a temporary directory, writes specified files into it, then runs a fresh pytest subprocess against it. It has helper functions for writing files, including Python files (makepyfile), a conftest.py file (makeconftest), and plain text files (maketxtfile). When we’re testing a fixture, we can create a conftest.py file that imports that fixture, then reference it in the tests. Here’s a more complicated example, where we import one of my Playwright fixtures in my conftest.py, write an HTML file into the temporary directory, then use them both in the test: from pytest import Pytester def test_browser_fixture(pytester: Pytester): """ Try testing the browser fixture with pytester. """ # Make a conftest.py file pytester.makeconftest(""" from chives.browser_fixtures import browser """) # Make an HTML file (pytester.path / "greeting.html").write_text(""" <p>Hello world!</p> """) # Make a temporary pytest test file pytester.makepyfile( """ from chives.browser_fixtures import file_uri from playwright.sync_api import Browser, expect def test_browser_fixture(browser: Browser) -> None: uri = file_uri("greeting.html") p = browser.new_page() p.goto(uri) expect(p.get_by_text("Hello world!")).to_be_visible() """ ) # Run the isolated test suite with pytest result = pytester.runpytest() # Check that one test passed result.assert_outcomes(passed=1) This pattern is sufficient for many fixtures, but it doesn’t work for Playwright – if you run this test, the isolated test suite gives an error rather than a passing test. Playwright needs you to install a web browser to work (for example, playwright install webkit), and Pytester runs in a sufficiently isolated environment that Playwright can’t find the browsers you already have installed. We could run the install command inside the temporary directory, but that would be slow and inefficient – it would be better if we could tell Playwright to look for the already-installed browsers elsewhere. If we set the PLAYWRIGHT_BROWSERS_PATH environment variable inside our isolated test suite, Playwright will look there for browsers. First, we need to work out where browsers are installed – we could hard-code the location, or we could inspect the executable_path property property on a browser: from pathlib import Path from playwright.sync_api import sync_playwright import pytest @pytest.fixture(scope="session") def playwright_browsers_path() -> str: """ Return the cache directory where Playwright browsers are installed. """ with sync_playwright() as p: # In my local builds, this returns a path like: # # ~/Library/Caches/ms-playwright/webkit-2272/pw_run.sh # # Unwrap two levels to get to the `ms-playwright` folder. return str(Path(p.webkit.executable_path).parent.parent) Then we need to set this as an environment variable inside the Pytester test suite. I couldn’t find an easy way to set an environment variable; the best approach I came up with was to modify os.environ inside the conftest.py file. (Perhaps we could access the MonkeyPatch object and set more environment variables, but using private attributes is icky.) Here’s how the new test starts: def test_browser_fixture(pytester: Pytester, playwright_browsers_path: str): """ Test the browser fixture with pytester. """ # Make a conftest.py file pytester.makeconftest(f""" from chives.browser_fixtures import browser import os os.environ["PLAYWRIGHT_BROWSERS_PATH"] = {playwright_browsers_path!r} """) ... and now the overall test passes. Here’s the complete code for the new test: test_browser_fixture.py from pathlib import Path from playwright.sync_api import sync_playwright import pytest from pytest import Pytester @pytest.fixture(scope="session") def playwright_browsers_path() -> str: """ Return the cache directory where Playwright browsers are installed. """ with sync_playwright() as p: # In my local builds, this returns a path like: # # ~/Library/Caches/ms-playwright/webkit-2272/pw_run.sh # # Unwrap two levels to get to the `ms-playwright` folder. return str(Path(p.webkit.executable_path).parent.parent) def test_browser_fixture(pytester: Pytester, playwright_browsers_path: str): """ Test the browser fixture with pytester. """ # Make a conftest.py file pytester.makeconftest(f""" from chives.browser_fixtures import browser import os os.environ["PLAYWRIGHT_BROWSERS_PATH"] = {playwright_browsers_path!r} """) # Make an HTML file (pytester.path / "greeting.html").write_text(""" <p>Hello world!</p> """) # Make a temporary pytest test file pytester.makepyfile( """ from chives.browser_fixtures import file_uri from playwright.sync_api import Browser, expect def test_browser_fixture(browser: Browser) -> None: uri = file_uri("greeting.html") p = browser.new_page() p.goto(uri) expect(p.get_by_text("Hello world!")).to_be_visible() """ ) # Run the isolated test suite with pytest result = pytester.runpytest() # Check that one test passed result.assert_outcomes(passed=1) The full test suite is more extensive, and checks that certain scenarios fail or error – will the fixtures spot the mistakes I expect them to? For example, my Page fixture is meant to load a page and fail the test if there are any console warnings or errors; does it actually fail the test correctly? I don’t expect to use Pytester very often, because it’s rare for me to write fixtures complex enough to need their own test suite – but sometimes I do, and it’s good to know how to create another layer of safety net. [If the formatting of this post looks odd in your feed reader, visit the original article]

I build a lot of static websites – including this site and all of my local media archives – and I want to test them. Most of my pages are static HTML and I can write automated tests that analyse the HTML, but for more complex sites I have JavaScript that runs in the browser and modifies the page. The only way to test that functionality is to open the page in a browser, click around, and see what happens. I could do that manually, but it quickly gets tedious. To automate this process, I’ve been using a testing framework called Playwright, which is designed for this sort of end-to-end testing. It’s a tool that allows you to programatically control a web browser, look at the contents of a page, and make assertions about what’s there. Playwright can be used to test or script any kind of web app; I’m using it for static sites because those are the only web apps I have. Playwright is available as a CLI, or there are libraries to use it with TypeScript, Python, .NET, and Java. All my other tests are written in Python, so that’s what I’m using. Writing a basic test with Playwright To set up Playwright with Python, you install the playwright library using pip or uv, then install a web browser for Playwright to control. (You can’t use Playwright with the browser you use day-to-day; you need special binaries with control hooks.) I use Safari as my main browser, and Safari is based on WebKit, so let’s install that: $ uv pip install playwright $ python3 -m playwright install webkit Then we can start writing tests. Here’s a basic test in which Playwright launches WebKit, opens example.com, and checks the text Example domain is visible on the page: from playwright.sync_api import expect, sync_playwright def test_basic_playwright() -> None: """ Run a basic test with Playwright: load a web page and check it contains the expected text. """ with sync_playwright() as p: browser = p.webkit.launch() page = browser.new_page() page.goto("https://example.com/") expect(page.get_by_text("Example domain")).to_be_visible() browser.close() For a larger app, you might run your tests with multiple browsers to check compatibility – Playwright supports lots of other browsers, including Chromium, Firefox, and Mobile Safari in emulation. I’m just testing private sites where I’m the only user, so a single browser is fine. This test passes in about half a second on my computer. That’s fine for a single test, but it would add up if I had lots of tests, each starting and stopping the browser every time. It would be nice to make that process faster, and to reduce some of the boilerplate as well. A pair of Playwright fixtures To reduce the repetition and reuse the browser instance, I have a couple of pytest fixtures to simplify things. The first is a session-scoped fixture that starts the browser at the start of the test run, and closes it when I’m done: from collections.abc import Iterator from playwright.sync_api import Browser, sync_playwright import pytest @pytest.fixture(scope="session") def browser() -> Iterator[Browser]: """ Launch an instance of WebKit to interact with in tests. """ with sync_playwright() as p: webkit = p.webkit.launch() yield webkit webkit.close() Because this is a session-scoped fixture, it only runs once per test suite – that means the browser is only started once, then the same instance is reused for all the tests. This makes a large test suite significantly faster. My other fixture is a bit more complicated – it gives you a page to interact with, and at the end of the test it checks the page didn’t have any warnings or errors. This is a strict approach, which helps me spot errors in areas I wasn’t explicitly testing. Here’s the fixture: from collections.abc import Iterator from playwright.sync_api import Browser, Page import pytest @pytest.fixture(scope="function") def page(browser: Browser) -> Iterator[Page]: """ Open a new page in the browser. If there are any errors or warnings when loading the page, the test will fail when this fixture is cleaned up. """ p = browser.new_page() # Capture anything that gets logged to the console. console_messages = [] p.on("console", lambda msg: console_messages.append(msg)) # Capture any page errors page_errors = [] p.on("pageerror", lambda err: page_errors.append(err)) yield p # Check there weren't any console errors logged to the page. console_errors = [ msg.text for msg in console_messages if msg.type == "error" or msg.type == "warning" ] assert console_errors == [] # Check there weren't any page errors assert page_errors == [] These two fixtures allow for tighter, faster tests, focusing on what the test is actually checking. Here’s the example test, rewritten to use this fixture: def test_playwright_with_fixture(page: Page) -> None: """ Run a test using my Playwright fixture: load a web page, check it contains the expected test, and check it loads without errors. """ page.goto("https://example.com/") expect(page.get_by_text("Example domain")).to_be_visible() I use the page fixture for most tests, where I want to spot any unexpected errors or warnings. If I’m testing error handling specifically, I use the browser fixture and create a new page which isn’t treated as strictly. Getting file:/// URIs for Playwright Normally Playwright is used with http: and https: URLs, but my static websites are stored as HTML files on my local disk, and I often open them with file: URLs. I could spin up a web server in my tests, but that’s extra overhead and might affect the results – there are subtle differences between how browsers handle pages opened with file: vs http:. To convert file paths to file: URLs, I use the pathname2url function from the urllib.request module. I combine this with os.path.abspath to get a full URL I can pass to Playwright: >>> from os.path import abspath >>> from urllib.request import pathname2url >>> path = "index.html" >>> pathname2url(abspath(path), add_scheme=True) 'file:///Users/alexwlchan/repos/alexwlchan.net/index.html' Assertions in Playwright Playwright has a different set of assertion helpers to regular Python tests, and it takes some getting used to – I still have to consult the documentation when I write new tests. Here are examples of assertions I’ve written using Playwright: Testing that a redirect is working: resp = page.goto("https://alexwlchan.net/projects/chives/files/doesnotexist.txt") assert resp is not None assert resp.status == 200 assert resp.url == "https://alexwlchan.net/projects/chives/files/?missing=doesnotexist.txt" Test that text does or does not appear on a page: from playwright.sync_api import expect page.goto("https://www.example.com") expect(page.get_by_text("Example Domain")).to_be_visible() expect(page.get_by_text("Alex Chan")).not_to_be_visible() or: assert "Example Domain" in page.content() assert "Alex Chan" not in page.content() Locate an element with a CSS selector, and check it does or doesn’t appear on a page: page.goto("https://www.example.com") expect(page.locator("h1")).to_be_visible() expect(page.locator("h2.title")).not_to_be_visible() Locate an element, and make assertions about its attributes: page.goto("https://www.example.com") href = page.locator("a").first.get_attribute("href") assert href == "https://iana.org/domains/example" Locate an element, and make assertions about the text it contains: page.goto("https://www.example.com") assert page.locator("a").inner_text() == "Learn more" Check that an element with particular inner text is visible on the page: page.goto("https://www.example.com/") expect(page.locator('//h1[text()="Example Domain"]')).to_be_visible() Locate an element immediately following a different element. I’ve used this a couple of times when I have tables or definition lists with a label in one element, and a value in another: dt_locator = page.locator('//dt[text()="Profile page:"]') next_dd = dt_locator.locator("xpath=following-sibling::*") assert ( next_dd.inner_html().strip() == '<a href="https://www.flickr.com/photos/nasahqphoto/">NASA HQ PHOTO</a>' ) Check the number of matching elements on a page; for example, the length of a list: page.goto("https://alexwlchan.net/articles/") assert page.locator("#list_of_posts li").count() >= 10 Check the title of the page: page.goto("https://www.example.com/") assert page.title() == "Example Domain" Check the behaviour of the page when JavaScript is disabled: context = browser.new_context(java_script_enabled=False) page = context.new_page() expect(page.locator("noscript .error")).to_be_visible() noscript_elem = page.locator("noscript .error") assert noscript_elem.inner_text() == "You must enable JavaScript to use this page." This is just a fraction of what Playwright can do; it can be used to build far more complicated tests that walk through a web app and test multi-step user flows. I’m only using it to make assertions about snippets of JavaScript, but it’s still useful. For a long time, I told myself that my static sites were simple enough not to need testing, but that didn’t prevent bugs from slipping in, and it limited what I could build. Now I can write proper tests for my sites, I can be more confident I haven’t broken anything, I can experiment faster, and I can try more ambitious ideas. [If the formatting of this post looks odd in your feed reader, visit the original article]