Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
49

from Civic Hax [alt+shift+b] in programming

11th Nov 2018

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Civic Hax

Chicago Parking Ticket Visualization

Intro Hi there! In this post, I want to show off a fun little web app I made for visualizing parking tickets in Chicago, but because I've spent so much time on the overall project, I figured I'd share the story that got me to this point. In many ways, this work is the foundation for my interest in public records and transparency, so it has a very special place in my heart. If you're here just to play around with a cool app, click here. Otherwise, I hope you enjoy this post and find it interesting. Also - please don't hesitate to share harsh criticism or suggestions. Enjoy! Project Beginnings Back in 2014, while on vacation, my car was towed for allegedly being in a construction zone. The cost to get the car out was quoted at around $700 through tow fees and storage fees that increase each day by $35. That's obviously a lot of money, so the night I noticed the car was towed, I began looking for ways to get the ticket thrown out in court. After some digging, I found a dataset on data.cityofchicago.org which details every street closure, including closures for construction. As luck would have it, the address that my car was towed at had zero open construction permits, which seemed like a good reason to throw out the fines. The next day I confirmed the lack of permits by calling Chicago's permit office - they mentioned they only thing found was a canceled construction permit by Comed. That same day, I scheduled a court date for the following day. When I arrived in the court room, I was immediately pulled aside by a city employee and was told that my case was being thrown out. They explained that the ticket itself "didn't have enough information", but besides that I wasn't told much! The judge then signed a form that allowed the release of my car at zero cost. At this point, I'd imagine most folk would just walk away with some frustration, but would be overall happy and move on. For whatever reason, though, I'm too stubborn for that - I couldn't get over the fact that Chicago didn't just spend five minutes checking the ticket before I appeared in court. And what bothered me more was that there isn't some sort of sorts automated check. Surely Chicago has some process in place within its $200 million ticketing system to make sure it's not giving out invalid tickets... Spoiler: it doesn't. The more I thought about it, the more I started wondering why Chicago's court system only favors those who have the time and resources to go to court and started wondering if it was possible to automate these checks myself using historical data. The hunch started based on what I'd seen from the recent "Open Gov" movement in to start making data open to the public by default. When I first started, I mostly just worked with some already publicly available datasets previously released by several news organizations. WBEZ in particular released a fairly large dataset of towing data that, while it was a compelling dataset, it unfortunately ultimately wasn't useful for my little project. So instead, I started working on a similar goal to programatically find invalid tickets in Chicago. That work and its research led to my very first FOIA request, where after some trial and error, received the records for 14m parking tickets. Trial Development and Trial Geocoding With the data at hand, I started throwing many, many unix one liners and gnuplot at the problem. This method worked well enough for small one-off checks, but for anything else, it's a complete mess of sed and awk statements. After unix-by-default I moved on to Python, where I made a small batch of surprisingly powerful, but scraggly scripts. One script, for example, would check to see whether a residential parking permit ticket was valid by comparing a ticket's address against a set of streets and its ranges found on data.cityofchicago.org. The script ended up finding a lot of residential permit tickets given outside of the sign's listed boundaries. Sadly though, none of the findings were relevant, since a ticket is given based on the physical location of the sign, not what the sign's location data represents. These sorts or problems ended up putting a halt to these sorts of scripts. With the next development iteration, I switched to visualizing parking tickets, where I essentially continued until today. Problem was, the original data had no lat/lng and had to be geocoded.. So, back when I first started doing this, there weren't that many geocoding services that were reasonably priced. Google was certainly possible, but its licensing and usage limits made Google a total no-go. The other two options I found were from Equifax (expensive) and the US Post Office (also expensive). So I went down a several hundred hour rabbit hole of attempting to geocode all addresses myself - entirely under the philosophy that all tickets had a story to tell, and anything lower than 90% geocoding was inexecusable. The first attempt at geocoding was decent, but only matched about 30% of tickets. It worked by tokenizing addresses using a wonderful python library called usaddress, then comparing the "string distance" of a street address against a known set of address already paired with lat/long. It got me to a point where I could make my first map visualization using some simple plotting: Jupyter Notebook Still, with only 30% geocoded, I felt that I could do better through many different methods that continued to use string distance - some implementations much fancier than others. Other methods, ahem, used lots of sed and vim, which I'm still not proud of. There were some attempts where I got close to 90% of addresses geocoded, though I ended up throwing each one out in fear that the lack of validation would bite me in the future. Spoiler: it did. Eventually, my life was made much easier thanks to the folks at SmartyStreets, who set me up an account with unlimited geocoding requests. By itself, SmartyStreets was able to geocode around 50% of the addresses, but when combined with some of my string-distance autocorrection, that number shot up to 70%. It wasn't perfect, but it was "good enough" to move on, and didn't really require me to trust my own set corrections and the anxiety that comes with that. Another New Dataset Fast forward about six months - I hadn't gotten much further on visualization, but instead started focusing on my first blog post. A bit around that post, ProPublica published some excellent work on parking tickets and ended up releasing its own tickets datasets to the public, one of which was geocoded. I'm happy to say I had a small part in helping out with, and then started using it for my own work in the hopes of using a common dataset between groups. More Geocoding Fun A high level of ProPublica's geocoding process is described in pretty good detail here, so if you're interested in how the geocoding was done, you should check that out. In brief, the dataset's addresses were geocoded using Geocodio. The geocoding process they used was dramatically simplified by replacing each address's last two digits with zeroes. It worked surprisingly well, and brought the percent of geocoded tickets to 99%, but this method had the unfortunate side effect of reducing mapping accuracy. So, because I have more opinions than I care to admit about geocoding, I had to look into how well the geocoding was done. Geocoding Strangeness After a few checks against the geocoded results, I noticed that a large portion of of the addresses wasn't geocoded correctly - often in strange ways. One example comes from the tendency of geocoders to favor one direction over the other for some streets. As can be seen below, the geocoding done on Michigan and Wells is completely demolished. What’s particularly notable here is that Michigan Ave has zero instances of “S Michigan” swapped to “N Michigan”. N Wells S Wells Total Total Tickets (pre-geocoding) 280,160 127,064 407,224 Unique Addresses (pre-geocoding) 3,254 3,190 6,444 Total Tickets (post-geocoding) 342,020 37,611 379,631 Unique Addresses (post-geocoding) 40 57 97 Direction is swapped 246 54,309 54,555 N Michigan S Michigan Total Total Tickets (pre-geocoding) 102,225 392,391 494,616 Unique Addresses (pre-geocoding) 2,916 16,585 19,501 Total Tickets (post-geocoding) 15,760 509,485 512,401 Unique Addresses (post-geocoding) 136 144 280 Direction is swapped 102,225 0 102,225 Interestingly, the total ticket count for Wells drops by about 30k after being geocoding. This turned out to be from Geocodio strangely renaming ‘200 S Wells’ to ‘200 W Hill St’. Similar also happens with numbered addresses similar to “83rd St”, where a steet name is often renamed to "100th". 100th street's ticket count appears to have six times as many tickets as it actually does because of this. These are just a few examples, but it's probably safe to say it's tip of the iceberg. My gut tells me that the address simplification had a major effect here. I think a simple solution of replacing the last two digits with 01 if odd, and 02 if even - instead of 0s - would do the trick. Quick Geocoding Story A few years ago, I approached the (ex) Chicago Chief of Open Data and asked if he, or someone in Chicago, was interested in a copy of geocoded addresses I’d worked on, since they don’t actually have anything geocoded (heh). He politely declined, and mentioned that since Chicago has its own geocoder, my geocoded addresses weren’t useful. I then asked if Chicago could run the parking ticket addresses through their geocoder – something that would then be requestable through FOIA. His response was, “No, because that would require a for loop”. Hmmmmm. Visualization to Validate Geocoding In one of my stranger attempts to validate the geocoding results, I wanted to see whether it was possible to use the distance a ticketer traveled to check if a particular geocode was botched or not. It ended up being less useful I'd hoped, but out of it, I made some pretty interesting looking visualizations: Left: Ticketer paths every 15 minutes with 24 hours decay (Animated). Right: All ticketer paths for a full month. This validation code, while it wasn't even remotely useful, ended up being the foundation for what I'm calling my final personal attempt to visualize Chicago's parking tickets. Final Version And with that, here is the final version of the parking ticket visualization app. Click the image below to navigate to the app. Bokeh Frontend The frontend for the app uses a python framework named Bokeh. It's an extremely powerful interactive visualization framework that dramatically reduces how much code it takes to make both simple and complex interfaces. It's not without its (many) faults, but in python land, I personally consider it to be the best available. That said, I wish the Bokeh devs would invest a lot more time improving the serverside documentation. There's just way too much guesswork, buggy behavior, and special one-offs required to match Bokeh's non-server functionality. Flask/Pandas Backend The original backend was PostgreSQL along with PostGIS (for GeoJSON creation) and TimeScaleDB (for timeseries charting). Sadly, PostgreSQL started struggling with larger queries, combined with memory exhausting itself quicker than I'd hoped. In the end, I ended up writing a Flask service that runs in PyPy with Pandas as an in-memory data store. This ended up working surprisingly well in both speed and modularity. That said, there’s still a lot more room for speed improvement - especially in caching! If I ever revisit the backend, I'll try giving PostgreSQL another chance, but throw a bunch more optimization its way. Please feel free to check out the application code. Interesting Findings To wrap this post up, I wanted to share some of the things I've found while playing around with the app. Everything below comes from screenshots from the app, so pardon any loss of information. Snow Route Tickets Outside 3AM-7AM If you park your car in a "snow route" between 3AM-7AM, you will get a ticket. Interestingly, between the hours of 8am to 11pm, over 1,000 tickets have been given over the past 15 years - mostly by CPD. Of these, only 6% have been contested – 6 of which the car owner was still liable for some reason. The other 94% should have automatically been thrown out, but Chicago has no systematic way of doing this. Shame. Highest value line = CPD's tickets. Expired Meter Within Central Business District – Outside of CBD. Downtown Chicago has an area that's officially described as the “Central Business District”. Within this area, tickets for expired meters end up costing $65 instead of the normal $50. In the past 13 years, over 52k of these tickets have been given outside of the Central Business District. From those 52k, only 5k tickets were taken to court, and 70% were thrown out. Overall, Chicago made around $725k off the $15 difference. Another instance of where Chicago could have systematically thrown out many tickets, but didn't. Downtown removed to highlight non-central business district areas. City sticker expiration tickets.. in the middle of July? Up until 2015, Chicago's city stickers expired at the end of June. A massive spike of tickets is bound to happen, but what's strange is that the spike happens in the middle of July, not the start of it. At the onset of each spikes, Chicago is making somewhere between $500k and $1m. This is a $200 ticket that has a nonstop effect on Chicago's poorer citizens. For more on this, check out this great ProPublica article. (The two different colors here come from the fact that Chicago changed the ticket description sometime in 2012.) Bike Lane Tickets With the increasing prevalence of bike lanes and the danger from cars parked in them, I was expecting to see a lot of tickets for parking in a bike lane. Unfortunately, that’s not the case. For example, in 2017, only around 3,000 tickets for parking in a bike lane were given. For more on this, click here. Wards Have different Street Cleaning times I found some oddities while searching for street cleaning tickets given at odd hours. Wards 48, 49 and 50 seem to give out parking tickets much later than the rest of Chicago. Then, starting at 6am, Ward 40 starts giving out tickets earlier than the rest of Chicago. Posted signs probably make it relatively clear that there’s going to be street cleaning, but it’s also probably fair to say that this is understandably confusing. Hopefully there’s a good reason these wards decided to be special. What’s Next? That's hard to say. There's a ton of work that needs to be done to address the systemic problems with Chicago's parking, but there just aren't enough people working on the problem. If you're interested in helping out, you should definitely download the data and start playing around with it! Also, if you're in Chicago, you should also check out ChiHackNight on Tuesday nights. Just make sure to poke some of us parking ticket nerds beforehand in #tickets on CHiHackNight's slack. That said, a lot of these issues should really be looked at by Chicago in more depth. The system used to manage parking ticket information - CANVAS - has cost Chicago around $200,000,000, and yet it's very clear that nobody from Chicago is interested in diving into the data. In fact, based on an old FOIA request, Chicago has only done a single spreadsheet's worth of analysis with its parking ticket data. I hope that some day Chicago starts automating some ticket validation workflows, but it will probably take a lot of public advocacy to compel them to do so. Unrelated Notes Recently I filed suit against Chicago after a FOIA request for the columns and table names of CANVAS was was rejected with a network security exemption. If successful, I hope that the information can be used to submit future FOIA requests using the database schema as the foundation. I'm extremely happy with the argument I made, and I plan on writing about it as the case comes to a close. Anywho, if you like this post and want to see more like it, please consider donating to my non-profit, Free Our Info, NFP which recently became a 501(c)(3). Much of the FOIA work from this blog (much of which is unwritten) is being filtered over to the NFP, so donations will allow similar work to continue. A majority of the proceedings will go directly into paying for FOIA litigation. So, the more donations I receive, the more I can hold government agencies accountable! Hope you enjoyed. Tags: tickets, foia, chicago, bokeh, visualization

27th Mar 2019 • 50 votes

More in programming

Bikepacking across Kyushu

Five days and 540km on a bike across Kyushu, without a laptop. Notes on the peace of riding all day, convenience store dinners, and Japan's emptying countryside.

6 hours ago • 1 votes
Modifications to add a booty to Loops by Kat's Ditto amigurumi

I found a wonderful free crochet pattern for the Pokémon Ditto from Loops by Kat, but I decided I wanted to make a version with more than 3 "leg" blobs like the original pattern had. Specifically, I wanted to add two more smaller blobs in the back to help stabilize Ditto when they're sitting and also add a butt to our favorite transforming blob. Hazel Knits Cadence, a worsted weight superwash merino, in Terrywinkle that I had in my stash (I used about a third of the skein/36 grams/66 yards/60 meters), and I crocheted with a 2.5 mm hook. I make very tightly stitched amigurumi - do whatever is a good fit for you and your yarn! I used two 6 mm round safety eyes, and I embroidered the mouth with scraps of Hazelknits Cadence in Coral, Bloom, and Signature Black. The above photo is more true to color than some of the ones I took where Ditto looks more pink. I made Ditto's 2 additional booty blobs intentionally off-center by embracing the natural lean of crochet stitches - this gives a more animated or "in motion" feel to Ditto. As an added bonus, this simplifies the pattern as everyone's tension will be a little different, and thus everyone's off-centering a little delightfully different, too, but centering it instead wouldn't be the same for everyone! Here's all the information to modify Kat's crochet pattern to create a 5-blobbed Ditto. I use US crochet terminology. This pattern modification is also available as a pdf. Follow Kat's pattern through round 13. At round 14, it's time to start modifying to add the space and shaping for the additional blobs. There are also 3 more rows in this section to better fit the new 5-blob proportions. Modified increases in body, beginning at round 14 (with additional rounds, too): Round 14: 3 sc, 1 inc, 46 sc, 1 inc, 3 sc (56) Round 15: 56 sc (56) Round 16: 2 sc, 1 inc, 2 sc, 1 inc, 44 sc, 1 inc, 2 sc, 1 inc, 2 sc (60) Round 17: 60 sc (60) Round 18: inc, 4 sc, inc, 48 sc, inc, 5 sc (63) Round 19: 63 sc (63) Round 20: dec (through front loop of first stitch and both loops of second stitch), inc, 9 sc, inc, 38 sc, inc, 11 sc (65) At this point, the original pattern suggests placing the safety eyes, but I think it makes more sense to do this after you've made the "butt" and divided the stitches the 3 main "front" "leg" blobs from the original pattern. You'll still have plenty of access to place them then. Roadmap for doing 5 "leg" blobs instead of 3 To make 5 blobs, you use some of the main body's stitches and some created stitches for each. Created stitches are a combination of chains, pickups on the back of chains, pickups in the connections between the chains and the main body, and pickups in the connections between the chains. The "butt" "leg"/mounds are created before the 3 original "legs," and the left "butt" "leg"/mound is made before the right one. 1. Left "butt" "leg"/mound: This is the left "butt" cheek when viewed from the bottom. (While this is really the right butt cheek from Ditto's perspective, I'm calling it left because we're crocheting while viewing from below so it's backwards.) The start of round is the same as the start of round from making the body of the blob, so you can continue as usual. Round 1: 13 sc, chain 11 (24) Round 2: connect chain at previous start of round (skipping 52 stitches) then 24 sc (24) Round 3: (6 sc, dec) x 3 (21) Round 4: (2 sc, dec, 3 sc) x 3 (18) Round 5: (sc, dec) x 6 (12) Round 6: 6 dec (6) Finish off by looping through front loops of all 6 stitches and pull tight, etc. 2. Right "butt" "leg"/mound: This is the right "butt" cheek when viewed from the bottom. The start of round for this "butt" "leg" is 12 stitches before the body/left "butt" leg start of round. This is the second stitch in an increase from the last round of the body. Round 1: 12 sc, 1 dec in the corner of body connected to first "butt" "leg" (pictured below), 4 sc in the bottom loops in the existing chain, chain 7 (24) Round 2: connect chain at start of round for this "butt" "leg" (skipping other blob's additional 7 chains and 40 stitches in the main body) then 24 sc (24) Round 3: (6 sc, dec) x 3 (21) Round 4: (2 sc, dec, 3 sc) x 3 (18) Round 5: (sc, dec) x 6 (12) Round 6: 6 dec (6) Finish off by looping through front loops of all 6 stitches and pull tight, etc. Getting back to the original pattern for the 3 main "legs": After both "butt" "legs"/mounds are done, there are 40 stitches on the original body, 7 chain stitches that can be picked up from each "butt" leg" that haven't been picked up in a second location, which totals the original 54 stitches - this sets you up for the 3 blobs as described in the original pattern. Between each "butt" "leg" and the main body and between the two "butt" "legs" themselves are an additional 3 connections that are also picked up and decreased appropriately to keep the seams between "legs" tight. As a reminder, here's the road map, which shows how to divide up these stitches for the Ditto's main three "legs:" You might find it helpful to visualize this division by temporarily stringing up some scrap yarn to preview the divisions: If you'd like, place stitch markers like I did to delineate the division: Now that you've visualized how you're dividing the main "leg" blobs, it's a good time to place safety eyes if you're using safety eyes for eyes. I placed mine between rows 7 and 8 counted down from the small head blob and left 8 stitches between the eyes. Now you can proceed to make the main legs in the same order as the original pattern. 3. Main "leg" blob 1: This is the left "leg" blob labeled 1 in the original pattern. To start, connect your yarn into the first stitch in the main body after where the main body meets the left "butt" mound. Round 1: 18 sc, chain 9, 1 sc in the 2nd bottom loop in the chain right of the body/left "butt" mound connection, dec through the leftmost bottom loop in the chain on the left "butt" mound and the loop on top of the meeting of the main body/left "butt" mound (pictured below). (29) In the start of the original pattern "leg" 1, here are the two loops in the decrease through the left most chain loop on the left "butt" mound and the loop on top of the meeting of the main body/left "butt" mound. Follow the original pattern's Rounds 19 to 23 and finish off blob "leg" 1 as written. 4. Main "leg" blob 2: This is the right "leg" blob labeled 2 in the original pattern. To start, connect your yarn into 15th stitch from where the main body meets the right "butt" mound - there should be 7 skipped stitches along the front of the Ditto between this start of round and the left "leg" blob's last stitch on the front side of Ditto before the chaining. Round 1: 15 sc, dec through the loop on top of the meeting of the main body/right "butt" mound and the rightmost bottom loop in the chain on the right "butt" mound (pictured below), 4 sc in the bottom loops in the chain on the right "butt" mound, chain 9 (29) In the start of the original pattern "leg" 2, here are the two loops in the decrease through the loop on top of the meeting of the main body/right "butt" mound and the rightmost chain loop on the right "butt" mound. Connect chain at start of round for this blob "leg" at the beginning of round 2's first sc and continue the original pattern's Rounds 2 to 6 for blob "leg" 2 as written. As in the original pattern, now is a good time to stuff your Ditto most of the way! 5. Main "leg" blob 3: This is the center "leg" blob labeled 3 in the original pattern. Round 1: Start in the corner next to left "leg" 1 on the front side (pictured below). 37 sc into the following stitches you pick up: 7 stitches along the main body 1 stitch in the corner of the main body and right "leg" 2 9 stitches in the bottom loops of the chain along right "leg" 2 1 stitch in the corner of right "leg" 2 and the right "butt" mound 2 stitches in the bottom loops of the chain along the right "butt" mound 1 stitch in the corner between the two "butt" mounds 5 stitches in the bottom loops of the chain along the left "butt" mound 1 stitch in the corner of the left "butt" mound and left "leg" 1 9 stitches in the bottom loops of the chain along left "leg" 1 1 stitch in the corner of left "leg" 1 and the main body (37) My lime green running stitch marker indicates the start of round pickup location for making the main blob "leg" 3. To compensate for the extra 5 stitches from those corners, I decreased in the second round at the corresponding locations instead of just single crocheting the whole way around to ensure the end of my round 2 of this section had the specified 32 stitches: Round 2: 6 sc, dec, 8 sc, dec, sc, dec, 4 sc, dec, 8 sc, dec (32) After round 2, I continued with the rest of the original pattern to finish the middle blob "leg" 3. Make the two arms according to the original pattern. I attached them less symmetrically than the original suggested. Finally, embroider a mouth for Ditto. You can either do this in black to match the safety eyes or create a two-toned mouth in red and pink to make a very happy looking Ditto like I did. FYI, this is also documented as a project on Ravelry (though it's a lot harder to follow since photos couldn't be inline).

13 hours ago • 1 votes
Tyding up my side projects

Being focused is about doing less things, and giving more space to the stuff that matters. I am spread out on too many projects and I can’t give enough attention to all of them. It’s time to clearly sort out which one I will keep maintaining and which I should drop entirely. Chuttersnap Mikochi Mikochi is a self-hostable remote file manager I’ve started in 2023. It’s my most successful open source project. I personally use it every day, and thousands of people have adopted it and started using it on their own servers. This year the project has been receiving more and more attention. Maybe it just reached a critical mass of users. In practice, that means a bit more maintenance work, since people have been pointing out bugs and suggesting new features. At the same time, working on Mikochi is fairly enjoyable. It’s a small, self-contained project made with technologies I enjoy working with. I’ve got no complaints about spending some time working on it. Verdict: Keep maintaining Mikochi and actively continue improving it. Itako Itako is an investment portfolio tracking SaaS. I built it for the most part between 2023 and 2025. In terms of scope, it’s the largest project I’ve done entirely by myself. Is it a success? Not really. There are definitely a lot of people who tried Itako, and some who’ve kept using it a bit. But it’s not engaging enough for anyone to consider getting the premium plan. And honestly I don’t find myself using it that much. My view of investing and portfolio management has significantly shifted since I started this project. While I used to think that more sophisticated analysis would lead to better returns, I now realize that the only strategies that consistently make money are honestly stupidly simple. Or maybe it’s a U curve: on one end you’ve got a monkey throwing darts, on the other end you’ve got Jane Street, and everyone in-between fails to beat the market. Anyway, the biggest issue with Itako is how dull working on it is. The interesting parts of the projects are working on statistics and data visualization. But in practice, 90% of the time is spent tediously fixing data, understanding why some German stocks have an exchange labelled XETRA instead of XETR, why some Belgian stocks still have quotes while Euronext shows them as delisted… Verdict: Shut down Itako today. Milou Milou is static site generator for press kits. It started as a fork of presskit.html which was completely abandoned at the time. It’s a very useful tool. At Tsukumogami, it’s part of our Cloudflare pipeline which builds our website in a few minutes whenever we need to change something. It seems like a few other game studios have picked it up, although I rarely get any feedback about it. I’ve not spent too much time working on it in 2026. I think it could be interesting to come back to it and add a feature or two. Better support for multiple languages could be an idea. At least, I need to make sure we don’t fall into the good old Node.js dependency rot. Verdict: Spend some time on Milou. Luluka Earlier this year, I created Luluka. It’s a CLI that allows previewing the output of Ebitengine shaders. I do still think Ebitengine is a great engine and Go is better suited for game development than most people realize. However, after quite a long period of experimentation around it, we’ve decided to remain a Godot studio. On top of having quite a large sunk cost in infrastructure surrounding Godot, the tooling and possibility of doing 3D games still makes it a more attractive engine for our use cases. And I’ve been doing a lot of 3D work lately, so better put those hours to use. Verdict: Archive the project. The rest I’ve had a lot of good ideas that stayed as almost empty repositories in my GitHub. For example, Hakumei was a terminal companion to Mikochi. I made a few commits on this project and never found the time to work on it again. I have to admit I’m probably never going to have the time to work on all those repositories. Time to archive them all!

4 days ago • 1 votes
First Strike

Internalize this graph, we post it on this blog a lot. If trends continue, China will have 10x the electricity of the US in 10 years. Whenever you think about geopolitical strategy, always keep it in mind, because whatever your plans are must consider this. Here’s a strategy that would make sense for America long term. Sell as many AI chips into China as possible, use the money to purchase solar panel and battery factories, and build out electricity capacity as quickly as possible. Fundamentally, compute is a function of electricity. That’s the sort of long term thinking that gives America a chance. With robotic labor, immigrants like Jensen and Elon, a business friendly culture, and an incredible chunk of land, it’s not so clear who is ahead in 50 years. Regardless, I do know this strategy would lead to massive prosperity for the citizens of both countries. However, that’s not close to what we see happening. We see a lot of short term zero sum thinking in action. It’s very scary what the US “AI race” mentality leads to. When exactly does the race end? Whatever capabilities you have today will eventually be matched, unless… …you believe that you can use ASI in a decapitating first strike. I’m worried that this is what these people see as the only way out. It probably doesn’t work, can you outthink steel and drone production? I don’t see how, see my critiques here. But I’m not totally sure, I’m not a superintelligence. And if it does work, that may be the end. My likelihood of imminent WWIII went up a lot with the nationalization of NVIDIA.

5 days ago • 1 votes
Not everybody is an AI expert - and that’s fine

This post was previously published through my newsletter on November 9, 2026. From time to time, I will republish newsletter issues on my blog here if I think people (and search engines) might benefit from it. If you want to read everything I’ve written once it is posted, I recommend signing up for the newsletter. If you’re doing anything to stay on top of what is happening in the world of software testing and test automation, like browsing or even posting on LinkedIn, which I stopped doing for the foreseeable future, or reading blog posts - my preferred way of keeping up these days, and I highly recommend Dawid Dylowicz’s excellent Software Testing Weekly aggregator mailing list if you prefer longer-form reading, too you’d be thinking that AI has consumed our entire field, and that everybody (or at least those who post or write in public) is an AI expert these days. The reality, though, is far from that. Yes, some people use a lot of AI, even when I would only call a small fraction of them ‘experts’. I definitely do not classify myself as an ‘AI expert’, by the way. A much larger group uses AI on a less frequent basis, and they’re getting some value out of it. Sometimes more, sometimes less. That’s the group I’m probably in. And then there’s a group of people who haven’t really been using a lot of AI at all so far, beyond maybe the odd prompt to ChatGPT or Claude. I don’t have the actual numbers, but I suspect that this group is a lot larger than you would think from browsing your feed or surfing the blogosphere. You just don’t see them, because they’re typically not the people who do a lot of work in public. A few weeks ago, I started working with a small group of testers to teach them how to use AI in their daily work. This work covers both test automation as well as broader software testing tasks, as some of them contribute to automation on a daily basis, but not all of them do. I’m working with them one day per week, on average, for three months, and we’re making small steps towards making AI a more prominent tool in their tool belt. So, what are we working on right now? I’m pulling ideas from my ‘Valuable Feedback, Fast with AI’ course, and so far, we’ve spent some time setting up Claude, going over some important concepts such as context windows, models, tokens, and (of course!) the probabilistic nature of LLMs, and what that means for the results they provide. We have gone through several initial conversations, using Claude to learn more about and review specific user stories, which was an excellent segue into using talking about MCPs and setting up a connection to both Jira and Confluence using the Atlassian MCP. Last week, we also started experimenting with writing straightforward skills to make it easier for Claude to repeat certain actions in a structured way. These actions ranged from reviewing stories and acceptance criteria to refactoring small chunks of code in their (Robot Framework-based) test automation suite. What I loved about this is the avalanche of ideas that popped up in their heads when they saw how easy it was to use Claude. We made a short to-do list of ideas to try out for the next couple of weeks, and I’m looking forward to seeing them getting more confident around using Claude. One thing I have noticed is that their level of comfort around using AI varies quite a bit. Some of them are naturally quite cautious, which means they’ll ask ‘can we really do this?’ and ‘should we really do this?’ frequently before even submitting a single prompt. This is a good habit, I think, but we’ll have to do some work to get them more comfortable with using LLMs, without losing too much of that caution. Others in the group are the opposite: I need to hold them back a little, because otherwise they’ll create a beautiful mess within days, simply because they see so many opportunities to use Claude in their work. That, too, isn’t inherently bad, but I do want to teach them not just how to get the most out of an LLM, but also (more so) how to use AI prudently. Over the next couple of months, we’ll dive deeper into writing better prompts, creating and structuring skills, better context management, and creating and using agents, and probably a few more things, too. I’m looking forward to seeing these people grow more comfortable around using AI in their daily work, and figuring out where AI genuinely adds value and where it doesn’t in their own context. Is there a deeper message in this newsletter? Maybe not, and I’m just happy to tell you about this project, and how it is a great example of a what I call ‘corporate mentoring’, a very efficient way of teaching a small group of engineers new skills, within their own context. But maybe it’s also to show you that, despite what you might see on the Internet, especially those places featuring a lot of self-congratulation (I’m looking at you, LinkedIn), not everybody out there is an AI expert. At least not yet. So, if you feel like you’re ‘behind’ simply because you’re not running at least 17 agents continuously, creating 5400 merge requests per day, know that you’re not alone. Quite far from it, I would say.

5 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in