Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
65

Finding near-duplicates with Jaccard similarity and MinHash

from Posts on Made of Bugs [alt+shift+b] in technology

Suppose we have a large collection of documents, and we wish you identify which documents are approximately the same as each other. For instance, we may have crawled the web over some period of time, and expect to have fetched the “same page” several times, but to see slight differences in metadata, or that we have several revisions of a page following small edits. In this post I want to explore the method of approximate deduplication via Jaccard similarity and the MinHash approximation trick.
3rd Jul 2024

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Posts on Made of Bugs

From error-handling to structured concurrency

How should we think about error-handling in concurrent programs? In single-threaded programs, we’ve mostly converged on a standard pattern, with a diverse zoo of implementations and concrete patterns. When an error occurs, it is propagated up the stack until we find a stack frame which is prepared to handle it. As we do so, we unwind the stack frames in-order, giving each frame the opportunity to clean up or destroy resources as appropriate.

23rd Mar 2026 • 1 votes
Solving Regex Crosswords with Z3

For a while now, I’ve been fascinated by Z3 and by SMT solving more broadly. While on pat leave recently, I was reminded of the existence of regular-expression crossword puzzles, and allowed myself to get nerdsniped by writing a Z3-backed solver. I expected to spend perhaps an afternoon cranking out a quick solver; I ended up getting sucked into understanding and debugging Z3 performance, and learning far more about Z3 and about SMT than I expected.

21st Oct 2025 • 1 votes
The ITTAGE indirect branch predictor

While investigating the performance of the new Python 3.14 tail-calling interpreter, I learned (via this very informative comment from Sam Gross) new (to me) piece of performance trivia: Modern CPUs mostly no longer struggle to predict the bytecode-dispatch indirect jump inside a “conventional” bytecode interpreter loop. In steady-state, assuming the bytecode itself is reasonable stable, modern CPUs achieve very high accuracy predicting the dispatch, even for “vanilla” while / switch-style interpreter loops1!

4th Jul 2025 • 1 votes
Performance of the Python 3.14 tail-call interpreter

About a month ago, the CPython project merged a new implementation strategy for their bytecode interpreter. The initial headline results were very impressive, showing a 10-15% performance improvement on average across a wide range of benchmarks across a variety of platforms. Unfortunately, as I will document in this post, these impressive performance gains turned out to be primarily due to inadvertently working around a regression in LLVM 19. When benchmarked against a better baseline (such GCC, clang-18, or LLVM 19 with certain tuning flags), the performance gain drops to 1-5% or so depending on the exact setup.

9th Mar 2025 • 62 votes
Building personal software with Claude

Earlier this month, I used Claude to port (parts of) an Emacs package into Rust, shrinking the execution time by a factor of 1000 or more (in one concrete case: from 90s to about 15ms). This is a variety of yak-shave that I do somewhat routinely, both professionally and in service of my personal computing environment. However, this time, Claude was able to execute substantially the entire project under my supervision without me writing almost-any lines of code, speeding up the project substantially compared to doing it by hand.

27th Jan 2025 • 71 votes

More in technology

Three years later

Reflections on October 7th

22 hours ago • 1 votes
It's a Gas!

If everything worked as well as the product called Evapo-Rust, the world would be a much better place. That’s just one of the many lessons learned during my recent — successful! — project to transform my old, nonfunctioning gasoline-powered generator into something much better.

yesterday • 1 votes
I deleted my favorite feature, and the image got better: Direct Binary Search (DBS) in OpenDithering

When I created OpenDithering back in June, the feature I was proudest of was the Auto-tune button. It did the slider work for me: it compared the dithered result to the original image, nudged the settings, and repeated until the image stopped improving. Over the summer it grew into a three-step process (Auto Expose, Color-tune, […] The post I deleted my favorite feature, and the image got better: Direct Binary Search (DBS) in OpenDithering appeared first on Style over Substance.

2 days ago • 1 votes
You Won’t Hear About These, Even In Myths (Atlassian Jira, Confluence (and more) Pre-Auth Arbitrary File Read CVE-2026-21589)

Welcome back to yet another episode of "security was taken seriously". Being who we are (and constantly being exposed to what we see…), we recognize we have been doomed to eternal damnation as we keep on watching security best practices crumble behind “secure by design”

2 days ago • 1 votes
Beeper MCP

Fellow Automattician Job Thomas, down in Cape Town, South Africa, has a lovely story about using Claude and the Beeper MCP to wrangle all the group chats for his daughter’s sports.

2 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in