Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
55
Data engineering is a field I would categorize as a subspecialty of software engineering. It shares the same concerns as software engineering—scalability, maintainability, and other “-ilities”—but its primary focus is on data. It’s a unique discipline because data is inherently messy, and as a result, no standard enterprise framework has emerged to dominate the space—and […]
3 months ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from Dan Quach Blog

Digital Ghosts, Wisdom, and Tennis Matchmaking

Digital Ghosts My mom recently had a free consultation from her electric company to assess replacing her propane water heater with an electric water pump heater.  She forwarded the assessment report to me, and I spent some time reviewing and researching the program. Despite living quite far away, I have been surprised by how much […]

4 weeks ago 16 votes
Europe

These are a mix of stories from Norway, Croatia, and Slovenia from some past trips. Make Serbia Great Again Our first stop is the city of Dubrovnik in Croatia, which has become well known being the primary filming location for Game of Thrones seasons 2 – 8.  I watched the series, and am kind of […]

5 months ago 48 votes
State of Data Engineering 2025 Q1

AI Updates There is a lot of chatter about 2025 being the year of agentic frameworks.  To me, this means a system in which a subset can allow AI models to take independent actions based on their environment, typically interacting with external APIs or interfaces.  The terminology around this concept is still evolving, and definitions […]

5 months ago 41 votes
State of Data Engineering Q3 2024

Prompt Engineering – Meta Analysis Whitepaper One of my favorite AI podcasts, Latent Space, recently featured Sander Schulhoff, one of the authors of a comprehensive research paper on prompt engineering. This meta-study reviews over 1,600 published papers, with co-authors from OpenAI, Microsoft, and Stanford.  [podcast] https://www.latent.space/p/learn-prompting   [whitepaper] https://arxiv.org/abs/2406.06608   The whitepaper is an interesting academic deep dive into prompting, […]

8 months ago 71 votes

More in programming

Measurement and Numbers

Here’s Jony Ive talking to Patrick Collison about measurement and numbers: People generally want to talk about product attributes that you can measure easily with a number…schedule, costs, speed, weight, anything where you can generally agree that six is a bigger number than two He says he used to get mad at how often people around him focused on the numbers of the work over other attributes of the work. But after giving it more thought, he now has a more generous interpretation of why we do this: because we want relate to each other, understand each other, and be inclusive of one another. There are many things we can’t agree on, but it’s likely we can agree that six is bigger than two. And so in this capacity, numbers become a tool for communicating with each other, albeit a kind of least common denominator — e.g. “I don’t agree with you at all, but I can’t argue that 134 is bigger than 87.” This is conducive to a culture where we spend all our time talking about attributes we can easily measure (because then we can easily communicate and work together) and results in a belief that the only things that matter are those which can be measured. People will give lip service to that not being the case, e.g. “We know there are things that can’t be measured that are important.” But the reality ends up being: only that which can be assigned a number gets managed, and that which gets managed is imbued with importance because it is allotted our time, attention, and care. This reminds me of the story of the judgement of King Solomon, an archetypal story found in cultures around the world. Here’s the story as summarized on Wikipedia: Solomon ruled between two women who both claimed to be the mother of a child. Solomon ordered the baby be cut in half, with each woman to receive one half. The first woman accepted the compromise as fair, but the second begged Solomon to give the baby to her rival, preferring the baby to live, even without her. Solomon ordered the baby given to the second woman, as her love was selfless, as opposed to the first woman's selfish disregard for the baby's actual well-being In an attempt to resolve the friction between two individuals, an appeal was made to numbers as an arbiter. We can’t agree on who the mother is, so let’s make it a numbers problem. Reduce the baby to a number and we can agree! But that doesn’t work very well, does it? I think there is a level of existence where measurement and numbers are a sound guide, where two and two make four and two halves make a whole. But, as humans, there is another level of existence where mathematical propositions don’t translate. A baby is not a quantity. A baby is an entity. Take a whole baby and divide it up by a sword and you do not half two halves of a baby. I am not a number. I’m an individual. Indivisible. What does this all have to do with software? Software is for us as humans, as individuals, and because of that I believe there is an aspect of its nature where metrics can’t take you.cIn fact, not only will numbers not guide you, they may actually misguide you. I think Robin Rendle articulated this well in his piece “Trust the vibes”: [numbers] are not representative of human experience or human behavior and can’t tell you anything about beauty or harmony or how to be funny or what to do next and then how to do it. Wisdom is knowing when to use numbers and when to use something else. Email · Mastodon · Bluesky

yesterday 4 votes
The 6 Hours of Lex

When I drive the 24 Hours of Le Mans, I spend a total of about 6-9 hours in the car, divided into stints of roughly two hours at a time. It's intense. But talking with Lex Fridman in Austin on his podcast? Over six hours straight! We only interrupted the session for five minutes total to take three bathroom breaks. All that endurance training has clearly paid off! But the magic of a good conversation, like the magic of driving at Le Mans, is that time flies by. Those six hours felt more like sixty minutes. This is what flow does: it compresses the moment. Besides, we had plenty to talk about. Lex prepares like no other podcast I've ever been on. Pages and pages of notes. Deep questions, endless attention for tangents. We covered the beauty of Ruby for half an hour alone! But also the future of AI, small teams, why we left the cloud, Elon Musk, fatherhood, money and happiness, and a million other topics (which Lex mercifully timestamps, so listeners without six hours to spare can hop around). It was a privilege to appear. If you're interested, the conversation is on YouTube, on Spotify, on X, and as a regular podcast.

yesterday 7 votes
Computers Are a Feeling

Exploring diagram.website, I came across The Computer is a Feeling by Tim Hwang and Omar Rizwan: the modern internet exerts a tyranny over our imagination. The internet and its commercial power has sculpted the computer-device. It's become the terrain of flat, uniform, common platforms and protocols, not eccentric, local, idiosyncratic ones. Before computers were connected together, they were primarily personal. Once connected, they became primarily social. The purpose of the computer shifted to become social over personal. The triumph of the internet has also impoverished our sense of computers as a tool for private exploration rather than public expression. The pre-network computer has no utility except as a kind of personal notebook, the post-network computer demotes this to a secondary purpose. Smartphones are indisputably the personal computer. And yet, while being so intimately personal, they’re also the largest distribution of behavior-modification devices the world has ever seen. We all willing carry around in our pockets a device whose content is largely designed to modify our behavior and extract our time and money. Making “computer” mean computer-feelings and not computer-devices shifts the boundaries of what is captured by the word. It removes a great many things – smartphones, language models, “social” “media” – from the domain of the computational. It also welcomes a great many things – notebooks, papercraft, diary, kitchen – back into the domain of the computational. I love the feeling of a personal computer, one whose purpose primarily resides in the domain of the individual and secondarily supports the social. It’s part of what I love about the some of the ideas embedded in local-first, which start from the principle of owning and prioritizing what you do on your computer first and foremost, and then secondarily syncing that to other computers for the use of others. Email · Mastodon · Bluesky

5 days ago 10 votes