Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
24

Luigi: complex pipelines of tasks in Python

from Home on Erik Bernhardsson [alt+shift+b] in technology

I'm shamelessly promoting my first major open source project. Luigi is a Python module that helps you build complex pipelines of batch jobs, handle dependency resolution, and create visualizations to help manage multiple workflows. It also comes with Hadoop support built in (because that's where really where its strength becomes clear).
21st Oct 2012

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Home on Erik Bernhardsson

It's hard to write code for computers, but it's even harder to write code for humans

Writing code for a computer is hard enough. You take something big and fuzzy, some large vague business outcome you want to achive. Then you break it down recursively and think about all the cases until you have clear logical statements a computer can follow.

27th Sep 2024 • 39 votes
Predicting solar eclipses with Python

As I am en route to see my first total solar eclipse, I was curious how hard it would be to compute eclipses in Python. It turns out, ignoring some minor coordinate system head-banging, I was able to get something half-decent working in a couple of hours.

7th Apr 2024 • 41 votes
Simple sabotage for software

CIA produced a fantastic book during the peak of World War 2 called Simple Sabotage. It laid out various ways for infiltrators to ruin productivity of a company. Some of the advice is timeless, for instance the section about “General interference with Organizations and Production”:

13th Dec 2023 • 40 votes
What I have been working on: Modal

Long story short: I'm working on a super cool tool called Modal. Please check it out — it lets you run things in the cloud without having to think about infrastructure. Scaling out, scheduling, containerization, using GPUs, setting up webhooks, and all kinds of other stuff.

7th Dec 2022 • 42 votes
We are still early with the cloud: why software development is overdue for a change

This is is in many respects a successor to a blog post I wrote last year about what I want from software infrastructure, but the ideas morphed in my head into something sort of wider.

19th Oct 2022 • 42 votes

More in technology

The Topology of Contemporary Politics

I’ve been getting back into the politics fandom this past month.

2 hours ago • 1 votes
So all software is free as in money now?

A few weeks ago I was up late in my kitchen. I've always been a bad sleeper, starting from when I was 10 and would spend whole nights reading books and watching the clock slowly wind it way to when I had to get up and go to

4 hours ago • 1 votes
Nathaniel at WordCamp Canada

Nathaniel Christopher has a lovely recap of his time at WordCamp Canada. Very cool that the dashboard widget helped connect him to the local community. Hat tip: Anne.

2 days ago • 1 votes
Our current screen time policies

Managing a house with five kids can be challenging so I like to share what’s working for us when I can. This post is how we currently handle screen time. Screens can be addictive, and it’s easy for anyone (both kids and adults) to fall into unhealthy patterns using them. Historically, I’ve been uncomfortable with screen time restrictions because I don’t want to restrict all the useful, creative, and productive opportunities that screens can provide. We’ve tried zero restrictions in the past (alongside Daily Demos as a creativity incentive), which worked ok, but at some point we wanted bit more structure. Here’s what that structure currently looks like. Device use categories For family members without phones, we have carved out the following categories of time: “Free Time” - Kids are free to do whatever activities they want (bounded only by what we’ve made available for them at the house). Video games, movies, board games, homework, run around outside, whatever. Often, this time trends towards consumption on devices, but it doesn’t have to. We call it “free time” instead of “screen time,” because we want to emphasize that the opportunities are wider than just screens. “Project Time” - Kids are encouraged to do projects, which are usually defined as creative activities of all kinds (with or without screens). This category can be blurry so we’ve often put together lists of approved project time activities, including things like: arts and crafts, coding (including “Scratch create”), playing musical instruments, recording videos, Minecraft (creative mode), the Powder Toy, Dojo Islands, and anything on Let’s Get Creative. Device-based consumption is not allowed. Offline activities are usually considered fine (reading a book, doing a puzzle, etc). “No devices” - No device use allowed. Addendum: Grant Proposals If a kid wants a one-time exception to one of the categories above, we tell them to “submit a grant proposal.” This means they need to get some paper, and write out what they want to do, and why they think it is justified. Example 1: “I want to play Minecraft creative after dinner because I am building a full-scale piano, I want to demo it tonight, and I only have 10 more minutes of work until it’s done.” Example 2: “I want to play Terraria during project time because I am making an tutorial on how to beat Moon Lord, and I need to collect screenshots from the game for my tutorial.” We usually approve grant proposals, as long as they aren’t incomplete. Schedule With these categories defined, we just need to decide how much time to allocate, and when to schedule it. For example, here’s our current “no devices” schedule: "No Devices" times Reason After Dinner Reduce stimulation before bed Sundays Protected for church and family Saturdays, before chores are done Incentivize quick completion of chores, so our Saturday afternoons remain available The categories + schedule constitutes our device policy, in addition to Daily Demos, which we continue to hold on the side. Summary Addictive device consumption can easily crowd out activities that are important long-term but deliver less dopamine short-term. Our device-time categories create a protected space for those activities on a consistent schedule that our kids can expect and make plans around. This is what works for us. Your milage may vary! Thanks for reading this post via RSS! Let me know your thoughts by leaving a comment on the original post or sending me an email.

2 days ago • 1 votes
Introducing The 4-Year Battery Keychain for AirTag

There are few things worse than a dead AirTag when you need it. That's why we designed the The 4-Year Battery Keychain for AirTag. It replaces the low power CR2032 coin cell with an included Toshiba CR2477 battery for over 4x more capacity. And does so in a minimal form factor.  Assembles in seconds. Split key ring + braided cord included. Easy to open with included tool. Perfect on a keychain.  And great for backpacks or a purse.  Also available on Amazon.

3 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in