Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
25
I had a bunch of thoughts yesterday about the Zettelkasten method and how I could use it effectively to manage my knowledge base. I started the day by dumping my thoughts into Logseq. Here they are. These are open questions for now. I plan to investigate this further and try out different iterations to see what works for me. I’ve been in a place before where I used Roam to gather a small number of notes (> 100) but then found all of that to be an unmanageable mess. Issues that I see with this setup With notes spread all over the place how do I find anything to link to? I can’t go through 200 notes every time I add a new one. All notes are in the same “directory”. Because there is no hierarchy, my notes about productivity are in the same place as my notes about data structures & algorithms. This seems unsustainable. There is 1 benefit I see to this. With everything being in the same place, I can find unexpected connections. Unfortunately, that doesn’t work for me because I don’t go...
over a year ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from Jibran’s Perspective

Project 2: Gift cards to Pakistan

I’ve completed a freelance project I was working on for a few months, and have started saying no to new opportunities. It’s time to work on one of my own ideas again. This is part of my plan to start failing more. I’ve decided to build a business sending gift cards to Pakistan - and eventually other countries in that corner of the world. Why? A few years ago I had sent a gift card to a colleague in the UK. I found a number of very good options. They all had websites that inspired confidence, and used robust payment methods (Stripe in my example) that I could trust with my credit card. I recently had to send a gift card to a colleague in Pakistan. I was confident that I would find a bunch of great options; instead I only found one that I could think of trusting with my money. I ended up using their services and the card was delivered, but there were a number of problems I saw: No trust building around card payments. There was no clear mention of which provider they used. I did a bank transfer instead of using a CC. This meant my payment was manually verified and the card was only sent after a few hours. There was no confirmation email about my order. I was worried enough to call their helpline to confirm that my order had gone through. Once they had sent the card (which I also had to confirm via phone), I only got a confirmation email the next day. To get an invoice to expense this, I had to send them an email. I’m still waiting on an invoice. There were multiple colleagues who chipped in on this gift card. I had to collect the money from them and then pay for the card myself. In my previous experience of sending a gift card to the UK, I was able to include my colleagues in the process. They were able to add their contributions directly to the gift card I selected and a card of the total amount was sent to the recipient. Finally, there was no option for the receiver to choose which gift card they wanted. Instead I had to choose for them. There is a “Universal Gift Card” they claim works at all merchants and is the one I got, but redeeming that would be slightly more complicated. Interestingly, my colleague didn’t open the email they received with the gift card because they thought it was a spam/scam/malicious email. Only after I asked if they had received the card did they end up opening it. I know a better user experience exists. I want to bring the same to Pakistan and solve my own problem at the same time. Is there a market for this? I believe so, because: It’s a problem I’ve just faced. I’ve seen my wife having to deal with low-trust companies sending gifts to Pakistan. Gift cards are different, but eventually I could also add the option to send physical gifts to the recipient. I’ve seen my employer deal with this. Recently a baby gift basket arrived 2 months after the baby was born. 🤯 This is a recurring problem. People & companies need to send gift cards on birthdays, weddings, etc. With more companies starting to hire remotely in Pakistan, this could be a valuable service for businesses to subscribe to. Validation? I haven’t found an easy way to validate this idea. There is no community of “people sending gift cards to Pakistan” that I can tap into. That isn’t a cohort I can find in one place. I could make a list of B2B customers; companies that hire remotely in Pakistan. However, I want to start with individual customers - because I’m starting from a place of solving my own problem. It should be possible to pivot to B2B if I don’t find any interest from individual customers. Validation then involves me starting with a blog - suggesting gift cards to send to Pakistan. I’ll use SEO to bring in traffic. If I see enough visitors, I could start building a business. This also means that if/when the actual product launches, I’ll have a distribution channel already working. What if I’m wrong? There’s a very strong possibility that I’m wrong about this idea. That I’ll spend a bunch of time for it to get nowhere, or that I have picked a problem that isn’t very valuable to solve. This is my unique brand of fear of failure. I used to think I didn’t fear failing, because I had already failed many times. Instead, my fear of failure manifests as a fear of picking the wrong thing and wasting time on it. The way I am dealing with this is to realize that if I don’t pick anything - which I have frequently done in the past - I have an exactly 0% chance of succeeding. Just trying something makes that probability > 0%. You miss 100% of the shots you don’t take. Another thing that’s helping me is to time box this idea. I will spend 6 weeks on building the blog and populating it with as much useful content as possible. After that I can spend an hour or two every week to add a few more pieces of content. I can start researching and working on a different idea after the 6 week period and wait for the SEO to have an impact before making a decision to continue or abandon this.

8 months ago 72 votes
Deploying Ruby on Rails to AWS with Kamal

As part of a contracting project, I’ve been building an analytics dashboard for a feedback collection SaaS. The app is built in Ruby on Rails and given all the nice things I’ve heard about Kamal; I decided to use it for deploying the app. The experience has been phenomenal; outside of some frustration with the initial deployment. The app is deployed on a pretty standard AWS setup; a couple of EC2 servers hosting the web app running inside Docker containers, and a load balancer in front. One of the problems I faced during the initial deployment was forwarding headers from the AWS application load balancer to the RoR server running in the Docker container. The challenge with Kamal is that it relies heavily on Traefik, and while Traefik is a great tool, it takes some getting used to. It’s configuration is not very intuitive, and there’s no easy way to see how things are configured outside of looking at the text logs. The Traefik document is pretty thorough, so a bit of searching led me to this CLI argument which needs to be passed to the Traefik container: entrypoints.http.forwardedheaders.insecure: true However, no matter what I tried, when I added this, the app container would stop responding to web requests. Without the config the container would work but throw an exception related to the Origin header not matching the configured hosts. After a lot of experimentation, I stumbled upon the other config I needed to add by pure luck. entrypoints.http.address: ":80" As far as I can tell, when I added the forwardedheaders config, the entrypoint no longer got the correct address configuration. I’m not sure if this is related to Kamal or Traefik. Kamal deploy.yml If you’re looking to replicate a similar setup, here’s the Kamal deploy.yml file that I am using with this project to deploy to AWS, with a load balancer terminating the SSL connection and forwarding traffic to web servers that are configured via Kamal. As a bonus, this config also deploys Sidekiq for background tasks. service: <SERVICE NAME> image: <IMAGE NAME> ssh: user: ubuntu proxy: "ubuntu@A.B.C.D" servers: web: hosts: - "A.B.C.D" - "A.B.C.D" labels: traefik.http.routers.<SERVICE NAME>-web.rule: Host(`<YOUR HOST NAME>`) sidekiq: hosts: - "A.B.C.D" - "A.B.C.D" traefik: false cmd: bundle exec sidekiq registry: server: <AWS ACCOUNT ID>.dkr.ecr.<AWS REGION>.amazonaws.com username: AWS password: <%= %x(aws ecr get-login-password --region <AWS REGION>) %> builder: local: arch: amd64 # Because I develop on a Apple Silicon machine, I need to use a build target env: clear: - DATABASE_URL: <DATABASE URL> secret: - RAILS_MASTER_KEY - DB_PASSWORD traefik: args: entrypoints.http.address: ":80" entrypoints.http.forwardedheaders.insecure: true log.level: DEBUG accesslog: true accesslog.format: json

a year ago 43 votes
Failure 1: Django + NextJS Boilerplate

I have failed, and that is exactly what I had hoped for a few months ago in this blog post. This is a good failure. It has taught me things, lessons I can use in the future to avoid failing this way again. But first a bit of context. What did I fail at? In February of 2024 I decide to try my hands on my first “Indie Hacker” hustle, something that would make me money on the internet without having to trade my time for it. A product instead of consultancy services that I usually provide. I had seen a number of people on Twitter (X) rave about how well their bootstrap templates were doing; and I had just gotten out of a consultancy project where I needed to connect a Next.js frontend to a Django backend. I thought it was the perfect project to start my indie hacking journey. I put up a launch post and started working, updating a build log as I went along. I gave myself until 28th March 2024 to finish it. That of course did not happen. Let’s talk about why I failed and what I learned. Episode 1: The one where I don’t understand the meaning of MVP My initial plan was to build a Django+Next.js boilerplate template the provided all of these: the base template that provided a Django backend & Next.js frontend working authentication b/w the backend & frontend Dockerfile that would create the backend & frontend containers for deployment Terraform scripts to setup an infrastructure on AWS Celery + Redis for background task processing TailwindCSS for the frontend (comes mostly for free with Next.js) social auth This looks like something achievable in a week or two of work - but only if you’re working full time on this. I failed to consider that I have a day job and a life. I was barely able to tick of the first two of these deliverables by the time my 6 week deadline came up. As a good friend told me later, I should have focused on the minimum amount of value I could deliver. Just having the first two things on my list be done would have been enough. I couldn’t charge the $20 I had planned for, but I could have charged $1-$5 for just that. And if no one was interested in spending the cost of a coffee on the MVP of the template, that would have been a good signal that this wasn’t going anywhere in it’s current shape. Instead, by focusing on building something much bigger, I robbed myself of the ability to validate the idea quickly. I spent all my available time coding the template instead of trying to talk to potential customers about it. Lesson 1: Scope down aggressively. Episode 2: Where I jumped on the hype-wagon I settled on building a boilerplate template because that’s what I had seen a lot of people on Twitter/X doing lately; I’m chalking this down to recency bias. I had no personal interest in a boilerplate template. It’s also not a product that I would personally use. I have so far made one project that uses this tech stack. Most of my other projects are Django, and Ruby on Rails. The most successful boilerplate templates I come across are from people who made a bunch of projects in 1 tech stack then realized they needed to do the same thing over-and-over again; which they then packaged into a template they could use. Selling to others was a bonus at first I guess. I was very enthusiastic about the project at the start, but as time went on I had to force myself to work on it. My lack of interest in this type of project was a big factor. Another factor was there being no way to see the fruits of my labor. I am currently working on an analytics dashboard for another client (a RoR project) and every time I build a feature, I love to play around with it in my free time. I test how it works, make sure the UX is a good one, and just play around and admire the app I’ve made. Without me using my template to build new projects, I lacked that feedback loop. Without the loop, I quickly lost interest. Lesson 2: Build something I can use myself. This isn’t a job I’m getting paid for, so the only motivation I have initially until it starts generating money is to build something interesting for myself. Episode 3: Where I had nothing for potential customers to play around with This is related to the 1st lesson. Because I didn’t have a path to quickly get something out there, there was no way for me to get my “product” into the hands of people who could test and provide feedback. I think the problem with a boilerplate template style of product is that you can’t give people a half-backed thing and ask them to test it. Unlike other SaaS apps, there’s no mid-way version of a template. Customers have to “buy-in” to use your template with any project they are starting. With SaaS, users can sign up and test, and then leave if they don’t like it. There’s no easy way of testing with a template. Lesson 3: Build something that can be tested by potential customers easily. For now, I’m going to stick with SaaS style web apps. Conclusion Moving forward: I’ll be working on web app products that users can sign up for and test very quickly. My next few experiments/products will be things that I can use myself as well. I’ll post what I’m going to work on next when I decide and have some time away from my job & freelance projects that are currently in progress.

a year ago 39 votes
Cookie Based Auth for Django and NextJS

If you’re just looking for implementation instructions, skip my ramblings and go straight to the code here. I’m currently working on my first project after deciding that I needed to fail more and practice finishing projects instead of abandoning them midway once they got “boring”. Anyways… This one is till in it’s interesting phase, so here’s a blog post with some things I learned yesterday while working on it. The project is a boilerplate template that should make it easy for devs. to start a new project with a Django backend and a Next.js frontend, something I had to struggle with recently. The problem The first thing I’m looking to solve is authentication. That was my biggest challenge when working on the contracting project that inspired this template. While there are a number of good posts around how to setup authentication b/w Django & Next.js, nothing “definitive” came up and I had to cobble together a weird mess of Django+DRF (Django Rest Framework) and Next.js+NextAuth, sharing a token from Django that was masquarading as a JWT token for Next.js. It wasn’t pretty and I knew I could do better. The options I considered 2 options for authenticating the Next.js frontend with the Django backend: Token based auth. On logging in, a user receives a token that is stored in local storage by the frontend and send with every request to the backend. Session/Cookie based auth. This is how authentication works in Django by default and is very easy to get started with - it basically comes for free out of the box when you start a new Django project. While token based auth. is what almost everyone suggests to use when using a Next.js frontend with any backend technology, I wanted to give session based auth. a try. I was curious what it would take to make it work - if it was even possible. tl;dr: It was possible to use cookie/session auth. b/w Django & Next.js - though with a few constraints which make it less appealing than the token based solution What follows are my notes on how to set it up, the problems I faced, and why for the template I’m going to go with token based auth. instead. Learning how CORS & Set-Cookie works It took me a few hours to get my head around how cross-origin requests and cookies work together, but the actual implementation was surprisingly straight forward. This “mini-quest” gave me a chance to learn a lot about how CORS and cookies work, and I’m happy with the time I spent on this. These are the resources which helped me the most (all are from MDN): Cross-Origin Resource Sharing Same-origin policy Using HTTP cookies Set-Cookie And finally, there was a surprise waiting for me! Browsers are almost universally making changes to restrict 3rd party or cross-domain cookies because of their privacy implications. Here’s a nice article from MDN about it: Saying goodbye to third-party cookies in 2024. This is the reason why; while this approach works, I won’t be using it in the template. More on that later. Implementation Implementing the session based auth. b/w Django & Next.js is pretty simple. Django configuration Install the django-cors-headers Python package. Add "corsheaders", to your INSTALLED_APPS. Add the "corsheaders.middleware.CorsMiddleware", middleware, right above the existing CommonMiddleware. Set CORS_ALLOWED_ORIGINS = ["http://localhost:3000"], replacing the URL with your frontend URL. Set CORS_ALLOW_CREDENTIALS = True Configure settings.py to allow cross-domain access for the session cookie. Set SESSION_COOKIE_SAMESITE = "None" Set SESSION_COOKIE_SECURE = True Next.js configuration No configuration is needed on the frontend. However, you do need to use the credentials: "include", option when using the fetch() API to access your backend. Here’s a minimal example. "use client"; import { BACKEND_URL } from "@/constants"; async function signIn() { const loginData = new FormData(); loginData.append("username", "admin"); loginData.append("password", "admin"); return await fetch(`${BACKEND_URL}/accounts/login/`, { method: "POST", body: loginData, credentials: "include", }); } async function whoAmI() { console.log( await fetch(`${BACKEND_URL}/accounts/me/`, { method: "GET", credentials: "include", }), ); } export default function Home() { return ( <main className="flex min-h-dvh w-full flex-col justify-around"> <h1 className="text-center">Home</h1> <button className="" onClick={signIn}> Sign In </button> <button onClick={whoAmI}>Who Am I</button> </main> ); } That’s it. That simple piece of code & configuration took me hours to find. Hopefully you can use this example to skip all that time spent trying to figure things out. Side quest log: Initially, I was not using the credentials: "include" option in the signIn() function above; thinking that I didn’t need to send any cookies with the login call, only the second API call to the /accounts/me endpoint. That mistake cost me about 2 hours of debugging time. If I had RTFM correctly the first time, I would have seen this: include: Tells browsers to include credentials in both same- and cross-origin requests, and always use any credentials sent back in responses. The credentials: "include" not only controls if cookies are sent, but also if they are saved when returned by the server. Why I won’t use this solution in the template Browsers are phasing out 3rd party cookies (Saying goodbye to third-party cookies in 2024) and adding features to work around that restriction where needed. The simplest way that doesn’t require much change is to use Cookies Having Independent Partitioned State (CHIPS). To enable CHIPS, you simply put a Partitioned flag on your Set-Cookie header, like so: Set-Cookie: session_id=1234; SameSite=None; Secure; Path=/; Partitioned; Unfortunately, there’s no straight forward way to do this in Django for now. There’s an open issue to resolve this, but looking at the comments, it won’t likely be solved anytime soon. Considering this, I opted to use the token based auth. method for my template. I’ll write a blog on that once I get it working over the next few days.

a year ago 36 votes
Project 1: Django + NextJS Boilerplate

Links: Gumroad page Build Log My accidental new years resolution was to work on the 1 problem that has plagued me for my entire adult life; failure to commit and focus. I decided to work in 6 week “sprints” (inspired by Shape Up) and complete the projects I start - for some known definition of complete. This is the 1st project I have decided to work on. I’ll work on this from today (15th Feb 2024) to (28th Mar 2024). I’ll follow-up then with another post talking about how it went. The project The goal is to make & sell a Django + NextJS boilerplate template. What’s a boilerplate template? It’s the source code for a project that’s already setup with many things that are needed in a new project; for example: Stripe subscriptions functionality Background jobs CSS framework User/team management A great example is Saas Pegasus, which seems like an amazing boilerplate loved by many people. My boilerplate is going to be much simpler - and also much cheaper. SaaS Pegasus comes with so many features that it’s worth the $249 starting price. I’m aiming for $5-$10. Goals My goal is to sell this boilerplate to at least 10 people - and have them be happy using it. This means: talking to prospective customers and seeing if this can be useful to them. People will have the option of scheduling a 15 minute pre-purchase call with me for $5 to see if this would be useful to them. The payment is purely to make sure that I only spend time talking to people who are somewhat serious about purchasing. providing excellent after sales support. I’ll include a 60 minute setup call with me for any purchase. While a 60 minute call for a $10 sale isn’t scalable, it’s a great way for me to talk to customers at the start. having a no questions asked refund policy. My experiences with running an e-commerce store in the past tell me this is an amazing way to build trust. provide on-going support, updates, and fixes over email. build a mailing list of people interested in my work who I can email when I launch my future projects. The deliverable The boilerplate will allow developers to quickly start a project that uses Django for the backend and NextJS for the frontend. My recent experiences with another project in this tech stack required me to spend significant time on: figuring out how to setup authentication b/w Django & NextJS (this took the most time & effort) setting up Django Rest Framework so I could write APIs that would be used by the frontend writing Docker files that would build 2 containers - backend & frontend writing Terraform scripts to deploy those containers to AWS ECS writing config & scripts to run the project on Gitpod so it could be easily worked on by my team members My plan is to build a boilerplate that already has most those features built in, plus a few extras: Celery with Redis for background task processing Tailwind CSS for the frontend (in my project I used ChakraUI but Tailwind would be a better option for a boilerplate) If there’s demand for it, a stretch goal is to include social auth (sign-in with Google/Apple/etc) Once complete, I’ll put this on Gumroad and create a landing page there. From then on, it’s all about marketing it; that’s the part which I have no experience with and hope to learn the most from. The marketing plan This is the area where I lack any experience; so I’m not sure how I’m going to market this. Some ideas I have: build it in public on Twitter. I have a tiny Twitter following (312 followers) so not sure how useful this could be. But I have to try something. share it with people asking how to setup Django & NextJS on forums like Reddit, Stackoverflow, and others. maybe write a blog post on how to setup Django & NextJS and then link to the boilerplate from there. The blog post would provider all the steps necessary for the basic setup and the boilerplate would go beyond that with something that’s ready to use. The build log I’d also like to create a build log with this project. This will be a daily note of what I did for this project. I’ll keep it in my notes app Reflect and periodically put it here in this blog post. These daily notes might also serve as content for my build-in-public marketing strategy.

a year ago 37 votes

More in programming

Executives should be the least busy people

If your executive calendar is packed back to back, you have no room for fires, customers, or serendipities. You've traded all your availability for efficiency. That's a bad deal. Executives of old used to know this! That's what the long lunches, early escapes to the golf course, and reading the paper at work were all about. A great fictional example of this is Bert Cooper from Mad Men. He knew his value was largely in his network. He didn't have to grind every minute of every day to prove otherwise. His function was to leap into action when the critical occasion arose or decision needed to be made. But modern executives are so insecure about seeming busy 24/7 that they'll wreck their business trying to prove it. Trying to outwork everyone. Sacrificing themselves thin so they can run a squirrel-brain operation that's constantly chasing every nutty idea. Now someone is inevitably going to say "but what about Elon!!". He's actually a perfect illustration of doing this right, actually. Even if he works 100-hour weeks, he's the CEO of 3 companies, has a Diablo VI addiction, and keeps a busy tweeting schedule too. In all of that, I'd be surprised if there was more than 20-30h per company per week on average. And your boss is not Elon. Wide open calendars should not be seen as lazy, but as intentional availability. It's time we brought them back into vogue.

2 days ago 4 votes
Dispatch 012: Local-first talks, Automerge 3, and Scribbling on a Google Calendar

A secret master plan, the official launch of Automerge 3, and an update on Sketchy Calendars

2 days ago 2 votes
React Server Components with Vite and React-Router (tip)

Create a small example app and send payloads from the server to the client using RSC's

3 days ago 9 votes
2000 words about arrays and tables

I'm way too discombobulated from getting next month's release of Logic for Programmers ready, so I'm pulling a idea from the slush pile. Basically I wanted to come up with a mental model of arrays as a concept that explained APL-style multidimensional arrays and tables but also why there weren't multitables. So, arrays. In all languages they are basically the same: they map a sequence of numbers (I'll use 1..N)1 to homogeneous values (values of a single type). This is in contrast to the other two foundational types, associative arrays (which map an arbitrary type to homogeneous values) and structs (which map a fixed set of keys to heterogeneous values). Arrays appear in PLs earlier than the other two, possibly because they have the simplest implementation and the most obvious application to scientific computing. The OG FORTRAN had arrays. I'm interested in two structural extensions to arrays. The first, found in languages like nushell and frameworks like Pandas, is the table. Tables have string keys like a struct and indexes like an array. Each row is a struct, so you can get "all values in this column" or "all values for this row". They're heavily used in databases and data science. The other extension is the N-dimensional array, mostly seen in APLs like Dyalog and J. Think of this like arrays-of-arrays(-of-arrays), except all arrays at the same depth have the same length. So [[1,2,3],[4]] is not a 2D array, but [[1,2,3],[4,5,6]] is. This means that N-arrays can be queried on any axis. ]x =: i. 3 3 0 1 2 3 4 5 6 7 8 0 { x NB. first row 0 1 2 0 {"1 x NB. first column 0 3 6 So, I've had some ideas on a conceptual model of arrays that explains all of these variations and possibly predicts new variations. I wrote up my notes and did the bare minimum of editing and polishing. Somehow it ended up being 2000 words. 1-dimensional arrays A one-dimensional array is a function over 1..N for some N. To be clear this is math functions, not programming functions. Programming functions take values of a type and perform computations on them. Math functions take values of a fixed set and return values of another set. So the array [a, b, c, d] can be represented by the function (1 -> a ++ 2 -> b ++ 3 -> c ++ 4 -> d). Let's write the set of all four element character arrays as 1..4 -> char. 1..4 is the function's domain. The set of all character arrays is the empty array + the functions with domain 1..1 + the functions with domain 1..2 + ... Let's call this set Array[Char]. Our compilers can enforce that a type belongs to Array[Char], but some operations care about the more specific type, like matrix multiplication. This is either checked with the runtime type or, in exotic enough languages, with static dependent types. (This is actually how TLA+ does things: the basic collection types are functions and sets, and a function with domain 1..N is a sequence.) 2-dimensional arrays Now take the 3x4 matrix i. 3 4 0 1 2 3 4 5 6 7 8 9 10 11 There are two equally valid ways to represent the array function: A function that takes a row and a column and returns the value at that index, so it would look like f(r: 1..3, c: 1..4) -> Int. A function that takes a row and returns that column as an array, aka another function: f(r: 1..3) -> g(c: 1..4) -> Int.2 Man, (2) looks a lot like currying! In Haskell, functions can only have one parameter. If you write (+) 6 10, (+) 6 first returns a new function f y = y + 6, and then applies f 10 to get 16. So (+) has the type signature Int -> Int -> Int: it's a function that takes an Int and returns a function of type Int -> Int.3 Similarly, our 2D array can be represented as an array function that returns array functions: it has type 1..3 -> 1..4 -> Int, meaning it takes a row index and returns 1..4 -> Int, aka a single array. (This differs from conventional array-of-arrays because it forces all of the subarrays to have the same domain, aka the same length. If we wanted to permit ragged arrays, we would instead have the type 1..3 -> Array[Int].) Why is this useful? A couple of reasons. First of all, we can apply function transformations to arrays, like "combinators". For example, we can flip any function of type a -> b -> c into a function of type b -> a -> c. So given a function that takes rows and returns columns, we can produce one that takes columns and returns rows. That's just a matrix transposition! Second, we can extend this to any number of dimensions: a three-dimensional array is one with type 1..M -> 1..N -> 1..O -> V. We can still use function transformations to rearrange the array along any ordering of axes. Speaking of dimensions: What are dimensions, anyway Okay, so now imagine we have a Row × Col grid of pixels, where each pixel is a struct of type Pixel(R: int, G: int, B: int). So the array is Row -> Col -> Pixel But we can also represent the Pixel struct with a function: Pixel(R: 0, G: 0, B: 255) is the function where f(R) = 0, f(G) = 0, f(B) = 255, making it a function of type {R, G, B} -> Int. So the array is actually the function Row -> Col -> {R, G, B} -> Int And then we can rearrange the parameters of the function like this: {R, G, B} -> Row -> Col -> Int Even though the set {R, G, B} is not of form 1..N, this clearly has a real meaning: f[R] is the function mapping each coordinate to that coordinate's red value. What about Row -> {R, G, B} -> Col -> Int? That's for each row, the 3 × Col array mapping each color to that row's intensities. Really any finite set can be a "dimension". Recording the monitor over a span of time? Frame -> Row -> Col -> Color -> Int. Recording a bunch of computers over some time? Computer -> Frame -> Row …. This is pretty common in constraint satisfaction! Like if you're conference trying to assign talks to talk slots, your array might be type (Day, Time, Room) -> Talk, where Day/Time/Room are enumerations. An implementation constraint is that most programming languages only allow integer indexes, so we have to replace Rooms and Colors with numerical enumerations over the set. As long as the set is finite, this is always possible, and for struct-functions, we can always choose the indexing on the lexicographic ordering of the keys. But we lose type safety. Why tables are different One more example: Day -> Hour -> Airport(name: str, flights: int, revenue: USD). Can we turn the struct into a dimension like before? In this case, no. We were able to make Color an axis because we could turn Pixel into a Color -> Int function, and we could only do that because all of the fields of the struct had the same type. This time, the fields are different types. So we can't convert {name, flights, revenue} into an axis. 4 One thing we can do is convert it to three separate functions: airport: Day -> Hour -> Str flights: Day -> Hour -> Int revenue: Day -> Hour -> USD But we want to keep all of the data in one place. That's where tables come in: an array-of-structs is isomorphic to a struct-of-arrays: AirportColumns( airport: Day -> Hour -> Str, flights: Day -> Hour -> Int, revenue: Day -> Hour -> USD, ) The table is a sort of both representations simultaneously. If this was a pandas dataframe, df["airport"] would get the airport column, while df.loc[day1] would get the first day's data. I don't think many table implementations support more than one axis dimension but there's no reason they couldn't. These are also possible transforms: Hour -> NamesAreHard( airport: Day -> Str, flights: Day -> Int, revenue: Day -> USD, ) Day -> Whatever( airport: Hour -> Str, flights: Hour -> Int, revenue: Hour -> USD, ) In my mental model, the heterogeneous struct acts as a "block" in the array. We can't remove it, we can only push an index into the fields or pull a shared column out. But there's no way to convert a heterogeneous table into an array. Actually there is a terrible way Most languages have unions or product types that let us say "this is a string OR integer". So we can make our airport data Day -> Hour -> AirportKey -> Int | Str | USD. Heck, might as well just say it's Day -> Hour -> AirportKey -> Any. But would anybody really be mad enough to use that in practice? Oh wait J does exactly that. J has an opaque datatype called a "box". A "table" is a function Dim1 -> Dim2 -> Box. You can see some examples of what that looks like here Misc Thoughts and Questions The heterogeneity barrier seems like it explains why we don't see multiple axes of table columns, while we do see multiple axes of array dimensions. But is that actually why? Is there a system out there that does have multiple columnar axes? The array x = [[a, b, a], [b, b, b]] has type 1..2 -> 1..3 -> {a, b}. Can we rearrange it to 1..2 -> {a, b} -> 1..3? No. But we can rearrange it to 1..2 -> {a, b} -> PowerSet(1..3), which maps rows and characters to columns with that character. [(a -> {1, 3} ++ b -> {2}), (a -> {} ++ b -> {1, 2, 3}]. We can also transform Row -> PowerSet(Col) into Row -> Col -> Bool, aka a boolean matrix. This makes sense to me as both forms are means of representing directed graphs. Are other function combinators useful for thinking about arrays? Does this model cover pivot tables? Can we extend it to relational data with multiple tables? Systems Distributed Talk (will be) Online The premier will be August 6 at 12 CST, here! I'll be there to answer questions / mock my own performance / generally make a fool of myself. Sacrilege! But it turns out in this context, it's easier to use 1-indexing than 0-indexing. In the years since I wrote that article I've settled on "each indexing choice matches different kinds of mathematical work", so mathematicians and computer scientists are best served by being able to choose their index. But software engineers need consistency, and 0-indexing is overall a net better consistency pick. ↩ This is right-associative: a -> b -> c means a -> (b -> c), not (a -> b) -> c. (1..3 -> 1..4) -> Int would be the associative array that maps length-3 arrays to integers. ↩ Technically it has type Num a => a -> a -> a, since (+) works on floats too. ↩ Notice that if each Airport had a unique name, we could pull it out into AirportName -> Airport(flights, revenue), but we still are stuck with two different values. ↩

3 days ago 8 votes
Our $100M Series B

We don’t want to bury the lede: we have raised a $100M Series B, led by a new strategic partner in USIT with participation from all existing Oxide investors. To put that number in perspective: over the nearly six year lifetime of the company, we have raised $89M; our $100M Series B more than doubles our total capital raised to date — and positions us to make Oxide the generational company that we have always aspired it to be. If this aspiration seems heady now, it seemed absolutely outlandish when we were first raising venture capital in 2019. Our thesis was that cloud computing was the future of all computing; that running on-premises would remain (or become!) strategically important for many; that the entire stack — hardware and software — needed to be rethought from first principles to serve this market; and that a large, durable, public company could be built by whomever pulled it off. This scope wasn’t immediately clear to all potential investors, some of whom seemed to latch on to one aspect or another without understanding the whole. Their objections were revealing: "We know you can build this," began more than one venture capitalist (at which we bit our tongue; were we not properly explaining what we intended to build?!), "but we don’t think that there is a market." Entrepreneurs must become accustomed to rejection, but this flavor was particularly frustrating because it was exactly backwards: we felt that there was in fact substantial technical risk in the enormity of the task we put before ourselves — but we also knew that if we could build it (a huge if!) there was a huge market, desperate for cloud computing on-premises. Fortunately, in Eclipse Ventures we found investors who saw what we saw: that the most important products come when we co-design hardware and software together, and that the on-premises market was sick of being told that they either don’t exist or that they don’t deserve modernity. These bold investors — like the customers we sought to serve — had been waiting for this company to come along; we raised seed capital, and started building. And build it we did, making good on our initial technical vision: We did our own board designs, allowing for essential system foundation like a true hardware root-of-trust and end-to-end power observability. We did our own microcontroller operating system, and used it to replace the traditional BMC. We did our own platform enablement software, eliminating the traditional UEFI BIOS and its accompanying flotilla of vulnerabilities. We did our own host hypervisor, assuring an integrated and seamless user experience — and eliminating the need for a third-party hypervisor and its concomitant rapacious software licensing. We did our own switch — and our own switch runtime — eliminating entire universes of integration complexity and operational nightmares. We did our own integrated storage service, allowing the rack-scale system to have reliable, available, durable, elastic instance storage without necessitating a dependency on a third party. We did our own control plane, a sophisticated distributed system building on the foundation of our hardware and software components to deliver the API-driven services that modernity demands: elastic compute, virtual networking, and virtual storage. While these technological components are each very important (and each is in service to specific customer problems when deploying infrastructure on-premises), the objective is the product, not its parts. The journey to a product was long, but we ticked off the milestones. We got the boards brought up. We got the switch transiting packets. We got the control plane working. We got the rack manufactured. We passed FCC compliance. And finally, two years ago, we shipped our first system! Shortly thereafter, more milestones of the variety you can only get after shipping: our first update of the software in the field; our first update-delivered performance improvements; our first customer-requested features added as part of an update. Later that year, we hit general commercial availability, and things started accelerating. We had more customers — and our first multi-rack customer. We had customers go on the record about why they had selected Oxide — and customers describing the wins that they had seen deploying Oxide. Customers starting landing faster now: enterprise sales cycles are infamously long, but we were finding that we were going from first conversations to a delivered product surprisingly quickly. The quickening pace always seemed to be due in some way to our transparency: new customers were listeners to our podcast, or they had read our RFDs, or they had perused our documentation, or they had looked at the source code itself. With growing customer enthusiasm, we were increasingly getting questions about what it would look like to buy a large number of Oxide racks. Could we manufacture them? Could we support them? Could we make them easy to operate together? Into this excitement, a new potential investor, USIT, got to know us. They asked terrific questions, and we found a shared disposition towards building lasting value and doing it the right way. We learned more about them, too, and especially USIT’s founder, Thomas Tull. The more we each learned about the other, the more there was to like. And importantly, USIT had the vision for us that we had for ourselves: that there was a big, important market here — and that it was uniquely served by Oxide. We are elated to announce this new, exciting phase of the company. It’s not necessarily in our nature to celebrate fundraising, but this is a big milestone, because it will allow us to address our customers' most pressing questions around scale (manufacturing scale, system scale, operations scale) and roadmap scope. We have always believed in our mission, but this raise gives us a new sense of confidence when we say it: we’re going to kick butt, have fun, not cheat (of course!), love our customers — and change computing forever.

3 days ago 11 votes