Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
37
It's been almost 2 years since I moved to a team lead role, then to a full-time engineering management position after the expansion of our team. I've been a front-end developer for 7 years before that, and initially I took the "advanced individual contributor" career track before doing the management turnaround. How's it been? Bumpy, but fun. In this article, I'll share the things I love and hate about my current job. Love Let's start with the positive side of management positions. There's plenty to love, honestly. Impact First things first, I adore the power to improve the product we're building, and the overall team well-being that comes with a management position. As an engineer, you'll sometimes find yourself in a tough spot with little to no power to change things. Early morning standups are a chore? Code quality sucks? The new feature makes no sense? As a manager, the power to change is yours: you get both the formal and informal authority to change things for the better, and to...
a year ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from Vladimir Klepov as a Coder

Growing my team 4x has been a pain. Can we do better?

My name is Vladimir, and I'm an engineering manager of a team building a banking app. Following the success of our core banking product, we've decided to expand to other financial services. In the last four months my team has grown 4x, going from a 4-person team to 4 teams totalling 15 people. It was, frankly, a shitshow, and now I see many things we could have done better. At first sight, increasing your team is a perfect way to speed up your product development. In practice, scaling is very challenging. Today, I'll share the pains of growth we've ran into: Disbalanced growth across functions. Teams becoming too large to manage. Processes breaking down. Mass onboarding challenges. Seniority skew. Now that it's all done, I can't say these problems were unexpected, or that our solutions have been incredibly inventive, but a first-hand account is worth writing down. Whether you're a leader expecting your team to scale, or just curious about the daily challenges of an engineering manager, hope you'll find something for yourself. Let's go! Scale evenly across functions You lead a 5-man engineering team. You'd like to build 2x the stuff you're building. The obvious solution is to hire 5 more engineers. Problem solved? Not so fast. Anything you build flows across several stages, each handled by a certain role, for example: product manager -> design -> engineering -> QA -> product analytics -> (back to) product manager. If you have a balanced flow with your current team, enlarging just the engineering will not make you more productive: Downstream functions (QA / analytics) can't keep up with the increased production — they must either put in overtime or downgrade quality standards. Upstream functions (product / design) can't fill the backlog fast enough, and the eng team has nothing to focus on, either slacking or refactoring the refactorings. With rapid growth, some temporary disbalances are unavoidable. It's fine to spend a few weeks or months in a disproportionate state, but overall aim for balanced scaling. Here are some tips to smooth the transition and even use the disbalance to your advantage: If you have any control over it, give product & design a hiring head-start. You can get their artifacts ready for development with a limited series of grooming meetings, and once the new engineers are on board, you'll have some great useful tasks to feed them. Catch up on your tech debt. Have too many engineers and not enough product tasks? Don't despair: use the time to clean up some old bugs, do the overdue refactorings, and prepare the codebase for the speed-up. Expand the area of responsibility. If you lack QA specialists, it might be time for the engineers to practice their testing skills. Oversized non-technical component (product / design) is more problematic, but you can give them some no-code tools to replace the eng team in some scenarios — e.g. build an admin UI where PM can edit the texts, create new banners, and so on, without involving your team. Update team structure Say you have a normal-sized team (4–7 engineers) with your average meeting structure (whole-team planning, grooming, and retro + weekly 1x1 with every team member). Making it a 15-person team won't work at all. An hour-long 4-person retro has 10 minutes of speaking time per member — enough to make a point. In a 15-person retro, it's 3 minutes — not practical. You either exclude some people, or extend the meetings — both poor options. It's harder to agree on any decision, because you now have 3x the possible objections. The 1x1s alone eat up 7 hours (almost a full day!) of your time a week. Managing communication of 15 people plus all the external stakeholders is time-consuming. The team must be split. What does it mean to be a separate team, anyways? Ever heard of "high cohesion and low coupling" principle in software architecture? I think this also applies to teams: Shared information space. Team members know what you're working on, where you're going, who your peers are, the system structure and so on. Own a well-defined part of the product and the codebase. A look at a random feature is enough to guess the responsible team. Control your processes — meeting structure, releases, etc. You can't be responsible for what you can't control. Have all the capabilities needed for your day-to-day work. Begging someone each time you need to deploy, change the API or add a banner is not very effective. It's best to split by product domain: customer acquisition team owns the landing page and signup, daily banking team owns the main app, and so on. You could split by layer (product + infra) or by function (backend + frontend + mobile) — I feel these compromise points 2 and 4, but let's not die on this hill today. At any rate, a team over 8–10 people must be split into sane-sized chunks to keep going. A note on grouping You can split a team by (A) building the new team out of newbies or (B) mixing newbies and oldies in each new part. Prefer mixing: it distributes the knowledge across the organization, and the social connections from the original team prevent siloing. You could argue that (A) keeps the original high-performing team intact, but it does so by slowing down the new team and undermining your long-term flexibility. Where to get new leaders? Splitting a team into 3 parts creates 2–4 leadership positions, depending on your place in the new structure. Ideally, you have senior members of the original team to lead the new teams, because it's a rare opportunity for career growth on a management track, and they can easily hire and onboard new members of their teams. If you don't have a suitable candidate (everyone is either very junior, or hates management), it's fine to hire externally — following a few hiccups, I recommend hiring people with prior leadership experience, because adjusting to a new product and a new role at once can be too much. Split iteratively You don't have to produce a fully separate team right away. As usual, move step by step — you get faster results, and can adapt to the issues that arise. Here's one possible sequence: Assemble a domain team, appoint one as the lead (you can call it trial-lead, to give them a chance to cop out). See how they like their new roles, and if the headcount needs tweaking. Run a retro for the new team to catch communication issues or cross-team dependencies early. Separate the domain backlog and kanban boards (or wherever you track the tasks). You'll need it for further process splitting, and to assess the team's load and velocity. Split planning and daily meetings, so that the teams don't waste each other's time on discussing irrelevant tasks. Gradually transfer the remaining processes (1x1s, onboarding, postmortems) to the new lead. Split the codebase, so that the new team can fully own its service. Update your processes Just like your single-team structure, the processes you have will likely fail for a larger team, especially if you have many newbies. Example: we had a liberal release process — if you want some feature in production, you deploy it. As the team grew, the release frequency dropped — the newbies were afraid to touch prod, the oldies were hoping one of the other 14 people would do it. Before I give you my solution to this puzzle, let's look at the general advice for process scaling. Localize processes to the new teams. Owning processes makes a team more effective. Retros, plannings, kickoffs, demos, daily stand-ups, releases, documenting, on-call duty, whatever you can split, do split. Yes, your overall team loses sight of the stuff going on across the system, causing duplication and poor decisions, but in return we can focus on a specific business area, and maximize productive time instead of drowning in discussions. If you need to offset the downsides, introduce a cross-team sync here and there (still working on this one). Let the new teams experiment with processes. What worked for your original team doesn't matter, that team is dead and gone. What works for one of your teams won't necessarily work for the other, because they work in different conditions. For example, our CA team has many time-bound tasks from marketing. The core banking team focuses on building quality software, and fixing the bugs as they arise. Very different teams. Start with a copy of your current processes (just to start somewhere!), and introduce team retros as early as possible to tweak the process as needed. Stricten the centralized processes. Back to our problem with deployment — we couldn't isolate the release process to sub-teams, because splitting a monolithic front-end into independently deployable parts is technically challenging. We introduced a more structured release process: The releases are automatically built and ready to deploy every morning. No more decisions to make. Daily rotation of release managers responsible for getting the release to production. With regular training, you get better at releasing. The release process is clearly documented. The newbies have a clear path to follow, making it less stressful. This works for other centralized processes — writing documentation, debugging with customer support, maintaining shared libs. Overall: hand over as much process as you can to the sub-teams, and introduce clear rules for the remaining centralized processes. The onboarding valley Surprisingly, fast hiring can reduce your team's productivity in the short term. The newbies are not yet up to speed, and the oldies now spend time explaining your codebase and reviewing code. This will fix itself over time, but here are a few strategies to get past the bump faster. Prefer slower growth. Adding one person every few weeks is much better than adding 6 people simultaneously, because: The "onboarding load" stretched over time occupies a smaller share of your team's resource. A few weeks in, new hires can already help onboard someone else. In some cases, they'll do a better job than any oldie, because their memories of one-off tasks like setting up the dev environment are fresh. Every onboarding exposes new roadblocks in your process, helping you smooth the next ones. On paper, batch onboarding might seem like a time-saver, as you can make a lecture explaining the basics to many people at once. In practice, unless your product is very small, or the tasks are very repetitive, every newbie faces very different challenges, drowning you with a wave of questions. Encourage peer-to-peer onboarding. As a leader, you might think onboarding is your personal responsibility. I call BS — peer-to-peer onboarding is clearly better: More "onboarding resource" leads to faster, better onboarding. The load on you, personally, decreases, freeing time to do other impactful things and, you know, live. Team members get a safe environment to practice their mentorship skills. People get to know each other, instead of only talking to you. ICs with recent hands-on experience do a better job at explaining the specifics than you. You can pair a formal "mentor" to every newbie, or direct questions to a team group chat. If you want to control the overall onboarding, at least route specific questions to team members experienced in that area instead of trying to come up with all the answers yourself. Write the docs. The best way to make onboardings cheaper is writing stuff down instead of explaining it over and over again, with your mouth. Some particular things to focus on: Onboarding checklist — the things every new team member must do: get a VPN certificate from the security dude, join this and that chat, clone a repo here and there, boom you're done. Only include essential steps — adding somewhat useful stuff obscures the actually important things. Document your existing business processes, system architecture, technical conventions, team and communication structure. It's better than explaining in real-time, because you get higher-quality charts, relevant links, and you can collaborate to put the knowledge of multiple team members in one place. Tooling and automation. The more automated a process or convention is, the less onboarding you need. Example: if you build your releases locally and upload somewhere via FTP using the keys you get from Piotr the devops, it's time to set up decent CD instead of documenting the current state of affairs. Pro tip: encourage newbies to improve and update the docs as they follow along — it's a great first contribution to your team! Senior vs junior hires It seems sensible to focus exclusively on senior hires. Experienced engineers get up to speed quicker, because they're already familiar with the basics, have a lower risk of making catastrophically poor decisions, and can bring good practices and ideas from across the industry to your team. Not so fast — here are some reasons to hire junior developers. With little exposure to the industry, they can easily adapt to whatever culture and processes you have. Anyone on your team can mentor a junior hire, while getting e.g. a junior engineer with decent knowledge of the product to mentor a newly hired senior engineer might be awkward, not very productive, or even taken as an insult. And of course, you can hire more junior engineers on the same budget. Overall, aim for a balanced team composition in the mid-term. You don't want your team to be a kindergarten, but a nursing home is no good either. Remember that people tend to gain experience, so the junior engineers you hire will become middle in no time. Today, we've discussed the challenges of rapid team growth — and ways to address them: Hiring more engineers won't speed you up unless product, design and QA grow to match. Start by growing product & design. If product lags behind, use the spare time to clean up the tech debt. Expand the area of responsibility of the oversized roles. Teams over 8–10 people are hard to manage. Split into chunks of 3–7 people, preferably by business domain. Mix old and new members in each team. Split step-by-step instead of going all in. The processes of your original team won't accommodate a larger team. Localize the processes to the sub-teams as much as possible. Let the teams tweak their processes to suit their needs. Stricten the remaining centralized processes. Onboarding is time-consuming, and can slow you down. Go slow: onboarding a person every week is easier than 6 people at once. Write the docs instead of explaining stuff over and over. Peer-to-peer knowledge transfer is better than onboarding everyone personally. Hiring only senior engineers is not a silver bullet. Aim for a healthy experience distribution in the mid-term. Hope these tips help you get past the scaling issues and up to speed in no time.

a year ago 41 votes
The most useful programming language

Aspiring developers often ask me what's the best programming language to learn. Personally, I mostly work with JS — solid choice, but everyone and their dog learns JS these days, so it might be time to add some diversity. I'm curious — which single programming language covers the most bases for you, and gives you most career opportunities for years to come? That's the question we'll try to answer today. Here's the plan. I made a list of 8 tech specializations: 2 web development areas: back- and front-end. Both pretty big areas, and ones I have most experience with. Mobile and desktop native app development. Native app development (especially desktop apps) seems to have fallen out of favor, but there's still enough work in these areas. Quality assurance automation. QA grows along with engineering, and increasingly relies on automated tests. Embedded systems. We'll focus on microcontroller programming, not fat boxes with a full windows / linux OS. Quite a promising area with the growth of IoT. Game development. Granted, I don't know much about this area, but I'll do my best to cover it as well, as many developers dream of building a fun game someday. Data analysis and Machine Learning. One of the most hyped areas of the last decade. The contenders are the usual suspects from TIOBE top 20: python, C, C++, Java (grouped with Kotlin and other JVM languages), C# (again, throw in VB and other .NET languages), JavaScript (and TypeScript), PHP, Go, Swift, Ruby, Rust. I left out SQL and Scratch, because they're not general-purpose languages, and Fortan with Matlab, because they aren't really used outside of scientific / engineering computing. A language scores 1 point by being the industry standard in the area — vast community and ecosystem, abundant jobs. Being useful for certain tasks in the area gets you 0.5 points. So, let's see what languages will make you the most versatile engineer, shall we? Backend Let's start with the simple one — Java, C#, Python, PHP, Go and Ruby are all excellent back-end programming languages. Of these, I'd say PHP is slightly more useful as many low-code solutions rely on it, and Ruby is steadily declining. Still, all these languages have earned 1 point. Next, 0.5 points go to: C++, used in high-load and time-critical scenarios, JS — node.js is often used to support front-end, but there aren't that many strictly back-end jobs for JS developers. Rust — still not that widely used, but growing fast. The only languages to fail here are Swift (technically usable on server via e.g. vapor, but I couldn't find any jobs in this stack) and C. Frontend Obviously, JavaScript is the language for front-end developers, which runs natively in browsers. But, surprise, other languages still qualify! All solid back-end languages (Java, C#, Python, PHP, Go, Ruby) get 0.5 points, because you can solve many UI problems by rendering HTML server-side the old-school way. C# has a slight edge here, since blazor is quite smart and popular. C, C++, and Rust score 0.5 points because they can be compiled to WebAssembly and run in the browser — just look at figma. Rust also powers some cool JS tooling, like biome and swc The only language to fail here is, again, Swift. QA automation The topic of QA automation is really simple. Java and python get the cake — Allure, Selenium, JUnit, and pytest are the most sought-after automation tools on the market right now. JS gets 0.5 points for playwright and cypress — the preferred tools for testing complex web front-ends. A few automation tools support C# — worth 0.2 points. Mobile apps Another straightforward area. Android apps are written in JVM languages (Java / Kotlin), iOS is integrated with Swift (finally). JS scores 0.5 points, because you can effectively build apps with React Native, and you can get pretty far with PWA or a good old WebView. Another 0.5 point for C#, thanks to Xamarin and MAUI. Desktop apps (windows / linux / MacOS) The three kings here are C++, C#, and Java. JS gets 0.5 points, again, for electron — disgusting or not, it's widely used. Another 0.5 points for Swift, because that's what you build MacOS apps with, but MacOS computers are relatively niche. Rust has the highly-hyped Tauri project for building desktop apps, but it's not that widespread, and I'm not aware of any high-profile apps using it. Let's give each 0.2 points for the effort and check back later. Embedded systems Embedded systems are usually tight on resources, so compiled languages are the way to go here. Basically any embedded job requires C and C++. Rust is, as usual, very promising, but not that popular yet, so 0.5 points. Another half-point for Python — used for edge computer vision and prototyping, but struggling with high memory requirements. Game development The primary languages in big gamedev are C++ (used in Unreal Engine) and C# (for Unity). Since mobile games are a thing, Java and Swift get 0.5 points each, because that's what you'll likely use here. Another 0.5 points for JS (browser games). Rust should be quite a good fit for games, but (as expected by now) it's not quite there yet. Data Analysis & Machine Learning It's no secret that Python is the language of choice for anything data-related, and most of the cutting edge stuff happens, well deserved 1 point here. But do you know there's another top language to get your piece of Data & ML hype? Big companies have a lot of data, right? And big companies love Java. So, many big data tools (especially coming from Apache — Hadoop, Spark, Jena) work with Java, and most data jobs require experience with python or java, so another 1 point for java. On to more surprises. Large chunks of data-heavy python libraries are actually written in C / C++ — e.g. over a third of numpy, or most of LlamaCPP — which earns both half-a-point. As you'd expect, Rust is also gaining traction for this use case with stuff like pola.rs, so another 0.2 points! The final half-a-point goes to JS for powering much of the UI / visualization stuff (see e.g. bokeh). Before we reveal our final ranking, let's weigh the categories, because they're not the same size. I've used some back-of-the-napkin analysis of job postings and sizing of reddit / linkedin groups and my personal experience. With backend as our reference, I'd say frontend is roughly the same size. Mobile development is surprisingly sizable — let's give it a 0.6 weight. For QA, I'd say 0.2 makes sense, as 1 QA per 3–5 devs is a normal ratio, and manual QA is still a thing. Desktop is easily the smallest area, looks like a 0.1 to me. For gamedev, 0.5 is just my random guess. Finally, there are surprisingly many data people — with the good salaries, let's make it a 0.6. Putting it all together: Java takes the first spot by a good margin by topping 5 categories, and having some gamedev / frontend capabilities. Place other JVM languages (especially Kotlin) around here, but with a discount since they're not as widely used. The next three are really close, but JS gets slighly ahead by being average at everything except embedded, even though it's only the top choice for front-end development. Python and C# tie for the third place. Both are top-tier backend languages with other strong areas (QA / ML for python, desktop and gamedev for C#). C++ is not that far behind either, as it's still the top language when it comes to efficiency. It also steps into other languages' realms when they need some speedup (WebAssembly / ML). Next come "three backend friends" — Go, PHP, and Ruby. All top-notch languages for building web backends, but not much else beyond that. Of these, Ruby is on the decline, and PHP and Go both have their separate niches. Rust does not score that well, but still makes it into the top 10 — not bad for such a new language. It has great growth potential by eating at the traditional C++ areas, super excited to see where it gets in 3–5 years. We all love good old C, but C++ looks like a better fit for complex systems. Swift comes in last — fair enough for a language that's only useful for the products of one single company. Perhaps surprisingly, the single most useful language is Java. Python and JS, beginner favorites, come strong, with a very different focus. C# perhaps deserves a bit more attention. Overall, today we've learnt about many amazing technologies that allow languages to sneak into each other's territory. If you were to start anew, what language would you learn?

a year ago 34 votes
I conducted 60 interviews in 2 months — here's what I learned

It's hard to believe, but, starting mid-october 2023 I conducted 60 technical interviews and hired 10 people into our team. It's been extremely tiring: around 80 hours of active interviewing, plus writing interview reports, plus screening CVs and take-home assignments, plus onboarding new members — all while doing my normal work stuff. Still, I feel like I learnt a lot in the process — things that would help me as a candidate in the future, and might help you land your next job. Note that I'm a fairly relaxed interviewer, and, as an internal startup of a large tech company, we generally have a more humane hiring process, so your mileage may vary. Still, I've done my best to pick the tips that I feel are universally applicable. Here are nine insights I took out of this experience, in no particular order: Be generous with your "expected income". Say you're a solid higher-middle engineer, and you ask for a senior salary. My thought process: OK buddy, it's a bit more than reasonable now, but I won't have to fight for your promotion 8–12 months from now when you get there, and I don't have to spend another 12 hours of my own time (and leave my team understaffed for another few weeks) looking for a real hardcore senior, so I'll let you have it. Now suppose you ask for a junior salary. It's suspicious — why is your bar so low? Is there someting about your work performance you're not telling us? So, do your research on reasonable salaries for your level of experience, and aim slightly above that. Ask the right questions. I always leave time for the candidate to ask me questions — obviously, this lets the candidate probe what it's like to work at our team, but it's also the best opportunity for me to learn what really matters to the candidate. I've never been much of an asker myself, but now I see that "Thanks, I have no questions" does not look good — if anything, it paints you as someone who doesn't care. Here's a short list of good questions: What does the daily work in this role look like? Harmless. What features are you building next? Caring about the overall product, nice. Sometimes the answer is "I can't disclose this secret", but not that often. Anyting about processes or team structure: how many people are on the team? How often do you release? What regular meetings do you have? Interested in organization, might want to be a team lead someday, great. Anything tech-related: which framework do you use? Why did you pick framework X? How do you test your app? Especially suitable for junior- to middle developers who are most involved in hands-on work. What kind of tasks do you see me doing? Again, just a good neutral question, because responsibilities for any role differ wildly between companies. What growth / promotion opportunities does this position have? Cool trick, flipping the feared "where do you see yourself in 5 years" question against the hiring manager. Here are a few questions that are not very good: Do you use jira and github? It's a minor detail, won't you be able to work with youtrack and gitlab? Do you sometimes work late? Only if something breaks, but overall this question makes you seem a bit lazy. People on poor teams that routinely overtime aren't likely to answer this question honestly, at any rate. Social skills matter. I understand that not everybody is super outgoing, but if we already feel awkward 1 hour into our acquaintance, why work together — to feel awkward for months to come? Just a few tips anyone can follow: Be energetic. You're tired, I'm tired, we're all tired of endless interviews. Are you just tired today, or generally always too tired to get anything done? I know it's easier said than done, but try and show me all the energy you have left. Show respect. People enjoy being respected. Very easy one: you have a great product. Sounds like you have a great engineering culture. This is one of the most interesting interviews I've ever seen. Like, I know you don't necessarily mean that, but subconsciously I'm very pleased: "oh yes, I'm very proud of my interview process, thanks for noticing" On a related note... Provide conversation opportunities. Q: Do you use TDD? Bad answer: "no". Good answer: "no, but I've heard of it. Interesting approach. Does your team use TDD?" Now you get to spend 5 minutes talking on your terms instead of being bombarded with random questions, and you come off as someone curious about stuff. On another related note... It's easy to hurt people. People normally ask you about stuff because they care about it. So, again, the interviewer askning "do you use TDD?", presumably, likes TDD and uses it. So, the worst answer: "no, TDD sucks, it's pure waste of time for idiots." A rare interviewer might appreciate you having a strong opinion on a topic, but to most this just paints you as a jerk, kinda like "Here's a photo of my children — "I hate children, and yours are especialy horrible". Not smart. Smart talk is not your friend. Saying stuff like "our front-end guild evaluated several cutting-edge approaches to testing universal applications" only makes you seem smart if you can elaborate on that topic: what these approaches were, the pros and cons you found, what tradeoffs you made for your final decision. If you can't answer a follow-up question beside "we settled on jest, not sure why", it was better to stay away from that topic altogether. Related: "in code reviews, I always consider the optimality of the algorithm selected" (proceeds to estimate the time complexity of comparison-based sorting as O(1). I never ask this unless the candidate boasts about her algo skillz). Admit your mistakes. Don't know an answer? Your code has a bug? It's always better to admit it and then try to come up with something at the spot than trying to talk your way out of it. Event loop? Sure thing, I'm an expert on loops. It's the way events are looped. Uses logarithmic weighing. Again, this makes you look like a candidate with big mouth and small hands. I have seen a couple of people who could talk their way out of any situation, but I honestly think with such skills you'd do better in a different line of work, like international relations, or selling financial services. Note that you really should give it your best shot — giving up at the first sign of trouble is not a good impression. If you genuinely have no idea — see conversation opportunity: "Event delegation? Tough luck, never heard of it. Would you tell me about it so that I learn something new today?" Make yourself memorable. It's hard to keep detailed profiles of 10 candidates in mind — after a good interview streak all I remember is the general impression (great / OK / horrible) and a few truly notable things. This guy worked for some crypto scam that went bust, that girl had a cute dog that was trying to eat the camera. The worst you can do is be a totally neutral candidate — we've had an interview, but I can't remember any details. So try and sneak some anecdote, or wear a silly scarf — something to remember. This point is especially important for intern / junior positions — online JS bootcamps do a good job of covering the basics, and it's really hard to differentiate these candidates. The memorable thing doesn't have to be professional, or even positive (even though it sure won't hurt) — your best bet would be some original personal project. Ask for feedback on the spot. Asking how you did at the end of the interview doesn't hurt. Yes, some interviewers will be hesitant to answer — at large companies, the feedback is normally sent through the recruiter, and you're never sure if sidestepping this process would get you into trouble. Besides, if the feedback is not complimentary, you're essentially asking for conflict at the spot, and people normally avoid conflict when possible. Still, it's a chance to adjust your expectations (if the interviewer says, looking you in the eyes, that you've done great, it's a good sign), and you might get actually useful tips that would probably get lost passing through the written report, and then through the non-technical recruiter.

a year ago 33 votes
Svelte stores: the curious parts

We've already learnt a lot about svelte's reactivity system — the primary way to work with state in svelte components. But not all state belongs in components — sometimes we want app-global state (think state manager), sometimes we just want to reuse logic between components. React has hooks, Vue has composables. For svelte, the problem is even harder — reactive state only works inside component files, so the rest is handled by a completely separate mechanism — stores. The tutorial does a decent job of covering the common use cases, but I still had questions: What's the relationship between the stores? Are they built on some common base? Is it safe to use { set } = store as a free function? How does get(store) receive the current value if it's not exposed on the object? Does set() trigger subscribers when setting the current value? What's the order of subscriber calls if you set() inside a subscriber? Does derived listen to the base stores when it's not observed? Will changing two dervied dependencies trigger one or two derived computations? Why does subscribe() have a second argument? What is $store sytax compiled to? In this article, I explore all these questions (and find a few svelte bugs in the process). writable is the mother store Svelte has 3 built-in store types: writable, readable, and derived. However, they are neatly implemented in terms of one another, taking only 236 lines, over half of which is TS types and comments. The implementation of readable is remarkably simple — it creates a writable, and only returns its subscribe method. Let me show it in its entirety: const readable = (value, start) => ({ subscribe: writable(value, start).subscribe }); Moreover, derived is just a special way of constructing readable: export function derived(stores, fn, initial_value) { // ...some normalization return readable(initial_value, /* some complex code */); } While we're at it, note that update method of a writable store is a very thin wrapper over set: fn => set(fn(value)). All in all: writable is the OG store, readable just removes set & update methods from a writable, derived is just a predefined readable setup, update is just a wrapper over set. This greatly simplifies our analysis — we can just investigate writable arguments, subscribe, and set — and our findings also hold for other store types. Well done, svelte! Store methods don't rely on this Writable (and, by extension, readable and derived) is implemented with objects and closures, and does not rely on this, so you can safely pass free methods around without dancing with bind: const { subscribe, set } = writable(false); const toggle = { subscribe, activate: () => set(true) }; However, arbitrary custom stores are not guaranteed to have this trait, so it's best to stay safe working with an unknown store-shaped argument — like svelte itself does with readonly: function readonly(store) { return { subscribe: store.subscribe.bind(store), }; } Subscriber is invoked immediately As svelte stores implement observable value pattern, you'd expect them to have a way to access current value via store.get() or store.value — but it's not there! Instead, you use the special get() helper function: import { get } from 'svelte/store' const value = get(store); But, if the store does not expose a value, how can get(store) synchronously access it? Normally, the subscribers are only called on change, which can occur whenever. Well, svelte subscribe is not your average subscribe — calling subscribe(fn) not only starts listening to changes, but also synchronously calls fn with the current value. get subscribes to the store, extracts the value from this immediate invocation, and immediately unsubscribes — like this: let value; const unsub = store.subscribe(v => value = v); unsub(); The official svelte tutorial section on custom stores says: as long as an object correctly implements the subscribe method, it's a store. This might bait you into writing "custom stores" with subscribe method, not based off of writable. The trick word here is correctly implements — even based on the tricky subscribe self-invocation it's not an easy feat, so please stick to manipulations with readable / writable / derived. set() is pure for primitives writable stores are pure in the same sense as svelte state — notifications are skipped when state is primitive, and the next value is equal to the current one: const s = writable(9); // logs 9 because immediate self-invocation s.subscribe(console.log); // does not log s.set(9); Object state disables this optimization — you can pass a shallow equal object, or the same (by reference) object, the subscribers will be called in any case: const s = writable({ value: 9 }); s.subscribe(console.log); // each one logs s.update(s => s); s.set(get(s)); s.set({ value: 9 }); On the bright side, you can mutate the state in update, and it works: s.update(s => { s.value += 1; return s }); Subscriber consistency Normally, store.set(value) synchronously calls all subscribers with value. However, a naive implementation will shoot you in the foot when updating a store from within a subscriber (if you think it's a wild corner case — it's not, it's how derived stores work): let currentValue = null; const store = naiveWritable(1); store.subscribe(v => { // let's try to avoid 0 if (v === 0) store.set(1); }) store.subscribe(v => currentValue = v); If we now call set(0), we intuitively expect both the store's internal value and currentValue to be 1 after all callbacks settle. But in practice it can fail: Store value becomes 0; First subscriber sees 0, calls set(1), then: Store value becomes 1; set(1) synchronously invokes all subscribers with 1; First subscriber sees 1, does nothing; Second subscriber is called with 1, sets currentValue to 1; First subscriber run for 0 is completed, continuing with the initial updates triggered by set(0) Second subscriber is called with 0, setting currentValue to 0; Bang, inconsistent state! This is very dangerous territory — you're bound to either skip some values, get out-of-order updates, or have subscribers called with different values. Rich Harris has taken a lot of effort to provide the following guarantees, regardless of where you set the value: Every subscriber always runs for every set() call (corrected for primitive purity). Subscribers for one set() run, uninterrupted, after one another (in insertion order, but I wouldn't rely on this too much). Subscribers are invoked globally (across all svelte stores) in the same order as set calls, even when set calls are nested (called from within a subscriber). All subscribers are called synchronously within the outermost set call (the one outside any subscriber). So, in our example, the actual callback order is: subscriber 1 sees 0, calls set(1) subscribers calls with 1 are enqueued subscriber 2 sets currentValue = 0 subscriber 1 runs with 1, does nothing subscriber 2 sets currentValue = 1 Since the callback queue is global, this holds even when updating store B from a subscriber to store A. One more reason to stick with svelte built-in stores instead of rolling your own. Derived is lazy derived looks simple on the surface — I thought it just subscribes to all the stores passed, and keeps an up-to-date result of the mapper function. In reality, it's smarter than that — subscription and unsubscription happens in the start / stop handler, which yields some nice properties: Subscriptions to base stores are automatically removed once you stop listening to the derived store, no leaks. Derived value and subscriptions are reused no matter how many times you subscribe to a derived store. When nobody is actively listening to a derived store, the mapper does not run. The value is automatically updated when someone first subscribes to the derived store (again, courtesy of subscribe self-invocation). Very, very tastefully done. Derived is not transactional While lazy, derived is not transactional, and not batched — synchronously changing 2 dependencies will trigger 2 derivations, and 2 subscriber calls — one after the first update, and one after the second one. In this code sample, we'd expect left + right to always be 200 (we synchronously move 10 from left to right), there's a glimpse of 190 (remember, the subscribers are synchronously called during set): const left = writable(100); const right = writable(100); const total = derived([left, right], ([x, y]) => { console.log('derive', x, y); return x + y; }); total.subscribe(t => console.log('total', t)); const update = () => { // try to preserve total = 200 left.update(l => l - 10); // ^^ derives, and logs "total 190" right.update(r => r + 10); // ^^ derives, and logs "total 200" }; This isn't a deal breaker, svelte won't render the intermediate state, but it's something to keep in mind, or you get hurt: const obj = writable({ me: { total: 0 } }); const key = writable('me'); const value = derived([obj, key], ([obj, key]) => obj[key].total); // throws, because { me: ... } has no 'order' field key.set('order'); obj.set({ order: { total: 100 } }); The mysteryous subscriber-invalidator Looking at subscribe() types, you may've noticed the mysterious second argument — invalidate callback. Unlike the subscriber, it's not queued, and is always called synchronously during set(). The only place I've seen an invalidator used in svelte codebase is inside derived — and, TBH, I don't understand its purpose. I expected it to stabilize derived chains, but it's not working. Also, the TS types are wrong — the value is never passed to invalidator as an argument. Verdict: avoid. $-dereference internals As you probably know, svelte components have a special syntax sugar for accessing stores — just prefix the store name with a $, and you can read and even assign it like a regular reactive variable — very convenient: import { writable } from 'svelte/store'; const value = writable(0); const add = () => $value += 1; <button on:click={add}> {$value} </button> I always thought that $value is compiled to get, $value = v to value.set(v), and so on, with a subscriber triggering a re-render in some smart way, but it's not the case. Instead, $value becomes a regular svelte reactive variable, synchronized to the store, and the rest is handled by the standard svelte update mechanism. Here's the compilation result: // the materialized $-variable let $value; // the store const value = writable(0); // auto-subscription const unsub = value.subscribe(value, value => { $$invalidate(0, $value = value) }); onDestroy(unsub); const add = () => { // assign to variable $value += 1; // update store value.set($value); }; In plain English: $store is a real actual svelte reactive variable. store.subscribe updates the variable and triggers re-render. The unsubscriber is stored and called onDestroy. AFAIK, store.update is never used by svelte. Assignments to $store simultaneously mutate $store variable without invalidating and triggering re-render and call store.set, which in turn enqueues the update via $$invalidate The last point puts us in a double-source-of-truth situation: the current store value lives both in the $store reactive variable, and inside store itself. I expected this to cause some havok in an edge case, and so it does — if you patch store.set method to skip some updates, the $-variable updates before your custom set runs, and the two values go out of sync as of svelte@3.59.1: const value = { ...writable(0), // prevent updates set: () => {} }; const add = () => $value += 1; let rerender = {}; $: total = $value + (rerender ? 0 : 1); {total} <button on:click={add}>increment</button> <button on:click={() => rerender = {}}> rerender </button> To summarize: Both readable and derived are built on top of writable — readable only picks subscribe method, derived is a readable with a smart start / stop notifier. Built-in stores don't rely on this, so you can safely use their methods as free functions. Calling subscribe(fn) immediately invokes fn with the current value — used in get(store) to get the current value. Calling set() with the current value of the store will skip notifying subscribers if the value is primitive. set() on object state always notifies, even if the object is same, by reference, as the current state. The subscribers for a single set() run after one another. If a subscriber calls set, this update will be processed once the first set() is fully flushed. derived only subscribes to the base stores and maps the value when someone's actively listening to it. When synchronously changing two dependencies of derived, the mapper will be called after the first change. There's no way to batch these updates. subscribe() has a second argument — a callback that's called synchronously during set(). I can't imagine a use case for it. $store syntax generates a regular svelte reactive variable called $store, and synchronizes it with the store in a subscriber. If you learn one thing from this article — svelte stores are thoughtfully done and help you with quite a few corner-cases. Please avoid excessive trickery, and build on top of the svelte primitives. In the next part of my svelte series, I'll show you some neat tricks with stores — stay tuned on twitter!

over a year ago 29 votes

More in programming

That boolean should probably be something else

One of the first types we learn about is the boolean. It's pretty natural to use, because boolean logic underpins much of modern computing. And yet, it's one of the types we should probably be using a lot less of. In almost every single instance when you use a boolean, it should be something else. The trick is figuring out what "something else" is. Doing this is worth the effort. It tells you a lot about your system, and it will improve your design (even if you end up using a boolean). There are a few possible types that come up often, hiding as booleans. Let's take a look at each of these, as well as the case where using a boolean does make sense. This isn't exhaustive—[1]there are surely other types that can make sense, too. Datetimes A lot of boolean data is representing a temporal event having happened. For example, websites often have you confirm your email. This may be stored as a boolean column, is_confirmed, in the database. It makes a lot of sense. But, you're throwing away data: when the confirmation happened. You can instead store when the user confirmed their email in a nullable column. You can still get the same information by checking whether the column is null. But you also get richer data for other purposes. Maybe you find out down the road that there was a bug in your confirmation process. You can use these timestamps to check which users would be affected by that, based on when their confirmation was stored. This is the one I've seen discussed the most of all these. We run into it with almost every database we design, after all. You can detect it by asking if an action has to occur for the boolean to change values, and if values can only change one time. If you have both of these, then it really looks like it is a datetime being transformed into a boolean. Store the datetime! Enums Much of the remaining boolean data indicates either what type something is, or its status. Is a user an admin or not? Check the is_admin column! Did that job fail? Check the failed column! Is the user allowed to take this action? Return a boolean for that, yes or no! These usually make more sense as an enum. Consider the admin case: this is really a user role, and you should have an enum for it. If it's a boolean, you're going to eventually need more columns, and you'll keep adding on other statuses. Oh, we had users and admins, but now we also need guest users and we need super-admins. With an enum, you can add those easily. enum UserRole { User, Admin, Guest, SuperAdmin, } And then you can usually use your tooling to make sure that all the new cases are covered in your code. With a boolean, you have to add more booleans, and then you have to make sure you find all the places where the old booleans were used and make sure they handle these new cases, too. Enums help you avoid these bugs. Job status is one that's pretty clearly an enum as well. If you use booleans, you'll have is_failed, is_started, is_queued, and on and on. Or you could just have one single field, status, which is an enum with the various statuses. (Note, though, that you probably do want timestamp fields for each of these events—but you're still best having the status stored explicitly as well.) This begins to resemble a state machine once you store the status, and it means that you can make much cleaner code and analyze things along state transition lines. And it's not just for storing in a database, either. If you're checking a user's permissions, you often return a boolean for that. fn check_permissions(user: User) -> bool { false // no one is allowed to do anything i guess } In this case, true means the user can do it and false means they can't. Usually. I think. But you can really start to have doubts here, and with any boolean, because the application logic meaning of the value cannot be inferred from the type. Instead, this can be represented as an enum, even when there are just two choices. enum PermissionCheck { Allowed, NotPermitted(reason: String), } As a bonus, though, if you use an enum? You can end up with richer information, like returning a reason for a permission check failing. And you are safe for future expansions of the enum, just like with roles. You can detect when something should be an enum a proliferation of booleans which are mutually exclusive or depend on one another. You'll see multiple columns which are all changed at the same time. Or you'll see a boolean which is returned and used for a long time. It's important to use enums here to keep your program maintainable and understandable. Conditionals But when should we use a boolean? I've mainly run into one case where it makes sense: when you're (temporarily) storing the result of a conditional expression for evaluation. This is in some ways an optimization, either for the computer (reuse a variable[2]) or for the programmer (make it more comprehensible by giving a name to a big conditional) by storing an intermediate value. Here's a contrived example where using a boolean as an intermediate value. fn calculate_user_data(user: User, records: RecordStore) { // this would be some nice long conditional, // but I don't have one. So variables it is! let user_can_do_this: bool = (a && b) && (c || !d); if user_can_do_this && records.ready() { // do the thing } else if user_can_do_this && records.in_progress() { // do another thing } else { // and something else! } } But even here in this contrived example, some enums would make more sense. I'd keep the boolean, probably, simply to give a name to what we're calculating. But the rest of it should be a match on an enum! * * * Sure, not every boolean should go away. There's probably no single rule in software design that is always true. But, we should be paying a lot more attention to booleans. They're sneaky. They feel like they make sense for our data, but they make sense for our logic. The data is usually something different underneath. By storing a boolean as our data, we're coupling that data tightly to our application logic. Instead, we should remain critical and ask what data the boolean depends on, and should we maybe store that instead? It comes easier with practice. Really, all good design does. A little thinking up front saves you a lot of time in the long run. I know that using an em-dash is treated as a sign of using LLMs. LLMs are never used for my writing. I just really like em-dashes and have a dedicated key for them on one of my keyboard layers. ↩ This one is probably best left to the compiler. ↩

22 hours ago 3 votes
AmigaGuide Reference Library

As I slowly but surely work towards the next release of my setcmd project for the Amiga (see the 68k branch for the gory details and my total noob-like C flailing around), I’ve made heavy use of documentation in the AmigaGuide format. Despite it’s age, it’s a great Amiga-native format and there’s a wealth of great information out there for things like the C API, as well as language guides and tutorials for tools like the Installer utility - and the AmigaGuide markup syntax itself. The only snag is, I had to have access to an Amiga (real or emulated), or install one of the various viewer programs on my laptops. Because like many, I spend a lot of time in a web browser and occasionally want to check something on my mobile phone, this is less than convenient. Fortunately, there’s a great AmigaGuideJS online viewer which renders AmigaGuide format documents using Javascript. I’ve started building up a collection of useful developer guides and other files in my own reference library so that I can access this documentation whenever I’m not at my Amiga or am coding in my “modern” dev environment. It’s really just for my own personal use, but I’ll be adding to it whenever I come across a useful piece of documentation so I hope it’s of some use to others as well! And on a related note, I now have a “unified” code-base so that SetCmd now builds and runs on 68k-based OS 3.x systems as well as OS 4.x PPC systems like my X5000. I need to: Tidy up my code and fix all the “TODO” stuff Update the Installer to run on OS 3.x systems Update the documentation Build a new package and upload to Aminet/OS4Depot Hopefully I’ll get that done in the next month or so. With the pressures of work and family life (and my other hobbies), progress has been a lot slower these last few years but I’m still really enjoying working on Amiga code and it’s great to have a fun personal project that’s there for me whenever I want to hack away at something for the sheer hell of it. I’ve learned a lot along the way and the AmigaOS is still an absolute joy to develop for. I even brought my X5000 to the most recent Kickstart Amiga User Group BBQ/meetup and had a fun day working on the code with fellow Amigans and enjoying some classic gaming & demos - there was also a MorphOS machine there, which I think will be my next target as the codebase is slowly becoming more portable. Just got to find some room in the “retro cave” now… This stuff is addictive :)

14 hours ago 2 votes
An Analysis of Links From The White House’s “Wire” Website

A little while back I heard about the White House launching their version of a Drudge Report style website called White House Wire. According to Axios, a White House official said the site’s purpose was to serve as “a place for supporters of the president’s agenda to get the real news all in one place”. So a link blog, if you will. As a self-professed connoisseur of websites and link blogs, this got me thinking: “I wonder what kind of links they’re considering as ‘real news’ and what they’re linking to?” So I decided to do quick analysis using Quadratic, a programmable spreadsheet where you can write code and return values to a 2d interface of rows and columns. I wrote some JavaScript to: Fetch the HTML page at whitehouse.gov/wire Parse it with cheerio Select all the external links on the page Return a list of links and their headline text In a few minutes I had a quick analysis of what kind of links were on the page: This immediately sparked my curiosity to know more about the meta information around the links, like: If you grouped all the links together, which sites get linked to the most? What kind of interesting data could you pull from the headlines they’re writing, like the most frequently used words? What if you did this analysis, but with snapshots of the website over time (rather than just the current moment)? So I got to building. Quadratic today doesn’t yet have the ability for your spreadsheet to run in the background on a schedule and append data. So I had to look elsewhere for a little extra functionality. My mind went to val.town which lets you write little scripts that can 1) run on a schedule (cron), 2) store information (blobs), and 3) retrieve stored information via their API. After a quick read of their docs, I figured out how to write a little script that’ll run once a day, scrape the site, and save the resulting HTML page in their key/value storage. From there, I was back to Quadratic writing code to talk to val.town’s API and retrieve my HTML, parse it, and turn it into good, structured data. There were some things I had to do, like: Fine-tune how I select all the editorial links on the page from the source HTML (I didn’t want, for example, to include external links to the White House’s social pages which appear on every page). This required a little finessing, but I eventually got a collection of links that corresponded to what I was seeing on the page. Parse the links and pull out the top-level domains so I could group links by domain occurrence. Create charts and graphs to visualize the structured data I had created. Selfish plug: Quadratic made this all super easy, as I could program in JavaScript and use third-party tools like tldts to do the analysis, all while visualizing my output on a 2d grid in real-time which made for a super fast feedback loop! Once I got all that done, I just had to sit back and wait for the HTML snapshots to begin accumulating! It’s been about a month and a half since I started this and I have about fifty days worth of data. The results? Here’s the top 10 domains that the White House Wire links to (by occurrence), from May 8 to June 24, 2025: youtube.com (133) foxnews.com (72) thepostmillennial.com (67) foxbusiness.com (66) breitbart.com (64) x.com (63) reuters.com (51) truthsocial.com (48) nypost.com (47) dailywire.com (36) From the links, here’s a word cloud of the most commonly recurring words in the link headlines: “trump” (343) “president” (145) “us” (134) “big” (131) “bill” (127) “beautiful” (113) “trumps” (92) “one” (72) “million” (57) “house” (56) The data and these graphs are all in my spreadsheet, so I can open it up whenever I want to see the latest data and re-run my script to pull the latest from val.town. In response to the new data that comes in, the spreadsheet automatically parses it, turn it into links, and updates the graphs. Cool! If you want to check out the spreadsheet — sorry! My API key for val.town is in it (“secrets management” is on the roadmap). But I created a duplicate where I inlined the data from the API (rather than the code which dynamically pulls it) which you can check out here at your convenience. Email · Mastodon · Bluesky

3 hours ago 2 votes
Implementation of optimized vector of strings in C++ in SumatraPDF

SumatraPDF is a fast, small, open-source PDF reader for Windows, written in C++. This article describes how I implemented StrVec class for efficiently storing multiple strings. Much ado about the strings Strings are among the most used types in most programs. Arrays of strings are also used often. I count ~80 uses of StrVec in SumatraPDF code. This article describes how I implemented an optimized array of strings in SumatraPDF C++ code . No STL for you Why not use std::vector<std::string>? In SumatraPDF I don’t use STL. I don’t use std::string, I don’t use std::vector. For me it’s a symbol of my individuality, and my belief in personal freedom. As described here, minimum size of std::string on 64-bit machines is 32 bytes for msvc / gcc and 24 bytes for short strings (15 chars for msvc / gcc, 22 chars for clang). For longer strings we have more overhead: 32⁄24 bytes for the header memory allocator overhead allocator metadata padding due to rounding allocations to at least 16 bytes There’s also std::vector overhead: for fast appends (push()) std::vectorimplementations over-allocated space Longer strings are allocated at random addresses so they can be spread out in memory. That is bad for cache locality and that often cause more slowness than executing lots of instructions. Design and implementation of StrVec StrVec (vector of strings) solves all of the above: per-string overhead of only 8 bytes strings are laid out next to each other in memory StrVec High level design of StrVec: backing memory is allocated in singly-linked pages similar to std::vector, we start with small page and increase the size of the page. This strikes a balance between speed of accessing a string at random index and wasted space unlike std::vector we don’t reallocate memory (most of the time). That saves memory copy when re-allocating backing space Here’s all there is to StrVec: struct StrVec { StrVecPage* first = nullptr; int nextPageSize = 256; int size = 0; } size is a cached number of strings. It could be calculated by summing the size in all StrVecPages. nextPageSize is the size of the next StrVecPage. Most array implementation increase the size of next allocation by 1.4x - 2x. I went with the following progression: 256 bytes, 1k, 4k, 16k, 32k and I cap it at 64k. I don’t have data behind those numbers, they feel right. Bigger page wastes more space. Smaller page makes random access slower because to find N-th string we need to traverse linked list of StrVecPage. nextPageSize is exposed to allow the caller to optimize use. E.g. if it expects lots of strings, it could set nextPageSize to a large number. StrVecPage Most of the implementation is in StrVecPage. The big idea here is: we allocate a block of memory strings are allocated from the end of memory block at the beginning of the memory block we build and index of strings. For each string we have: u32 size u32 offset of the string within memory block, counting from the beginning of the block The layout of memory block is: StrVecPage struct { size u32; offset u32 } [] … not yet used space strings This is StrVecPage: struct StrVecPage { struct StrVecPage* next; int pageSize; int nStrings; char* currEnd; } next is for linked list of pages. Since pages can have various sizes we need to record pageSize. nStrings is number of strings in the page and currEnd points to the end of free space within page. Implementing operations Appending a string Appending a string at the end is most common operation. To append a string: we calculate how much memory inside a page it’ll need: str::Len(string) + 1 + sizeof(u32) + sizeof(u32). +1 is for 0-termination for compatibility with C APIs that take char*, and 2xu32 for size and offset. If we have enough space in last page, we add size and offset at the end of index and append a string from the end i.e. `currEnd - (str::Len(string) + 1). If there is not enough space in last page, we allocate new page We can calculate how much space we have left with: int indexEntrySize = sizeof(u32) + sizeof(u32); // size + offset char* indexEnd = (char*)pageStart + sizeof(StrVecPage) + nStrings*indexEntrySize int nBytesFree = (int)(currEnd - indexEnd) Removing a string Removing a string is easy because it doesn’t require moving memory inside StrVecPage. We do nStrings-- and move index values of strings after the removed string. I don’t bother freeing the string memory within a page. It’s possible but complicated enough I decided to skip it. You can compact StrVec to remove all overhead. If you do not care about preserving order of strings after removal, I haveRemoveAtFast() which uses a trick: instead of copying memory of all index values after removed string, I copy a single index from the end into a slot of the string being removed. Replacing a string or inserting in the middle Replacing a string or inserting a string in the middle is more complicated because there might not be enough space in the page for the string. When there is enough space, it’s as simple as append. When there is not enough space, I re-use the compacting capability: I compact all existing pages into a single page with extra space for the string and some extra space as an optimization for multiple inserts. Iteration A random access requires traversing a linked list. I think it’s still fast because typically there aren’t many pages and we only need to look at a single nStrings value. After compaction to a single page, random access is as fast as it could ever be. C++ iterator is optimized for sequential access: struct iterator { const StrVec* v; int idx; // perf: cache page, idxInPage from prev iteration int idxInPage; StrVecPage* page; } We cache the current state of iteration as page and idxInPage. To advance to next string we advance idxInPage. If it exceeds nStrings, we advance to page->next. Optimized search Finding a string is as optimized as it could be without a hash table. Typically to compare char* strings you need to call str::Eq(s, s2) for every string you compare it to. That is a function call and it has to touch s2 memory. That is bad for performance because it blows the cache. In StrVec I calculate length of the string to find once and then traverse the size / offset index. Only when size is different I have to compare the strings. Most of the time we just look at offset / size in L1 cache, which is very fast. Compacting If you know that you’ll not be adding more strings to StrVec you can compact all pages into a single page with no overhead of empty space. It also speeds up random access because we don’t have multiple pages to traverse to find the item and a given index. Representing a nullptr char* Even though I have a string class, I mostly use char* in SumatraPDF code. In that world empty string and nullptr are 2 different things. To allow storing nullptr strings in StrVec (and not turning them into empty strings on the way out) I use a trick: a special u32 value kNullOffset represents nullptr. StrVec is a string pool allocator In C++ you have to track the lifetime of each object: you allocate with malloc() or new when you no longer need to object, you call free() or delete However, the lifetime of allocations is often tied together. For example in SumatraPDF an opened document is represented by a class. Many allocations done to construct that object last exactly as long as the object. The idea of a pool allocator is that instead of tracking the lifetime of each allocation, you have a single allocator. You allocate objects with the same lifetime from that allocator and you free them with a single call. StrVec is a string pool allocator: all strings stored in StrVec have the same lifetime. Testing In general I don’t advocate writing a lot of tests. However, low-level, tricky functionality like StrVec deserves decent test coverage to ensure basic functionality works and to exercise code for corner cases. I have 360 lines of tests for ~700 lines of of implementation. Potential tweaks and optimization When designing and implementing data structures, tradeoffs are aplenty. Interleaving index and strings I’m not sure if it would be faster but instead of storing size and offset at the beginning of the page and strings at the end, we could store size / string sequentially from the beginning. It would remove the need for u32 of offset but would make random access slower. Varint encoding of size and offset Most strings are short, under 127 chars. Most offsets are under 16k. If we stored size and offset as variable length integers, we would probably bring down average per-string overhead from 8 bytes to ~4 bytes. Implicit size When strings are stored sequentially size is implicit as difference between offset of the string and offset of next string. Not storing size would make insert and set operations more complicated and costly: we would have to compact and arrange strings in order every time. Storing index separately We could store index of size / offset in a separate vector and use pages to only allocate string data. This would simplify insert and set operations. With current design if we run out of space inside a page, we have to re-arrange memory. When offset is stored outside of the page, it can refer to any page so insert and set could be as simple as append. The evolution of StrVec The design described here is a second implementation of StrVec. The one before was simply a combination of str::Str (my std::string) for allocating all strings and Vec<u32> (my std::vector) for storing offset index. It had some flaws: appending a string could re-allocate memory within str::Str. The caller couldn’t store returned char* pointer because it could be invalidated. As a result the API was akward and potentially confusing: I was returning offset of the string so the string was str::Str.Data() + offset. The new StrVec doesn’t re-allocate on Append, only (potentially) on InsertAt and SetAt. The most common case is append-only which allows the caller to store the returned char* pointers. Before implementing StrVec I used Vec<char*>. Vec is my version of std::vector and Vec<char*> would just store pointer to individually allocated strings. Cost vs. benefit I’m a pragmatist: I want to achieve the most with the least amount of code, the least amount of time and effort. While it might seem that I’m re-implementing things willy-nilly, I’m actually very mindful of the cost of writing code. Writing software is a balance between effort and resulting quality. One of the biggest reasons SumatraPDF so popular is that it’s fast and small. That’s an important aspect of software quality. When you double click on a PDF file in an explorer, SumatraPDF starts instantly. You can’t say that about many similar programs and about other software in general. Keeping SumatraPDF small and fast is an ongoing focus and it does take effort. StrVec.cpp is only 705 lines of code. It took me several days to complete. Maybe 2 days to write the code and then some time here and there to fix the bugs. That being said, I didn’t start with this StrVec. For many years I used obvious Vec<char*>. Then I implemented somewhat optimized StrVec. And a few years after that I implemented this ultra-optimized version. References SumatraPDF is a small, fast, multi-format (PDF/eBook/Comic Book and more), open-source reader for Windows. The implementation described here: StrVec.cpp, StrVec.h, StrVec_ut.cpp By the time you read this, the implementation could have been improved.

22 hours ago 1 votes
The parental dead end of consent morality

Consent morality is the idea that there are no higher values or virtues than allowing consenting adults to do whatever they please. As long as they're not hurting anyone, it's all good, and whoever might have a problem with that is by definition a bigot.  This was the overriding morality I picked up as a child of the 90s. From TV, movies, music, and popular culture. Fly your freak! Whatever feels right is right! It doesn't seem like much has changed since then. What a moral dead end. I first heard the term consent morality as part of Louise Perry's critique of the sexual revolution. That in the context of hook-up culture, situationships, and falling birthrates, we have to wrestle with the fact that the sexual revolution — and it's insistence that, say, a sky-high body count mustn't be taboo — has led society to screwy dating market in the internet age that few people are actually happy with. But the application of consent morality that I actually find even more troubling is towards parenthood. As is widely acknowledged now, we're in a bit of a birthrate crisis all over the world. And I think consent morality can help explain part of it. I was reminded of this when I posted a cute video of a young girl so over-the-moon excited for her dad getting off work to argue that you'd be crazy to trade that for some nebulous concept of "personal freedom". Predictably, consent morality immediately appeared in the comments: Some people just don't want children and that's TOTALLY OKAY and you're actually bad for suggesting they should! No. It's the role of a well-functioning culture to guide people towards The Good Life. Not force, but guide. Nobody wants to be convinced by the morality police at the pointy end of a bayonet, but giving up on the whole idea of objective higher values and virtues is a nihilistic and cowardly alternative. Humans are deeply mimetic creatures. It's imperative that we celebrate what's good, true, and beautiful, such that these ideals become collective markers for morality. Such that they guide behavior. I don't think we've done a good job at doing that with parenthood in the last thirty-plus years. In fact, I'd argue we've done just about everything to undermine the cultural appeal of the simple yet divine satisfaction of child rearing (and by extension maligned the square family unit with mom, dad, and a few kids). Partly out of a coordinated campaign against the family unit as some sort of trad (possibly fascist!) identity marker in a long-waged culture war, but perhaps just as much out of the banal denigration of how boring and limiting it must be to carry such simple burdens as being a father or a mother in modern society. It's no wonder that if you incessantly focus on how expensive it is, how little sleep you get, how terrifying the responsibility is, and how much stress is involved with parenthood that it doesn't seem all that appealing! This is where Jordan Peterson does his best work. In advocating for the deeper meaning of embracing burden and responsibility. In diagnosing that much of our modern malaise does not come from carrying too much, but from carrying too little. That a myopic focus on personal freedom — the nights out, the "me time", the money saved — is a spiritual mirage: You think you want the paradise of nothing ever being asked of you, but it turns out to be the hell of nobody ever needing you. Whatever the cause, I think part of the cure is for our culture to reembrace the virtue and the value of parenthood without reservation. To stop centering the margins and their pathologies. To start centering the overwhelming middle where most people make for good parents, and will come to see that role as the most meaningful part they've played in their time on this planet. But this requires giving up on consent morality as the only way to find our path to The Good Life. It involves taking a moral stance that some ways of living are better than other ways of living for the broad many. That parenthood is good, that we need more children both for the literal survival of civilization, but also for the collective motivation to guard against the bad, the false, and the ugly. There's more to life than what you feel like doing in the moment. The worst thing in the world is not to have others ask more of you. Giving up on the total freedom of the unmoored life is a small price to pay for finding the deeper meaning in a tethered relationship with continuing a bloodline that's been drawn for hundreds of thousands of years before it came to you. You're never going to be "ready" before you take the leap. If you keep waiting, you'll wait until the window has closed, and all you see is regret. Summon a bit of bravery, don't overthink it, and do your part for the future of the world. It's 2.1 or bust, baby!

yesterday 2 votes