Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
37

How I run LLMs locally

from Abishek Muthian [alt+shift+b] in technology

A HN user asked me0 how I run LLMs locally with some specific questions, I’m documenting it here for everyone. Before I begin I would like to credit the thousands or millions of unknown artists, coders and writers upon whose work the Large Language Models(LLMs) are trained, often without due credit or compensation. Get Started r/LocalLLaMA subreddit1 & Ollama blog2 are great places to get started with running LLMs locally.
29th Dec 2024

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Abishek Muthian

Multiple $20 AI Plans Are Better Than a Single $100 AI Plan

After using the $200 and $100 Anthropic monthly plans for Claude, I have now settled on $20 Anthropic + $20 OpenAI + $10 OpenCode Go monthly plans for coding and design work. I also use local AI occasionally for non-coding-related tasks. By using multiple entry-level AI plans from different providers, I can assign different tasks to different models based on their strengths and also hedge against service disruption by not relying on a single provider.

1st Jul 2026 • 2 votes
Access Ring

Discover and invoke UI elements near your cursor, without precise mouse targeting. A Logi Actions SDK plugin for MX Creative Console and Actions Ring. Buttons, checkboxes, menus, tabs, dropdowns: tap to interact with whatever is nearby. The problem with precise clicking Every toolbar button, checkbox, and menu item demands pixel-perfect mouse aim. For everyone, it is slow. For people living with conditions that affect motor function, such as Parkinson's disease, Alzheimer's, essential tremor, multiple sclerosis, muscular dystrophy, ALS, repetitive strain injury, spinal injury, arthritis etc. precise mouse targeting can range from difficult to impossible.

16th Apr 2026 • 1 votes
Building an offline AI stoic chatbot

Inspiration Many teens are turning to AI chatbots for friendship and emotional support and online AI chatbots are harnessing their data. As an alternate, I propose Epictetus- A 100% offline chatbot built on the wisdom of Epictetus-- A Stoic philosopher who taught ~2000 years ago that true freedom and happiness are found by focusing solely on what is within our power (our judgments, intentions, and responses) while calmly accepting everything else as beyond our control.

28th Nov 2025 • 1 votes
AI for generating 2D animations

I wanted to add some frills in Memory Hammer app1 when the user completes review of their flashcards. I decided to add a simple 2D animation where a brain is sulking and after being hit by the hammer (memory hammer's mascot) the brain becomes happy and jubilant. I made a simple animation with Lottie first, When I imported the .lottie file for the animation into my flutter app, I noticed that it's skipping frames. There's no official Lottie library for flutter, I saw someone else facing the same issue2 with the 3rd party Lottie library.

26th Oct 2025 • 1 votes
Owning my Social Network

After yet another Mastodon drama1 and tired of migrating from one Mastodon instance to another I have decided to self-host a Mastodon instance. But Mastodon is an overkill for a single user instance, Luckily GoToSocial2 an ActivityPub social network server written in Go serves my needs perfectly. I deployed GoToSocial on Railway, The developer of GoToSocial patiently helped me to fix few quirks with my deployment. Thanks to the federated nature of ActivityPub protocol, I was able to migrate my existing Fosstodon account to my new GoToSocial account3.

14th May 2025 • 1 votes

More in technology

The cryptographic world computer
11 hours ago • 1 votes
FLIP Fluid on Flip Dots

[Hardware] Electromechanical Fluid Simulation

3 days ago • 1 votes
Is This A Joke? In The Auth Header? (F5 BIG-IP UnAuth Heap-Overflow to RCE CVE-2026-94127)

Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’

4 days ago • 1 votes
The Reason You Prohibit Things

You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.

4 days ago
Solitaire Alone Together

Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together

6 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in