Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Cognitive Computations
NIST’s September 30 report on DeepSeek not about security, it’s about control. I am here to tell you what the man behind the curtain doesn’t want you to pay any attention to. NIST's DeepSeek "Evaluation" is a Hit Piece NIST’s recent report on DeepSee...
With the recent update to OpenAI's Terms of Use on October 23, 2024, there’s been a flurry of online discussions around what these terms mean for developers, businesses, and everyday users of AI tools like ChatGPT. Much of the conversation, especiall...
Wanna chat with Dolphin locally? (no internet connection needed) Here is the easy way - Ollama. install ollama. after you finsh you should be able to run ollama from the command line. Also you will see the ollama icon up top like this: Iff you are curious - anytime you see that icon, that means ollama is running in the background and it also has a port open (11434) that can accept api calls. Sadly, as of this writing there is no swagger. Back to business. download your dolphin. You can choose which size you want from here. I am choosing size Q5_K_M which is just over 5gb. mkdir ~/models cd ~/models wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf Now you need a Modelfile. vim Modelfile.dolphin.mistral Add the following text to Modelfile.dolphin.mistral FROM ./dolphin-2.2.1-mistral-7b.Q5_K_M.gguf TEMPLATE """{{ if and .First .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}<|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ SYSTEM """You are Dolphin, an uncensored, unbiased AI assistant""" PARAMETER stop <|im_end|> Now you can use Ollama to install this model. ollama create dolphin.mistral -f Modelfile.dolphin.mistral Now look, you can run it from the command line. Which is cool enough. But we are just getting started. If you want, you can install samantha too so you have two models to play with. wget https://huggingface.co/TheBloke/samantha-1.2-mistral-7B-GGUF/resolve/main/sama ntha-1.2-mistral-7b.Q5_K_M.gguf vim Modelfile.samantha.mistral And enter the following into Modelfile.samantha.mistral FROM ./samantha-1.2-mistral-7b.Q5_K_M.gguf TEMPLATE """{{ if and .First .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}<|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ SYSTEM """You are Samantha, an AI companion""" PARAMETER stop <|im_end|> Then install the model ollama create samantha -f Modelfile.samantha.mistral And now you can also chat with Samantha from the command line. Cool yeah? We are just getting started. Let's get Ollama Web UI installed. cd ~ git clone https://github.com/ollama-webui/ollama-webui.git cd ollama-webui npm i npm run dev Now you can open that link http://localhost:5173 in your web browser. now you can choose dolphin or samantha from the dropdown (I have installed a few others too) Well talking to these models from the command line and the web ui is just the beginning. Also, frameworks such as langchain, llamaindex, litellm, autogen, memgpt all can integrate with ollama. Now you can really play with these models. Here is a fun idea that I will leave as an exercise - given some query, ask dolphin to decide whether a question about coding, a request for companionship, or something else. If it is a request for companionship then send it to Samantha. If it is a coding question, send it to deepseek-coder. Otherwise, send it to Dolphin. And just like that, you have your own MoE.
When a16z generously sponsored Dolphin, I had some compute budget, and because the original dolphin-13b was a flop, I had some time to go back to the drawing board. When I was ready to train the next iteration, I reconsidered whether to rent or buy the compute for the build. I ultimately decided to buy, because I have the skill and interest, I'm good at finding deals, and owning a cluster would give me the ability to continue executing on future projects beyond Dolphin, not to mention the satisfaction of building the AI end-to-end, like a baker baking bread from scratch. Artisan AI. What to build? I am building 4 servers of 8x AMD Instinct MI100 and 4 servers of 8x NVIDIA GeForce RTX 4090. But for now, I'm just going to get 2 servers up and running at a time. Later, after I've built out all of the servers and tested them, then I will get the full 8 server setup running. (that'll require extra electrical work) I went with the MI100s because I got a killer deal with Rhino Technology (who have an excellent sales and technical team, I highly recommend) on some refurbished Gigabyte G482-Z53 servers that came preinstalled each with 4x MI100s. As a scrappy guy building in my garage, I gotta roll with the deals that I can find. And the servers support 8x PCI-e gen4 x16. Exactly what I needed. This capability is pretty hard to find, by the way. Usually, the PCI slots are bifurcated and don't get all 16 lanes. And for training AI models for bandwidth I really need each card to get all 16 lanes. The servers also came preinstalled with dual AMD EPYC 7742 64-core CPUs (wow!) and 256GB RAM. Which is plenty to start with. I had a very good start to my cluster with these servers. So I'm starting with the easy ones, the MI100s. They are easy because they fit in the server, unlike the 4090s, which are larger and won't fit in the chassis. My inspiration: https://www.pugetsystems.com/labs/articles/1-7x-nvidia-geforce-rtx-4090-gpu-scaling/ https://nonint.com/2022/05/30/my-deep-learning-rig/ The care and feeding of servers Then I had to do some math. I planned to limit each card to 300 watts each. So I figured, I need 240 volts and 25 amps per server. For the cooling, for each pair of servers I got a 25000 BTU air conditionter (240v, 30 amps) So for my "building" phase when I only need to power 2 servers at a time, I will have 2 breakers. One 240v 50 amps for the two servers, and one 240v 30 amps for the air conditioner. Later, when all the servers are ready for operation, I will need 4x 50 amp breakers and 4x 30 amp breakers. And I figured with the amps and the length of the wire, I needed an 8-gauge 3 conductor wire that I got from the hardware store. And a couple of flush-mounted outlets, one 6-50 for the pair of servers and one 6-30 for the air conditioner. I found these PDUs that can each handle 50 amps and power 2 servers that each have 3 power supplies. Perfect. As I have never done this before, it took a few weeks of researching and ordering things to get all of this put together before I was able to power up the first server. Getting Ubuntu Server installed Pretty straightforward, just put Ubuntu on a thumb drive and boot from that. Of course, I needed to get a monitor and a keyboard working too. And one of my servers turned out to have a nonfunctional VGA output so i switched to another server for now. Getting it on the network So I didn't wanna run a wire ALL the way to my living room. I bought these TP-Link AC600 wireless adapters but of course they didn't just work when I plug them in. So I had to first get the server on the network so I could install the driver. So I hooked it up with ethernet to my workstation and used Windows wifi sharing to bridge the wifi to the ethernet. That got the server on the internet. After that, I was able to download updates. I installed links2 web browser because Ubuntu Server has no GUI. After that, I was able to install the driver for the TP-Link AC600 and get it connected. Then I give the mac address an assigned ip address on my router's dhcp, so that I can forward port 22 and dynamic dns so I can SSH to it from the outside. (of course, I add my public ed25519 key and disable password login) This server will act as my bastion, I will ssh from this to the other servers in my cluster. Installing drivers, ROCm, and HIP I was told I should use the docker image, but I couldn't get that working. Instead I installed the drivers from the repository. The trick is this: don't try to install multiple versions of ROCm, just install version 5.7, and install the nightly version of Pytorch that works with 5.7. That's the combination that made everything work for me. Actually, AMD's install experience is better than NVidia's. Also you gotta install NVidia's Cuda too before you install HIP. https://docs.amd.com/projects/HIP/en/docs-5.3.0/how_to_guides/install.html Inference with Oobabooga Setup Oobabooga as normal, using requirements-rocm.txt then duplicate this drive for the other servers Future Plans I am going to make any changes required to get Axolotl running on these servers. I am going to put a Lustre cluster on these servers, I plan to do 7x 2tb ssd on each server and using 100GbE so it should be fast enough and able to handle the nodes saving and loading checkpoints. This is very important for multinode training. I am going to train more versions of dolphin and other models using these servers. I am eventually going to train my own base model.
More in programming
Understanding how assigning anthropomorphic traits to agents can improve their effectiveness and interaction.
The Ware for September 2026 is a tire pressure monitor from a late 80’s to early 90’s model Porsche, made by Bosch. According to the submitter: Kind of a crazy piece of hardware for the late 80’s – this would be pretty cutting edge for the time. Nobody quite got what the part was – […]
I’ve been taking a break from Mastodon as my social home on the web for about six weeks. It has been more difficult than I’d usually admit. I have an amazing local community in my town that I love and am deeply involved in. But nobody here is a programmer, or even a computer nerd, so it doesn’t serve my need for a tech community. I work alone, for myself, in my house. It’s a fortunate lifestyle with many upsides, but it means I don’t get that form of community at work, either. I’ve had this feeling before… as a teenager in the 1990s. It’s profoundly lonely. But when I got internet access, that started to change. I found my people. First in the Total Annihilation community (RIP, Annihilated), then the HardOCP Forum, then the Something Awful Forums, then Tumblr, then Twitter and our community of Apple bloggers, and then Mastodon. Communities form, change, and dissolve over time. None of those are, today, anything like what they were when I most enjoyed them. We change, too. I didn’t move on from most of those solely because they changed — it was usually because we both changed, and it was no longer a good fit. My break from Mastodon has been a relief in some ways, as I’ve felt increasing divergence between what I wanted and what I was getting from it, but I didn’t know where else to go. Instagram, TikTok, and YouTube are all more video-broadcast platforms than communities. All I’ve found on Bluesky is even more anger and politics, the last things I need or want. Threads hasn’t yet stuck for me, and it’s owned by an asshole. Twitter was an amazing product and community that was previously owned by assholes, and now it’s X, which has a strong but different tech community, and is owned by different assholes. So I’ve just been… nowhere, for the first time since the 1990s. It has been, once again, quite lonely. Sure, it’s been relaxing, too, but in the way a vacation is relaxing. It’s great to take breaks, but after a long vacation, I’m ready to turn my brain back on. Without participating in a large, broad tech community, I’m exposed to fewer new ideas, both of my main jobs (app development and podcasting) are much harder, the quality and reach of my work have suffered, my skills aren’t improving as much, my mind is less stimulated, and I feel disconnected from my professions and passions. My world has gotten smaller. I could keep going like this to prove a stubborn victory or purity test, but I’d mostly just be hurting myself. Communities change, conditions change, and we change. We’re all better off if we’re willing to change our minds, and if our communities let us, no matter how strongly our younger selves declared a previous position. So I’m going to wade back in and see how it goes, trying to find a better balance to hopefully have better results, including on Mastodon. My people are spread across multiple services, and I’m going to explore them all, despite most of them being owned by assholes. I’d love to participate in communities without any assholes benefiting from it, but in the modern world, that simply isn’t possible. The only alternative is to isolate myself and not have a community at all (or a job, or a smartphone, or a computer, or internet access, or a car, or most of my food, or my country). That’s neither a price I’m willing to pay nor a condition I’d try to impose on anyone else. I’d love to have your support, forgiveness, and grace as I try to find my people again, and I strive to offer the same to all of you. Everyone deserves to be part of a community. And as I grow, I’m learning to declare fewer absolutes in a complex, imperfect, and constantly changing world.
My now-traditional “summary” of the conference along with my own commentary and thoughts.
