Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Cognitive Computations
NIST’s September 30 report on DeepSeek not about security, it’s about control. I am here to tell you what the man behind the curtain doesn’t want you to pay any attention to. NIST's DeepSeek "Evaluation" is a Hit Piece NIST’s recent report on DeepSee...
With the recent update to OpenAI's Terms of Use on October 23, 2024, there’s been a flurry of online discussions around what these terms mean for developers, businesses, and everyday users of AI tools like ChatGPT. Much of the conversation, especiall...
Gratitude to https://tensorwave.com/ for giving me access to their excellent servers! Few have tried this and fewer have succeeded. I've been marginally successful after a significant amount of effort, so it deserves a blog post. Know that you are in for rough waters. And even when you arrive - There are lots of optimizations tailored for nVidia GPUs so, even though the hardware may be just as strong spec-wise, in my experience so far, it still may take 2-3 times as long to train on equivalient AMD hardware. (though if you are a super hacker maybe you can fix it!) Right now I'm using Axolotl. Though I am probably going to give LlamaFactory a solid try in the near future. There's also LitGpt and TRL. But I kind of rely on the dataset features and especially the sample packing of Axolotl. But more and more LlamaFactory is interesting me, it supports new features really fast. (like GaLore is the new hotness at the moment). This blog post will be about getting Axolotl up and running in AMD, and I may do one about LlamaFactory if there is demand. I am using Ubuntu 22.04 LTS, and you should too. (unless this blog post is really old by the time you read it). Otherwise you can use this post as a general guide. Here are all the environment variables I ended up setting in my .bashrc and I'm not exactly sure which ones are needed. You better set them all just in case. export GPU_ARCHS="gfx90a" # mi210 - use the right code for your GPUexport ROCM_TARGET="gfx90a"export HIP_PATH="/opt/rocm-6.0.0"export ROCM_PATH="/opt/rocm-6.0.0"export ROCM_HOME="/opt/rocm-6.0.0"export HIP_PLATFORM=amdexport DS_BUILD_CPU_ADAM=1 export TORCH_HIP_ARCH_LIST="gfx90a" Part 1: Driver, ROCm, HIP Clean everything out. There shouldn't be any trace of nvidia, cuda, amd, hip, rocm, anything like that. This is not necessarily a simple task, and of course it totally depends on the current state of your system. and I had to use like 4 of my daily Claude Opus questions to accomplish this. (sad face) By the way Anthropic Claude Opus is the new king of interactive troubleshooting. By far. Bravo. Don't nerf it pretty please! Here are some things I had to do, that might help you: sudo apt autoremove rocm-core sudo apt remove amdgpu-dkms sudo dpkg --remove --force-all amdgpu-dkms sudo apt purge amdgpu-dkms sudo apt remove --purge nvidia* sudo apt remove --purge cuda* sudo apt remove --purge rocm-* hip-* sudo apt remove --purge amdgpu-* xserver-xorg-video-amdgpu sudo apt clean sudo reboot sudo dpkg --remove amdgpu-install sudo apt remove --purge amdgpu-* xserver-xorg-video-amdgpu sudo apt autoremove sudo apt clean rm ~/amdgpu-install_*.deb sudo reboot sudo rm /etc/apt/sources.list.d/amdgpu.list sudo rm /etc/apt/sources.list.d/rocm.list sudo rm /etc/apt/sources.list.d/cuda.list sudo apt-key del A4B469963BF863CC sudo apt update sudo apt remove --purge nvidia-* cuda-* rocm-* hip-* amdgpu-* sudo apt autoremove sudo apt clean sudo rm -rf /etc/OpenCL /etc/OpenCL.conf /etc/amd /etc/rocm.d /usr/lib/x86_64-linux-gnu/amdgpu /usr/lib/x86_64-linux-gnu/rocm /opt/rocm-* /opt/amdgpu-pro-* /usr/lib/x86_64-linux-gnu/amdvlk sudo reboot I love Linux (smile with tear) Now finally do like sudo apt-get updatesudo apt-get upgrade and sudo apt-get dist-upgrade and make sure there's no errors or warnings! You should be good to begin your journey. Install AMD drivers, ROCm, HIP wgethttps://repo.radeon.com/amdgpu-install/23.40.2/ubuntu/jammy/amdgpu-install_6.0.60002-1_all.deb (at time of this writing). But you should double check here. And the install instructions here. sudo apt-get install ./amdgpu-install_6.0.60002-1_all.deb sudo apt-get update sudo amdgpu-install -y --accept-eula --opencl=rocr --vulkan=amdvlk --usecase=workstation,rocm,rocmdev,rocmdevtools,lrt,opencl,openclsdk,hip,hiplibsdk,mllib,mlsdk If you get error messages (I did) try to fix them. I had to do this: sudo dpkg --remove --force-all libvdpau1 sudo apt clean sudo apt update sudo apt --fix-broken install sudo apt upgrade and then, again, I had to run sudo amdgpu-install -y --accept-eula --opencl=rocr --vulkan=amdvlk --usecase=workstation,rocm,rocmdev,rocmdevtools,lrt,opencl,openclsdk,hip,hiplibsdk,mllib,mlsdk Check Installation rocm-smirocminfo/opt/rocm/bin/hipconfig --full I hope that worked for you - if not, I suggest asking Claude Opus about the error messages to help you figure it out. If that doesn't work, reach out to the community. Part 2: Pytorch, BitsAndBytes, Flash Attention, DeepSpeed, Axolotl Conda mkdir -p ~/miniconda3wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.shbash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3rm -rf ~/miniconda3/miniconda.sh~/miniconda3/bin/conda init bash Exit your shell and enter it again. conda create -n axolotl python=3.12conda activate axolotl Pytorch I tried the official install command from pytorch's website, and it didn't work for me. Here is what did work: pip install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/rocm6.0python -c "import torch; print(torch.version.hip)" This tests both Torch, and Torch's ability to interface with HIP. If it worked, it will print HIP version. Otherwise, it will print None. BitsAndBytes BitsAndBytes is by Tim Dettmers, an absolute hero among men. It lets us finetune in 4-bits. It gives us qLoRA. It brings AI to the masses. There is a fork of BitsAndBytes that supports ROCm. This is provided not by Tim Dettmers, and not by AMD, but by a vigilante superhero, Arlo-Phoenix. In appreciation, here is a portrait ChatGPT made for Arlo-Phoenix, vigilante superhero. I hope you like it, if you see this Arlo-Phoenix. <3 git clone https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6cd bitsandbytes-rocm-5.6git checkout rocmROCM_TARGET=gfx90a make hip # use the ROCM_TARGET for your GPUpip install . Flash Attention This fork is maintained by AMD git clone --recursive https://github.com/ROCmSoftwarePlatform/flash-attention.gitcd flash-attentionexport GPU_ARCHS="gfx90a" # use the GPU_ARCHS for your GPUpip install . DeepSpeed Microsoft included AMD support in DeepSpeed proper, but there's still some undocumented fussiness to get it working, and there is a bug I found with DeepSpeed, I had to modify it to get it to work. git clone https://github.com/microsoft/DeepSpeedcd DeepSpeedgit checkout v0.14.0 # but check the tags for newer version Now, you gotta modify this file: vim op_builder/builder.py Replace the function assert_no_cuda_mismatch with this: (unless they fixed it yet) def assert_no_cuda_mismatch(name=""): cuda_available = torch.cuda.is_available() if not cuda_available and not torch.version.hip: # Print a warning message indicating no CUDA or ROCm support print(f"Warning: {name} requires CUDA or ROCm support, but neither is available.") return False else: # Check CUDA version if available if cuda_available: cuda_major, cuda_minor = installed_cuda_version(name) sys_cuda_version = f'{cuda_major}.{cuda_minor}' torch_cuda_version = torch.version.cuda if torch_cuda_version is not None: torch_cuda_version = ".".join(torch_cuda_version.split('.')[:2]) if sys_cuda_version != torch_cuda_version: if (cuda_major in cuda_minor_mismatch_ok and sys_cuda_version in cuda_minor_mismatch_ok[cuda_major] and torch_cuda_version in cuda_minor_mismatch_ok[cuda_major]): print(f"Installed CUDA version {sys_cuda_version} does not match the " f"version torch was compiled with {torch.version.cuda} " "but since the APIs are compatible, accepting this combination") return True elif os.getenv("DS_SKIP_CUDA_CHECK", "0") == "1": print( f"{WARNING} DeepSpeed Op Builder: Installed CUDA version {sys_cuda_version} does not match the " f"version torch was compiled with {torch.version.cuda}." "Detected `DS_SKIP_CUDA_CHECK=1`: Allowing this combination of CUDA, but it may result in unexpected behavior." ) return True raise CUDAMismatchException( f">- DeepSpeed Op Builder: Installed CUDA version {sys_cuda_version} does not match the " f"version torch was compiled with {torch.version.cuda}, unable to compile " "cuda/cpp extensions without a matching cuda version.") else: print(f"Warning: {name} requires CUDA support, but torch.version.cuda is None.") return False return True pip install -r requirements/requirements.txtHIP_PLATFORM="amd" DS_BUILD_CPU_ADAM=1 TORCH_HIP_ARCH_LIST="gfx90a" python setup.py install Axolotl Installing Axolotl might overwrite BitsAndBytes, DeepSpeed, and PyTorch. Be prepared for things to break, they do often. Your choice is either modify the setup.py and requirements.txt (if you are confident to change those things) or pay attention to what libraries get deleted and reinstalled, and just delete them again and reinstall the correct ROCm version that you installed earlier. If Axolotl complains about incorrect versions - just ignore it, you know better than Axolotl. Right now, Axolotl's Flash Attention implementation has a hard dependency on Xformers for its SwiGLU implementation, and Xformers doesn't work with ROCm, you can't even install it. So, we are gonna have to hack axolotl to remove that dependency. https://github.com/OpenAccess-AI-Collective/axolotl.gitcd axolotl from requirements.txt remove xformers==0.0.22 from setup.py make this change (remove any mention of xformers) $ git diff setup.pydiff --git a/setup.py b/setup.pyindex 40dd0a6..235f1d0 100644--- a/setup.py+++ b/setup.py@@ -30,7 +30,7 @@ def parse_requirements(): try: if "Darwin" in platform.system():- _install_requires.pop(_install_requires.index("xformers==0.0.22"))+ print("hi") else: torch_version = version("torch") _install_requires.append(f"torch=={torch_version}")@@ -45,9 +45,6 @@ def parse_requirements(): else: raise ValueError("Invalid version format")- if (major, minor) >= (2, 1):- _install_requires.pop(_install_requires.index("xformers==0.0.22"))- _install_requires.append("xformers>=0.0.23") except PackageNotFoundError: pass And then in src/axolotl/monkeypatch/llama_attn_hijack_flash.py make this change: --- a/src/axolotl/monkeypatch/llama_attn_hijack_flash.py+++ b/src/axolotl/monkeypatch/llama_attn_hijack_flash.py@@ -22,7 +22,9 @@ from transformers.models.llama.modeling_llama import ( apply_rotary_pos_emb, repeat_kv, )-from xformers.ops import SwiGLU+class SwiGLU:+ def __init__():+ print("hi") from axolotl.monkeypatch.utils import get_cu_seqlens_from_pos_ids, set_module_name@@ -45,15 +47,7 @@ LOG = logging.getLogger("axolotl") def is_xformers_swiglu_available() -> bool:- from xformers.ops.common import get_xformers_operator-- try:- get_xformers_operator("swiglu_packedw")()- return True- except RuntimeError as exc:- if "No such operator xformers::swiglu_packedw " in str(exc):- return False- return True+ return False Now you can install axolotl pip install -e .accelerate launch -m axolotl.cli.train examples/openllama-3b/lora.yml Welcome to finetuning on ROCm!
Wanna chat with Dolphin locally? (no internet connection needed) Here is the easy way - Ollama. install ollama. after you finsh you should be able to run ollama from the command line. Also you will see the ollama icon up top like this: Iff you are curious - anytime you see that icon, that means ollama is running in the background and it also has a port open (11434) that can accept api calls. Sadly, as of this writing there is no swagger. Back to business. download your dolphin. You can choose which size you want from here. I am choosing size Q5_K_M which is just over 5gb. mkdir ~/models cd ~/models wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf Now you need a Modelfile. vim Modelfile.dolphin.mistral Add the following text to Modelfile.dolphin.mistral FROM ./dolphin-2.2.1-mistral-7b.Q5_K_M.gguf TEMPLATE """{{ if and .First .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}<|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ SYSTEM """You are Dolphin, an uncensored, unbiased AI assistant""" PARAMETER stop <|im_end|> Now you can use Ollama to install this model. ollama create dolphin.mistral -f Modelfile.dolphin.mistral Now look, you can run it from the command line. Which is cool enough. But we are just getting started. If you want, you can install samantha too so you have two models to play with. wget https://huggingface.co/TheBloke/samantha-1.2-mistral-7B-GGUF/resolve/main/sama ntha-1.2-mistral-7b.Q5_K_M.gguf vim Modelfile.samantha.mistral And enter the following into Modelfile.samantha.mistral FROM ./samantha-1.2-mistral-7b.Q5_K_M.gguf TEMPLATE """{{ if and .First .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}<|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ SYSTEM """You are Samantha, an AI companion""" PARAMETER stop <|im_end|> Then install the model ollama create samantha -f Modelfile.samantha.mistral And now you can also chat with Samantha from the command line. Cool yeah? We are just getting started. Let's get Ollama Web UI installed. cd ~ git clone https://github.com/ollama-webui/ollama-webui.git cd ollama-webui npm i npm run dev Now you can open that link http://localhost:5173 in your web browser. now you can choose dolphin or samantha from the dropdown (I have installed a few others too) Well talking to these models from the command line and the web ui is just the beginning. Also, frameworks such as langchain, llamaindex, litellm, autogen, memgpt all can integrate with ollama. Now you can really play with these models. Here is a fun idea that I will leave as an exercise - given some query, ask dolphin to decide whether a question about coding, a request for companionship, or something else. If it is a request for companionship then send it to Samantha. If it is a coding question, send it to deepseek-coder. Otherwise, send it to Dolphin. And just like that, you have your own MoE.
More in programming
This post was previously published through my newsletter on October 19, 2026. From time to time, I will republish newsletter issues on my blog here if I think people (and search engines) might benefit from it. If you want to read everything I’ve written once it is posted, I recommend signing up for the newsletter. For the last 5-6 years or so, I’ve been talking about, teaching and helping teams implement contract testing on a regular basis. To make that more concrete: at the moment, I’m having ongoing conversations with four different potential clients about helping them in some way with their contract testing implementation, and so far, I’ve run my contract testing workshop four times this year. It probably doesn’t come as a surprise to you when I say that I really like the concept of contract testing as a means of addressing integration testing challenges in distributed software systems. However, no matter if I’m speaking, teaching or implementing contract testing, certain challenges almost always come up at some point, and I want to highlight some of those today. Understanding the concept of contract testing First of all, it often takes people a while to understand the concept of contract testing, especially when they’re used to what I call the ‘traditional’ approach to integration testing: deploying the systems and services in scope to a shared testing environment and then testing them ‘as a whole’. Of course, the more distributed your system and your development process is, and the higher the deployment frequency, the more complex and the slower this integration testing approach is going to be. The premise behind contract testing is to break that integration testing puzzle down into small chunks of individual consumer-provider pairs, and then test the integration between those two in an asynchronous manner. By asynchronous, I mean that there is no need to deploy consumer and provider into a shared environment; both parties perform their part of the process in their own, local development environment. This means you can detect potential integration issues before you deploy, instead of stumbling onto them after you do so. This makes contract testing (or contract-based integration testing, to use its proper name) a significantly different approach to integration testing compared to the more ‘traditional’ approach. In my experience, it usually takes people time to wrap their head around how the process actually works, which steps you should take and in which order, and what you (do not yet) know after each step. It often takes going through several contract testing cycles before people start to ‘get it’. That’s not their fault, that is to be expected, since the contract-based approach to integration testing is, again, quite different from what they’ve been doing for so long. To better cater to this, I redesigned my contract testing workshop a year and a half or so ago. It is now a full-day workshop (half a day is simply too short), and I take much smaller steps and repeat them more often. From what I see, this has led to a better understanding of the concepts with my audience, and that’s the whole point. Contract testing === Pact I’ve talked to quite a few teams that had heard of contract testing somewhere, decided that that was the way out of their integration testing problems, and then immediately started writing contract tests with Pact. Understandable, because Pact is a very useful, very powerful and very popular tool, plus, it’s open source, which makes it easy for teams to get started. Often, they skipped a step in the process. Pact is very much a consumer-driven contract testing tool by nature, and while consumer-driven contract testing is the richest and most powerful of the different approaches to contract testing, it is not necessarily the best fit for every team and every context. It’s a classic example of using the wrong tool for the problem at hand, because they didn’t take enough time looking at the problem and the available solutions. That’s not the problem of Pact, even though it is often framed that way (‘Pact didn’t solve my problems, therefore Pact is not a good tool’). That’s a ‘we didn’t fully understand the problem and the context’ problem. Contract testing removes the need for communication Another challenge that I often see when I talk about or help teams with contract testing is the fact that teams don’t talk to each other. This then leads to integration issues, because team A expects something from team B that team B doesn’t supply. Contract testing is not the solution to these problems. It can’t be - tools alone do not solve problems, let alone communication problems. What contract testing does do is make potential integration issues visible early on in the development process, often earlier than (again) more ‘traditional’ approaches to integration testing. That’s where the value of contract testing can be found. It is not a means to replace communication between teams. In fact, and this is especially true for consumer-driven contract testing, contract testing cannot succeed without good communication between consumer and provider teams. Contract testing replaces all other types of integration testing This one comes up less often than it used to, but sometimes, I still hear people saying that when they will have contract-based integration tests in place, they don’t need other types of integration tests anymore - including E2E tests. That’s simply not true. While contract testing is a great way to find out earlier about the impact of a change in a specific consumer or provider further downstream, in many cases, I wouldn’t rely on these tests alone before moving to production. Because of its asynchronous nature, contract-based integration testing only verifies whether a consumer and a provider speak the same language - can a consumer understand the message returned by a provider in a specific situation? It does not verify that the request or the response message reaches the other party in the first place, for example. In other words, it doesn’t test connectivity between a consumer and a provider. And because integration testing is broken down into individual consumer-provider pairs, you don’t see the entire system working ‘as a whole’. This means you might want to keep a few end-to-end tests in place, even when you’ve got good contract testing coverage across the board. The above is not a complete list of challenges with contract testing, but the ones I mentioned here are those that come up most often.
Have you seen this? If you haven't, you should. I think everyone needs to. Wealth inequality on a scale this extreme is almost impossible to explain. You have to experience it to understand. Go ahead. Try it out. Where is the top of this graph? Good question. Keep
Slop is without a doubt everywhere. Be it in restaurant’s menus, music playlists, or online game storefronts, it is now impossible to escape it. Yet, I’ve never met anyone who was actually looking forward to consuming that. No matter how hard it’s shoved down our throats, the human longing to connect with other people through art still remains. Raymond Kotewicz They will not let you filter A simple solution would be to have slop filters. Automatically label AI-generated trash, and let users hide it if they don’t want to see it. So far, I don’t think any of the major internet platforms has implemented this. The common excuse for why they don’t let you easily filter out slop is that it would be technically too difficult. Let me be clear with everyone here: this is complete bullshit. We’re talking about companies that can hire some of the most talented engineers in the world. I don’t think anyone seriously believes they can’t come up with a way to guess if a file has been generated by AI or not. 90% of the slop out there stands out so much everyone can see how it was made after looking at it for 5 seconds. A more plausible reason for the lack of slop filter is that they are simply testing out the waters. They are checking if users will tolerate it. The question tech executives want answered is: will people buy this garbage? And if the answer to that question is yes, here’s what they will do: they will start generating it themselves. They don’t need the useless meat proxies who think typing ten words and pressing enter makes them creative. They definitely don’t want to verse anyone royalties when they could just be making a one time purchase of tokens. If you think this is not real, just ask Tim Sweeney. I bet he will be more than happy to confirm. Roman Navarro Curation by humans for humans So now that it’s clear that tech companies will not save us from drowning in slop, it’s time to turn back to actual humans instead. The good news is they are a lot of people who are here to help you. There are thousands of people on the internet who will search through mountains of content to find the perfect piece of art they want to share with the world. We don’t explicitly call them curators. They are streamers, bloggers, journalists, DJs, overly enthusiast fans, or simply your friends. Some of those people are seriously amazing. There are DJs out there who will spend thousands of hours digging through crates to craft you the perfect ninety minutes of music. Why do you still need those machine-generated playlists? The alternative to blindly trusting the output of a recommendation algorithm isn’t always to become an expert in every field of art. Sometimes you just have to put the trust you had in a machine back into another person. Back to the real world? The real long term solution would be to stop relying only on the internet for distribution. This is already a thing in the music industry. Every city has a record shop that will sell you a carefully selected set of vinyls. For many musicians, physical sales (and concerts) are a much better source of income than streaming. The same goes for the movie. While text-to-video AI is a thing, the real money isn’t made by generating 30 second TikTok dramas. The real money comes from people paying tickets to see a full-length movie in a theater. Sadly I’m not too optimistic about my own field: video games. Somehow it seems the direction we’ve taken is the opposite. We’re all in on online distributions platforms. We have fully accepted them as our new digital overlords. Next month GTA V is coming out without a disk in the box. Sony already announced that the next Playstation will not support physical media. Physical distribution is not even mentioned in the pitch of the average indie publisher. In Breda, the small Dutch city where I live, we used to have two dedicated video games shop. Higher interests rates, expensive rents, and various policy failures have done their jobs: they are both closed. I don’t think this is a good sign for the survivability of the gaming industry.
Understanding how assigning anthropomorphic traits to agents can improve their effectiveness and interaction.
