Software Choices for a Multi-GPU AI Workstation

In my previous articles I discussed the balance between model quality and hardware requirements for running AI models locally.  In this context it is more about the relationship between the AI model and overall memory consumption and understanding the limits that still provide good performance.  In my next article I put those principles to work in building a Mutli-GPU AI workstation on a budget.  Of course, “budget” is very subjective and really comes down to how deep your pockets are.  For instance, a $50k workstation can buy a LOT of cloud time whereas the estimated $4000 I spent on my workstation isn’t bad by comparison.

For this article I wanted to discuss software and how my AI Workstation build was designed to work with the software packages and how I can switch between them to complete certain tasks.  Below you will find the software list and links to the various webpages and github repos.

• Ollama - https://ollama.com/
• Llama.cpp - https://github.com/ggml-org/llama.cpp
• OpenWebUI - https://openwebui.com/
• ComfyUI - https://www.comfy.org/
• N8N - https://n8n.io/
• OpenClaw - https://openclaw.ai/
• Aphrodite Engine - https://github.com/aphrodite-engine/aphrodite-engine
• Augment Toolkit - https://augmentoolkit.com/
• Axolotl LLM - https://axolotl.ai/
• Deepspeed - https://www.deepspeed.ai/
• Phison Aidaptiv - https://github.com/aiDAPTIV-Phison/aiDAPTIV

Keep in mind this is just the software I am currently using.  AI is a changing landscape and what is current as of this article, may not be the same 6 months to 3 years from now.  Likewise, I find that the opinions around AI software to vary quite a bit and what might work for me may not be compatible with what someone else is doing.

A good example is LMStudio.  This is a desktop application and a perfect thing to have on an AI Workstation.  However, for me I would rather treat my AI Workstation as a remote server and thus I will often prefer web enabled apps over desktop versions.

I blame my background as a web application developer for my bias.

Another example would be Langchain.  I have heard good things about Langchain, just about everyone is using it and it is quite powerful.  Though, I don’t like the idea that it is a licensed open-source project.  I can fully respect that people want to get paid for their good work but the whole SASS model for software is getting out of hand.

N8N does have a paid tier and so far, I haven’t used that platform enough to warrant needing to pay anything and the Phison aiDAPTIV product is a closed source software platform supported by open-source scripts and dependent on a very expensive SSD.  Phison calls this drive an AI accelerator and the marketing often confuses what the system is designed to do.  Some users think it will accelerate inference when really, it’s just a memory extension for LLM training.  It doesn’t really accelerate anything, at least in the traditional sense and is something I will dive into later.

For the most part this article will be discussing the software I have been using at a high level along with how I use them on my AI Workstation.  If I was to take a deep dive, this would be a very different article and would be exponentially longer in length.  In addition, most everything I will talk about in this article are based on my personal experience and information you can find on the various homepages and repos.  Unless relevant, I won’t cover how to install or update the software but will include things that caused me grief in the past.

Like a review, let’s get started