Local LLMs Run on Your Computer

Large language models (LLMs) that power chatbots like ChatGPT can be run on your own computer. This provides offline access and greater privacy compared to cloud-based services, as no data is sent to external servers for analysis.

You also avoid paying monthly subscriptions to AI companies. Numerous LLMs are available for free download, including from major firms like Meta and Google. While these local models may not be as advanced or fast as premium cloud offerings, they are capable enough for everyday tasks.

There is more maintenance involved, such as handling updates yourself. However, you gain a personalized and private AI system.

What You Need to Get Started

Local LLMs can run on Windows, macOS, and Linux. macOS is often the preferred platform for AI enthusiasts due to its unified hardware and Apple Silicon chips, which combine CPU, GPU, and RAM in a way AI models favor.

Plenty of RAM is beneficial. A bare minimum is 8 GB, but this limits model size and speed. Sixteen gigabytes is better, and 32 or more is required for the biggest and fastest models. For optimal performance, a dedicated GPU with ample VRAM (above 8 GB) is recommended, as this memory is optimized for AI tasks.

On Windows, a dedicated Nvidia GPU significantly helps because graphics chips are better for AI processes than standard processors. These cards provide additional RAM for the models.

There is no strict minimum specification, but more RAM and a discrete graphics card are advantageous. You then need software to run the model and the model itself.

For beginners, LM Studio Bionic is generally considered the best free AI app choice on Windows and macOS. Other popular options include vLLM, Llama.cpp, Ollama, and GPT4All, which are available for multiple operating systems but are more technical.

After choosing an app, you must select an LLM. The software often guides users to options, and online repositories like Hugging Face host over 3 million models.

An Example Local LLM Setup

Many setups are possible. As an example, setting up LM Studio Bionic on Windows involves downloading the installation file, launching the app, and clicking Create Project to name a new project.

A blank conversation window opens. Clicking Choose a model in the prompt box and then Get local models takes you to the model picker. Available models are listed with their size, popularity, and details. Staff picks offer starting recommendations. Smaller models are less capable but download faster and use less disk space.

Back in the chat interface, operations resemble a standard AI chatbot. The prompt box lets you choose between installed AI models. A + (plus) icon next to it allows image and file submission if the model supports it.

The left navigation pane lets you switch between projects. A Settings link in the lower left opens a preferences screen for customizing software behavior, from handling deleted chats to interface elements.

Clicking Library in Settings manages existing models, while the Explore menu finds new ones. For image and document support, look for multimodal models. These build on LLM capabilities but are often called LLMs.

The final important interface element is the right-hand sidebar, toggled via a button in the top-right corner. It offers further options for managing files across projects and granting file system access.