Home Server & AI: Ollama

Ollama is an open-source machine learning model that works similarly to OpenAI’s ChatGPT. It makes it easy to get up and running with large language models locally, making it accessible for users who prefer not to rely on cloud-based solutions. We are going to configure a NixOS environment for running Ollama with GPU support using Docker..

GPU access for Docker containers

To leverage GPU acceleration for running Ollama in Docker containers on NixOS, you need to configure the system properly. Here’s a step by step breakdown of the configuration settings in configuration.nix:

# OpenGL Settings
hardware.opengl = {
    enable = true; # Activates OpenGL support, essential for graphical applications and GPU computations.
    driSupport = true; # Enables Direct Rendering Infrastructure (DRI) to allow direct access to the GPU.
    driSupport32Bit = true; # Provides support for 32-bit applications, which is important if you run legacy or specific software that requires it.
};

# NVIDIA GPU Configuration
hardware.nvidia = {
    modesetting.enable = true; # Allows the use of the modesetting driver, which can improve performance.
    open = false; # Indicates that proprietary NVIDIA drivers should be used instead of the open-source alternatives.
    nvidiaSettings = true; # Enables the NVIDIA settings utility for managing GPU settings.
    package = config.boot.kernelPackages.nvidiaPackages.stable;
    # Specifies the version of the NVIDIA driver package to use.
  };

# Library Path and Video Drivers:
hardware.opengl.setLdLibraryPath = true; # Ensures that the library path is set correctly for OpenGL drivers.
services.xserver.videoDrivers = [ "nvidia" ];
# Specifies that the NVIDIA driver should be used for the X server, enabling graphical applications to utilize the GPU.

Creates and starts a new container Ollama in Docker

# To run the Ollama mode:
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

• -d: Runs the container in detached mode as a demon in the background.
• ‐‐gpus=all: Grants the container access to all available GPUs. It does not work in a macOS and you need to be sure that the NVIDIA Container Toolkit is installed in your system.
• -v ollama:/root/.ollama: Create a volume name ollama (to persist data for Ollama) and mount it on /root/.ollama inside the container (-v left side(host):right side (container)). If the left side is a word (e.g., ollama) and not a path, then the Docker process will create a new directory where its default location is for volumes (e.g., sudo ls /var/lib/docker/volumes).
• -p 11434:11434: Maps port 11434 (Ollama’s default API port) on the host to the container, enabling access to the model. In other words, we want the port 11434 inside the container to be accessible outside the container, too.
• ‐‐name ollama: Names the container “ollama” for easy reference.
• ollama/ollama: is the Docker image being used.

docker exec -it ollama ollama run llama3

docker ps # Check if container is running
docker logs ollama # View container logs
docker stop ollama # Stop the container
docker rm ollama # Remove the container
docker pull ollama/ollama # Pull a new image
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama #

• exec -it: Executes a command in an interactive terminal inside the running container.
• ollama run llama3: This command runs the specified model (in this case, llama3).