Setting Up a Private Self-Hosted Ollama Server with Open-WebUI on Docker (Zero Data Leaks)

Commercial cloud AI providers like OpenAI, Anthropic, and Google offer powerful foundation models, but their cloud-based nature creates insurmountable compliance barriers for healthcare, financial institutions, legal practices, and defense contractors. Transmitting confidential patient records, customer credit histories, or proprietary source code across public API endpoints risks regulatory fines under GDPR, HIPAA, and SOC 2 frameworks.

The open-source generative AI ecosystem has reached a historic inflection point. High-capability models like Meta Llama 3.1, Mistral NeMo, and DeepSeek R1 rival commercial models across reasoning and coding benchmarks. By uniting Ollama as the backend model runner with Open-WebUI as the frontend interface inside hardened Docker containers, organizations can deploy a private on-premises ChatGPT alternative that guarantees zero data egress and mathematical privacy.

In this comprehensive systems engineering blueprint, we will configure an enterprise self-hosted Ollama and Open-WebUI environment on a Linux VPS or on-prem GPU workstation. We will implement NVIDIA GPU container passthrough, configure role-based access control (RBAC), enable private document RAG pipelines, enforce SSL termination via Nginx reverse proxy, and establish automated model quantization workflows.

1. System Prerequisites and Hardware Sizing

Before provisioning containers, ensure your physical host or cloud instance satisfies VRAM requirements based on model weight quantization:

# Clone and compile llama.cpp conversion toolkit
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Convert HuggingFace checkpoint to 16-bit GGUF
python3 convert_hf_to_gguf.py /models/custom-model/ --outfile model-f16.gguf

# Quantize to 4-bit medium (Q4_K_M)
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

# Open-WebUI OAuth2 / Microsoft Entra ID Config
ENABLE_OAUTH_SIGNUP: "true"
OAUTH_MERGE_ACCOUNTS_BY_EMAIL: "true"
OAUTH_PROVIDER_NAME: "Enterprise SSO"
OAUTH_CLIENT_ID: "your-azure-client-id"
OAUTH_CLIENT_SECRET: "your-azure-client-secret"
OPENID_PROVIDER_URL: "https://login.microsoftonline.com/your-tenant-id/v2.0/.well-known/openid-configuration"

2. Installing NVIDIA Container Toolkit on Ubuntu 22.04 / 24.04

Docker containers cannot access host GPU hardware unless the NVIDIA Container Toolkit is installed and registered with the Docker daemon:

# 1. Configure the production NVIDIA package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# 2. Update and install toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# 3. Configure Docker runtime and restart service
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 4. Verify GPU passthrough inside a test container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

3. Complete Production Docker Compose Deployment

We orchestrate both services using a unified docker-compose.yml file. Notice that Ollama's port 11434 is isolated to the internal Docker network, ensuring external internet bots cannot invoke models without authenticating through Open-WebUI:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama_backend
    restart: always
    volumes:
      - /opt/ollama/models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=4
    networks:
      - ai_internal

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open_webui_frontend
    restart: always
    ports:
      - "127.0.0.1:3000:8080" # Bind strictly to localhost; proxy via Nginx
    volumes:
      - /opt/open-webui/data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
      - ENABLE_SIGNUP=false # Disable public user registration
      - DEFAULT_MODELS=llama3.1:8b
      - RAG_EMBEDDING_ENGINE=ollama
      - RAG_OLLAMA_BASE_URL=http://ollama:11434
      - RAG_OLLAMA_MODEL=nomic-embed-text
    depends_on:
      - ollama
    networks:
      - ai_internal

networks:
  ai_internal:
    driver: bridge

4. Pulling Recommended Production Models

Execute model downloads directly into the running Ollama container:

# Primary General Reasoning Model
docker exec -it ollama_backend ollama pull llama3.1:8b

# High-Precision Embedding Model for Document RAG
docker exec -it ollama_backend ollama pull nomic-embed-text

# Specialized Code Generation Model
docker exec -it ollama_backend ollama pull qwen2.5-coder:7b

5. Hardening Security: Nginx Reverse Proxy & SSL Termination

Never expose port 3000 directly over plain HTTP. Configure an Nginx reverse proxy with TLS 1.3 encryption and Let's Encrypt certificates:

# /etc/nginx/sites-available/ai.yourdomain.com
server {
    server_name ai.yourcompany.internal;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade \$http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host \$host;
        proxy_set_header X-Real-IP \$remote_addr;
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto \$scheme;

        # Disable buffering for real-time token streaming
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 600s;
    }

    listen 443 ssl http2;
    ssl_certificate /etc/letsencrypt/live/ai.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.yourdomain.com/privkey.pem;
}

6. Enterprise Document RAG Built Directly into Open-WebUI

One of Open-WebUI's crowning features is its native document retrieval engine. Users can drag and drop internal spreadsheets, PDFs, or software architecture manuals directly into the chat box. Open-WebUI automatically chunks the document, converts text into high-dimensional vectors via nomic-embed-text inside Ollama, stores them in its embedded ChromaDB instance, and injects relevant passages into the model's context window.

Because the embedding model and the reasoning model both run on your local GPU, not a single byte of sensitive corporate documentation ever leaves your server rack.

Summary: The Sovereign AI Architecture

# Clone and compile llama.cpp conversion toolkit
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Convert HuggingFace checkpoint to 16-bit GGUF
python3 convert_hf_to_gguf.py /models/custom-model/ --outfile model-f16.gguf

# Quantize to 4-bit medium (Q4_K_M)
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

# Open-WebUI OAuth2 / Microsoft Entra ID Config
ENABLE_OAUTH_SIGNUP: "true"
OAUTH_MERGE_ACCOUNTS_BY_EMAIL: "true"
OAUTH_PROVIDER_NAME: "Enterprise SSO"
OAUTH_CLIENT_ID: "your-azure-client-id"
OAUTH_CLIENT_SECRET: "your-azure-client-secret"
OPENID_PROVIDER_URL: "https://login.microsoftonline.com/your-tenant-id/v2.0/.well-known/openid-configuration"

7. Custom GGUF Quantization: Converting HuggingFace Checkpoints to Ollama

If your data science team trains custom domain models on PyTorch, converting them into quantized GGUF binaries allows direct deployment into your Ollama cluster without recompiling software:

# Clone and compile llama.cpp conversion toolkit
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Convert HuggingFace checkpoint to 16-bit GGUF
python3 convert_hf_to_gguf.py /models/custom-model/ --outfile model-f16.gguf

# Quantize to 4-bit medium (Q4_K_M)
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

8. Enterprise Single Sign-On (SSO) with Active Directory & OAuth2

In corporate environments, individual password management introduces security vulnerabilities. Open-WebUI natively supports OAuth2, OpenID Connect (OIDC), and Microsoft Entra ID (Azure AD). Configure SSO authentication by populating environment variables inside your docker-compose.yml:

# Open-WebUI OAuth2 / Microsoft Entra ID Config
ENABLE_OAUTH_SIGNUP: "true"
OAUTH_MERGE_ACCOUNTS_BY_EMAIL: "true"
OAUTH_PROVIDER_NAME: "Enterprise SSO"
OAUTH_CLIENT_ID: "your-azure-client-id"
OAUTH_CLIENT_SECRET: "your-azure-client-secret"
OPENID_PROVIDER_URL: "https://login.microsoftonline.com/your-tenant-id/v2.0/.well-known/openid-configuration"