Ollamav0.32BEGINNER

Ollama

Quick reference for Ollama — run large language models locally with CLI commands, REST API, Modelfiles, vision, tool calling, and OpenAI-compatible endpoints.

8 min read

ollamallmailocal-aicliapimachine-learning

Loading your progress

Installation & Setup

Install Ollama on macOS, Linux, and Windows and start the server.

Install Ollama

Install Ollama on your platform using the official installer or package manager.

bash

# macOS / Linux — one-line install
curl -fsSL https://ollama.com/install.sh | sh

# macOS — via Homebrew
brew install ollama

# Windows — download installer from https://ollama.com/download

# Docker
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

💡 Ollama runs as a background service automatically after installation

⚡ The Docker image supports GPU passthrough with --gpus=all for NVIDIA

📌 Default API server runs on http://localhost:11434

🟢 After install, run "ollama pull llama3.2" to get your first model

installsetup

Environment Configuration

Configure Ollama behavior with environment variables.

bash

# Common environment variables
export OLLAMA_HOST=0.0.0.0:11434    # Listen on all interfaces
export OLLAMA_MODELS=/path/to/models # Custom model storage path
export OLLAMA_KEEP_ALIVE=5m          # How long to keep models loaded

💡 Ollama runs as a system service automatically — no need to manually start it

⚡ Set OLLAMA_HOST=0.0.0.0 to expose the API on your local network

📌 OLLAMA_KEEP_ALIVE controls how long models stay loaded in memory (default 5m)

🟢 Use OLLAMA_MODELS to store models on a different drive if disk space is limited

serverconfig

Running Models

Run and interact with models from the command line.

Run a Model

Start an interactive chat session with a model.

bash

# Run a model (downloads if not present)
ollama run llama3.2

# Run a specific size variant
ollama run llama3.2:1b
ollama run gemma3:27b

# Run with a one-off prompt (no interactive session)
ollama run llama3.2 "Explain Docker in one sentence"

💡 Models are automatically downloaded on first run if not already pulled

⚡ Use the tag (e.g., :1b, :8b, :27b) to pick a specific model size

📌 Interactive mode supports multi-line input with triple quotes (""")

🟢 Vision models like gemma3 can accept image file paths directly

runchatcli

Model Management

Pull, list, copy, and remove models.

Pull & List Models

Download models from the registry and view installed models.

bash

# Pull a model from the registry
ollama pull llama3.2

# List all downloaded models
ollama list

# Show model details (parameters, template, license)
ollama show llama3.2

# Show the Modelfile of a model
ollama show --modelfile llama3.2

💡 Use "ollama ps" to see which models are currently loaded in memory

⚡ Models are stored in ~/.ollama/models by default (configurable via OLLAMA_MODELS)

📌 The :latest tag is used by default if no tag is specified

🟢 Browse available models at https://ollama.com/library

pulllistmodels

Copy & Remove Models

Create model aliases and delete models from local storage.

bash

# Copy/alias a model to a new name
ollama cp llama3.2 my-model

# Remove a model
ollama rm my-model

💡 Copying a model creates a lightweight alias — it does not duplicate the model files

⚡ Use cp to alias models to OpenAI-compatible names (e.g., gpt-3.5-turbo)

📌 Removing a model frees up disk space by deleting its layers

🟢 You can always re-pull a removed model from the registry later

copyremovealias

Modelfile

Create custom models with system prompts, parameters, and adapters using Modelfiles.

Create a Custom Model

Define a Modelfile with a base model, system prompt, and parameters, then build it.

bash

# Modelfile
FROM llama3.2
SYSTEM "You are a helpful coding assistant."
PARAMETER temperature 0.7
PARAMETER num_ctx 4096

# Build the model
ollama create my-coder -f ./Modelfile

💡 FROM is the only required instruction — everything else is optional

⚡ Use PARAMETER to tune behavior: temperature, num_ctx, top_p, repeat_penalty

📌 MESSAGE pre-seeds the conversation to set the model's behavior through examples

🟢 View any model's Modelfile with "ollama show --modelfile <model>"

modelfilecustomcreate

Modelfile Parameters

Reference for common model parameters and their effects.

bash

# Key parameters for PARAMETER instruction
PARAMETER temperature 0.7    # Creativity (0.0-2.0, default ~0.8)
PARAMETER num_ctx 4096       # Context window size in tokens
PARAMETER top_p 0.9          # Nucleus sampling threshold
PARAMETER top_k 40           # Top-K sampling
PARAMETER repeat_penalty 1.1 # Penalize repetition (1.0 = off)
PARAMETER seed 42            # Reproducible output

💡 temperature 0 gives deterministic output; 1.0+ increases creativity

⚡ Increase num_ctx for longer conversations but be aware of memory limits

📌 Use seed for reproducible responses — same seed + prompt = same output

🟢 stop sequences tell the model when to stop generating text

parametersconfig

REST API

Interact with models programmatically using the Ollama REST API.

Generate Completions

Generate text completions using the /api/generate endpoint.

bash

# Generate a completion
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

💡 Set "stream": false to get the complete response in a single JSON object

⚡ The "context" field from a response can be passed back to maintain conversation state

📌 Use "options" to override model parameters like temperature and num_ctx per request

🟢 Default streaming returns newline-delimited JSON objects as tokens are generated

apigeneraterest

Chat Completions

Use the /api/chat endpoint for multi-turn conversations.

bash

# Chat with message history
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is Docker?"}
  ],
  "stream": false
}'

💡 The chat endpoint manages conversation history via the messages array

⚡ Roles are "system", "user", and "assistant" — same as OpenAI format

📌 Set "format": "json" to force the model to respond with valid JSON

🟢 Include the full message history in each request for context continuity

apichatrest

Embeddings

Generate vector embeddings for text using embedding models.

Generate Embeddings

Use the /api/embed endpoint to create text embeddings for RAG and similarity search.

bash

# Generate embeddings for a single text
curl http://localhost:11434/api/embed -d '{
  "model": "all-minilm",
  "input": "Ollama is great for local AI"
}'

💡 Use /api/embed (not /api/embeddings) for the latest API with batch support

⚡ all-minilm is fast and lightweight (23MB) — great for getting started

📌 Embeddings are used for RAG, semantic search, and similarity comparisons

🟢 Batch multiple inputs in a single request for better performance

embeddingsapirag

Vision & Multimodal

Use vision models to analyze images alongside text prompts.

Image Analysis

Send images to vision-capable models via CLI or API.

bash

# CLI — pass an image file to a vision model
ollama run gemma3 ./photo.png "What's in this image?"

# API — send base64-encoded image
curl http://localhost:11434/api/chat -d '{
  "model": "gemma3",
  "messages": [{
    "role": "user",
    "content": "Describe this image",
    "images": ["<base64-encoded-image>"]
  }],
  "stream": false
}'

💡 Vision models include gemma3, llava, and moondream — pull one to get started

⚡ The CLI handles image encoding automatically — just pass the file path

📌 API requires base64-encoded images in the "images" array; SDKs accept file paths

🟢 You can combine multiple images in a single request for comparison tasks

visionmultimodalimages

Structured Output

Force models to respond with structured JSON or schema-based output.

JSON Mode & Schema

Use the format parameter to get structured JSON responses.

bash

# JSON mode — free-form JSON output
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Tell me about Canada"}],
  "format": "json",
  "stream": false
}'

💡 "format": "json" ensures the model always returns valid JSON

⚡ Pass a JSON Schema object as "format" to enforce an exact output structure

📌 Include format instructions in your prompt too — the model responds better with both

🟢 Schema-based output works best with larger models (7B+) for complex structures

jsonstructuredschema

Tool Calling

Let models call external functions and tools to augment their capabilities.

Function Calling with Tools

Define tools that models can invoke, then process the tool calls.

python

# Python — pass functions directly as tools
from ollama import chat

def get_weather(city: str) -> str:
  """Get current weather for a city"""
  return f"Sunny, 22°C in {city}"

response = chat(
  model='qwen3',
  messages=[{'role': 'user', 'content': 'Weather in NYC?'}],
  tools=[get_weather],
)

💡 Python SDK auto-generates tool schemas from type hints and docstrings

⚡ Tool calling uses a two-step flow: model requests tools, you execute and return results

📌 Models like qwen3, llama3.2, and command-r support tool calling

🟢 Add role: "tool" messages with results so the model can form its final answer

toolsfunction-calling

OpenAI Compatibility

Use the OpenAI-compatible API to drop in Ollama as a local replacement.

OpenAI SDK Drop-in

Point the OpenAI SDK at Ollama for local inference with no code changes.

python

# Python — use OpenAI SDK with Ollama
from openai import OpenAI

client = OpenAI(
  base_url='http://localhost:11434/v1/',
  api_key='ollama',  # required but ignored
)

response = client.chat.completions.create(
  model='llama3.2',
  messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.choices[0].message.content)

💡 Just change base_url to localhost:11434/v1/ — the API key is required but ignored

⚡ Supports /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models

📌 Use "ollama cp" to alias models to OpenAI names for zero-code-change migration

🟢 Great for testing apps locally before switching to cloud API providers

openaicompatibilitysdk

Official SDKs

Use the official Python and JavaScript libraries for a native Ollama experience.

Python SDK

Install and use the official Ollama Python library.

python

# Install
pip install ollama

# Basic chat
from ollama import chat
response = chat(model='llama3.2', messages=[
  {'role': 'user', 'content': 'Hello!'},
])
print(response.message.content)

💡 The Python SDK provides both sync and async clients (AsyncClient)

⚡ Pass stream=True for token-by-token streaming in real time

📌 Functions like chat(), generate(), embed(), list(), pull() mirror the REST API

🟢 Type hints are included — full autocomplete support in modern editors

pythonsdk

JavaScript SDK

Install and use the official Ollama JavaScript/TypeScript library.

typescript

// Install
// npm install ollama

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'llama3.2',
  messages: [{ role: 'user', content: 'Hello!' }],
})
console.log(response.message.content)

💡 The JS SDK works in Node.js and supports ESM and CommonJS imports

⚡ Streaming uses async iterators — use "for await...of" to process chunks

📌 Full TypeScript support with typed responses out of the box

🟢 Use ollama.pull() with stream: true to show download progress

javascripttypescriptsdk

GPU & Performance

Configure GPU acceleration and optimize model performance.

GPU Configuration

Configure GPU layers, parallel requests, and memory management.

bash

# Check which models are loaded and GPU usage
ollama ps

# Environment variables for GPU/performance
export OLLAMA_NUM_PARALLEL=4         # Concurrent requests
export OLLAMA_MAX_LOADED_MODELS=2    # Models in memory at once
export OLLAMA_FLASH_ATTENTION=1      # Enable flash attention
export OLLAMA_GPU_OVERHEAD=0         # Reserved GPU memory (bytes)

💡 Ollama automatically detects and uses available GPUs (NVIDIA, AMD, Apple Silicon)

⚡ Use OLLAMA_FLASH_ATTENTION=1 for faster inference on supported hardware

📌 Send keep_alive: 0 to immediately unload a model and free GPU memory

🟢 Preload models by sending an empty prompt — great for reducing first-response latency

gpuperformancememory

Ollama

Table of Contents

Installation & Setup

Running Models

Model Management

Modelfile

REST API

Embeddings

Vision & Multimodal

Structured Output

Tool Calling

OpenAI Compatibility

Official SDKs

GPU & Performance