Running Llama 3 locally with Ollama and a REST API wrapper
Why run a model locally at all
Cloud-hosted LLM APIs are convenient, but there are scenarios where local inference is the right choice. Privacy-sensitive applications cannot send user data to third-party endpoints. Offline-capable tools need a model that works without internet access. Cost modelling for high-volume workloads sometimes shows that owning inference hardware becomes cheaper past a certain request volume. Development iteration is faster when you are not rate-limited or paying per token.
Llama 3 from Meta is one of the most capable open-weight models available. The 8B parameter variant runs comfortably on a MacBook Pro with 16 GB of unified memory. The 70B variant needs a machine with at least 40 GB of available VRAM or RAM. For most developer use cases — local assistants, code generation, document processing — the 8B model is the right starting point.
Ollama is the tool that makes the entire process approachable. It handles model downloads, quantisation, hardware detection, and serving through a local HTTP API. You do not need to install CUDA drivers, configure Python environments, or understand tensor parallelism.
Installing Ollama and pulling Llama 3
On macOS, installation is a single command:
brew install ollama
On Linux, use the official install script:
curl -fsSL https://ollama.com/install.sh | sh
On Windows, download the installer from ollama.com. All three platforms receive the same CLI and API surface.
Start the Ollama server in one terminal:
ollama serve
Pull the Llama 3.1 8B model (approximately 4.7 GB):
ollama pull llama3.1:8b
Verify it works with a quick chat:
ollama run llama3.1:8b "Explain the CAP theorem in two sentences"
Ollama stores models in ~/.ollama/models on macOS and Linux and exposes a REST API on http://localhost:11434. The API is partially compatible with the OpenAI API format, which means libraries written for OpenAI can often point at Ollama without modification.
Querying the Ollama API directly
Before wrapping Ollama in your own server, understand its native API. The primary endpoint for chat is POST /api/chat. For completions it is POST /api/generate.
# Non-streaming chat request
curl http://localhost:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.1:8b",
"stream": false,
"messages": [
{ "role": "user", "content": "What is the difference between TCP and UDP?" }
]
}'
The response is a JSON object:
{
"model": "llama3.1:8b",
"message": {
"role": "assistant",
"content": "TCP (Transmission Control Protocol) is connection-oriented..."
},
"done": true,
"total_duration": 4820398000,
"eval_count": 143
}
For streaming, set "stream": true. Ollama sends newline-delimited JSON, with each line being a partial message until "done": true.
Ollama also exposes an OpenAI-compatible endpoint at /v1/chat/completions, which accepts the exact same payload format as the OpenAI API:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.1:8b",
"messages": [
{ "role": "user", "content": "What is the capital of Ghana?" }
]
}'
This compatibility layer means you can swap OPENAI_BASE_URL=http://localhost:11434/v1 in any OpenAI SDK integration and immediately use your local model.
Building a production REST wrapper with Hono
Ollama is a local-only service by default. If you need to expose it over a network — to your frontend, to a mobile app, to a team of developers — you should build a wrapper that adds authentication, rate limiting, request logging, and model selection logic. Hono is a lightweight web framework well-suited for this.
mkdir llm-gateway && cd llm-gateway
npm init -y
npm install hono @hono/node-server
npm install -D typescript @types/node tsx
Create src/index.ts:
import { serve } from "@hono/node-server";
import { Hono } from "hono";
import { cors } from "hono/cors";
import { logger } from "hono/logger";
const OLLAMA_BASE = process.env.OLLAMA_URL ?? "http://localhost:11434";
const API_KEY = process.env.GATEWAY_API_KEY ?? "";
const DEFAULT_MODEL = process.env.DEFAULT_MODEL ?? "llama3.1:8b";
const app = new Hono();
app.use("*", logger());
app.use("*", cors());
// Simple API key auth middleware
app.use("/v1/*", async (c, next) => {
if (!API_KEY) {
await next();
return;
}
const authHeader = c.req.header("Authorization");
if (authHeader !== `Bearer ${API_KEY}`) {
return c.json({ error: "Unauthorized" }, 401);
}
await next();
});
// Health check
app.get("/health", (c) => {
return c.json({ status: "ok", model: DEFAULT_MODEL });
});
// List available models
app.get("/v1/models", async (c) => {
const res = await fetch(`${OLLAMA_BASE}/api/tags`);
const data = (await res.json()) as { models: Array<{ name: string }> };
return c.json({
object: "list",
data: data.models.map((m) => ({
id: m.name,
object: "model",
owned_by: "ollama",
})),
});
});
// Chat completions — proxies to Ollama with streaming support
app.post("/v1/chat/completions", async (c) => {
const body = await c.req.json<{
model?: string;
messages: Array<{ role: string; content: string }>;
stream?: boolean;
temperature?: number;
max_tokens?: number;
}>();
const ollamaPayload = {
model: body.model ?? DEFAULT_MODEL,
messages: body.messages,
stream: body.stream ?? false,
options: {
temperature: body.temperature ?? 0.7,
num_predict: body.max_tokens ?? 2048,
},
};
const ollamaRes = await fetch(`${OLLAMA_BASE}/api/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(ollamaPayload),
});
if (!ollamaRes.ok) {
const err = await ollamaRes.text();
return c.json({ error: err }, 500);
}
// Streaming path: forward the NDJSON stream as SSE
if (body.stream) {
return new Response(
new ReadableStream({
async start(controller) {
const reader = ollamaRes.body!.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) {
controller.enqueue(
new TextEncoder().encode("data: [DONE]\n\n")
);
controller.close();
break;
}
const chunk = decoder.decode(value);
for (const line of chunk.split("\n").filter(Boolean)) {
try {
const parsed = JSON.parse(line) as {
message?: { content: string };
done: boolean;
};
if (!parsed.done && parsed.message) {
const ssePayload = JSON.stringify({
choices: [
{
delta: { content: parsed.message.content },
finish_reason: null,
},
],
});
controller.enqueue(
new TextEncoder().encode(`data: ${ssePayload}\n\n`)
);
}
} catch {
// Skip malformed lines
}
}
}
},
}),
{
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
Connection: "keep-alive",
},
}
);
}
// Non-streaming path: collect full response and return OpenAI-compatible JSON
const ollamaData = (await ollamaRes.json()) as {
message: { role: string; content: string };
eval_count: number;
prompt_eval_count: number;
};
return c.json({
id: `chatcmpl-${Date.now()}`,
object: "chat.completion",
model: body.model ?? DEFAULT_MODEL,
choices: [
{
index: 0,
message: ollamaData.message,
finish_reason: "stop",
},
],
usage: {
prompt_tokens: ollamaData.prompt_eval_count ?? 0,
completion_tokens: ollamaData.eval_count ?? 0,
total_tokens:
(ollamaData.prompt_eval_count ?? 0) + (ollamaData.eval_count ?? 0),
},
});
});
serve({ fetch: app.fetch, port: 3001 }, (info) => {
console.log(`LLM gateway running on http://localhost:${info.port}`);
});
Run it:
GATEWAY_API_KEY=supersecret npx tsx src/index.ts
Now any OpenAI SDK call pointed at http://localhost:3001 will route through your gateway to the local Ollama instance with authentication applied.
Using the wrapper from client applications
With the OpenAI-compatible endpoint, existing SDKs work without modification. In Node.js:
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:3001/v1",
apiKey: "supersecret",
});
const response = await client.chat.completions.create({
model: "llama3.1:8b",
messages: [{ role: "user", content: "Write a haiku about databases" }],
});
console.log(response.choices[0].message.content);
For streaming responses:
const stream = await client.chat.completions.create({
model: "llama3.1:8b",
messages: [{ role: "user", content: "List the SOLID principles with examples" }],
stream: true,
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content ?? "";
process.stdout.write(delta);
}
console.log();
In Python with the openai package:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:3001/v1",
api_key="supersecret",
)
response = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "Explain ACID properties"}],
)
print(response.choices[0].message.content)
Model selection and quantisation trade-offs
Ollama downloads quantised versions of models by default. Quantisation reduces model size and memory requirement by storing weights in lower precision formats (4-bit or 8-bit integers instead of 16-bit floats). The trade-off is a small reduction in output quality.
The Llama 3.1 family on Ollama offers several quantisation levels:
# 8B model at different quantisation levels
ollama pull llama3.1:8b # Default Q4_K_M — 4.7 GB
ollama pull llama3.1:8b-q8_0 # 8-bit — 8.5 GB, better quality
ollama pull llama3.1:8b-fp16 # Full precision — 16 GB, maximum quality
# 70B model for higher capability
ollama pull llama3.1:70b # Default Q4_K_M — 40 GB
For most developer workflows, Q4_K_M (the default) offers the best balance. If you are doing tasks where output quality is critical — code generation, reasoning tasks, structured extraction — and you have the RAM, Q8_0 is worth the extra memory.
You can also create custom Modelfiles to set system prompts, temperature defaults, and other model parameters at the Ollama level:
FROM llama3.1:8b
SYSTEM """
You are a senior software engineer specialising in distributed systems.
Answer questions concisely, use code examples when relevant, and acknowledge uncertainty honestly.
"""
PARAMETER temperature 0.3
PARAMETER num_predict 2048
Save this as Modelfile and create a named variant:
ollama create engineering-assistant -f Modelfile
ollama run engineering-assistant "What are the trade-offs of eventual consistency?"
Performance tuning for your hardware
Ollama automatically detects available GPU hardware and offloads computation to it. On Apple Silicon Macs, it uses Metal. On machines with NVIDIA GPUs, it uses CUDA. On CPU-only machines, it falls back to optimised BLAS routines.
To check what hardware Ollama detected and how tokens per second you are achieving, look at the eval_duration and eval_count in the API response:
const response = await fetch("http://localhost:11434/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "llama3.1:8b",
stream: false,
messages: [{ role: "user", content: "Hello" }],
}),
});
const data = await response.json() as {
eval_count: number;
eval_duration: number; // nanoseconds
};
const tokensPerSecond = data.eval_count / (data.eval_duration / 1e9);
console.log(`Inference speed: ${tokensPerSecond.toFixed(1)} tokens/second`);
On a MacBook Pro M3 Max with 36 GB unified memory, Llama 3.1 8B achieves around 60-80 tokens per second. The 70B model on the same machine runs at around 15-20 tokens per second. For interactive use, 20+ tokens per second is comfortable. Below 10 tokens per second, streaming feels slow.
If you need faster inference on constrained hardware, consider smaller models like llama3.2:3b (approximately 2 GB) which runs at 100+ tokens per second on the same hardware.
Ollama also supports concurrent requests. Set OLLAMA_NUM_PARALLEL to control how many requests can be processed simultaneously, though each concurrent request shares the available memory and reduces throughput per request.
Deploying the gateway to a server
For team use or production deployment, you can run both Ollama and the gateway on a dedicated machine. On a Linux server with an NVIDIA GPU:
# Start Ollama bound to all interfaces so the gateway can reach it
OLLAMA_HOST=0.0.0.0 ollama serve &
# Pull the model
ollama pull llama3.1:8b
# Run the gateway
OLLAMA_URL=http://localhost:11434 \
GATEWAY_API_KEY=your-production-key \
npx tsx src/index.ts
The gateway handles authentication and sits between your application and Ollama. You should not expose Ollama's port directly to the internet — it has no authentication by default.
For containerised deployments, Ollama provides an official Docker image:
# docker-compose.yml
services:
ollama:
image: ollama/ollama
volumes:
- ollama_data:/root/.ollama
environment:
- OLLAMA_HOST=0.0.0.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
gateway:
build: .
ports:
- "3001:3001"
environment:
- OLLAMA_URL=http://ollama:11434
- GATEWAY_API_KEY=${GATEWAY_API_KEY}
depends_on:
- ollama
volumes:
ollama_data:
Local LLM inference with Ollama removes the hard dependency on cloud APIs. The REST wrapper layer means you can swap the underlying model, add custom logging, enforce usage policies, and evolve your infrastructure independently of your application code.