Local RAG pipelines: Build fast, private AI with Ollama and Python

Building AI features that respect privacy and avoid cloud costs is now within reach using local resources. A recent surge in open-source tools like Ollama has made it possible to run retrieval-augmented generation (RAG) pipelines entirely offline—no third-party APIs required.

This approach eliminates latency spikes from network calls and ensures sensitive documents never leave your infrastructure. While cloud-based LLM services offer convenience, their pricing models and data handling policies can introduce friction for production systems. By shifting inference and embeddings to local hardware, teams can build high-performance AI agents with predictable performance and full control over their data lifecycle.

Why Local RAG Beats Cloud Services for Production Workloads

Cloud-based LLM APIs introduce two persistent challenges: inconsistent response times and compliance overhead. Every call to a remote endpoint adds unpredictable latency, making user experience dependent on network conditions. Additionally, sending proprietary or confidential documents to external servers requires extensive data governance and security reviews.

Running your entire RAG stack locally removes these barriers. All model inference, embedding generation, and retrieval happen within your infrastructure, giving you complete control over performance, security, and cost. With the right setup, a local RAG pipeline can match or exceed the responsiveness of cloud services while keeping operational expenses predictable and transparent.

Designing a Privacy-First RAG Architecture with Ollama

A local RAG system follows a clear data flow: document ingestion, text segmentation, vector embedding, semantic retrieval, and final answer generation. Each component can run on standard hardware, provided sufficient memory and processing power are available.

The architecture relies on three core components:

• Ollama – Manages local model inference and embedding generation.
• Chunking engine – Splits documents into logical segments for processing.
• Vector store – Stores embeddings for fast similarity search.

By keeping the entire pipeline local, you avoid network overhead and maintain data sovereignty. The only external dependency is the Ollama runtime, which runs as a local service on your machine.

Step-by-Step Setup: From Zero to Local RAG in 15 Minutes

Getting started requires only two prerequisites: a supported operating system and Ollama installed. Begin by downloading and running the Ollama service on your local machine.

Pull the Required Models

Open a terminal and fetch the models needed for inference and embeddings:

ollama pull llama3
ollama pull nomic-embed-text

These commands download the Llama 3 model for generation and the Nomic embedding model, both optimized for local use. Once downloaded, Ollama runs these models entirely offline, ensuring no data leaves your system.

Choose Your Orchestration Language

You can implement the orchestration layer in TypeScript or Python, depending on your project’s tech stack. Both environments provide client libraries that connect to the Ollama HTTP API.

#### TypeScript Implementation

Create a new TypeScript project and install the Ollama client:

// index.ts
import { Ollama } from 'ollama';

const ollama = new Ollama({
  host: '
});

async function generateLocalEmbedding(text: string): Promise<number[]> {
  const response = await ollama.embeddings({
    model: 'nomic-embed-text',
    prompt: text,
  });
  return response.embedding;
}

This snippet initializes the client and defines a function to compute embeddings locally. The client connects to the Ollama service running on port 11434, ensuring all operations stay within your network.

#### Python Implementation

For Python-based orchestration, install the official Ollama client and set up an asynchronous client:

# orchestrator.py
import asyncio
from ollama import AsyncClient

client = AsyncClient(host=')

async def generate_local_embedding(text: str) -> list[float]:
    response = await client.embed(
        model='nomic-embed-text',
        input=text
    )
    return response['embeddings'][0]

The Python client follows a similar pattern, using async/await for non-blocking operations. Both implementations return dense vector representations of input text, ready for semantic search.

Optimizing Retrieval: Cosine Similarity for Local Search

Once embeddings are generated, the next step is to find the most relevant document chunks. Cosine similarity measures the angle between two vectors, indicating how closely their meanings align.

Here’s how to compute similarity in both languages:

#### TypeScript Cosine Similarity Function

function cosineSimilarity(vecA: number[], vecB: number[]): number {
  const dotProduct = vecA.reduce((sum, a, i) => sum + a * vecB[i], 0);
  const normA = Math.sqrt(vecA.reduce((sum, a) => sum + a * a, 0));
  const normB = Math.sqrt(vecB.reduce((sum, b) => sum + b * b, 0));
  return dotProduct / (normA * normB);
}

This function calculates the dot product of two vectors and divides it by their magnitudes, returning a score between -1 and 1. Higher values indicate greater semantic alignment.

#### Python Equivalent for Local Use

import math

def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
    dot_product = sum(a * b for a, b in zip(vec_a, vec_b))
    norm_a = math.sqrt(sum(a * a for a in vec_a))
    norm_b = math.sqrt(sum(b * b for b in vec_b))
    
    if norm_a == 0 or norm_b == 0:
        return 0.0
    
    return dot_product / (norm_a * norm_b)

The Python version includes a safeguard against division by zero, ensuring robustness when comparing empty or zero vectors.

Critical Pitfalls to Avoid in Local RAG Deployments

Running large models locally demands careful resource management. Memory constraints can crash processes if too many embeddings are generated concurrently. Monitor system usage and cap batch sizes accordingly.

Another common issue arises from improper text chunking. Arbitrary segment sizes can split context mid-sentence, leading to incoherent retrievals. Always implement overlap between chunks—typically 50 to 100 characters—to preserve semantic continuity.

Finally, validate model performance on your specific hardware. Some consumer GPUs struggle with larger embedding models, so benchmark inference speeds before deploying to production.

The Future of Local AI: Beyond In-Memory Search

This guide demonstrates how to build a lightweight RAG pipeline using only memory and local models. For scalability, consider integrating a persistent vector database like Chroma or Milvus to store embeddings long-term and enable concurrent queries.

As open-source models improve, local RAG systems will become even faster and more capable. Teams can now build AI features that respect privacy without sacrificing performance—or their budget.

The choice between cloud APIs and local inference is no longer binary. With tools like Ollama, you can have both control and cost efficiency.