All Projects
Artificial Intelligence & RAG Production Verified · 100% Offline

RAG-Powered Document Assistant

Context-Grounded Academic Literature Intelligence with ChromaDB & Local LLM Orchestration

Discipline Artificial Intelligence & RAG
Role AI/ML Engineer & Backend Architect
Timeline / Year 2026
Organization / Context Independent AI Systems Engineering

Executive Overview

Axiom RAG is an enterprise-grade Retrieval-Augmented Generation system designed for grounded question answering across academic computer science literature and research papers. Operating 100% locally with zero cloud dependencies, the architecture eliminates hallucination through dense semantic embeddings, persistent vector indexing, and zero-shot grounding guardrails that enforce verbatim page-level citations.

The Challenge & Solution

The Core Problem: Standard Large Language Models frequently invent convincing but factually incorrect citations, misquote algorithms, and require expensive API tokens. Furthermore, academic research institutions and enterprise clients cannot upload proprietary research or confidential internal documentation to public cloud models.

The Engineering Solution: Constructed a robust offline RAG pipeline using Sentence Transformers (all-MiniLM-L6-v2) to generate dense vector embeddings, ChromaDB for high-throughput persistent vector indexing, and Ollama to orchestrate local quantized models (LLaMA 3 / Mistral). The prompt architecture strictly penalizes ungrounded speculation and forces the LLM to output source page numbers and exact passage citations.

System Architecture & Data Flow

01
Document Ingestion
PyPDF extracts raw text and metadata from academic PDF papers with custom header and footer sanitation.
02
Sliding-Window Chunking
Documents are partitioned into 800-character chunks with 150-character semantic overlaps to preserve cross-boundary technical definitions.
03
Vector Embedding & Indexing
Sentence Transformers generate 384-dimensional dense vectors stored inside ChromaDB persistent collections with HNSW indexing.
04
Hybrid Retrieval & Reranking
Cosine similarity top-k semantic search extracts the highest-relevance context passages.
05
Grounded Generation
Ollama executes local model inference using a grounded prompt template that mandates page-level citations and admits ignorance if context is absent.
06
Dual-Interface Serving
FastAPI asynchronous REST API with Lifespan handlers paired with an interactive Streamlit reactive chat dashboard.

Key Engineering Highlights

Zero Cloud Leaks & Privacy Preservation

Operates 100% offline with Ollama and ChromaDB; zero confidential data ever leaves local infrastructure.

Granular Citation Tracking

Every generated answer includes verified document citations, exact page numbers, and cosine similarity relevance metrics.

High-Throughput FastAPI Backend

Engineered with modern async FastAPI Lifespan architecture, health check endpoints, and input sanitization.

Dual UI & Docker Deployment

Includes an interactive Streamlit chat interface and multi-container Docker Compose configuration for one-command deployment.

Automated Test Suite

Full test coverage using Pytest verifying chunking boundaries, embedding matrix dimensions, and API response contracts.

Technical Challenges & Resolutions

Engineering Challenge
Hallucinations on Technical Proofs: Vanilla LLMs tended to fabricate mathematical proofs when given partial context.
Architectural Solution
Engineered a strict system prompt with zero-shot guardrails that penalizes unverified speculation and requires verbatim citations.
Engineering Challenge
Inference Latency on Consumer GPUs: Full precision 16-bit models resulted in 4+ second response times on workstation GPUs.
Architectural Solution
Switched to 4-bit quantized GGUF weights running through Ollama's Metal/CUDA acceleration, dropping TTFT (time-to-first-token) to under 450ms.

Have a Project in Mind?

Whether you're building production AI pipelines, full-stack applications, or enterprise business software, let's connect and discuss your vision.

Get in Touch with Martin