RAG on the edge — Llama-3.1 8B + OpenVINO INT4 quantization for document Q&A.
A complete retrieval-augmented generation system pairing Llama-3.1 8B Instruct with Intel's OpenVINO GenAI toolkit. INT4 quantization cuts model size by 73% with under 5% quality loss; FAISS-GPU vector search and a heterogeneous Intel-plus-NVIDIA pipeline give 2–5 second query latency on a single workstation, with no cloud dependency.
This paper is published in full on ReadyTensor. What follows is a working summary — abstract, headline results, and the architecture moves we found most interesting. Click through for the full methodology, references, and appendix.
Why this work
Most production RAG conversation today assumes you’re calling a frontier-model API over the network. That’s the easy path — and the wrong one for a meaningful subset of clients. Three settings where sending data to a hosted model is not an option:
- Healthcare and finance teams with explicit jurisdictional constraints
- Defence and government contracts where the air gap is not optional
- Field deployments — ships, mines, rural infrastructure — where the network is the least reliable thing in the system
The question becomes: how close can a fully local stack get to the latency and quality of a managed API, on hardware a customer already owns? This paper answers that for one realistic configuration.
The stack
| Component | Choice | Why |
|---|---|---|
| Generator | Llama-3.1 8B Instruct | Best open weights at this size for instruction-following |
| Quantization | OpenVINO GenAI INT4 | 73% size reduction, <5% quality loss measured on our eval |
| Embeddings | all-MiniLM-L6-v2 (sentence-transformers) | 384-dim, fast, well-understood |
| Vector store | FAISS-GPU, IndexFlatIP | Exact search, GPU acceleration, simple to reason about |
| Document processing | Custom semantic chunker | Section-aware boundaries, configurable overlap |
| Inference target | Intel UHD Graphics (LLM) + NVIDIA RTX 3080 (embeddings) | Heterogeneous use of hardware most workstations already have |
The deliberate move here is the heterogeneous GPU split. Most “local LLM” advice assumes you have one big NVIDIA card. We found that pinning LLM inference to the Intel iGPU via OpenVINO and the embedding model to the NVIDIA dGPU via CUDA freed each device to do what it does best — and let us hit our latency target on a workstation that costs ~$2,500 instead of one that costs ~$8,000.
Headline results
- Model size: 15.8 GB → 4.2 GB after INT4 quantization (73% reduction)
- Query latency: 2–5 seconds end-to-end after warm-up; 30–60s cold start
- Memory footprint: 8–12 GB stable during active use
- Quality: 92% factual accuracy, 87% on complex analytical questions, 95% on procedural questions across 100 manually-evaluated test queries; <5% delta vs. the unquantized baseline
- Concurrency: stable up to 10 simultaneous users with <20% latency degradation
- Cost (12-month TCO): roughly 60–70% cheaper than the equivalent cloud-API setup once we accounted for hardware amortisation and electricity
Three findings worth keeping
1. Section-aware semantic chunking matters more than chunk size
Fixed 512-token chunks with sliding windows are the default in most tutorials. We found a 15% relevance lift by chunking at section boundaries instead — using the document’s own structure (headers, paragraph breaks, formatting) as the boundary signal, with sliding-window overlap within sections. The retrieval payload is no smarter; the boundaries are just less arbitrary.
This is one of those tweaks that costs nothing to implement and quietly compounds across every query forever.
2. INT4 is the right Pareto point — for now
We tested INT8, INT4, and FP16 side-by-side. INT8 won on accuracy (1–2% better than INT4) but the model is 50% larger, which mattered both for memory headroom and for cache behaviour. INT4 was 15–20% faster than FP16 at inference, primarily because of reduced memory bandwidth — the quantization paid for itself twice over.
If you are not memory-bound, INT8 is a reasonable choice. If you are memory-bound — and on edge hardware you almost always are — INT4 wins.
3. The boring engineering is most of the work
The interesting research moves above are real, but they are not where the implementation time went. The implementation time went into:
- 103 unit and integration tests covering the pipeline end-to-end
- Pinned Conda dependencies and
run_demo.{sh,bat}scripts that work on Windows, Linux, and macOS - Graceful degradation when the GPU isn’t available — full CPU fallback, 3–5x slower but functional
- Real error handling for the things that actually happen: corrupted PDFs, network drops mid-download, models that fail integrity checks, GPU memory exhaustion
This is the part that separates a research demo from something a client can run in their own environment without us. None of it is glamorous; all of it is necessary.
Where this goes next
The current implementation is single-document and single-tenant. The obvious next steps:
- Multi-document collections with cross-document retrieval and confident provenance
- Domain-specific chunking strategies — the section-aware approach generalises poorly to legal contracts, structured forms, and conversational data
- Smaller hardware — getting this onto a 16 GB RAM laptop with a modest iGPU is the next bar
Each of these is a known problem with active research; none is easy. We’ll publish on the first one we ship.
Read the full paper
The complete paper — methodology, experimental setup, comparative analysis, references, and appendix — is published on ReadyTensor. Read it on ReadyTensor →
For commercial questions about deploying a system like this in your environment, info@theelixirslab.com.