RAG on the edge — Llama-3.1 8B + OpenVINO INT4 quantization for document Q&A.
A complete retrieval-augmented generation system pairing Llama-3.1 8B Instruct with Intel's OpenVINO GenAI toolkit. INT4 quantization cuts model size by 73% with under 5% quality loss; FAISS-GPU vector search and a heterogeneous Intel-plus-NVIDIA pipeline give 2–5 second query latency on a single workstation, with no cloud dependency.