A step-by-step guide to building a local RAG pipeline that lets you chat with any PDF document using LangChain, FAISS, and Groq's Llama 3.1 — no cloud storage required.
Was this helpful?
Retrieval-Augmented Generation (RAG) is one of the most practical patterns in applied AI. Instead of fine-tuning a model on your data, you give the model relevant context at query time — so it can answer questions grounded in your actual documents.
In this post, I'll walk through the exact architecture I used to build my PDF Q&A Agent: a fully local RAG pipeline that lets you chat with any PDF, with cited source pages.
The pipeline has three stages:
PDF → chunks → embeddings → FAISS index
↓
query → embedding → top-3 chunks → Groq Llama 3.1 → answer
I'm using uv for fast dependency management:
uv init pdf-qa-agent
cd pdf-qa-agent
uv add langchain langchain-community faiss-cpu sentence-transformers pypdf langchain-groqCreate a .env file:
GROQ_API_KEY=your_groq_api_key_herefrom langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
def load_and_chunk(pdf_path: str):
loader = PyPDFLoader(pdf_path)
pages = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=120, # overlap preserves context across chunks
separators=["\n\n",
The chunk_overlap=120 is important — without it, a sentence split across a chunk boundary loses context on both sides.
I'm using all-MiniLM-L6-v2 from HuggingFace — it's fast, lightweight, and runs entirely local.
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
import os
CACHE_DIR = ".faiss_cache"
def get_vector_store(chunks, pdf_name: str):
cache_path = os.path.join(CACHE_DIR, pdf_name)
embeddings = HuggingFaceEmbeddings(
model_name="sentence-transformers/all-MiniLM-L6-v2"
)
if os.path.exists(cache_path):
print("Loading cached index..."
Caching the index to disk means subsequent runs are instant — no re-embedding needed.
from langchain_groq import ChatGroq
from langchain.prompts import ChatPromptTemplate
from langchain.schema.runnable import RunnablePassthrough
from langchain.schema.output_parser import StrOutputParser
def build_qa_chain(vector_store):
retriever = vector_store.as_retriever(search_kwargs={"k": 3})
llm = ChatGroq(
model_name="llama-3.1-8b-instant",
temperature=
import sys
def main():
if len(sys.argv) < 2:
print("Usage: python main.py path/to/document.pdf")
sys.exit(1)
pdf_path = sys.argv[1]
pdf_name = os.path.basename(pdf_path).replace(".pdf", "")
print(f"Loading: {pdf_path}")
chunks
uv run python main.py research-paper.pdf
# Output:
# Loading: research-paper.pdf
# Created 47 chunks
# Building index...
# Ready. Type your question (Ctrl+C to exit)
# Q: What is the main contribution of this paper?
# A: The paper proposes a novel attention mechanism...RetrievalQAWithSourcesChain returns which page each answer came fromstream=True — important for long answersThe full source is on GitHub. If you have questions or want to extend this, drop a comment below.