Skip to content
EMARQUE AI

Private RAG on-prem: reference architecture.

This is a reference design for letting staff ask questions of the company's own documents, with the model, the retrieval stack and the application all running on hardware you own. It is a starting point to adapt to your corpus and your permissions model, not a product you order off this page.
Published . Updated .
The idea

What is RAG?

RAG stands for retrieval-augmented generation. Before the AI model answers a question, the system searches your own documents for the passages that matter and gives them to the model with the question. The model then answers from those passages, and the answer can point to the documents it came from.

It lets a general model answer questions about your company's policies, contracts or manuals without retraining it. Run privately, the documents, the search index and the model all stay on hardware you own.

The stack

Six layers, from the hardware up.

1. Hardware

AI PRO 500 (up to four RTX PRO 6000 Blackwell, 384 GB) for a single department; EMARQUE AI Server (eight RTX PRO 6000 Blackwell Server Edition, 768 GB) where the whole organisation uses it. ECC memory throughout and a Gen5 NVMe pool so the vector index sits on local disk. These cards talk over PCIe Gen5 with no NVLink between them, so a model kept inside one card's 96 GB avoids paying for a split.

2. Inference runtime

vLLM in production, Ollama for smaller deployments, NVIDIA Triton where one endpoint has to serve mixed model types. Pinned versions, NUMA-aware placement, paged attention on. An open-weight model (Qwen3.8 27B, Gemma 4, gpt-oss-120b, or the sparse Qwen3.6 35B-A3B) sized to fit GPU memory at the concurrency you expect.

3. Embedding and retrieval

Qwen3 Embedding or EmbeddingGemma 2 for dense vectors, multilingual where the corpus needs it. Hybrid search: dense plus BM25 plus a reranker. The vector store runs on disk inside the network (pgvector, Qdrant, Weaviate), with no SaaS retrieval step.

4. Document pipeline

Tika or unstructured for parsing, chunking at semantic boundaries, dedup, and a link back to the source of record. Permission tags come across from your SSO or IAM, and retrieval filters results by each user's permissions at query time.

5. Application layer

Streaming chat over SSE, citations in the response object, retrieval traces logged for audit. Light orchestration (LangGraph, LlamaIndex Workflows, or a thin in-house layer). Add orchestration steps only where a single retrieve-then-answer pass is not enough.

6. Evaluation and observability

A golden set of 100 to 500 representative questions with curated answers, run nightly against retrieval recall and generation faithfulness. Prometheus and Grafana on the inference path, with an alert on first-token latency drift.

Sizing

What each machine suits.

Model size and the number of people using it at once are the two figures that decide the machine. Use this as a starting point and refine it with the full sizing guide. Every system is quoted on request.

DGX SparkAI PRO 500AI ServerAI HGX B300
GPU memory128 GB unifiedUp to 384 GB (4 × 96 GB)768 GB (8 × 96 GB)2.1 TB HBM3e
GPU interconnectSingle GB10PCIe Gen5PCIe Gen5NVLink on the HGX baseboard
Model size it suitsUp to 200B parameters (NVIDIA figure)27B to 70B class, plus QLoRA fine-tunes70B-class serving, or several models side by sideTraining at scale, multi-tenant serving
Network10 GbE, ConnectX-7 at 200 Gb/s10 GbEConnectX SuperNICConnectX-8, 8 × 800 Gb/s OSFP
Where it sitsA desk5U rack6.5U rackData centre rack
Before you commit hardware

Pre-deployment checklist.

  • Source-of-record corpus identified: file paths, owners, refresh cadence
  • Permission model documented: ACL fields, SSO group structure
  • Latency budget agreed (first token, full response)
  • Evaluation golden set drafted with stakeholders
  • Air-gap and data residency requirements signed off
  • Network and firewall posture confirmed with IT

Tell us about your workload.

Tell us what it needs to do and who will use it. A few lines is enough. We reply by email, and every quote comes in writing, in MYR.