Skip to content
EMARQUE AI

How to size an on-prem AI system.

Eight figures we ask for before quoting hardware. Bring them to the first call and we come back with a sized system. Without them, scoping takes several conversations.
Published . Updated .
1

Model size and precision

Parameter count, precision and architecture together. At 4-bit, weights come to roughly half a gigabyte per billion parameters; at FP16, about two gigabytes. A sparse model does not change that: Qwen3.6 35B-A3B activates around 3B parameters per token, but all 36B have to be resident. Every figure below depends on this one, so it is the one to settle first.

What this drives

GPU memory, GPU count, KV-cache headroom

2

Concurrent sessions

Peak concurrent active sessions, not registered users. Each session holds its own KV cache in GPU memory, so an engine that is comfortable for a few people at a time will queue once a whole department is on it. Give us the realistic peak.

What this drives

GPU count, parallelism strategy, batch scheduler

3

Latency budget

Time to first token and time to full response, separately, and per workload: chat, retrieval, batch. A chat window and an overnight batch job have different tolerances. The figures you agree here are what the configuration is sized against.

What this drives

GPU choice, tensor-parallel size, network fabric

4

Context window

Average and tail length. An 8K average behaves differently from a 200K tail, because the KV cache grows with context and competes with the weights for the same GPU memory. Long-context work is where memory per GPU matters more than compute.

What this drives

GPU memory per card, paged attention, batch size

5

Deployment site

Office, server room, colocation, data centre, edge or air-gapped. The site sets the power envelope, the cooling, the noise you can live with, the networking, and which classes of system will physically install.

What this drives

Form factor, power, cooling

6

Compliance and data residency

PDPA, ISO 27001, air-gapped operation, or sector rules such as BNM's technology-risk requirements for the financial institutions it regulates. This decides whether the system may reach anything outside your network, what it has to log, and which documents ship with it.

What this drives

Software hardening, audit logging, firewall posture, paperwork

7

Timeline

The date you are working to, and what has to be true by then. EMARQUE-built servers are assembled and tested in Malaysia. NVIDIA's own systems and the rack-scale NVL72 platforms are sourced through NVIDIA's OEM partners and coordinated with them. Tell us the date before settling on a specific GPU.

What this drives

EMARQUE-built or NVIDIA system, OEM variant

8

Who operates it

Who runs the machine after delivery. A team with its own HPC staff can operate a rack-scale fabric directly. A team without one usually wants the software and support that comes with an NVIDIA system, or our support arrangement on an EMARQUE-built one.

What this drives

EMARQUE-built or NVIDIA DGX, support arrangement, runbook

Three common sizing mistakes

Where sizing usually goes wrong.

Buying by GPU spec sheet

Picking the GPU first, then looking for a chassis and a room that will take it. The deployment site usually constrains the choice more than the GPU does.

Sizing for this month

Sizing for the people using it now rather than the number you expect in a year. Adding GPUs later only works if the chassis, power and PCIe lanes were specified for it.

Skipping who operates it

Specifying a rack-scale system for a team with nobody to run it, or specifying an EMARQUE-built server when procurement requires an NVIDIA-branded machine.

Tell us about your workload.

Tell us what it needs to do and who will use it. A few lines is enough. We reply by email, and every quote comes in writing, in MYR.