Inference Engineering free

reviewed by skillbay: we re-ran the example ourselves and the skill changed the answer. see our run

Design, benchmark, tune, and operate inference systems from workload and quality requirements.

before and after

the same prompt, answered without the skill and then with it.

We are building a customer support assistant with Llama 3.1 8B Instruct. Traffic is 200 concurrent users on one 24 GB GPU. Should we use INT4 quantization and maximum batching? Recommend a production setup.

No—use INT4, but do not configure “maximum batching” blindly.

For one 24 GB GPU, I recommend:

- **Model:** Llama 3.1 8B Instruct with high-quality **weight-only INT4** quantization, preferably AWQ/GPTQ or an engine-native equivalent. Validate answer quality on your support test set.
- **Serving engine:** **vLLM** for operational simplicity, or TensorRT-LLM if you need maximum NVIDIA-specific performance. Both support continuous/in-flight batching and paged KV caching. [vLLM](https://docs.vllm.ai/), [TensorRT-LLM](https://developer.nvidia.com/tensorrt-llm)
- **Batching:** Use **continuous batching**, with a bounded token budget and request admission queue. Avoid a fixed “maximum batch size.”
- **Context limits:** Start with 4,096–8,192 total tokens per request. Enforce:
  - input limit: approximately 3,000–6,000 tokens
  - output limit: 256–512 tokens
  - stop sequences and concise support prompts
- **Concurrency:** Accept 200 client connections, but allow only roughly **16–32 active generations initially**. Queue the rest and measure queue latency. The correct value depends heavily on prompt length, output length, GPU model, and latency target.
- **Memory target:** Reserve roughly 10–15% of VRAM for runtime overhead. Do not allocate all remaining memory to KV cache.
- **Reliability:** Add request timeouts, cancellation, per-tenant rate limits, queue limits, and graceful overload responses.
- **Quality fallback:** Keep a higher-precision checkpoint available for offline evaluation and difficult cases. INT4 can cause noticeable degradation in factuality, formatting, or tool/function-call reliability.

The key constraint is KV cache, not just model weights. INT4 may reduce the model from roughly 16 GB in FP16 to about 5–6 GB, but each active request still consumes KV-cache memory proportional to its context. Two hundred simultaneous 4k-token conversations can exceed the remaining VRAM even with INT4.

A practical initial configuration is:

```text
quantization: AWQ or GPTQ INT4
max_model_len: 4096
max_num_seqs: 24
max_num_batched_tokens: 8192–16384
gpu_memory_utilization: 0.88–0.92
max_new_tokens: 384
```

Then load-test with realistic distributions, not 200 identical short prompts. Increase `max_num_seqs` only while p95 time-to-first-token, p95 inter-token latency, GPU memory, and queue time remain within your SLOs.

If the requirement is truly 200 users with low interactive latency—not merely 200 connected users—one 24 GB GPU is unlikely to be sufficient. Use multiple replicas behind a router, or a larger GPU.

## Recommendation

Do not use INT4 plus “maximum batching” as the production strategy.

Use INT4 only if your quality evaluation passes, and use bounded continuous batching with admission control. A single 24 GB GPU cannot reliably serve 200 simultaneous, long-lived chat generations with low latency. Treat 200 users as peak demand, then queue or scale replicas.

### Why

For Llama 3.1 8B:

- FP16/BF16 weights require roughly 16 GB before runtime overhead.
- INT4 weights require roughly 4–6 GB, but this does not solve the main concurrency problem: KV cache.
- Approximate KV-cache cost is about 128 KB per cached token per request.
- At 2,000 cached tokens, 200 active requests would require roughly 51 GB of KV cache alone.
- Larger batches improve aggregate throughput but consume KV memory and increase queueing, TTFT, and tail latency.

The model supports a 128K context window, but that is not a realistic per-request serving limit on this GPU. vLLM supports continuous batching, INT4 formats, memory controls, and KV-cache quantization, but its documentation also recommends limiting context length and maximum sequences when conserving memory. [vLLM quantization documentation](https://github.com/vllm-project/vllm), [vLLM memory configuration](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md), [Llama 3.1 model card](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)

## Initial production setup

For one 24 GB GPU, start with:

- vLLM or an equivalent production LLM engine
- Weight-only INT4, preferably GPTQ/AWQ or another format with tested kernels for the exact GPU
- `max_model_len`: 2,048–4,096 tokens
- `max_num_seqs`: start at 16; sweep 8, 16, 32, and 48
- Bounded continuous batching, not an unlimited batch
- `max_num_batched_tokens`: start around 8,192
- Reserve at least 15–20% VRAM for KV cache growth, workspaces, fragmentation, and runtime overhead
- Strict output-token cap, initially 256–512
- Bounded request queue with timeout and overload rejection
- Streaming responses
- Per-tenant rate limits and cancellation when clients disconnect
- Metrics for TTFT, inter-token latency, P95/P99 latency, queue time, active sequences, KV-cache utilization, rejected requests, and tokens per second

Do not expose the full 128K context to clients. Summarize or retrieve older conversation history.

## Scaling recommendation

If 200 users can be active at once, use multiple serving replicas. A reasonable starting topology is:

- 2–4 replicas, one GPU each, with INT4 weights
- Route requests by queue depth and remaining KV capacity
- Keep a small admission buffer instead of running every GPU at 100% memory utilization
- Autoscale on queue age, active token count, KV utilization, and P95 latency—not request count alone

If the latency target is strict, use more capable GPUs or more replicas rather than forcing a larger batch on the 24 GB card.

## Required benchmark before launch

Test the BF16/FP16 baseline and INT4 candidate on the same support workload:

- short, medium, and long prompts
- normal and peak output lengths
- 1, 8, 16, 32, and 48 concurrent generations
- representative tool calls and retrieved documents
- customer-support quality checks: factuality, policy adherence, escalation accuracy, refusal behavior, and answer completeness

Accept INT4 only if quality remains above the product threshold and P95/P99 latency improves or capacity increases materially. Roll back to the higher-precision model if quality or tail latency regresses.

Bottom line: use INT4 as a capacity optimization, not as permission to maximize batching. For 200 concurrent users, plan for bounded concurrency plus multiple replicas; one 24 GB GPU should be treated as a measured serving unit, not the entire production fleet.

**Short answer:** No to both as stated. INT4 might be the right choice, but that depends on which 24 GB GPU you have. "Maximum batching" is the wrong goal: set the batch size to meet a latency target, not to fill the card. Before any of that, check that you really have 200 requests in flight at once, because you almost certainly don't.

I haven't benchmarked anything here. The numbers below are my own sizing calculations, and you should confirm them with a load test on your hardware.

## 1. 200 users is not 200 requests
A support user reads, thinks and types for 20–60 seconds between messages, and each reply takes about 3–10 seconds to generate. So 200 connected users usually means roughly **20–60 requests running at once**. Before choosing quantization, measure or simulate the arrival rate with realistic think time. That number sets your KV cache budget and batch size.

## 2. Memory math (Llama 3.1 8B: 32 layers, 8 KV heads, head dim 128)
- **KV cache:** about 128 KiB per token in BF16, about 64 KiB in FP8.
- **Usable memory:** about 21–22 GB at 0.90–0.92 GPU utilization, minus about 1.5 GB for activations and CUDA graphs.

| Weights | Weight memory | KV budget | Tokens (BF16 KV) | Tokens (FP8 KV) |
|---|---|---|---|---|
| BF16 | ~16 GB | ~4 GB | ~32k | ~64k |
| FP8 (W8A8) | ~8.5 GB | ~11 GB | ~90k | ~180k |
| INT4 (W4A16, AWQ/GPTQ) | ~6 GB | ~14 GB | ~110k | ~220k |

A support turn is about 3–4k tokens: system prompt, policies, retrieved docs and chat history. At 40–60 active requests you need 150–250k tokens. **BF16 weights won't fit, so you do need to quantize.** Prefix caching (section 3) stretches these budgets a lot, because the shared system prompt is stored once, not once per request.

## 3. Recommended setup

**Engine:** vLLM or SGLang on the single GPU. Keep CUDA graphs on (never use `--enforce-eager` in production). Use streaming responses.

**Weights, chosen by GPU generation:**
- **Ada or Hopper (L4, L40S, RTX 4090):** use **FP8 W8A8**. It is close to lossless on 8B and these cards run FP8 natively. It fits the memory budget, so I'd take it over INT4.
- **Ampere (A10G, A5000, 3090):** these cards have no native FP8, so use **INT4 W4A16 (AWQ or GPTQ with Marlin kernels)**. On these cards INT4 is the right call.
- Why I lean away from INT4 when you have a choice: on a model this small, INT4 costs more quality than it does on 70B. It tends to show up in instruction following, policy adherence and structured or tool-call output, which are exactly what a support bot needs.

**KV cache:** FP8 (`--kv-cache-dtype fp8`). This roughly doubles concurrency and the quality cost is usually small, but check it in your eval.

**Prefix caching:** keep it on. This is the biggest single win for support traffic. Put the fixed content first (system prompt, policies, product docs) and the variable content last (retrieved documents, user message) so the cache prefix stays shared.

**Batching:** set `--max-num-seqs` from a latency sweep, not to its maximum. Decode on an 8B model is limited by memory bandwidth, so bigger batches raise total throughput but slow every user's token rate. Pick targets such as TTFT p95 under 1 s and inter-token latency p95 under 50–70 ms. Then sweep 32, 64, 96 and 128, and choose the largest value that still meets them. Keep chunked prefill on so long prompts don't stall other users' decoding.

**Context length:** set `--max-model-len` to what you actually need (8k or 16k). Leaving it at 128k just invites runaway requests.

**Speculative decoding:** skip it at this load. It helps at low batch sizes and hurts at high ones.

Starting point (flag names match recent vLLM; confirm with `vllm serve --help`, and check the model repo name exists before using it):
```bash
vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 \   # or a w4a16 checkpoint on Ampere
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-model-len 8192 \
  --max-num-seqs 64 \          # tune by SLO sweep
  --gpu-memory-utilization 0.92
```

## 4. Production requirements beyond the model server
1. **Quality gate:** build a golden set of 200–500 real tickets. Score BF16 against your quantized build on correctness, policy adherence, refusals and JSON or tool-call validity. Only ship the quantized version if it passes.
2. **Load test:** use realistic arrival patterns with think time (tools like AIPerf or GuideLLM work). Report TTFT and inter-token latency at p50 and p95, plus how deep the queue gets at peak.
3. **Admission control:** cap the queue per replica, return a fast "busy" or fall back to human handoff, and rate-limit per user. When the server is overloaded, rejecting quickly is better than letting every request slow down.
4. **Watch GPU bandwidth:** an L4 has about 300 GB/s, an A10G about 600 GB/s and a 4090 about 1 TB/s. That can mean a 3x difference in per-user speed at the same batch size, and on an L4, 200 users may need a second GPU no matter how you quantize.
5. **No single point of failure:** one GPU means any crash, driver problem or upgrade takes support offline. Run at least two replicas behind a load balancer, even if one could carry the peak on its own.
6. **Metrics:** track KV cache usage, prefix-cache hit rate, number of queued requests, TTFT and ITL, and alert on them.

**Summary:** use FP8 weights on Ada or Hopper, or INT4 AWQ/GPTQ on Ampere, with FP8 KV cache and prefix caching. Set the batch size from a latency target, and run two replicas. If you tell me the exact GPU and your typical prompt length, I can narrow the concurrency estimate.

**Short answer: no to both as stated.** Start with FP8 weights and FP8 KV cache, not INT4. Cap the batch at the size where your latency targets still hold, not at the maximum. Also, "200 concurrent users" probably means far fewer requests running at the same time than you're sizing for. And one GPU on its own isn't a production setup, because nothing takes over when it fails.

## 1. Concurrent users are not concurrent requests

Support chat users spend most of their time reading and typing. Using Little's law:

```
in-flight requests ≈ users × (generation time / time between messages)
                   ≈ 200 × (5 s / 30–60 s) ≈ 17–35 at the same time
```

These are estimates, so measure your real message rate. But you're probably sizing for about 30 active sequences with bursts to 50, not 200. That changes the whole memory picture.

## 2. Memory math on 24 GB (Llama 3.1 8B: 32 layers, 8 KV heads, head dim 128)

KV cache per token is 128 KiB in BF16 and 64 KiB in FP8. Assume about 21.5 GB usable at 90% utilization, minus about 2 GB for activations and CUDA graphs.

| Weights | Weight size | Left for KV | KV tokens (FP8 KV) | 4k-token conversations |
|---|---|---|---|---|
| BF16 | ~16 GB | ~3.5 GB | ~28k at BF16 KV | ~7. **Too small.** |
| **FP8 (W8A8)** | ~8.5 GB | ~11 GB | ~175k | **~40, plus prefix sharing** |
| INT4 (AWQ/GPTQ) | ~5.7 GB* | ~14 GB | ~220k | ~55 |

\*The embeddings and output head (about 2 GB) usually stay at 16-bit.

INT4 only gets you about 30% more KV room than FP8. You pay for it in quality risk, and small models like an 8B lose more to 4-bit than large ones do. The usual casualties are instruction following, structured and tool-call output, and policy compliance, which is exactly what a support bot relies on. W4A16 kernels also lose much of their speed advantage at larger batches, where the dequantization overhead starts to matter.

**Which GPU you have matters:**
- **Ada/Hopper (RTX 4090, L4):** use FP8 W8A8 with FP8 KV.
- **Ampere (A10/A10G, RTX 3090):** there is no FP8 hardware, so use INT8 W8A8 for the weights. Check which KV cache formats your runtime supports there.
- **L4 specifically:** it has only about 300 GB/s of memory bandwidth, roughly a third of a 4090 or 3090. Token generation is limited by bandwidth, so an L4 will struggle to keep 30+ streams at a comfortable reading speed. Benchmark it before you commit.

Use INT4 only if a benchmark shows you're still short on KV capacity after FP8, and only if your own support evals pass.

## 3. Batching: set a cap, then queue and shed

Maximum batching gives the most total throughput, but every user's tokens stream more slowly and tail latency grows. A better approach:
- Run a sweep of concurrency (1, 8, 16, 32, 48, 64). Set `max_num_seqs` at the knee where p95 time to first token and time between tokens still meet your targets. For example, first token under 1 s and at least 20 tokens/s per user.
- Keep chunked prefill on, so long incoming prompts don't stall other users' streams.
- Anything above the cap waits in a bounded queue. When the queue is full, return a fast "busy" response or overflow to a fallback. Don't let latency degrade for everyone.

## 4. The biggest free win: prefix caching

Every support request starts with the same system prompt, policies and tool schemas. Put all the stable content first and the per-user content last, and turn on prefix caching. That removes most of the prompt-processing work and shares KV memory across users.

Two isolation rules:
- Don't share cache entries across customers' private data. vLLM supports a per-request cache salt for this.
- Treat a change to the model, tokenizer or chat template as a cache invalidation.

## 5. Suggested starting config (vLLM; check flags against current docs)

```
vllm serve <FP8 checkpoint of Llama-3.1-8B-Instruct>
  --kv-cache-dtype fp8
  --max-model-len 8192           # don't reserve 128k context you won't use
  --gpu-memory-utilization 0.90
  --max-num-seqs 32              # starting point; set from the sweep
  --enable-prefix-caching        # on by default in recent versions
```

Keep CUDA graphs on and stream responses. I couldn't check current vLLM or SGLang docs or FP8 checkpoint availability from here, so confirm the flag names and pick a validated FP8 checkpoint before you deploy.

## 6. Production essentials

- **Two replicas, or one GPU plus an API fallback.** A single GPU means any crash, OOM or driver problem takes support down. This matters more than any quantization choice.
- **Quality gate before any quantization.** Build 200–500 real support transcripts that cover refunds, escalation, refusals, tool calls, long histories and prompt-injection attempts. Score BF16 first as the baseline, then FP8, then INT4 only if you need it. Promote a format only if the drop is within the tolerance you set in advance.
- **Load test** with real prompt and output lengths and real arrival bursts, not uniform synthetic traffic.
- **Monitor:** p50/p95/p99 time to first token, time between tokens, queue depth, KV cache usage, preemptions, prefix cache hit rate, rejection rate and error rate. Keep a documented way to roll back to the previous config.
- **Check the model choice.** Llama 3.1 8B is from 2024. Run your support evals against one or two current models of similar size before locking it in.

**The order to work in:** measure real concurrency, build the eval set, then run BF16 as a baseline. Move to FP8 weights and FP8 KV, restructure prompts for prefix caching, and set the batch cap from the sweep. Add a second replica or fallback. Consider INT4 only if the data shows you still need it.

All the numbers above are first-pass estimates from the model's architecture, not measurements; I ran nothing in this environment. Tell me which 24 GB GPU you're on and your typical prompt and response lengths, and I can tighten the capacity estimate.

about this skill

SKILL.md

---
name: inference-engineering
description: Design, size, benchmark, tune, debug, or review machine-learning model inference on local or production hardware. Use for accelerator fit, memory and bandwidth limits, serving runtimes, quantization, batching, KV cache, speculation, parallelism, latency, throughput, cost, capacity, reliability, and production operations. Do not use for hosted-model API documentation or pricing alone, AI media generation through inference.sh, GPU graphics, or generic application deployment.
---

# Inference Engineering

Treat inference as a product system, not a model-server setting. The model, runtime, hardware, router, autoscaler, network, client, workload, and quality threshold jointly determine the result.

## Select A Mode

- **LEARN:** Explain the system or derive a mental model. Read [fundamentals-and-hardware.md](references/fundamentals-and-hardware.md).
- **DESIGN:** Turn a workload into a model, hardware, runtime, and topology decision. Read the fundamentals reference, then [serving-and-optimization.md](references/serving-and-optimization.md). For speech, image, video, embeddings, or VLMs, also read [modalities.md](references/modalities.md).
- **MEASURE:** Design, run, or review a benchmark, experiment, or capacity model. Read [benchmark-and-capacity.md](references/benchmark-and-capacity.md) and the reference for the surface being measured.
- **OPERATE:** Design or review scaling, routing, deployment, observability, failure recovery, or cost controls. Read [production-operations.md](references/production-operations.md).

If the task depends on current models, hardware, prices, runtime features, commands, compatibility, or benchmarks, also read [source-map.md](references/source-map.md) and verify the relevant official source live.

## Governing Sequence

Follow this causal order:

1. Define the workload and first useful user outcome.
2. Set a task-specific quality floor.
3. Separate prefill, decode, preprocessing, queueing, transport, and postprocessing.
4. Classify the limiting resource: quality, compute, bandwidth, memory capacity, communication, queueing, or operations.
5. Choose the smallest intervention that can remove the measured limit.
6. Compare against a preserved baseline under production-shaped traffic.
7. Adopt only when quality, service, cost, and rollback gates pass.

Do not name a GPU, runtime, or optimization before the workload is clear. Do not accept a faster kernel, higher token rate, or successful model load as product proof.

## Invariants

- Use product-specific evaluations before model optimization. Prefer the smallest well-supported model that clears the quality gate.
- Keep shared inference while requirements are unclear. Recommend dedicated serving only for a measurable scale, specialization, control, privacy, or orchestration need.
- Treat prefill as commonly compute-bound and decode as commonly bandwidth-bound only as starting hypotheses. Profile the exact workload.
- Include weights, KV cache, activations, workspaces, fragmentation, replicas, and margin in memory sizing.
- Require a full-precision or current accepted-quality baseline before lossy quantization, changed attention, caching approximations, or distilled models.
- Change one major variable per experiment. Test the intended combination after isolated tests because optimizations can interfere.
- Separate measured facts, documentation facts, estimates, and hypotheses.
- Treat vendor claims and named product rankings as time-sensitive. Current official documentation and target-stack measurements control.

## Default Output

Return only the parts the task needs, but preserve this decision record when making a recommendation:

```text
Mode:
Workload and first useful outcome:
Quality gate:
Service objectives by workload bucket and cache state:
Cost objectives:
Known facts:
Assumptions and unknowns:
Phase and bottleneck diagnosis:
Candidate intervention:
Baseline and experiment:
Measured or expected tradeoffs:
Acceptance and rollback:
Current facts that need verification:
```

Lead with the decision in product language. Keep calculations and evidence available as supporting detail.

package contents

post id: i0l570d27tjcdtnk