#Gemma-4-31B-it

Gemma4ForConditionalGeneration is a dense multimodal model. This validation covers its text-generation serving path: the text backbone has 60 layers, hidden size 5,376, 32 attention heads / 16 KV heads, a 262,144-token vocabulary, and a native 262,144-token context window. The validated deployment is P4+D4: one 4-card Prefill engine and one 4-card Decode engine, coordinated by KServe LLMInferenceService and the InferNex Bridge. It is an 8 × Ascend 910B3 64 GB composite service, not a 4-card replica.

#Model identity

#Validated hardware × stack

#Model configuration

#Why P4+D4

The earlier aggregated TP4 deployment and P4+D4 are different resource envelopes. The aggregate service uses four NPUs and mixes long Prefill work with Decode; P4+D4 uses eight NPUs and removes Prefill dynamic steps from the Decode process. In the earlier capacity study, aggregate TP4 reached only 2.554 completed RPS on the 512/256 chatbot workload at target rate 3 over 900 seconds, with Running=104 and Waiting=294 at the end. P4+D4 is therefore the production candidate, but its throughput must never be presented as a four-card aggregate result.

#Deploy

The deployment spec is a self-contained KServe LLMInferenceService template with a TP4 Decode template, a TP4 Prefill template, InferNex router/scheduler, InferencePool, and the external route.

Before applying it, set the namespace, the two node names, the model PVCs, and registry references for your cluster. The manifest assumes the P/D pair has equal TP size and that the two engines can use Ascend Direct transport.

base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/plan/validated_models/assets/gemma4-31b-it
kubectl apply -f $base/gemma4-31b-it-pd-p4d4-llmisvc.yaml

# Production MaaS ingress:
curl -s https://<maas-host>/v1/chat/completions \
  -H "Authorization: Bearer $MAAS_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"google/gemma-4-31B-it","messages":[{"role":"user","content":"hello"}]}'

#Benchmark results

Closed-loop aiperf 0.7.0, driven through MaaS with its API key. Both scenarios run 480 requests per concurrency tier at concurrency 8 / 16 / 32, with warmup equal to the tier's concurrency. TTFT / E2E are seconds, ITL is ms, Decode is output-only token throughput, and TPS is input-plus-output token throughput.

Scenario ① — fixed-length system-prompt reuse (ISL 8,013 / OSL 128)

Scenario ② — multi-turn dialogue (average ISL 17,933 / OSL 128)

Every tier completed 480 requests with zero errors. Throughput and ITL are flat from concurrency 8 onward in both scenarios, while TTFT grows with concurrency. The service is Prefill-capacity-bound at roughly 8k input-dominated tokens/s: increasing average ISL from 8,013 to 17,933 lowers request throughput from 1.02 to 0.44 requests/s, while total token throughput changes by about 4.8%. Decode ITL remains stable near 29 ms.

#Real coding-agent validation

Gemma-4 was also driven as a real coding agent through the same MaaS gateway, with reasoning/thinking enabled. Pi 0.80.6 ran the fixed 14-task capacity14 Terminal-Bench selection at concurrency 1 / 8 / 32. Concurrency 1 and 8 use one attempt per task; concurrency 32 uses three attempts per task, for 42 trials. An objective task verifier produced the scores below.

The harness reports Pass@2 = 54.8% for the concurrency-32 run. Concurrency 8 is the best observed Pi operating point: increasing to 32 did not improve pass rate or correct-task throughput and introduced 15 AgentTimeoutError trials. During the run the Gemma Prefill, Decode, scheduler, and proxy Pods remained Ready with zero restarts and no serving 5xx/traceback, so these timeouts are recorded as Agent/task-level failures rather than an engine crash. OpenCode capacity14 is still running and will be added separately; the earlier mini7 result is smoke evidence and is not used as the formal Agent score.