[NMCUR-421] Add inference-server PDF benchmark path

[praateekmahajan]

Description

Adds reusable benchmark utilities for starting Ray Serve or Dynamo inference servers and moves the existing NDD benchmark onto them. Shared inference-server CLI JSON arguments are validated through one reusable parse_json_object helper.

Adds NemotronParseHTTPClientStage to the production Nemotron-Parse stage package. It sends OpenAI-compatible multimodal page requests through Curator's existing AsyncOpenAIClient, including the client's bounded concurrency and exponential-backoff retries, while preserving response order, raw completion usage, and finish metadata. inference_batch_size means pages per GPU forward pass for HF or maximum concurrent page requests for the HTTP client.

NemotronParsePDFReader creates exactly one inference stage: the existing in-process stage when no endpoint is supplied, or the HTTP client stage when inference_server_endpoint is set. The composite and tutorial factory accept a direct inference_server_client_num_workers value and do not model inference-server replicas. In-process proc_size remains internally derived from AutoProcessor; the HTTP client stage retains the model-default (2048, 1664) metadata needed by postprocessing.

The in-process vLLM stage and HTTP client stage derive their generation settings from one canonical plain mapping, including a shared parameterized max_tokens default of 8192. The HTTP adapter only reshapes vLLM-specific fields into the OpenAI-compatible extra_body wire format.

Server replicas default to get_available_cpu_gpu_resources()[1] and can be overridden with --num-replicas. The PDF benchmark exposes --inference-server-client-workers-per-replica, set explicitly to 4 in YAML, computes num_replicas * client_workers_per_replica, and passes that total to the composite as inference_server_client_num_workers.

The existing Xenna and Ray Data in-process workloads and throughput thresholds are retained. One Ray Data inference-server entry per server type covers Ray Serve and Dynamo, both with inference batch size 32. Dynamo uses its TCP request plane. PDF benchmark Slack sinks are disabled for now, and both inference-server entries are registered in benchmarking/4xGB200-64CPU.yaml.

For a startup-independent comparison between in-process and inference-server execution, the benchmark reports exactly three inference-stage metrics:

• inference_stage_pages_per_sec_per_gpu
• inference_stage_input_tokens_per_sec_per_gpu
• inference_stage_output_tokens_per_sec_per_gpu

They use the inference stage's task process-time sum normalized by stage parallelism and inference GPU count. Model/server startup is outside this stage timing. The intermediate process-time, active-time, GPU-time, and parallelism values are calculation details and are not emitted as benchmark metrics.

Usage

python benchmarking/scripts/nemotron_parse_pdf_benchmark.py \
  --benchmark-results-path=/tmp/results \
  --executor=ray_data \
  --inference-server-type=dynamo \
  --inference-server-client-workers-per-replica=4 \
  --inference-batch-size=32 \
  --manifest=/path/to/manifest.jsonl \
  --pdf-dir=/path/to/pdfs \
  --output-dir=/tmp/results/output \
  --model-path=/path/to/NVIDIA-Nemotron-Parse-v1.2 \
  --model-id=nvidia/NVIDIA-Nemotron-Parse-v1.2 \
  --backend=vllm

Four-way 8*H100 comparison

All four entries succeeded on PR head 12bccbdd, processed 5,794 pages, and produced zero request errors in the same run.

Entry	inference_stage_pages_per_sec_per_gpu	Mean GPU power draw (W)
nemotron_parse_pdf_xenna	1.995	216.3
nemotron_parse_pdf_raydata	2.063	190.5
nemotron_parse_pdf_inference_server_ray_serve_batch_32	3.929	234.9
nemotron_parse_pdf_inference_server_dynamo_batch_32	3.968	250.0

The inference-stage metric excludes model/server startup. Mean GPU power draw is the arithmetic mean across all eight GPUs and all gpustats.csv samples over each complete benchmark entry. Full results are available in the run viewer.

Validation

• Latest Curator nightly (nightly-2026-08-27): Ruff and formatting passed; focused OpenAI-client, inference, HTTP-client, and pipeline-factory coverage passed 25/25 tests.
• The broader targeted run passed 41 tests and had six unrelated nightly-environment failures from the changed pypdfium2.save() API and missing cv2 extra.
• Synthetic normalization confirmed identical three per-GPU inference-stage metrics for equivalent in-process and HTTP stage work at different worker counts.
• EOS Dynamo batch-32 and batch-64 runs both completed all 5,794 requests with no pre-shutdown errors. Batch 32 was retained because batch 64 improved aggregate throughput by only about 1.5% while increasing mean request latency by about 41%.
• The final four-way EOS run completed all four entries successfully at identical page counts with zero request errors.
• Scoped repository pre-commit hooks passed for the signed-off commits.
• No benchmark tests were added, per task direction.

Checklist

• I am familiar with the Contributing Guide.
• Existing tests cover the production-stage and pipeline-factory changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[sarahyurick]

Fyi @praateekmahajan for GB200s, I got it to work by setting up TIKTOKEN_RS_CACHE_DIR - #2348.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-inference-server-pdf-nmcur-421.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/curate-text/load-data/nemotron-parse-pdf
• nemo/curator/curate-text/synthetic/inference-server

[praateekmahajan]

/claude review
@greptileai review

[praateekmahajan]

/ok to test b901d27

[praateekmahajan]

/claude review
@greptileai review

[praateekmahajan]

/ok to test 4fc7724

[claude]

num_request_errors is hardcoded to 0.0 here and never incremented — a request that fails after retries propagates out of _query_pages/process and raises the whole task (as the class docstring intends). So this metric is always 0 on any run that completes.

That has two downstream effects worth confirming as intended:

• The docs table describes it as "Page requests that still failed after client retries," which will never be observable via this metric.
• In nemotron_parse_pdf_benchmark.py, the elif pdf_parse_metrics["num_request_errors"]: branch (line 265) is effectively dead, and the benchmark requirement num_request_errors exact_value: 0 is trivially satisfied.

If the intent is truly "raise on any failure," consider dropping the metric/branch or adjusting the docs; if partial-failure counting was intended, the gather path would need return_exceptions handling.