disaggregated-serving

A Datacenter Scale Distributed Inference Serving Framework

Composable and Embeddable Communication Runtime for Distributed AI Services

Benchmark & decision framework for KV cache transfer compression in disaggregated LLM serving — 14+ compressors, 6 models, GPU-calibrated timing, decision flowchart

A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.

Production-grade Java 25 Virtual Thread inference gateway bridging NVIDIA Triton → Dynamo with Earliest Deadline First (EDF) priority queuing, adaptive batching, and async shadow validation.

Reproducible capacity and throughput modeling for co-located vs disaggregated LLM Prefill/Decode serving

A/B profiling: aggregated vs disaggregated prefill/decode LLM serving on equal hardware, measured per-device with inferscope

Prefix KV sharing in disaggregated LLM serving: transfer vs recompute vs hotset/LRU replication under decode memory pressure.

Adaptive disaggregated inference on a role-free fleet.

Measuring the KV-transfer tax of disaggregated prefill/decode LLM serving (vLLM + NIXL, 4x A10G): on PCIe-only hardware, disagg loses to plain data-parallel replication — measured, committed, reproducible.

A fault-tolerant LLM routing system that decouples inference from AWS Bedrock by routing prefill and decode tasks through SQS and ensuring zero-downtime scaling with a graceful drain sidecar.

Automate protein design cycles from research goals to ranked wet-lab shortlists with git-like version history.