Siddhant Ray

I am a fourth year PhD student in Computer Science at the University of Chicago, advised by Junchen Jiang and Nick Feamster. Overall, I am interested in building efficient inference systems for LLM and Physical AI applications, using application-specific feedback.

My current research focuses on two directions. First, I am developing new Vision-Language-Action (VLA) model architectures and serving systems for resource-efficient inference of long-horizon physical AI workloads. Second, I am exploring how to achieve better cost–quality tradeoffs in long-horizon LLM agent serving systems through joint agent-memory and KV Cache management.

Earlier I have worked on joint quality-delay optimizations in Retrieval-Augmented-Generation systems with query level configuration selection and resource management (METIS, SOSP'25). I also worked on Transformer models for per-packet latency prediction enabling tail-latency reduction for latency sensitive applications (SwiftQueue, NINeS'26).

In the past, I have worked on advances in Software Defined Networking, programmable networks and cloud computing. Additionally I have spent some time developing NLP techniques to analyse political corpora.

I'm fortunate to be additionally supported by the Liew Family Graduate Fellowship. Prior to starting my PhD, I earned my MSc in Electrical Engineering and Information Technology at ETH Zurich and my B.Tech in Electronics and Communication Engineering at VIT Vellore.

News

Sep, 2026	Argo: Efficient Importance Labeling for Enterprise Email Systems accepted at PACMI’26.
Jun, 2026	Started my research internship at Microsoft, Redmond working on efficient Physical AI inference.
Jan, 2026	SwiftQueue: Optimizing Low-Latency Applications with Swift Packet Queuing accepted at NINeS’26.
Oct, 2025	METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation accepted at ACM SOSP’25.
Sep, 2025	Serving as a Reviewer for AAAI’26 and ICLR’26 .

Selected publications

NINeS
SwiftQueue: Optimizing Low-Latency Applications with Swift Packet Queuing

Siddhant Ray, Xi Jiang, Jack Luo, Nick Feamster, and Junchen Jiang

In Proceedings of the New Ideas in Networked Systems (NINeS) Conference 2026

Abs arXiv Bib Slides Talk Cited by -

Low Latency, Low Loss, and Scalable Throughput (L4S), as an emerging router-queue management technique, has seen steady deployment in the industry. An L4S-enabled router assigns each packet to the queue based on the packet header marking. Currently, L4S employs per-flow queue selection, i.e. all packets of a flow are marked the same way and thus use the same queues, even though each packet is marked separately. However, this may hurt tail latency and latency-sensitive applications because transient congestion and queue buildups may only affect a fraction of packets in a flow. We present SwiftQueue, a new L4S queue-selection strategy in which a sender uses a novel per-packet latency predictor to pinpoint which packets likely have latency spikes or drops. The insight is that many packet-level latency variations result from complex interactions among recent packets at shared router queues. Yet, these intricate packet-level latency patterns are hard to learn efficiently by traditional models. Instead, SwiftQueue uses a custom Transformer, which is well-studied for its expressiveness on sequential patterns, to predict the next packet’s latency based on the latencies of recently received ACKs. Based on the predicted latency of each outgoing packet, SwiftQueue’s sender dynamically marks the L4S packet header to assign packets to potentially different queues, even within the same flow. Using real network traces, we show that SwiftQueue is 45-65% more accurate in predicting latency and its variations than state-of-art methods. Based on its latency prediction, SwiftQueue reduces the tail latency for L4S-enabled flows by 36-45%, compared with the existing L4S queue-selection method.
@inproceedings{ray2026swiftqueueoptimizinglowlatencyapplications, title = {{S}wift{Q}ueue: {O}ptimizing {L}ow-{L}atency {A}pplications with {S}wift {P}acket {Q}ueuing}, author = {Ray, Siddhant and Jiang, Xi and Luo, Jack and Feamster, Nick and Jiang, Junchen}, year = {2026}, eprint = {2410.06112}, archiveprefix = {arXiv}, primaryclass = {cs.NI}, booktitle = {Proceedings of the New Ideas in Networked Systems (NINeS) Conference}, url = {2410.06112}, talk = {https://www.youtube.com/watch?v=hr9fJZHlSJ8}, scholar_query = {SwiftQueue Optimizing Low-Latency Applications with Swift Packet Queuing Siddhant Ray} }
SOSP
METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation

Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Shaoting Feng, Ganesh Ananthanarayanan, Ravi Netravali, and Junchen Jiang

In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles 2025

Abs arXiv Bib Slides Talk Cited by -

RAG (Retrieval Augmented Generation) allows LLMs (large language models) to generate better responses with external knowledge, but using more external knowledge causes higher response delay. Prior work focuses either on reducing the response delay (e.g., better scheduling of RAG queries) or on maximizing quality (e.g., tuning the RAG workflow), but they fall short in systematically balancing the tradeoff between the delay and quality of RAG responses. To balance both quality and response delay, this paper presents METIS, the first RAG system that jointly schedules queries and adapts the key RAG configurations of each query, such as the number of retrieved text chunks and synthesis methods. Using four popular RAG-QA datasets, we show that compared to the state-of-the-art RAG optimization schemes, METIS reduces the generation latency by 1.64 – 2.54\texttimes without sacrificing generation quality.
@inproceedings{10.1145/3731569.3764855, author = {Ray, Siddhant and Pan, Rui and Gu, Zhuohan and Du, Kuntai and Feng, Shaoting and Ananthanarayanan, Ganesh and Netravali, Ravi and Jiang, Junchen}, title = {METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation}, year = {2025}, isbn = {9798400718700}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3731569.3764855}, doi = {10.1145/3731569.3764855}, booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles}, pages = {606–622}, numpages = {17}, keywords = {RAG systems, LLM inference, scheduling}, location = {Lotte Hotel World, Seoul, Republic of Korea}, series = {SOSP '25}, talk = {https://drive.google.com/file/d/1bMZf-38ubFO6hZk6ivZcsaL9kC72GEpa/view?usp=sharing}, scholar_query = {METIS Fast Quality-Aware RAG Systems with Configuration Adaptation Siddhant Ray} }
EuroSys
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang

In Proceedings of the Twentieth European Conference on Computer Systems 2025

Abs arXiv Bib Cited by - 🏅Best Paper

Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized.This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? This challenge naturally arises in retrieval-augmented generation (RAG) where the input is supplemented with multiple retrieved texts as the context. We present CacheBlend, a scheme that reuses the precomputed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime, the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job, allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art KV cache reusing schemes on three open-source LLMs of various sizes and four popular benchmark datasets of different tasks, we show that CacheBlend reduces time-to-first-token (TTFT) by 2.2-3.3\texttimes and increases the inference throughput by 2.8-5\texttimes from full KV recompute without compromising generation quality. The code is available at https://github.com/LMCache/LMCache.
@inproceedings{10.1145/3689031.3696098, author = {Yao, Jiayi and Li, Hanchen and Liu, Yuhan and Ray, Siddhant and Cheng, Yihua and Zhang, Qizheng and Du, Kuntai and Lu, Shan and Jiang, Junchen}, title = {{C}ache{B}lend: {F}ast {L}arge {L}anguage {M}odel {S}erving for {RAG} with {C}ached {K}nowledge {F}usion}, year = {2025}, isbn = {9798400711961}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3689031.3696098}, doi = {10.1145/3689031.3696098}, booktitle = {Proceedings of the Twentieth European Conference on Computer Systems}, pages = {94–109}, numpages = {16}, keywords = {KV Cache, Large Language Models, Retrieval-Augmented-Generation}, location = {Rotterdam, Netherlands}, series = {EuroSys '25}, award_name = {Best Paper}, scholar_query = {CacheBlend Fast Large Language Model Serving for RAG with Cached Knowledge Fusion} }
SIGCOMM
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang

In Proceedings of the ACM SIGCOMM 2024 Conference 2024

Abs arXiv Bib Talk Cited by -

s large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache’s distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. % When available bandwidth drops, CacheGen may raise the compression level for a part of the context or recompute its KV cache on the fly. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x with negligible impact on the LLM response quality. Our code is at: this https URL.
@inproceedings{10.1145/3651890.3672274, author = {Liu, Yuhan and Li, Hanchen and Cheng, Yihua and Ray, Siddhant and Huang, Yuyang and Zhang, Qizheng and Du, Kuntai and Yao, Jiayi and Lu, Shan and Ananthanarayanan, Ganesh and Maire, Michael and Hoffmann, Henry and Holtzman, Ari and Jiang, Junchen}, title = {Cache{G}en: KV {C}ache {C}ompression and {S}treaming for {F}ast {L}arge {L}anguage {M}odel {S}erving}, year = {2024}, isbn = {9798400706141}, publisher = {ACM}, url = {https://doi.org/10.1145/3651890.3672274}, doi = {10.1145/3651890.3672274}, booktitle = {Proceedings of the ACM SIGCOMM 2024 Conference}, keywords = {large language models, KV cache, compression}, location = {Sydney, NSW, Australia}, talk = {https://www.youtube.com/watch?v=H4_OUWvdiNo&list=PLU4C2_kotFP3N52c6xy4kOrapJaY50Iqt&index=4&t=90s}, scholar_query = {CacheGen KV Cache Compression and Streaming for Fast Large Language Model Serving} }