Siddhant Ray

I am a second year PhD student in Computer Science at the University of Chicago, advised by Junchen Jiang and Nick Feamster. I am interested in machine learning methods for performance improvement in computer networks and efficient serving systems for Large Language Models, with a focus on Retrieval-Augmented-Generation (RAG) systems.

Currently I work on joint optimizations in RAG for quality and delay with query level configuration selection and resource scheduling . I also work on using Transformer models for per-packet latency prediction to improve queue selection and reduce tail-latency for latency sensitive applications.

In the past, I have worked on advances in Software Defined Networking, programmable networks and cloud computing. Additionally I have spent some time working on developing NLP techniques to analyse political corpora.

I'm fortunate to be additionally supported by the Liew Family Graduate Fellowship. Prior to starting my PhD, I earned my MSc in Electrical Engineering and Information Technology at ETH Zurich and my B.Tech in Electronics and Communication Engineering at VIT Vellore.

News

Dec, 2024	RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation on ArXiv.
Oct, 2024	SwiftQueue: Optimizing Low-Latency Applications with Swift Packet Queuing on ArXiv.
Aug, 2024	CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving was presented at ACM SIGCOMM’24 .
Sep, 2023	Joined the University of Chicago as a PhD student in Computer Science.

Selected publications

SIGCOMM
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang

In Proceedings of the ACM SIGCOMM 2024 Conference 2024

Abs arXiv Bib

s large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache’s distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. % When available bandwidth drops, CacheGen may raise the compression level for a part of the context or recompute its KV cache on the fly. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x with negligible impact on the LLM response quality. Our code is at: this https URL.
@inproceedings{10.1145/3651890.3672274, author = {Liu, Yuhan and Li, Hanchen and Cheng, Yihua and Ray, Siddhant and Huang, Yuyang and Zhang, Qizheng and Du, Kuntai and Yao, Jiayi and Lu, Shan and Ananthanarayanan, Ganesh and Maire, Michael and Hoffmann, Henry and Holtzman, Ari and Jiang, Junchen}, title = {Cache{G}en: KV {C}ache {C}ompression and {S}treaming for {F}ast {L}arge {L}anguage {M}odel {S}erving}, year = {2024}, isbn = {9798400706141}, publisher = {ACM}, url = {https://doi.org/10.1145/3651890.3672274}, doi = {10.1145/3651890.3672274}, booktitle = {Proceedings of the ACM SIGCOMM 2024 Conference}, keywords = {large language models, KV cache, compression}, location = {Sydney, NSW, Australia}, }
HOTNETS
A New Hope for Network Model Generalization

Alexander Dietmüller, Siddhant Ray, Romain Jacob, and Laurent Vanbever

In Proceedings of the 21st ACM Workshop on Hot Topics in Networks 2022

Abs arXiv Bib Code

Generalizing machine learning (ML) models for network traffic dynamics tends to be considered a lost cause. Hence, for every new task, we often resolve to design new models and train them on model-specific datasets collected, whenever possible, in an environment mimicking the model’s deployment. This approach essentially gives up on generalization. Yet, an ML architecture called_Transformer_ has enabled previously unimaginable generalization in other domains. Nowadays, one can download a model pre-trained on massive datasets and only fine-tune it for a specific task and context with comparatively little time and data. These fine-tuned models are now state-of-the-art for many benchmarks. We believe this progress could translate to networking and propose a Network Traffic Transformer (NTT), a transformer adapted to learn network dynamics from packet traces. Our initial results are promising: NTT seems able to generalize to new prediction tasks and contexts. This study suggests there is still hope for generalization, though it calls for a lot of future research.
@inproceedings{dietmuller2022new, author = {Dietm\"{u}ller, Alexander and Ray, Siddhant and Jacob, Romain and Vanbever, Laurent}, title = {A New Hope for Network Model Generalization}, year = {2022}, isbn = {9781450398992}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3563766.3564104}, doi = {10.1145/3563766.3564104}, booktitle = {Proceedings of the 21st ACM Workshop on Hot Topics in Networks}, pages = {152–159}, numpages = {8}, keywords = {packet-level modeling, transformer}, location = {Austin, Texas}, series = {HotNets '22}, }