Siddhant Ray

Selected publications

1. SIGCOMM

   CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

   Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang

   In Proceedings of the ACM SIGCOMM 2024 Conference 2024

   Abs arXiv Bib

   s large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache’s distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. % When available bandwidth drops, CacheGen may raise the compression level for a part of the context or recompute its KV cache on the fly. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x with negligible impact on the LLM response quality. Our code is at: this https URL.

   @inproceedings{10.1145/3651890.3672274,
     author = {Liu, Yuhan and Li, Hanchen and Cheng, Yihua and Ray, Siddhant and Huang, Yuyang and Zhang, Qizheng and Du, Kuntai and Yao, Jiayi and Lu, Shan and Ananthanarayanan, Ganesh and Maire, Michael and Hoffmann, Henry and Holtzman, Ari and Jiang, Junchen},
     title = {Cache{G}en: KV {C}ache {C}ompression and {S}treaming for {F}ast {L}arge {L}anguage {M}odel {S}erving},
     year = {2024},
     isbn = {9798400706141},
     publisher = {ACM},
     url = {https://doi.org/10.1145/3651890.3672274},
     doi = {10.1145/3651890.3672274},
     booktitle = {Proceedings of the ACM SIGCOMM 2024 Conference},
     keywords = {large language models, KV cache, compression},
     location = {Sydney, NSW, Australia},
   }

2. HOTNETS

   A New Hope for Network Model Generalization

   Alexander Dietmüller, Siddhant Ray, Romain Jacob, and Laurent Vanbever

   In Proceedings of the 21st ACM Workshop on Hot Topics in Networks 2022

   Abs arXiv Bib Code

   Generalizing machine learning (ML) models for network traffic dynamics tends to be considered a lost cause. Hence, for every new task, we often resolve to design new models and train them on model-specific datasets collected, whenever possible, in an environment mimicking the model’s deployment. This approach essentially gives up on generalization. Yet, an ML architecture called_Transformer_ has enabled previously unimaginable generalization in other domains. Nowadays, one can download a model pre-trained on massive datasets and only fine-tune it for a specific task and context with comparatively little time and data. These fine-tuned models are now state-of-the-art for many benchmarks. We believe this progress could translate to networking and propose a Network Traffic Transformer (NTT), a transformer adapted to learn network dynamics from packet traces. Our initial results are promising: NTT seems able to generalize to new prediction tasks and contexts. This study suggests there is still hope for generalization, though it calls for a lot of future research.

   @inproceedings{dietmuller2022new,
     author = {Dietm\"{u}ller, Alexander and Ray, Siddhant and Jacob, Romain and Vanbever, Laurent},
     title = {A New Hope for Network Model Generalization},
     year = {2022},
     isbn = {9781450398992},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3563766.3564104},
     doi = {10.1145/3563766.3564104},
     booktitle = {Proceedings of the 21st ACM Workshop on Hot Topics in Networks},
     pages = {152–159},
     numpages = {8},
     keywords = {packet-level modeling, transformer},
     location = {Austin, Texas},
     series = {HotNets '22},
   }