LLM Examples

Basics

• Generate text
• Generate text asynchronously
• Generate text in streaming
• Distributed LLM Generation

Customization

• Generate text with guided decoding
• Control generated text using logits processor
• Generate text with multiple LoRA adapters
• Sparse Attention
• Speculative Decoding
• KV Cache Compression
• KV Cache Connector
• KV Cache Connector under VSWA
• KV Cache Offloading
• Runtime Configuration Examples
• Sampling Techniques Showcase
• LMCache KV Cache Connector

Slurm

• Run LLM-API with pytorch backend on Slurm
• Run trtllm-bench with pytorch backend on Slurm
• Run trtllm-serve with pytorch backend on Slurm