scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-12 23:01:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 87
litellm/prompt_compression/README.md
Krrish Dholakia 65b8660c80 feat(prompt-compression): add headroom compression plugin 
Microservice that wraps headroom-ai as a LiteLLM pre_call guardrail.
Compresses tool outputs and JSON arrays before they reach the LLM
provider using headroom's smart_crusher (JSON dedup + schema compression)
and kompress (prose/log compression). Returns compressed structured_messages
via the generic guardrail API, which required adding structured_messages
support to GenericGuardrailAPIResponse.
2026-06-25 19:49:26 -07:00

1.3 KiB
Raw Permalink Blame History

Prompt Compression Plugins

Microservices that plug into the LiteLLM gateway as pre_call guardrails to compress incoming prompts before they reach the LLM provider. Each runs independently and speaks the LiteLLM generic guardrail API.

How it works

LiteLLM sends the full message array to the microservice before forwarding the request to the LLM provider. The microservice compresses it and returns the compressed messages. LiteLLM then sends the smaller payload to the provider — reducing cost and latency transparently to the caller.

Available plugins

Plugin Compression approach
headroom ML-based; deduplicates JSON arrays, compresses tool outputs and prose via a trained model

Adding a plugin to your LiteLLM config

guardrails:
  - guardrail_name: headroom-compression
    litellm_params:
      guardrail: generic_guardrail_api
      mode: pre_call
      api_base: http://localhost:8100
      # api_key: your-secret-key

The mode: pre_call ensures compression runs before the request reaches the provider. The plugin returns GUARDRAIL_INTERVENED with compressed structured_messages when savings are found, or NONE to pass through unchanged.