diff --git a/docs/my-website/docs/completion/prompt_compression.md b/docs/my-website/docs/completion/prompt_compression.md index 0ea679fb6b7..2d999291af6 100644 --- a/docs/my-website/docs/completion/prompt_compression.md +++ b/docs/my-website/docs/completion/prompt_compression.md @@ -46,7 +46,7 @@ response = litellm.completion( - `messages` (`List[dict]`, required): input conversation messages - `model` (`str`, required): model name used for token counting - `compression_trigger` (`int`, default `200000`): compress only if input token count exceeds this -- `compression_target` (`Optional[int]`, default `compression_trigger // 2`): desired post-compression token budget +- `compression_target` (`Optional[int]`, default `70% of compression_trigger`): desired post-compression token budget - `embedding_model` (`Optional[str]`): if set, combines BM25 + embedding relevance scoring - `embedding_model_params` (`Optional[dict]`): additional kwargs passed to `litellm.embedding()` - `compression_cache` (`Optional[DualCache]`): optional cache used by embedding scoring @@ -70,9 +70,53 @@ args = json.loads(tool_call.function.arguments) full_content = compressed["cache"][args["key"]] ``` -## Evaluate Compression Quality +## Performance -You can benchmark baseline vs compressed behavior with: +Benchmarked on [SWE-bench Lite](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite_bm25_27K) (real GitHub issues with ~27k tokens of BM25-retrieved repo context per problem). + +### Claude Opus — 5 problems, trigger=10k + +| Metric | Baseline | Compressed | Delta | +|---|---|---|---| +| File overlap | 1.000 | 1.000 | +0.000 | +| Exact file match | 100% | 100% | +0.0% | +| Hunk overlap | 0.582 | 0.361 | -0.221 | +| Content similarity | 0.367 | 0.373 | +0.006 | +| Avg prompt tokens | 30,828 | 6,890 | -77.7% | +| Avg cost/problem | $0.488 | $0.136 | **-72.0%** | + +**Key takeaways:** + +- **File-level targeting is fully preserved** — the model edits the same files with or without compression. +- **Content similarity matches baseline** — the actual lines changed are comparable. +- **Hunk overlap drops modestly** (-0.221) — the model targets the right files but may edit slightly different line ranges with less surrounding context. +- **72% cost savings** with 78% token reduction. + +### Metrics explained + +| Metric | What it measures | +|---|---| +| **File overlap** | Fraction of gold-patch files present in the generated patch | +| **Exact file match** | Whether the generated patch touches exactly the same set of files | +| **Hunk overlap** | Fraction of gold hunk line ranges covered by generated hunks | +| **Content similarity** | Jaccard similarity of changed lines (added/removed) between gold and generated patches | + +### Running the SWE-bench eval + +```bash +# 5-problem quick check +python tests/eval_swe_bench.py --model claude-opus-4-20250514 --problems 5 + +# Custom trigger/target +python tests/eval_swe_bench.py --model gpt-4o --problems 20 \ + --compression-trigger 15000 --compression-target 10000 + +# With embedding scoring +python tests/eval_swe_bench.py --model gpt-4o --problems 10 \ + --embedding-model text-embedding-3-small +``` + +### Running the HumanEval-style eval ```bash python scripts/eval_compression.py --model gpt-4o --problems 5