mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
e2691a7031
* feat(gateway): hooks and observers around inference requests, with usage and cost Adds the litellm-gateway-hooks crate: request facts built only when something reads them, on-path hooks that declare what they read and what they may change and run in config order, off-path observers on bounded queues, a usage tap for SSE and JSON responses, and per-deployment prices from the model cost map overlaid by config prices. The messages and chat handlers run through it when anything is installed; with nothing installed requests take the old path. The key store now also returns the key alias, user, team alias and key and team metadata, and router attempts carry the deployment id. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * fix(model-catalog): read token counts the cost map writes as whole floats The published cost map now has prompt_cache_min_tokens: 4096.0, which the typed u64 fields rejected, so the gateway dropped the whole map at startup and priced every request at zero. Counts accept any whole non-negative number, as Python reads any JSON number, and a test keeps the checked-in map parsing. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| adrs | ||
| customer-proxy | ||