mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-28 06:23:07 +00:00
dfac770e2b
* feat(proxy): map baggage values to budget tags * fix(proxy): preserve mapped tags in the parsed request cache * fix(proxy): preserve mapper accounting and tag policy * fix(proxy): preserve baggage tags across native transports * fix(proxy): keep mapper rejection and passthrough logging safe * fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing Mapped tags now apply only to billed LLM API routes, skipping token counting, model listing, MCP and A2A. They classify spend and budgets but never select deployments unless a key, team or client set the same tag. WebSocket inference returns 403 only while a required rule exists. A tag created with /tag/new resets its spend counter so its budget counts from creation. The General settings editor keeps a new mapping unsaved until every required field is filled in. * fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews * fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403 * fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values * test(proxy): cover baggage tag mappings end to end Adds integration cells for request_tag_mappings across chat, messages, responses, A2A and custom pass-through spend, required and budgeted rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags, tag routing and policy isolation, a dropped tag read with and without allow_requests_on_db_unavailable, and the /tag/new spend counter reset * fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags. * fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering. * feat(proxy): parse baggage headers into request tag mappings * fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes * fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request * fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag. * test(spend): cover string metadata.tags under mapped baggage * fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings Mapped baggage tags are applied again after the header and key tag merges, so a header or key tag that replaces a string metadata.tags no longer hides the mapped tag from the tag budget check. An empty or false metadata.tags is read like a missing one, and a tags list with non-string items is kept as sent with the mapped tags added after it, instead of failing auth with a 401 |
||
|---|---|---|
| .. | ||
| _experimental | ||
| a2a | ||
| agent_endpoints | ||
| analytics_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_management_endpoints | ||
| config_resolvers | ||
| container_endpoints | ||
| credential_endpoints | ||
| custom_hooks | ||
| db | ||
| decisions_endpoints | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| example_config_yaml | ||
| fine_tuning_endpoints | ||
| google_endpoints | ||
| guardrails | ||
| health_check_utils | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| lens | ||
| list_api | ||
| logging_endpoints | ||
| management | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| ocr_endpoints | ||
| openai_evals_endpoints | ||
| openai_files_endpoints | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| rerank_endpoints | ||
| response_api_endpoints | ||
| response_polling | ||
| roi_calculator | ||
| search_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| swagger | ||
| telemetry | ||
| test_prompts | ||
| types_utils | ||
| ui_crud_endpoints | ||
| vector_store_endpoints | ||
| vector_store_files_endpoints | ||
| vertex_ai_endpoints | ||
| video_endpoints | ||
| workflows | ||
| .gitignore | ||
| __init__.py | ||
| _lazy_features.py | ||
| _lazy_openapi_snapshot.json | ||
| _lazy_openapi_snapshot.py | ||
| _new_new_secret_config.yaml | ||
| _new_secret_config.yaml | ||
| _super_secret_config.yaml | ||
| _types.py | ||
| admin_mcp.py | ||
| bug_report_config.py | ||
| cached_logo.jpg | ||
| caching_routes.py | ||
| collector.py | ||
| common_request_processing.py | ||
| compliance_checks.py | ||
| custom_auth_auto.py | ||
| custom_prompt_management.py | ||
| custom_sso.py | ||
| custom_validate.py | ||
| dd_span_tagger.py | ||
| dev_config.yaml | ||
| enterprise | ||
| health_check.py | ||
| lambda.py | ||
| litellm_pre_call_utils.py | ||
| llamaguard_prompt.txt | ||
| logo.png | ||
| logo_dark.png | ||
| logo_monogram.png | ||
| logo_monogram_dark.png | ||
| mcp_registry.json | ||
| mcp_tools.py | ||
| model_config.yaml | ||
| model_insights_tasks.json | ||
| moyai_endpoints.py | ||
| native_compaction.py | ||
| openapi.json | ||
| openapi_registry.json | ||
| plugin_routes.py | ||
| post_call_rules.py | ||
| prisma_migration.py | ||
| prometheus_cleanup.py | ||
| prometheus_metrics_server.py | ||
| proxy_cli.py | ||
| proxy_config.yaml | ||
| proxy_server.py | ||
| read_model_list.py | ||
| README.md | ||
| route_llm_request.py | ||
| route_priority.py | ||
| schema.prisma | ||
| start.sh | ||
| tracing_endpoints.py | ||
| tracing_runtime.py | ||
| utils.py | ||
| wildcard_config.yaml | ||
litellm-proxy
A local, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.
usage
$ uv tool install litellm
$ litellm --model ollama/codellama
#INFO: Ollama running on http://0.0.0.0:8000
replace openai base
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:8000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
])
print(response)
See how to call Huggingface,Bedrock,TogetherAI,Anthropic, etc.
Folder Structure
Routes
proxy_server.py- all openai-compatible routes -/v1/chat/completion,/v1/embedding+ model info routes -/v1/models,/v1/model/info,/v1/model_group_inforoutes.health_endpoints/-/health,/health/liveliness,/health/readinessmanagement_endpoints/key_management_endpoints.py- all/key/*routesmanagement_endpoints/team_endpoints.py- all/team/*routesmanagement_endpoints/internal_user_endpoints.py- all/user/*routesmanagement_endpoints/ui_sso.py- all/sso/*routes