litellm

mirror of https://github.com/BerriAI/litellm.git synced 2026-10-07 02:59:05 +00:00

History

Mateo Wang ff7dc86947 Some checks failed CI Coverage / assert-ci-coverage (push) Waiting to run Details CodeQL / Analyze (python) (push) Waiting to run Details CodeQL / Analyze (actions) (push) Waiting to run Details CodeQL / Analyze (javascript-typescript) (push) Waiting to run Details CodSpeed Benchmarks / benchmarks (push) Waiting to run Details Helm unit test / unit-test (push) Waiting to run Details Publish basedpyright base counts / publish (push) Waiting to run Details Scorecard supply-chain security / Scorecard analysis (push) Waiting to run Details Code Quality Checks / code-quality (push) Waiting to run Details Code Quality Checks / python-310-import-smoke (push) Waiting to run Details UI Unit Tests / ui-unit-tests (push) Waiting to run Details Postgres Tests / proxy-security (push) Waiting to run Details Postgres Tests / schema-migration (push) Waiting to run Details Postgres Tests / proxy-behavior (push) Waiting to run Details LiteLLM Rust / rust-lint (push) Waiting to run Details LiteLLM Rust / rust-test (push) Waiting to run Details LiteLLM Rust / rust-wheel (push) Waiting to run Details Unit Tests: Documentation Validation / documentation (push) Waiting to run Details Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run Details Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions Details Unit Tests / misc (push) Waiting to run Details Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions Details Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions Details Unit Tests / caching-local (push) Waiting to run Details Unit Tests / core-utils (push) Waiting to run Details Unit Tests / enterprise-package (push) Waiting to run Details Unit Tests / enterprise-routing (push) Waiting to run Details Unit Tests / integrations (push) Waiting to run Details Unit Tests / All Other Providers (push) Waiting to run Details Unit Tests / Vertex AI (push) Waiting to run Details Unit Tests / proxy-auth (push) Waiting to run Details Unit Tests / proxy-endpoints (push) Waiting to run Details Unit Tests / proxy-extras (push) Waiting to run Details Unit Tests / proxy-infra (push) Waiting to run Details Unit Tests / proxy-server (push) Waiting to run Details Unit Tests / responses-caching-types (push) Waiting to run Details GitHub Actions Security Analysis / zizmor (push) Waiting to run Details Issue label sync / sync-issue-labels-tests (push) Has been cancelled Details Issue label sync / sync-issue-labels (push) Has been cancelled Details Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled Details Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled Details VS Code Extension / vscode-extension (push) Has been cancelled Details Merge pull request #41892 from BerriAI/litellm_gemini_contentless_candidate_finish_reason fix(gemini): preserve candidates with finishReason and no content (#40477)		2026-09-18 16:34:46 -07:00
..
audio_utils	fix(audio_utils): validate MPEG frame headers before labeling sniffed audio	2026-08-29 14:20:40 -07:00
llm_cost_calc	Merge pull request #39424 from emerzon/litellm_azure_ai_flux_2_flex	2026-09-18 14:30:25 -07:00
llm_response_utils	fix(convert_dict_to_response): name the non-list choices type in the converter error	2026-09-09 15:12:13 -07:00
prompt_templates	Merge pull request #41275 from BerriAI/litellm_azure_strip_file_format	2026-09-18 11:46:23 -07:00
specialty_caches	feat(newrelic): per-team cost and usage metrics via team callbacks (#37610 )	2026-08-26 23:42:02 -07:00
tokenizers	Code Quality Improvement - remove `tokenizers/` from /llms (#7163 )	2024-12-10 23:50:15 -08:00
agentic_loop_settings.py	fix: keep accepting a loop ceiling that spells a whole number	2026-08-22 11:24:22 -07:00
api_route_to_call_types.py	fix(guardrails): resolve generateContent routes and async-first passthrough call types	2026-08-29 21:35:19 -07:00
app_crypto.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
asyncify.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
aws_partition.py	fix(aws): build every AWS endpoint and ARN from the region partition	2026-08-29 01:21:59 -07:00
cached_imports.py	refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets	2026-08-01 15:43:29 -07:00
chat_completion_agentic_loop.py	refactor(agentic-loop): narrow logging_obj before building the fake stream wrapper	2026-08-27 23:19:17 +00:00
classifier_logging.py	fix(router): merge staging and retain native classifier audits	2026-09-10 14:24:56 -07:00
cli_keyring.py	fix(cli): stop asking a keychain that already stopped answering	2026-08-20 04:12:23 -07:00
cli_token_utils.py	fix(cli): keep the refresh token in the OS keychain, not in token.json	2026-08-20 11:43:53 -07:00
cloud_storage_security.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
completion_timeout.py	chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom	2026-08-05 02:37:24 -07:00
core_helpers.py	fix(gemini): map every documented finishReason and reset per-candidate state	2026-09-18 14:54:18 -07:00
coroutine_checker.py	fix lint review feedback	2026-09-14 15:09:53 +08:00
credential_accessor.py	fix(proxy): load db credentials in the model reconcile so a worker never serves a model before its credential (#39876 )	2026-09-08 10:08:24 -07:00
custom_logger_registry.py	feat(pointfive): register the pointfive callback	2026-09-10 14:02:35 +03:00
dd_tracing.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
default_encoding.py	feat(rust): count tiktoken cl100k_base admission tokens in Rust (#40777 )	2026-09-11 23:47:05 +00:00
dot_notation_indexing.py	refactor(typing): replace Any with proven types in 89 more backend files	2026-09-02 23:08:42 +00:00
duration_parser.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
env_utils.py	fix(token_counter): stop large token counts from blocking the proxy event loop (#37697 )	2026-08-20 16:09:07 -07:00
exception_mapping_utils.py	fix(exceptions): keep repeated litellm_proxy response headers on the rebuilt response	2026-09-14 22:59:40 -07:00
fallback_generalizations.py	refactor(model_info): drop types import from fallback_generalizations	2026-09-14 18:58:53 +00:00
fallback_utils.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
get_blog_posts.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
get_litellm_params.py	Merge pull request #41138 from BerriAI/litellm_bedrock_files_s3_endpoint_url	2026-09-18 15:04:01 -07:00
get_llm_provider_logic.py	fix(spend_tracking): honour the global litellm_proxy override when inferring a model group provider	2026-09-17 00:03:01 +00:00
get_model_cost_map.py	Merge pull request #40444 from BerriAI/litellm_annotate_strict_budget_helpers	2026-09-09 13:11:05 -07:00
get_provider_specific_headers.py	fix(proxy): stop forwarding a client Anthropic OAuth token to Bedrock and Vertex	2026-08-21 18:02:40 -07:00
get_supported_openai_params.py	fix(vertex_ai): keep reasoning_effort unsupported on Vertex AI Mistral partner models	2026-09-18 13:16:56 -07:00
health_check_helpers.py	fix(ocr): send each provider a health-check document it accepts	2026-09-04 22:52:12 -07:00
health_check_utils.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
initialize_dynamic_callback_params.py	feat(langfuse): support langfuse_environment as a per-key dynamic callback param (#38264 )	2026-08-26 16:56:55 -07:00
internal_call_metadata.py	fix(auto_router): bill the routing embedding to the caller's key and team (#39532 )	2026-09-03 13:53:30 -07:00
json_fragment_accumulator.py	perf(streaming): add shared JSONFragmentAccumulator for Vertex and Anthropic (#36610 )	2026-08-25 16:15:47 -07:00
json_validation_rule.py	refactor(typing): replace Any with proven types in 89 more backend files	2026-09-02 23:08:42 +00:00
litellm_logging.py	refactor(rust): route-neutral callback contract	2026-09-18 15:43:08 -07:00
llm_judge.py	fix(shadow_eval): refuse a judge model that also serves one of the arms it grades (#38589 )	2026-08-27 18:44:44 -07:00
llm_request_utils.py	fix(hosted_vllm): forward Omni media URLs instead of downloading them	2026-08-24 17:59:52 -04:00
logging_callback_manager.py	fix(proxy): surface runtime-registered callbacks in UI Logging page (#38974 )	2026-09-09 22:03:16 -07:00
logging_utils.py	fix(logging): scan each log record once and collapse base64 payloads before the secret regex	2026-09-12 19:17:26 -07:00
logging_worker.py	style(logging): drop explanatory comments per repo no-comments policy	2026-09-12 17:03:40 -07:00
mock_functions.py	refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets	2026-08-01 15:43:29 -07:00
model_param_helper.py	merge litellm_internal_staging into litellm_anthropic_messages_response_cache	2026-08-11 20:48:41 +00:00
model_response_utils.py	chore(techdebt): clear fresh debt from the 2026-08-29 window	2026-08-30 07:57:16 +00:00
private_json.py	feat(auto-router): show routed model and savings in Claude Code and Codex (#40330 )	2026-09-11 12:52:42 -07:00
ptu_pricing.py	feat(spend-logs): record Azure spillover source deployment in spend log metadata	2026-09-17 05:48:51 +00:00
README.md	Guardrails API - add streaming support (#17400 )	2025-12-02 22:52:09 -08:00
realtime_errors.py	fix(realtime): relay the upstream websocket close to the client instead of hanging	2026-09-04 19:25:12 -07:00
realtime_streaming.py	refactor(realtime): move session ownership marker keys into constants	2026-09-14 21:33:48 +00:00
reasoning_effort_utils.py	fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke (#31364 )	2026-06-27 11:35:36 -07:00
redact_messages.py	fix(logging): reconstruct classifier audit redaction payloads	2026-09-10 13:33:36 -07:00
request_timeout_resolver.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
response_header_helpers.py	fix(utils.py): guarantee openai-compatible headers always exist in response	2024-09-28 21:08:15 -07:00
rules.py	chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom	2026-08-05 02:37:24 -07:00
safe_json_dumps.py	fix(logging): contain every extra serializer failure and skip rescanning a stamped record	2026-09-16 17:22:12 -07:00
safe_json_loads.py	refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets	2026-08-01 15:43:29 -07:00
secret_redaction.py	fix(utils): redact credential kwargs from the set_verbose request line	2026-09-03 01:55:56 -07:00
sensitive_data_masker.py	fix(router): redact cookies and preserve legacy classifier logs	2026-09-10 13:12:04 -07:00
service_tier_utils.py	feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011)	2026-08-04 12:54:39 -07:00
streaming_chunk_builder_utils.py	refactor(streaming): compute the reasoning token cap once	2026-09-16 23:28:28 +00:00
streaming_handler.py	chore(streaming): remove retired ai21/maritalk/baseten/azure raw-bytes handlers and dead palm completion code	2026-09-17 20:07:51 +00:00
thread_pool_executor.py	fix(logging): bound the shared logging executor backlog (#37694 )	2026-08-20 16:04:43 -07:00
token_counter.py	perf: defer fastapi and tiktoken BPE imports out of import litellm	2026-09-17 08:59:48 +00:00
url_utils.py	fix(image_handling): answer every SSRF rejection with one message so error text cannot probe internal hostnames	2026-09-05 00:16:14 -07:00

README.md

Folder Contents

This folder contains general-purpose utilities that are used in multiple places in the codebase.

Core files:

streaming_handler.py: The core streaming logic + streaming related helper utils
core_helpers.py: code used in types/ - e.g. map_finish_reason.
exception_mapping_utils.py: utils for mapping exceptions to openai-compatible error types.
default_encoding.py: code for loading the default encoding (tiktoken)
get_llm_provider_logic.py: code for inferring the LLM provider from a given model name.
duration_parser.py: code for parsing durations - e.g. "1d", "1mo", "10s"
api_route_to_call_types.py: mapping of API routes to their corresponding CallTypes (e.g., /chat/completions -> [acompletion, completion])