scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-09 08:18:12 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 67
litellm/litellm/litellm_core_utils
History
Kerry Lu 7acdf9f00d
fix(anthropic): count search results and reject unsupported token-count blocks (#48699) 
* fix(anthropic): count search results and reject unsupported token-count blocks

Co-authored-by: Prasanna Jain <prassu017@gmail.com>

* fix(anthropic): avoid a Final assignment inside the token-count loop

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Prasanna Jain <prassu017@gmail.com>
2026-10-28 02:33:38 +00:00
..
audio_utils perf(types): defer pydantic schema builds via shared LiteLLMBaseModel (#44720) 2026-10-05 22:10:08 -07:00
llm_cost_calc feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278) 2026-10-27 12:10:45 -07:00
llm_response_utils fix(logging): spilled over PTU requests without base_model log the served model's rates (#47655) 2026-10-20 22:28:19 +00:00
prompt_templates fix: preserve media across Responses and Gemini conversions (#48484) 2026-10-27 18:07:10 -07:00
specialty_caches refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
tokenizers fix(token_counter): count DeepSeek V4 text with DeepSeek's own tokenizer (#47353) 2026-10-20 03:43:35 +00:00
agentic_followup_kwargs.py chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
agentic_loop_settings.py fix: keep accepting a loop ceiling that spells a whole number 2026-08-22 11:24:22 -07:00
api_route_to_call_types.py fix(guardrails): resolve generateContent routes and async-first passthrough call types 2026-08-29 21:35:19 -07:00
app_crypto.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
asyncify.py fix(chatgpt,github_copilot): refuse device-code login inside an event loop or worker thread (#39585) 2026-10-06 03:20:07 +00:00
aws_partition.py fix(aws): build every AWS endpoint and ARN from the region partition 2026-08-29 01:21:59 -07:00
bug_report.py feat(proxy): admin-only /debug/report sharing the bug report environment (#42440) 2026-09-22 12:05:23 -07:00
cached_imports.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
chat_completion_agentic_loop.py fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream (#45433) 2026-10-09 22:16:02 -07:00
classifier_logging.py fix(router): merge staging and retain native classifier audits 2026-09-10 14:24:56 -07:00
cli_keyring.py fix(cli): stop asking a keychain that already stopped answering 2026-08-20 04:12:23 -07:00
cli_token_utils.py perf(cli): start lite without initializing the litellm SDK (#46803) 2026-10-17 04:53:32 +00:00
cloud_storage_security.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
cold_storage_handler.py refactor(responses): move ColdStorageHandler out of proxy into litellm_core_utils (#45783) 2026-10-10 19:08:04 +00:00
completion_timeout.py fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162) 2026-10-17 22:08:02 +00:00
core_helpers.py fix(opencode): bill OpenCode Go requests from the reported list_cost (#48254) 2026-10-24 16:44:29 -07:00
coroutine_checker.py Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
cost_discount.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
credential_accessor.py fix(secret_managers)!: resolve oidc/file and oidc/env refs in admin-configured api_key per request (#47341) 2026-10-20 01:43:59 +00:00
custom_logger_registry.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
dd_tracing.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
decisions_pricing.py feat(decisions): bill decisions and systemone from _decisions cost map fields (#45904) 2026-10-12 14:31:12 -07:00
default_encoding.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
deprecated_package.py fix(mcp): load deprecated package aliases on demand (#48615) 2026-10-27 19:46:19 +00:00
dot_notation_indexing.py refactor(typing): widen delete_nested_value to Mapping and drop two unused Span aliases (#48001) 2026-10-24 19:39:27 +00:00
duration_parser.py fix(proxy)!: gate key budget_duration and budget_id updates like max_budget (#47681) 2026-10-26 16:45:35 -07:00
env_utils.py fix(config): treat blank env values as unset so the proxy neither crash-loops nor silently drops a guardrail (#46341) 2026-10-14 12:49:31 -07:00
error_normalization.py feat(auth): deny search tools by default when search_tool_deny_by_default is set (#44490) 2026-10-06 12:50:26 -07:00
exception_mapping_utils.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
fallback_generalizations.py refactor(model_info): drop types import from fallback_generalizations 2026-09-14 18:58:53 +00:00
fallback_utils.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
get_blog_posts.py feat(proxy): add LITELLM_OFFLINE_MODE, one switch for the fetches LiteLLM makes on its own (#48096) 2026-10-23 16:54:19 -07:00
get_litellm_params.py feat(chatgpt): forward client subscriptions through the gateway (#41522) 2026-10-26 15:44:05 -07:00
get_llm_provider_logic.py feat(chatgpt): forward client subscriptions through the gateway (#41522) 2026-10-26 15:44:05 -07:00
get_model_cost_map.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
get_provider_specific_headers.py feat(chatgpt): forward client subscriptions through the gateway (#41522) 2026-10-26 15:44:05 -07:00
get_supported_openai_params.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
health_check_helpers.py feat(extraction): add /v1/extraction with Cloudflare Clairic support (#47473) 2026-10-20 15:51:45 -07:00
health_check_utils.py feat(extraction): add /v1/extraction with Cloudflare Clairic support (#47473) 2026-10-20 15:51:45 -07:00
hidden_params.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
initialize_dynamic_callback_params.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
internal_call_metadata.py fix(responses): write one spend-log row per native WebSocket turn (#46768) 2026-10-15 19:05:22 -07:00
json_fragment_accumulator.py chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
json_validation_rule.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
litellm_logging.py fix(security)!: build proxy-only failure logs from an allowlist and drop caller guardrail info (#48599) 2026-10-27 12:40:59 -07:00
llm_judge.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
llm_request_utils.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
logging_callback_manager.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
logging_utils.py refactor(types): replace Any with proven types in 24 files (#46663) 2026-10-15 06:10:12 -07:00
logging_worker.py feat(realtime): proxy OpenAI Live sessions on /v1/live/sessions (#43621) 2026-10-13 13:59:27 -07:00
mock_functions.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
model_param_helper.py feat(gemini): allow disabling thought summaries while keeping thinking on (#36460) 2026-10-16 09:01:33 +00:00
model_response_utils.py chore(techdebt): clear fresh debt from the 2026-08-29 window 2026-08-30 07:57:16 +00:00
oauth_token_exchange.py feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange (#45158) 2026-10-09 15:19:07 -07:00
offline_mode.py feat(proxy): add LITELLM_OFFLINE_MODE, one switch for the fetches LiteLLM makes on its own (#48096) 2026-10-23 16:54:19 -07:00
optional_imports.py refactor(sdk): separate core AWS and tokenizer dependencies (#44447) 2026-10-08 19:07:57 -07:00
path_utils.py refactor(core): move path_utils from proxy common_utils to litellm_core_utils (#45775) 2026-10-10 19:07:22 +00:00
private_json.py feat(auto-router): show routed model and savings in Claude Code and Codex (#40330) 2026-09-11 12:52:42 -07:00
provider_affinity.py fix(opencode): forward stable sessions automatically across inference routes (#48121) 2026-10-23 17:38:47 -07:00
ptu_pricing.py feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043) 2026-10-15 18:44:30 +00:00
README.md feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
realtime_errors.py fix(realtime): surface an upstream handshake refusal as an error event and policy close (#42388) 2026-09-22 11:31:15 -07:00
realtime_streaming.py fix(bedrock)!: block realtime replay after an input-first start and reject malformed Nova Sonic session settings (#46936) 2026-10-17 00:29:50 +00:00
reasoning_effort_utils.py feat(router)!: apply each deployment's reasoning_effort_levels on every provider and route (#46601) 2026-10-20 04:34:55 +00:00
redact_messages.py feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278) 2026-10-27 12:10:45 -07:00
renamed_package_shim.py refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510) 2026-10-27 10:08:25 -07:00
request_timeout_resolver.py feat(router): let a deployment set a timeout per service_tier (#47406) 2026-10-19 14:27:29 -07:00
resource_ownership.py refactor(core_utils): move resource_ownership out of the proxy (#45805) 2026-10-10 21:02:53 +00:00
response_header_helpers.py fix(utils.py): guarantee openai-compatible headers always exist in response 2024-09-28 21:08:15 -07:00
rules.py chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom 2026-08-05 02:37:24 -07:00
safe_json_dumps.py refactor(types): replace Any with proven types in 32 files 2026-09-21 10:51:46 +00:00
safe_json_loads.py refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574) 2026-10-15 21:08:47 +00:00
secret_redaction.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
sensitive_data_masker.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00
sentry_scrubbing.py fix(proxy): cooldown errors carry 429, Sentry starts at boot without request bodies, tracing failures log at ERROR, exporter counters exported (#46344) 2026-10-16 00:53:52 +00:00
served_output_texts.py refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510) 2026-10-27 10:08:25 -07:00
service_tier_utils.py feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
streaming_chunk_builder_utils.py feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278) 2026-10-27 12:10:45 -07:00
streaming_handler.py fix(streaming): dict-usage arm drops prompt_tokens_details/completion_tokens_details its sibling arms preserve (#36678) 2026-10-27 15:21:11 -07:00
thread_pool_executor.py fix(logging): bound the shared logging executor backlog (#37694) 2026-08-20 16:04:43 -07:00
token_counter.py fix(anthropic): count search results and reject unsupported token-count blocks (#48699) 2026-10-28 02:33:38 +00:00
tokenizer.py refactor(sdk): separate core AWS and tokenizer dependencies (#44447) 2026-10-08 19:07:57 -07:00
tool_fields.py fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583) 2026-10-16 14:57:53 -07:00
url_utils.py refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) 2026-10-24 11:31:09 -07:00

README.md

Folder Contents

This folder contains general-purpose utilities that are used in multiple places in the codebase.

Core files:

  • streaming_handler.py: The core streaming logic + streaming related helper utils
  • core_helpers.py: code used in types/ - e.g. map_finish_reason.
  • exception_mapping_utils.py: utils for mapping exceptions to openai-compatible error types.
  • default_encoding.py: code for loading the default Python tokenizer and bundled cache
  • get_llm_provider_logic.py: code for inferring the LLM provider from a given model name.
  • duration_parser.py: code for parsing durations - e.g. "1d", "1mo", "10s"
  • api_route_to_call_types.py: mapping of API routes to their corresponding CallTypes (e.g., /chat/completions -> [acompletion, completion])

Tokenizer factories return Python tokenizer objects by default. Set LITELLM_RUST=1 or call litellm.rust(True) before constructing tokenizers to select the Rust backend through Route.TOKENIZER in the Rust catalog. Missing native bindings or unsupported native features fall back to Python. Existing tokenizer objects keep their selected backend. Rust-backed tokenizer objects carry the read-only tiktoken.Encoding / tokenizers.Tokenizer surface and are immutable: enable_padding, enable_truncation and add_tokens stay on the Python tokenizer.