scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-11 03:38:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 82
litellm/litellm/llms
History
devin-ai-integration[bot] 9d16412341
fix(caching): count tool_call cache_control marks in the injection census (#43556) 
* fix(caching): count tool_call cache_control marks in the injection census

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): remove cache census casts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): only skip injection on message or content marks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): skip injection on messages whose tool calls carry marks

Reverts 9f08d8aef8. A default 5m mark injected on assistant text lands
before the client's 1h tool_use mark, which Anthropic rejects with a 400
because a 1h breakpoint must not follow a 5m one. Keeping the full census
in the skip check leaves the client's tool_call breakpoint as the only
one on that message.

* fix(caching): count every client tool_call cache mark in the breakpoint census

The census gated tool_call marks on type function and dict shape, so a client mark on a call without a type or with a string cache_control slipped past the count and injection overflowed the 4 breakpoint cap. Count any non-None tool_call mark, keep the server tool exclusion, and add integration cells for the capped surfaces, the yaml stand-down, Bedrock and Gemini, and router affinity

* test(integration): hold the upstream so the worker kill lands mid-burst

* refactor(caching): reuse the transform's server tool lookup in the breakpoint census

The census now calls the same helper the Anthropic transform uses to decide
whether a marked tool call becomes a server tool block, so the two cannot
drift apart. The owned-proxy burst test waits up to 90 seconds for the burst
to reach the wire before it kills a worker

* refactor(anthropic): move the server tool rebuild check under llms/anthropic

* test(integration): audit the tool call mark census across chat, messages, responses, and chaos

Adds the /audit cells for the breakpoint census on assistant tool_calls marks: the
Responses stream bridge, the OpenAI and Anthropic SDK clients, in-process Pydantic
messages, response cache twins, request-level points, a provider 401 on a capped request,
malformed provider_specific_fields and tool_call ids, null or empty points, a points
update mid-burst, a proxy restart mid-burst, and a worker SIGKILL that picks the worker
holding the burst's upstream connections

* test(integration): close the SDK clients the cache census cells open

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-03 17:33:48 -07:00
..
a2a chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
ai21/chat
aiml fix(cost): honor deployment pricing for image generation (#39311) 2026-09-22 11:40:39 -07:00
aiohttp_openai/chat feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
amazon_nova feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
anthropic fix(caching): count tool_call cache_control marks in the injection census (#43556) 2026-10-03 17:33:48 -07:00
apiserpent
aws_polly refactor(typing): replace Any with proven types in 89 more backend files 2026-09-02 23:08:42 +00:00
azure chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
azure_ai feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
base_llm feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
baseten
bedrock fix(bedrock): honor the per-request timeout on Converse and Invoke streaming (internal copy of #38210) (#44134) 2026-10-03 19:29:15 +00:00
bedrock_mantle fix(health): probe Bedrock Mantle Claude deployments over the Anthropic Messages API (#44419) 2026-10-03 17:05:47 -07:00
black_forest_labs feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
brave/search feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
bytez fix(bedrock): honor the per-request timeout on Converse and Invoke streaming (internal copy of #38210) (#44134) 2026-10-03 19:29:15 +00:00
cerebras fix(cerebras): add max_retries and extra_headers to get_supported_openai_params (#36601) 2026-08-25 14:10:56 -07:00
chatgpt fix(chatgpt): preserve requested service tier in Responses calls (#40108) 2026-10-02 10:57:37 -07:00
clarifai/chat feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
claude_code refactor(types): replace Any with proven types in 9 files (#44389) 2026-10-03 21:03:16 +00:00
cloudflare feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236) 2026-10-03 17:38:38 +00:00
codestral/completion refactor(types): replace Any with proven types in 32 files 2026-09-21 10:51:46 +00:00
codex feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
cohere refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
cometapi fix(cost): honor deployment pricing for image generation (#39311) 2026-09-22 11:40:39 -07:00
compactifai feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
custom_httpx feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
dashscope chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
databricks fix(streaming): keep the served service_tier on streamed chunks and spend rows (#42870) 2026-09-29 12:54:17 -07:00
dataforseo/search refactor(typing): replace Any with proven types in 42 more backend files 2026-09-02 15:35:01 +00:00
datarobot/chat
deepagents feat(harness): add Harness.TOOL_LOOP, a minimal in-process tool-calling loop (#44391) 2026-10-03 17:33:01 +00:00
deepgram fix(deepgram): forward only the first model and language values to /listen 2026-09-18 21:29:29 +00:00
deepinfra refactor(types): replace Any with precise types across 73 modules 2026-09-01 11:05:02 +00:00
deepseek chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
deprecated_providers feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
docker_model_runner/chat
duckduckgo/search
e2b Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_decrease_anys_opus5 2026-08-29 06:03:33 -07:00
edenai chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
elevenlabs fix(params): keep _litellm_* kwargs out of provider request bodies by construction (#43221) 2026-09-26 15:20:09 -07:00
empower/chat
exa_ai/search fix(exa): fall back to highlights/summary when text is missing (#42213) 2026-10-02 14:15:15 -07:00
fal_ai chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
fastcrw
featherless_ai/chat
firecrawl
fireworks_ai chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
friendliai/chat
galadriel/chat
gdc refactor(types): replace Any with real types across 29 more backend files 2026-09-08 10:14:07 +00:00
gemini feat(interactions): durable cross-pod settlement for background interaction billing (#41955) 2026-10-02 21:08:38 -07:00
gigachat chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
github/chat
github_copilot refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
google_pse/search
gradient_ai/chat
groq chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
heroku/chat
hosted_vllm chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
huggingface feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
hyperbolic
inception
infinity Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_decrease_anys_opus5 2026-08-29 06:03:33 -07:00
jina_ai refactor(types): replace Any with proven types in 32 files 2026-09-21 10:51:46 +00:00
lambda_ai
langflow feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
langgraph fix(bedrock): honor the per-request timeout on Converse and Invoke streaming (internal copy of #38210) (#44134) 2026-10-03 19:29:15 +00:00
laya feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
lemonade feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
linkup
litellm_proxy chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
llamafile/chat
lm_studio
manus
meta/realtime chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
meta_llama/chat
milvus/vector_stores refactor(s3_vectors): embed search queries through the shared vector store executor 2026-09-02 19:26:31 -07:00
minimax refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
mistral chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
modelscope
mongodb chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
moonshot/chat fix(moonshot, together_ai): send the reasoning effort Kimi K3 accepts (#38611) 2026-08-27 20:46:46 -07:00
morph
nadir/chat chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
nebius
nimble chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
nlp_cloud feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
novita/chat
nscale/chat
nvidia_nim chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
nvidia_riva refactor(types): replace Any with real types across 11 more backend files 2026-09-08 14:09:22 +00:00
oci fix(bedrock): honor the per-request timeout on Converse and Invoke streaming (internal copy of #38210) (#44134) 2026-10-03 19:29:15 +00:00
ollama refactor(types): replace Any with proven types in 9 files (#44389) 2026-10-03 21:03:16 +00:00
oobabooga feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
openai feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
openai_like chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
opencode feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
openrouter feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236) 2026-10-03 17:38:38 +00:00
opensandbox Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_decrease_anys_opus5 2026-08-29 06:03:33 -07:00
ovhcloud
parallel_ai/search style(websearch): apply ruff format to changed files 2026-09-09 08:55:25 -04:00
pass_through refactor(typing): replace Any with proven types in 42 more backend files 2026-09-02 15:35:01 +00:00
perplexity feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236) 2026-10-03 17:38:38 +00:00
petals feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
pg_vector/vector_stores refactor(s3_vectors): embed search queries through the shared vector store executor 2026-09-02 19:26:31 -07:00
predibase feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
ragflow Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_decrease_anys_opus5_r4 2026-09-03 06:44:40 +00:00
recraft fix(cost): honor deployment pricing for image generation (#39311) 2026-09-22 11:40:39 -07:00
reducto refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
replicate feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
runwayml fix(fal_ai): reuse the status client and headers on the video result probe (#42511) 2026-09-22 13:12:51 -07:00
s3_vectors fix(s3_vectors): embed registered-store ingests with the store's embedding model 2026-09-19 04:31:15 -07:00
sagemaker fix(bedrock): honor the per-request timeout on Converse and Invoke streaming (internal copy of #38210) (#44134) 2026-10-03 19:29:15 +00:00
sail chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
sambanova
sap refactor(types): replace Any with proven types in 5 files (#42722) 2026-09-23 03:14:45 -07:00
scaleway refactor: clean up fresh tech debt from 2026-10-02 (#44362) 2026-10-03 02:12:03 -07:00
searchapi
searxng
serper/search
snowflake chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
soniox refactor(typing): replace Any with proven types in 89 more backend files 2026-09-02 23:08:42 +00:00
stability feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
strands_decider/decisions feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236) 2026-10-03 17:38:38 +00:00
tavily/search
tencent refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
tinyfish/search chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
together_ai chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
tool_loop feat(harness): add Harness.TOOL_LOOP, a minimal in-process tool-calling loop (#44391) 2026-10-03 17:33:01 +00:00
topaz feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
triton feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
typesafe/decisions feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236) 2026-10-03 17:38:38 +00:00
v0
valkey chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
vercel_ai_gateway refactor: daily fresh tech debt cleanup, rolling PR (#42710) 2026-09-24 04:17:23 -07:00
vertex_ai fix(vertex_ai): forward system and tools to partner model count_tokens (#43900) 2026-10-03 16:36:14 -07:00
vllm
volcengine refactor(types): replace Any with real types across 54 more backend files 2026-09-08 12:41:07 +00:00
voyage Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
wandb chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
watsonx feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
xai chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
xinference/image_generation
you_com
zai
__init__.py fix(gemini): bill Google Maps grounding as its own SKU 2026-08-26 15:31:27 -07:00
base.py refactor(types): replace Any with real types across 178 backend files 2026-08-27 10:14:58 +00:00
compaction.py feat(router): native compact-to-fit across conversation APIs (#42074) 2026-09-21 22:52:29 -07:00
custom_llm.py refactor(types): replace Any with real types across 178 backend files 2026-08-27 10:14:58 +00:00
maritalk.py
oss_decision.py feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
README.md

README.md

File Structure

August 27th, 2024

To make it easy to see how calls are transformed for each model/provider:

we are working on moving all supported litellm providers to a folder structure, where folder name is the supported litellm provider name.

Each folder will contain a *_transformation.py file, which has all the request/response transformation logic, making it easy to see how calls are modified.

E.g. cohere/, bedrock/.