mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-12 08:42:18 +00:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yujonglee
|
11af2eb22d
|
refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822)
* refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
yujonglee
|
523b3efea2
|
refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501)
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
JeongCheolLee
|
febbe38326
|
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls Anthropic server-executed tools (web search, tool search, code execution) were returned in chat completions tool_calls with srvtoolu_ ids, so OpenAI-protocol agent clients treated them as pending work and re-invoked the model until their recursion limit. Move them to provider_specific_fields.server_tool_calls on both the non-streaming and streaming paths. History replay, the /v1/responses bridge and the /v1/messages streaming guardrail read that field, so multi-turn reconstruction, web_search_call output items and the guardrail's fail-closed check on server tool blocks keep working. * fix(responses): return Claude web_fetch calls as web_search_call open_page items * fix(responses): keep Claude server web calls without a result as hosted items A non-streamed Claude turn that calls web_search or web_fetch next to a client tool can return the server call before its result. Build an in_progress web_search_call for it, so the Responses bridge no longer returns it as a client function_call * fix(anthropic): keep pending server calls native and hide server calls from client tool paths Replaying a server call that has no result yet now sends it back as server_tool_use, so Anthropic accepts the next turn. The tool permission guardrail checks server calls again and rewrite mode removes a denied one. The /v1/messages adapter keeps Claude's server blocks, and the Responses bridge never returns a server call as a client function_call. * fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls, so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own server_tool_use instead of running again * fix(anthropic): let guardrails and loggers read Claude's server tool calls again Guardrails, tool policies, tracing loggers and the Responses guardrail translation read only client tool calls after server calls moved to provider_specific_fields. They now read server calls first, then client calls, through one shared helper, and the Responses guardrail treats Claude's web_search_call items as web_search or web_fetch calls. Non-streaming splits server calls by block type, as streaming does. * fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy, streamed or not. Redaction with message logging off hides server call arguments and hosted web call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls. * fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use. * fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy. * test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions * test(guardrails): type the fake Pillar client like its Protocol * fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or web_search still needs that call and its thinking replayed, or Claude loses the fetch result * fix(guardrails): scan replayed Claude server calls and hosted web calls Request guardrails now see and rewrite server_tool_calls replayed in chat messages. Cato and Purview scan Claude web_search and web_fetch calls on Responses output, and a function tool call with an extra custom key is no longer dropped from Cisco, Purview, Repello and PANW scans. * fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks * fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows * test: annotate the new server tool call test locals with Final and type their parameters * test: type the new guardrail test helpers with litellm's tool call types instead of Any * test: narrow the remaining new test helper parameters from bare dict and object * fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls * fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera * fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
norkelbyte
|
e0778bc9dd
|
refactor(types): replace Any with proven types in 21 files (#48921)
* refactor(types): replace Any with proven types in 35 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): keep anthropic chat transformation json path untyped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): accept null gemini vector store fields and load encryption TypedDicts on python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): validate Python client user responses and type delete_user as the count it returns * fix(types): keep loose payloads working under the sweep's validators An off-schema Anthropic stream event is now skipped instead of ending the guardrail's text and stop_reason scan, and MAI image usage accepts the loose numbers it did before. New dict and list adapters hide their input from validation errors, RerankResponse indexing keeps its field types, and the delete_user test mocks the int /user/delete returns. * fix(types): make the changed DynamoAI fields ReadOnly and skip non-object A2A stream items * refactor(types): tighten round 2 parse sites and keep payloads out of errors Parse Anthropic SSE events with validate_json and ignore extra fields, validate admin team and key rows per row with hidden inputs, type rerank get and __getitem__ per key, and move the Phoenix prompt version response to a frozen BaseModel * fix(types): keep MAI image calls alive on unreadable usage counts An unreadable MAI usage count now counts as absent instead of failing a billed image call, and the provider's usage fields pass through as sent. The Anthropic stream scan warnings no longer log raw model output, and the off-schema skip has tests that fail on the old scan. * fix(types): count non-object MAI usage as absent and parse Anthropic SSE with json.loads --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Max Azatian
|
77880a0653
|
fix(otel)!: leave out span cost when the price lookup failed (#44187)
* fix(otel): leave out span cost when the price lookup failed A model missing from the price map gets response_cost 0.0 in the standard logging payload, with response_cost_failure_debug_info set. The OpenInference writer (arize, phoenix, langfuse_otel) and the OTel v2 span data copied that 0.0 into llm.cost.total, litellm.cost.total and langfuse.observation.cost_details, so backends showed the call as free. Skip the cost when the lookup failed, except on cache hits, which are free. * test(otel): keep the guardrail charge on the guardrail span for unpriced calls * refactor(otel): type the Arize cost writer input and reuse the cache_hit flag * fix(otel): keep a billed guardrail fee on an unpriced call's Arize span cost Only the placeholder 0 is left out when the price lookup failed, so a guardrail fee LiteLLM billed still reaches Arize, Phoenix, Langfuse OTel and W&B Weave OTel. Adds unit cases for that fee and a malformed cost. * fix(arize): leave out the span cost on any failed price lookup, matching OTel v2 * fix(logging): clear a stale cost failure flag when a later pricing succeeds Stream-start pricing on the request model name can fail and set the cost failure flag even when the final pricing succeeds (OpenRouter usage.cost, an Azure deployment name priced from the response model). The new span gate then dropped a real cost. Each pricing now resets the flag, and uncached_price restores whatever flag it found. * test(logging): pin the start time in the stale cost flag tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Max Azatian
|
f43f2d8c15
|
fix(otel): put the error message on every failed span status (#44185)
* fix(otel): put the error message on the failed span status The legacy OTel logger (used by otel, langfuse_otel, arize and others) ended a failed call with Status(StatusCode.ERROR) and no description, so backends that read the span status, such as Langfuse's status message, showed an empty error. The text was only in the error.message attribute. Use the same message as that attribute, as the OTel v2 emitter already does. * fix(otel): take the failure status message from _record_exception_on_span Instead of reading the logging payload a second time in _handle_failure, _record_exception_on_span returns the error message it already records as error.message, and _handle_failure puts it on the ERROR status. The other callers ignore the return value, so their spans are unchanged. Its input is now a typed view of what it reads (exception, error_information, error_str) instead of a bare dict. * fix(otel): put the error message on every failed span status Covers the proxy SERVER span, management endpoint spans, service spans, and Arize Phoenix spans, so no ERROR span ships with an empty status description * fix(otel): cut the error status to the span attribute length limit and describe dangling server spans The status description now uses the error text as stored on the span, so OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT applies to it too. A server span the proxy closes on a 4xx or 5xx outside the request path gets the same description, and an empty error message falls back to the error class * test(otel): type the validation-handler status test * test(otel): cover the Phoenix parent span and error_str length-limit status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dfe4df8c8a
|
test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)
* test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
46d2c2a6ea
|
fix(otel): honor per-team Arize sampling rates in OTel v2 fan-out (#44595)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d80f8c28ca
|
refactor(types): replace Any with proven types in 137 files (#44478)
* refactor(types): prove runtime types at harness, search, rag and client boundaries Replace Any with adapter-validated types in the litellm.agent() harness, the search provider transformations, RAG ingestion and query, the vector store pre-call hook and registry, the galileo and opik logging integrations and the proxy client CLI. Each boundary gets unit tests for well-formed and malformed payloads. * chore(typing): prove types at more provider boundaries and restore search transformations Second pass over a2a, embedding, rerank, image, audio and small provider modules. Search transformations go back to their previous form because validating their response bodies would change the proxy status for malformed upstream bodies from 400 to 500. * refactor(types): prove types at logging, files, rerank, image, audio and management boundaries Replace Any with validated or annotation-only types in 37 more files: logging integrations, token counters, provider files/rerank/image generation/audio transcription transformations, pass-through logging handlers and management endpoints. No proxy HTTP status or error type changes. * refactor(types): prove types at repository, spend, files and router boundaries Replace Any with repository table accessors, validated mappings and annotation-only types in 33 more files: Prisma repositories, the enterprise batch and responses cost checkers, budget reservation, files endpoints, management endpoints, the policy registry, the adaptive and complexity routers and the secret managers. No proxy HTTP status or error type changes. * test(types): run the aiohttp transformation test in-process and cover repository row conversion The aiohttp chat transformation test no longer starts a server. It feeds the transformation a response whose json() returns the body under test. The proxy unit shards now exercise stored model rows whose params are JSON strings and the object permission create and update paths. |
||
|
yuneng-jiang
|
cf491d1df9
|
test: move tests/test_litellm integrations and secret_managers into tests/unit (#43194)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: move tests/test_litellm integrations and secret_managers into tests/unit Rename-only. Mirrors the old paths, including the directory conftests and the prompt and JSON fixtures. Follow-up commits prune and wire them. * test: prune and repoint the moved integrations tests Deletes the 7 audited tests a stronger test in the same tree already covers, imports the TLS sink helpers from their new conftest path, and restores os.environ after each integrations test. Some presets write OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the legacy tree's test ordering that header leaked into the AgentOps tests. * ci: run the moved integrations tests under their legacy flag The integrations GHA shard and a new CircleCI job run the integrations unit selection. secret_managers joins the misc selection. * docs: point integrations and secret_managers references at tests/unit * test: make the moved integrations directories packages * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: keep the job's UNIT_FLAG out of the shard-script tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |