Commit graph

96 commits

Author SHA1 Message Date
zoveki
c8cef1a95b
fix(a2a)!: bill completion-bridge agents at their own prices on /a2a send and stream (#49150)
* fix(a2a): bill completion-bridge agents at their own prices on /a2a send and stream

message/send on an agent served through the completion bridge (custom_llm_provider
agents such as LangGraph or databricks_agent) logged a $0, 0-token row, and
message/stream logged no row at all, so cost_per_query and per-token prices never
reached spend.

The bridge send path now records counted usage, the agent's pricing keys, and the
agent id on its logging object, the same way URL agents do. The bridge stream is
wrapped in the A2A streaming logger, which now reads the bridge's artifact event
for the reply text and gets the agent name from the proxy. Streamed calls also
count prompt tokens again, since the message text reader now reads pydantic
messages as dicts.

* fix(a2a): count chunked and Task replies once and skip agent prices on error replies

Bridge streams now follow the A2A append flag per artifact, so chunked
pydantic-ai, watsonx, and AgentCore replies count the whole reply once.
Send replies that are a Task count their artifact text. A JSON-RPC error
reply is logged at cost 0 instead of the agent's price.

* fix(a2a): keep one reply text per stream and bill every artifact and status reply

* fix(a2a): ignore a null error field on streams and run the stream per-token billing test

A stream event whose error field is null no longer zeroes the call's cost. The integration test for per-token stream billing now runs instead of skipping, and the URL agent SDK case skips when the optional a2a-sdk is not installed.

* fix(a2a): bill streams the client drops and skip the bridge's working placeholder

A stream the client closes or cancels mid-reply now logs its spend row at the agent's prices on both the bridge and URL paths, and a stream dropped before the agent sends anything still logs nothing. The bridge's "Processing request..." status no longer counts as reply text, so an empty bridge reply bills no completion tokens, as send does. Adds proxy-level tests that a bridge stream bills under the agent's card name.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-30 19:43:45 -07:00
rj-cairn
d06b889d41
fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream

The Gemini and Vertex AI streaming path read the SSE body with httpx
iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks
on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085,
U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds
one of them was cut in two. The tail is not a data: line and the head never
parses, and the accumulate-and-retry fallback then carries the unparsed
prefix forward, so every later event is appended to it and the stream ends
with no more content and no usage.

SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines
split decoded text on exactly those, treating a CR at the end of one chunk and
an LF at the start of the next as one break. The first request and every
long-decode hop now read through them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF

Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks.

* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator

* fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed

Switch the remaining httpx stream readers (Anthropic chat, Google GenAI,
Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent
Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and
others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a
raw U+2028 or U+2029 inside a JSON string no longer splits an event.

Keep the chunk that closes a think block in the logged and cached response
when a stream ends without a finish_reason, close the block before a legacy
function_call, and make sync Codestral FIM streaming send stream=true and use
the text-completion-codestral stream parser.

Test fakes that only mocked iter_lines now stream real httpx bytes and
assert the content they feed in.

* fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests

Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on
U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use
split_sse_lines from sse_lines.py, which also gains a str.split fast path.
Adds tests/unit/llms/codestral to the unit workflow shard.

* fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines

* test: type the new stream tests, drop a comment, and benchmark the SSE line splitter

* test: type the iterator class parameters in the line-separator stream tests

* fix(responses): stream tool call arguments opened in a text chunk

* fix(responses): hold a tool call that shares a chunk with text until the text ends

* test: type the a2a SSE handler helper and the sandbox fake responses

---------

Co-authored-by: Tim Vink <vinktim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
2026-10-28 17:00:29 -07:00
Mateo Wang
464932b1d1
refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789)
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1)

* refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2)

* refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts

* fix: restore pass-through behavior on paths the typing pass tightened

MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites

* fix(runwayml): keep a malformed 2xx task body on the 500 path

* fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report

EdenAI's embedding and image generation parse dropped a non-object body from the error
they raised, so the message no longer named what the provider sent. A helper now raises
with the body in the message, and both paths have a regression test.

`lite encryption migrate` exited 0 with no message when the proxy's report was not an
object or its residual_legacy was not a number, where base crashed. It now fails with a
message naming the value.

The CompactifAI and Gemini image-edit non-object body tests accept TypeError or
AttributeError, which both base and this branch raise, so they guard against a
ValidationError instead of pinning one class. RunwayML's async parse gets the same
non-object body test as the sync one.

* fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed

* test: annotate the new regression tests' locals with Final

* test: annotate the remaining new test locals with Final
2026-10-24 11:31:09 -07:00
Mateo Wang
04eaa6b4a8
refactor(types): replace Any with proven types in 15 files (#47509)
* refactor(types): replace Any with proven types in 9 shared files

* refactor(types): replace Any with proven types in the remaining sweep files

* refactor(types): keep unproven integrations on the merge base and pin malformed-body handling

* refactor(types): leave five request paths on the merge base so non-object bodies keep their 400

* refactor(types): drop a docstring that repeats the helper name

* refactor(types): leave six files with no live malformed-payload proof on the merge base

* refactor(types): leave Anthropic pass-through SSE parsing on the merge base

* refactor(types): leave Responses output walking, the retry ceiling and concatenated tool JSON on the merge base

* fix(types): keep main's parsing in the budget window reset, utils loaders and Langfuse attributes

The new strict checks rejected inputs main skips or keeps, such as a JSONB
string in budget_limits, so these three files go back to main's version

* refactor(types): return the request tag parse failure as a value
2026-10-20 19:02:18 +00:00
Max Azatian
5145250d83
fix(langfuse_otel)!: send the completion start time (#45597)
* fix(langfuse_otel): send the completion start time

langfuse_otel never set langfuse.observation.completion_start_time, so
Langfuse showed no time to first token for any call, streamed or not.
The legacy langfuse callback already sends kwargs["completion_start_time"].
Send the same value as an ISO timestamp with an explicit offset.

* fix(langfuse_otel): type kwargs and test against a real span

Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any],
the declared type of model_call_details, so the new completion start time
read adds no unknown-type errors. The tests now check a real SDK span
instead of patching safe_set_attribute, and the parameter is typed.

* fix(otel): send the completion start time from the v2 Langfuse mapper

With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper,
which had no completion start time either. Carry the payload's
completionStartTime on LLMCallSpanData for streamed calls, the same calls
that get time_to_first_chunk_seconds, and map it to
langfuse.observation.completion_start_time.

* fix(langfuse_otel): take the logging kwargs as a read-only Mapping

The new kwargs annotation tripped LIT001. Both helpers only read the
logging kwargs, so type them Mapping[str, Any] instead of dict.

* fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter

* fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON

The native Gemini, Interactions and A2A streams never recorded a first chunk, so with
completion_start_time now sent both emitters showed TTFT equal to latency on them. They
now record it, and Gemini and Interactions take the call's start time like /v1/messages.

Both emitters send the completion start time as a JSON string, the form the Langfuse SDK
and the langfuse callback send, through one serializer.

* fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event

A2A streams took their start time when the iterator was built, so their latency left out
the work before it. They now use the call's start time like every other route.

Cached /v1/messages stream replays recorded no first event, so their time to first token
fell back to the end time. The replay now records it on the first event it yields.

The tests use the real Interactions config and typed logger hooks.

* fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free

* test(a2a): type the card fetch test and stop rebinding the seed helper's recorder

* test(otel): expect completion_start_time and cache_hit in the B3 span keys

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-20 13:52:26 +00:00
Mateo Wang
130699ca23
refactor(typing): clear 1143 basedpyright Any errors across 273 files (#46574)
* refactor(typing): narrow Any sources in proxy, guardrails, bedrock and responses code

Turns untyped json.loads, response.json(), header lookups and forwarded
kwargs into object or precise types at their source, so downstream code
type-checks without reportAny / reportExplicitAny. No runtime behavior
changes.

* refactor(typing): narrow Any sources in caching, MCP, providers and proxy request code

Decoded JSON, cache reads and header lookups now enter as object or
precise types, prisma tables used by the shadow-eval logger get typed
Protocol views, and a few SDK parameters move from dict[str, Any] to
Mapping[str, object]. No runtime behavior changes.

* refactor(typing): narrow Any sources in proxy server, router, logging and shared utils

* refactor(typing): narrow Any sources in provider transformations, integrations and proxy endpoints

* refactor(typing): narrow Any sources across providers, proxy and router

Deepens the Any cleanup on the hottest files: typed helpers in
core_helpers for attribute and item access, identity TypeAdapter(object)
wraps at untyped boundaries, explicit return annotations on shared
provider and proxy helpers, and Protocols in place of Any parameters.
No intended behavior change.
2026-10-15 21:08:47 +00:00
devin-ai-integration[bot]
da2f5a498f
feat(agents): support Databricks Agent Platform agents in the agent gateway (#45681)
* feat(agents): support Databricks Agent Platform agents in the agent gateway

Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire
(Responses-shaped input, output_text items, output_text.delta SSE) to Model
Serving endpoints (/invocations or the unified /serving-endpoints/responses)
and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials

Register it through the agents config block or the Admin UI agent catalog and
call it through /a2a/<agent> (message/send, message/stream) or model:
a2a/<agent> chat completions. The chat route now runs every completion-bridge
agent through its own provider, and attaches the minted Databricks OAuth token
for both bridge and URL agents. Consumed OAuth settings never reach the
provider call

* fix(agents): harden databricks_agent input, output, and OAuth handling

Non-text content parts answer 400 unless drop_params is set, a reply without
an output list answers 502, flat OAuth fields resolve an os.environ api_base,
the bridge debug log no longer echoes the request model, routing tests stub
only the HTTP boundary, and the catalog description fits the dropdown

* fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block

* fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App

* refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name

* fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty

* test(agents): drive the flat OAuth a2a send through the real bridge

* fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change

* fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents

* fix(databricks_agent): address review caveats on chat bridging, params, and auth

Map max_tokens and max_completion_tokens to max_output_tokens and pass
temperature, top_p, and user, so /v1/messages works without drop_params.
Send A2A message metadata only with string values, answering 400 or
dropping the rest under drop_params. Join message items with a blank line
in both stream and non-stream replies. Cache the env service principal
token until shortly before it expires. Keep databricks_agent replies out
of the response cache.

On the chat bridge, let request fields win over the agent's stored params
while pinning the agent's api_base and api_key, and apply the agent's
static_headers between client and agent extra_headers, as /a2a does.
Resolve os.environ/ only for databricks_agent api_base and api_key instead
of every bridge provider. Ignore an unusable databricks_oauth block on URL
agents on chat, as at the merge base, instead of answering 400.

* test(internal_context): list the databricks agent token cache as process-local

* fix(databricks_agent): keep the agent's endpoint, input, and target out of client control

extra_body model and input answer 400, or are dropped under drop_params.
A client api_base or base_url never replaces the agent's card URL on the
bridge. Chat to a URL agent keeps its merge-base call, and an incomplete
databricks_oauth block on a URL agent keeps its merge-base ValueError.

* test(databricks): expect the merge-base 400 for a client api_base on agent chat

* fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach

* fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-15 12:42:06 -07:00
Mateo Wang
0ebec123fd
chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors

Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy
modules with one pydantic validation at the point the payload enters the
module, so every value read afterwards carries a real type.

reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright,
ruff-strict or type-discipline rule is higher repo-wide or in any single
file, and no cast, ignore or suppression marker was added.

* chore(typing): replace 47 Any annotations that every caller already satisfies with object

Each annotation here is a parameter or return type whose value is only
stored, compared, logged or forwarded, so object describes it exactly and
no function body changes. A site was kept only if no basedpyright,
ruff-strict or type-discipline rule rose in any file that references it.

reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655.

* chore(typing): type upstream responses and loose annotations across providers, proxy and integrations

Second pass of the Any reduction. Decoded upstream JSON is validated once at
the provider boundary and read through typed views afterwards, annotations
that every caller already satisfies move from Any to a concrete type, and the
nine search transformations validate each result item on its own so error
text no longer carries the item index.

No cast, no type: ignore, no suppression marker and no budget file edit

* ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard
2026-10-13 15:35:23 -07:00
devin-ai-integration[bot]
6b7d1928b8
fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards (#44463)
* fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(a2a): cover streamed 0.3 replies through the real client

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(a2a): accept new kwargs in create_a2a_client fake; add protocol-version integration tests

The create_a2a_client fake in tests/agent_tests/test_a2a_agent.py listed a
fixed signature, so asend_message's new protocol_version= kwarg raised
TypeError in test_a2a_non_streaming and test_a2a_streaming. Accept and
ignore extra kwargs with **_, matching the new _capture fake in test_main.py.

Add tests/integration/compatibility cells for litellm_params.a2a_protocol_version
against a real 2-worker proxy with Postgres and Redis and scripted A2A peers
modeled on langgraph-api 0.15.4: pinned send/stream on all three routes, aliases,
the 1.0 opt-out, unchanged genuine v1 agents, PATCH taking effect on both
workers, persistence via /v1/agents, the localhost retry, invalid values,
upstream errors, the dialect hint, and the completion bridge stripping the key.

Merge base 5f969982fc: 33 red, 7 green. Head ba709d3de5: 40/40 green twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 09:04:36 -04:00
devin-ai-integration[bot]
77005080f3
refactor(types): replace Any with proven types in 6 files (#46042)
* refactor(types): replace Any with proven types in 8 files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(types): keep azure search and google_genai seams untyped

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 10:00:52 +00:00
devin-ai-integration[bot]
e2971e0af4
refactor(llms): expose public names for private provider helpers (#45037)
* refactor(litellm): migrate private usage in llms

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(litellm): preserve Bedrock batch signature marker

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(litellm): migrate llms private usage symbols

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(litellm): preserve UUID Watsonx project IDs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(litellm): retarget llms mocks to public names

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(litellm): limit llms changes to renames and forwarders

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(litellm): keep GCS mock client patching private Vertex auth methods

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(litellm): assert forwarder arguments and type forwarder signatures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-07 08:49:25 -07:00
Mateo Wang
d80f8c28ca
refactor(types): replace Any with proven types in 137 files (#44478)
* refactor(types): prove runtime types at harness, search, rag and client boundaries

Replace Any with adapter-validated types in the litellm.agent() harness, the
search provider transformations, RAG ingestion and query, the vector store
pre-call hook and registry, the galileo and opik logging integrations and the
proxy client CLI. Each boundary gets unit tests for well-formed and malformed
payloads.

* chore(typing): prove types at more provider boundaries and restore search transformations

Second pass over a2a, embedding, rerank, image, audio and small provider
modules. Search transformations go back to their previous form because
validating their response bodies would change the proxy status for
malformed upstream bodies from 400 to 500.

* refactor(types): prove types at logging, files, rerank, image, audio and management boundaries

Replace Any with validated or annotation-only types in 37 more files: logging
integrations, token counters, provider files/rerank/image generation/audio
transcription transformations, pass-through logging handlers and management
endpoints. No proxy HTTP status or error type changes.

* refactor(types): prove types at repository, spend, files and router boundaries

Replace Any with repository table accessors, validated mappings and
annotation-only types in 33 more files: Prisma repositories, the enterprise
batch and responses cost checkers, budget reservation, files endpoints,
management endpoints, the policy registry, the adaptive and complexity routers
and the secret managers. No proxy HTTP status or error type changes.

* test(types): run the aiohttp transformation test in-process and cover repository row conversion

The aiohttp chat transformation test no longer starts a server. It feeds the
transformation a response whose json() returns the body under test.

The proxy unit shards now exercise stored model rows whose params are JSON
strings and the object permission create and update paths.
2026-10-05 11:01:45 -07:00
yuneng-jiang
bba85f0b6c
chore(lint): remove the LIT002 mutable-construction rule (#43971)
* chore(lint): remove the LIT002 mutable-construction rule

Drop LIT002 from scripts/check_type_discipline.py along with its helpers,
its budget entry, its unit tests, and the AGENTS.md and gate docstring
mentions. `# mutable-ok` now only suppresses LIT001, so the markers that
only existed to silence LIT002 became LIT013 stale suppressions and are
removed. The files whose layout depended on those trailing comments are
reformatted with ruff format.

Every other LIT rule count is unchanged and the ASTs of all touched
litellm/ files match main apart from one docstring.

* chore(lint): keep the leftover mutable-ok markers for a follow-up

Restore the ~1.4k `# mutable-ok` markers stripped in the previous commit
so this PR only touches the checker, its tests, the budget, and docs.
Those markers no longer suppress anything, so `# mutable-ok` is exempt
from LIT013 until a follow-up strips them.

* Revert "chore(lint): keep the leftover mutable-ok markers for a follow-up"

This reverts commit c35bc0b84e.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-01 12:24:02 -07:00
mateo-berri
c62c187054 Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5
# Conflicts:
#	litellm/integrations/otel/model/metadata.py
#	litellm/litellm_core_utils/coroutine_checker.py
#	litellm/litellm_core_utils/exception_mapping_utils.py
#	litellm/litellm_core_utils/llm_response_utils/response_metadata.py
#	litellm/llms/bedrock_mantle/responses/transformation.py
#	litellm/llms/openai/chat/guardrail_translation/handler.py
#	litellm/proxy/common_utils/reset_budget_job.py
#	litellm/proxy/db/db_spend_update_writer.py
#	litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py
#	litellm/proxy/openai_files_endpoints/storage_backend_service.py
#	litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py
#	litellm/realtime_api/main.py
2026-09-21 12:57:38 -07:00
yassin
6786eb0131 fix(a2a): send message/stream for Bedrock AgentCore streaming requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 13:39:32 +00:00
Devin AI
8795be0a65 refactor(types): replace Any with proven types in 32 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 10:51:46 +00:00
Devin AI
ccd8997b08 refactor(types): replace Any with proven types in 34 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-20 10:16:48 +00:00
mateo-berri
44fff9297d fix(a2a): narrow discovery status codes through a structural protocol
basedpyright cannot narrow the probe error through
isinstance(error, AgentCardResolutionError) while that class is imported
inside try/except ImportError, which left three reportAttributeAccessIssue
errors over the budget main now carries. A runtime-checkable Protocol with
the same status_code contract carries the narrowing instead, so the check
no longer depends on the possibly unbound SDK name and the import goes.
2026-09-17 17:17:21 -07:00
Devin AI
75eec8712c fix(a2a): keep the upstream status on card discovery failures and inject the card client in tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-16 23:55:57 +00:00
mateo-berri
e243237a7c feat(a2a): reach Microsoft Foundry agents with Entra auth and versioned card discovery
Foundry serves its agent card only at agentCard/v1.0, accepts only an Entra ID
bearer, and defaults to a non-blocking send, so the A2A relay and the chat
completions route could not use it.

The relay gains an agent_card_path litellm_param plus agentCard/v1.0 as a third
discovery probe, mints a bearer from flat Entra fields on the agent
(tenant_id, client_id, client_secret, azure_ad_token, azure_username,
azure_password, azure_scope) for https://ai.azure.com/.default, and sends it on
the card fetch, message/send, message/stream, tasks/* and the chat bridge.

Chat completions look the registered agent up by its provider-stripped name so
its api_key and headers reach the request, tag every message with its kind, ask
for a blocking send, fall back to a blocking send when the registered card says
streaming: false, and fail the call on a JSON-RPC error inside a stream instead
of yielding an empty one. Entra fields stay out of the chat bridge's logged
parameters.

Resolves LIT-5122
2026-09-16 16:36:47 -07:00
Yassin Kortam
cdb4eaac59
Merge pull request #40262 from BerriAI/litellm_lit_7190_inline_token_counts
fix(proxy): run the remaining inline token counts off the event loop
2026-09-15 13:38:00 -07:00
mateo-berri
6df5eef768 Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5
# Conflicts:
#	litellm/llms/bedrock/files/transformation.py
#	litellm/proxy/_lazy_features.py
#	litellm/proxy/analytics_endpoints/cache_activity.py
2026-09-14 07:27:54 +00:00
mateo-berri
d93baa3f2c fix(bedrock): sign on a dedicated executor instead of the shared default one
asyncio.to_thread puts every Bedrock signing on the loop's default executor, the same pool every provider's async entry point hops through, so signings parked on botocore's refresh lock queued unrelated providers behind Bedrock. run_aws_signing runs them on a 16-thread pool only AWS signing uses
2026-09-09 18:52:00 -07:00
mateo-berri
89c3a8216b fix(bedrock): sign requests off the event loop on every async path
SigV4 signing resolves AWS credentials, and botocore refreshes expiring
credentials inside that signing with a blocking HTTP call. Every async
Bedrock path that still signed on the event loop (/v1/messages, Converse,
count tokens, the agent-runtime and Comprehend Medical pass-throughs,
async-invoke status polling, realtime, AgentCore, SQS, S3) now signs on a
worker thread, so one Bedrock request no longer stalls the whole worker.

Fixes #40165
2026-09-08 11:54:43 -07:00
yassin
b5a7032eb4 fix(proxy): run the remaining inline token counts off the event loop
Wrap the context-management editors, the end-of-stream chunk builder,
acount_tokens, the compression interception hook, the passthrough
interrupted-stream recovery, the A2A usage counters, and the semantic
cache embedding truncation in asyncify so a multi-megabyte payload no
longer stalls the worker's event loop while it is tokenized

The pass-through suite now drains the process-global logging worker
from an autouse conftest fixture so work queued on one test's loop
cannot fire against the next test's callbacks

Resolves LIT-7190

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 18:42:38 +00:00
mateo-berri
1ec5083ab4 refactor(types): replace Any with real types across 54 more backend files
Third batch of the fifth basedpyright Any reduction round. Every change is
typing-only and leaves runtime behavior identical.

Provider transformation configs, video and rerank base classes, OTel metadata
and the guardrail and realtime type modules move their payload, header and
optional-parameter annotations from Any to object, Mapping[str, object] or the
concrete model the call site already produces. Repositories and endpoints that
reached Prisma through an untyped handle now name the actions they call with the
repo's own TableActions protocol. The pydantic field retypes were checked
against pydantic to confirm object and Any validate, serialize and generate JSON
schema identically.
2026-09-08 12:41:07 +00:00
mateo-berri
066652c194 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_decrease_anys_opus5_r4
# Conflicts:
#	basedpyright-code-budget.json
#	litellm/a2a_protocol/utils.py
#	litellm/llms/azure_ai/vector_stores/transformation.py
#	litellm/llms/milvus/vector_stores/transformation.py
#	litellm/llms/openai/vector_stores/transformation.py
#	litellm/llms/ragflow/vector_stores/transformation.py
#	litellm/proxy/container_endpoints/endpoints.py
#	ruff-strict-budget.json
#	type-discipline-budget.json
2026-09-02 23:32:10 +00:00
mateo-berri
459858829e refactor(typing): replace Any with proven types in 89 more backend files 2026-09-02 23:08:42 +00:00
devin-ai-integration[bot]
cde9d94c36
feat(agentcore-a2a): derive runtime session id from A2A message.contextId (#39371)
Native AgentCore A2A always sent either a fresh generated runtime session id
or the single configured runtimeSessionId, so related turns lost context and
unrelated callers shared one AgentCore microVM. The runtime session id is now
params.message.contextId scoped to the calling key hash, then runtimeSessionId,
then generated, and is length-validated (33-256) before the header is signed.
Invalid ids surface as JSON-RPC -32602 / HTTP 400 instead of a 500.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-02 12:40:24 -07:00
mateo-berri
362fb4cffe refactor(typing): replace Any with proven types in 42 more backend files 2026-09-02 15:35:01 +00:00
mateo-berri
f94bd6d903 refactor(typing): replace Any with proven types in 65 backend files
Typing-only pass over backend modules that carried the most reportAny and
reportExplicitAny errors. Every new annotation is backed by a construction
site, a call site, or an isinstance narrowing that already existed; untyped
JSON boundaries were left alone rather than declared without validation.

Tree-wide basedpyright errors drop 138,481 to 138,007. reportAny drops 8,854
to 8,645 and reportExplicitAny drops 3,119 to 2,814.
2026-09-02 09:11:36 +00:00
mateo-berri
b74e696615 refactor(types): replace Any with real types across 178 backend files
Types provider request and response bodies at their boundaries with
TypedDicts and Protocols instead of dict[str, Any], so the untyped-to-typed
crossing is paid once per boundary rather than once per field read.

Removes 1,700 reportAny/reportExplicitAny errors and 1,953 basedpyright
errors overall, plus 310 ruff strict-rule and 106 LIT-rule violations.
No cast, type: ignore, noqa, or new Any annotations anywhere in the diff.

Ratchets the basedpyright, ruff-strict, and type-discipline budgets to the
new counts so the cleared headroom cannot silently grow back.
2026-08-27 10:14:58 +00:00
mateo-berri
fe567bd846 fix(a2a): speak the 0.3 dialect to servers with mis-cased protocol bindings 2026-08-24 10:51:14 -07:00
milan
de0d8ceb25 refactor(a2a): return a normalized card copy instead of mutating the resolved card
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-23 03:20:53 +00:00
milan
0697188be4 refactor(a2a): use direct typed access in protocol binding normalization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-23 03:14:19 +00:00
milan
d6f9bce4bb fix(a2a): normalize agent card protocolBinding casing before transport match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-22 04:05:02 +00:00
mateo-berri
11552dbafc chore(typing): drop 1.3k basedpyright errors across 42 Any hotspot files
Replace implicit and explicit Any with real types across the highest-density
reportAny/reportExplicitAny files: module-private TypedDicts for dict payloads,
Protocols for duck-typed collaborators, and existing litellm/types models where
they already describe the shape

No new cast(), no # type: ignore, no # pyright: ignore, no # noqa, and no
new suppressions. Diagnostics that could not be resolved without one were left in
place rather than hidden
2026-08-19 01:12:41 +00:00
mateo-berri
6b5249bcce refactor: replace Any with precise types across responses, proxy, and llms modules
Types 28 files with Protocols, TypedDicts, and Pydantic validation in place
of Any, cutting basedpyright reportAny by 974 and reportExplicitAny by 261
(1411 errors total across 48 rules), and ratchets the basedpyright, ruff
strict, and type discipline budgets down to match
2026-08-13 03:59:26 -07:00
mateo-berri
20eb7bb437 chore(typing): clear 1.4k basedpyright Any errors across 21 hotspot files
Typing-only pass over the 21 files with the highest reportAny and
reportExplicitAny density among self-contained modules: management
endpoints, guardrails, streaming internals, response transformations,
MCP server, enterprise managed files, and vector store management.

Whole-tree basedpyright drops from 148,648 to 146,984 errors (-1,664),
with reportAny -1,111 and reportExplicitAny -296. No rule increased
repo-wide and no file regressed on any rule. No cast(), type: ignore,
noqa, suppression comments, or new Any annotations anywhere in the diff,
and no runtime behavior changes.

Budgets ratcheted by make lint-budget-update: basedpyright -1,663 across
48 rules, ruff-strict -86, type-discipline -110.
2026-08-08 08:14:29 -07:00
Yassin Kortam
ae1d1cb05e
fix(http): stop pooled clients persisting cookies on the aiohttp jar too (#36149)
#35978 stopped the pooled A2A client replaying one upstream's Set-Cookie to
another by installing a blocking policy on that client's httpx cookie jar. That
covers only one of the two jars on the request path. AiohttpTransport is the
default transport unless it is explicitly disabled, and the aiohttp ClientSession
behind it keeps its own cookie jar which no httpx-level assertion can observe, so
the leak is still live on the default path: a live proxy on that commit still
delivers agent-alpha's session cookie to agent-beta's card fetch and JSON-RPC
call.

The reason it looked fixed is that aiohttp's default CookieJar is built with
unsafe=False and refuses to store cookies for IP hosts, so a proof addressed to
127.0.0.1 comes back clean whether or not that jar is blocked.

Cookie persistence is now blocked where the clients are built rather than at one
call site: blocked_cookie_jar() gives every httpx client, async and sync, a jar
whose DefaultCookiePolicy(allowed_domains=()) rejects every domain in both
directions, and both ClientSession constructions litellm owns, the transport's
session factory and the proxy's shared startup session, get a DummyCookieJar.
LiteLLM reads a response cookie nowhere, and an explicitly supplied Cookie header
still goes out, so passthrough forwarding and an agent's extra_headers are
unaffected. The A2A-scoped policy #35978 added is removed, since it is now dead.

The two suites that drive the aiohttp session factory synchronously mock
ClientSession because a real one needs a running event loop; DummyCookieJar has
the same requirement, so they mock it for the same reason.
2026-08-07 11:05:59 -07:00
Yassin Kortam
2b38991df9
fix(a2a): stop writing per-caller state onto the shared cached httpx client (#35978)
create_a2a_client took the raw client off a process-wide cached handler and
called headers.update() on it, then leaned on folding the header set into the
cache key (through the unrelated disable_aiohttp_transport field) to keep one
caller's credentials away from the next.

Per-caller headers now ride with each request through the a2a SDK's call
context, and the agent card fetch gets them through resolver_http_kwargs, so
the shared client is never written to and its cache key no longer varies by
header set. Since the proxy puts a fresh trace id in every request's headers,
that key previously changed on every call, giving each request its own httpx
client and flushing the 200-entry client cache that every other provider
shares. All A2A callers on one timeout now reuse a single pooled client.

Sharing that client also means sharing its httpx cookie jar, which httpx fills
from every Set-Cookie and replays on any later request to a matching domain, so
one agent's session cookie would arrive at another agent on the same host. The
pooled client now carries a cookie policy that stores and sends nothing, which
neither litellm nor the a2a SDK relies on: the SDK's auth interceptor skips
cookie-borne API keys outright.
2026-08-06 18:58:26 -07:00
mateo-berri
2bd6af1b64 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_daily_any_cleanup_08_04_2026 2026-08-05 12:43:11 -07:00
mateo-berri
6c76f5f9c6 chore(lint): clear grandfathered over-limit lint drift and ratchet budgets down
Every ruff-strict rule that sat above its budget limit (FURB188, RUF022,
SIM118, UP007, UP032, UP037) is now at zero, LIT001 and LIT006 are back
under their ceilings, and the freed headroom is ratcheted out of
ruff-strict-budget.json, type-discipline-budget.json, and
basedpyright-code-budget.json so the gates take the fast path again
2026-08-05 12:18:13 -07:00
mateo-berri
d259070fdf Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_daily_any_cleanup_08_04_2026 2026-08-05 12:13:11 -07:00
mateo-berri
338e411103 chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom 2026-08-05 02:37:24 -07:00
mateo-berri
42564e896f
chore(typing): replace Any seams with real types across responses, proxy, and provider adapters
Replace Any-typed payload dicts, record shapes, and provider request/response
seams with TypedDicts, Protocols, and precise annotations in the ten litellm/
files carrying the highest combined basedpyright reportAny + reportExplicitAny
counts. No behavior changes.

Adds a regression test covering the managed-id list path so a prisma client
missing the managed tables keeps returning a fail-closed empty page.
2026-08-04 20:15:49 +00:00
mateo-berri
2708620d6a feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
Classic298
c9887a1f94
perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
mateo-berri
b604e2b20c refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007
modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and
friends), removal of the 1,296 typing imports the rewrite orphaned, and
hand fixes for what the fixers could not see: five star-import
freeloaders of typing names, two F823 late-import annotations, the
/get/config/list introspection crash on types.UnionType, redundant
function-local RoleMappings imports in ui_sso.py that shadowed the
module-level name once the annotation lost its quotes, and one FURB168
tautology.

B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes
rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms
basedpyright then rejects (283 new errors measured), so their budgets
stay at base values.

ruff-strict-budget.json drops by 39,579 this commit (39,968 across the
branch) with 28 rules at an actual 0 and 9 more sharply down.
type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001
moves to the now-honest total: the checker matches the spelling `set`
but not the alias `Set`, so the 160 typing.Set annotations rewritten to
set[...] were always mutable-set annotations and only now count.
2026-08-01 15:43:29 -07:00
mateo-berri
397e8e4918 refactor(imports): move collections.abc names out of typing and ratchet the lint budgets 2026-08-01 13:17:42 -07:00