* fix(spend_tracking): leave SpendLogs.session_id null when no client session id was established
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(lint): ratchet basedpyright budget after session_id fix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): ignore trace ids as session ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): gate null SpendLogs.session_id behind missing_session_id: omit
Unset, generate and reject keep the legacy trace id fallback. omit records only
metadata.session_id, the key Langfuse reads, so a trace id copied into
litellm_session_id by get_litellm_params never becomes a session.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): stamp the omit decision on the request so a config reload cannot fabricate a session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): keep omit covering requests the pre-call stamp never reaches
Router-model provider pass-through calls allm_passthrough_route directly and
skips add_litellm_data_to_request, so those requests never run the pre-call
helper and carry no omit stamp. Reading only the stamp made POST
/anthropic/v1/messages write a fabricated uuid into SpendLogs.session_id under
missing_session_id: omit while its Langfuse trace had no session, the exact
divergence the policy exists to remove.
The stamp now only pins omit on, and an unstamped request falls back to the
configured policy, so a config reload still cannot fabricate a session for a
request that was decided pre-call.
* fix(spend_tracking): make the session-omission marker proxy-owned so clients cannot forge it
* fix(spend_tracking): strip the client-sent omission marker from both metadata buckets before they merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): strip the session-omission marker from both metadata buckets
The pre-call policy ran before litellm_metadata is merged into metadata, so a
client that planted the marker in litellm_metadata had it copied back into the
route's own bucket after the strip and still got a null SpendLogs.session_id.
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): keep passthrough logging metadata and model_info dicts when team callbacks are wired
Passing team callback vars into Logging(kwargs=...) makes get_litellm_params materialize a full litellm_params, where metadata and model_info default to None instead of being absent. Readers that resolve them as .get(key, {}).get(...) then raise, so any passthrough request from a team with logging callbacks 500s once a pre-call guardrail is on, and the router strategy loggers log a traceback per request.
* test(proxy): annotate the closure dicts the passthrough logging tests record into
* fix(proxy): wire team-level logging callbacks into passthrough endpoints
LIT-5152: passthrough routes now wire dynamic team-level callbacks
(success_callback, failure_callback, callback_vars) into Logging constructor,
mirroring the add_litellm_data_to_request behavior. Three hardening fixes:
1. Catch TypeError/AttributeError in _get_validated_callback_metadata when
team logging metadata has wrong shape (e.g., logging list instead of dict),
preventing HTTP 500 on passthrough routes with malformed config.
2. Wrap websocket passthrough logging initialization in try/except, since
the socket is already accepted at that point; errors after accept() yield
abrupt close (1006/1011) rather than clean HTTP error response.
3. Handle malformed deprecated callback_settings gracefully with try/except.
4. Wrap HTTP passthrough callback resolution in try/except to prevent 500 on
malformed team metadata (backward-compatibility fix).
Changes:
- pass_through_endpoints.py: wire dynamic callbacks in HTTP+WS paths, handle
malformed metadata gracefully with try/except fallbacks
- litellm_pre_call_utils.py: expand exception handling in validators
- test file: regression test for happy-path team callback wiring
* refactor(proxy): share passthrough team-callback resolution and cover its fail-open path
Collapse the duplicated callback wiring on the HTTP and websocket passthrough
paths into one helper that returns a frozen wiring value, log resolution
failures at error level so a broken logging config stays visible, and add
regression tests for malformed team metadata and an operational lookup failure.
Reverts the _get_validated_callback_metadata except widening: it changed
behavior for normal LLM routes, which is outside this ticket's scope.
* fix(proxy): keep passthrough alive when team callback vars hold env references
The deprecated team_metadata.callback_settings branch builds
TeamCallbackMetadata directly, skipping the AddTeamCallback validation
that strips os.environ/ references from the newer logging list. Stamping
those vars onto the Logging object made its constructor raise, so a team
on the legacy shape got HTTP 500 on every passthrough call. Validate the
resolved vars inside the fail-open boundary instead, so the request goes
through with dynamic callbacks skipped and the reason logged.
* fix(proxy): lint violations in team callback wiring helper
* style: format lint
precached_prompt_tokens is a subset of prompt_tokens (OpenAI cached_tokens
semantics), so map it to prompt_tokens_details.cached_tokens instead of
adding it on top of prompt/total. Emit stream usage from any final chunk
carrying it rather than only finish_reason stop, which dropped tokens for
function_call and length streams. Merge auth metadata into a new dict in
the gigachat router handler instead of mutating the shared parsed-body
cache in place.
- sync llm_passthrough_route: read and close an error-status streaming
response before mapping it, so upstream 4xx/5xx surface as the provider
error instead of httpx.ResponseNotRead
- AsyncPassthroughStreamingResponse: expose aiter_bytes() and carry
_hidden_params so the router attaches headers in place instead of
wrapping the stream in HiddenParamsAsyncIteratorWrapper, which 500'd
every streaming azure router-model passthrough request
- logging: swap the passthrough httpx result for the transformed
ModelResponse/EmbeddingResponse when firing success callbacks
- get_llm_provider: resolve gigachat from its api base and drop the dead
gigachat_models elif branch
- constants: register the gigachat api base in openai_compatible_endpoints
- forward unrouted /gigachat/* requests with env credentials like other passthrough providers (the old fallback returned 400 on any request without a routed model, /gigachat/models included)
- fix basedpyright budget breaches across the gigachat provider, common_request_processing, and llm_passthrough_endpoints with real narrowing, no new suppressions
- add regression tests for the fallback target, auth header, and model-less endpoints
Adds litellm/litellm_core_utils/aws_partition.py mapping a region to its
AWS partition (aws, aws-cn, aws-us-gov, and the iso partitions), its DNS
suffix, and its ARN prefix, and uses it at every AWS host and ARN build
site: bedrock (runtime, agent, agentcore, legacy client, batches, files,
realtime), sagemaker, polly, secrets manager, s3 log uploads, bedrock
passthrough routes, and rag ingestion. ARN detection now accepts
arn:aws-cn: and arn:aws-us-gov: prefixes.
STS region resolution now falls back to the configured aws_region_name
after the aws_sts_endpoint host and the AWS_REGION/AWS_DEFAULT_REGION env
vars, so cn and gov role assumption no longer silently signs against
us-west-2.
A partition sweep test walks every endpoint builder with cn regions and
asserts no amazonaws.com host or arn:aws: prefix comes out, plus an AST
guard that fails on any new f-string hardcoding either literal.
PR #38114 dropped whichever header user_api_key_auth would read the caller's
key from, by precedence. Under custom_auth, JWT auth, or no master key that
header is the caller's own Google token, so the bring-your-own-credentials
Vertex branch answered 401 to every valid request.
A header value is now dropped only when it is the master key or when its
hash is the api_key that authenticated the request, so a Google token that
auth never consumed keeps flowing while a LiteLLM key still never reaches
Google.
test_passthrough_post_call_guardrails.py no longer plants a MagicMock
proxy_server module in sys.modules at import, which poisoned sibling tests
that read module globals at call time.
Staging split batch output-line costing into _safe_output_line_stats /
_compute_output_line_stats / _output_line_cost so one uncostable line can no
longer zero a whole batch, and added _provider_output_file_id so model-encoded
output file ids decode before the fetch. This branch's pass/fail counting was
written against the pre-split shape, where every None line meant a provider
failure.
Keep staging's structure and layer the counts on a three-way classification: a
provider-reported failure yields PROVIDER_FAILED, a provider-successful line
litellm cannot price yields UNCOSTABLE and stays in successful_requests billed
at $0. Without that split a litellm-side pricing gap would be reported to the
customer as a failed request and the counts would stop reconciling with the
provider's own request_counts.
Route the error-file fetch through _provider_output_file_id too, and carry the
new dataclass return through the callers staging added after this branch
forked.
On mapped pass-through routes, of which /vertex_ai is one,
user_api_key_auth accepts the caller key from a header literally named
litellm_user_api_key and applies it last, so it overrides every other source.
The credential-less filter neither dropped it nor resolved the caller key from
it, so a virtual key there reached Google past a real x-goog-api-key, and a
bring-your-own Authorization could be stripped when auth actually came from that
header. Drop it by name and resolve it at highest precedence.
The resolver placed both operator-configured key headers at the top of its
precedence, but user_api_key_auth only overrides with litellm_key_header_name;
a pass_through_endpoints litellm_user_api_key is checked last. So a request that
authenticated via Authorization while also sending a pass-through header could
have the wrong value chosen, leaving the authenticated Authorization key
forwarded. Order the resolver exactly like get_api_key: override first, built-in
headers next, pass-through header last.
user_api_key_auth also accepts the caller key from a pass_through_endpoints
entry's headers.litellm_user_api_key, not just litellm_key_header_name. Drop
every operator-configured caller-key header by name and treat them as
top-precedence caller-key sources, so a virtual key sent through one is never
forwarded to Google.
The LIT-4761 streaming-classification tests passed only the bring-your-own
Google OAuth token in Authorization and mocked get_litellm_virtual_key, a shape
that cannot authenticate in production. The credential-less filter now resolves
the caller key by auth precedence, so a lone Authorization value reads as the
key and is stripped. Send the virtual key in x-litellm-api-key, matching a real
request, so Authorization is preserved and the classification assertions run.
The filter's own Bearer-only stripping missed the other schemes
user_api_key_auth accepts, so a virtual key echoed as `Authorization: Basic
<key>` alongside a higher-precedence auth header did not match the caller key
and was forwarded to Google. Reuse the auth module's _get_bearer_token so the
comparison strips exactly what authentication does (Bearer / bearer / Basic /
AWS4-HMAC-SHA256), falling back to the raw value for a bare token.
The credential-less filter derived the caller key only from x-litellm-api-key,
Authorization, and the custom header, but the route authenticates through
Depends(user_api_key_auth), which also accepts the key from x-goog-api-key. A
virtual key sent only in x-goog-api-key therefore authenticated yet was kept as
a preserved upstream header and forwarded to Google. Resolve the caller key by
the same precedence get_api_key uses and value-strip exactly that, so a key in
x-goog-api-key is stripped while a real Google key alongside a higher-precedence
virtual key is preserved.
The hand-rolled drop set missed Ocp-Apim-Subscription-Key, so a caller
Azure APIM secret in that header was forwarded to Google on the
credential-less branch. Derive the name-drop set from the canonical
SpecialHeaders.litellm_credential_header_names(), minus Authorization and
x-goog-api-key which double as real Google credentials and are value-stripped
instead. New credential headers added there are now dropped automatically.
The router hop _ageneric_api_call_with_fallbacks canonicalises the passthrough
call type onto litellm_metadata, and the cost callback reads spend attribution
from that bucket while only backfilling user_api_key* keys from metadata. The
helper was building on metadata, so agent_id and user_api_end_user_max_budget
were silently dropped before the callback ever saw them. Build and pass the
attribution under litellm_metadata so every field survives.
user_api_key_auth also authenticates a caller from the operator-configured
general_settings.litellm_key_header_name, reading that header straight off
the request, so a virtual key sent there survived the credential-less Vertex
forwarding filter and reached Google alongside a real bring-your-own
credential. Value-strip every header whose value matches the caller's key
from any accepted source, including that custom header.
On the credential-less Vertex passthrough branch, drop every header that
can only carry LiteLLM caller auth (x-litellm-api-key, api-key, x-api-key)
by name, since Google never consumes them, and strip the virtual key by
value from Authorization / x-goog-api-key, which may instead hold a genuine
bring-your-own Google credential. This closes the residual leak where a
distinct caller secret in api-key or x-api-key still reached upstream.
Adds a regression asserting the value-based strip also drops the caller's
virtual key when it is duplicated into the api-key and x-api-key headers,
while a genuine bring-your-own Google credential still forwards.
The credential-less Vertex passthrough dropped the caller's LiteLLM
virtual key only from Authorization by exact match. A caller who sent
the same key in x-goog-api-key (which doubles as a real Google
credential) had it accepted as a credential and forwarded upstream.
Drop the virtual key by value across every forwarded header, normalizing
any Bearer prefix, so no header name carries it to Google.
When no Vertex credential is configured (no default_vertex_config, no matching
use_in_pass_through deployment, no vector-store credential), the Vertex passthrough
took the bring-your-own-credentials branch and forwarded the entire incoming header
set upstream to Google. That set included whichever header carried the caller's
LiteLLM virtual key: x-litellm-api-key, or Authorization when get_litellm_virtual_key
read the key from there. The proxy's own secret was sent to a third-party provider.
The credential-less branch now drops x-litellm-api-key and the Authorization value
that equals the virtual key, keeping a genuine bring-your-own Google credential
(an OAuth token in Authorization, or x-goog-api-key) so real BYO passthrough still
works. When neither survives, the request fails with a clean 401 telling the operator
no credential is configured, instead of forwarding the virtual key.
Regression coverage in the mapped test path asserts the 401-and-never-forwarded
behavior for both leak vectors and that a real Google credential still passes through
with the virtual key stripped.
The /vllm and /azure router-model passthrough branches called
llm_router.allm_passthrough_route directly with no request metadata,
so the cost callback saw no user_api_key and no
user_api_key_budget_reservation. Spend for a budgeted virtual key hit
neither the key's spend nor the spend logs, and the reservation minted
at auth into the shared Redis counter was never released, drifting the
counter up until the key falsely tripped BudgetExceededError.
Thread the authenticated key's attribution metadata into both calls via
the same builder add_litellm_data_to_request uses, so the cost callback
attributes spend and reconciles the reservation. Regression tests cover
both branches.
POST /v1/videos without an input_reference file now goes out as
multipart/form-data the way the OpenAI SDK always sends it, instead of a
JSON body that OpenAI-compatible backends (SGLang Diffusion, vLLM-Omni)
reject; gemini, vertex, and runwayml keep their JSON bodies
/v1/images/edits on the openai/azure/openai-compatible path now forwards
unknown provider params (e.g. seed) and honors extra_body, matching
/v1/images/generations, and aimage_edit forwards
extra_headers/extra_query/extra_body instead of dropping them
Generic pass-through no longer downgrades a file-less multipart form to
application/x-www-form-urlencoded
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
