- sync llm_passthrough_route: read and close an error-status streaming
response before mapping it, so upstream 4xx/5xx surface as the provider
error instead of httpx.ResponseNotRead
- AsyncPassthroughStreamingResponse: expose aiter_bytes() and carry
_hidden_params so the router attaches headers in place instead of
wrapping the stream in HiddenParamsAsyncIteratorWrapper, which 500'd
every streaming azure router-model passthrough request
- logging: swap the passthrough httpx result for the transformed
ModelResponse/EmbeddingResponse when firing success callbacks
- get_llm_provider: resolve gigachat from its api base and drop the dead
gigachat_models elif branch
- constants: register the gigachat api base in openai_compatible_endpoints
Adapts streaming_model_restamp.py to the LIT001/LIT010 gates that landed
on staging since this branch was cut (Final annotations, Mapping in
annotations instead of dict).
Assistant list content was forwarded to /v1/responses/input_tokens as chat
`text` blocks, which the Responses API rejects (it accepts only output_text
and refusal inside an assistant turn). The 400 sent the whole request to the
local tokenizer, so any conversation with an assistant turn silently lost
provider-exact counting, including the image counting added in 73ab647b1c.
Assistant content now collapses to the plain string the Responses API counts
identically, and image parts are kept to user turns where they are legal.
* fix(vertex_ai): graft default vertex path when api_base has a version-only path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): keep query and fragment placement when grafting vertex path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): merge alt=sse into existing query when streaming
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend_tracking): persist router metadata in spend logs for internal router models
* test(spend_tracking): expect router_metadata key in exact-payload tests, type the routed-kwargs helper
When the upstream errors while the client is still connected, the pump
forwards the exception through the relay queue so the proxy's failure
handling re-raises it. If the client disconnects before consuming that
queued exception, neither the failure hook nor billing ran and the spend
row was lost. The pump now waits for client detach and, if the exception
was never consumed, salvages partial spend like the post-disconnect
error path.
Also rewrites the bedrock disconnect logging test to the detached-pump
contract: billing fires after the upstream drain completes, not
synchronously at aclose().
The chat-to-Responses reverse transform kept only text blocks, so an image
input was dropped before the count went to OpenAI. A 256x256 image request
counted 13 tokens instead of 268.
/v1/responses/input_tokens returned 200 with a count for an empty
"input" ("" or []), while OpenAI returns a 400 missing_required_parameter.
The route also went through optimistic budget reservation, which is only
released by LLM success/failure callbacks that a token count never
reaches, so every call leaked a reservation until TTL expiry and could
429 real traffic. Both routes plus the /openai alias now join
/utils/token_counter in the reservation exemption set.
Resolving github_copilot/chatgpt names through get_llm_provider runs the
provider's OAuth device flow synchronously on the event loop. Adopt the
declared provider in PatternMatchRouter.get_pattern, which the auth
layer's zero-cost budget check walks on every request against wildcard
routers, and in /utils/supported_openai_params.
When the pump finishes draining while the client is still connected,
billing is deferred to the proxy's post-response hook, which only fires
on a normally completed response. A client disconnect before the relay
consumed the queued tail tore the generator down past that hook, so the
request logged no spend at all. The relay teardown now dispatches the
stored deferred billing whenever it never reached the end-of-stream
sentinel.
Also drops the live pass_through_tests script: that CI job runs against
a fixed config with no Bedrock model or AWS credentials, so it could
only fail there. The scenario is covered by unit tests on the
relay/pump seam.
* test(newrelic): cover static default_team_settings per-team routing
The dynamic POST /team/{team_id}/callback path for New Relic is tested, but
the static default_team_settings twin had no regression coverage. Add a test
that drives default_team_settings -> add_team_based_callbacks_from_config and
asserts the resolved trusted vars dispatch to BOTH the per-team metrics logger
(cost/usage) and the trace logger (LLM/agent spans), so a config-file customer
gets the same per-team routing as the API customer.
Also correct the /team/callback docstring: callback_name is a str validated
against the credential-capable callbacks, not a fixed langfuse/langsmith/gcs
Literal, and document the newrelic_api_key / newrelic_region vars.
* chore(ui): sync schema.d.ts with the /team/callback docstring
Regenerate the dashboard OpenAPI types for the add_team_callbacks description
change: callback_name is a validated str (not a langfuse/langsmith/gcs
Literal) and the newrelic_api_key / newrelic_region vars are documented.
* docs(newrelic): note LITELLM_OTEL_V2 prerequisite, trim test comments
Address review: team-scoped New Relic config is rejected with a 400 unless the
proxy runs with LITELLM_OTEL_V2=true, so document that in the /team/callback
endpoint and sync schema.d.ts. Drop the narrative setup comments in the new
test per the repo comment convention; the test name and docstring already say why.
* fix(proxy): deliver budget alerts on webhook-only alerting and accept ALERTING_WEBHOOK_URL
ProxyLogging.budget_alerts forwarded to the alerting pipeline only when
'slack' was in general_settings.alerting, so alerting: ['webhook'] plus
WEBHOOK_URL silently never delivered a budget alert (the config
/health/services?service=webhook exists to test). Forward when 'webhook'
is present too; SlackAlerting.send_alert already fans out per channel.
Also accept a provider-neutral ALERTING_WEBHOOK_URL env fallback for the
Slack-format channel (any Slack-compatible receiver works), mark it as a
sensitive var, and de-brand the admin UI alerting copy.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): format settings.tsx with prettier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for updated alerting description
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger checks after ALERTING_WEBHOOK_URL docs merged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A window whose start equals its end is the natural way to spell off-peak all
day, and the docstring's promise that a window may wrap past midnight invites
it. It took the non-wrap branch instead, where start <= now < end can never
hold, so it matched nothing. It parses cleanly, so it never reached the branch
that ignores malformed windows: no exception, no log, and the model billed at
standard rates around the clock while the config said otherwise. Let equality
fall through to the wrap branch, which covers every instant, and say so in the
docstring.
Reported by @xyzs996 in review.
Tiered pricing resolves its own base rates and returns early, before the
off-peak swap ran, so a model carrying both tiered_pricing and off_peak_pricing
billed the tier rate around the clock. Route every base-cost path through one
helper so the window applies wherever the rates came from, and say plainly in
the docstring that an off-peak rate replaces the rate it lands on rather than
discounting it
They sat at the end of the file, which is where everyone else appends
too, so this branch picked up a conflict there on nearly every rebase.
Grouping them with the other _get_token_base_cost test keeps them clear
of that churn and next to the code they cover. Pure move, no test changes
_is_within_off_peak_window used current_time.time(), which drops tzinfo, so a caller passing a non-UTC aware datetime had the window compared against local wall-clock instead of UTC. That silently mispriced off-peak requests. Normalize aware datetimes to UTC before comparing; naive datetimes stay as-is per the documented UTC contract. Added a regression test with a UTC+8 datetime that fails without the fix
Some providers charge different per-token rates depending on the time of
day. DeepSeek, for example, has historically discounted its chat and
reasoner models during an off-peak window (16:30-00:30 UTC). LiteLLM's
cost map only modeled static per-token pricing, so cost tracking could
not stay accurate for these providers.
This adds optional off-peak pricing to a model entry: input_cost_per_token_off_peak,
output_cost_per_token_off_peak, cache_read_input_token_cost_off_peak, and an
off_peak_hours_utc window expressed as "HH:MM-HH:MM" in UTC (the window may
wrap past midnight). When the current UTC time falls inside the window, the
cost calculator uses the off-peak rates and otherwise falls back to the
standard rates, so existing models are unaffected. The fields are also
accepted as custom pricing on a deployment, so they can be set from the
proxy config or the SDK.
The window check is a pure function that takes the current time as an
argument, which keeps the regression tests deterministic without patching
the clock.
When a pre-call filter left no order-2 deployments, target_order matching
fell through to the remaining healthy list and reselected the failed
primary. Prompt-cache and deployment affinity also pinned that hop back
to order 1. Match the requested order strictly, skip those pins while
target_order is set, and keep target_order across retries of that hop.
Default remains true so lite claude turns tool search back on through
a proxy. An explicit false or auto in the env or settings is left alone
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Claude Code turns tool search off when ANTHROPIC_BASE_URL is a proxy.
lite claude, lite up, login --config-claude, and autoroute now force
ENABLE_TOOL_SEARCH=true so MCP tools stay deferred through the proxy
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
_image_sources had no test asserting what it extracts. The existing image tests
live on the Bedrock side and all use base64 without a media_type, which is the one
path the fix left unchanged, so both behaviors it does change went unverified: the
url shape reaching the guardrail at all, and base64 arriving as a data URI.
Against the pre-fix extractor the url case sees [] and the media_type case sees
['AAAA'] instead of ['data:image/png;base64,AAAA'].
The remaining three assert behavior the fix deliberately preserves -- bare base64
passed through, a file source yielding nothing, a malformed source dropped rather
than handed on for a consumer to choke on.
Each message carries a text block because a message with no text never reaches the
guardrail, which would make every source shape look equally dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
