* fix(proxy): leave cache-read tokens out of the priority limiter's token count
The priority limiter counted cache-read tokens that the TPM limiter leaves out, so mostly cached prompts were throttled early. Usage dicts in Anthropic's own shape now count uncached input plus cache writes.
Fixes#51970
Co-authored-by: yifei tang <fl2326151118@163.com>
* fix(parallel_request_limiter): count cache writes and cache reads in the prompt of an Anthropic-shaped usage dict
A usage dict in Anthropic's own shape reports input_tokens without its cache reads and
cache writes. The Rust reconciliation read input_tokens as the full prompt and subtracted
cache reads from it, so billable input dropped the cache writes and clamped to 0.
Fixes#51970
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yifei tang <fl2326151118@163.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: safely reuse and retire generic API loggers
* test: cover generic API logger reuse and rotation
* fix(proxy): unregister the current generic API logger after its headers rotate
Removing a database-configured generic API callback after a header rotation put the retired logger back, and it kept sending logs through its replacement. The database callback sync now compares and removes callbacks by the logger that is currently active
* fix(logging): let a retired generic API logger finish its sends and move its cache tests
A header change used to give the old logger 30 seconds to finish, then cancel it and drop any logs still queued on it, even when its own timeout and retries allowed longer. The old logger now finishes its last sends under its own send settings
The cache tests move into the existing logging callback manager test file, with typed parameters and settings swapped in through monkeypatch
* fix(logging): hand a retired generic API logger's queued logs to its replacement
Logs still queued when a header change retires a logger now go out through the new logger with the new header. Also drop the flush_queue test double, add Final to the proxy test, and cover both current-logger mappings in DB callback sync
* fix(logging): send queued logs with the old config when the new logger filters their event types
* test(logging): read delivered logs and headers through one helper
* fix(logging): check the hand-off target's event types under the flush lock
* fix(logging): stop retired loggers from resending queued logs or rejoining callback lists
* fix(logging): read the cached generic API logger under the callback lock
Two threads rotating the same logger could each build a replacement, leaving a listed logger that is not the cached one. Also drop a no-op mapping of displaced database callbacks, which are only ever strings, and add a regression test for a file callback that rotates during database registration
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): answer every /scim/v2 error with the RFC 7644 SCIM Error schema
Okta reads the detail of an RFC 7644 SCIM Error when it rejects a provisioning call, and the /scim/v2 routes answered with the proxy's OpenAI error shape or FastAPI's detail object instead, so a rejected create showed nothing after "Errors reported by remote server"
A route class on the SCIM router now turns every HTTPException, ProxyException and request validation error, auth and premium dependency failures included, into a SCIM Error served as application/scim+json with the same HTTP status. Duplicate userName and duplicate group creates carry scimType uniqueness, and a failed user create logs the userName, status and reason
* fix(scim): keep placeholder admin errors in the proxy shape and never drop an error's reason
The /scim/v2/placeholders list and merge routes are LiteLLM admin tools, so they
go back to the proxy's error body. An error detail SCIM cannot read now answers
its own text instead of the bare status phrase. The integration tests that read
SCIM error bodies now expect the SCIM Error schema.
* fix(scim): answer every SCIM route error as a SCIM Error and keep the span's error type
Every 409 on a SCIM route now answers scimType uniqueness, so a duplicate email
that only differs in case gets it too. An unexpected error answers a SCIM Error
with the status phrase and is logged like the handlers' own errors. The OTel
server span reports the ProxyException or a public RequestValidationError the
route raised instead of ScimError
* fix(scim): keep unexpected 5xx text out of SCIM detail and trace an HTTPException as itself
* fix(scim): keep deliberate 5xx reasons and log every refused SCIM create with its userName
A 5xx answers the status phrase only when its text is the raw text of an error raised without a status. The roster sync retry hint, the auth database outage 503, and other deliberate reasons keep their text, and each failed roster write is named with the same rule. The refused-create warning moves to the SCIM route, so a create refused by the key check or the body schema logs its userName too
* fix(scim): answer the status phrase for a 5xx whose reason quotes an unexpected error
A team write that wraps a database error in its own 500 put the table name in the SCIM roster sync detail. A 5xx now answers the status phrase whenever its reason carries the text of an error raised without a status, and keeps a reason that does not quote one
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(prompt_management): apply a prompt manager once per request on acompletion and SDK fallbacks
acompletion ran the prompt manager's async hook and then completion() ran its sync hook on the same request, so a template was sent twice, and once more per attempt under SDK fallbacks. acompletion now passes an internal marker to completion() and to each fallback attempt when its manager has its own async hook, and both skip their pass when it is set. The proxy strips a client-sent marker so a caller cannot skip an admin's prompt manager
* fix(prompt_management): reuse the primary's template on fallbacks and apply it once on Responses-bridged chat calls
Fallback attempts reuse the rendered messages and the settings the template returned, an attempt whose model names a prompt integration prefix still runs it, the chat-to-Responses bridge marks its call so the Responses side skips the template, and the applied marker is an enum value a JSON request cannot carry
* fix(prompt_management): template each fallback attempt once from the caller's own messages
With SDK fallbacks the outer call no longer applies a registered prompt manager, so each
attempt renders its own template once, keeping a fallback's own prompt_id and the
template's precedence over the fallback's settings. A prefix-routed model gets no marker,
so a registered manager still applies after litellm_agent/ resolves the model, and
completion(fallbacks=...) skips its discarded outer pass
* fix(prompt management): skip the template on a chat call the Responses API hands back to chat, and keep a missing model a 400
* fix(prompt management): apply the template once on the Responses-to-chat bridge, keep cache points, and detect an applied async hook by its result
* fix(responses): build the empty prompt marker default with a factory so litellm imports on Python 3.11
* fix(fallbacks): keep the caller's credentials on a fallback behind the same prompt prefix
The SDK fallback runner compared providers on the prompt prefix (litellm_agent/, dotprompt/), so a
litellm_agent/openai fallback of an OpenAI primary counted as a provider switch and dropped the
caller's api_key and client
* fix(fallbacks): keep a fallback's own prompt template and drop credentials when its template can switch provider
* fix(fallbacks): resolve a prompt-prefixed fallback's provider from its own model name
* fix(responses): treat a prompt hook that only changed parameters as applied
A Responses call handed to chat now marks the template as applied when the
hook changed only parameters, so chat skips re-running it and still places
the cache points
* fix(responses): pass a template's chat-only settings to the bridged chat call
A Responses call handed to chat now forwards the template settings the
Responses request cannot carry, such as max_tokens, the way the chat side
applies a hook's settings
* fix(responses): keep the Responses call's own parameters out of a template's bridged chat settings
A template that set timeout or extra_headers made the bridged chat call get that keyword twice and fail. Those stay the caller's, as on base and on the native Responses path
* fix(prompt management): keep a template from sending the deployment's key to another host or provider
A Responses call handed to chat took a template's api_base and sent the deployment's key there, and a litellm_agent/ fallback that names its own prompt kept the caller's key when that prompt picked another provider. The bridged chat call now takes only a template's chat inference settings, and that fallback drops the caller's credentials, both as on main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): record the guardrail verdict on a request dict that starts empty
The Bedrock guardrail and the shared `log_guardrail_information` wrapper both picked the request dict with `or {}`. An empty dict is falsy, so a caller that passed one had the verdict written to a throwaway dict. /apply_guardrail passes an empty dict for a request that carries no messages and no metadata, so nothing downstream could see what the guardrail decided. The verdict now goes to the dict the caller passed, and only a missing dict falls back to a new one.
* fix(guardrails): log requests that a guardrail blocks on /apply_guardrail
A request a guardrail blocked on /apply_guardrail was missing from the logs. The endpoint skipped the failure handlers for an HTTPException, the spend log failure hook wrote no row because the route is not an LLM API route, and the guardrail's verdict stayed on a request dict no logger reads.
The endpoint now runs the failure handlers for every exception and copies the verdict onto the logged request, so allowed and blocked requests both carry it along with the guardrail's cost. It marks the failure of a request it had set up with a sentinel, and the spend log failure hook writes its row for that mark. A request rejected before setup, such as one with a bad key or an unknown guardrail, is still not logged. Guardrail records a caller sends in metadata are dropped for both response formats so a verdict cannot be forged in the logs.
* feat(guardrails): scan an image on /apply_guardrail with a Bedrock guardrail
/apply_guardrail only took text. It now takes `images`, one base64 data URI per request, and `text` becomes optional. The image is validated with the attachment reader the Bedrock guardrail already uses for chat completions and sent to ApplyGuardrail through the same call, with source OUTPUT for a response-direction scan. The image is scanned first, and the text scan is skipped only when the request has neither text nor messages.
A request the guardrail cannot judge answers 422 instead of 200 or 400. That covers an unreadable image, a guardrail that is not a Bedrock ApplyGuardrail guardrail, and an answer in which Bedrock does not report the image as guarded.
* test(guardrails): run the image tests through the endpoint's real request setup
The endpoint test helper on main now goes through the real request setup and guardrail lookup, and expects each request to leave one log entry. A body the request model refuses never reaches the endpoint, so those two cases are checked on the model. The guardrail that cannot scan images now carries the name the request asks for.
* fix(guardrails): log an image Bedrock did not evaluate as not run
An ApplyGuardrail image call that Bedrock answered without guarding the image
was logged as a guardrail success, so the request log read as a pass for an
image nobody checked. It now logs as not_run. A Bedrock error inside a 200
body on /apply_guardrail answers 502 naming the AWS error type instead of the
Image modality message
* fix(guardrails): refuse message text a Bedrock guardrail would not scan
* fix(guardrails): answer 422 when Bedrock cannot evaluate an image on /apply_guardrail
* refactor(guardrails): return image scan failures as values and type the test override
* fix(guardrails): keep Bedrock API errors reporting as HTTPException and log image-only prompts
* fix(guardrails): keep the scanned image in the apply_guardrail spend log request
* fix(guardrails): redact the apply_guardrail image when message logging is off and keep Bedrock API errors a plain HTTPException
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(tts): stream audio from the provider as it arrives on /v1/audio/speech
The proxy read the whole audio body from the provider before sending the
first byte to the client, so time-to-first-byte was the full synthesis
time. Pass stream=True to get the bytes forwarded as they arrive.
The response type does not change: it is still HttpxBinaryResponseContent,
only unread, so every existing caller, logging path and spend hook works
the same. Providers opt in with supports_streaming on their config, the
others ignore the flag and buffer as before.
* fix(tts): keep streamed speech inside its concurrency and cleanup guarantees
Streaming the audio out moved two things that used to be covered by the
buffered response, and the review on #46222 caught both.
A deployment's max_parallel_requests counted syntheses that had started
rather than syntheses in flight, because aspeech now returns at the
response headers with the audio still arriving. The slot is held until
the response is closed, the way the chat streaming path already does it.
A client that walks away mid-clip leaves the body unread, and httpx only
releases the connection when it is read to the end. The body iterator
closes it on the way out, and the route arms the same close as a
background task for a response whose body is never read at all. These
are two different paths on purpose: Starlette raises ClientDisconnect
out of the response before background tasks run, so neither one covers
both cases.
Also pulls the speech body cast into one helper beside
speech_request_body, which drops a banned typing.cast import.
* test(tts): cover the slot-holding helper directly
router_code_coverage.py wants every function in router.py called by name
from a unit test, and _hold_slot_until_closed was only reached through
router.aspeech(). Test it on its own instead of adding it to the ignore
list: which responses keep their slot is the part worth pinning, so the
cases are buffered body, non-binary response and unread body.
* fix(tts): stream ElevenLabs speech from its /stream endpoint
ElevenLabs' convert endpoint synthesizes the whole clip before it sends the
first byte, so stream: true still waited for full synthesis. Requests with
stream: true now call /v1/text-to-speech/{voice_id}/stream.
* fix(tts): log streamed speech like buffered speech
A deployment-hook sync copied stream=True onto the logging object for
speech, so the success handler waited for an assembled chat stream that
never comes and streamed speech never reached spend logs or callbacks.
* fix(tts): forward streamed speech audio as it arrives
* fix(tts): log streamed speech when the audio ends
Streamed speech now logs success when the caller reads the audio to the end or closes it, and logs a failure when the upstream breaks mid-clip. Key max_parallel_requests slots stay held for the whole stream, the logged duration and bytes match what was sent, and the router slot is no longer pinned by a pending logging task.
* fix(tts): bill and map streamed speech failures, defer logging only when streaming
A streamed clip that breaks mid-way is now logged as a failure at the clip's cost, since speech is priced per input character and the provider bills the whole request. Transport errors map to litellm.Timeout or litellm.APIConnectionError, and the proxy runs post_call_failure_hook with the clip's cost. Success logging waits for the stream end only when the caller asked for stream=True.
* fix(tts): stream only on a literal true, mark hang-ups, and cut off without a fake DB outage
* fix(tts): put the 499 hang-up marker on the spend row
* fix(tts): keep the caller's metadata and positional base_url intact, add speech stream integration cells
* fix(speech): keep stream keyword-only and buffer the speech health check
Passing stream positionally skipped the logging wrapper, and a deployment
with stream: true left the health check's provider stream open. Also gives
each scripted chat reply its own id so the non-boolean stream cases do not
collide on the spend log key, and adds the audit's round-2 cells
* test(speech): check the health check calls the buffered path
* test(integration): read the speech chaos spend rows from the worker database
Main now gives each owned proxy a private database unless the test file opts
into the worker's state. The chaos tests read billed rows from the worker
database, so they opt in with the shares_worker_state marker like the other
providers chaos files
* fix(tts): release the router slot when a streamed read fails
The deployment slot was only released through Response.aclose, which httpx
does not call when the body read raises. A provider cutting a clip off
mid-read left the slot held, so later Router.aspeech calls got 429.
Release the slot on any stream end, and close the upstream stream when a
read fails.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(tts): drive speech stream tests with events instead of sleeps
Co-Authored-By: Claude <noreply@anthropic.com>
* test(utils): capture streamed sync speech usage when the audio ends, not at the headers
* fix(proxy): close streamed speech when the route fails before responding
A response headers hook, or any other step between aspeech returning an unread
audio stream and the route returning its StreamingResponse, could raise and leave
that stream open. The upstream connection and the deployment's
max_parallel_requests slot then stayed held, so later clips got 429s. The route
now closes the audio on any error before it returns
* fix(proxy): end streamed speech as failed when the route errors before responding
Closing the unread audio on a pre-response error went through aclose, which always
ends as StreamClosedEarly: the clip was success-logged with a 499 ClientDisconnected
marker while the route returned 500. The route now ends the audio with the error,
through every OnEndAsyncByteStream wrapper, so it is logged as a failure. A
cancellation still closes it as a hang-up.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(proxy): close unread speech audio when a deployment guardrail rejects it
A guardrail's post-call deployment hook runs before the audio stream's end logger is attached, so a rejection left the unread provider response open and held its connection. The failure path of the async wrapper now closes it, and a close that fails never replaces the rejection
Also annotates the new test locals in test_routes_audio.py with Final
* fix(proxy): free the key's parallel slot before a streamed speech clip ends
A streamed clip released its key's max_parallel_requests slot only from the
success logging task, which defers the Redis write until after the caller has
seen the end of the audio. A next clip sent to another instance at that moment
could still be counted against the key and get 429. The speech route now
releases the slot when the audio finishes, before the response ends, the same
way chat streams do.
* fix(speech): log a failed streamed clip once per failure hook
A logger with both failure hooks got both for one failed aspeech stream. The
stream's failure path now marks its sync run as part of an async request, as the
non-stream path does. Also type the route test helpers and check that the speech
health check leaves no provider connection open.
* test(speech): expect one async failure event from a cut-off aspeech stream
* fix(proxy): shield the speech upstream close and tag cut-off rows as aspeech
A hang-up while the provider is silent no longer lets the anyio cancel abort the upstream close
on servers that cancel the stream on disconnect. Mid-clip cut-off failure rows now store the
aspeech call type instead of an empty one.
* refactor(logging): record a speech client disconnect through the logging object
* test(speech): hold the provider mid-clip in the guardrail-rejection wire test
The test sent the whole short clip at once on a keep-alive connection, so when the clip reached the HTTP client before the guardrail rejected it, closing the response handed the finished connection back to the pool and the test timed out waiting for a close. The peer now holds after the first audio chunk until the rejection is raised, as a real provider still sending a long clip would
* fix(speech): close the unread audio when a sync speech call fails after its headers
The async wrapper already closed it. The sync wrapper left the provider
connection open until garbage collection when post-call processing raised.
* fix(router): keep hung-up speech clips out of latency samples
A streamed clip the caller hangs up on ends early, so its short duration made
the deployment look fast to latency-based routing. The clip still counts as a
request.
* test(proxy): type the speech request helper in the audio route tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(responses): write one failure row per refused Responses WebSocket turn
A turn the deployment refuses on an open native or managed Responses WebSocket now writes its own spend
log failure row when it is refused, with its own request id, the turn's start time and request, the
socket's resolved model group and the websocket call type. Before, refused turns wrote nothing until the
socket closed, and then only one row for the whole socket, carrying the socket's open time and first frame.
Guardrail-blocked turns go through the same row builder, so their rows now carry the model group and call
type too. Turns refused or crashed in the turn check before a session-limit cancel also keep their rows.
The socket's close-time failure hooks still run once to settle the admission budget reservation, but when
the close failure is the first refused turn the spend logger skips the duplicate row.
* fix(responses): keep forged logging state out of refused-turn rows and wait for each row
A refused turn's failure row now leaves out the top-level fields the HTTP path strips from a client body, plus standard_callback_dynamic_params and litellm_params, so a frame cannot forge its own logging state or turn off message redaction. The relay now waits for each refused turn's row before it reads the next frame, and the write still finishes if the socket is cancelled meanwhile
* ci: start the unit workflow GitHub never created for the previous push
* fix(responses): keep the client's request tags on refused-turn rows
A refused turn's row now carries the key's tags plus the client's User-Agent tags, computed from the socket's request headers like its success rows, while the headers themselves stay out of the row
* fix(responses): keep a refused turn's frame from forging its spend row and store no blocked prompt
A refused turn's failure row now takes only the turn's model and call id from the frame, ignores frame metadata tags and model_group, and logs non-object metadata as empty instead of dropping the row. Guardrail-blocked frames no longer carry a stored request.
* fix(proxy): keep the socket's session id, guardrail records, and router-key strip on refused WebSocket turn rows
* fix(proxy): use the public router-reserved metadata field list, as main renamed it
* fix(responses): give refused WebSocket turn rows the socket's session id
When a socket's request carries no trace id, the router minted its own for the relayed turns, so success rows and refused-turn rows had different session ids. The endpoint now hands the router the socket's trace id
The refused-turn row test's session-limit case jumps the loop clock past the limit once the relay is parked, instead of racing a 50 ms timer
* test(responses): expect the parallel-slot refusal at the WebSocket handshake
Main now refuses a cached key that is over its limits before the request body is read, so a second socket on a key whose only parallel slot an open socket holds is refused at the upgrade instead of with a 429 frame. The test still checks that the refused turn's failure row keeps the slot held until the first socket closes
* fix(responses): take a refused WebSocket turn's agent and session fields from the socket
A frame's metadata.agent_id, billing_agent_id or session_id no longer lands on the refused turn's spend row. The row copies those fields from the socket, like its success rows
The integration test no longer pins the reopened socket's 1011 close, which is incidental to the held slot it checks
* test(responses): expect a 403 when a held parallel slot refuses a second WebSocket handshake
* test(responses): expect a held parallel slot to refuse a second socket's first turn with a 429 frame
Main's #51675 skips the headers-only limit check for a bodyless GET, and every WebSocket upgrade is one, so a second socket on a key whose only parallel slot is held now opens and its first turn gets a 429 max_parallel_requests frame and close 1008 instead of a 403 at the handshake
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep both errors when a streamed cross-group fallback also fails
On /v1/messages and /v1/responses streams, a recovery walk appended the
fallback outcome to the MidStreamFallbackError envelope and the stream then
raised the inner provider error without it. A fallback hop whose stream opened
and then failed before any content never reached the walk, so the primary's
error was dropped. The outcome now lands on the surfaced error, and a fallback
hop's pre-content failure reads as the walk's own failure would.
* fix(proxy): keep the router fallback outcome in Responses response.failed events
The response.failed formatter preferred the upstream body message, so the
primary error and fallback outcome the router put on the exception never
reached the client. Use the exception message when it carries that outcome.
* fix(router): only add the primary error to a fallback stream's failure before any content
Count any chunk past message_start and pings, split or not, as content on
/v1/messages, track forwarded output events on Responses, and switch the
response.failed message only on a fallback-also-failed outcome.
* fix(router): scope the streamed fallback outcome to pre-content messages and Responses failures
Leave the router core and chat streams as they were. Carry the outcome the
walk put on the stream's envelope onto the provider error only before any
content, and read a fallback hop's /v1/messages stream as whole SSE frames so
split or merged chunks are judged by event, not by chunk.
* fix(router): keep a Responses stream's own error once it has sent output
A primary Responses stream that already sent output no longer lets its
fallback stream's failure read as the primary's. Keep CRLF line endings in a
rewritten error frame, and read a bridge chunk's type through a typed mapping.
* test(router): cover partial frames on a fallback stream's end and raise
* fix(router): carry the primary error through a fallback stream's retry and redact the combined frame
A fallback group's same-group stream retry keeps the fallback's error context,
every fallback-level /v1/messages stream is read as whole frames, and the
rewritten error frame goes through the client message redactor.
* fix(router): treat a cut-off message_start on a fallback stream as before content
* fix(router): redact the primary error and the fallback outcome separately before joining them
A traceback in the primary's error made the client redactor drop the fallback
outcome that followed it. Make the fallback-retry tests deterministic with a
single-deployment fallback group.
* test(router): cover a split error frame on a fallback group's retried stream
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(copilot): accept empty or whitespace stored key with key ref (#52014)
* fix(copilot): treat blank GitHub App fields as unset and keep other values fail-closed
Booleans and other non-string App field values stay configured so a
non-App request carrying them still fails closed, and a non-string key
ref no longer passes as a blank one
* fix(copilot): never store blank GitHub App fields on a credential
POST /credentials drops a blank or whitespace-only App field instead of
storing it, and PATCH treats one as a deletion. A stored blank key came
back masked as ***** and the dashboard then edited it as an uploaded key
* fix(copilot): keep non-admin credential creates with blank App fields forbidden
* test(copilot): pin that a null App key on update keeps the stored key
* test(copilot): pin that a false or 0 key ref next to an uploaded key fails closed
* fix(copilot): keep signing a whitespace-only key that has no key ref
A named credential with a whitespace-only App key and no key ref answered
400 at the head, which the router never retries. The merge base reached JWT
signing and answered 401, so a model group with a working sibling kept
serving. Whitespace still counts as unset next to a key ref.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(shadow-eval): let reverse jobs skip samples the baseline group served
A reverse shadow eval judges the router's response against baseline_model.
When the router itself sent the request to the baseline's model group, the
comparison only produces ties and still pays for a shadow and a judge call.
Reverse jobs can now opt in with skip_same_model. A sampled request whose
served group (router alias resolved, fallback hops respected) is the
baseline's own group is counted in a new same_model funnel stage instead of
being judged. It makes no provider calls, writes no attempt row, and leaves
max_turns and the eval budget untouched. When the served group is unknown
the sample is judged as before. The job detail reports
same_model_skipped_count even when nothing has been judged yet.
The flag defaults to false and is rejected on forward jobs, so existing
jobs and requests behave exactly as before. Adds two columns through an
additive, schema-only migration.
* test(shadow-eval): drive same-model skip tests off explicit completion signals
Judged cases now wait on the attempt write the injected prisma reports,
then await the pipeline task that made it. Skip cases drain only the tasks
the hook created, through a task factory scoped to the hook call, before
asserting that no provider call, attempt row, or spend was recorded. This
replaces a whole-loop task diff and the assertion on the number of tasks.
Adds a skip case with tier parameter overrides, which the policy still
treats as the same group, and a case showing a lost sampling roll counts as
not_sampled before same_model. Drops the class docstring claim that a same
group comparison only produces ties.
* style(tests): remove extra shadow evaluation test spacing
* fix(ui): send skip_same_model false from the shadow eval start form
The generated request type now requires skip_same_model, which broke the
dashboard typecheck and builds. The form sends false, the API default, so
jobs started from the UI behave exactly as before.
* fix(migrations): bound shadow evaluation schema lock waits
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
gateway-auth-core gains plan/ and credential/, which plan and decide key and
JWT authentication on top of its route rules. The Python proxy now runs that
core through python-bridge with a PythonAuthHost, which replaces
handle_jwt.py and the key and JWT paths of user_api_key_auth.py.
The core defines Token, MasterKey and KeyHash, and config gains the JWT auth
settings. Route groups named in allowed_routes match as starlette's
compile_path compiles them, like _allowed_routes_check, and a team's
allowed and denied pass-through routes read as RouteLists. allow_user_auth,
allow_requests_on_db_unavailable and enforce_rbac read through python-compat's
IsTrue, StrBool and Truthy adapters, and StrBool reads text with
flags::str_to_bool.
Fixes that landed on main in the code this moves carry over: every user read
the core makes from a JWT hands the raw email_verified claim to the host,
which reads it with read_email_verified, the impersonation header applies
after a master key, cached admin, virtual key or mapped JWT key is accepted,
and the team member budget step uses the native budget decision.
A JWT email claim that is not a string reads as no email, the same as a
missing claim, instead of being stored as the number's text.
The native extension size cap rises from 55 MB to 65 MB, since the auth core
brings the Linux build to 56.85 MB.
* fix(anthropic): close /v1/messages thinking blocks with a signature and keep bridged output
* fix(anthropic): close rebuilt-stream thinking blocks with a signature_delta
* fix(anthropic): drop unsigned thinking before Claude, stream raw reasoning_text, close rebuilt thinking blocks with signature_delta
A Claude target on /v1/messages now gets unsigned thinking blocks left out of the request, so a switch from a reasoning_content model makes one call and keeps thinking, except when that would leave the final tool turn without its opening thinking block
Raw response.reasoning_text deltas from the Responses bridge stream as thinking, and a stream rebuilt from a full answer or rewritten by a guardrail closes each thinking block with a signature_delta, the real signature or an empty string
* fix(anthropic): stream a whole reasoning item as a closed thinking block
A Responses stream can send a reasoning item whole in output_item.added and output_item.done with no text deltas, as litellm's own stream does for gpt-oss on OpenRouter. The /v1/messages stream now builds that item's thinking block with the same helper the non-streamed response uses, so streamed and non-streamed answers carry the same reasoning
* test(anthropic): mark the new streaming test locals Final
* fix(anthropic): drop unsigned thinking before every Claude call, a pending tool turn included
Anthropic answers 400 to a thinking block with an empty signature but accepts a
tool turn with no thinking, so the strip no longer keeps the block that opens the
pending tool turn. The check now looks for "claude" in the model name, which covers
Claude behind every messages route and the chat adapter, not only four providers
* fix(anthropic): strip unsigned thinking when a deployment's base_model names Claude
* fix(anthropic): drop unsigned thinking for an alias whose base_model names Claude on the native messages path
The chat adapter already read base_model before leaving unsigned thinking out of a Claude request. The native messages transform and the Rust request transform now read it too, so an aliased Claude deployment gets one call instead of a 400 and a retry
* test(anthropic): build the pending tool turn without mutating it in the Rust strip test
The Rust test now builds its expected messages from the same helper instead of editing them afterward, and a doc comment that only restated a helper's name is gone
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend-logs): match key aliases exactly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(spend-logs): drop unused key alias ORM lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate exact key alias API descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): keep key alias filter matching CLI session logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): classify short asks in long conversations
* fix(complexity_router): validate active context limits without recursion
* test(rag): wait for keepalive frames before releasing retrieval
* test(ui): control availability debounce in context-limit regressions
* fix(auth): apply current team limits in the pre-body rate limit check
The check that refuses an exhausted key before reading the request body
used the team limits copied onto the cached key. Raising a team's rpm or
tpm limit with /team/update did not reach that copy, so a throttled key
stayed throttled until the key cache expired, and the full auth path that
does refresh team fields never ran.
The check now overlays the cached team's newer fields the same way full
auth does, through one shared helper. When the team is not in the cache,
as on a peer right after an invalidation, it defers to full auth instead
of judging the key on stale limits.
* fix(auth): keep key limits in the early check when the team is not cached
On a team cache miss the early check skipped every pre-body limit, so an
exhausted key read its body before being refused. Apply the key's own
limits and defer to full auth only when the cached team limit refuses.
* fix(mcp): keep an MCP server's own Authorization key when the JWT signer is on
With mcp_jwt_signer enabled, a server whose configured authentication_token is sent in
Authorization (bearer_token, token, authorization, basic) had that key replaced by the
signer's JWT on tools/list and tools/call, so the upstream returned 401. Both signer guards
now also treat the server's own key as occupying Authorization, using the same
to_server_spec mapping that sends the key upstream.
Resolves LIT-10782
Fixes#47988
Co-authored-by: Arvind Kumar <arvindkumar.acad@gmail.com>
* fix(mcp): keep a deferred oauth2-delegate server's own token when the JWT signer is on
Servers that defer to the v1 client, such as oauth2 with delegate_auth_to_upstream,
send their authentication_token in Authorization (or upstream_token_header), but the
signer guard only read the v2 spec and let the gateway JWT overwrite it. The guard
now falls back to v1's own header rule when to_server_spec returns None. BYOK keeps
its existing behavior.
* test(mcp): cover a SigV4 server's token leaving Authorization to the JWT signer
* fix(mcp): only let upstream_token_header claim Authorization for auth types v1 writes a key for
A SigV4 server with a stray authentication_token and upstream_token_header
set to Authorization lost the signer JWT, though v1 never sends that token.
The v1 rule now resolves the header v1 would write the key to, or none, before
applying the override.
* fix(mcp): keep the signer JWT on tools/call when the caller's auth header displaces the server key
A per-request MCP auth header replaces the server's authentication_token, so
the configured key is not what reaches Authorization. The tools/call guard now
only consults the server's configured key when the caller sent none, matching
the tools/list guard, so the upstream no longer ends up with no Authorization.
---------
Co-authored-by: Arvind Kumar <arvindkumar.acad@gmail.com>
* fix(mcp): let zero-budget users call MCP tools that cost nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): price a tool call made by its display name as the tool it runs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep budget checks for bare display names a paid server can run and for cut non-tool-call bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): budget check reserved tool names and display names that resolve on to a paid tool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): follow chained display names without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): price a scoped tool call on the servers it is scoped to and follow display names as execution does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep the budget check for MCP servers that use display names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): add Parallel Search to discovery registry
* fix(mcp): point the Parallel catalog entry at mcp.parallel.ai
Parallel's docs call search.parallel.ai/mcp legacy and say new installs should use mcp.parallel.ai/mcp, which keeps web_search and web_fetch and stays free without a key
* test(mcp): cite the Parallel docs beside the pinned catalog URL
---------
Co-authored-by: georgeatparallel <george@parallel.ai>
* feat(ui): show the usage spend total in another currency at ECB reference rates
Adds GET /get/reference_exchange_rates, which returns the current ECB
reference rates as USD per unit for every feed currency. It refreshes at
most once a day under a lock, keeps fresh rates when a refresh fails and
answers 503 instead of serving rates older than seven days.
The usage overview's spend card gets a Spend display selector. It asks for
rates only once opened and then shows an estimated amount with the rate date
under the unchanged USD total. Every other figure stays in USD.
* fix(ui): fall back to USD when spend display rates stop being usable
The selector now shows USD whenever no estimate can be shown, so a failed
refetch never leaves EUR selected beside a USD-only notice, and an opened
selector refreshes its rates hourly. The endpoint gets its HTTP client
inside the async fetch so the client belongs to the serving event loop.
* fix(proxy): import the ECB rates endpoint HTTP client helper at module level
Binds get_async_httpx_client at import so its type is known. The client is
still built inside the awaited fetch on the serving event loop.
* fix(proxy): fetch display rates with a short-lived handler closed after each refresh
Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, and drops the descriptive comment on the spend display component.
* chore(ui): regenerate API types for the reference exchange rates endpoint
* fix(proxy): use the cached async client factory for display rate refreshes
Direct AsyncHTTPHandler construction fails the async client check, so the
refresh fetch goes back to get_async_httpx_client, called inside the awaited
fetch, with a named type suppression for the factory's untyped params.
* fix(ui): tidy spend display helper, hook options and fallback test
Groups the rate snapshot into one estimatedSpend argument, names the rates
query options, names the missing-estimate condition, and waits for the
unavailable notice before asserting the USD fallback.
* chore(ui): drop a restating comment on the spend estimate helper
* test(proxy): prove concurrent first rate requests share one ECB fetch
* fix(proxy): wait a minute before retrying a failed display rate refresh
A failed ECB refresh left rates due, so every request after it waited on
the lock and fetched again, and N queued callers paid N timeouts during an
outage. The endpoint now skips refreshing for 60 seconds after a failed
attempt, serving held rates or a 503, and retries on the first request
after that. Concurrent waiters share the failed attempt. Also drops the
usd_rates docstring.
* fix(ui): retry unavailable currency estimates on reopen
* refactor(ui): name the currency selector open handler
* style(ui): format currency selector props
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache
Behind general_settings.prompt_cache_alerts, which is off by default, a
logger watches native Anthropic chat completions that send exactly one
explicit five-minute cache breakpoint. A cache write the provider reported
for that prefix proves it. Three later requests that each report zero
cache reads and writes, all started after the proof and within its five
minutes, save one alert for the deployment and configuration through the
existing alert repository, so a dismissed or resolved alert is never
reopened.
Only the provider's raw response counts. A missing or contradictory usage
is unknown, never zero. Streaming requests, failures and unknown usage end
the streak, and so does a caching request whose prefix can't be compared.
A read or a new write resets it. The prefix is a keyed hash, and the
configuration covers the model, endpoint and a keyed digest of the API key,
so nothing from the prompt or credentials is kept.
Streaks live in the proxy's shared Redis, one Lua script per update, and
requests are ordered by their own start and end so replays and overlaps
can't add misses. Without Redis nothing is tracked. A Redis or database
error never affects the request, and evidence lost to Redis ends its
scope's streak before anything else counts.
* fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration
Streamed attempts log an empty API key, so their evidence could not reach
the streak of the assessed configuration. Streams, failures, missing usage
and prefixes that can't be compared now reset every streak on their
deployment. Assessed traffic keeps its credential, model and endpoint
specific configuration.
After an alert is saved, its configuration stays quiet in Redis for a day,
so later misses don't write the same incident again. The quiet window is
only set once the save succeeds, and a failed save can trigger again.
Evidence lost to Redis now resets its deployment from a loop instead of
recursion, and a lost reset older than any proof is dropped. Asyncio
timeouts count as store failures on Python 3.10, and the module no longer
uses StrEnum. The proxy imports and registers the logger only when
prompt_cache_alerts is on, and removes it from the callback lists when
the lifespan ends.
* fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading
Registering prompt cache alerts had moved the asynccontextmanager decorator
from the proxy lifespan onto the new helper, which broke every proxy start.
The decorator is back on _proxy_lifespan, and a lifespan test now checks
that registration runs on entry and its cleanup on exit.
A config reload can leave the shared Redis getter empty for a moment.
Evidence seen then is remembered like evidence lost to a Redis error, so
its deployment is reset before anything else reaches Redis once it is back.
* test(alerts): let a failed completion raise its mapped error before asserting the deployment reset
* fix(alerts): describe prompt cache alerts as advisory and type their boundaries
Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view.
* fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof
A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry.
* fix(alerts): serialize each message's fields once in the prompt cache prefix
Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request.
* test(proxy): keep a leaked worker config out of the prompt cache lifespan test
An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs.
* test(alerts): clean each prompt cache deployment tag in its own comprehension
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(guardrails): record the guardrail verdict on a request dict that starts empty
The Bedrock guardrail and the shared `log_guardrail_information` wrapper both picked the request dict with `or {}`. An empty dict is falsy, so a caller that passed one had the verdict written to a throwaway dict. /apply_guardrail passes an empty dict for a request that carries no messages and no metadata, so nothing downstream could see what the guardrail decided. The verdict now goes to the dict the caller passed, and only a missing dict falls back to a new one.
* fix(guardrails): log requests that a guardrail blocks on /apply_guardrail
A request a guardrail blocked on /apply_guardrail was missing from the logs. The endpoint skipped the failure handlers for an HTTPException, the spend log failure hook wrote no row because the route is not an LLM API route, and the guardrail's verdict stayed on a request dict no logger reads.
The endpoint now runs the failure handlers for every exception and copies the verdict onto the logged request, so allowed and blocked requests both carry it along with the guardrail's cost. It marks the failure of a request it had set up with a sentinel, and the spend log failure hook writes its row for that mark. A request rejected before setup, such as one with a bad key or an unknown guardrail, is still not logged. Guardrail records a caller sends in metadata are dropped for both response formats so a verdict cannot be forged in the logs.
* fix(guardrails): keep Presidio and Gray Swan verdicts on empty request data
Presidio's text-only path and Gray Swan's API-failure path replaced an empty request dict with a throwaway one, so the verdict never reached the logged request. The apply_guardrail logging tests now run the real request setup and read what a logging callback receives.
* test(guardrails): cover the endpoint failure row, the sync decorator, and an unreachable Gray Swan without real sockets
* fix(logging): redact prompt text in guardrail records when message logging is off
* fix(logging): redact the failure payload when message logging is off
* fix(logging): keep guardrail details for other callbacks when one opts out of message logging
* fix(logging): redact failure events per callback and keep guardrail compression stats
A callback that turns off message logging now gets the same redacted copy on failure events as on success events, so a blocked /apply_guardrail call no longer hands it the raw prompt and verdict
Guardrail redaction keeps the numeric compression stats (tokens saved, compression ratio) that spend tracking reads, through one helper shared with the spend-log sanitizer
* fix(policy): test each batch input on its own copy of the request data
The guardrail logging decorator now records a verdict on an empty request dict instead of dropping it. The policy test endpoint shared one request dict across every input and merged it into the agent call, so each later input and agent request carried the earlier inputs' verdicts. Each input now gets its own copy, which keeps the caller's request data unchanged
* test(policy): look up the counting guardrail through the real registry
The batch isolation test now registers its guardrail as a litellm callback, so the endpoint finds it through the real guardrail registry instead of a mocked one, and its apply_guardrail override carries the same concrete types as the base class
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): convert per-request token prices from pricing_currency to USD
A trusted chat completion request can now send pricing_currency with
input_cost_per_token and/or output_cost_per_token. The proxy converts the
supplied prices to USD once at ECB reference rates in pre-call, right after
the existing pricing strip, and removes the field before routing. Requests
without the field, and USD, behave as before and make no network call.
The field sits behind the same two gates as request prices. Other routes,
invalid codes or prices, other pricing fields alongside it, and rates that
are unavailable, stale or out of range are rejected instead of being priced
as USD. Cold lookups share one bounded refresh and fresh rates never wait.
* test(proxy): type pricing currency tests and cover failed rate refreshes
Types the new pre-call tests, waits for the shared refresh to finish even
when the due-refresh test fails, and covers stale rates with a failed
refresh (503), a failed background refresh keeping fresh rates, and a
refresh that raises.
* test(proxy): restore an unrelated import line reformatted by mistake
* test(proxy): use the current chat model in request pricing fixtures
* fix(proxy): reject pricing_currency on generic pass-through requests
An authorized key keeps request prices on a generic pass-through, which
bills them as USD, while pricing_currency was forwarded upstream untouched.
The pass-through body boundary now rejects a non-null pricing_currency with
a 400 and drops a null one, matching the chat-only conversion.
* fix(proxy): type the request pricing conversion call and ECB client import
Validates the request body to a typed mapping only when pricing_currency is
present, converts the supplied prices from that view, and writes them back
onto the original request. Imports the async HTTP client helper at module
level so its binding is typed. The client is still built inside the fetch.
* test(proxy): cover a null pricing_currency through the pre-call pipeline
* fix(proxy): fetch ECB rates with a short-lived handler closed after each refresh
Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, instead of binding the untyped cached client factory.
* fix(proxy): wait a minute before retrying a failed ECB rate refresh
A failed refresh left rates due, so every later non-USD request started a new
ECB fetch during an outage. The shared lookup now reuses a failed refresh
for 60 seconds from its start, so requests in that window get the held
rates or a 503 without fetching again, and the next request after it
retries. It also goes back to the cached async client factory, with a named
type suppression for the factory's untyped params argument.
* fix(proxy): convert pricing_currency on every chat completions alias
The /engines/{model}/chat/completions and /openai/deployments/{model}/chat/completions
aliases reach the same chat handler but were rejected, because the check
compared the literal request path. It now compares the registered route
template, falling back to the path when no template is available. Also
covers a cancelled refresh not blocking the next lookup.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): rewrite batch JSONL rows to the deployment model on model-routed uploads
* fix(proxy): rewrite only alias rows and authorize them against the alias at batch create
* fix(proxy): keep batch create working when the credentials lookup names no model
* fix(proxy): spool the batch alias rewrite to disk and close it after upload
* fix(proxy): rewrite only alias rows in model-routed batch uploads and charge both project quotas
Rows that do not name the alias now reach the provider exactly as uploaded, and a file with no alias row is sent unchanged. Rewritten rows are compact UTF-8 JSON. Rows routed through the alias count against the project quota of both the alias and the model the row names
* fix(proxy): keep every byte of non-alias batch rows and skip files with nothing to rewrite
* fix(proxy): rewind batch uploads instead of calling seekable() so Python 3.10 works
* fix(vertex_ai): honor deployment and row drop_params for sampling params in batch file rows
* fix(batches): rewrite escaped alias rows and charge shared quota counters once
Parse each row's model to find rows naming the alias, so an alias written
with JSON escapes is rewritten too. Merge a row's alias and real model into
one project quota descriptor per counter, so a model_group_alias that puts
both in one group charges the row once against the lower limit.
* test(batches): prove the shared quota counter is charged once on the Redis store
* test(files): capture model-routed batch uploads at the provider HTTP boundary
* fix(batches): charge a shared project counter once per batch and keep rows JSON cannot re-encode
A batch file that mixes rows naming the proxy alias and rows naming the
deployment model built one project ITPM/OTPM descriptor per row group, so
the in-memory limiter checked each group against the quota on its own and
let the batch through. Descriptors now merge by counter with summed tokens,
and the 429 detail reports the merged total.
A row whose numbers JSON cannot write back, such as 1e400, was rewritten as
Infinity. It now stays byte-identical, the same as rows the rewrite skips.
* fix(batch_rate_limiter): hold a shared project counter to its strictest limit
When a batch file mixes rows naming the alias and rows naming a model in the
same group, both row groups charge one project ITPM/OTPM counter. The merged
descriptor kept the first group's limit, so a stricter limit set on the other
name was skipped. Merge them with the limiter's coalesce_rate_limit_descriptors,
which keeps the lowest limit, the same rule a single request already follows.
* chore(rust): restore generate_stdlib_fixtures.py as it is on main
An earlier merge commit reformatted this ruff-excluded fixture script. Restoring main's bytes keeps the PR to the batch alias fix
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve team aliases in realtime websocket routing
* test(proxy): cover translations and live sessions team alias routing
* test(proxy): cover a team alias that differs from the provider model on realtime routes
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): serve image n with n single-image requests
Gemini image models reject a candidate count above one on the images routes,
so n > 1 failed with a provider 400 on AI Studio and Vertex AI. The image
generation and edit handlers now ask the provider config how many identical
requests to send, and merge the returned images and token usage into one
response. Vertex AI Gemini image edits also accept n now.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): cap image fan-out and bill served images when one request fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): price each Gemini fan-out request on its own usage and give the merged response its own hidden params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Bill completed Gemini image calls when a generations fan-out request fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Expect the shared logging object in the image generation router call test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): keep counting and reporting priority requests after a priority pool's tokens run out
* fix(dynamic_rate_limiter): return the tracked priority statuses as a tuple
* fix(dynamic_rate_limiter): read optional rate limit fields without assuming they are set
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(token_counter): memoize token counts of long texts per tokenizer
* test(prompt_caching): count a prompt no other test has counted in the loop lag checks
* fix(token_counter): key the count memo on the string hash instead of blake2b
* test(router): embed distinct texts so the batches are counted in full
* test: time token counts over text no earlier count has memoized
* fix(logging): retain streaming success logging tasks until they finish
Streaming chat, A2A, Gemini/Vertex and the proxy's deferred chat stream
logging scheduled their success dispatch with a bare asyncio.create_task.
The loop only holds weak references to tasks, so a dispatch suspended on
an awaitable nothing else reaches could be garbage collected mid-flight
and its spend log never written. Route them through spawn_logging_task,
which keeps a strong reference until completion, logs failures and is
drained at proxy shutdown.
* test(logging): release parked logging callbacks on every exit path
Wrap the garbage collection regressions in a context manager that releases
the parked callback, waits for its task and drains the logging registry even
when the test fails, assert exactly one parked callback, and leave the
orphaned closure dispatch on its original schedule.
* test(logging): cancel a parked logging callback that never finishes
Cancel and await the parked task if it is still pending after release, fail
the test in that case, and leave the streaming cache write on its original
schedule since it hands off to an already tracked write without suspending.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Fourteen tests carried an unconditional skip and could never run. The unit
conftest deletes DATABASE_URL and REDIS_* before every test, so the prisma
and Redis ones would fail in tests/unit even without the marker. Each
behavior they claimed is already covered by mocked unit tests or by the
integration suite against real Postgres and Redis, and the gpt-4o token
counter test patched a function removed in #10409.
The Redis buffer drain test was the one with unique value, so it is
rewritten in test_redis_update_buffer.py against the fakeredis-backed
RedisCache fixture. It fills every queue a spend flush writes, drains the
buffer, and asserts no Redis key is left behind.
config-load now resolves every os.environ/ reference after the include merge
and before Config::from_value, porting ProxyConfig._check_for_os_environ_vars
with no secret manager. References are read through a SecretResolver the host
injects, with the config's environment_variables layered in front of that
resolver's environment. A deployment's api_key arrives resolved instead of as
literal text, and oidc/ references see the config's variables too
The resolver parses a name into a SecretReference once and routes it with a
single match (OIDC, manager or environment) instead of repeating prefix checks