JWT auth caches the user row before it adds the user to the JWT's team, and admission checks the credential's team against the cached row on whichever worker takes the next request. On a two-worker gateway the credential minted for a newly joined team answered 403 "not in your team memberships" until the management-object TTL ran out, because /team/member_add only evicted the membership spend sentinel. The add now evicts the added members' cached user rows and broadcasts the eviction to the other workers, the way /team/member_delete already did
The mint test now also covers a user SCIM deactivated after the cache last saw them active: the database read refuses the mint while the cached row still says active
The code-quality check refuses recursive functions and the walk that inspected
extras was one, so the filter no longer walks anything itself. safe_dumps now
builds its JSON-native structure through safe_json_structure, the filter scrubs
the extra through that, and the original object is kept only when the scrubbed
copy compares equal to it. Anything the serializer skipped (non-string keys,
nests past its depth, fields a repr hides) makes the copy differ, so the copy
wins. A host object whose equality raises, as numpy arrays and torch tensors
do, counts as changed instead of breaking the caller's log call
UserApiKeyCache's batch delete ran the two partitions in sequence, so a Redis
failure on the hashed token partition returned before the ordinary management
keys were touched. Both partitions are attempted now and the first failure is
re-raised for the caller to report.
The customer walk kept its position in two locals it reassigned each page. It
now mirrors the window walk in the same file: a page helper returns where the
walk goes next, and the driver rebinds one value.
Team admins are refused until a proxy admin enables a field, then limited to the enabled fields, and resending unchanged budget settings keeps the team's budget reset times
Greptile review follow-ups on the paged end-user cache invalidation.
UserApiKeyCache keeps hashed token keys in a second in-memory partition, and
routes delete_cache / async_delete_cache there. It inherited the new batch
delete unchanged, so a budget cascade cleared the main partition and left the
key object sitting on its pre-reset spend. Override it the way
async_set_cache_pipeline already partitions its entries.
The spend counters and the management cache shared one exception handler, so a
Redis failure on the counters returned before the management cache was touched
at all. Each cache gets its own await and its own handler now.
A failed page read returned the same empty tuple that ends the walk normally,
so a truncated pass was reported as a complete one. The window is advanced by
then and no later tick comes back for the customers past that page, so the walk
now says it was cut short and the service log carries it.
A request for a configured model group that the router rejects before picking a deployment (all deployments in cooldown, no healthy deployment) never gets a custom_llm_provider in its logging kwargs. The spend log payload persisted an empty provider, the daily spend tables carried it through, and the Admin UI Usage page rendered those requests under unknown even though every model in the group has a provider
get_logging_payload now takes the proxy router and, when the logged provider is missing, infers it from the model group's deployments. It only attributes when every deployment in the group resolves to the same provider; mixed groups, unknown groups and a missing router leave the value empty as before. Explicitly logged providers keep precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Vertex AI file content retrieval downloaded the whole GCS object into memory
before responding, which made large batch output files (hundreds of MB, image
generation JSONL past 4 GiB) impractical to fetch through the proxy.
Add BaseLLMHTTPHandler.async_retrieve_file_content_streaming, an httpx
stream=True path that hands the byte iterator to the provider config through
the new BaseFilesConfig.transform_file_content_stream hook and closes the
response on completion, early close, and HTTP error. VertexAIFilesConfig peeks
at the first JSONL row: Generate Content batch output is converted to OpenAI
batch format one row at a time (content-length dropped since it changes),
embeddings output stays buffered so fanned-out rows can be regrouped, and
anything else passes through with the upstream content-type and content-length.
vertex_ai joins FILE_CONTENT_STREAMING_PROVIDERS, so the proxy returns a
StreamingResponse for it while OpenAI-compatible providers and the buffered
Vertex path are unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The agent-runtime branch of /bedrock/{endpoint} (agents, knowledgebases, flows,
retrieveAndGenerate, rerank, generateQuery, optimize-prompt) forwarded every
caller header to AWS next to the SigV4 signature, so a LiteLLM key presented in
x-api-key or x-litellm-api-key reached bedrock-agent-runtime verbatim. Build the
upstream header set explicitly: drop LiteLLM credential headers by name and any
authenticated secret by value, keep the remaining caller headers, and let the
signed headers win on collisions.
It drives the real CLI for several seconds. The edge stamps every
upstream call with PYTEST_CURRENT_TEST, a process-global that names
whichever test the worker is in when the call arrives rather than the one
that made it, so a test that holds a worker that long collects other
tests' in-flight calls. Build 234's key report credits this test with 20
Bedrock and 7 Anthropic misses, and it makes no provider call at all.
Those misattributed calls take the wrong test id into the cache key and
write recordings under it, so the test was polluting the shared corpus it
exists to protect.
Deselected unless E2E_CLI_DETERMINISM is set, the same opt-in shape the
managed-files, prompt-caching and redis-chaos markers already use. The
attribution bug itself is older than this branch and is reported, not
fixed here.
Interrupted Anthropic streams that die before message_delta were billed at
the message_start placeholder (any value above 1 was trusted) or at 0 when
the partial response was reasoning-only, because the token_counter fallback
only looked at visible text. Reset the placeholder whenever no finish_reason
or second usage event arrived, fold the already-counted reasoning tokens into
the fallback estimate, and drop the stale completion_tokens_details so cost
is computed from the recovered count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_PROXY_MaxBudgetLimiter re-checked spend:user:{id} against user_max_budget in
async_pre_call_hook without the zero-cost model exemption that
_user_max_budget_check applies in auth, so free models were rejected with
"Max budget limit reached." once a user was over budget. Auth already owns
this check, so the hook is deleted rather than taught the exemption again
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
update_team loaded the org without its budget row, so the org max_budget,
tpm_limit and rpm_limit checks silently passed. It now loads the budget the
same way /team/new does
tpm_limit is the first field in the team-admin allow-list registry. The team
settings tab gives a team admin a form with only the enabled fields and sends
only those on save, and UI Settings labels the checkbox the same way
The dashboard gated the team settings form on a role it guessed from the
is_* props, the members list and the org list. The org list is premium
gated and empty while loading, so a team admin who is also an org admin
was told team admins cannot edit, although /team/update accepts them as
an org admin
/team/info now returns caller_edit_access, resolved by the same helper
/team/update uses, and TeamInfo keys the form and the toast off that
field. The org list is only read for the organization dropdown now, and
general_settings is read through one validated accessor in both handlers
assert_never landed in typing in 3.11, so importing it from typing broke the
3.10 import smoke job and the py310 typing gate. Take it from typing_extensions
like the rest of the proxy does.
Also drop /team/update from test_neighbouring_team_routes_stay_closed. The route
is self-managed now, so the coarse gate admits the caller and update_team decides,
which the sibling test in the same file already asserts.
Claude-Session: https://claude.ai/code/session_018PUCupsaarVLJy4iDFx256
