Greptile review follow-ups on the paged end-user cache invalidation.
UserApiKeyCache keeps hashed token keys in a second in-memory partition, and
routes delete_cache / async_delete_cache there. It inherited the new batch
delete unchanged, so a budget cascade cleared the main partition and left the
key object sitting on its pre-reset spend. Override it the way
async_set_cache_pipeline already partitions its entries.
The spend counters and the management cache shared one exception handler, so a
Redis failure on the counters returned before the management cache was touched
at all. Each cache gets its own await and its own handler now.
A failed page read returned the same empty tuple that ends the walk normally,
so a truncated pass was reported as a complete one. The window is advanced by
then and no later tick comes back for the customers past that page, so the walk
now says it was cut short and the service log carries it.
A request for a configured model group that the router rejects before picking a deployment (all deployments in cooldown, no healthy deployment) never gets a custom_llm_provider in its logging kwargs. The spend log payload persisted an empty provider, the daily spend tables carried it through, and the Admin UI Usage page rendered those requests under unknown even though every model in the group has a provider
get_logging_payload now takes the proxy router and, when the logged provider is missing, infers it from the model group's deployments. It only attributes when every deployment in the group resolves to the same provider; mixed groups, unknown groups and a missing router leave the value empty as before. Explicitly logged providers keep precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Vertex AI file content retrieval downloaded the whole GCS object into memory
before responding, which made large batch output files (hundreds of MB, image
generation JSONL past 4 GiB) impractical to fetch through the proxy.
Add BaseLLMHTTPHandler.async_retrieve_file_content_streaming, an httpx
stream=True path that hands the byte iterator to the provider config through
the new BaseFilesConfig.transform_file_content_stream hook and closes the
response on completion, early close, and HTTP error. VertexAIFilesConfig peeks
at the first JSONL row: Generate Content batch output is converted to OpenAI
batch format one row at a time (content-length dropped since it changes),
embeddings output stays buffered so fanned-out rows can be regrouped, and
anything else passes through with the upstream content-type and content-length.
vertex_ai joins FILE_CONTENT_STREAMING_PROVIDERS, so the proxy returns a
StreamingResponse for it while OpenAI-compatible providers and the buffered
Vertex path are unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The agent-runtime branch of /bedrock/{endpoint} (agents, knowledgebases, flows,
retrieveAndGenerate, rerank, generateQuery, optimize-prompt) forwarded every
caller header to AWS next to the SigV4 signature, so a LiteLLM key presented in
x-api-key or x-litellm-api-key reached bedrock-agent-runtime verbatim. Build the
upstream header set explicitly: drop LiteLLM credential headers by name and any
authenticated secret by value, keep the remaining caller headers, and let the
signed headers win on collisions.
It drives the real CLI for several seconds. The edge stamps every
upstream call with PYTEST_CURRENT_TEST, a process-global that names
whichever test the worker is in when the call arrives rather than the one
that made it, so a test that holds a worker that long collects other
tests' in-flight calls. Build 234's key report credits this test with 20
Bedrock and 7 Anthropic misses, and it makes no provider call at all.
Those misattributed calls take the wrong test id into the cache key and
write recordings under it, so the test was polluting the shared corpus it
exists to protect.
Deselected unless E2E_CLI_DETERMINISM is set, the same opt-in shape the
managed-files, prompt-caching and redis-chaos markers already use. The
attribution bug itself is older than this branch and is reported, not
fixed here.
Interrupted Anthropic streams that die before message_delta were billed at
the message_start placeholder (any value above 1 was trusted) or at 0 when
the partial response was reasoning-only, because the token_counter fallback
only looked at visible text. Reset the placeholder whenever no finish_reason
or second usage event arrived, fold the already-counted reasoning tokens into
the fallback estimate, and drop the stale completion_tokens_details so cost
is computed from the recovered count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_PROXY_MaxBudgetLimiter re-checked spend:user:{id} against user_max_budget in
async_pre_call_hook without the zero-cost model exemption that
_user_max_budget_check applies in auth, so free models were rejected with
"Max budget limit reached." once a user was over budget. Auth already owns
this check, so the hook is deleted rather than taught the exemption again
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
update_team loaded the org without its budget row, so the org max_budget,
tpm_limit and rpm_limit checks silently passed. It now loads the budget the
same way /team/new does
tpm_limit is the first field in the team-admin allow-list registry. The team
settings tab gives a team admin a form with only the enabled fields and sends
only those on save, and UI Settings labels the checkbox the same way
The dashboard gated the team settings form on a role it guessed from the
is_* props, the members list and the org list. The org list is premium
gated and empty while loading, so a team admin who is also an org admin
was told team admins cannot edit, although /team/update accepts them as
an org admin
/team/info now returns caller_edit_access, resolved by the same helper
/team/update uses, and TeamInfo keys the form and the toast off that
field. The org list is only read for the organization dropdown now, and
general_settings is read through one validated accessor in both handlers
assert_never landed in typing in 3.11, so importing it from typing broke the
3.10 import smoke job and the py310 typing gate. Take it from typing_extensions
like the rest of the proxy does.
Also drop /team/update from test_neighbouring_team_routes_stay_closed. The route
is self-managed now, so the coarse gate admits the caller and update_team decides,
which the sibling test in the same file already asserts.
Claude-Session: https://claude.ai/code/session_018PUCupsaarVLJy4iDFx256
The persisted UI settings were read once at startup, so a proxy admin
flipping a runtime flag through PATCH /update/ui_settings only changed the
pod that served the request. Every other pod kept serving the old value
until it restarted.
add_deployment, the reload the scheduler runs every 30s, now re-reads the
row and applies the runtime flags before it takes the model reconcile lock,
so a change made through one pod reaches the rest within one reload
interval. The startup hook and the two settings endpoints share that helper
instead of each repeating the flag copy.
Claude-Session: https://claude.ai/code/session_018PUCupsaarVLJy4iDFx256
Team admins could never reach POST /team/update: the route gate answered
401 before the handler's team-admin branch ran. This moves /team/update
into the self-managed routes, resolves proxy admin, org admin or team
admin inside the handler, and filters team admins through a new
proxy-wide UI setting, team_admin_editable_team_fields. The setting is
an allow-list of team fields. Empty means team admins cannot edit team
settings and get a 403 pointing at the proxy admin, and changing a field
outside the list fails 403 naming that field. Only values that differ
from what is stored count, since the dashboard resends the whole form.
The registry of fields the setting accepts ships empty on purpose. Each
field lands in its own follow-up PR with its value diff and dashboard
wiring. The Admin UI gains a "Team admin editable fields" section under
Settings > UI and a toast on the team page while editing is disabled.
Refs LIT-5722
Claude-Session: https://claude.ai/code/session_01A6SkwJdfZUmkzfUkrEkqX8
Keeps the original CustomStreamWrapper so response headers and the
correlation-context cleanup in __del__ are untouched when a deployment
hook rewrites the converted response. Covers the early-return branches
for real provider streams and unmapped call types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The token exchange mint keeps reading the user row from the database, since JWT auth caches the user it creates before adding it to the JWT's team and a mint off that cached row refused the first exchange for a new user. Introspection and the refresh revalidation go back to the cache read, so a resource server calling /introspect per request pays no database read.
