Vertex multi-region endpoints (e.g. us, eu) use the rep host pattern, not
{geo}-aiplatform.googleapis.com. Regional IDs still contain a hyphen.
common_utils.get_vertex_base_url centralizes the rule for SDK/API URL building.
Proxy pass-through duplicates the same branching in a local get_vertex_base_url
(with trailing slashes) to avoid importing from common_utils there; live
WebSocket passthrough uses the same multi-region host logic for wss://.
Tests cover us/eu for the common_utils helper.
Made-with: Cursor
Relative labels ("today", "in 2 days", "on May 12, 2026") mixed three
shapes in one column, breaking scannability. Always render MMM D, YYYY
for consistency and easier at-a-glance comparison across members.
Two changes:
1. workers: 8 -> 4 on every non-serial proxy-db shard. ubuntu-latest is a
4-core runner; -n 8 oversubscribes 2x and workers block each other
during their cold-start imports (pytest-cov instruments every litellm
module per worker). Measured ~441% CPU locally with -n 8 on 8 cores
(i.e. ~55% effective). Matching -n to physical cores should give
~2x faster worker startup, which is where most of the ~9m wall-clock
per shard goes (7+ minutes is plugin load + xdist imports before any
test runs).
2. Revert the -k split on test_proxy_utils.py. It was split into
proxy-utils-a-h / proxy-utils-i-z as a semantic-adjacent hack; merge
back to a single proxy-utils shard. Still uses --dist=worksteal so
xdist can balance the 188 parametrized cases across workers.
Also drops the now-unused `keyword` input from _test-unit-services-base.yml
and its matching matrix field across all proxy-db entries.
Shard count: 14 -> 13 (+ the assert-shard-coverage guard).
Sibling tests were mutating litellm.proxy.proxy_server.master_key and
prisma_client with raw setattr. Values leaked across tests in the same
xdist worker, flipping the auth short-circuit in user_api_key_auth and
causing unrelated tests (e.g. test_ui_view_session_spend_logs_pagination)
to return 401 instead of 200.
Replace raw setattr with monkeypatch in the two offending files and add
an autouse conftest fixture that snapshots/restores the known-leaky
module globals for every proxy test.
Default GHA matrix job names join every matrix field, producing unreadable
check labels like:
'proxy-db (logging-misc, tests/proxy_unit_tests/test_proxy_reject_logging.py
tests/proxy_unit_tests/test_audit_logs_proxy.py ..., 8, loadscope, "", 15)'
Set the job's display name to '${{ matrix.test-group }}' so each check
shows just 'logging-misc', 'proxy-utils-a-h', etc.
Previous run (13.8m total) was bottlenecked by shards with 9-12m wall-clock.
Setup + xdist spawn + coverage teardown is ~3m per shard, so each shard's
pytest runtime must stay under ~4m to fit inside 7m total.
Observed per-shard pytest times (before split):
db-and-spend 9:08 (170s outlier: test_aaaasschema_migration_check)
proxy-server 7:15
logging-and-callbacks 6:45
guardrails-budget-hooks 6:37
proxy-utils 6:23
auth-and-jwt 6:54
Split 6 shards into 12, keeping key-generation and endpoints-and-responses
(already <7m). Adds a `keyword` input to _test-unit-services-base.yml so
test_proxy_utils.py can be split by -k expression (same file, two runners).
New matrix entries:
auth-and-jwt -> auth-checks + jwt-and-keys
proxy-server -> proxy-server-core + proxy-runtime
logging-and-callbacks -> custom-logging + logging-misc
db-and-spend -> schema-migration (isolated 170s test) + db-and-spend
guardrails-budget-hooks-> guardrails-hooks + budgets
proxy-utils -> proxy-utils-a-h + proxy-utils-i-z (-k split)
The -k expression split is verified to cover every one of the 64 test
functions in test_proxy_utils.py exactly once. The assert-shard-coverage
guard still catches any file not in any shard.
PR feedback (greptile P1 / veria high): with the previous change, a team
storing mcp_tool_permissions={"my-alias": ["read_file"]} would pass the
server-access check (because the alias expanded to a concrete id in the
allowed-servers list) but the per-server tool lookup still did
dict.get(server_id) against the raw name-keyed dict — missing, returning
None, which callers treat as "no restrictions" → all tools allowed instead
of only the declared ones.
Add MCPServerManager.expand_tool_permissions() that rewrites the dict so
every key is a concrete server_id where possible (tool lists from keys
pointing at the same server are unioned). Unresolved keys pass through
unchanged so stale id-keyed restrictions still apply when the same string
is used for lookup. Wire the helper into the four dict-lookup sites:
get_allowed_tools_for_server (key + team paths), the agent tool lookup,
and the rest_endpoints.py tool filter.
Also switch expand_permission_list to pass through unresolved entries
(rather than dropping them) so existing test fixtures that use bare string
placeholders continue to work. The downstream access check denies unknown
entries when compared to the concrete request server_id, so security
posture is unchanged.
Sanitize the debug log to use %r formatting so an admin-controlled
identifier with newlines can't forge log entries (CodeQL log-injection
warning).
The create-branch job in create-release.yml calls the reusable
create-release-branch.yml workflow, which requires contents: write.
The top-level permissions: {} blocks the inherited default, and only
the release job overrode it, so the nested call failed with:
The nested job 'create-branch' is requesting 'contents: write',
but is only allowed 'contents: none'.
Add the permission at the calling job level so the reusable
workflow is granted what it needs.
The previous `while not self._queue.empty(): await self._queue.join()`
pattern skipped the join entirely when the worker had already dequeued a
task but not yet called task_done(). asyncio.Queue.join() tracks
_unfinished_tasks (incremented by put, decremented by task_done), not
queue depth, so it already handles that case on its own.
Two fixes to proxy-db CI:
1. test_realtime_webrtc_endpoints.py's `proxy_app` fixture mutated the
module-global `proxy_server.master_key` without restoring it, leaking
state into any test that shared the same xdist worker. Under
--dist=loadscope with 2 workers (GHA proxy-endpoints), this caused the
google_endpoints tests to fail with "No api key passed in." because
user_api_key_auth saw a set master_key and a missing API key on the
test request. The fixture now saves and restores the original value.
2. Address the Greptile note that the semantic shard design has no
catch-all, so a new test file added to tests/proxy_unit_tests/ without
a matrix entry would silently skip CI. Adds an assert-shard-coverage
job that enumerates test_*.py files and fails the workflow if any are
not referenced by a matrix entry, with a clear message telling the
author which semantic shard to place it in. All proxy-db shards now
depend on this guard.
Split into two related cleanups:
1. Delete CCI jobs that duplicate GHA coverage:
- mcp_testing (tests/mcp_tests) — already run by test-mcp.yml
- litellm_mapped_tests_proxy_part1/part2 (tests/test_litellm/proxy) —
already run across test-unit-proxy-auth.yml, test-unit-proxy-endpoints.yml,
and test-unit-proxy-infra.yml
Add rag_endpoints and realtime_endpoints to test-unit-proxy-endpoints.yml
(they were only covered by the deleted CCI part2 job).
Remove the corresponding workflow wiring, coverage combine entries, and
upload-coverage dependencies in .circleci/config.yml.
2. Re-shard test-unit-proxy-db.yml from 4 alphabetic buckets to 8 semantic
ones (auth-and-jwt, proxy-server, logging-and-callbacks, db-and-spend,
guardrails-budget-hooks, endpoints-and-responses, plus the existing
serial key-generation and test_proxy_utils.py shards). New test files are
placed in whichever group they belong to instead of reshuffling slices.
Add a dist input to _test-unit-services-base.yml so the test_proxy_utils.py
shard can use --dist=worksteal to spread its ~64 (many parametrized)
functions across workers; the default --dist=loadscope pins a single file
to a single worker, which was the root cause of that shard running 10m+.
The mocked async_increment_cache_pipeline is invoked from Router's
deployment_callback_on_success, registered as an async success callback.
Those callbacks are enqueued to GLOBAL_LOGGING_WORKER and run on a
background task, so the mock may not have been called yet when the test
asserts on it. Flush the worker before asserting.
Previous commit from greptile-apps added a new `when: always` teardown
step without removing the prior `name:`-only step, leaving a `- run`
block with no `command:` — CircleCI config validation rejects that.
Collapse back to a single teardown step that runs on success and
failure.
Two independent deflakes:
1. test_ui_view_spend_logs_unauthorized (unit) was returning 400 instead
of 401/403 when earlier tests in the file left proxy-auth globals
(prisma_client, master_key, user_custom_auth, general_settings,
user_api_key_cache) in a state that let invalid tokens pass auth and
fall through to the endpoint's own start_date/end_date validation.
Add an autouse fixture that pins those globals to their import-time
defaults for every test in the file. Harden the assertion to include
response body so future flakes are diagnosable.
2. test_basic_spend_accuracy (CI job proxy_spend_accuracy_tests) depends
on the Redis transaction buffer flushing spend to Postgres. The buffer
uses a single global pod-lock key (cronjob_lock:db_spend_update_job)
and a single global buffer list key. Pointing the proxy at the shared
remote Redis means concurrent CI pipelines contend for the same lock
and can drain each other's buffer into the wrong database. Add a
start_redis reusable command that boots a per-job redis:7-alpine
container (digest-pinned), and switch proxy_spend_accuracy_tests to
REDIS_HOST=host.docker.internal:6379 so lock and buffer state are
isolated per CI run.
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
* feat: add gpt-5.5 to model cost map
Add gpt-5.5 entry with pricing from OpenAI flagship page:
input $5/1M, cached input $0.50/1M, output $30/1M, 272K context.
* test: add gpt-5.5 coverage for model cost map and gpt-5 routing
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
The periodic budget-window reset job filtered keys/teams with
`where={"budget_limits": {"not": None}}`. The prisma-client-python
library does not support null-filtering on `Json?` columns (no
DbNull/JsonNull sentinel — upstream issue #714). The client drops the
`None` value during serialization and the engine rejects the query with
`MissingRequiredValueError: where.budget_limits.not: A value is
required but not set`, so neither the key nor team reset path runs.
Switch those two `find_many` calls to `query_raw` with
`WHERE budget_limits IS NOT NULL`, selecting only the PK and the
`budget_limits` column. Writes still go through the ORM. Add unit tests
covering the expired/unexpired paths for keys and teams, string-encoded
JSON payloads, empty payloads, error isolation between the two paths,
and a regression guard asserting the query still uses `IS NOT NULL`.
Extracts release branch creation into a separate reusable workflow
(create-release-branch.yml) that can be triggered independently via
workflow_dispatch or called from other workflows via workflow_call.
create-release.yml now dispatches it as a dependent job after the
release publishes, keeping both workflows decoupled.
token_counter.py: the previous size-limit raises were inside except Exception: pass,
so they were silently swallowed. The post-read raise was worse — img_data was already
assigned the full body before the raise, so the oversized value was used downstream.
Restructured to only assign img_data when the body is within bounds.
vertex_ai/common_utils.py and llm_passthrough_endpoints.py: the is-not-None guard
skipped validation for None, falling through to produce "https://None-aiplatform..."
Added explicit None check that raises before the regex guard.
team.object_permission.mcp_servers (and the per-key equivalent) previously
only accepted server_id strings. For config-loaded MCP servers, the id is
derived from a hash that includes the server URL, so the same logical
server in two regions ends up with two different ids in a shared database.
Permission lists had to enumerate every region's id.
Add a single MCPServerManager.expand_permission_list() helper that resolves
each entry against the current region's config + DB registry union: entries
that match a server_id pass through, entries that match an alias/server_name/
name expand to every matching id, and unresolved entries drop with a debug
log so stale or typo entries are diagnosable. Wire it into the four
_get_allowed_mcp_servers_for_* helpers so direct server entries and
mcp_tool_permissions dict keys are both expanded before the intersection.
Access-check outcomes are unchanged for existing id-based permissions;
name-based entries now resolve instead of being silently denied.
The rvm/install script sources scripts/functions/installer using
paths relative to the caller's working directory (not $0), so
invoking /tmp/rvm/install from /home/circleci/project fails with
'No such file or directory'. Switch to (cd /tmp/rvm && ./install).
Align the Ruby, Node.js, and npm install path with the rest of the
config. Three separate upstream installers were being invoked via
\`curl ... | bash\` or unlocked \`npm install\`:
- RVM's \`get.rvm.io/stable\` installer (mutable upstream script).
Replace with a shallow git clone of the rvm/rvm repo at tag 1.29.12
and verify HEAD matches the published commit SHA before running the
local \`./install\` script. Same pattern already used for the
helm-unittest plugin in .github/workflows/helm_unit_test.yml.
- NodeSource's \`deb.nodesource.com/setup_18.x\` piped into sudo bash.
Replace with a direct download of the Node.js 18.20.8 linux-x64
tarball from nodejs.org, verified against the published
SHASUMS256.txt digest before extraction.
- \`npm install @google-cloud/vertexai @google/generative-ai\` and
\`--save-dev jest\` resolved fresh from the npm registry on every
run. Add \`tests/pass_through_tests/package.json\` with pinned
direct-dep versions and commit the generated package-lock.json, then
switch CI to \`npm ci\` (exact lockfile install, fails on drift).
Also scopes the Ruby+JS test runners to \`tests/pass_through_tests/\`
so they pick up the committed package.json rather than writing
node_modules at repo root.
Stragglers from the 2026-04-21 Python 3.12 standardization:
- .github/workflows/check_duplicate_issues.yml (was 3.11)
- .github/workflows/llm-translation-testing.yml (was 3.11)
- .github/workflows/scan_duplicate_issues.yml (was 3.13)
- .circleci proxy_build_from_pip_tests (was 3.13)
The only intentional non-3.12 CI job is installing_litellm_on_python_3_13,
which exists as an explicit "latest supported Python" smoke matrix.
The Windows uv install step was piping a remote install.ps1 into
Invoke-Expression without any integrity check, while the Linux
install steps (install_uv command, line 89) download to a file,
verify SHA-256 against a hardcoded digest, and only then execute.
Bring the Windows path to the same pattern.
Also hardcode the kubectl v1.31.4 checksum in helm_chart_testing
instead of fetching kubectl.sha256 from the same origin as the
binary — if dl.k8s.io were ever to serve a tampered pair, a
co-hosted checksum provides no additional integrity.
The same branch filter block appeared 46 times in the workflow
declaration:
filters:
branches:
only:
- main
- /litellm_.*/
And the same pinned Python docker image appeared 29 times in jobs:
- image: cimg/python:3.12@sha256:9c796c...
auth:
username: ${DOCKERHUB_USERNAME}
password: ${DOCKERHUB_PASSWORD}
Replace with YAML anchors declared at first use:
- `&main_branches` on using_litellm_on_windows's filters block;
all other job entries reference it as `filters: *main_branches`.
- `&python312_image` on local_testing_part1's first docker image
entry; all other jobs reference `- *python312_image`, including
the multi-image jobs (auth_ui_unit_tests,
installing_litellm_on_python_v2_migration_resolver) which keep
their postgres sidecar entry inline afterwards.
Net result: one place to change when the image digest rolls or
the branch-filter convention changes. No behavior change — YAML
anchor resolution produces identical config at parse time.
Also adds Docker Hub auth block to upload-coverage (previously
pulled anonymously). No functional difference for a public
image, but avoids Docker Hub rate limits now that we reuse the
same entry.
Clean out copy-paste debug and workaround lines that serve no
purpose:
- `pwd && ls` echoes at the top of 30 "Run tests" steps (CCI
already logs working_directory on every step).
- "Show git commit hash" in local_testing_part1/part2 and
langfuse_logging_unit_tests (CCI shows the SHA in every job
header).
- "Verify Docker is available" stubs in 6 machine-executor
jobs (machine executors always have Docker).
- `sudo systemctl restart docker` in proxy_store_model_in_db_tests
(one-off workaround; not used anywhere else).
- Duplicated Black formatting step in local_testing_part1 and
local_testing_part2 — Black runs in the lint job, no reason to
run it again here.
- Second back-to-back `helm test litellm --logs` invocation in
helm_chart_testing (one call is enough).
No behavior change — these are all log-only or no-op steps.
Address codex review P1 + P2 findings:
- BYOK /token now accepts OAuth 2.1 clients that omit redirect_uri
(draft-15 §4.1.3 dropped the requirement). When the client does
submit a value, equality is still enforced vs the /authorize record.
PKCE + client_id binding cover the security role redirect_uri
played under RFC 6749.
- _user_id_from_session_cookie requires the ``exp`` claim on the UI
session JWT (PyJWT options={"require": ["exp"]}) so leaked cookies
have a bounded lifetime.
- validate_loopback_redirect_uri rejects URIs with a fragment
(RFC 6749 §3.1.2) and catches malformed-URI ValueError so
unparseable input surfaces as 400 invalid_request instead of 500.
The HTTPException arm in _run_centralized_common_checks assumed the
exception came from the team-object fetch, but asyncio.gather raises
the first exception from any of the five gathered coroutines. If
get_user_object / get_project_object / get_end_user_object raises
HTTPException on a token with team_id=None, the assert inside
_team_obj_from_token fires and the outer auth-exception handler
mishandles it.
Guard on team_id before calling _team_obj_from_token and default
team_object to None otherwise. (Greptile P1.)
Address codex review P0 + P1 findings on the discoverable OAuth proxy:
- /callback now re-validates that the decoded base_url is loopback before
302-redirecting to it. State is encrypted but pre-existing states minted
before the /authorize validation was added have no expiry and remain
valid; validating at the sink closes the open-redirect + code-theft
primitive for those stale states too. (VERIA-57 root cause B, P0.)
- /token responses now set Cache-Control: no-store + Pragma: no-cache
per RFC 6749 §5.1 (P1).
- Move TOKEN_NO_CACHE_HEADERS constant from byok_oauth_endpoints into
the shared oauth_utils module so both endpoints use the same value.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
