Mock proxy now serializes responses using snake_case keys (agent_id,
created_at, system_prompt, run_id, etc.) and reads request bodies as
snake_case so it matches the real backend that the SDK now talks to via
the new transform layer. Also update the status string literals to the
new SessionStatus and RunStatus values, and read the followup body as
{prompt: {text}}.
There is no per-run conversation endpoint on the backend; conversation
history is session-scoped. Callers should use SessionHandle.conversation()
instead. Also update TERMINAL_STATES to the new RunStatus values
(finished/cancelled/error).
The backend's followup endpoint expects a FollowupCreate payload with a
nested prompt object ({prompt: {text: ...}}), not a flat {message: ...}.
Public method signature followup(message: string) is unchanged - only
the wire body changes.
Backend speaks snake_case (Python idiom) while the SDK's public TS API
is camelCase. Add recursive snakeToCamel and camelToSnake helpers and
wire them into the HTTP layer so request bodies are camel->snake before
JSON.stringify and response JSON is snake->camel before being returned
to callers. Single-word keys like id/type/data/seq/status pass through
unchanged in both directions, and non-object values are not touched.
Backend (LIT-2890) emits provisioning/ready/busy/error/terminated for
session status and queued/running/finished/cancelled/error for run status.
Update the SDK type aliases so callers compare against the actual values
the proxy returns over the wire.
Greptile P2: a misbehaving or adversarial server returning
'Retry-After: 9999999' could stall the SDK indefinitely. Cap the
honored delay at MAX_RETRY_AFTER_MS (60s).
Greptile P1: the reconnects counter accumulated across the entire
stream lifetime — for a long-running stream with several transient
drops over hours, the budget would be exhausted even though every
individual reconnect succeeded. Now the counter tracks *consecutive*
failures: once a connection delivers at least one new event, the
counter resets to zero on the next drop, so only a sustained outage
trips sse_reconnect_exhausted.
Greptile P1: wait() polled indefinitely with no escape hatch — a stuck
or partitioned server would hang the caller forever. Now accepts
{ signal, timeoutMs, pollMs } and throws LiteLLMAgentError with codes
wait_aborted / wait_timeout. Forwards the signal to the underlying
requestJson call and to the inter-poll sleep.
Greptile P2: runFromInfo was exported but had no internal callers
(agent.ts and session.ts both build Run directly). Removed to shrink
the surface and avoid leaking resolveClient as a construction detail.
* fix(auth): pass team_id in member-level model access check
_check_team_member_model_access calls _can_object_call_model without
team_id, so access groups defined via model_info.access_groups cannot
resolve for team-scoped DB models (their internal router name is
model_name_<team>_<uuid>, not the public name). The team-level check
already passes team_id; this mirrors that.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(auth): add tests for member-level access group resolution with team_id
Eight tests covering _can_object_call_model and
_check_team_member_model_access with team-scoped DB models:
- access group resolves when team_id is passed
- access group fails without team_id (pre-fix behavior)
- literal model name still works with team_id (no regression)
- denied model still denied with team_id
- second model in group also reachable
- end-to-end member access via access group (mocked membership)
- end-to-end member denied for model not in allowed list
- no-override member inherits team-level check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* proxy: hot-reload config YAML when --reload is set
Uvicorn's --reload only watches *.py by default, so editing the
--config YAML did not restart the proxy. _get_reload_options() now
extends reload_dirs/reload_includes with the config file's directory
and basename when --config is provided.
* proxy: qualify reload_includes with absolute config path
Address Greptile review on PR #27274. When the --config file lives
outside cwd, reload_includes previously stored only the basename, which
meant uvicorn/watchfiles would also reload on edits to any same-named
file inside cwd. Use the absolute config path as the include pattern in
that case so only the actual proxy config triggers a restart.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy): use basename for reload_includes config pattern
Uvicorn's resolve_reload_patterns() calls pathlib.Path.glob(), which
raises NotImplementedError on absolute patterns (uvicorn discussion
2156). Passing config_abs (an absolute path) when the config file lived
outside cwd crashed startup under --reload. The config_dir is already
added to reload_dirs, so using just the basename as the include pattern
is sufficient to match the specific config file.
* fix: make it reload app when yaml changes
* style: remove unneeded comments
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Include model name + configured TPM/RPM in priority rate-limit 429 errors (#27215)
* Include model name + configured TPM/RPM in priority rate-limit 429 errors
The current 429 message ('Priority-based rate limit exceeded. Priority: prod,
Rate limit type: tokens, Remaining: -664145, Model saturation: 86.3%') doesn't
tell the operator which model was hit or what the configured limit is, so they
can't tell whether the priority allocation needs tuning or the model TPM is
just too small.
Add Model, Model TPM, and Model RPM to both the priority-based 429 and the
sibling Model-capacity 429 in dynamic_rate_limiter_v3._check_rate_limits.
Pure error-message change — no behavior or schema impact.
* test: assert priority 429 includes model name + configured TPM/RPM
Adds a regression test for the new fields in the priority-based 429 detail
('Model:', 'Model TPM:', 'Model RPM:'). Verified locally that the test
fails against the unpatched dynamic_rate_limiter_v3.py and passes after
the patch.
---------
Co-authored-by: shin-watcher <ext-agent-shin@berri.ai>
* Update litellm/proxy/hooks/dynamic_rate_limiter_v3.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/hooks/dynamic_rate_limiter_v3.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: shin-watcher <ext-agent-shin@berri.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* ci(circleci): enable Rerun Failed Tests for all pytest suites
Migrated every pytest-based CircleCI job that uploads JUnit results to use
'circleci tests run' instead of invoking pytest directly. This is the
prerequisite for CircleCI's 'Rerun failed tests' feature to be available
on each job in the pipeline.
For each job:
- Glob test files via 'circleci tests glob' and pipe them into
'circleci tests run --command="xargs ... pytest ..."' so the agent can
feed the failed-test subset on rerun.
- Preserve all original pytest flags (parallelism, timeouts, retries,
coverage, junit output paths).
- For jobs that previously lacked 'store_test_results' (proxy spend
accuracy, proxy_build_from_pip, db_migration_disable_update_check),
add the step so JUnit XML is uploaded and rerun is actually wired up.
- Replace the dynamic IGNORE_DIRS shell array in llm_translation_testing
with a 'grep -v' filter on the glob output, matching the previous
behavior of skipping tests/llm_translation/realtime.
- For 'build_and_test', glob 'tests/test_*.py' (top-level only) which
matches the prior 'tests/*.py' shell glob; the long list of
'--ignore=tests/<subdir>' flags was vestigial and is dropped.
Jobs already using 'circleci tests run' (local_testing_part1/2,
litellm_router_testing) are unchanged.
* fix(ci): convert classnames to file paths on rerun
CircleCI's Rerun Failed Tests sends each previously failed test as a
JUnit classname (e.g. 'tests.otel_tests.test_key_logging_callbacks'),
but pytest needs a file path. Without the awk preprocess step, rerun
runs fail with 'file or directory not found'.
Mirror the awk transform that local_testing_part1, local_testing_part2,
and litellm_router_testing already use, so rerun works in every job
that this PR migrated to 'circleci tests run'.
* ci: drop -x from OTEL pytest run so all failures are reported
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
