Reproduces the original race deterministically: a session has both
an OLDER active run AND a NEWER terminal run. The buggy code path
used latest_run (newest by created_at) to decide whether to fall
through to the create-new-run branch — and since the newest is
terminal, it skipped the busy check and would have inserted a
duplicate run.
Three tests:
* 409 run_busy when an older active run exists.
* Happy path: empty session creates the first run.
* Happy path: latest run is active so /followup appends a
user_message event (unchanged by the busy guard).
Greptile P1 regression coverage (validation #16).
Exercise every state-mutating endpoint as PROXY_ADMIN_VIEW_ONLY and
confirm they all return 403:
* POST /v2/agents
* PATCH /v2/agents/{id}
* DELETE /v2/agents/{id}
* POST /v2/sessions
* DELETE /v2/sessions/{id}
* POST /v2/sessions/{id}/runs
* POST /v2/sessions/{id}/runs/{rid}/cancel
* POST /v2/sessions/{id}/followup
Plus a happy-path test confirming view-only admins still get
cross-tenant READ access (the whole point of the role).
Greptile P1 SECURITY regression coverage (validation #14).
Asserts:
* _get_signing_secret raises AgentJWTSecretNotConfiguredError when
LITELLM_AGENT_JWT_SECRET is unset, even with LITELLM_MASTER_KEY set
(no silent fallback).
* is_agent_jwt_secret_configured returns False for unset / empty,
True for any non-empty value.
* mint_daemon_token and decode_daemon_token both surface the same
error when the env var is missing.
Greptile P1 SECURITY regression coverage (validation #15).
When LITELLM_AGENT_JWT_SECRET is not set, the daemon JWT auth layer
cannot operate safely (no master-key fallback). Refuse to mount the
four /v2/agents+/v2/sessions routers in that case and log a clear
error pointing operators at the env var. Mounting them anyway would
expose an auth surface that can never validate a token and crash on
every request.
Greptile P1 SECURITY follow-up.
asyncio.get_event_loop() is deprecated inside a running coroutine
(Python 3.10+). Replace the two call sites in
daemon_get_next_queued_run with asyncio.get_running_loop().
Greptile P2.
Two fixes in this file:
1. asyncio.get_event_loop() is deprecated inside a running coroutine
(Python 3.10+). Replace both call sites in _stream_run_events with
asyncio.get_running_loop().
2. Call assert_caller_can_mutate on create_run and cancel_run so
view-only admins get 403 instead of bypassing ownership.
Greptile P2 (deprecation) + P1 SECURITY (view-only admin write bypass).
Two fixes in this file:
1. /followup new-run branch was bypassing the run-busy concurrency
guard. When latest_run was terminal-or-absent, the endpoint went
straight to litellm_agentrun.create without calling
_has_active_run. Two concurrent /followup calls on an idle session
both passed the latest_run.status check and both inserted runs,
breaking the 'one active run per session' invariant that POST /runs
enforces via 409 run_busy. Add the same _has_active_run check
before the fallthrough create, plus a defensive insert-time retry
that re-queries for active runs after IntegrityError and surfaces
409 run_busy if it lost the race.
2. Call assert_caller_can_mutate on create_session, delete_session,
and followup so view-only admins get 403 instead of bypassing
ownership.
Greptile P1 (concurrency) + P1 SECURITY (view-only admin write bypass).
Block PROXY_ADMIN_VIEW_ONLY from create_agent / update_agent /
delete_agent. Reads (GET) still pass through is_proxy_admin_read
so view-only admins keep cross-tenant visibility for the support UI.
Greptile P1 SECURITY follow-up (view-only admin write bypass).
is_proxy_admin previously returned True for both PROXY_ADMIN and
PROXY_ADMIN_VIEW_ONLY, letting view-only admins skip
assert_caller_owns_agent / assert_caller_owns_session on every write
endpoint and create / update / delete other tenants' agents,
sessions, and runs.
Split the helpers:
* is_proxy_admin: now full-admin only (used for write paths via the
fall-through to per-tenant ownership; view-only fails and gets 404).
* is_proxy_admin_read: full + view-only, used on read paths so the
support UI can still render any tenant's resources.
* assert_caller_can_mutate: explicit 403 guard for view-only on
every state-mutating endpoint.
The mutating endpoints in agent/session/run files call
assert_caller_can_mutate before any DB write — see follow-up commits.
Greptile P1 SECURITY (review #PRR_kwDOKALCgc78uM7F).
The daemon JWT secret must be a SEPARATE credential from the proxy
master key. The previous fallback to LITELLM_MASTER_KEY conflated
two distinct auth surfaces — a captured daemon JWT could be used
to mint regular API keys with master-key authority.
Replace _get_signing_secret with a strict check that raises
AgentJWTSecretNotConfiguredError if the dedicated env var is unset.
Add is_agent_jwt_secret_configured() so proxy_server.py can refuse
to mount the routers when the secret is missing.
Greptile P1 SECURITY (review #PRR_kwDOKALCgc78uM7F).
EC2 `ModifyImageAttribute` rejects "Character sets beyond ASCII are not
supported" when registering the AMI description. The whole AMI rolls back
on this error. Replace em-dash with hyphen.
LIT-2878
Built by `packer build` against the BYOC PoC account (us-west-2). Customers
running their own BYOC account should re-run the Packer build and replace
this value.
LIT-2878
`creds` was reassigned from `AwsCreds` to `Optional[AwsCreds]` in the
fallback branch — rename the second binding so mypy can narrow the type.
LIT-2878
Avoids the PytestUnknownMarkWarning when collecting
`tests/test_litellm/proxy/agent_session_endpoints/vm_providers/test_ec2_provider_real.py`
without `-m slow`.
LIT-2878
The cnf-update-db post-invoke hook fails (exit 100) when we install a
non-default python3 alongside, because cnf-update-db imports apt_pkg which
is bound to /usr/bin/python3 -> python3.12. Disabling the hook makes apt
update + install idempotent for AMI builds.
Also stop remapping /usr/bin/python3 via update-alternatives — it breaks
Ubuntu's python-coupled apt tooling. Tools that need 3.13 invoke it
explicitly; the systemd unit already uses /usr/bin/python3.13.
LIT-2878
Validations #3 (real boot), #11 (real BYOC fail-fast), and the real-cloud
piece of #8. Skipped by default; enable with `pytest -m slow` when the
`LITELLM_AGENT_AWS_*` + `LITELLM_TEST_*` env vars are set.
Each test wraps RunInstances in try/finally with TerminateInstances and
installs a 60-min process watchdog (per the AWS safety boundary) so a
hung test cannot leak an instance.
LIT-2878
Mocked Prisma client driving each sweeper through its happy path plus the
optimistic-lock branch (sweeper skips a session whose status changed
underneath it).
LIT-2878
Covers validation #11 (no creds = fail-fast, no instance launched), #12
(two teams resolve to two distinct creds objects), and the env-var fallback
path used in local dev before the LiteLLM_AgentVMConfig table is populated.
LIT-2878
Covers: prerequisites (Packer + AWS profile), `packer init` + `packer
build` invocation, sharing the AMI cross-account via `ami_users`, the
`LITELLM_AGENT_MODE` boot-mode contract, the Epic C migration path, and
the leak-cleanup one-liner that targets `LitellmManagedBy=agent-vm-provider`.
LIT-2878
Behaviour:
- reads runtime config from systemd's EnvironmentFile
- session mode: bootstrap → heartbeat every 30s, exit cleanly on HTTP 410
- warm mode: idle (real warm-pool hydrate lands in B2)
- redacts JWT in log output
Zero non-system deps (only `requests`, installed by the AMI builder).
Replaced wholesale by Epic C.
LIT-2878
Reads runtime config from /etc/litellm-agent/runtime.env (written by
EC2 user-data, mode 600). Restart=on-failure with a 5s backoff so transient
network blips during bootstrap don't permanently kill the session.
LIT-2878
Provisioner script driven by `litellm-agent-runtime.pkr.hcl`. Each tool is
pinned to a specific version and verified against a SHA-256 sidecar where
upstream provides one (uv) — see CLAUDE.md "CI Supply-Chain Safety".
The bun installer has no checksum sidecar, so we pin a version and pull the
artifact directly (not the install script).
LIT-2878
Builds an Ubuntu 24.04 AMI with node 24, python 3.13, git, gh, uv, bun, and
the agent-runtime systemd unit autostarted on boot. The daemon honours
`LITELLM_AGENT_MODE` from EC2 user-data: `session` for cold-boot,
`warm` for warm-pool prewarming (B2).
Uses IMDSv2 only. Tags every resource Packer creates for easy cleanup.
`ami_users` lets us share the AMI cross-account without rebuilding.
Validation #2 (`packer build`) covers this file.
LIT-2878
Documents the agent_settings YAML shape consumed by `get_vm_provider`.
B0's AWS resource IDs are referenced via `default_ami_id: ami-CHANGEME`;
the user fills in the real AMI after running `packer build`.
LIT-2878
Three sweepers run on the same 30s tick:
- bootstrap_timeout — sessions stuck in `provisioning` past the timeout
- heartbeat_timeout — `ready` sessions whose daemon stopped checking in
- max_session_minutes — sessions older than the configured ceiling
Each sweeper:
- bounds its batch to 100 rows so a backlog doesn't stall the loop
- re-fetches the row (optimistic lock) before terminating so multiple
proxy replicas don't double-terminate
- treats terminate failures as non-fatal (retry next tick)
Uses Prisma model methods (`find_many` / `find_unique` / `update`); no
raw SQL per project rules.
Validations covered: #7 (max_session_minutes), #9 (bootstrap_timeout), #10
(heartbeat_loss).
LIT-2878
One EC2 per session, launched in the team's AWS account using the team's
BYOC creds (decrypted at use, never logged). Spot first, on-demand fallback
when capacity unavailable. Per-session tags (litellm-session-id,
litellm-team-id, litellm-agent-id) for cleanup.
Safety:
- `set_stream_logger('botocore', WARNING)` so SigV4 payloads can't leak the
access key into proxy logs (regression-tested in #13)
- creds enter via ProvisionContext, never leave this module
- `_safe_aws_error` formats ClientError without echoing the request payload
- `InvalidClientTokenId` / `SignatureDoesNotMatch` → fail-fast InvalidCredentialsError
- terminate is idempotent on already-gone instances
Validations covered: #5 (spot fallback), #8 (terminate idempotent), #11
(invalid creds fail-fast), #13 (creds never logged).
LIT-2878
Reads the team's BYOC AWS creds from `LiteLLM_AgentVMConfig` (decrypts
each field individually) and falls back to `LITELLM_AGENT_AWS_*` env vars
for local dev. Raises `InvalidCredentialsError` if neither path yields
creds (validation #11 fail-fast).
Falls back gracefully when the table doesn't exist yet (Epic G hasn't
shipped its migration), so this code can land before LIT-2891.
LIT-2878
Reads `agent_settings.vm_provider` and `agent_settings.<provider>` from
the loaded proxy config and builds the matching provider. Defaults to
`noop`. Unknown values raise `ValueError` listing the supported
providers so config typos surface fast.
Validation #1 (test_factory) covers this path. Validation #6 (provider
swap is config-only) is also exercised here.
LIT-2878
In-memory `AgentVMProvider` used by the unit tests and as the default when
`agent_settings.vm_provider` is unset. The factory returns `NoopProvider`
when no AWS-backed provider is configured so the proxy boots cleanly without
AWS credentials.
LIT-2878
