* test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy (#47152)
* test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy
These two files register deployments with their own model settings on the
gateway. They now boot a module-scoped proxy on a scratch database, so their
setup cannot affect other tests that share the worker's proxy
* test(integration): seed the isolated gateway with the same router settings as the worker gateway
(cherry picked from commit 603ba6fe00)
* test(e2e): enable prometheus_system on the redis chaos proxy config (#47160)
#46484 moved the litellm_redis_circuit_breaker_* metrics behind service_callback prometheus_system, so the chaos proxy, which only set callbacks prometheus, stopped exposing them and the breaker assertion read 0
(cherry picked from commit 519906e6c2)
* fix(tests): wait for every thread of a killed worker before treating it as gone (#47148)
WorkerKill returned once the victim's main thread was a zombie. On Linux the
process keeps its sockets open until its last thread exits, so a pooled
keep-alive connection to the killed worker still looked idle and the next
request through it got ECONNRESET. A zombie now counts as gone only once
num_threads() is at most 1. Where psutil cannot read threads of a zombie
(macOS) it raises ZombieProcess, a NoSuchProcess, so the old behavior holds
(cherry picked from commit 0431e75716)
* test(caching): assert sync cache-hit logging without depending on thread order (#47093)
Sync success callbacks run on a thread pool, so the miss and the two
hits can reach the logger in any order. Compare counts instead of the
list order.
(cherry picked from commit 77c2ac18bd)
* test(e2e): hold failing-request RSS to a loose ceiling over two measured batches (#47170)
The 48 MB budget on the third batch kept failing on the x86 release gate with no
leak behind it. Isolated rc/1.107.0 and v1.105.1 stacks both grew about 60 MB on
the third batch with gc-tracked objects flat, and both levelled off to a couple of
MB per batch after twenty batches. Measure batches 3-4 against a 200 MB per-worker
ceiling instead, which a native leak of 400 KB per failed request still trips. The
stored-request size check stays the deterministic catch for the v1.100.0 leak
(cherry picked from commit e0514d92b8)
* fix(router): read an application profile's base model from metadata on a cache hit (#47144)
* fix(router): read an application profile's base model from metadata on a cache hit
The 1h prompt-cache pin check read base_model only from the top level of
litellm_params. On a response-cache hit the params it sees carry the base
model only under metadata.model_info, so an application-profile ARN
resolved to itself and the pin was written for 5 minutes while the wire
carried a 1h TTL. Read it through get_base_model_from_metadata, which
also checks metadata.model_info
* test(router): pin top-level base_model precedence over metadata model_info
(cherry picked from commit 1e3641a52a)
ab58c55d4b