mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-15 04:19:08 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
7efce7f8e0
|
test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)
* test(integration): give chaos and timing tests a harness API and gate raw signals Chaos tests killed and suspended proxy workers with raw os.kill and psutil calls and bounded elapsed time with fixed literals, so they raced the uvicorn worker healthcheck and failed on slow CI boxes OwnedProxy now offers kill_worker and replacing_worker, which return only once a replacement answers readiness with its own pid, and suspended, which is refused unless the proxy was launched with a lenient worker healthcheck. timing.py bounds durations against a measured control or the injected delay. run.py measures the proxy import time once and derives readiness deadlines and the per-test timeout from it, writing boot-calibration.json check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time asserted under a numeric literal) for tests/integration outside _support, and every call site is migrated so both report zero with no suppressions * test(integration): steer the assistants wire burst onto both workers The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split * test(integration): restore product-enforced timing bounds lost in the migration Some TQ011 migrations dropped or loosened a bound whose time is the behaviour under test. The websocket session cap only kept its floor, so a cap closing at 300 s would have passed, and several timeouts were bounded by the injected stall instead of the configured product value assert_enforced_within bounds an elapsed time by the configured value plus explicit slack, with an optional floor. The websocket cap, the MCP validation timeout, the guardrail litellm_params timeout, the prisma command timeouts, the baseline projection lock timeout and the key metadata statement timeout use it. The device code refusal keeps its 15 s margin, and the langfuse retried prompt fetch is now bounded by a cold fetch of a steady prompt as well as by the ignored retry-after * fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat in a ping on the killed worker for up to that long before respawning it. It now uses the graceful stop budget, and a worker kill returns only after the victim is reaped, so pooled connections no longer race the dead worker's socket. Several tests on main picked the multiprocessing resource tracker as their victim and never killed a worker. Now that they kill a real one, presidio counts streams cut by the death, delete_model waits for the config reload that removes a deleted model from a replacement worker, and the wedged coordination Redis test kills before the wedge and waits for every parked publish to fail before resuming Redis. The redis cluster test imports the harness through the tests package so it collects in the unit job, and the per-model fallback rig exposes the owned proxy it now needs. * test(observability): only require S3 deliveries the killed worker could not have buffered A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
56bafc29cf
|
feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305)
* feat(gemini): accept file content blocks with video_metadata on multimodal embeddings * test(gemini): annotate the embedding mock helper and keep the type comment short * fix(gemini-embeddings): keep file blocks through a partial embedding cache hit and validate video_metadata strictly * test(caching): type the partial-hit cache test and pin its clock * fix(gemini-embeddings): accept the Files API URI /v1/files returns as a file block's file_id * fix(gemini-embeddings): 400 on bad input shapes, drop unknown block keys under drop_params, cache multi-block inputs A bare object `input` answered 500 from both the caching handler and the transformation; both now answer 400. An unresolved `files/` reference on vertex_ai/ answered a ValueError 500; it now answers 400 naming the gemini/ provider. An empty `format` passed through to the provider; it now answers 400 naming file.format. Unknown block keys (`detail`, an unknown video_metadata key, a top-level block key) are dropped under drop_params, global or per-request, and still answer 400 without it. The embedding cache counted file blocks against `max_messages`, so a request with 5 or more blocks was never cached; file blocks no longer count. * fix(caching): answer 400 for an object embedding input on the cache lookup * test(gemini-embeddings): annotate the new embedding tests with return types and Final * test(gemini-embeddings): annotate the transformation and files batch tests with return types and Final * test(gemini-embeddings): audit file content blocks on the wire, in the cache, in batch uploads and under chaos Integration cells for gemini/ and vertex_ai/ embedding file blocks with video_metadata: the batchEmbedContents and embedContent wire shapes, format overrides, nested and repeated inputs, every malformed block answering 400 before any provider call, drop_params at the request, deployment and YAML levels, the OpenAI SDK sync and async clients, Redis cache fills, hits, partial hits and the metadata in the key with zero-spend hit rows, chat, responses and messages controls, the vertex_ai/ batch JSONL upload, a mixed fast/slow/malformed/dropped burst and a worker kill. The wire peer now reads chunked request bodies, which the streamed GCS media upload sends * test(integration): treat an upload the client abandons before its terminating chunk as a disconnect, never a stored request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
77fc3315e5
|
fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts (#43878)
* fix(caching): keep tool calls and tool results in semantic cache prompts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): keep semantic tool prompt helpers within lint budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): keep structured function_call_output text in semantic prompts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): split Responses text-field collection to stay within complexity budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): tag each tool result with the position of the call it answers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): encode tool result position and output together so tool text cannot forge result tags Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): expect encoded tool result record in qdrant semantic prompt parity case Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): cover tool result arrangements, SDK clients, concurrency and qdrant outage for semantic cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(caching): embed every semantic cache prompt field except volatile ones Replace the per-shape allowlist in the Python and Rust semantic cache prompt walkers with one include-by-default walker. Plain text keeps its old concatenation; any other block or message is embedded as compact JSON with call ids mapped to ordinals, cache_control dropped, and signatures, encrypted content and base64 data replaced with a short sha256 digest. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(code-quality): allow the bounded semantic cache prompt walkers in the recursion check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust): expect structured JSON for unknown fields in redis and valkey semantic prompts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "test(rust): expect structured JSON for unknown fields in redis and valkey semantic prompts" This reverts commit |
||
|
devin-ai-integration[bot]
|
abd3af422d
|
fix(caching): never store or serve a chat completion with no choices (#44709)
* fix(caching): never store or serve a chat completion with no choices A provider response with empty choices was written to the response cache and served on every identical request until the TTL ended, with no provider call in between. The cache now skips storing such a response and treats an already stored one as a miss, so the next request goes back to the provider and its answer replaces the entry. * fix(caching): skip responses with no output on the Responses API and Anthropic Messages too * fix(caching): skip streams and stored entries that carry no output A chat or text completion stream whose chunks carried no choice is closed by the stream wrapper with one empty choice of its own, so the assembled response passed the choices check and was cached. The assembled stream is now judged on its content: a stream with no text, tool call, or other output in any choice is never stored, on the async and sync writers alike. The Responses API stream writer and the Anthropic Messages stream writer apply the same no-output check before storing. A stored entry with no output read through the worker memory tier is now evicted from that tier on the miss, so the next read reaches Redis where the refill lands; the text completion and messages writers only write to Redis, and the memory copy otherwise kept missing until its own TTL. * test(integration): response cache cells for answers without output Deterministic cells for the response cache on every unified endpoint, streamed and not, through the OpenAI and Anthropic SDKs and raw httpx, plus the sync SDK paths, stale entries, malformed answers, per-request TTLs, cache delete, and chaos (Redis stopped or paused mid burst, a worker killed, in-memory cache mode). The scripted upstream counts only POSTs as deployment calls, since the proxy's boot-time GET /v1/models discovery of a config deployment is not one. * test(caching): pin the stored entry timestamp in the worker-copy test * test(integration): drop the restating comments from the chaos cells * test(integration): close the breaker on the first call after the Redis restart --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |