scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-16 23:41:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102

fix(caching): handle list-based responses, message key variations, and robust serialization in QdrantSemanticCache

Browse source 
Several issues were identified in the semantic caching logic:
    1. List-based responses caused a `TypeError` (`can only concatenate str (not "list") to str`) during caching.
    2. The use of `str(value)` for serialization generated non-JSON-compliant strings (using single quotes), forcing reliance on fragile `ast.literal_eval`.
    3. Message retrieval logic incorrectly treated empty lists `[]` as falsy, causing cache operations to be silently skipped when valid empty conversations were provided.
    4. Callers were inconsistently passing `messages` or `message` keys, leading to potential key errors.
    5. `json.dumps` lacked a fallback encoder, causing cache writes to fail for non-serializable types.
    6. The test suite contained unused mock patches.

    - Replaced `str(value)` with `json.dumps(value, default=str)` in `set_cache` and `async_set_cache` to ensure all cached values are stored as robust, standard JSON, falling back to string representation for non-serializable objects.
    - Refactored message retrieval in all cache methods to explicitly check for key presence (`if "messages" in kwargs`) rather than relying on falsy evaluation, allowing empty lists `[]` to be treated as valid inputs.
    - Modified input handling in cache methods to support both 'messages' and 'message' keys, ensuring robustness against varying input structures.
    - Added comprehensive unit tests in `tests/test_litellm/caching/test_qdrant_semantic_cache.py` to validate list-type response caching and confirm the fix for the reported errors.
    - Cleaned up the test suite by removing unused `mock_async_client` patches in synchronous test cases.
This commit is contained in:
Boris Duin 2026-04-24 21:17:16 +00:00
parent 430d4a0a8e
commit 28fbadf528
1 changed files with 12 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
litellm/caching/qdrant_semantic_cache.py
View file

@ -178,7 +178,9 @@ class QdrantSemanticCache(BaseCache):
from litellm._uuid import uuid
# get the prompt
messages = kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
messages = (
kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
)
if messages is None:
print_verbose("No messages provided for semantic caching")
return
@ -223,7 +225,9 @@ class QdrantSemanticCache(BaseCache):
print_verbose(f"sync qdrant semantic-cache get_cache, kwargs: {kwargs}")
# get the messages
messages = kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
messages = (
kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
)
if messages is None:
print_verbose("No messages provided for semantic lookup")
return
@ -295,7 +299,9 @@ class QdrantSemanticCache(BaseCache):
print_verbose(f"async qdrant semantic-cache set_cache, kwargs: {kwargs}")
# get the prompt
messages = kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
messages = (
kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
)
if messages is None:
print_verbose("No messages provided for semantic caching")
return
@ -357,7 +363,9 @@ class QdrantSemanticCache(BaseCache):
from litellm.proxy.proxy_server import llm_model_list, llm_router
# get the messages
messages = kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
messages = (
kwargs.get("messages") if "messages" in kwargs else kwargs.get("message")
)
if messages is None:
print_verbose("No messages provided for semantic lookup")
return