scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-07 08:26:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 89

(docs) setting cache paras on proxy + Openai client

Browse source
This commit is contained in:
ishaan-jaff 2024-02-22 13:17:30 -08:00
parent ede57d4ac2
commit 580f9e3097
1 changed files with 16 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
docs/my-website/docs/proxy/caching.md
View file

@ -238,9 +238,11 @@ chat_completion = client.chat.completions.create(
}
],
model="gpt-3.5-turbo",
cache={
"no-cache": True # will not return a cached response
}
extra_body = { # OpenAI python accepts extra args in extra_body
cache: {
"no-cache": True # will not return a cached response
}
}
)
```
@ -264,9 +266,11 @@ chat_completion = client.chat.completions.create(
}
],
model="gpt-3.5-turbo",
cache={
"ttl": 600 # caches response for 10 minutes
}
extra_body = { # OpenAI python accepts extra args in extra_body
cache: {
"ttl": 600 # caches response for 10 minutes
}
}
)
```
@ -288,13 +292,15 @@ chat_completion = client.chat.completions.create(
}
],
model="gpt-3.5-turbo",
cache={
"s-maxage": 600 # only get responses cached within last 10 minutes
}
extra_body = { # OpenAI python accepts extra args in extra_body
cache: {
"s-maxage": 600 # only get responses cached within last 10 minutes
}
}
)
```
## Supported `cache_params`
## Supported `cache_params` on proxy config.yaml
```yaml
cache_params: