mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
docs(index.md): add key highlights to docs
This commit is contained in:
parent
bccab23a6a
commit
914ab00805
1 changed files with 28 additions and 1 deletions
|
|
@ -41,7 +41,16 @@ pip install litellm==1.68.0.post1
|
|||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Bedrock Vector Stores
|
||||
## Key Highlights
|
||||
|
||||
LiteLLM v1.68.0-stable will be live soon. Here are the key highlights of this release:
|
||||
|
||||
- **Bedrock Knowledge Base**: You can now call query your Bedrock Knowledge Base with all LiteLLM models via `/chat/completion` or `/responses` API.
|
||||
- **Rate Limits**: This release brings accurate rate limiting across multiple instances, reducing spillover to at most 10 additional requests in high traffic.
|
||||
- **Meta Llama API**: Added support for Meta Llama API [Get Started](https://docs.litellm.ai/docs/providers/meta_llama)
|
||||
- **LlamaFile**: Added support for LlamaFile [Get Started](https://docs.litellm.ai/docs/providers/llamafile)
|
||||
|
||||
## Bedrock Knowledge Base (Vector Store)
|
||||
|
||||
<Image img={require('../../img/release_notes/bedrock_kb.png')}/>
|
||||
<br/>
|
||||
|
|
@ -57,6 +66,24 @@ For the next release we plan on allowing you to set key, user, team, org permiss
|
|||
|
||||
[Read more here](https://docs.litellm.ai/docs/completion/knowledgebase)
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
This release brings accurate multi-instance rate limiting across keys/users/teams. Outlining key engineering changes below:
|
||||
|
||||
- **Change**: Instances now increment cache value instead of setting it. To avoid calling Redis on each request, this is synced every 0.01s.
|
||||
- **Accuracy**: In testing, we saw a maximum spill over from expected of 10 requests, in high traffic (100 RPS, 3 instances), vs. current 189 request spillover
|
||||
- **Performance**: Our load tests show this to reduce median response time by 100ms in high traffic
|
||||
|
||||
This is currently behind a feature flag, and we plan to have this be the default by next week. To enable this today, just add this environment variable:
|
||||
|
||||
```
|
||||
export LITELLM_RATE_LIMIT_ACCURACY=true
|
||||
```
|
||||
|
||||
[Read more here](../../docs/proxy/users#beta-multi-instance-rate-limiting)
|
||||
|
||||
|
||||
|
||||
## New Models / Updated Models
|
||||
- **Gemini ([VertexAI](https://docs.litellm.ai/docs/providers/vertex#usage-with-litellm-proxy-server) + [Google AI Studio](https://docs.litellm.ai/docs/providers/gemini))**
|
||||
- Handle more json schema - openapi schema conversion edge cases [PR](https://github.com/BerriAI/litellm/pull/10351)
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue