scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-12 23:01:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 88

(docs) update encode/decode doc

Browse source
This commit is contained in:
ishaan-jaff 2023-10-30 14:13:06 -07:00
parent c61fa70ba0
commit 47ac4fa58f
1 changed files with 9 additions and 30 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

39
docs/my-website/docs/completion/token_usage.md
View file

@ -31,21 +31,10 @@ Encoding has model-specific tokenizers for anthropic, cohere, llama2 and openai.
```python
from litellm import encode, decode
def test_encoding_and_decoding():
try:
sample_text = "Hellö World, this is my input string!"
# openai tokenizer
openai_tokens = token_counter(model="gpt-3.5-turbo", text=sample_text)
openai_text = encode(model="gpt-3.5-turbo", tokens=openai_tokens)
assert openai_text == sample_text
except:
pass
test_encoding_and_decoding()
sample_text = "Hellö World, this is my input string!"
# openai encoding + decoding
openai_tokens = encode(model="gpt-3.5-turbo", text=sample_text)
print(openai_tokens)
```
### 2. `decode`
@ -55,21 +44,11 @@ Decoding is supported for anthropic, cohere, llama2 and openai.
```python
from litellm import encode, decode
def test_encoding_and_decoding():
try:
sample_text = "Hellö World, this is my input string!"
# openai tokenizer
openai_tokens = token_counter(model="gpt-3.5-turbo", text=sample_text)
openai_text = decode(model="gpt-3.5-turbo", tokens=openai_tokens)
assert openai_text == sample_text
except:
pass
test_encoding_and_decoding()
sample_text = "Hellö World, this is my input string!"
# openai encoding + decoding
openai_tokens = encode(model="gpt-3.5-turbo", text=sample_text)
openai_text = decode(model="gpt-3.5-turbo", tokens=openai_tokens)
print(openai_text)
```
### 3. `token_counter`