mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-09 22:31:41 +00:00
(docs) lm harness
This commit is contained in:
parent
53abc31c27
commit
68e668df13
1 changed files with 25 additions and 0 deletions
|
|
@ -41,6 +41,31 @@ python3 -m lm_eval \
|
|||
|
||||
```
|
||||
|
||||
## FLASK - Fine-grained Language Model Evaluation Based on Alignment Skill Sets
|
||||
Use litellm to evaluate any LLM on FLASK https://github.com/kaistAI/FLASK
|
||||
|
||||
**Step 1: Start the local proxy**
|
||||
```shell
|
||||
$ litellm --model huggingface/bigcode/starcoder
|
||||
```
|
||||
|
||||
**Step 2: Set OpenAI API Base & Key**
|
||||
```shell
|
||||
$ export OPENAI_API_BASE=http://0.0.0.0:8000
|
||||
```
|
||||
|
||||
**Step 3 Run with FLASK**
|
||||
|
||||
```shell
|
||||
cd FLASK
|
||||
cd gpt_review
|
||||
```
|
||||
|
||||
Run the eval
|
||||
``shell
|
||||
python gpt4_eval.py -q '../evaluation_set/flask_evaluation.jsonl'
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### Making a test request to your proxy
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue