> For the complete documentation index, see [llms.txt](https://evalverse.gitbook.io/evalverse-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://evalverse.gitbook.io/evalverse-docs/documents/evaluation.md).

# Evaluation

## Common Arguments

These arguments are common across all benchmarks. Note that you must select at least one benchmark from the list provided in [this section](#arguments-for-each-benchmark).

<table><thead><tr><th width="190">args</th><th>default</th><th>description</th></tr></thead><tbody><tr><td>ckpt_path</td><td>upstage/SOLAR-10.7B-Instruct-v1.0</td><td>Model name or ckpt_path to be evaluated</td></tr><tr><td>output_path</td><td>./results</td><td>Path to save evaluation results</td></tr><tr><td>model_name</td><td>SOLAR-10.7B-Instruct-v1.0</td><td>Model name used in saving eval results</td></tr><tr><td>use_fast_tokenizer</td><td>False</td><td>Flag to use fast tokenizer</td></tr><tr><td>use_flash_attention_2</td><td>False</td><td>Flag to use flash attention 2 (highly suggested)</td></tr></tbody></table>

### Example

> By running the code below, <mark style="color:blue;">`h6_en`</mark>, <mark style="color:blue;">`mt_bench`</mark>, <mark style="color:blue;">`ifeval`</mark>, <mark style="color:blue;">`eq_bench`</mark> benchmarks will execute.

```
python3 evaluator.py \
    --h6_en \
    --data_parallel 8 \
    --mt_bench \
    --num_gpus_total 8 \
    --parallel_api 4 \
    --ifeval \
    --eq_bench \
    --devices 0,1,2,3,4,5,6,7 \
    --ckpt_path upstage/SOLAR-10.7B-Instruct-v1.0
```

## Arguments for Each Benchmark

{% tabs %}
{% tab title="H6" %}
Evaluate a LLM with the benchmarks in Open LLM Leaderbaord

<table><thead><tr><th width="167">args</th><th width="86">default</th><th>description</th></tr></thead><tbody><tr><td>h6_en</td><td>False</td><td>ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K </td></tr><tr><td>use_vllm</td><td>False</td><td>Inference with vLLM</td></tr><tr><td>model_parallel</td><td>1</td><td>Size of model_parallel</td></tr><tr><td>data_parallel</td><td>1</td><td>Size of data_parallel</td></tr></tbody></table>

## Example

#### Library

```
import evalverse as ev

evaluator = ev.Evaluator()
evaluator.run(
    model="upstage/SOLAR-10.7B-Instruct-v1.0",
    benchmark="h6_en",
    data_parallel=8,
    output_path="./results"
)
```

#### CLI

<pre><code><strong>python3 evaluator.py \
</strong>    --ckpt_path upstage/SOLAR-10.7B-Instruct-v1.0 \
    --h6_en \
    --data_parallel 8 \
    --output_path ./results
</code></pre>

## Result

```
📦results
 ┗ 📂SOLAR-10.7B-Instruct-v1.0
   ┗ 📂h6_en
     ┣ 📜arc_challenge_25.json
     ┣ 📜gsm8k_5.json
     ┣ 📜hellaswag_10.json
     ┣ 📜mmlu_5.json
     ┣ 📜truthfulqa_mc2_0.json
     ┗ 📜winogrande_5.json
```

{% endtab %}

{% tab title="MT-bench" %}
To run this `mt_bench` benchmark, OpenAI API Key in `.env` file is required.

| args                  | default | description                                        |
| --------------------- | ------- | -------------------------------------------------- |
| mt\_bench             | False   | Evaluate a LLM with MT-Bench                       |
| baselines             | None    | The baseline LLMs to compare                       |
| judge\_model          | gpt-4   | The model used for judge                           |
| num\_gpus\_per\_model | 1       | The number of GPUs per model (for gen\_answer)     |
| num\_gpus\_total      | 1       | The total number of GPUs (for gen\_answer)         |
| parallel\_api         | 1       | The number of concurrent API calls (for gen\_judge |

## Example

#### Library

```
import evalverse as ev

evaluator = ev.Evaluator()
evaluator.run(
    model="upstage/SOLAR-10.7B-Instruct-v1.0",
    benchmark="mt_bench",
    num_gpus_total=8,
    parallel_api=4,
    output_path="./results"
)
```

#### CLI

<pre><code><strong>python3 evaluator.py \
</strong>    --ckpt_path upstage/SOLAR-10.7B-Instruct-v1.0 \
    --mt_bench \
    --num_gpus_total 8 \
    --parallel_api 4 \
    --output_path ./results
</code></pre>

## Result

```
📦results
 ┗ 📂SOLAR-10.7B-Instruct-v1.0
   ┗ 📂mt_bench
     ┣ 📂model_answer
     ┃ ┗ 📜SOLAR-10.7B-Instruct-v1.0.jsonl
     ┣ 📂model_judgment
     ┃ ┗ 📜gpt-4_single.jsonl
     ┗ 📜scores.txt
```

{% endtab %}

{% tab title="IFEval" %}
Evaluate a LLM with Instruction Following Eval

<table><thead><tr><th width="183">args</th><th width="86">default</th><th>description</th></tr></thead><tbody><tr><td>ifeval</td><td>False</td><td>Evaluate a LLM with Instruction Following Eval</td></tr><tr><td>gpu_per_inst_eval</td><td>1</td><td>Num GPUs per ifeval process. Set to > 1 for larger models. Keep <code>len(devices) / gpu_per_inst_eval = 1</code> for repeatable results</td></tr></tbody></table>

## Example

#### Library

```
import evalverse as ev

evaluator = ev.Evaluator()
evaluator.run(
    model="upstage/SOLAR-10.7B-Instruct-v1.0",
    benchmark="ifeval",
    devices="0,1,2,3,4,5,6,7",
    output_path="./results"
)
```

#### CLI

```
python3 evaluator.py \
    --ckpt_path upstage/SOLAR-10.7B-Instruct-v1.0 \
    --ifeval \
    --devices 0,1,2,3,4,5,6,7 \
    --output_path ./results
```

## Result

```
📦results
 ┗ 📂SOLAR-10.7B-Instruct-v1.0
   ┗ 📂ifeval
     ┣ 📜eval_results_loose.jsonl
     ┣ 📜eval_results_strict.jsonl
     ┣ 📜output.jsonl
     ┗ 📜scores.txt
```

{% endtab %}

{% tab title="EQ-Bench" %}
Evaluate a LLM with EQ-Bench

<table><thead><tr><th width="183">args</th><th width="104">default</th><th>description</th></tr></thead><tbody><tr><td>eq_bench</td><td>False</td><td>Evaluate a LLM with EQ-Bench</td></tr><tr><td>eq_bench_prompt_type</td><td>"ChatML"</td><td>Set chat template</td></tr><tr><td>eq_bench_lora_path</td><td>None</td><td>Path to LORA adapters</td></tr><tr><td>eq_bench_quantization</td><td>None</td><td>Whether to quantize the LLM when loading</td></tr></tbody></table>

## Example

#### Library

```
import evalverse as ev

evaluator = ev.Evaluator()
evaluator.run(
    model="upstage/SOLAR-10.7B-Instruct-v1.0",
    benchmark="eq_bench",
    eq_bench_prompt_type="Solar-v1",
    devices="0,1,2,3,4,5,6,7",
    output_path="./results"
)
```

#### CLI

```
python3 evaluator.py \
    --ckpt_path upstage/SOLAR-10.7B-Instruct-v1.0 \
    --eq_bench \
    --eq_bench_prompt_type Solar-v1 \
    --devices 0,1,2,3,4,5,6,7 \
    --output_path ./results
```

## Result

```
📦results
 ┗ 📂SOLAR-10.7B-Instruct-v1.0
   ┗ 📂eq_bench
     ┣ 📜benchmark_results.csv
     ┗ 📜raw_results.json
```

{% endtab %}
{% endtabs %}