Skip to content

Latest commit

 

History

History
176 lines (160 loc) · 4.95 KB

README.md

File metadata and controls

176 lines (160 loc) · 4.95 KB
Raw
language license tags datasets base_model model-index
en
llama2
Medicine
yahma/alpaca-cleaned
epfl-llm/meditron-7b
name results
meditron-7b-chat
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
AI2 Reasoning Challenge (25-Shot)
ai2_arc
ARC-Challenge
test
num_few_shot
25
type value name
acc_norm
50.77
normalized accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type split args
HellaSwag (10-Shot)
hellaswag
validation
num_few_shot
10
type value name
acc_norm
75.37
normalized accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
MMLU (5-Shot)
cais/mmlu
all
test
num_few_shot
5
type value name
acc
40.49
accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
TruthfulQA (0-shot)
truthful_qa
multiple_choice
validation
num_few_shot
0
type value
mc2
48.56
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
Winogrande (5-shot)
winogrande
winogrande_xl
validation
num_few_shot
5
type value name
acc
73.16
accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
GSM8k (5-shot)
gsm8k
main
test
num_few_shot
5
type value name
acc
9.17
accuracy

Model Card for Model ID

meditron-7b-chat is a finetuned version of epfl-llm/meditron-7b using SFT Training on the Alpaca Dataset. This model can answer information about different excplicit ideas in medicine (see epfl-llm/meditron-7b for more info)

Model Description

Prompt Template

### Instruction:

<prompt> (without the <>)

### Response:

How to Get Started with the Model

Use the code sample provided in the original post to interact with the model.

from transformers import AutoTokenizer,AutoModelForCausalLM
 
model_id = "malhajar/meditron-7b-chat"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             torch_dtype=torch.float16,
                                             revision="main")

tokenizer = AutoTokenizer.from_pretrained(model_id)

question: "what is tract infection?"
# For generating a response
prompt = '''
### Instruction:
{question} 

### Response:'''
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output = model.generate(inputs=input_ids,max_new_tokens=512,pad_token_id=tokenizer.eos_token_id,top_k=50, do_sample=True,
        top_p=0.95)
response = tokenizer.decode(output[0])

print(response)

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 49.59
AI2 Reasoning Challenge (25-Shot) 50.77
HellaSwag (10-Shot) 75.37
MMLU (5-Shot) 40.49
TruthfulQA (0-shot) 48.56
Winogrande (5-shot) 73.16
GSM8k (5-shot) 9.17