Output is incohesive
The output is incohesive and wrong, I thought it corrupted or something, redownloaded, nothing changed
I'm using pipeline, updated transformers today
pipe = pipeline("text-generation", model=model, torch_dtype=torch.bfloat16, device_map="auto", pad_token_id=0)
pipe(messages, max_new_tokens=1000, do_sample=True, temperature=0.5, repetition_penalty=1.18)
repetition penalty may be too high
repetition penalty may be too high
Doesn't seem to change with penalty
There is a couple things you need to do. First off you should make your sending in the proper prompts with system commands. The other issue is phi really doesn't play well with the huggingface templates for chat. What that means is you get extra garbage in your chats. My suggestion is use the ONNX model or convert this one to ONNX and use my scripts to fix the build. Once when you do this the model works well.