mlx-community/

Qwen2.5-7B-Instruct-kowiki-qa-8bit

Qwen2.5-7B-Instruct-kowiki-qa-8bit mlx convert model

• Original model is beomi/Qwen2.5-7B-Instruct-kowiki-qa

Requirement

• pip install mlx-lm

Usage

• Generate with CLI
  mlx_lm.generate --model mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-8bit --prompt "하늘이 파란 이유가 뭐야?"
• In Python
  python

  1from mlx_lm import load, generate 2 3model, tokenizer = load( 4 "mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-8bit", 5 tokenizer_config={"trust_remote_code": True}, 6) 7 8prompt = "하늘이 파란 이유가 뭐야?" 9 10messages = [ 11 {"role": "system", "content": "당신은 친철한 챗봇입니다."}, 12 {"role": "user", "content": prompt}, 13] 14prompt = tokenizer.apply_chat_template( 15 messages, 16 tokenize=False, 17 add_generation_prompt=True, 18) 19 20text = generate( 21 model, 22 tokenizer, 23 prompt=prompt, 24 # verbose=True, 25 # max_tokens=8196, 26 # temp=0.0, 27)
• OpenAI Compitable HTTP Server
  mlx_lm.server --model mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-8bit --host 0.0.0.0
  python

  1import openai 2 3 4client = openai.OpenAI( 5 base_url="http://localhost:8080/v1", 6) 7 8prompt = "하늘이 파란 이유가 뭐야?" 9 10messages = [ 11 {"role": "system", "content": "당신은 친절한 챗봇입니다.",}, 12 {"role": "user", "content": prompt}, 13] 14res = client.chat.completions.create( 15 model='mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-8bit', 16 messages=messages, 17 temperature=0.2, 18) 19 20print(res.choices[0].message.content)