Views
No views yet
1# Serve
2vllm serve demonlxrd/olmoe-openhermes-ultrafeedback-dora-dpo-merged \
3 --max-model-len 4096 \
4 --dtype bfloat16
5
6# Inference
7curl -s http://localhost:8000/v1/chat/completions \
8 -H "Content-Type: application/json" \
9 -d '{
10 "model": "demonlxrd/olmoe-openhermes-ultrafeedback-dora-dpo-merged",
11 "messages": [
12 {"role": "user", "content": "Explain machine learning in simple terms."}
13 ],
14 "max_tokens": 200,
15 "temperature": 0.7
16 }' | jq -r '.choices[0].message.content'1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
3
4tokenizer = AutoTokenizer.from_pretrained("demonlxrd/olmoe-openhermes-ultrafeedback-dora-dpo-merged")
5model = AutoModelForCausalLM.from_pretrained(
6 "demonlxrd/olmoe-openhermes-ultrafeedback-dora-dpo-merged",
7 device_map="auto",
8 torch_dtype=torch.bfloat16
9)
10
11messages = [{"role": "user", "content": "What is quantum computing?"}]
12prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
13inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
14
15with torch.inference_mode():
16 outputs = model.generate(**inputs, max_tokens=200, temperature=0.7)
17
18print(tokenizer.decode(outputs[0], skip_special_tokens=True))1from openai import OpenAI
2
3client = OpenAI(
4 base_url="http://localhost:8000/v1",
5 api_key="dummy"
6)
7
8response = client.chat.completions.create(
9 model="demonlxrd/olmoe-openhermes-ultrafeedback-dora-dpo-merged",
10 messages=[
11 {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."}
12 ],
13 max_tokens=300,
14 temperature=0.7
15)
16
17print(response.choices[0].message.content)| Parameter | Value |
|---|---|
| Architecture | OLMoE (Mixture of Experts) |
| Parameters | ~1B active, 7B total |
| Precision | bfloat16 |
| Context Length | 4096 tokens |
| Training | SFT + DPO with DoRA adapters |
| Base Model | 1024m/OLMoE-1B-7B-0924-Base |
User:
<message>
Assistant:
<response>system, user, assistant