-
Efficient Knowledge Distillation:
Carefully distilled from DeepSeek's QWEN 1.5B model to preserve capabilities while reducing computational requirements.
-
Mobile-First Design:
Architected specifically for the constraints of mobile devices, with optimizations for both inference speed and memory usage.
-
Balanced Performance:
Maintains a good balance between model size and language generation capabilities, making it practical for real-world mobile applications.
This model is part of our ongoing effort to bring powerful language models to edge devices. Upcoming releases will include:
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("ijktech/ByteGPT-r1", trust_remote_code=True)
4tokenizer = AutoTokenizer.from_pretrained("ijktech/ByteGPT-r1")
5
6input_text = "What is the capital of France?"
7inputs = tokenizer(input_text, return_tensors="pt")
8outputs = model.generate(**inputs, max_new_tokens=100)
9
10print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The model is also available in ONNX format, and can be used with the ONNX Runtime:
1import onnxruntime as ort
2import numpy as np
3
4# Create ONNX Runtime session
5ort_session = ort.InferenceSession("model.onnx")
6
7# Helper function to generate text using the ONNX model
8def generate_with_onnx(prompt_ids, max_new_tokens=50, temperature=1.0):
9 input_ids = prompt_ids.clone()
10
11 for _ in range(max_new_tokens):
12 # Get the last block_size tokens if input is too long
13 if input_ids.shape[1] > model.block_size:
14 input_ids = input_ids[:, -model.block_size:]
15
16 # Run inference
17 ort_inputs = {
18 'input': input_ids.cpu().numpy()
19 }
20 logits = ort_session.run(None, ort_inputs)[0]
21
22 # Get predictions for the next token
23 logits = torch.from_numpy(logits)
24 logits = logits[:, -1, :] # Only take the last token's predictions
25
26 # Apply temperature
27 if temperature != 1.0:
28 logits = logits / temperature
29
30 # Sample from the distribution
31 probs = torch.nn.functional.softmax(logits, dim=-1)
32 next_token = torch.multinomial(probs, num_samples=1)
33
34 # Append the new token
35 input_ids = torch.cat([input_ids, next_token], dim=1)
36
37 return input_ids
38
39# Test the generation
40prompt = "Hello"
41prompt_ids = tok(prompt, return_tensors="pt")["input_ids"]
42generated_ids = generate_with_onnx(prompt_ids)
43generated_text = tok.decode(generated_ids[0], skip_special_tokens=True)
44print(f"Generated text: {generated_text}")
45#Generated text: Hello there! How can I assist you today? I'm a helpful AI assistant trained to provide information and answer questions on a wide range of topics.
💼
Commercial Use: Contact IJK Technology Ltd for licensing at
james@ijktech.com.
IJK Technology Ltd (IJKTech) develops innovative machine learning models optimized for on-device inference. Our focus is on efficiency, privacy, and usability across mobile and embedded platforms.
