You can deploy this model and start earning money today!

mlx-community/
RakutenAI-2.0-8x7B-MLX-4bit

Name: RakutenAI-2.0-8x7B-MLX-4bit
Availability: InStock

Rakuten/RakutenAI-2.0-8x7B

quantized

apache-2.0

4-bit

Views

No views yet

This model was converted to MLX format from Rakuten/RakutenAI-2.0-8x7B using mlx-lm version 0.31.0. Refer to the original model card for more details on the model.

Use with mlx

pip install mlx-lm

python

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/RakutenAI-2.0-8x7B-MLX-4bit")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)

Inference: M3 Ultra
==========
 world - CSDN
精华内容
下载资源
问答
我要提问
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11:20:00
Hello World
展开全文
Hello World
2019-04-09 11
==========
Prompt: 2 tokens, 7.076 tokens-per-sec
Generation: 256 tokens, 70.175 tokens-per-sec
Peak memory: 26.421 GB