This repository hosts a conversion ofMediPhi-Instructfor use on Rockchip RK3588 single-board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using the RKNN-LLM toolkit
w8a8_g128.rkllm artifactw8a8_g128) may slightly impact model fidelity vs. upstream FP16..rkllm converted model is using an example script provided in the toolkit in this directory: rknn-llm/examples/rkllm_server_demo1python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
2 --rkllm_model_path <MODEL_PATH>/MediPhi-Instruct_w8a8_g128_rk3588.rkllm \
3 --target_platform rk35881{
2 "model":"MediPhi-Instruct",
3 "messages":[{
4 "role":"user",
5 "content":"<YOUR_PROMPT_HERE>"}],
6 "stream":false
7}curl:1curl -s -X POST <SERVER_IP_ADDRESS>:8080/rkllm_chat \
2 -H 'Content-Type: application/json' \
3 -d '{"model":"MediPhi-Instruct","messages":[{"role":"user","content":"What is the role of protein folding in disease?"}],"stream":false}'1{
2 "choices":[{
3 "finish_reason":"stop",
4 "index":0,
5 "logprobs":null,
6 "message":{
7 "content":"<MODEL_REPLY_HERE">,
8 "role":"assistant"}}],
9 "created":null,
10 "id":"rkllm_chat",
11 "object":"rkllm_chat",
12 "usage":{
13 "completion_tokens":null,
14 "prompt_tokens":null,
15 "total_tokens":null}
16}{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":" Protein misfolding and aggregation are central to many diseases, including neurodegenerative disorders like Alzheimer's Disease (AD), Parkinson\u2019s Disease (PD), Huntington\u2019s Disease (HD) etc.\n\nIn these conditions, proteins fail to fold into their correct three-dimensional structures and instead aggregate or misfold in the cellular environment leading to formation of insoluble protein aggregates known as amyloid fibrils which are highly resistant to degradation by lysosomal enzymes.\n\nThese aberrant proteins can disrupt normal physiological processes, interfere with neuronal communication and eventually lead to cell death or apoptosis in the affected tissues/organs thereby causing disease symptoms associated with these conditions like memory loss (in Alzheimer\u2019s), movement difficulties etc.\n\nTherefore understanding protein folding pathways is crucial for developing therapeutic strategies aimed at preventing misfolding or promoting proper refolding of aberrant proteins back into their native conformations thereby restoring normal cellular function and alleviating disease symptoms associated with these conditions.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}http://<SERVER_IP_ADDRESS>:8080 and use the endpoint: /rkllm_chatmodel field matches the converted model’s name, for example:1{
2 "model": "MediPhi-Instruct",
3 "messages": [{"role":"user","content":"Hello!"}],
4 "stream": false
5}