Qwen2-7B-FlagOS-Arm provides an all-in-one deployment solution, enabling execution of arm-Qwen2-7B on Amazon . As the first-generation release for the Amazon Graviton4 CPUs, this package delivers two key features:
We use FlagScale as the serving engine to improve the portability of distributed inference.
FlagScale is an end-to-end framework for large models across multiple chips, maximizing computational resource efficiency while ensuring model effectiveness. It ensures both ease of use and high performance for users when deploying models across different chip architectures:
We validate the execution of arm-Qwen2-7B model with a Triton-based operator library as a PyTorch alternative.
We use a variety of Triton-implemented operation kernels to run the arm-Qwen2-7B model. These kernels come from two main sources:
-
Most Triton kernels are provided by FlagGems (
https://github.com/FlagOpen/FlagGems). You can enable FlagGems kernels by setting the environment variable USE_FLAGGEMS.
-
Also included are Triton kernels from vLLM, such as fused MoE.
Qwen2-7B is a pre-o1 model without "thinking" or "reasoning" abilities. We use MMLU instead of AIME or GPQA-diamond to evaluate Qwen2-7B
1pip install modelscope
2modelscope download --model Qwen/Qwen2-7B-Instruct --local_dir /nfs/Qwen/Qwen2-7B-Instruct
1docker run --rm --init --detach \
2 --net=host --uts=host --ipc=host \
3 --security-opt=seccomp=unconfined \
4 --privileged=true \
5 --ulimit stack=67108864 \
6 --ulimit memlock=-1 \
7 --ulimit nofile=1048576:1048576 \
8 --shm-size=32G \
9 -v /nfs:/nfs \
10 --gpus all \
11 --name flagos \
12 flagrelease-registry.cn-beijing.cr.aliyuncs.com/flagrelease/flagrelease:flagrelease-arm \
13 sleep infinity
14
15docker exec -it flagos bash
