ChessLM Qwen3 - Neuron Traced (AWS Format Structure)
This is a Neuron-traced version of karanps/ChessLM_Qwen3 optimized for AWS Trainium (trn1) and Inferentia (inf2) instances using vLLM with continuous batching enabled.
This model follows the AWS Neuron repository structure with separate directories for compiled artifacts.
The AWS Neuron reference models (e.g., aws-neuron/Qwen3-1.7B-TP2-BS8-SEQ4096) typically have:
A weights/ directory with separate safetensors files (e.g., tp0_sharded_checkpoint.safetensors)
A smaller model.pt (e.g., ~100MB) containing just the model structure
This model has:
Weights embedded within model.pt (17GB)
An empty weights/ directory (preserved for format compatibility)
This is because Neuron-compiled models with optimum-neuron[vllm]==0.3.0 bundle weights within the compiled format. The weights are optimized and embedded in the NEFF (Neuron Executable File Format) during compilation. This is a valid alternative implementation that provides the same functionality.
Compilation logs: Detailed logs from neuronx-cc compiler
Metadata: Configuration and metrics from compilation
Continuous Batching
This model is compiled with continuous batching enabled, which allows vLLM to:
Process multiple requests simultaneously with dynamic batch sizes up to 4
Optimize throughput by batching requests with different sequence lengths
Reduce latency for concurrent inference workloads
Note: On-device sampling is disabled due to a known Neuron runtime limitation when using tensor parallelism with 2 cores. Sampling is handled on the host instead.
