Views
No views yet
transformers repack of EssentialAI/rnj-1.5-instruct intended for quick inference tests, downstream software compatiblity (transformers/torch itself, llama.cpp, etc):layer_types: chunked_attention, which modeling_gemma3.py doesn't implement - inference crashes with KeyError: 'chunked_attention'. This repo swaps those entries to sliding_attention so the model loads under stock transformers. Weights unchanged, resaved in bf16.[!IMPORTANT] Sliding window (8192) is not identical to the original block-local attention - equivalent for prompts ~< 8192 tokens, divergent beyond that. For faithful long-context inference, use vLLM 0.20.0 against the upstream repo.
| Field | Upstream | Here |
|---|---|---|
| Local layer type | chunked_attention | sliding_attention |
| RoPE params for locals | under chunked_attention key | moved to sliding_attention key |
| Dtype | float32 | bfloat16 |
| Architecture string | Rnj1ForCausalLM | Gemma3ForCausalLM |
LLLGLLLGLLLGLGGGGGLGLLLGLLLGLLLL) preserved.1import torch
2from transformers import pipeline
3
4pipe = pipeline(
5 "text-generation",
6 model="pszemraj/rnj-1.5-instruct",
7 dtype=torch.bfloat16,
8 device_map="auto",
9)
10res = pipe([{"role": "user", "content": "Who are you?"}])
11print(res)