Llamion (Llamafied Orion) is derived from transforming the
Orion model
into
the standard LLaMA architecture
through parameter mapping and offline knowledge transfer.
Further technical specifications and study results are detailed in
our paper.
Notably, the LongChat model supports an extensive text range of 200K tokens.
The following figure shows the perplexity of models
on
English Wikipedia corpus
and
Korean Wikipedia corpus, respectively.
