The model was developed as part of a DL course at HSE university. The task was to create a model capable of generating anecdotes in Russian. This involved writing a tokenizer and implementing Byte Pair Encoding (BPE), followed by building a custom Transformer model. The model incorporates SwiGLU activation functions, Grouped Query Attention for optimization, and ALiBI positional embeddings. It was then trained on a dataset of Russian anecdotes.
Tokenizer consists of 2048 tokens. Model window is 256 tokens.
usage:
Right now I suppose only snapshot_download from huggingface_hub works.
After that:
1tokenizer = AutoTokenizer.from_pretrained(model_path)
2model = AutoModelForCausalLM.from_pretrained(model_path)
3inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
4with torch.no_grad():
5 outputs = model.generate(**inputs)
Or you can just use .safetensors weights wherever you want.
