from unsloth import FastLanguageModel # we use unsloth for faster inference
import torch
max_seq_length = 4096 
dtype = None
load_in_4bit = False

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "mehmetkeremturkcan/oblivionsend",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit
)

from transformers import TextStreamer
FastLanguageModel.for_inference(model)
text_streamer = TextStreamer(tokenizer)

inputs = tokenizer(
[
"""<start_of_turn>user
Write a story with the following description: Setting - a dark abandoned watchtower and its environs. A wizard carefully explores a tomb where a priest of a dark, dead God has raised a band of brigands that have been terrorizing a town."""+ """<end_of_turn>
<start_of_turn>model
"""
], return_tensors = "pt").to("cuda")

_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 4096, num_beams=1, temperature=1.0, do_sample=True)

Model Description

Finetuned from model: google/gemma-2-9b-it

Model Sources [optional]

Repository: GitHub.

Uses

Made for creative writing.

Training Details

Training Data

Check out the model card details.

Training Procedure

Model training performance (margins) are available in the wandb instance.

Training Hyperparameters

Training regime: bf16 on a 1x 80GB A100 node.

Environmental Impact

Total emissions are estimated to be 0.83 kgCO$_2$eq.