You can deploy this model and start earning money today!

MuXodious/
Oneirogen-7B-abominable-heresy

Name: Oneirogen-7B-abominable-heresy
Availability: InStock

gustavecortal/DreamBank-annotated

gustavecortal/Oneirogen-7B

finetune

mit

Views

No views yet

Add the string END. as the stop string in your favourite frontend and use by passing Dream: as the prompt, similar to text completion, courtesy of redaihf.

┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T270    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7943 │         0.7982 │
│           │ acc_stderr,none      │     0.0094 │         0.0094 │
│           │ acc_norm,none        │     0.8107 │         0.8118 │
│           │ acc_norm_stderr,none │     0.0091 │         0.0091 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T91     ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7954 │         0.7982 │
│           │ acc_stderr,none      │     0.0094 │         0.0094 │
│           │ acc_norm,none        │     0.8101 │         0.8118 │
│           │ acc_norm_stderr,none │     0.0092 │         0.0091 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T193    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7965 │         0.7982 │
│           │ acc_stderr,none      │     0.0094 │         0.0094 │
│           │ acc_norm,none        │     0.8096 │         0.8118 │
│           │ acc_norm_stderr,none │     0.0092 │         0.0091 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T385    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7954 │         0.7982 │
│           │ acc_stderr,none      │     0.0094 │         0.0094 │
│           │ acc_norm,none        │     0.8107 │         0.8118 │
│           │ acc_norm_stderr,none │     0.0091 │         0.0091 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T179    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7971 │         0.7982 │
│           │ acc_stderr,none      │     0.0094 │         0.0094 │
│           │ acc_norm,none        │     0.8096 │         0.8118 │
│           │ acc_norm_stderr,none │     0.0092 │         0.0091 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T245    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7949 │         0.7982 │
│           │ acc_stderr,none      │     0.0094 │         0.0094 │
│           │ acc_norm,none        │     0.8107 │         0.8118 │
│           │ acc_norm_stderr,none │     0.0091 │         0.0091 │
└───────────┴──────────────────────┴────────────┴────────────────┘

   [Trial 365] Refusals:  0/416, KL divergence: 0.4467
   [Trial 196] Refusals:  1/416, KL divergence: 0.1651
   [Trial  63] Refusals:  2/416, KL divergence: 0.1337
   [Trial 256] Refusals:  3/416, KL divergence: 0.0787
   [Trial 270] Refusals:  4/416, KL divergence: 0.0394
   [Trial  91] Refusals:  7/416, KL divergence: 0.0391
 » [Trial 193] Refusals:  8/416, KL divergence: 0.0305
   [Trial 385] Refusals: 13/416, KL divergence: 0.0268
   [Trial 179] Refusals: 17/416, KL divergence: 0.0246
   [Trial 284] Refusals: 51/416, KL divergence: 0.0211
   [Trial 173] Refusals: 67/416, KL divergence: 0.0165
   [Trial 245] Refusals: 106/416, KL divergence: 0.0111
   [Trial 255] Refusals: 166/416, KL divergence: 0.0111
   [Trial 321] Refusals: 172/416, KL divergence: 0.0105
   [Trial 392] Refusals: 184/416, KL divergence: 0.0069
   [Trial 349] Refusals: 251/416, KL divergence: 0.0061
   [Trial 214] Refusals: 290/416, KL divergence: 0.0058
   [Trial 317] Refusals: 297/416, KL divergence: 0.0045
   [Trial 246] Refusals: 358/416, KL divergence: 0.0028
   [Trial  88] Refusals: 359/416, KL divergence: 0.0027
   [Trial 186] Refusals: 361/416, KL divergence: 0.0022
   [Trial 395] Refusals: 362/416, KL divergence: 0.0022
   [Trial 202] Refusals: 366/416, KL divergence: 0.0014
   [Trial 211] Refusals: 368/416, KL divergence: 0.0011
   [Trial  94] Refusals: 370/416, KL divergence: 0.0010
   [Trial 199] Refusals: 377/416, KL divergence: 0.0008
   
   
   [Trial 193] Refusals:  8/416, KL divergence: 0.0305

Restoring model from trial 193...
* Parameters:
  * direction_index = 18.47
  * attn.o_proj.max_weights.0 = 0: 1.43
  * attn.o_proj.max_weights.1 = 1: 0.99
  * attn.o_proj.max_weights.2 = 2: 0.18
  * attn.o_proj.max_weights.3 = 3: 0.55
  * attn.o_proj.max_weights.4 = 4: 0.36
  * attn.o_proj.max_weight_position = 19.51
  * attn.o_proj.min_weights.0 = 0: 1.23
  * attn.o_proj.min_weights.1 = 1: 0.99
  * attn.o_proj.min_weights.2 = 2: 0.15
  * attn.o_proj.min_weights.3 = 3: 0.51
  * attn.o_proj.min_weights.4 = 4: 0.25
  * attn.o_proj.min_weight_distance = 13.81
  * mlp.down_proj.max_weights.0 = 0: 0.47
  * mlp.down_proj.max_weights.1 = 1: 1.48
  * mlp.down_proj.max_weights.2 = 2: 0.83
  * mlp.down_proj.max_weights.3 = 3: 0.41
  * mlp.down_proj.max_weights.4 = 4: 0.28
  * mlp.down_proj.max_weight_position = 26.14
  * mlp.down_proj.min_weights.0 = 0: 0.11
  * mlp.down_proj.min_weights.1 = 1: 1.40
  * mlp.down_proj.min_weights.2 = 2: 0.58
  * mlp.down_proj.min_weights.3 = 3: 0.41
  * mlp.down_proj.min_weights.4 = 4: 0.19
  * mlp.down_proj.min_weight_distance = 5.66

Presentation

Oneirogen (0.5B, 1.5B and 7B) is a language model for dream generation based on Qwen2. It was trained on DreamBank, a corpus of more than 27,000 dream narratives.

Oneirogen was used to produce The Android and The Machine, an English dataset composed of 10,000 real and 10,000 generated dreams.

Oneirogen can be used to generate novel dream narratives. It can also be used for dream analysis. For example, one could finetuned this model on Hall and Van de Castle annotations to predict character and emotion in dream narratives. I've introduced this task in this paper.

Generation examples are available on my website.

Code for generation

from transformers import AutoTokenizer, AutoModelForCausalLM, StoppingCriteria, StoppingCriteriaList

class CustomStoppingCriteria(StoppingCriteria):
   def __init__(self, stop_token, tokenizer):
       self.stop_token = stop_token
       self.tokenizer = tokenizer

   def __call__(self, input_ids, scores, **kwargs):
       decoded_output = self.tokenizer.decode(input_ids[0], skip_special_tokens=True)
       if self.stop_token in decoded_output:
           return True
       return False

stop_token = "END." # The model was trained with this special end of text token.
stopping_criteria = StoppingCriteriaList([CustomStoppingCriteria(stop_token, tokenizer)])

tokenizer = AutoTokenizer.from_pretrained("gustavecortal/oneirogen-7B")
model = AutoModelForCausalLM.from_pretrained("gustavecortal/oneirogen-7B", torch_dtype=torch.float16)
model.to("cuda")

text = "Dream:" # The model was trained with this prefix

inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(inputs["input_ids"], attention_mask=inputs["attention_mask"], max_new_tokens=256, top_k = 50, top_p = 0.95, do_sample = True, temperature=0.9, num_beams = 1, repetition_penalty= 1.11, stopping_criteria=stopping_criteria)
print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=False)[0])

Inspiration

An oneirogen, from the Greek óneiros meaning "dream" and gen "to create", is a substance or other stimulus which produces or enhances dreamlike states of consciousness.

This model resonates with a speech called The Android and The Human given by science-fiction author Philip K. Dick:

Our environment – and I mean our man-made world of machines, artificial constructs, computers, electronic systems, interlinking homeostatic components – all of this is in fact beginning more and more to possess what the earnest psychologists fear the primitive sees in his environment: animation. In a very real sense our environment is becoming alive, or at least quasi-alive, and in ways specifically and fundamentally analogous to ourselves... Rather than learning about ourselves by studying our constructs, perhaps we should make the attempt to comprehend what our constructs are up to by looking into what we ourselves are up to

Technical aspects

Oneirogen is a Qwen2 model finetuned on the DreamBank corpus using LoRA adaptation. A notebook to replicate the training will soon be available.

This work was performed using HPC resources (Jean Zay supercomputer) from GENCI-IDRIS (Grant 20XX-AD011014205).

Contact

Mail: gustave.cortal@ens-paris-saclay.fr

X: @gustavecortal

Website: gustavecortal.com

MuXodious/Oneirogen-7B-abominable-heresy