"emotion|anger|increases_a_lot\nemotion|hurt|increases_a_little\nrelationship|closeness|increases_a_little\nbelief|user_is_frustrated|increases_a_little"

Model's prediction :

yaml

state_changes:
  emotion:
    anger: increases_a_lot
    hurt: increases_a_little
  relationship:
    closeness: increases_a_little
  belief:
    user_is_frustrated: increases_a_little
  memory: null
  reflection: null

Output of the dataset :

yaml

state_changes:
  relationship:
    trust: increases_a_little
  emotion:
    anger: decreases
  belief:
    people_can_change: increases_a_little
  memory: reinforce_previous_conflict
  reflection: required

The model didn't really integrate the difference between relationship and emotion, didn't really integrate the subtleties about the character personnality, but didn't totally hallucinate : it integrate the output format, didn't repeat himself, and proposed a pretty relevant IR (not totally absurd). The result is quite the same as for the first model, but a little better (more content).

Prediction - held-out example

Model's raw output :

text

RAW OUTPUT:
"emotion|anger|increases\nemotion|hope|increases\nemotion|hope|increases\nrelationship|trust|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelations"

Model's output :

yaml

state_changes:
  emotion:
    anger: increases
    hope: increases
  relationship:
    trust: increases
    trust_is_frustration: increases
  belief: null
  memory: null
  reflection: null

Output of the test dataset :

yaml

state_changes:
  relationship:
    respect: increases_a_little
    disappointment: decreases
  emotion:
    irritation: decreases
    pride: increases_a_little
  belief:
    effort_matters: increases
  memory: reinforce_previous_conflict
  reflection: required

The problem is that here, the model seems worst than the one with 135 examples... so I decided to run other tests :

Prediction - held-out example 2

RAW OUTPUT: "no_changes"

Model's output :

state_changes: no_changes

Output of the dataset :

yaml

state_changes:
  emotion:
    contentment: increases_a_little

This time, the model didn't hallucinate. It only decided that no changes were required, which is not absurd.

![NOTE] This examples and the following ones are from the dataset, but the model wasn't trained on these.

Prediction - held-out example 3

Model's output :

yaml

state_changes:
  emotion:
    contentment: increases_a_little
    warmth: increases_a_little
  relationship: null
  belief: null
  memory: null
  reflection: null

Output from the dataset :

yaml

state_changes:
  emotion:
    contentment: increases_a_little

This time, the result is very good. While the model wasn't trained over this examples (he didn't know this one), its prediction is adequate : he predicted contentment, but also added warmth, which is not absurd. We still have a problem : the hallucination. The model repeated himself. Anyway, thanks to the internal parser, this doesn't affect the final output.

Prediction - held-out example 4

RAW OUTPUT: "no_changes"

Output of the model :

state_changes: no_changes

Output of the dataset :

state_changes: no_changes

While the model wasn't trained over this example, it prediction is correct. He 'understands' that there are no changes or minor changes in basic conversations.

Prediction - held-out example 5

Model's output :

yaml

state_changes:
  emotion:
    determination: increases
  relationship:
    determination: increases
  belief:
    exhaustion: increases
  memory: null
  reflection: required

Output of the dataset :

yaml

state_changes:
  emotion:
    exhaustion: increases_a_little
    relief: increases
    warmth: increases_a_little
  relationship:
    closeness: increases_a_little

This examples is more complex. It manages to predict something coherent (exhaustion, determination, null memory is right...), but not totally right : exhaustion is NOT a belief, reflection wasn't really required, and closeness was required.

Current model's behaviour :

Situation type	Behaviour
Very simple	Almost always correct
Casual conversation	Very good, sometimes even better than the dataset (`warmth`)
Situation complex with emotions	Understands concepts but makes magnitude and classification errors
Situation very complex	Begins mixing classes, forgets elements, hallucinate more

Next steps

I think the real bottleneck is the model size : a 360M parameters could be more appropriate. Also, a larger dataset would make the model understand more situations and more complex ones.

Next things to do are :

train a larger model (~360M parameters) with a larger dataset / the same one (~500 examples) and ~100 epochs (to avoid overtraining and memorization)
train a LoRA instead of finetuning all the model.
train with an even datasets : 500, 1500, 2500 examples.
evaluate the model's predictions over many tests
integrate the IR in the engine

CogStateIR/CogStateIR-Supra50M-v0.0.3-medium_dataset

Supra 50 M Instruct (third time, medium dataset)

Training setup

Results

Prediction - training example

Prediction - held-out example

Prediction - held-out example 2

Prediction - held-out example 3

Prediction - held-out example 4

Prediction - held-out example 5

Current model's behaviour :

Next steps

CogStateIR/
CogStateIR-Supra50M-v0.0.3-medium_dataset