You can deploy this model and start earning money today!

gautambhetanabhotla/
inlp-a3-task2-ssm

Name: inlp-a3-task2-ssm
Availability: InStock

Views

No views yet

Task-2 SSM Language Model

Mamba-inspired S6 language model for next-word prediction.

Training Summary

Best epoch: 6
Best validation loss: 6.396149
Best validation perplexity: 599.531929
Local checkpoint path: models/task2_ssm.pt

Training Configuration

yaml

seed: 2498374213
data:
  plain_path: data/plain.txt
  append_eos: true
  val_fraction: 0.1
  test_fraction: 0.1
  min_freq: 1
  max_vocab_size: null
train:
  seq_len: 256
  stride: 32
  batch_size: 128
  epochs: 8
  lr: 0.001
  weight_decay: 0.0
  clip_grad_norm: 1.0
model:
  d_model: 48
  d_state: 12
  n_layers: 12
  conv_kernel: 6
  dropout: 0.2
checkpoint:
  path: models/task2_ssm.pt
eval:
  batch_size: 128
  output_path: outputs/task2_ssm_eval.txt
  sample_prompts:
  - the dominion
  - we can
  - language modeling
device: auto
wandb:
  enabled: true
  project: inlp-a3-task2-ssm
  run_name: task2-ssm
huggingface:
  enabled: true
  repo_id: gautambhetanabhotla/inlp-a3-task2-ssm
  token_env: HF_TOKEN
  checkpoint_path_in_repo: task2_ssm.pt
  config_path_in_repo: training_config.yaml
  readme_path_in_repo: README.md