Views
No views yet
scrape_ski_data.py script.
skimag.compowder.combraveskimom.commabeyski.comsnowbrains.comtheskidiva.compurepowder.comrobots.txt allowed).a-z, space, comma, period; URLs and other characters removed.⏯️ start, ⏹️ end), adding 2 tokens per sentence.scrape_ski_data.py to recreate the web-scraped portion from the original sources, and combine it with your own synthetic data.⏯️, ⏹️, space, a-z, ., ,). Tokenizer normalizes text, maps to base tokens, pairs them into 2-character tokens, pads with space if odd-length.TransformerLM is a decoder-only transformer:embed_dim) and positional (max_seq_len → embed_dim).num_layers blocks, each with:
num_heads), Feedforward (embed_dim → feedforward_dim → embed_dim).dropout).embed_dim → 961).scrape_ski_data.py to collect ~17M tokens:python3 scrape_ski_data.pyski_sentences.txt. Takes hours/days depending on network and site response.webscrape.txt for training.train_skiLM.py for the training process. It processes the dataset into batches and trains the TransformerLM.model_weights. generate text with inference_skiLM.py.inference_skiLM.py: Inference script.scrape_ski_data.py: Web scraping script.train_skiLM.py: Training script.model_metadata.json: Training results, loss functions.