Views
No views yet
transformers and use Galeras dataset for sampling the model:1from transformers import AutoTokenizer, AutoModelWithLMHead
2
3tokenizer = AutoTokenizer.from_pretrained("semeru/compatible-codeparrot-small")
4model = AutoModelWithLMHead.from_pretrained("semeru/compatible-codeparrot-small")
5
6df_sampled_code['size'] = df_sampled_code['ground_truth'].map(lambda code: len(tokenizer(code)['input_ids']))
7df_sampled_code['input_ids'] = tokenizer(df_sampled_code['prompt'].tolist())['input_ids']
8| Config | Value |
|---|---|
| Batch size | 192 |
| Context size | 1024 |
| Training steps | 150'000 |
| Gradient accumulation | 1 |
| Gradient checkpointing | False |
| Learning rate | 5e-4 |
| Weight decay | 0.1 |
| Warmup steps | 2000 |
| Schedule | Cosine |
| Metric | Value |
|---|---|
| pass@1 | 3.80% |
| pass@10 | 6.57% |
| pass@100 | 12.78% |