Views
No views yet
1{user_c_code}
2
3[/INST]"""
4
5Generate the analysis
6sequences = pipe(
7prompt,
8do_sample=True,
9temperature=0.2,
10top_p=0.9,
11num_return_sequences=1,
12eos_token_id=pipe.tokenizer.eos_token_id,
13max_new_tokens=512,
14)
15
16Print the model's output
17for seq in sequences:
18print(seq['generated_text'])
19
20
21## Training Procedure
22
23The model was fine-tuned using a script that leverages PEFT (Parameter-Efficient Fine-Tuning) with LoRA and 4-bit quantization to make the process memory-efficient.
24
25### Data Preparation
261. The `oussama120/misra-c-instructions` dataset was loaded, which contains a single `text` column.
272. Each sample in the `text` column was parsed to extract the `###Instruction:` and `###Response:` sections.
283. Any rows that could not be parsed correctly were filtered out and excluded from training.
294. The extracted instruction and response pairs were formatted into the CodeLlama prompt format: `<s>[INST] {instruction} [/INST] {response} </s>`.
30
31### Fine-tuning Technique
32- **Quantization:** The base model was loaded in 4-bit precision using the `BitsAndBytesConfig` from the `transformers` library. This dramatically reduces the GPU memory footprint. The quantization type used was `nf4`.
33- **PEFT with LoRA:** Low-Rank Adaptation (LoRA) was applied to the model. Instead of training all the model's weights, only small, low-rank adapter matrices were added to the attention layers and trained. This allows for efficient fine-tuning with far fewer trainable parameters.
34
35### Hyperparameters
36
37| Hyperparameter | Value |
38|-------------------------------|-----------------------------------------------------------------|
39| `base_model` | `codellama/CodeLlama-7b-Instruct-hf` |
40| `num_train_epochs` | 3 |
41| `per_device_train_batch_size` | 1 |
42| `gradient_accumulation_steps` | 8 |
43| `learning_rate` | 2e-4 |
44| `lr_scheduler_type` | `cosine` |
45| `warmup_ratio` | 0.03 |
46| `optimizer` | `paged_adamw_8bit` |
47| `lora_r` (rank) | 64 |
48| `lora_alpha` | 64 |
49| `lora_dropout` | 0.1 |
50| `lora_target_modules` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
51
52### Hardware
53- **GPU:** 1x NVIDIA A100 (48GB VRAM)
54- **Frameworks:** PyTorch, Transformers, PEFT, TRL, Accelerate
55
56## Testing Procedure
57
58The model was evaluated using two primary methods:
59
60### Method 1: Validation on the Dataset Test Split
611. **Model Loading:** The trained LoRA adapters were merged into the base model to create a standalone, fully fine-tuned model.
622. **Data Loading:** The `test` split of the `oussama120/misra-c-instructions` dataset was loaded.
633. **Inference:** An interactive script prompted the user for a row number from the test set.
644. **Comparison:** The script extracted the instruction from the specified row, generated a response from the model, and printed it side-by-side with the ground-truth response from the dataset.
655. **Export:** The results, including the instruction, model response, and actual response, were logged to a CSV file for later analysis.
66
67### Method 2: Interactive Analysis of Custom C Code
681. **Model Loading:** The merged model was loaded and prepared for inference.
692. **User Input:** An interactive script prompted the user to enter a multi-line C code snippet directly into the terminal.
703. **Prompt Engineering:** The user's code was wrapped in a standardized analysis prompt (`<s>[INST] Analyze the following C code...[/INST]`).
714. **Generation:** The model generated a detailed analysis of potential MISRA C violations.
725. **Export:** The user's input C code and the model's generated analysis were printed to the console and saved to a CSV file for record-keeping.
73
74## Citation
75
76```bibtex
77@misc{codellama,
78 title = {Code Llama: Open Foundation Models for Code},
79 author = {Rozière, Baptiste and Gehring, Jonas and Gloeckle, Fabian and So, Dmytro and Beeferman, David and Borges, Lui and Liptchinsky, Valentin and Lavril, Tristan and Izacard, Guillaume and Grave, Edouard and Lample, Guillaume and Touvron, Hugo},
80 year = {2023},
81 month = {August},
82 eprint = {2308.12950},
83 archivePrefix = {arXiv},
84 primaryClass = {cs.CL}
85}
86
87@misc{misra_c_dataset,
88 author = {Oussama},
89 title = {MISRA C Instructions},
90 year = {2023},
91 publisher = {Hugging Face},
92 journal = {Hugging Face repository},
93 howpublished = {\url{[https://huggingface.co/datasets/oussama120/misra-c-instructions](https://huggingface.co/datasets/oussama120/misra-c-instructions)}}
94}