This model is fine-tuned on a code-completion dataset collected from the open-source Code4Me plugin. The training objective is to have a small, lightweight transformer model to filter out unnecessary and unhelpful code completions. To this end, we leverage the in-IDE telemetry data, and integrate it with the textual code data in the transformer's attention module.

Developed by: AISE Lab @ SERG, Delft University of Technology
Model type: JonBERTa
Language: Code
Finetuned from model: CodeBERTa-small-v1.

Models are named as follows:

CodeBERTa → CodeBERTa-ft-coco-[1,2,5]e-05lr
- e.g. CodeBERTa-ft-coco-2e-05lr, which was trained with learning rate of 2e-05.
JonBERTa-head → JonBERTa-head-ft-[dense,proj,reinit]
- e.g. JonBERTa-head-ft-dense-proj, where all have 2e-05 learning rate, but may differ in the head layer in which the telemetry features are introduced (either head or proj, with optional reinitialisation of all its weights).
JonBERTa-attn → JonBERTa-attn-ft-[0,1,2,3,4,5]L
- e.g. JonBERTa-attn-ft-012L , where all have 2e-05 learning rate, but may differ in the attention layer(s) in which the telemetry features are introduced (either 0, 1, 2, 3, 4, or 5L).

Other hyperparameters may be found in the paper or the replication package (see below).

Sources

Replication Repository: Ar4l/curating-code-completions
Paper: "A Transformer-Based Approach for Smart Invocation of Automatic Code Completion"
Contact: https://huggingface.co/Ar4l

To cite, please use

bibtex

@misc{de_moor_smart_invocation_2024,
	title = {A {Transformer}-{Based} {Approach} for {Smart} {Invocation} of {Automatic} {Code} {Completion}},
	url = {http://arxiv.org/abs/2405.14753},
	doi = {10.1145/3664646.3664760},
	author = {de Moor, Aral and van Deursen, Arie and Izadi, Maliheh},
	month = may,
	year = {2024},
}

Training Details

This model was trained with the following hyperparameters, everything else being TrainingArguments' default. The dataset was prepared identically across all models as detailed in the paper.

python

num_train_epochs : int      = 3 
learning_rate    : float    = 2e-5
batch_size       : int      = 16

Model Configuration

python

num_telemetry_features  :int = 26

add_feature_embeddings :bool = True 
feature_hidden_size     :int = num_telemetry_features * 4
feature_dropout_prob  :float = 0.1
add_feature_bias       :bool = True

add_self_attn          :bool = True
self_attn_layers  :list[int] = search(sum(
    [[i,j,k] for i in range(6) for j in range(6) for k in range(6) if i < j < k], 
    [[i,j] for j in range(6) for i in range(6) if i < j],
    [[i] for i in range(6)],
    []
))

AISE-TUDelft/JonBERTa-attn-ft-coco-34L

JonBERTa-attn-ft-coco-34L

Description

Sources

Training Details

Model Configuration

AISE-TUDelft/
JonBERTa-attn-ft-coco-34L