You should visit that site and read all the details of the competition, which
include the task definition, how your model will be evaluated, the format of
submissions to the competition, etc.
Create a CodaBench account
You must create a CodaBench account and join the competition:
In the upper right corner of the page, you should see a "Sign Up" button.
Click that and go through the process to create an account.
Please use your @arizona.edu account when signing up.
Your username will be displayed publicly on a leaderboard showing everyone's
scores.
If you wish to remain anonymous, please select a username that does not reveal
your identity.
Your instructor will still be able to match your score with your name via your
email address, but your email address will not be visible to other students.
Return to the competition website and click the "My Submissions" tab, accept
the terms and conditions of the competition, and register for the task.
Wait for your instructor to manually approve your request.
This may take a few days.
You should then be able to return to the "My Submissions" tab and see a
"Submission upload" form.
That means you are fully registered for the task.
Clone the repository
The GitHub Classroom link for creating your project repository will be
provided in a course announcement (and on D2L).
Once you have created your repository using the provided GitHub Classroom
link, clone the repository to your local machine:
You are now ready to begin working on the assignment.
Download the data
Go to the "Get Started" tab on the CodaBench site, and click on the "Files"
sub-tab.
You should see a button to download the training and validation (info-557-2025spr-PUBLIC-train_dev ;
referred to below as just 'dev') data for the task.
Download and unzip that data into your cloned repository directory.
Please do NOT commit the data to the repository.
When the test phase of the competition (Test Phase) begins, you may return to the "Files"
tab to download the unlabeled test data for the task (info-557-2025spr-PUBLIC-test); this will
not be available until the test phase.
Write your code
You should design a neural network model to perform the task described on the
CodaBench site.
Your code should train a model on the provided training data and tune
hyper-parameters on the provided validation (dev) data.
Your code should be able to make predictions on either the dev data
or (once available) the test data.
Your code should package up its predictions in a submission.zip file,
following the formatting instructions on CodaBench.
You must create and train your neural network in the Keras framework.
You should train and tune your model using the training and development data
that you downloaded from the CodaBench site.
If you would like to use any additional resource to train your model, you must
first ask for permission by contacting the instructors by email
(Clay and Jiacheng)
There is some sample code in this repository you can use in any way you like to get started.
This code is described briefly on the CodaBench site.
If you prefer to delete this code entirely and start from scratch, that is fine, too!
Test your model predictions on the validation (dev) set
During the development phase of the competition, the CodaBench site will expect
predictions on the dev set.
To test the performance of your model, run your model on the dev data,
format your model predictions as instructed on the CodaBench site, and upload
your model's predictions (in the submission.zip file) on the "My Submissions" tab of the CodaBench site.
During the development phase, you are allowed to upload predictions many times.
You are strongly encouraged to upload your model's dev set
predictions to CodaBench after every significant change to your code to make
sure you have all the formatting correct.
Submit your model predictions on the test set
When the test phase of the competition begins (consult the CodaBench site for
the exact timing), the instructor will update the CodaBench site to expect
predictions on the test set, rather than predictions on the development set.
The instructor will also release the unlabeled test set on CodaBench as
described above under "Download the Data".
To test the performance of your model, download the test data, run your model on
the test data, format your model predictions as instructed on the CodaBench
site, and upload your model's predictions (again, in the submission.zip file)
on the "My Submissions" tab of the CodaBench site.
During the test phase, you are allowed to upload predictions only once.
This is why it is critical to debug any formatting problems during the
development phase.
Also be sure to push the final state of your model code to your GitHub
repository during this phase. This will constitute your code submission
for the project.
Grading
You will be graded first by your model's accuracy, and second on how well your
model ranks in the competition.
If your model achieves better accuracy on the test set than the baseline model
included in this repository, you will get at least a B.
If your model achieves better accuracy on the test set than a second baseline
that the instructor will reveal during the test phase of the competition, you will get an A.
All models within the same letter grade will be assigned a numeric grade based on how they are ranked across the
range, evenly distributing the ranked performance across the letter grade.
So for example, the highest ranked model in the A range will get 100%, and the
lowest ranked model in the B range will get 80%.
If you train your neural network with any library other than Tensorflow/Keras,
or you use an external resource that you do not obtain permission for from the
instructors (Clay and Jiacheng), you will lose 10% (a letter grade) from
your score.
