This model has been fine tuned to categorise user prompts into one of 47 Concepts or HCCH Principle Themes. The model should also be able to return “No Category” if no Concept or Theme is applicable.

The fine tuning data consists of 20 generated queries per Concept and Theme. Queries were generated using Meta Llama 3.2 (1B). Here’s one example:

instruction
Classify the following user query into one of the predefined categories and do not create new categories. Select from the following list: 'Revision/reform (concept)', 'Comparative method (concept)’, …, 'No Category'

user_query
What are potential solutions to address concerns about jurisdiction and transparency in revision or reform processes for existing international treaties?

category
Revision/reform (concept)

20 examples for “No Category” were generated using OpenAI ChatGPT-4o:

user_query
What was the main cause of the American Civil War?

category
No Category

In a further attempt to classify irrelevant queries, a further 20 “No Category” queries were generated ChatGPT-4o, which ask a legal question but have nothing to do with private international law:

user_query
What are the rights of tenants when facing eviction?

category
No Category

The fine tuning data contains a total of 980 query examples.

Evaluation

980 new, i.e., unseen, queries were generated (again using the same Llama and ChatGPT models in the same manner as above).

Overall Accuracy: 83.98%
Weighted Average F1-Score: 89.87%

F1-Score per Category
Sorted by descending F1-Score

Category	F1-Score
1 Arbitral awards vs. court decisions (concept)	1
2 Comparative method (concept)	1
3 Connection criteria/factors (concept)	1
4 Exclusion of renvoi (principle)	1
5 Freedom of choice (principle)	1
6 Neutral law / law of a 3rd country (concept)	1
7 PIL codification (concept)	1
8 Partial choice (principle)	1
9 Persuasive authority (concept)	1
10 Preamble (principle)	1
11 Principles/soft law (concept)	1
12 Refer to an instrument (concept)	1
13 Revision/reform (concept)	1
14 Scope of the chosen law (principle)	1
15 Severability (principle)	1
16 Tacit choice (concept)	1
17 Weaker/vulnerable parties (concept)	1
18 Arbitrators vs. State judges (concept)	0.974
19 Assignment (principle)	0.974
20 Express and tacit choice (principle)	0.974
21 Incorporate rules by way of reference (concept)	0.974
22 Modification of a choice of law clause (concept)	0.974
23 Respect choice of law clause (concept)	0.974
24 Rules of law (principle)	0.974
25 non-State law / rules of law (concept)	0.974
26 Absence of choice (concept)	0.947
27 Conventions/hard law (concept)	0.947
28 Express choice (concept)	0.947
29 Connection (concept)	0.919
30 Formal validity of the choice of law (principle)	0.919
31 Institutional rules (arbitration) (concept)	0.919
32 No Category	0.919
33 Agreement on the choice of law and battle of forms (principle)	0.889
34 Establishment (principle)	0.889
35 Legislation (concept)	0.889
36 International contracts (commercial) (concept)	0.824
37 Public policy (principle)	0.824
38 Mandatory rules (principle)	0.788
39 Public policy (concept)	0.788
40 Scope of the Principles (principle)	0.788
41 Absence of choice (principle)	0.75
42 Arbitration (principle)	0.75
43 Party autonomy (concept)	0.75
44 Dépeçage (principle)	0.710
45 Arbitration (concept)	0.667
46 Dépeçage (concept)	0.519
47 Party autonomy (principle)	0.519
48 Mandatory rules (concept)	0.462

Konfi7000/llama_3.2_rag_categorisation_fine_tuning

Uploaded model

Description

Evaluation

Konfi7000/
llama_3.2_rag_categorisation_fine_tuning