This model is a parameter-efficient fine-tuned version of Phi-3 Mini 4K trained for verbalized rebus solving in Italian, as part of the
release for our paper
Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses. The task of verbalized rebus solving consists of converting an encrypted sequence of letters and crossword definitions into a solution phrase matching the word lengths specified in the solution key. An example is provided below.
The model was trained in 4-bit precision for 5070 steps on the verbalized subset of the
EurekaRebus using QLora via
Unsloth and
TRL. This version has merged adapter weights in half precision, enabling out-of-the-box for usage with the
transformers library.
We also provide
adapter checkpoints through training and
8-bit GGUF versions of this model for analysis and local execution.
The following example shows how to perform inference using Unsloth or Transformers:
1
2# With Unsloth (efficient, requires GPU)
3from unsloth import FastLanguageModel
4
5model, tokenizer = FastLanguageModel.from_pretrained(
6 model_name = "gsarti/phi3-mini-rebus-solver-fp16",
7 max_seq_length = 1248,
8 load_in_4bit = True,
9)
10
11# Or with Transformers
12from transformers import AutoTokenizer, AutoModelForCausalLM
13
14tokenizer = AutoTokenizer.from_pretrained("gsarti/phi3-mini-rebus-solver-fp16")
15model = AutoModelForCausalLM.from_pretrained("gsarti/phi3-mini-rebus-solver-fp16")
16
17# Inference
18verbalized_rebus = "[Materiale espulso dai vulcani] R O [Strumento del calzolaio] [Si trovano ai lati del bacino] C I [Si ingrassano con la polenta] E I N [Contiene scorte di cibi] B [Isola in francese]"
19solution_key = "1 ' 5 6 5 3 3 1 14"
20template = """<s><|user|>
21Risolvi gli indizi tra parentesi per ottenere una prima lettura, e usa la chiave di lettura per ottenere la soluzione del rebus.
22
23Rebus: {rebus}
24Chiave risolutiva: {key}<|end|>
25<|assistant|>"""
26input = template.format(rebus=verbalized_rebus, key=solution_key)
27inputs = tokenizer(input, return_tensors="pt")["input_ids"]
28outputs = model.generate(input_ids = inputs, max_new_tokens = 500, use_cache = True)
29model_generations = tokenizer.batch_decode(outputs)
30print(model_generations[0])
31
32# Procediamo alla risoluzione del rebus passo per passo:
33# - [Materiale espulso dai vulcani] = lava
34# - R O = R O
35# - [Strumento del calzolaio] = lesina
36# - [Si trovano ai lati del bacino] = anche
37# - C I = C I
38# - [Si ingrassano con la polenta] = oche
39# - E I N = E I N
40# - [Contiene scorte di cibi] = silos
41# - B = B
42# - [Isola in francese] = ile
43#
44# Prima lettura: lava R O lesina anche C I oche E I N silos B ile
45#
46# Ora componiamo la soluzione seguendo la chiave risolutiva:
47# 1 = L
48# ' = '
49# 5 = avaro
50# 6 = lesina
51# 5 = anche
52# 3 = ciò
53# 3 = che
54# 1 = è
55# 14 = insilosbile
56#
57# Soluzione: L'avaro lesina anche ciò che è insilosbile
A ready-to-use local version of this model is hosted on the
Ollama Hub and can be used as follows:
Lexical overfitting: As remarked in the related publication, the model overfitted the set of definitions/answers for first pass words. As a result, words that were
explicitly witheld from the training set cause significant performance degradation when used as solutions for verbalized rebuses' definitions. You can compare model performances between
in-domain and
out-of-domain test examples to verify this limitation.
For problems or updates on this model, please contact
gabriele.sarti996@gmail.com.
1@article{sarti-etal-2024-rebus,
2 title = "Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses",
3 author = "Sarti, Gabriele and Caselli, Tommaso and Nissim, Malvina and Bisazza, Arianna",
4 journal = "ArXiv",
5 month = jul,
6 year = "2024",
7 volume = {abs/2408.00584},
8 url = {https://arxiv.org/abs/2408.00584},
9}
