This is a sentence-transformers model finetuned from BAAI/bge-m3 on the arcanum-quests-queries-synthetic-v2 dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: BAAI/bge-m3
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 1024 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text
Training Dataset:
- arcanum-quests-queries-synthetic-v2

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pameydorke/arcanum-quests-base-retriever-v3")
# Run inference
sentences = [
    'need help with torian kels curse',
    "Free Torian Kel from the Curse. Northeast of Tarant there are some ancient ruins. On the west end of the dungeon you will find the Torian Kel key. Use the key to open the door on the east end of the dungeon and speak to the crumbling skeleton. It will ask you to bring it some dragon's blood. It will mark the location of where to find some on your map. Go to the Dungeon of the Dragon Pool to the northwest. Click on the pool on the 2nd level of the dungeon to receive a Vial of Dragon's Blood. Return to the Ancient Temple and use the Vial of Dragon's Blood on the crumbling skeleton to release Torian Kel from the curse. He will tell you his story and may accept an offer to join you.",
    'Clean Up the Boil. Chairman Willoughsby wishes to clean up The Boil by ridding it of the the bulk of its criminal element. The player might be able to become involved',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6918, 0.2949],
#         [0.6918, 1.0000, 0.3334],
#         [0.2949, 0.3334, 1.0000]])

Evaluation

Metrics

Information Retrieval

Dataset: arcanum-test
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7237
cosine_accuracy@3	0.9167
cosine_accuracy@5	0.9605
cosine_accuracy@10	0.9868
cosine_precision@1	0.7237
cosine_precision@3	0.3056
cosine_precision@5	0.1921
cosine_precision@10	0.0987
cosine_recall@1	0.7237
cosine_recall@3	0.9167
cosine_recall@5	0.9605
cosine_recall@10	0.9868
cosine_ndcg@10	0.8675
cosine_mrr@10	0.8278
cosine_map@100	0.8285

Training Details

Training Dataset

arcanum-quests-queries-synthetic-v2

Dataset: arcanum-quests-queries-synthetic-v2 at d24544a
Size: 912 training samples
Columns: anchor and positive
Approximate statistics based on the first 912 samples:
anchor positive
type string string
details
min: 5 tokens
mean: 9.62 tokens
max: 19 tokens
min: 15 tokens
mean: 180.92 tokens
max: 1044 tokens

Samples:

anchor	positive
`kill king caladon guide`	Kill the King of Caladon. After talking with Mr Willoughsby, a man, Heinrich Jenks appears in the room outside. He wants you to assassinate the King Of Caladon, he tells you to meet Vernon in the Kitchen for more details. When the job is done he will meet you again in Grant’s Tavern and give you 20,000 gold.You can tell Mr Willoughsby and he will send Lorham, his bodyguard to deal with him (this botches the quest).Travelling to Caladon, you need to talk with Renard first and then you will find Vernon who gives you a key.You can also get into the palace by hiding in a crate on the Caladon waterfront.Upstairs you will find a guard patrolling, he will try and kill you on sight (getting past him depends on your style prowling and backstab work well here). Make sure you loot the rooms as there is a lot of jewellery. Go through the ventilation system and explore.It is worthwhile making your way to the treasure vault. The door can be difficult to pick (or break down without damaging weapons)....
`gambliing quest walkthrough`	Skill mastery quest. Gambling mastery. Gamble with Gurin Rockharrow. Women are refused entry to the Gentlemen's Club, but they still have a few avenues: Pick Pocket the key from the doorman Use Unlocking Cantrip to unbar the door Find out the name of the Club Owner from the doorman, getting his address from the Hall of Records, and then paying him a visit Once you meet with Wendell Wellington the options branch further: Pay a bribe and sleep with him for a special invitation Pick Pocket the invite Kill him and his bodyguard and loot his corpse for the invite With 8 points (2 ranks) of melee, you can threaten him for it With 4 points (1 rank) of persuasion and a Reaction Modifier of 81 you can sweet talk him for it Using this invitation will allow a lady to come and go as she pleases
`where is the key for big warehouse`	`Rid Mr. Plough's Warehouses of Rats. Talk to Simon Plough at the warehouse entrance and ask about his problem. Visit the smaller warehouse to the left first and kill the rats inside. You will also find a key in one of the barrels. Use the key to enter the bigger warehouse and kill everything inside. Return to Simon to finish the quest.`

Loss: GISTEmbedLoss with these parameters:
json
1{ 2 "guide": "SentenceTransformer('BAAI/bge-m3')", 3 "temperature": 0.01, 4 "margin_strategy": "absolute", 5 "margin": 0.0, 6 "contrast_anchors": true, 7 "contrast_positives": true, 8 "gather_across_devices": false 9}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
warmup_steps: 0.1
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 3
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: None
warmup_steps: 0.1
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	arcanum-test_cosine_ndcg@10
0.1754	10	0.4912	0.8698
0.3509	20	0.2105	0.8717
0.5263	30	0.2697	0.8649
0.7018	40	0.3948	0.8537
0.8772	50	0.2772	0.8565
1.0526	60	0.3912	0.8645
1.2281	70	0.1545	0.8461
1.4035	80	0.2246	0.8605
1.5789	90	0.1349	0.8693
1.7544	100	0.0925	0.8677
1.9298	110	0.1036	0.8589
2.1053	120	0.0441	0.8685
2.2807	130	0.0989	0.8711
2.4561	140	0.1541	0.8712
2.6316	150	0.0441	0.8705
2.8070	160	0.0109	0.8696
2.9825	170	0.0846	0.8675

Training Time

Training: 11.6 minutes

Framework Versions

Python: 3.12.13
Sentence Transformers: 5.4.0
Transformers: 5.0.0
PyTorch: 2.10.0+cu128
Accelerate: 1.13.0
Datasets: 4.0.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

GISTEmbedLoss

bibtex

@misc{solatorio2024gistembed,
    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
    author={Aivin V. Solatorio},
    year={2024},
    eprint={2402.16829},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

	anchor	positive
type	string	string
details	min: 5 tokens mean: 9.62 tokens max: 19 tokens	min: 15 tokens mean: 180.92 tokens max: 1044 tokens