This is a text embedding model finetuned from arthemislm-base on the all-nli-pair, all-nli-pair-class, all-nli-pair-score, all-nli-triplet, stsb, quora and natural-questions datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
The Arthemis Embedding model is a 155.8M parameter text embedding model that incorporates Spiking Neural Networks (SNNs) and Liquid Time Constants (LTCs) for enhanced temporal dynamics and semantic representation learning. This neuromorphic architecture provides unique advantages in classification tasks while maintaining competitive performance across various text understanding benchmarks.
This embedding model performs on par with jinaai/jina-embeddings-v2-base-en on MTEB
Model Details
Model Type: Text Embedding Supported Languages: English Number of Parameters: 155.8M Context Length: 1024 tokens Embedding Dimension: 768 Base Model: arthemislm-base Training Data: all-nli-pair, all-nli-pair-class, all-nli-pair-score, all-nli-triplet, stsb, quora, natural-questions
Architecture Features
Spiking Neural Networks in attention mechanisms for temporal processing
Liquid Time Constants in feed-forward layers for adaptive dynamics
12-layer transformer backbone with neuromorphic enhancements
RoPE positional encoding for sequence understanding
Surrogate gradient training for differentiable spike computation
Inference
In this gist you can find code for inference of this embedding model
1from transformers import AutoTokenizer
2import torch
3import numpy as np
45# Load model (using the custom MTEBLlamaSNNLTCEncoder)6from mteb_benchmark_snn_ltc import MTEBLlamaSNNLTCEncoder
78model = MTEBLlamaSNNLTCEncoder('rootxhacker/arthemis-embedding')910# Encode sentences11sentences =["This is an example sentence","Each sentence is converted"]12embeddings = model.encode(sentences, task_name="similarity")1314print(f"Embeddings shape: {embeddings.shape}")# (2, 768)15print(f"Embedding dimension: {embeddings.shape[1]}")
Usage (Custom Implementation)
For direct usage with the neuromorphic architecture:
python
1import torch
2import torch.nn as nn
3from transformers import AutoTokenizer
45# Initialize tokenizer6tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-small")7tokenizer.pad_token = tokenizer.eos_token
89# Load the model10model = MTEBLlamaSNNLTCEncoder('rootxhacker/arthemis-embedding')1112# Process text13sentences =['This is an example sentence','Each sentence is converted']14embeddings = model.encode(sentences, task_name="embedding_task")1516# Use embeddings for similarity17from scipy.spatial.distance import cosine
18similarity =1- cosine(embeddings[0], embeddings[1])19print(f"Cosine similarity: {similarity:.4f}")
Evaluation
The model has been evaluated on 41 tasks from the MTEB (Massive Text Embedding Benchmark):
MTEB Performance
Task Type
Average Score
Tasks Count
Best Individual Score
Classification
42.78
8
Amazon Counterfactual: 65.43
STS
39.96
8
STS17: 58.48
Clustering
28.54
8
ArXiv Hierarchical: 49.82
Retrieval
12.41
5
Twitter URL: 53.78
Other
13.07
12
Ask Ubuntu: 43.56
Overall MTEB Score: 27.05 (across 41 tasks)
Notable Individual Results
Task
Score
Task Type
Amazon Counterfactual Classification
65.43
Classification
STS17
58.48
Semantic Similarity
Toxic Conversations Classification
55.54
Classification
IMDB Classification
51.69
Classification
SICK-R
49.24
Semantic Similarity
ArXiv Hierarchical Clustering
49.82
Clustering
Banking77 Classification
29.98
Classification
STSBenchmark
36.82
Semantic Similarity
Model Strengths
Classification Excellence: Superior performance on text classification tasks with 42.78% average
