Text Encoding: Input text is encoded using the base sentence-transformer model
Ontological Embedding: Pre-trained GNN embeddings capture structural relationships
Fusion Layer: Simple concatenation of text and ontological embeddings

Embedding Flow:

Text: 384 dimensions → 1024 hidden → 128 output
Structure: 403 concepts → GNN → 128 output
Fusion: cross_attention → Final embedding

How It Works

This model combines:

Text Embeddings: Generated using the base sentence-transformer model
Ontological Embeddings: Created by training Graph Neural Networks on OWL ontology structure
Fusion Layer: Combines both embedding types using the specified fusion method

The ontological knowledge helps the model better understand domain-specific relationships and concepts.

Usage

python

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('amphx_all-MiniLM-L6-v2_cross_attention_gat_h1024_o128_cross_entropy_e128_early')

# Generate embeddings
sentences = ['Example sentence 1', 'Example sentence 2']
embeddings = model.encode(sentences)

# Compute similarity
from sentence_transformers.util import cos_sim
similarity = cos_sim(embeddings[0], embeddings[1])

Training Process

This model was created using the on2vec pipeline:

Ontology Processing: The OWL ontology was converted to a graph structure
GNN Training: Graph Neural Networks were trained to learn ontological relationships
Text Integration: Base model text embeddings were combined with ontological embeddings
Fusion Training: The fusion layer was trained to optimally combine both embedding types

Intended Use

This model is particularly effective for:

General domain text processing
Tasks requiring understanding of domain-specific relationships
Semantic similarity in specialized domains
Classification tasks with domain knowledge requirements

Limitations

Performance may vary on domains different from the training ontology
Ontological knowledge is limited to concepts present in the source OWL file
May have higher computational requirements than vanilla text models

Citation

If you use this model, please cite the on2vec framework:

bibtex

@software{on2vec,
  title={on2vec: Ontology Embeddings with Graph Neural Networks},
  author={David Steinberg},
  url={https://github.com/david4096/on2vec},
  year={2024}
}

Created with on2vec 🧬→🤖

ellisdoro/amphx-all-MiniLM-L6-v2_cross_attention_gat_h1024_o128_cross_entropy_e128_early-on2vec-koji-early

amphx_all-MiniLM-L6-v2_cross_attention_gat_h1024_o128_cross_entropy_e128_early

Model Details

Technical Architecture