This is a fine-tuned sentence-transformers model based on all-MiniLM-L12-v2 that has been specifically trained for agent routing and conversation matching. The model determines whether agents should respond to conversations based on semantic similarity.

Model Details

Base Model

Base Model: sentence-transformers/all-MiniLM-L12-v2
Model Architecture: MiniLM-L12 (Microsoft)
Embedding Dimensions: 384
Max Sequence Length: 256 tokens

Training Data

The model was fine-tuned on two custom datasets using triplet training:

semantic_triplet_training_data_round1.pkl: 469 samples
inverse_semantic_triplet_training_data.pkl: 475 samples

Each sample contains:

anchor: Conversation text or agent description
positive: Similar/relevant text to the anchor
negative: Dissimilar/irrelevant text to the anchor

Training Configuration

Loss Function: MultipleNegativesRankingLoss
Batch Size: 16
Learning Rate: 2e-5
Epochs: 1
Warmup Ratio: 0.1
Training Framework: sentence-transformers v2.7.0+

Performance

Evaluation results on held-out test sets:

Semantic Triplets Accuracy: 97.87%
Inverse Semantic Triplets Accuracy: 100.00%

Usage

Direct Usage (Sentence Transformers)

python

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('msugimura/gatekeeper_agent_responding')

# Example: Agent routing for conversation
conversation = "I've been feeling anxious and need help with stress management"
agent_descriptions = [
    "Licensed therapist specializing in anxiety and stress management",
    "Fitness trainer who creates workout routines for stress relief",
    "Financial advisor who helps with investment planning"
]

# Get embeddings
conversation_embedding = model.encode(conversation)
agent_embeddings = model.encode(agent_descriptions)

# Calculate similarities
from sentence_transformers.util import cos_sim
similarities = cos_sim(conversation_embedding, agent_embeddings)

print("Similarity scores:", similarities)
# Expected: Highest similarity with the therapist

API Usage (Portcullis Service)

python

import requests

# Example API call to Portcullis service
response = requests.post("http://localhost:8000/should_agents_respond", json={
    "conversation": "I've been feeling anxious and need help",
    "agent_descriptions": [
        "Licensed therapist specializing in anxiety treatment",
        "Fitness trainer for workout routines",
        "Financial advisor for investments"
    ],
    "threshold": 0.4
})

result = response.json()
print("Qualified agents:", result["qualified_agents"])

Intended Use Cases

Agent Routing: Automatically route conversations to appropriate specialist agents
Conversation Matching: Match user queries with relevant service providers
Semantic Search: Find similar conversations or agent descriptions
Content Recommendation: Recommend agents based on conversation context

Limitations

Domain Specific: Optimized for agent-conversation matching scenarios
English Only: Trained primarily on English text
Context Length: Limited to 256 tokens per input
Training Data: Performance depends on similarity to training domain

Technical Details

Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
  (2): Normalize()
)

Training Process

Data Preprocessing: Cleaned triplet datasets, removed extraneous columns
Multi-Dataset Training: Combined training on both semantic and inverse semantic data
Loss Function: MultipleNegativesRankingLoss with in-batch negatives
Evaluation: TripletEvaluator on held-out validation sets

Citation

If you use this model, please cite:

bibtex

@misc{gatekeeper_agent_responding_2024,
  title={Gatekeeper Agent Responding Model},
  author={Michael Sugimura},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/msugimura/gatekeeper_agent_responding}
}

Contact

For questions or issues, please contact [your-email] or open an issue in the model repository.