This is a fine-tuned DistilBERT model for detecting commission-related content in social media profiles and posts. The model classifies text to identify whether an artist's profile/bio/post content shows they are open or closed for commissions, or if the text is unclear.

Model Details

Model Architecture

Base Model: distilbert/distilbert-base-uncased
Model Type: Text Classification
Language: English
License: MIT

Training Data

Sources: Manually scraped profile names, bios, and posts from Bluesky and Twitter by a crowd of furries uploading classifications via a custom extension built specifically to make this dataset
Dataset: Custom dataset of ~1000 rows and user classifications with an equal amount of artificial data to boost pattern recognition

Performance

Metric	Value
Accuracy	95.06%
Precision	95.13%
Recall	95.06%
F1 Score	95.08%

Note: These metrics are not independently verified.

Usage

I recommend a high temperature when inferencing to lower the model's confidence. I use between 1.5 - 3.0.

python

# Example inference #

from transformers import DistilBertForSequenceClassification, DistilBertTokenizer
import torch

# Load model and tokenizer #
model_name = 'zohfur/distilbert-commissions'
tokenizer = DistilBertTokenizer.from_pretrained(model_name)
model = DistilBertForSequenceClassification.from_pretrained(model_name, num_labels=3)

# Example usage #
example_sentences = [
    "Commissions are currently closed.",
    "Check my bio for commission status.",
    "C*mms 0pen on p-site",
    "DM for comms",
    "Taking art requests, dm me",
    "comm completed for personmcperson, thank you <3",
    "open for trades",
    "Comms are not open",
    "Comms form will be open soon, please check back later",
    "~ Furry artist - 25 y.o - he/him - c*mms 0pen: 2/5 - bots dni ~"
]

# Map label integers back to strings #
label_map = {0: 'open', 1: 'closed', 2: 'unclear'}

def predict_with_temperature(model, tokenizer, sentences, temperature=1.5):
    # Prepare input #
    encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    encoded_input = {key: value.to(device) for key, value in encoded_input.items()}
    model.to(device)
    model.eval()

    # Make predictions with temperature scaling #
    with torch.no_grad():
        outputs = model(**encoded_input)
        logits = outputs['logits'] / temperature  # Apply temperature scaling #
        probabilities = torch.softmax(logits, dim=1)

        # Extract predictions and confidence scores #
        predicted_class_indices = torch.argmax(probabilities, dim=1)
        confidences = torch.max(probabilities, dim=1).values

        # Convert to CPU and prepare results #
        predictions = {
            'sentences': sentences,
            'labels': [label_map[idx.item()] for idx in predicted_class_indices],
            'confidences': [score.item() for score in confidences]
        }

        return predictions

def print_predictions(predictions):
    """Print formatted predictions with confidence scores."""
    print("\nClassification Results:")
    print("=" * 50)
    for i, (sentence, label, confidence) in enumerate(zip(
        predictions['sentences'],
        predictions['labels'],
        predictions['confidences']
    ), 1):
        print(f"\n{i}. Sentence: '{sentence}'")
        print(f"   Predicted Label: {label}")
        print(f"   Confidence Score: {confidence:.4f}")

# Make predictions with temperature scaling #
predictions = predict_with_temperature(model, tokenizer, example_sentences, temperature=1.5)

# Print results #
print_predictions(predictions)

Limitations and Biases

Limitations

Language: Only trained on English text
False Positives: Requires a high temperature to avoid false positives (particularly with the words "open" and "closed")
Platform Bias: Trained on Bsky and Twitter/X data, might not perform as well on other platforms like FurAffinity or Instagra

Training Details

Training Procedure

Base Model: DistilBERT base uncased
Fine-tuning: Finetuned using Huggingface's Trainer, evaluated using Trainer and sklearn.metrics
Optimization: Wandb hyperparameter sweep using bayers algorithm to reach highest f1 score

Data Preprocessing

Classifications uploaded voluntarily by crowdsourcing extension users
Problematic unicode characters cleaned from dataset
Label encoding for classification
Class weights computed to adjust weights inversely proportional to class frequencies

Model Card Authors

All credit to original author Zohfur. Base model attributed to distilbert.

Model Card Contact

For questions or concerns about this model, please contact: [ben@zohfur.dog]

zohfur/modernbert-commissions

DistilBERT Commission Detection Model

Model Description