This model is a direct copy from Mungert/all-MiniLM-L6-v2-GGUF
Credit goes entirely to him. I made this repo because of a need to have only one model in the repo.
all-MiniLM-L6-v2
This is a sentence-transformers model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
1from sentence_transformers import SentenceTransformer
2sentences =["This is an example sentence","Each sentence is converted"]34model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')5embeddings = model.encode(sentences)6print(embeddings)
Usage (HuggingFace Transformers)
Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
python
1from transformers import AutoTokenizer, AutoModel
2import torch
3import torch.nn.functional as F
45#Mean Pooling - Take attention mask into account for correct averaging6defmean_pooling(model_output, attention_mask):7 token_embeddings = model_output[0]#First element of model_output contains all token embeddings8 input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()9return torch.sum(token_embeddings * input_mask_expanded,1)/ torch.clamp(input_mask_expanded.sum(1),min=1e-9)101112# Sentences we want sentence embeddings for13sentences =['This is an example sentence','Each sentence is converted']1415# Load model from HuggingFace Hub16tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')17model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')1819# Tokenize sentences20encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')2122# Compute token embeddings23with torch.no_grad():24 model_output = model(**encoded_input)2526# Perform pooling27sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])2829# Normalize embeddings30sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)3132print("Sentence embeddings:")33print(sentence_embeddings)
Background
The project aims to train sentence embedding models on very large sentence level datasets using a self-supervised
contrastive learning objective. We used the pretrained nreimers/MiniLM-L6-H384-uncased model and fine-tuned in on a
1B sentence pairs dataset. We use a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset.
Our model is intended to be used as a sentence and short paragraph encoder. Given an input text, it outputs a vector which captures
the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.
By default, input text longer than 256 word pieces is truncated.
Training procedure
Pre-training
We use the pretrained nreimers/MiniLM-L6-H384-uncased model. Please refer to the model card for more detailed information about the pre-training procedure.
Fine-tuning
We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch.
We then apply the cross entropy loss by comparing with true pairs.
Hyper parameters
We trained our model on a TPU v3-8. We train the model during 100k steps using a batch size of 1024 (128 per TPU core).
We use a learning rate warm up of 500. The sequence length was limited to 128 tokens. We used the AdamW optimizer with
a 2e-5 learning rate. The full training script is accessible in this current repository: train_script.py.
Training data
We use the concatenation from multiple datasets to fine-tune our model. The total number of sentence pairs is above 1 billion sentences.
We sampled each dataset given a weighted probability which configuration is detailed in the data_config.json file.
The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder
💬 How to test:
Choose an AI assistant type:
TurboLLM (GPT-4.1-mini)
HugLLM (Hugginface Open-source models)
TestLLM (Experimental CPU-only)
What I’m Testing
I’m pushing the limits of small open-source models for AI network monitoring, specifically:
Function calling against live network services
How small can a model go while still handling:
Automated Nmap security scans
Quantum-readiness checks
Network Monitoring tasks
🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):
✅ Zero-configuration setup
⏳ 30s load time (slow inference but no API costs) . No token limited as the cost is low.
🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!
Other Assistants
🟢 TurboLLM – Uses gpt-4.1-mini :
**It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
Create custom cmd processors to run .net code on Quantum Network Monitor Agents
Real-time network diagnostics and monitoring
Security Audits
Penetration testing (Nmap/Metasploit)
🔵 HugLLM – Latest Open-source models:
🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.
💡 Example commands you could test:
"Give me info on my websites SSL certificate"
"Check if my server is using quantum safe encyption for communication"
"Run a comprehensive security audit on my server"
'"Create a cmd processor to .. (what ever you want)" Note you need to install a Quantum Network Monitor Agent to run the .net code on. This is a very flexible and powerful feature. Use with caution!
Final Word
I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.
If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.
I'm also open to job opportunities or sponsorship.
