NeuroBERT-Mini is a lightweight NLP model derived from google/bert-base-uncased, optimized for real-time inference on edge and IoT devices. With a quantized size of ~35MB and approximately 10 million parameters, it enables efficient contextual language understanding in resource-constrained environments such as mobile apps, wearables, microcontrollers, and smart home devices.
In addition to its edge-ready design, NeuroBERT-Mini is suitable for a wide range of general-purpose NLP tasks, including text classification, intent detection, semantic similarity, and information extraction. Its compact architecture makes it ideal for offline, privacy-first applications that demand fast, on-device language processing without relying on constant cloud connectivity.
Whether you're building a chatbot, a smart assistant, or an embedded NLP module, NeuroBERT-Mini offers a strong balance of performance and portability for both specialized and mainstream NLP applications.
Download quantized model weights from the Hugging Face model hub.
Extract and integrate into your edge/IoT application.
Quickstart: Masked Language Modeling
Predict missing words in IoT-related sentences with masked language modeling:
python
1from transformers import pipeline
23# Unleash the power4mlm_pipeline = pipeline("fill-mask", model="boltuix/NeuroBERT-Mini")56# Test the magic7result = mlm_pipeline("Please [MASK] the door before leaving.")8print(result[0]["sequence"])# Output: "Please open the door before leaving."
Quickstart: Text Classification
Perform intent detection or text classification for IoT commands:
python
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
34# 🧠 Load tokenizer and classification model5model_name ="boltuix/NeuroBERT-Mini"6tokenizer = AutoTokenizer.from_pretrained(model_name)7model = AutoModelForSequenceClassification.from_pretrained(model_name)8model.eval()910# 🧪 Example input11text ="Turn off the fan"1213# ✂️ Tokenize the input14inputs = tokenizer(text, return_tensors="pt")1516# 🔍 Get prediction17with torch.no_grad():18 outputs = model(**inputs)19 probs = torch.softmax(outputs.logits, dim=1)20 pred = torch.argmax(probs, dim=1).item()2122# 🏷️ Define labels23labels =["OFF","ON"]2425# ✅ Print result26print(f"Text: {text}")27print(f"Predicted intent: {labels[pred]} (Confidence: {probs[0][pred]:.4f})")
Output:
plaintext
1Text: Turn off the fan
2Predicted intent: OFF (Confidence: 0.5328)
Note: Fine-tune the model for specific classification tasks to improve accuracy.
Evaluation
NeuroBERT-Mini was evaluated on a masked language modeling task using 10 IoT-related sentences. The model predicts the top-5 tokens for each masked word, and a test passes if the expected word is in the top-5 predictions.
Test Sentences
Sentence
Expected Word
She is a [MASK] at the local hospital.
nurse
Please [MASK] the door before leaving.
shut
The drone collects data using onboard [MASK].
sensors
The fan will turn [MASK] when the room is empty.
off
Turn [MASK] the coffee machine at 7 AM.
on
The hallway light switches on during the [MASK].
night
The air purifier turns on due to poor [MASK] quality.
air
The AC will not run if the door is [MASK].
open
Turn off the lights after [MASK] minutes.
five
The music pauses when someone [MASK] the room.
enters
Evaluation Code
python
1from transformers import AutoTokenizer, AutoModelForMaskedLM
2import torch
34# 🧠 Load model and tokenizer5model_name ="boltuix/NeuroBERT-Mini"6tokenizer = AutoTokenizer.from_pretrained(model_name)7model = AutoModelForMaskedLM.from_pretrained(model_name)8model.eval()910# 🧪 Test data11tests =[12("She is a [MASK] at the local hospital.","nurse"),13("Please [MASK] the door before leaving.","shut"),14("The drone collects data using onboard [MASK].","sensors"),15("The fan will turn [MASK] when the room is empty.","off"),16("Turn [MASK] the coffee machine at 7 AM.","on"),17("The hallway light switches on during the [MASK].","night"),18("The air purifier turns on due to poor [MASK] quality.","air"),19("The AC will not run if the door is [MASK].","open"),20("Turn off the lights after [MASK] minutes.","five"),21("The music pauses when someone [MASK] the room.","enters")22]2324results =[]2526# 🔁 Run tests27for text, answer in tests:28 inputs = tokenizer(text, return_tensors="pt")29 mask_pos =(inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]30with torch.no_grad():31 outputs = model(**inputs)32 logits = outputs.logits[0, mask_pos,:]33 topk = logits.topk(5, dim=1)34 top_ids = topk.indices[0]35 top_scores = torch.softmax(topk.values, dim=1)[0]36 guesses =[(tokenizer.decode([i]).strip().lower(),float(score))for i, score inzip(top_ids, top_scores)]37 results.append({38"sentence": text,39"expected": answer,40"predictions": guesses,41"pass": answer.lower()in[g[0]for g in guesses]42})4344# 🖨️ Print results45for r in results:46 status ="✅ PASS"if r["pass"]else"❌ FAIL"47print(f"\n🔍 {r['sentence']}")48print(f"🎯 Expected: {r['expected']}")49print("🔝 Top-5 Predictions (word : confidence):")50for word, score in r['predictions']:51print(f" - {word:12} | {score:.4f}")52print(status)5354# 📊 Summary55pass_count =sum(r["pass"]for r in results)56print(f"\n🎯 Total Passed: {pass_count}/{len(tests)}")
Sample Results (Hypothetical)
Sentence: She is a [MASK] at the local hospital. Expected: nurse Top-5: [doctor (0.35), nurse (0.30), surgeon (0.20), technician (0.10), assistant (0.05)] Result: ✅ PASS
Sentence: Turn off the lights after [MASK] minutes. Expected: five Top-5: [ten (0.40), two (0.25), three (0.20), fifteen (0.10), twenty (0.05)] Result: ❌ FAIL
Total Passed: ~8/10 (depends on fine-tuning).
The model performs well in IoT contexts (e.g., “sensors,” “off,” “open”) but may require fine-tuning for numerical terms like “five.”
Evaluation Metrics
Metric
Value (Approx.)
✅ Accuracy
~92–97% of BERT-base
🎯 F1 Score
Balanced for MLM/NER tasks
⚡ Latency
<40ms on Raspberry Pi
📏 Recall
Competitive for lightweight models
Note: Metrics vary based on hardware (e.g., Raspberry Pi 4, Android devices) and fine-tuning. Test on your target device for accurate results.
Use Cases
NeuroBERT-Mini is designed for edge and IoT scenarios with constrained compute and connectivity. Key applications include:
Smart Home Devices: Parse commands like “Turn [MASK] the coffee machine” (predicts “on”) or “The fan will turn [MASK]” (predicts “off”).
IoT Sensors: Interpret sensor contexts, e.g., “The drone collects data using onboard [MASK]” (predicts “sensors”).
Wearables: Real-time intent detection, e.g., “The music pauses when someone [MASK] the room” (predicts “enters”).
Mobile Apps: Offline chatbots or semantic search, e.g., “She is a [MASK] at the hospital” (predicts “nurse”).
Voice Assistants: Local command parsing, e.g., “Please [MASK] the door” (predicts “shut”).
Toy Robotics: Lightweight command understanding for interactive toys.
Fitness Trackers: Local text feedback processing, e.g., sentiment analysis.
Car Assistants: Offline command disambiguation without cloud APIs.
Hardware Requirements
Processors: CPUs, mobile NPUs, or microcontrollers (e.g., ESP32, Raspberry Pi)
Storage: ~35MB for model weights (quantized for reduced footprint)
Memory: ~80MB RAM for inference
Environment: Offline or low-connectivity settings
Quantization ensures efficient memory usage, making it suitable for microcontrollers.
Trained On
Custom IoT Dataset: Curated data focused on IoT terminology, smart home commands, and sensor-related contexts (sourced from chatgpt-datasets). This enhances performance on tasks like command parsing and device control.
Fine-tuning on domain-specific data is recommended for optimal results.
Fine-Tuning Guide
To adapt NeuroBERT-Mini for custom IoT tasks (e.g., specific smart home commands):
Prepare Dataset: Collect labeled data (e.g., commands with intents or masked sentences).
Fine-Tune with Hugging Face:
python
1#!pip uninstall -y transformers torch datasets2#!pip install transformers==4.44.2 torch==2.4.1 datasets==3.0.134import torch
5from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments
6from datasets import Dataset
7import pandas as pd
89# 1. Prepare the sample IoT dataset10data ={11"text":[12"Turn on the fan",13"Switch off the light",14"Invalid command",15"Activate the air conditioner",16"Turn off the heater",17"Gibberish input"18],19"label":[1,1,0,1,1,0]# 1 for valid IoT commands, 0 for invalid20}21df = pd.DataFrame(data)22dataset = Dataset.from_pandas(df)2324# 2. Load tokenizer and model25model_name ="boltuix/NeuroBERT-Mini"# Using NeuroBERT-Mini26tokenizer = BertTokenizer.from_pretrained(model_name)27model = BertForSequenceClassification.from_pretrained(model_name, num_labels=2)2829# 3. Tokenize the dataset30deftokenize_function(examples):31return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=64)# Short max_length for IoT commands3233tokenized_dataset = dataset.map(tokenize_function, batched=True)3435# 4. Set format for PyTorch36tokenized_dataset.set_format("torch", columns=["input_ids","attention_mask","label"])3738# 5. Define training arguments39training_args = TrainingArguments(40 output_dir="./iot_neurobert_results",41 num_train_epochs=5,# Increased epochs for small dataset42 per_device_train_batch_size=2,43 logging_dir="./iot_neurobert_logs",44 logging_steps=10,45 save_steps=100,46 evaluation_strategy="no",47 learning_rate=3e-5,# Adjusted for NeuroBERT-Mini48)4950# 6. Initialize Trainer51trainer = Trainer(52 model=model,53 args=training_args,54 train_dataset=tokenized_dataset,55)5657# 7. Fine-tune the model58trainer.train()5960# 8. Save the fine-tuned model61model.save_pretrained("./fine_tuned_neurobert_iot")62tokenizer.save_pretrained("./fine_tuned_neurobert_iot")6364# 9. Example inference65text ="Turn on the light"66inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=64)67model.eval()68with torch.no_grad():69 outputs = model(**inputs)70 logits = outputs.logits
71 predicted_class = torch.argmax(logits, dim=1).item()72print(f"Predicted class for '{text}': {'Valid IoT Command'if predicted_class ==1else'Invalid Command'}")
Deploy: Export the fine-tuned model to ONNX or TensorFlow Lite for edge devices.
Comparison to Other Models
Model
Parameters
Size
Edge/IoT Focus
Tasks Supported
NeuroBERT-Mini
~10M
~35MB
High
MLM, NER, Classification
NeuroBERT-Tiny
~5M
~15MB
High
MLM, NER, Classification
DistilBERT
~66M
~200MB
Moderate
MLM, NER, Classification
TinyBERT
~14M
~50MB
Moderate
MLM, Classification
NeuroBERT-Mini offers a balance between size and performance, making it ideal for edge devices with slightly more resources than those targeted by NeuroBERT-Tiny.
