DistilBERT for Smart Home Joint Intent Classification and Slot Filling
Model Description
This model is a fine-tuned version of distilbert-base-uncased specifically adapted for joint intent classification and slot filling in the smart home domain. Given a user command related to controlling smart home devices (like lights or thermostats), the model simultaneously predicts:
- The user's intent (e.g.,
set_device_state, get_device_state).
- The relevant slots (entities like
device_name, location, state, attribute_value) within the command, using BIO tagging.
Intended Use and Limitations
Primary Intended Use: This model is intended for hobbyist experimentation and educational purposes related to Natural Language Understanding (NLU) for smart home applications. It can be used as a baseline or starting point for understanding how to build NLU components for simple device control.
Disclaimer: This model is NOT intended for use in production environments. Enfuse.io takes no responsibility for the performance, reliability, security, or any consequences arising from the use of this model in production systems or safety-critical applications. Use in such contexts is entirely at the user's own risk.
Out-of-Scope Use:
- The model is not designed for general conversation or tasks outside the specific smart home intents and slots it was trained on.
- It has no built-in mechanism for handling out-of-domain requests (e.g., asking about weather, playing music). It will likely attempt to classify such requests into one of the known smart home intents, potentially leading to incorrect behavior.
- It has not been evaluated for fairness, bias, or robustness against adversarial inputs.
Training Data
The model was fine-tuned on the enfuse/joint-intent-slot-smarthome dataset, specifically the generated_smarthome_2016_unique.jsonl version containing 2016 unique synthetic examples.
This dataset was generated by Enfuse.io using a combination of
mistralai/Mistral-7B-Instruct-v0.1 and
openai/gpt-4o, followed by validation and de-duplication. Please refer to the
dataset card for more details on the data generation process and limitations.
Training Procedure
Preprocessing
The text was tokenized using the distilbert-base-uncased tokenizer. Slot labels were converted to a BIO tagging scheme. Input sequences were padded and truncated to a maximum length of 128 tokens.
Fine-tuning
The model was fine-tuned using the Hugging Face transformers library Trainer on a single NVIDIA RTX 5090.
- Epochs: 10
- Batch Size: 16 (per device)
- Learning Rate: 5e-5 (with linear decay)
- Optimizer: AdamW
- Precision: FP16
- Dataset Split: 80% Train (1612), 10% Validation (202), 10% Test (202)
- Best Model Selection: The checkpoint with the highest
eval_intent_accuracy on the validation set during training was selected for the final model (corresponding to Epoch 8 or 10 in the 10-epoch run).
Evaluation Results
The following results were achieved on the test set (202 examples) using the best checkpoint saved during training:
- Intent Accuracy: 92.08%
- Slot F1 Score (Micro): 88.00%
- Slot Precision (Micro): 88.00%
- Slot Recall (Micro): 88.00%
(Note: These results are specific to this particular training setup and may vary with different hyperparameters or training runs.)
How to Use
(You would typically add code examples here showing how to load and use the model with the Transformers pipeline or custom code, similar to the logic in infer.py. Since the user requested no code, this section is omitted but would normally be present.)
Model Card Contact
Citation
If you use this model, please cite the dataset:
1@misc{enfuse_smarthome_intent_slot_2024,
2 author = {Enfuse.io},
3 title = {Enfuse Smart Home Joint Intent and Slot Filling Dataset},
4 year = {2024},
5 publisher = {Hugging Face},
6 journal = {Hugging Face Hub},
7 howpublished = {\url{https://huggingface.co/datasets/enfuse/joint-intent-slot-smarthome}}
8}
