Model Card for the Query Parser LLM using Falcon-7B-Instruct

• Implement a bunch of rules, regexps, or grammar parsers (like NLTK grammar parser).
• Collect search queries and to annotate some dataset for NER task.

Model Details

Model Description

• Developed by EmbeddingStudio team:
  • Aleksandr Iudaev [LinkedIn] [Email]
  • Andrei Kostin [LinkedIn] [Email]
  • ML Doom [AI Assistant]
• Funded by EmbeddingStudio team
• Model type: Instruct Fine-Tuned Large Language Model
• Model task: Zero-shot search query parsing
• Language(s) (NLP): English
• License: apache-2.0
• Finetuned from model: IlyaGusev/saiga_mistral_7b_lora
• !Maximal Length Size: we used 1024 for fine-tuning, this is highly different from the original model max_seq_length = 2048
• Tuning Epochs: 3 for now, but will be more later.

• Curating a specific dataset for more precise analysis.
• Exploring various approaches and Large Language Models (LLMs) to identify the most effective solution.
• Publishing a detailed paper to ensure our findings and methodologies can be thoroughly reviewed and verified.

Model Sources

• Repository: code of inference the model will be here
• Paper: Work In Progress
• Demo: Work In Progress

Uses

• Zero-shot Search Query Parsing with porived company market name and filters schema
• Search Query Spell Correction

Instruction format

• Name - name of filter (better to be meaningful).
• Representations - list of possible filter formats (dict):
  • Name - name of representation (better to be meaningful).
  • Type - python base type (int, float, str, bool).
  • Examples - list of examples.
  • Enum - if a representation is enumeration, provide a list of possible values, LLM should map parsed value into this list.
  • Pattern - if a representation is pattern-like (datetime, regexp, etc.) provide a pattern text in any format.

Direct Use

Bias, Risks, and Limitations

Bias

Risks and Limitations

1. Can add extra spaces or remove spaces: 1-2 -> 1 - 2.
2. Can add extra words: 5 -> 5 years.
3. Can not differentiate between <>= and theirs HTML versions <, >, &eq;.
4. Bad with abbreviations.
5. Can add extra .0 for floats and integers.
6. Can add extra 0 or remove 0 for integers with a char postfix: 10M -> 1m.
7. Can hallucinate with integers. For the case like list of positions exactly 7 openings available result can be {'Name': 'Job_Type.Exact_Match', 'Value': 'Full Time'}.
8. We fine-tuned this model with max sequence length = 1024, so it may happen that response will not be JSON-readable.

Recommendations

1. We used synthetic data for the first version of this model. So, we suggest you to precisely test this model on your company's domain, even it's in the list.
2. Use meaningful names for filters and theirs representations.
3. Provide examples for each representation.
4. Try to be compact, model was fine-tuned with max sequence length equal 1024.
5. During the generation use greedy strategy with tempertature 0.05.
6. The result will be better if you align a filters schema with a schema type of the training data.

How to Get Started with the Model

Training Details

Training Data

• We generated lists of possible filters for 72 company categories:
  • Raw version of filters dataset
  • Split by representations
• Select randomly up-to 150 possible combinations (1-3 filters in each combination) of filters, the way each filter's representation appears maximum twice.
• For a given category and combination we generated with GPT-4 Turbo:
  • 2 search queries and theirs parsed version with unstructured parts.
  • 2 search queries and theirs parsed version without unstructured part.
• Using filters, queries and parsed version we prepared 27.42k saiga format instruction

Principles of train / test splitting

• Ability to work well with unseen domain
• Ability to work well with unseen filters
• Ability to work well with unseen queries

1. We put into test split 5 categories, completely separared from train: Automotive, Educational Institutions, Enterprise Software Development, Payment Processing, Professional Social Networks.
2. Also out of each appearing in train company categories, we put aside / removed one filter and queries related to it.

Filters generation details

Queries generation details

• If passed schema doesn't contain possible filter, do not generate query itself or a possible filter
• If a selected representations combination contains enumeration, so we ask to map values in a search query and a parsed version.
• If a selected representations combination contains pattern, so we ask GPT-4 Turbo to be aligned with a pattern

Instructions generation details

1. Zero-Shot query parser should be schema agnostic. Cases like snake_case, CamelCase, http-headers-like should not ruin generation process.
2. Zero-Shot query parser should be spelling errors insensitive.
3. Training instructions should be in the following order:
   • Category
   • Schema
   • Query
   So LLM can be used in the following way: just generate embedding of category -> schema part, so inference will be faster.

1. For each query we generated a version with a mistake
2. Passed to each parsed version an additional field Correct, which contains a corrected version of a search query.
3. For each query we randomly selected and used a case for schema fields and a case for filter and representation names.
4. For each query we additionally generated two instuctions:

• Where did we remove from a provided schema and parsed version one filter
• Where did we remove from a provided schema and parsed version all related filters

Training Procedure

1. Mixed Precision Regime
2. Supervised Fine-Tuning
3. Three epochs with cosine scheduler

Preprocessing [optional]

Training Hyperparameters

Testing Data, Factors & Metrics

Testing Data

Factors Influencing Saiga-Mistral-7B Model Performance

1. Company Category and Domain Knowledge

• Performance may vary based on the specific company category or domain.
• Enhanced performance in domains specifically trained on, such as Educational Institutions, Banking Services, Logistics, etc.

2. Filter Schema Adaptability

• Ability to adapt to various filter schemas.
• Performance in parsing and organizing queries according to different schemas.

3. Handling of Spelling and Syntax Errors

• Robustness in handling spelling errors and syntax variations in queries.

4. Representation and Type Handling

• Capability to handle different data representations (e.g., date formats, enumerations, patterns).
• Accurate processing of various base types (int, float, str, bool).

5. Length and Complexity of Queries

• Impact of the length and complexity of queries on performance.
• Maximum sequence length of 1024 could pose limitations for longer or complex queries.

6. Bias and Ethical Considerations

• Inherited ethical biases from the original model.
• Importance of understanding these biases in different contexts.

7. Limitations in Fine-Tuning and Data Curation

• Limitations such as extra spaces, handling of abbreviations, etc.
• Influence of the extent of training data curation on model accuracy.

8. Specific Use Cases

• Recommended primarily for zero-shot search query parsing and search query spell correction.
• Performance in other use cases might be unpredictable.

9. Training Data Quality and Diversity

• Quality and diversity of synthetic training data.
• Influence on the model's effectiveness across different scenarios.

Testing Procedure

Metrics

Metric Overview

True Positives (TP)

• Definition: A True Positive in our model is counted when the model correctly identifies both the 'Name' and 'Value' in a query, matching the expected results.
• Measurement Method: The TP rate is quantified using the levenshtein_tokenwise function, which calculates the distance between predicted and actual key-value pairs at a token level. We consider a Levenshtein distance of 0.25 or less as acceptable for matching.
• Importance:
  • Token-Level Accuracy: We use token-wise accuracy over traditional character-level Levenshtein distance, which can be overly strict, especially for minor spelling variations. Our token-wise approach prioritizes semantic accuracy.
  • Relevance to Search Queries: Accuracy at the token level is more indicative of the model's ability to understand and parse user intent in search queries.

Generation Strategy

• Approach: The model generates responses based on input queries with a maximum token length set to 1000, employing a sampling strategy (do_sample=True), and a low temperature setting of 0.05. This controlled randomness in generation ensures a variety of accurate and relevant responses.
• Impact on TP:
  • The low temperature setting directly influences the TP rate by reducing the randomness in the model's predictions. With a lower temperature, the model is more likely to choose the most probable word in a given context, leading to more accurate and consistent outputs. This is particularly crucial in search query parsing, where understanding and interpreting user input with high precision is vital.

Additional Metrics

• False Positives (FP) and False Negatives (FN): These metrics are monitored to provide a comprehensive view of the model's predictive capabilities.
• Precision, Recall, F1 Score, Accuracy: These standard metrics complement our TP-focused assessment, providing a rounded picture of the model's performance in various aspects.

Motivation for Metric Choice

• Alignment with User Intent: Focusing on token-wise accuracy ensures the model's performance closely mirrors the structure and intent typical in search queries.
• Robustness Against Query Variations: This metric approach makes the model adaptable to the varied formulations of real-world search queries.
• Balancing Precision and Recall: Our method aims to balance the model's ability not to miss relevant key-value pairs (high recall) while not over-identifying irrelevant ones (high precision).

Total metrics

Unseen domains metrics

Environmental Impact

• Hardware Type: NVIDIA Tesla V100
• Hours used: 72
• Cloud Provider: Google Cloud
• Compute Region: us-west-1
• Carbon Emitted: 6.48

Technical Specifications

Model Architecture and Objective

• Base model: IlyaGusev/saiga_mistral_7b_lora.
• Quantization Configuration: Uses settings like load_in_4bit and bnb_4bit_quant_type for model quantization.

Compute Infrastructure

Hardware

Software

• Python 3.9+
• CUDA 11.7.1
• NVIDIA Compatible Drivers
• Torch 2.0.0

More Information / About us

Features

1. 🔄 Turn your vector database into a full-cycle search engine
2. 🖱️ Collect users feedback like clickstream
3. 🚀 (*) Improve search experience on-the-fly without frustrating wait times
4. 📊 (*) Monitor your search quality
5. 🎯 Improve your embedding model through an iterative metric fine-tuning procedure
6. 🆕 (*) Use the new version of the embedding model for inference

1. Data source
2. Vector database
3. Clickstream database
4. Embedding model

Model Card Authors and Contact

• Aleksandr Iudaev [LinkedIn] [Email]
• Andrei Kostin [LinkedIn] [Email]
• ML Doom [AI Assistant]

Framework versions

• PEFT 0.5.0
• Datasets 2.16.1
• BitsAndBytes 0.41.0
• PyTorch 2.0.0
• Transformers 4.36.2
• TRL 0.7.7