You can deploy this model and start earning money today!

am5uc/
ServiceNow_Table_Question_Answering

Name: ServiceNow_Table_Question_Answering
Rating: 2.5 (7 reviews)

transformers

safetensors

tapas

table-question-answering

endpoints_compatible

Views

No views yet

ServiceNow Table Answering

Introduction

ServiceNow is a platform that helps businesses automate their processes and workflows. They offer several solutions such as ITSM. Currently, users of ServiceNow generally need to apply filters and/or build dashboards to observe data about tables in ServiceNow, such as incidents and problems. Building dashboards and reports often require the help of developers and may be a hassle just for quick information. Dashboards are useful for visual representation, but it would be useful to be able to ask questions about the data just to a chatbot. I created some sample tables with some ServiceNow fields and used that as data. The task is to create a custom LLM chat/assistant that takes in data from tables ServiceNow uses such as incident, change, and problem, which can then be used to respond to user queries in natural language.

Training Data

For this project, the training data was structured around ServiceNow ITSM tables, specifically Incident, Change, and Problem tables. I used a certain subset fields from Incident, Change, and Problem tables. For example, Problem tables have a problem id, priority, status, root cause, and resolved at field. Since I can’t use official data from in-use ServiceNow instances, which contain private information, I generated a synthetic dataset with custom code. Then, I had to structure that code in sqa format, which is the best format for the model I was using, TAPAS. For this, I had to save each table in a CSV file. The final refined dataset that I would pass in would contain an id, uestion, table_file, answer_coordinates if the answer was in the table itself, the actual answer, and a float answer if the answer was a numeric value not in the data, such as a count. I do have an aggregation_label field as well, which I set right before the training process, but after the train_test_table split. I used the method train_test_split() to obtain the training, validation, and test data. I specifically used a seed of 42:

Example of how the training data appears:

python

train_val_data, test_data = train_test_split(data, test_size=0.1, random_state=42)
# Then split train+validation into train and validation
train_data, val_data = train_test_split(train_val_data, test_size=0.1, random_state=42)

Training Method

I used full-fine tuning. The model did not really need generalization abilities. Its primary purpose is to take ServiceNow Tables and answer queries based on those tables. Keeping some generalization ability would be nice, but isn't really that necessary. PEFT could work as well to prevent catastrophic overfitting, but generalization is not hugely important. The drawbacks I had expected was some generalization loss, but that wasn't really the case.

These were the arguments/hyperparameters, I used. I tried using higher epochs, but those usually caused worse results:

python

        num_train_epochs=1,  # Number of training epochs
    	per_device_train_batch_size=32,  # Batch size per device during training
    	per_device_eval_batch_size=64,   # Batch size per device during evaluation
    	learning_rate=0.00001,
    	warmup_steps=100,   # Number of warmup steps for learning rate scheduler
    	weight_decay=0.01,  # Strength of weight decay
    	evaluation_strategy="steps",  # Evaluate every 'eval_steps'
    	eval_steps=50,  # Evaluation frequency in steps
    	logging_steps=50,  # Log every eval_steps
    	save_steps=150,  # Save model every 500 steps
    	save_total_limit=2,
    	load_best_model_at_end=True,  # Load the best model when finished training
    	metric_for_best_model="eval_loss",  # Metric to use for best model selection

Evaluation

I had three benchmarks, the WikiTableQuestions dataset, the TabFact dataset, and SQA. Fine-tuning did not harm the results of on the WTQ Validation Set and the TabFact Dataset, in which I got accuracies of .3405 and .5005, respectively for both the pre-trained and fine-tuned model. It slightly improved the results on the SQA dataset. There were improvements in the test results after training though. On the test set, there was quite a large jump in accuracy from 0.2033 to 0.4667 after fine-tuning.

Model	Test Set of Synthetic Dataset	Benchmark 1 (WTQ Validation Set)	Benchmark 2 (TabFact)	Benchmark 3 (SQA)
google/tapas-base-finetuned-wtq (before Fine-tuning)	0.2933	0.3405	0.5005	0.2512
google/tapas-base-finetuned-wtq (Fine-tuned)	0.4667	0.3405	0.5005	0.2525
mistralai/Mistral-7B-Instruct-v0.3	0	Exact Match: 0.0346 / Fuzzy Match: 0.4744	0.4995	0.0296
meta-llama/Llama-3.2-1B	0.0133	Exact Match: 0.0593 / Fuzzy Match: 0.2769	0.4995	0.0238

Usage and Intended Uses

This model is designed for question answering over tabular data. It is mostly directed for querying ITSM tables (change, problem, and incident). It is used to answer questions, such as most common issues and number of records in varying categories.

python

saved_path = "am5uc/ServiceNow_Table_Question_Answering"
tokenizer = TapasTokenizer.from_pretrained(saved_path)
model = TapasForQuestionAnswering.from_pretrained(saved_path)

question = "How many Hardware Upgrade changes are still pending?"
table_df = pd.DataFrame({
    "change_id": [
        "CHG3000",
        "CHG3001",
        "CHG3002",
        "CHG3003"
    ],
    "category": [
        "Security Patch",
        "Software Update",
        "Hardware Upgrade",
        "Software Update"
    ],
    "status": [
        "Rejected",
        "In Progress",
        "In Progress",
        "Completed"
    ],
    "approved_by": [
        "",
        "Manager2",
        "",
        "Admin1"
    ],
    "implementation_date": [
        "",
        "",
        "",
        "2023-05-30"
    ]
})

# Tokenize both Question and Table together
inputs = tokenizer(table=table_df, queries=[question], padding='max_length', return_tensors='pt')

# Model prediction
# --- Helper function ---
def get_final_answer(model, tokenizer, inputs, table_df):
    outputs = model(**inputs)

    logits = outputs.logits
    logits_agg = outputs.logits_aggregation

    predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
        inputs,
        logits.detach(),
        logits_agg=logits_agg.detach()
    )

    aggregation_operators = ["NONE", "SUM", "AVERAGE", "COUNT"]

    agg_op_idx = predicted_aggregation_indices[0] if predicted_aggregation_indices else 0
    agg_op = aggregation_operators[agg_op_idx]

    predicted_cells = []
    for coord in predicted_answer_coordinates[0]:
        cell_value = table_df.iat[coord[0], coord[1]]
        predicted_cells.append(cell_value)

    if agg_op == "COUNT":
        answer = len(predicted_cells)
    elif agg_op == "SUM":
        try:
            answer = sum(float(cell) for cell in predicted_cells)
        except ValueError:
            answer = "Could not SUM non-numeric cells"
    elif agg_op == "AVERAGE":
        try:
            answer = sum(float(cell) for cell in predicted_cells) / len(predicted_cells)
        except ValueError:
            answer = "Could not AVERAGE non-numeric cells"
    else:  # NONE
        answer = predicted_cells

    return agg_op, answer, predicted_cells

_, answer, _ = get_final_answer(model, tokenizer, inputs, table_df)

print(answer)

Prompt Format

The prompt for the TAPAS model should be a natural language question paired with a structured table that can be passed in in dataframe format. TAPAS does not work with just one prompt and generally reqires a question and a table dataframe to work.

python

question = "How many Hardware Upgrade changes are still pending?"
table_df = pd.DataFrame({
    "change_id": [
        "CHG3000",
        "CHG3001",
        "CHG3002",
        "CHG3003"
    ],
    "category": [
        "Security Patch",
        "Software Update",
        "Hardware Upgrade",
        "Software Update"
    ],
    "status": [
        "Rejected",
        "In Progress",
        "In Progress",
        "Completed"
    ],
    "approved_by": [
        "",
        "Manager2",
        "",
        "Admin1"
    ],
    "implementation_date": [
        "",
        "",
        "",
        "2023-05-30"
    ]
})

inputs = tokenizer(table=table_df, queries=[question], padding='max_length', return_tensors='pt')

Or you could define table in json format and then have table = pd.DataFrame(table) in your tokenizer.

Expected Output Format

You tokenize the inputs and then perform a specific function to get outputs, which are the aggregation operation, answer, and predicted_cells. You can just grab the middle value which is the predicted answer.

python

# Tokenize both Question and Table together
inputs = tokenizer(table=table_df, queries=[question], padding='max_length', return_tensors='pt')

# Model prediction
 ##--- Helper function ---
 
def get_final_answer(model, tokenizer, inputs, table_df):
    outputs = model(**inputs)

    logits = outputs.logits
    logits_agg = outputs.logits_aggregation

    predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
        inputs,
        logits.detach(),
        logits_agg=logits_agg.detach()
    )

    aggregation_operators = ["NONE", "SUM", "AVERAGE", "COUNT"]

    agg_op_idx = predicted_aggregation_indices[0] if predicted_aggregation_indices else 0
    agg_op = aggregation_operators[agg_op_idx]

    predicted_cells = []
    for coord in predicted_answer_coordinates[0]:
        cell_value = table_df.iat[coord[0], coord[1]]
        predicted_cells.append(cell_value)

    if agg_op == "COUNT":
        answer = len(predicted_cells)
    elif agg_op == "SUM":
        try:
            answer = sum(float(cell) for cell in predicted_cells)
        except ValueError:
            answer = "Could not SUM non-numeric cells"
    elif agg_op == "AVERAGE":
        try:
            answer = sum(float(cell) for cell in predicted_cells) / len(predicted_cells)
        except ValueError:
            answer = "Could not AVERAGE non-numeric cells"
    else:  # NONE
        answer = predicted_cells

    return agg_op, answer, predicted_cells

_, answer, _ = get_final_answer(model, tokenizer, inputs, table_df)
print(answer)

If the question is asking for a count, like how many changes have been completed, the answer would just be one number. If it is asking a question about the most common incident status or root cause, the answer would be the root cause or status the model predicts.

Limitations

The model does still not come close to a 100% accuracy. Possibly using a larger model could help. I have ran into issues with larger table sizes before, so that may be an issue. Once again, possibly a larger model could help. Also this needs to take in a question and table in dataframe format, so more preocessing is necessary. The number of samples it is trained on could also be increased to possibly improve results.

Model Card Authors [optional]

Abhinandan Mekap