🧠 Joint fMRI-Text Model
This model jointly predicts cognitive response type, trial type, and generates a 3D fMRI-like brain activation tensor based on natural language input and user-level metadata.
🧩 Inputs
Text: a belief or statement in natural language (e.g., "Handwashing reduces disease risk."), processed using distilbert-base-uncased.
Metadata: a vector of 14 user features including:
Scaled continuous inputs: Age_scaled, Openness_scaled, Conscientiousness_scaled, Extraversion_scaled, Agreeableness_scaled, Neuroticism_scaled, ICAR_Total_scaled, MOCA_scaled, VMN_Sum_scaled
Encoded categorical features: Gender_encoded, Education_encoded, Ethnicity_fused_encoded, Income_level_encoded, VCBS_cat_encoded
🎯 Outputs
Response Type: probabilities over 4 possible response categories
Trial Type: probabilities over 3 trial categories
fMRI Tensor: synthetic output of shape (74 × 74 × 52) representing brain activity across four timepoints
🚀 Example Usage
from transformers import AutoTokenizer
from joint_fmri_model import JointFMRIModelWithHub
import torch
model = JointFMRIModelWithHub.from_pretrained("kenchenxingyu/joint-fmri-text-model")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
text = "I think regular hand washing reduces disease risk."
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
with torch.no_grad():
text_vec = model.text_encoder(inputs["input_ids"]).squeeze(0)
meta_input = torch.rand(1, 14) # Replace with realistic metadata
output = model(text_vec.unsqueeze(0), meta_input)
print("Response:", output["response_probs"])
print("Trial:", output["trial_probs"])
print("fMRI shape:", output["fmri"].shape)
📂 Files
pytorch_model.bin: trained model weights
config.json: model configuration
README.md: this file
🏷 License
This model is released under an open academic research license. For other use cases, please contact the author.
