Large Language Models (LLMs) have emerged as powerful tools for interpreting multimodal data (e.g., images, audio, text), often surpassing specialized models. In medicine, they hold particular promise for synthesizing large volumes of clinical information into actionable insights and patient-facing digital health applications. Yet, a major limitation remains their inability to handle time series data. To overcome this gap, we present OpenTSLM, a family of Time Series Language Models (TSLMs) created by integrating time series as a native modality to pretrained Large Language Models, enabling natural-language prompting and reasoning over multiple time series of any length [...] 🔗 Read the full paper
schematic_overview
Examples
OpenTSLM models can reason over multiple time series of any length at once, generating findings, captions, and rationales in natural language. We tested these models across a wide range of tasks spanning Human Activity Recognition (HAR) from 3-axis acceleration data, sleep staging from EEG readings, 12-lead ECG question answering, and time series captioning. Some examples are shown below, more are available in the paper.
OpenTSLM is designed to work with Llama and Gemma models, with Llama 3.2 1B as the default. These models are stored in Hugging Face repositories which may require access permissions. Follow these steps to gain access and download:
If you are using Apple's MPS (Metal Performance Shaders) backend (e.g., on Mac with Apple Silicon), you may encounter issues with training or inference. Checkpoints trained with CUDA (NVIDIA GPUs) may not yield good results or may not be fully compatible when loaded and run on MPS. For best results, use the same device type (CUDA or MPS) for both training and inference. CUDA is preferred in general.
Quick Start
bash
1# Run full curriculum with OpenTSLMFlamingo2python curriculum_learning.py --model OpenTSLMSP
34# Run full curriculum with OpenTSLMSP5python curriculum_learning.py --model OpenTSLMFlamingo
67# Run specific stages8python curriculum_learning.py --model OpenTSLMFlamingo --stages stage1_mcq
9python curriculum_learning.py --model OpenTSLMFlamingo --stages stage2_captioning
10python curriculum_learning.py --model OpenTSLMFlamingo --stages stage3_cot
11python curriculum_learning.py --model OpenTSLMFlamingo --stages stage4_sleep_cot
12python curriculum_learning.py --model OpenTSLMFlamingo --stages stage5_ecg_cot
1314# Run multiple stages15python curriculum_learning.py --model OpenTSLMFlamingo --stages stage1_mcq stage2_captioning stage3_cot
1617# Specify device18python curriculum_learning.py --model OpenTSLMFlamingo --device cuda
1920# Use different models21python curriculum_learning.py --model OpenTSLMFlamingo --llm_id meta-llama/Llama-3.2-1B
22python curriculum_learning.py --model OpenTSLMFlamingo --llm_id google/gemma-3-270m
2324# Run only evaluation25python curriculum_learning.py --model OpenTSLMFlamingo --eval_only
Command Line Arguments
--model: Model type (OpenTSLMSP or OpenTSLMFlamingo)
--stages: Stages to run (any combination of: stage1_mcq, stage2_captioning, stage3_cot, stage4_sleep_cot, stage5_ecg_cot)
--device: Device to use (cuda, mps, cpu)
--eval_only: Run evaluation only (requires an existing checkpoint for the stage)
--llm_id: Model ID (default: meta-llama/Llama-3.2-1B, supports Llama and Gemma models)
--batch_size: Batch size for training
--gradient_checkpointing: Enable gradient checkpointing for memory efficiency
--verbose: Enable verbose logging
📁 Results Structure
During training, the script creates a structured results directory:
Each stage automatically loads the best model from the previous stage, ensuring proper curriculum progression. Results are organized by model ID (sanitized), then by model type and stage. The {llm_id} directory name is derived from the --llm_id parameter (e.g., meta-llama/Llama-3.2-1B becomes Llama3_2_1B, google/gemma-3-1b-pt becomes gemma_3_1b_pt).
Authors
This work was made possible through the collaborative efforts of an interdisciplinary team of researchers from computer science, medicine, and engineering. Thank you to all of the Co-authors of the TSLM publication:
Patrick Langer (Stanford University, ETH Zurich)
Thomas Kaar (Stanford University, TUM)
Max Rosenblattl (Stanford University, TUM)
Maxwell A. Xu (Google Research, University of Illinois Urbana-Champaign)
Winnie Chow (Stanford University)
Martin Maritsch (Amazon)
Aradhana Verma (Stanford University)
Brian Han (Stanford University)
Daniel Seung Kim (University of Washington)
Henry Chubb (Stanford University)
Scott Ceresnak (Stanford University)
Aydin Zahedivash (Stanford University)
Alexander Tarlochan Singh Sandhu (Stanford University)
Fatima Rodriguez (Stanford University)
Daniel McDuff (Google Research, University of Washington)
Elgar Fleisch (ETH Zurich, University of St. Gallen)
You can find a list of all current contributors at CONTRIBUTORS.md.
Research Opportunities
Are you a student interested in advancing the frontiers of time-series language models and digital health research? We welcome students to get involved in our research projects!
Visit our Student Research Opportunities page to learn more about current projects and how you can contribute to cutting-edge research at the intersection of AI and healthcare.
For researchers and project partners interested in collaboration opportunities, please reach out to us at digitalhealthresearch@stanford.edu.
