WearableQA: Benchmarking LLM Reasoning on Real-World Wearable Data
WearableQA is a new benchmark designed to evaluate how well AI systems, particularly Large Language Models (LLMs), can reason about real-world health data collected from wearable devices. It uses a large dataset of continuous physiological and behavioral signals from actual users to test LLMs' ability to interpret complex, longitudinal health records, including device noise and individual variations. This benchmark helps identify the strengths and weaknesses of current AI models in health reasoning.
Reading in Plain English mode — toggle below to switch
WearableQA is a new benchmark designed to evaluate how well AI systems, particularly Large Language Models (LLMs), can reason about real-world health data collected from wearable devices. It uses a large dataset of continuous physiological and behavioral signals from actual users to test LLMs' ability to interpret complex, longitudinal health records, including device noise and individual variations. This benchmark helps identify the strengths and weaknesses of current AI models in health reasoning.
Imagine giving an AI a pop quiz based on a patient's entire medical history, including daily heart rate, sleep patterns, and blood tests, all collected over a year. WearableQA is like that pop quiz, but for wearable data, challenging AI to make sense of complex, noisy, and long-term health information, rather than just simple, clean datasets.
How WearableQA Works
- 11. Data Collection: Real-world wearable time series, blood biomarkers, and demographic data from 200 users, each with up to 500 days of daily measurements, are compiled.
- 22. Question Generation: 4,084 10-option multiple-choice questions are constructed using a dual-grounding framework, combining literature-grounded physiological findings with statistically validated population-grounded patterns.
- 33. Question Types: Questions are categorized into 16 types along two axes: data vs. health reasoning (computation vs. physiological interpretation) and single- vs. cross-signal reasoning (individual signal vs. multi-signal integration).
- 44. Model Evaluation: Various LLMs (proprietary and open-source) are tested against these questions, and their performance is measured against a 10% chance baseline to assess their reasoning capabilities.
Accuracy = (Correct Answers / Total Questions) × 100%
Accuracy measures the percentage of questions an LLM answers correctly, indicating its performance on the benchmark. A higher percentage means better reasoning ability.
One example question type might ask an LLM to predict the likelihood of a user experiencing a certain health event based on a week of their heart rate variability and sleep data, requiring the model to integrate multiple signals and interpret physiological patterns. Another might ask to identify an anomaly in a blood biomarker trend over several months.
WearableQA provides a realistic and diagnostic benchmark that effectively differentiates LLM capabilities in health reasoning over complex, real-world wearable data, highlighting that current models are far from solving this challenge.