Limitations
SymptomAI is an exploratory research effort that could represent a significant research advancement in AI-based symptom assessment and demonstrates the potential it could provide for the general public seeking understanding of their symptoms. While a population deployment evaluation reveals the accuracy of symptom assessment through remote patient interviews, there are nuanced limitations when comparing against clinician’s assessments.
Firstly, differential diagnosis itself is an ambiguous task and even reported diagnoses may change and develop longitudinally. A symptom assessment is a snapshot in time and captures the symptoms as they present in that moment. Due to the scale of our deployment, we were unable to control for frequency and timing of symptom reporting. As a result, some participants may have reported their symptoms well before more representative indicators developed, while others may have reported obvious indicators from an informed context after years of experience with chronic illness. Future work may focus on specific illnesses at specific points during symptom development such as early-onset metabolic syndrome or symptoms discussed at the start of respiratory infections. All diagnoses, labels, and disease associations generated during the study are AI-derived for research analysis only and do not constitute confirmed clinical diagnoses or official medical assessments.
Secondly, in our evaluation the clinicians reviewed static chat transcripts and were not given agency to ask their own follow-up questions. Clinicians may have intuitively sourced different information had they directed the symptom interview. Moreover, while recent research has shown that conversational AI systems can source clinical data with a clinician-level of detail and accuracy, such systems may miss alternative signals like body language, visual assessment, medical records, or in the context of primary care, existing rapport with the patient.
In conclusion, we introduce SymptomAI, an investigational conversational AI agent for conducting real-world patient interviews and symptom assessments. We demonstrate SymptomAI’s end-to-end real-world performance through DDx accuracy on a population sample, and show how SymptomAI diagnoses can enable analysis of population-scale signals like wearable biosignals for identifying associations in physiological signals with reported illness.