By Yonaira Rivera and Vivek Singh
In today’s digital world, large language models like ChatGPT are important sources of healthcare information, particularly for those who face barriers like insurance issues or language constraints. As part of the IMPACT-NJ project, we are working to make sure that chatbot technologies are accessible to everyone in New Jersey and beyond. We recently published a paper titled “Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation,” highlighting the quality and linguistic fairness of vaccination information provided by ChatGPT and the Centers for Disease Control and Prevention (CDC) in both English and Spanish.
Study Overview
The study evaluated the accuracy, readability, and understandability of vaccination information from ChatGPT and the CDC in both English and Spanish. This is critical as previous evaluations have mostly focused on English, overlooking the needs of non-English speakers in the U.S. We compared responses to common vaccination-related questions using both quantitative and qualitative methods.
Key Findings
- Accuracy: Both ChatGPT and the CDC provided highly accurate information, scoring over 95 out of 100. The trends were similar in both languages.
- Readability: Neither source met the American Medical Association’s recommended readability level of Grade 6. This was a bigger issue in English than in Spanish (average grade level for ChatGPT: English 12.84, Spanish 7.93; recommended: 6.0). Qualitative analysis showed that some Spanish responses used less common words (e.g., “patógeno” instead of “infección”), while some sentence structures read as direct translations from English, making them sound unnatural.
- Understandability: Both sources offered understandable information, but some Spanish texts included English words in quotations (e.g., “herd immunity” and “fake”).
Implications for New Jersey
Given New Jersey’s diverse population that speaks multiple languages, these findings have significant implications for public health policy:
- Enhanced Health Access: Understanding the strengths and weaknesses of AI tools like ChatGPT can help New Jersey enhance health information access, especially for non-English speakers.
- Equity in Health Communication: Ensuring linguistic equity is essential so that all residents, regardless of language proficiency, have access to accurate and understandable health information. This is vital for informed health decisions and improving public health outcomes.
- Contextual Evaluation: While readability scores were similar quantitatively, qualitative analysis revealed issues like unnatural phrasing due to direct translations. Stakeholder engagement and understanding the context are crucial for effective health information.
Moving Forward
To maximize the potential of AI in health information dissemination, addressing the challenges highlighted by the study is crucial. Policymakers in New Jersey should consider:
- Investing in AI improvements by supporting research and development to enhance the readability and linguistic equity of AI-generated health information.
- Fostering community engagement by working with diverse communities to understand their needs and tailor health information to their linguistic and cultural contexts.
- Regulatory oversight that implements guidelines to ensure AI tools used for health communication are accurate, fair, and accessible to all.
By taking these actions, New Jersey can lead the way in integrating advanced technologies into public health while ensuring equity and accuracy in health information access for all residents.