NJSPL: Ensuring Accurate & Equitable Vaccination Info

January 27, 2025

By Yonaira Rivera and Vivek Singh 

In today’s digital world, large language models like ChatGPT are important sources of healthcare information, particularly for those who face barriers like insurance issues or language constraints. As part of the IMPACT-NJ project, we are working to make sure that chatbot technologies are accessible to everyone in New Jersey and beyond. We recently published a paper titled Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation,” highlighting the quality and linguistic fairness of vaccination information provided by ChatGPT and the Centers for Disease Control and Prevention (CDC) in both English and Spanish.

Study Overview

The study evaluated the accuracy, readability, and understandability of vaccination information from ChatGPT and the CDC in both English and Spanish. This is critical as previous evaluations have mostly focused on English, overlooking the needs of non-English speakers in the U.S. We compared responses to common vaccination-related questions using both quantitative and qualitative methods.

Key Findings

  • Accuracy: Both ChatGPT and the CDC provided highly accurate information, scoring over 95 out of 100. The trends were similar in both languages.
  • Readability: Neither source met the American Medical Association’s recommended readability level of Grade 6. This was a bigger issue in English than in Spanish (average grade level for ChatGPT: English 12.84, Spanish 7.93; recommended: 6.0). Qualitative analysis showed that some Spanish responses used less common words (e.g., “patógeno” instead of “infección”), while some sentence structures read as direct translations from English, making them sound unnatural.
  • Understandability: Both sources offered understandable information, but some Spanish texts included English words in quotations (e.g., “herd immunity” and “fake”).

Implications for New Jersey

Given New Jersey’s diverse population that speaks multiple languages, these findings have significant implications for public health policy:

  • Enhanced Health Access: Understanding the strengths and weaknesses of AI tools like ChatGPT can help New Jersey enhance health information access, especially for non-English speakers.
  • Equity in Health Communication: Ensuring linguistic equity is essential so that all residents, regardless of language proficiency, have access to accurate and understandable health information. This is vital for informed health decisions and improving public health outcomes.
  • Contextual Evaluation: While readability scores were similar quantitatively, qualitative analysis revealed issues like unnatural phrasing due to direct translations. Stakeholder engagement and understanding the context are crucial for effective health information.

Moving Forward

To maximize the potential of AI in health information dissemination, addressing the challenges highlighted by the study is crucial. Policymakers in New Jersey should consider:

  • Investing in AI improvements by supporting research and development to enhance the readability and linguistic equity of AI-generated health information.
  • Fostering community engagement by working with diverse communities to understand their needs and tailor health information to their linguistic and cultural contexts.
  • Regulatory oversight that implements guidelines to ensure AI tools used for health communication are accurate, fair, and accessible to all.

By taking these actions, New Jersey can lead the way in integrating advanced technologies into public health while ensuring equity and accuracy in health information access for all residents.

Recent Posts

Lindenfeld Assesses Substance Use Services Ads in NY

What substance use services are advertised by local governments? An analysis of data from county websites in New York state Objective To assess the substance use disorder (SUD) prevention and response activities that county governments in New York advertise. Study...

A Valentine’s Inspired Interview with the Poppers

Frank J. Popper is Professor Emeritus in the Bloustein School of Planning and Public Policy at Rutgers University, where he also participated in the American Studies, Geography, and Political Science Departments and the School of Arts and Sciences Honors Program. He...

VTC-POET: Springwood Avenue Heritage Walk

Did you know that between the 1930s and 1970s, the West Side of Asbury Park, NJ, was a vibrant hub of African American life and culture? The community, shaped by Black and African American, Italian, Jewish, and Latino residents and business owners, created famous...

New Briefs: Rutgers Child Care Research Collaborative

The Rutgers Child Care Research Collaborative has released two new research briefs by Heldrich Center for Workforce Development researchers. New Jersey’s Childcare Workforce: An Examination of Internet Job Postings from October 2023, by Liana Lin, Brittney Donovan,...

Bhuyan Receives Leadership Excellence Award

  Soumitra Bhuyan, Executive Director of Health Administration Programs and Associate Professor at Rutgers University-New Brunswick’s Edward J. Bloustein School of Planning and Public Policy, is the recipient of a National Leadership Excellence Award. Jointly...

Upcoming Events

[tribe_events view="list" tribe-bar="false" events_per_page="3" filter-bar="false"]