Research: The Dark Side of Sentiment Analysis: An Exploratory Review Using Lexicons, Dictionaries, and a Statistical Monkey and Chimp

January 18, 2022

by Marcia Hannigan

Sentiment analysis (SA) uses a combination of natural language processing (NLP) methods to analyze a text to estimate the implied sentiment. Sentiment may be classified into categories such as positive, negative or neutral, or be measured by a range of numerical scores.  It is used frequently in business to determine consumer attitudes towards products where data such as customer reviews may be too voluminous for traditional analysis, but the use of SA and NLP helps to identify meaningful trends.  

SA can be used to analyze a wide range of texts, including short snippets of text such as Twitter feeds to generate meaningful insights. A new study by Jim Samuel (Rutgers University), Gavin C. Rozzi (Rutgers University),  Ratnakar Palle (Apple, Inc.) in SSRN (Jan. 2022) reviews known issues with SA as documented by prior research and then compares the application of multiple of-the-shelf lexicon and dictionary methods to stock market and vaccine tweets. The intention of this research is to identify and discuss critical aspects of the “dark side” of SA and develop a conceptual discussion of the characteristics of the dark side.

The study demonstrates flaws with a plug-and-play approach to SA and concludes with notes on conceptual solutions for the dark side of SA. It points to future strategies that could be used to improve the accuracy of SA. This research can help align researcher and practitioner expectations to understand the limits and boundaries of NLP-based solutions for sentiment analysis and estimation.

The study concludes that lexicons and dictionaries help in implementing sentiment analysis. While an in-depth analysis of SA is necessary before drawing conclusions, it is important to know the limits of SA methods and tools. SA modeling may need to be customized for some situations while acknowledging the absence of satisfactory SA solutions for other situations. SA tools are very useful and must continue to be used for research and practice – however, as demonstrated and described in this study, it is vital to understand the conflicts and ways to acknowledge and address them.

It is expected that this study will lead to deeper attention to applied SA and spur new strategies for the improvement of sentiment analysis research and practice.

Read the full study.

Recent Posts

Sophia Jones, Committed to Cultural Competence

Sophia Jones, PhD, has been a Public Health part-time lecturer at the Bloustein School since 2016. She was recently featured on "Meet the People of Rutgers." Sophia Jones, Committed to Cultural Competence Jeff Arban/Rutgers University The Basics Title: Program...

Upcoming Events

Event Series CAREERS

Career Virtual Drop-ins

Virtual

Bloustein Career Development Specialists Cheryl Egan and Andrea Garrido will be in a Zoom Room on Monday's beginning January 22, 2024 (excluding holidays and spring break) to answer questions, provide […]