I'm working on a sentiment analysis project with very messy social media data. Between emojis, slang, and weird punctuation, my tokenization is a disaster. What is your go-to pipeline for cleaning text before feeding it into a transformer model like BERT?
3 answers
For social media text, the cleaning pipeline should be cautious. If you are using BERT, don't remove stop words or punctuation too aggressively, as BERT relies on sentence structure for context. I recommend using the 'demoji' library to convert emojis into descriptive text . Handle contractions using a standard dictionary and use a library like 'cleantext' to remove URLs and mentions. Lemmatization is usually better than stemming for sentiment analysis because it preserves the semantic meaning of the words, which is vital for understanding nuanced emotions.
Do you find that removing social media handles and hashtags improves your model's performance, or does it lose too much contextual information?
I always suggest using a spell-checker specifically designed for social media. Slang is usually the biggest hurdle for pre-trained models to overcome correctly.
Very true. A custom dictionary for industry-specific slang can drastically increase the accuracy of the final sentiment classification.
I actually keep the hashtags but remove the '#' symbol. Often, the hashtag itself contains the most important sentiment keywords of the entire post.