Standard NLP models always fail at sarcasm because they only look at the text. I am trying to build a customer feedback tool that combines the text transcript with the speaker's "Prosody" or tone of voice. If a customer says "Great job" but with a high-pitch, fast-tempo acoustic profile, the model should flag it as negative. What are the best libraries for extracting acoustic features for multimodal sentiment? And how do we synchronize the timestamp of a specific word with its corresponding audio pitch?
3 answers
To handle the synchronization, you should use "Forced Alignment" tools like the Montreal Forced Aligner (MFA). This will give you word-level timestamps so you can map text tokens directly to audio segments. For feature extraction, Librosa or OpenSMILE are the industry standards for capturing "MFCCs" and pitch. The most effective architecture for this is a "Cross-Attention Transformer." The text encoder creates a query, and the audio encoder provides the keys and values. This allows the model to "hear" the sarcasm by checking if the acoustic intensity contradicts the positive sentiment of the words in the transcript.
Does your dataset include video data as well? Because facial micro-expressions often provide the "tie-breaker" when the audio and text signals are still ambiguous or neutral.
You should definitely look into "DeepMoji" or similar pre-trained models for the text side, and then use a simple Concatenation Fusion with your audio features as a baseline first.
Good suggestion, Karen. Simple concatenation is a great baseline. It's easy to debug before moving into the complex world of attention-based multimodal fusion.
Brian, that's the ultimate goal! We are looking at the "AffWild2" dataset which has labeled video, audio, and text. Adding the "Vision" modality allows us to detect a "smirk" which is a huge indicator of sarcasm. We use a "Coordinated Representation" approach where we keep the three models separate but use a "Contrastive Loss" to align them. If the face says 'happy' but the voice says 'angry' and the text is 'neutral,' the model learns to prioritize the conflicting signals as a sign of complex human emotion like irony or frustration.