In my machine learning classes, I hear a lot about feature engineering. But in NLP, it seems like we just throw text into a tokenizer and then into a model. Is feature engineering dead?
What kind of 'hand-crafted' features (like text length, sentiment scores, reading ease) are still useful to add to a model? I feel like I might be missing out on performance by not doing enough.
Feature engineering remains essential for injecting domain-specific constraints, business logic, and metadata that raw embedding models often fail to capture.
7 answers
The choice between raw text and feature-augmented models depends on the signal-to-noise ratio of your dataset. Using raw tokens is excellent for generalizable language tasks where the structure is consistent, but adding hand-crafted features is necessary when the model must account for external constraints that aren't present in the text stream itself. It is a balancing act between the inductive bias of your architecture and the empirical reality of your domain data.
Feature engineering is not dead, it has just shifted from linguistic rules to high-level metadata injection. While transformers capture syntax and semantics inherently, adding exogenous signals like transaction timestamps or account-specific flags remains critical in high-stakes environments where model interpretability is a prerequisite for production deployment.
I recall working on a clinical trial sentiment analysis project where the raw transformer output consistently failed to distinguish between patient frustration and legitimate adverse event reporting. We solved this by integrating length-of-stay and medication frequency as independent variables alongside the embeddings.
It taught me that domain-specific constraints provide a safety net that pure deep learning models simply cannot replicate on their own. Contextual metadata is often the deciding factor in whether a system is clinically valid or just a parlor trick.
You should absolutely continue to use auxiliary features because they steer your model toward business logic that embeddings ignore.
- Entity density ratios
- Reading level complexity scores
- Categorical metadata like region or source ID
- Temporal decay factors for news relevance
I apologize if I am overthinking this, Myrtle, but your mention of reading level complexity is quite intriguing. I have been hesitant to add variables, yet this seems very logically sound.
Myrtle, this list is a lifesaver. I’ve been struggling with my embeddings ignoring context, and my pipeline is barely holding together. These auxiliary features might actually save my upcoming deployment.
Finally, Myrtle! I’m drowning in manual data cleaning over here. Using temporal decay factors is exactly the kind of direct approach I need to get this off my plate immediately.
Feature engineering is definitely not dead, but most people are doing it wrong. Throwing random scores at a model usually creates noise, not performance gains. Unless you can prove that a specific variable provides a signal orthogonal to what the attention heads are already seeing, you are just bloating your feature space and increasing latency for zero real-world benefit.
Dwayne, I think I am definitely guilty of bloating my feature space. I get so nervous about missing a signal that I end up adding useless variables, which is probably quite foolish.
I’m so worried I’ve been doing this wrong, Dwayne. It’s hard to know if my variables are orthogonal to the attention heads, but your advice makes me feel a bit more grounded.
Look, modern NLP models represent the state space remarkably well, but they are notoriously bad at counting or handling hard logic. If your use case requires parsing formal documents or legal contracts, you have to build features that explicitly capture structural components like header hierarchies or bullet point depth. Relying solely on tokenization is a rookie mistake if your downstream task depends on rigid document architecture rather than just nuance or sentiment.
Engineered features act as anchors. When I build models for market microstructure, the transformer provides the sentiment, but the numeric features provide the hard boundaries. You need both to stop the model from hallucinating or ignoring specific numeric triggers. Stop thinking of it as an either-or scenario and start thinking of it as layering signals.
Stop worrying about what is dead and start worrying about what actually improves your F1-score. If adding text length or sentiment scores works, use them. If it doesn't, drop them. It really is that simple.
Jean, your distinction regarding inductive bias is remarkably astute. I often worry my own domain features introduce more noise than value, so I appreciate your formal analysis of this balancing act.