I see a lot of people using AI for poems and images, but can generative AI models actually help with predictive analytics or cleaning messy datasets? I tried using an LLM to write a cleaning script and it was okay, but I’m curious if these models can replace traditional machine learning workflows for structured data problems in a professional setting.
3 answers
While generative AI models aren't a replacement for XGBoost or Random Forests on structured data, they are incredible "assistants." They excel at generating synthetic data to balance skewed classes and can write complex SQL queries or Python scripts to automate tedious data cleaning. However, for the actual prediction, traditional ML is still more efficient and interpretable. A great use case for GenAI in your workflow is "Feature Engineering." You can ask the model to brainstorm potential interactions between variables based on domain knowledge, which you then test using standard statistical methods.
Are you worried that the "black-box" nature of generative AI models makes them unsuitable for the highly regulated audits common in data science?
I've had great success using generative AI models for "Entity Resolution"—matching similar records in messy datasets where exact matches don't exist.
That's a clever application, Donna! Using the semantic understanding of an LLM to realize "St." and "Street" are the same is a huge time-saver for data engineers.
Absolutely. In my field (insurance), if I can't provide a SHAP or LIME explanation for a prediction, the model is useless. GenAI just isn't there yet for explainability.