Our team is moving toward a Lakehouse architecture and we are looking to automate our transformation layers. How are you using Generative AI (ChatGPT, Gemini) to write dbt models or SQL transformations? Is it reliable for complex joins in Snowflake, or does it require too much manual fixing to be efficient for a data engineer?
3 answers
We have successfully integrated Generative AI (ChatGPT, Gemini) into our dbt workflow by using it to generate the initial schema documentation and base models. It saves a massive amount of "grunt work" when dealing with hundreds of staging tables. The trick is to provide the DDL as context in the prompt. While it occasionally misses specific window function nuances in Snowflake, it handles 90% of the boilerplate perfectly. It allows our senior engineers to focus on the orchestration logic and data quality testing rather than typing out repetitive SQL statements all day.
Are you finding that the AI-generated SQL follows your specific internal naming conventions and modularity standards?
I use it mainly for regex patterns in Python scripts. It’s a lifesaver because writing complex regex manually is always a headache.
Regex is definitely where the AI shines. It’s one of those tasks that is logically dense but easily handled by LLMs with the right context.
That’s a valid concern, Patrick. We actually solved this by creating a "style guide" prompt. We feed the AI our naming rules first, and then it adheres to them quite strictly, making the code review process much faster for our leads.