I've noticed dbt has introduced Python models recently. For those of you in heavy data engineering roles, have you found this to be a legitimate replacement for complex PySpark scripts, or is it better suited for simple transformations that SQL can't handle efficiently?
3 answers
In my experience, dbt Python models are a fantastic bridge for teams that are already centered around dbt's ecosystem but need libraries like Scikit-learn or Pandas for specific logic. However, they aren't a full replacement for standalone PySpark scripts when you need deep control over cluster configurations or hardware-level optimizations. dbt Python models run within the compute of your warehouse (like Snowflake or Databricks), which limits some flexibility. Use them for "last-mile" transformations, but keep your heavy-lifting, large-scale data ingestion in optimized Spark.
What specific libraries are you looking to use within your dbt models that you currently find difficult to manage in your existing Spark environment?
It’s mostly about the developer experience. dbt makes documentation and testing so much easier, but for massive sharding and partitioning, Spark is still the king of the hill.
Good point. If the dataset isn't multi-terabyte, the convenience of having everything—SQL and Python—in one dbt project is a massive win for maintainability.
We mainly need to use specialized NLP libraries for sentiment analysis on customer reviews before they hit our analytics table. Managing these dependencies in Spark has been a headache.