data science Community

We are struggling to keep our data catalog updated. Can we use Generative AI (ChatGPT, Gemini) to automatically generate descriptions for tables and columns based on the data profiling? I want to auto...

I'm trying to figure out the fastest way to grow my career in IT today by looking at the data sector. Is Data Engineering still the most in-demand skill, or has the focus shifted entirely toward D...

We are using Kafka to collect logs for training a predictive maintenance AI. Sometimes logs arrive out of order due to network issues on the edge devices. How does Kafka's Watermarking and windowi...

We use Fivetran to pull data from Salesforce and Zendesk into BigQuery. Every time their API updates or a user adds a custom field, our downstream dbt models break. How are you automating 'Schema ...

I’m seeing a lot of "AutoML" tools that claim to do everything from cleaning data to picking models. For those in the field, do you think we are at risk, or is there a layer of busines...

I have a transaction log where identical values appear across multiple rows. If I try to find the index of a specific element or value in a Pandas DataFrame, how do I ensure I capture every single occ...

I've heard the saying "garbage in, garbage out," but as a new analyst, I'm terrified of accidentally deleting important data while cleaning. What are the best practices for handling ...

I have been researching Data Science roles and noticed that almost every job description lists Python Programming as a core requirement. I am curious about the specific features of the language that m...

Our analytics group is launching a real-time fraud monitoring mechanism. What is Apache Spark used for when building continuous streaming software pipelines that require sub-second event ingestions an...

I'm building a Retrieval-Augmented Generation (RAG) system for our technical manuals. My current setup often retrieves irrelevant chunks, causing the LLM to give wrong answers. Should I focus on c...