We are building a centralized Data Lake on Google Cloud Storage but we need to restrict access so that analysts in Europe only see EU customer data. Since the data is just files (CSV/JSON), how can we enforce row-level and column-level security without moving everything into a relational database?
3 answers
Securing raw files is the hardest part of data lake management. On GCP, you should look into BigLake, which allows you to define fine-grained access control (FGAC) policies that apply even when querying files in Cloud Storage. You can define a "policy tag" for PII and then restrict access based on the user's IAM role. Another popular approach is using an abstraction layer like Trino or Starburst, which has a built-in security engine to mask columns or filter rows dynamically based on the user's location, ensuring you stay compliant with GDPR mandates.
Are you planning to handle the data masking at the storage layer via encryption keys, or do you prefer a dynamic masking approach at the query execution level?
Using Apache Ranger is the standard for Hadoop-based lakes. It provides a very robust framework for both row-level filtering and column-level masking across the stack.
Sarah is right. Ranger is incredibly powerful, though it can be a bit of a beast to set up if you aren't already running a managed service like EMR or Dataproc.
Brandon, we would much prefer dynamic masking at the query level. If we encrypt files with different keys, it makes our aggregation jobs across regions almost impossible to manage. We want a central policy engine where we can just say "If User.Region != Data.Region, then Nullify(Customer_Name)". That seems like the most scalable way to handle our upcoming audit without breaking our current ETL pipelines.