We are designing a Hybrid Cloud architecture that requires seamlessly integrating our existing on-premises file systems (NAS/SAN) with public Object Storage (AWS S3 or Azure Blob) for backup, archiving, and bursting compute. What are the most reliable Cloud Technology solutions (e.g., Cloud Storage Gateway, Data Sync Services) and best practices for achieving this integration while maintaining local performance and ensuring data consistency? I need advice on how to structure the Cloud Storage tiers and what key governance controls must be in place to manage security and Access Control across the hybrid boundary.
3 answers
Utilize a Cloud Storage Gateway for Hybrid Cloud local performance and Data Consistency with on-premises systems. Enforce strict Governance by centralizing Access Control through your Active Directory, and use Data Sync Services for bursting compute.
The key to successful Hybrid Cloud Storage integration is using a Cloud Storage Gateway (e.g., AWS Storage Gateway, Azure StorSimple). This solution provides a local cache for frequently accessed data, ensuring high local performance, while asynchronously replicating the data to Object Storage in the cloud for durability and archiving. Best practices include: 1) Data Tiering at the Source: Immediately move inactive or cold data to the gateway's Tape or Archive mode. 2) Centralized Governance: All Access Control must be managed centrally via your on-premises Active Directory or IAM system, with the Cloud Storage Gateway handling the secure identity mapping to the cloud Object Storage policies. 3) Data Consistency: The gateway or specialized Data Sync Services must provide versioning and checksum verification to guarantee Data Consistency between the on-premises file system and the remote Cloud Storage buckets.
The Cloud Storage Gateway for local performance sounds like the necessary link. However, what is the latency impact on bursting compute workloads (e.g., running analytics in the cloud on data sourced from on-premises)? If the Cloud Storage is acting as a data lake for cloud compute, is it more efficient to bypass the gateway entirely for certain workloads and use dedicated Data Sync Services that specialize in fast, bulk data movement directly to the cloud? Does the Hybrid Cloud model necessitate using both technologies simultaneously for different use cases?
Adam, you are absolutely right: the Hybrid Cloud necessitates using both technologies! The Cloud Storage Gateway is perfect for applications that need the local file interface and high local performance (e.g., end-user access). However, for bursting analytics (Data Science), where the goal is to run massive compute in the cloud, you should use Data Sync Services (or dedicated transfer appliances) to move the entire dataset directly to the cloud's Object Storage. This eliminates the gateway as a potential latency bottleneck for the cloud compute, ensuring the cloud-native applications get the best possible throughput and Data Consistency without being tied to on-premises performance.
Christopher's advice is solid. Remember to carefully select the right Object Storage class (Hot, Cool, Archive) in the cloud to align with the access frequency of the data being moved from the gateway to optimize for Cost Optimization in your Cloud Technology design.