Using a Self-serve Data Architecture to Share Data Across Development Teams

By: Adit Madan, Jasmine Wang and Bin Fan, Alluxio

This article highlights the synergy between the two widely adopted open-source projects,  and , and demonstrates how together they deliver a self-serve data architecture across clouds.

What Makes an Architecture Self-serve?

Condition 1: Evolution of the data platform does not require changes

All data platforms evolve over time, including the addition of a new data store, compute engine, or a new team that needs to access shared data. In either case, a data platform is self-serve if it does not require changes to accommodate evolution.

Condition 2: Isolation across Teams

Business units don’t step on each other with a self-serve platform. When a new team is introduced, data access by one team should have no impact on the existing usage of the shared data infrastructure.

The combination of the above two offers agility, which oftentimes is more important than the cost of physical infrastructure.

Data Platform Considerations

Below, we introduce some considerations when designing a self-serve platform, and architectural patterns for simple solutions.

Consideration 1: Data is shared

1. Between Compute Frameworks
   • There are a large number of specialized compute engines. Each engine is better suited for a specific task, which means there is a need to share data between engines. For example, ETL in a batch processing followed by Presto for interactive queries.
2. Between Different Teams
   • For example, a team is responsible for collection of operational data which is then consumed by multiple other business units.
3. Between Data Centers Across Regions and Cloud Providers
   • This offers the flexibility to choose the most optimal service across environments.

     
     

The solution for shared data is to have an abstraction layer across heterogeneous compute. Alluxio provides such an abstraction across clouds for seamless sharing of data between Presto and other compute engines regardless of the data store.

Consideration 2: Data has ownership domains and processing in place is simple

1. Although replication provides isolation, governance becomes complex as the owner of data enforces strict policies about the consumption of data.
2. Copies introduce redundancy, which is error-prone and has high resource requirements.

   
   

It may seem obvious that a solution is to not make copies of data, but what about performance when we don’t move data? This calls for a single abstraction layer that takes care of governance, performance, and movement of data across ownership domains.

The architecture below shows Presto using the Alluxio layer for access to data regardless of the location.

The above design can be broken down into a few simple cases

1. All within a single cloud or a datacenter
2. Shared across multiple data centers or a hybrid cloud

In all these cases, the separation of the CONSUMER from the PRODUCER of data is enabled by an abstraction layer that provides more than a simple cache. Advanced preloading and write capabilities guarantee SLAs even with the separation of data from compute.

Conclusion:

With a self-serve data architecture across clouds, we construct a solution that stands the test of time as a data platform evolves. Learn more from the whitepaper , and see how companies including Facebook, TikTok, Electronic Arts, Walmart, Tencent, Comcast, etc level up their current Presto platform leveraging Alluxio.

First Published