VP of Open Source and Founding Member @Alluxio
This is a guest blog contributed by datasapiens’ Juraj Pohanka, Koen Michiels and Sam Gilbert. This article described how engineers at datasapiens brought down S3 API costs by 200x by implementing Alluxio as a data orchestration layer between S3 and Presto.
There have been numerous articles and online webinars dealing with the benefits of using Alluxio as an intermediate storage layer between the S3 data storage and the data processing system used for ingestion or retrieval of data (i.e. Spark, Presto), as depicted in the picture below:
To name a few use cases
The main conclusion from these use cases is that using Alluxio as a data orchestration layer has the following benefits:
All these benefits are critical when deciding a production-grade data processing architecture, however one important benefit has so far not been sufficiently highlighted when choosing such architecture – cost reduction on the usage of the S3 API. This will be the focus of this article.
datasapiens is an international startup, based in Prague, that focuses primarily on helping companies to transform their business through data. We offer 3 key products:
The first generation of our infrastructure used AWS Redshift as the main data warehouse. However, when we started to acquire clients with larger data volumes (i.e. billions of records per year) we started to struggle with the following:
It was at this point that we decided to move to the Hadoop ecosystem.
The first version of our second generation was the common software stack for cloud-based data lake architectures:
With this storage-compute separated architecture, we achieved faster performance and reduced compilation times. However, the infrastructure costs were above expectations due to S3 API cost increases, making up a large proportion of the overall cost (described in detail below).
The second version of our second generation includes Alluxio as a data orchestration layer. We co-locate our data with our compute for in-memory data access. This architecture achieves even faster performance more consistently. In addition, we saw a drop in infrastructure costs due to S3 API costs dropping to a negligible level.
During the fine-tuning of our first Presto cluster, we conducted several performance tests of the cluster under various levels of user concurrency.
The setup of the performance tests was the following:
For the design of our DWH, we wanted to avoid using the traditional star/snowflake schemas. Instead, we could pre-join the table entities in our ETL into a few OLAP cubes to avoid expensive table joins in Presto.
Towards the end of the testing phase, we detected a dramatic increase in costs for the S3 API service. The following image depicts the incurred S3 API costs for each day of the performance tests:
The following table summarizes the total number of executed queries with the S3 API costs and S3 API request counts per each day of performance tests:
The next table shows the percentage of the S3 API costs out of the total costs:
During this testing period, we tested several configuration variations of the Presto cluster in order to achieve a satisfying query speed at a specific concurrency level. In the first table above, we can observe the variance in the S3 API costs/requests per day. We attribute this variance to the varying Presto cluster deployments for each day in the testing period.
After reviewing the incurred S3 API costs under the given workloads, we came to the conclusion that running a Presto cluster that read directly from S3 was not feasible, as it would impact our profit margin.
We considered deploying Hadoop as an intermediate layer between Presto and S3 and synchronizing through simple jobs.
We also looked at Alluxio, which provided similar functionality without the need for manual synchronization. Furthermore, Alluxio has additional benefits such as connections to various cloud providers & storage devices and having tiered storage functionality which we thought would improve query speed.
After deploying a co-located Alluxio-Presto cluster, the performance of queries improved (especially for those with a higher number of execution stages), and the S3 API costs dropped to below $1/day. We have since implemented co-located Alluxio-Presto clusters in production. The infrastructure is successfully serving hundreds of users per client.
We also learned that although manual synchronization is not needed. To prevent any performance degradation of the Presto clusters during startup and after ETL jobs it’s best to “hydrate” the cluster using “warm-up” jobs.
To get a better quantitative understanding of the S3 API costs and the cost reduction when using Alluxio, we conducted a simple experiment by querying data from S3 in two separate clusters:
In both clusters, we measured the API request count to Alluxio per each query.
The following are the setup schemas for each cluster:
Alluxio-Presto cluster setup:
EMR-Presto cluster setup:
Query execution setup:
The deployment configuration for each of the clusters, as well as the complete test results, can be found in our GitHub repository.
Here are the results from the conducted tests on the Alluxio-Presto cluster:
Here are the results from the conducted tests on the EMR-Presto cluster:
As we can observe, the cumulative count for the `GetObject` request type and the `File Infos Got` request type are more-less compatible. The different values for the storage API calls in both cases can be attributed to:
In the case of the S3 API, we also have an additional cost for the `ListBucket` request type which is more costly then the `GetObject` type.
Now, we will try to estimate the per-query S3 API costs for the EMR-Presto cluster. We are limited by the following facts:
To estimate the per-query S3 API costs, we will take a naïve approach and redistribute the costs to each query proportionally to the number of requests sent to Alluxio.
This gives us the following table with the top 10 request-expensive queries:
As can be seen, we can have quite significant costs for a single executed query. Let's do a simple comparison of the cluster infrastructure and S3 API costs for both cases with and without Alluxio:
It is apparent that the S3 API costs for the cluster with Alluxio form a negligible (0.58%) part of the total costs, whereas for the cluster without Alluxio, they form almost half (48.83%) of the total costs in our simple test case.
Looking at the pricing models for the storage services of three largest cloud providers (AWS S3, Azure Blob Store, and Google Cloud Storage), we see that they are very similar to each other, hence one can expect similar storage API costs when running analytical workloads with or without Alluxio. However, this claim still needs to be verified.
Our test case was a simple setup of a small cluster to which we were sending queries in a sequential manner. However, real-life production environments cope with workloads with far higher compute and storage demands. Frequently, we have:
For these types of cloud-based production environments, using no intermediate data storage layer between the cloud storage service and the data processing system can lead to tremendous costs.
Therefore we would be inclined to recommend that when running analytical workloads in a cloud-based environment, to have a data orchestration layer such as Alluxio.
The original article was published on Alluxio’s Engineering Blog.