AWS re:Invent: Availability of Amazon S3 Express One Zone

At AWS re:Invent, AWS announced the general availability of Amazon Simple Storage Service (Amazon S3) Express One Zone, a performance, single-zone Amazon S3 storage class purpose-built to deliver consistent, single-digit millisecond data access for customers’ most latency-sensitive applications.

Click to enlarge

Amazon S3 Express One Zone is the lowest latency cloud object storage available, with data access speed up to 10x faster and request costs up to 50% lower than Amazon S3 Standard, from any AWS Availability Zone within an AWS Region. For the most demanding workloads, customers can choose to collocate their Amazon S3 Express One Zone data in the same AWS Availability Zone as their compute resources, optimizing compute performance and costs while increasing data processing speed. Amazon S3 Express One Zone is a performant storage class for request-intensive operations such as ML training and inference, interactive analytics, and media content creation. With Amazon S3 Express One Zone, customers can scale their storage up or down as needed and work with the Amazon S3 APIs they use today.

Since its launch 17 years ago, Amazon S3 is a cloud storage service, with millions of customers around the world across all industries and use cases. With durability, availability, security, and scalability, it holds more than 350 trillion objects and averages more than 100 million requests/second. Offering a range of storage classes, it delivers performance at the lowest cost for frequent, infrequent, and archive data-access patterns for virtually any workload. However, emerging use cases, such as ML training and inference, interactive analytics, financial model simulations, real-time advertising, and media content creation, can write and access data millions of times per minute and require consistent, single-digit millisecond response times to meet performance needs. For example, AI and ML model training often requires millions of images and lines of text to be processed in minutes, and data access speed is critical to achieve the highest compute efficiency. In order to reduce storage latency, application run time, and compute costs for performance-critical applications, customers move their most frequently accessed data from Amazon S3 to custom caching solutions. However, this increases complexity, adds operational burden, and requires application modifications because customers must maintain additional storage infrastructure with multiple sets of APIs. This also leads to idle compute time because the compute infrastructure must run while data is copied into the instance, which can increase costs and create inefficiencies.

Amazon S3 Express One Zone is a performance storage class that delivers a fast data access speed and high performance of any cloud object storage for customers’ most latency-sensitive applications. By replicating and storing data on purpose-built hardware and software, it provides up to 10x faster access speed than Amazon S3 Standard, resulting in accelerated data processing. To further increase access speed and support millions of requests per minute, it stores data in a bucket type – an Amazon S3 directory bucket. Customers have always been able to choose a specific AWS Region to store their Amazon S3 data. Now, within an AWS Region, customers can choose to collocate their Amazon S3 Express One Zone data in the same AWS Availability Zone as their compute resources for their most demanding workloads, which helps them lower compute costs and run workloads faster. Amazon S3 Express One Zone allows performance-critical applications, such as query-intensive analytics, to achieve single-digit millisecond data access speed with durability and availability, while reducing request costs by up to 50% compared to Amazon S3 Standard, and reducing compute costs by up to 60%. For example, data analysts often query petabytes of data, and the speed of processing data is important to make timely business decisions. With Amazon S3 Express One Zone, data lake queries complete up to 4x faster because data access speed is up to 10x faster than Amazon S3 Standard, decreasing the workload time for analytics, so data analysts can derive insights and make decisions quicker. Amazon S3 Express One Zone enables customers to scale their storage up or down, and they no longer need to manage multiple storage systems for low latency workloads because their applications access data through the same Amazon S3 APIs used today.

“Millions of customers rely on Amazon S3 for everything from low-cost archival storage to PB-scale data lakes, and they want to expand their use to support their most performance-intensive applications where every millisecond counts,” said James Kirschner, GM, Amazon S3, AWS. “Amazon S3 Express One Zone delivers the fastest data access speed for the most latency-sensitive applications and enables customers to make millions of requests per minute for their highly accessed datasets, while also reducing request and compute costs.”

With a few clicks, customers can create a new Amazon S3 directory bucket and either upload new objects directly into the Amazon S3 directory bucket they created or copy objects from existing datasets in other Amazon S3 storage classes using Amazon S3 Batch Operations in the AWS Management Console.

Amazon S3 Express One Zone supports Amazon S3 features, including Mountpoint for Amazon S3, Amazon S3 Server-Side Encryption, and Amazon S3 Block Public Access, as well as AWS services, such as Amazon EMR, Amazon Redshift, Amazon SageMaker, and Amazon Bedrock.

It is available in the US East (N. Virginia), US West (Oregon), Europe (Stockholm, Sweden), and AsiaPac (Tokyo, Japan), with availability in additional AWS regions coming soon.

ChaosSearch, Inc. provides solutions that empower businesses to make data-driven decisions with unparalleled efficiency.

“Our uniqueness lies in our native Amazon S3 data lake, facilitating SQL, search, and large language model-based queries,” said Thomas Hazel, founder and CTO, ChaosSearch. “We already benefit from Amazon S3’s unmatched scalability, durability, availability, and security, and now it’s supercharged for high-performance analytics at scale. With Amazon S3 Express One Zone, we get rapid time to insight with 60% faster queries and substantial cost savings through lower query costs—without any code modifications.”

Cloudera, Inc. delivers a hybrid data platform with secure data management and analytics to customers across industries around the world.

“Customers use our platform to provide an open data lakehouse to get business value from all their data using AI and data analytics,” said Jim Clark, VP, product management, Cloudera. “With Amazon S3 Express One Zone’s data access speed alone, Cloudera customers can achieve up to 40% faster read and write performance, driving faster insights and value from their data. We are actively working to support Amazon S3 Express One Zone across Cloudera to improve performance and lower our customers’ cloud infrastructure costs.”

Colorfront develops high performance digital processing technologies for the motion picture industry.

“Colorfront integrates a high-performance GPU architecture with Amazon S3 Express One Zone to deliver accelerated processing to transcode, decode, convert, and wrap files to various high-end formats,” said Brandon Heaslip, director, solutions engineering, Colorfront. “With Amazon S3 Express One Zone’s single-digit millisecond data access speed, Colorfront customers can accelerate their digital video processing by up to 70% to meet diverse media and entertainment demands for streaming delivery while maintaining digital video standards.”

Pinterest is a visual inspiration platform where over 480 million people come each month to search, save, and shop the best ideas in the world. It leverages advanced ML models to analyze billions of objects and personalize recommendations for each of its users.

“Machine learning is at the heart of Pinterest and is powered by large scale ML training log collection. To accelerate data ingestion and distribution at Pinterest, we developed MemQ, an ultra-efficient open source PubSub system that powers all our ML training data and leverages Amazon S3 for supercharging data pipelines at tens of GBps while staying cost-efficient,” said Ambud Sharma, technology lead, data engineering, Pinterest. “During our evaluation of the Amazon S3 Express One Zone storage class with MemQ, we saw latency improvements of over 10 times while increasing our data processing transactions per second. Based on this order of magnitude performance improvement, Amazon S3 Express One Zone allows us to enable higher velocity data consumption and more frequent experimentations, while significantly reducing infrastructure costs. Amazon S3 Express One Zone is creating a paradigm shift making cloud applications more performant, scalable, flexible, and efficient.”

Resources :
Blog: Announcing the new Amazon S3 Express One Zone high performance storage class   
Video: Introduction to the Amazon S3 Express One Zone storage class
Blog: Amazon S3 Express One Zone delivers cost and performance gains for ChaosSearch customers   
Blog: Akridata accelerates processing of unstructured data with Amazon S3 Express One Zone    
Blog: lakeFS and Amazon S3 Express One Zone: Highly performant data version control for ML/AI    
Blog: ClickHouse Cloud & Amazon S3 Express One Zone: Making a blazing fast analytical database even faster