AWS: Availability of UltraWarm for Amazon Elasticsearch Service

Amazon Web Services(AWS), an Amazon.com company, announced the availability of UltraWarm for Amazon Elasticsearch Service, a fully managed, low-cost warm storage tier that provides fast, interactive analytics of log data at one-tenth the cost of existing storage options.

Click to enlarge

The Elasticsearch Service makes it simple to collect, analyze, and visualize machine-generated log data from websites, mobile devices, and sensors. UltraWarm for Elasticsearch Service gives Elasticsearch customers a warm storage tier that both stores large amounts of data cost-effectively and provides the type of snappy, interactive experience that Elasticsearch customers expect. There are no up-front investments required to use UltraWarm, and customers pay an hourly rate for the storage provisioned.

As more and more applications are built using microservices, containers, and purpose-built data stores, the volume of machine-generated log data is growing. Real-time analysis of this data has become essential to customers in order to quickly resolve operational and security issues. Elasticsearch Service is a service for log analytics because of its ability to ingest high volumes of log data and analyze it interactively. With this growth of log data, storing and analyzing months’ or even years’ worth of data is cost-prohibitive at scale. This has led customers to use multiple analytics tools, or delete valuable data, missing out on important insights that the longer-term data could yield.

To solve for this customer challenge, the company built UltraWarm, which gives Elasticsearch customers a warm storage tier that both stores large amounts of data cost-effectively, and provides the type of snappy, interactive experience that Elasticsearch customers expect.

Now Elasticsearch Service supports two storage tiers, hot and UltraWarm. The hot tier is used for indexing, updating, and providing the fastest access to data. UltraWarm provides a distributed cache for more frequently accessed data, while using advanced placement techniques to determine the blocks of data that are accessed less frequently, and can be moved outside of the cache to Amazon Simple Storage Service (Amazon S3). it stores data in Amazon S3, providing up to 50% faster query execution vs. competing warm-tier solutions, and 80% lower cost than the warm-tier storage from other managed Elasticsearch offerings. With UltraWarm, customers can manage current and historical log data for interactive operational analysis and visualization in a single cluster.

UltraWarm is an extension of the Elasticsearch Service. Customers can visualize search results across both their recent and longer-term operational dat, all from their Kibana interface. Additionally, it supports all of the Elasticsearch APIs, tools, and features, including enterprise-grade security with fine-grained access control, encryption at rest and in flight, integrated alerting, SQL querying, and more. This allows developers, DevOps engineers, and InfoSec experts to use Elasticsearch Service for the analysis of recent (weeks) and longer-term (months or years) operational data without needing to spend days restoring data from archives (Amazon S3 or Amazon Glacier) to an active searchable state in an Elasticsearch cluster.

“Our customers tell us that log data offers a wealth of operational and security insights, but that the storage of log data quickly adds up, and proves cost-prohibitive over the medium and long term,” said Raju Gulabani, VP, databases and analytics, AWS. “UltraWarm is the most cost-effective Elasticsearch-compatible storage solution available. It is also performance-optimized, so customers can investigate and interactively visualize their data while they embrace data at scale.“

UltraWarm can be enabled on existing or new domains using the AWS Management console, CLI, or SDK. It is available on Amazon Elasticsearch version 6.8 and above in US East (North Virginia, Ohio), US West (Oregon, North California), AWS GovCloud (US-Gov-East, US-Gov-West), Canada (Central), South America (Sao Paulo, Brazil), EU (Ireland, London, Frankfurt, Paris, Stockholm), AsiaPac (Singapore, Sydney, Tokyo, Seoul, Mumbai, Hong Kong), China (Beijing, Ningxia), and Middle East (Bahrain), with additional regions coming soon.

Sophos Group plc. is a provider in next-gen cybersecurity, protecting organizations of all sizes in more than 150 countries from today’s most advanced cyber threats.

“Sophos uses Amazon Elasticsearch Service to run a large-scale security monitoring and alerting system, because it is highly performant and scalable,” said Prakash Talreja, architect, Sophos. “We are excited that UltraWarm will enable us to retain log data for much longer in a cost-effective way. We see great value in leveraging UltraWarm to bring down cost and reduce operational overhead.“

Asurion LLC is a provider of device insurance, warranty, and support services for cell phones, consumer electronics, and home appliances.

“All our application and infrastructure logs are loaded into Amazon Elasticsearch Service for real-time data analysis, API performance metrics, and alerting based on log events,” said Shyam Rayaprolu, principal architect, retail and SBX platforms, Asurion. “We are always looking to bring costs down. Even though we automated our backup process operationally, it has become overhead for our DevOps and support teams. We are really excited that UltraWarm removes the operational overhead, and reduces cost. We were able to easily migrate our data from the hot nodes to UltraWarm nodes, and use our existing Kibana dashboards, without the need to make any changes.“

SparkPost is a predictive email intelligence platform, helping brands predict and optimize email performance with data analytics solutions.

“Amazon Elasticsearch Service powers one of our most used features – Event Search – which indexes billions of email message and engagement events in real-time every day. Our customers can quickly resolve support issues by easily looking up the delivery and open status for an email sent to a particular recipient,” says Chris McFadden, VP, engineering, SparkPost. “We love the simplicity and performance of Amazon Elasticsearch Service and are excited to use UltraWarm to further reduce our operational costs, while simultaneously offering our customers access to raw event data beyond our current ten day retention period. We believe offering up to 30 days of data will make our Event Search feature even more valuable as a go-to troubleshooting tool for our customers.“

Resource:
Blog: General Availability of UltraWarm for Amazon Elasticsearch Service by Martin Beeby

Read also:
From AWS, Low-Cost HDD Storage Option for Amazon FSx for Windows File Server
Single-AZ HDD storage at $0.013 and multi-AZ HDD storage at $0.025/GB/month
April 3, 2020 | Press Release