MinIO AIStor, Purpose Built for AI and Data Workloads

MinIO, Inc. announced AIStor, an evolution of its Enterprise Object Store designed for the exascale data infrastructure challenges presented by modern AI workloads. AIStor provides new features, along with performance and scalability improvements, to enable enterprises to store all AI data in one infrastructure.

Recent research from MinIO underscores the importance of object storage in AI and ML workloads. Polling more than 700 IT leaders, the company found that the top 3 reasons motivating organizations to adopt object storage were to support AI initiatives and to deliver performance and scalability modeled after the public clouds. The firm is the only storage provider solving this new class of AI-scale problems in private cloud environments. This has driven the company to build new AI-specific features, while also enhancing and refining existing functionality, specifically catered to the scale of AI workloads.

“The launch of AIStor is an important milestone for MinIO. Our object store is the standard in the private cloud and the features we have built into AIStor reflect the needs of our most demanding and ambitious customers,” said AB Perisamy, co-founder and CEO. “It is not enough to just protect and store data in the age of AI, storage companies like ours must facilitate an understanding of the data that resides on our software. AIStor is the realization of this vision and serves both our IT audience and our developer community.”

While MinIO was already designed to manage exascale data, these advances make it easier to extend the power of AI to introduce new functionality across applications without needing to manage separate tools for analysis.

Of particular note is the introduction of a new S3 API, promptObject. This API enables users to “talk” to unstructured objects in the same way one would engage an LLM moving the storage world from a PUT and GET paradigm to a PUT and PROMPT paradigm. Applications can use promptObject through function calling with additional logic. This can be combined with chained functions with multiple objects addressed at the same time. For example, when querying a stored MRI scan, one can ask “where is the abnormality?” or “which region shows the most inflammation?” and promptObject will show it. The applications are almost infinite when considering this extension. This means that application developers can exponentially expand the capabilities of their applications without requiring domain-specific knowledge of RAG models or vector databases. This will dramatically simplify AI application development while simultaneously making it more powerful.

Click to enlarge

Additional new and enhanced capabilities in MinIO AIStor include:

• AIHub: a private Hugging Face API compatible repository for storing AI models and datasets directly in AIStor, enabling enterprises to create their own data and model repositories on the private cloud or in air-gapped environments without changing a single line of code. This eliminates the risk of developers leaking sensitive data sets or models.
• Updated Global Console: a completely redesigned user interface for the company that provides extensive capabilities for Identity and Access Management (IAM), Information Lifecycle Management (ILM), load balancing, firewall, security, caching and orchestration, all through a single pane of glass. The updated console features a new MinIO Kubernetes operator that further simplifies the management of large scale data infrastructure where there are hundreds of servers and tens of thousands of drives.
• Support for S3 over Remote Direct Memory Access (RDMA): enables customers to take full advantage of their high-speed (400GbE, 800GbE, and beyond) Ethernet investments for S3 object access by leveraging RDMA’s low-latency, high-throughput capabilities, and provides performance gains required to keep the compute layer fully utilized while reducing CPU utilization.

“AMD EPYC processors provide unparalleled core density, exceptional storage performance and I/O, aligned with high performance compute requirements,” said Rajdeep Sengupta, director, systems engineering, Advanced Micro Devices, Inc. “We have deployed the MinIO offering to host our big data platform for structured, unstructured, and multimodal datasets. Our collaboration with MinIO optimizes AIStor to fully leverage our advanced enterprise compute technologies and address the growing demands of data center infrastructure.”

The AIStor launch follows a series of major announcements in the AI storage space starting with the release of its reference architecture for large scale data AI data infrastructure, MinIO DataPod. The vendor also demonstrated its compute extensibility with announcements supporting optimizations for Arm-based chipsets and a customer win announcing it was powering the Intel Tiber AI Cloud. Collectively, these announcements frame MinIO’s leadership in the AI storage space and its place at the center of the AI ecosystem.

More info on the blog.