SC24: Unstructured Data Management for HPC Gains Traction

HPC is once again in the spotlight at SuperCompute 2024 (SC24) in Atlanta, GA this month. AI and supporting digital business initiatives are paramount in enterprise IT. HPC is maturing and while it creates many opportunities for innovation and productivity, it also creates large quantities of unstructured data. Komprise, Inc. will be at the conference (booth #414) showcasing unstructured data management for HPC solutions. We see the intersection between enterprise IT organizations using HPC and the need to manage and protect that unstructured data in a new way.

Unstructured data management for HPC is growing.

Here’s why:

• Unstructured data is growing at a remarkable pace, and it’s not slowing down anytime soon. This data is costing enterprise IT organizations millions in storage, backup and data protection. Most organizations spend at least 30% of their IT budget on data storage, according to the Komprise 2024 State of Unstructured Data Management. These costs are growing and becoming unsustainable.
• Data is living in silos from the data center to edge and cloud and across multiple technologies and vendors, and therefore introduces risks: from compliance, to ransomware, to the missed opportunity of data analytics and AI and ML initiatives. You can’t manage nor analyze what you can’t see or don’t understand.
• Managing data using your storage vendor’s technology alone does not deliver the best ROI and limits your ability to see all data across the organization to be more proactive and sustainable. IT departments are beginning to get pressure from above to operate “greener.” That means not storing all your data on the highest-performing, energy hogging storage device. It also means cleaning up your data mess: data hoarding and duplicate data is a common problem.
• AI needs a different unstructured data strategy. Your organization might not be doing much with AI yet– other than experimenting with generative AI for writing and research support. Yet the time will come when you’ll need to act fast. That means having the right infrastructure. It’s not just the storage, security and networks but also having the right unstructured data management platform to govern and classify your data and set up automated AI data workflows.

Komprise can help HPC organizations in the following ways:

• Reduce 70-80% of annual storage costs: The Komprise Global File Index delivers a dashboard where you can see data growth rates, amount of data in storage, and time of last access so you can model plans to save. For instance, you can see potential savings of moving “cold” data that is one year or older and rarely accessed into secondary storage. Komprise patented Transparent Move Technology (TMT) tiers data across hybrid storage while maintaining native access to the tiered data both from the original location and from the cloud.
• Self-service access for research teams: Data stakeholders such as research directors need easier access to data and the ability to easily search for files and request workflows. They also need to understand department data usage so they can collaborate with storage teams on archiving strategies to free up space and/or help identify other areas for efficiency.
• Tag data for improved classification and segmentation: Metadata enrichment is increasingly valuable as unstructured data volumes grow into multiple petabytes in organizations. By adding tags to data, indicating file contents, location or project, data becomes more searchable. You can quickly identify sensitive data types, such as those containing PII, or curate specific data sets for use in AI and ML projects.
• Migrate faster and with lower risk: Large-scale data migrations are often painful, complex and may not deliver the expected ROI. Komprise has a proven process to analyze your environment and data prior to migration to ensure that you are moving just the right data to the right storage. Komprise Elastic Data Migration is significantly faster than many common tools and has built-in features for reliability and ease of use, such as by retaining all file permissions after a migration.
• Manage data across its lifecycle: One-size-fits-all storage is no longer viable in today’s world because of the size of data. Komprise unstructured data management analyzes HPC environments and executes data movement as it ages. You can automate policies to tier data from hot to warm to cold data tiers according to parameters that you set. Because of our patented TMT technology, you can access your data at any tier later, without expensive rehydration to the original storage.
• Expand your ransomware protection: By reducing the data stored on your expensive NAS though cold data tiering to immutable object storage in the cloud, you reduce your attack surface for ransomware actors. Unlike storage tiering solutions that lock the data into their file format and are incompatible with ransomware protection solutions like tamperproof snapshots, Komprise technology is transparent and fully compatible with ransomware protection and backup solutions.
• Prepare data for AI: Getting unstructured data ready to safely use in AI tools is one of the largest challenge for AI. Komprise offers a Google-like search across disparate data silos. You can tag your data via UI and API to enrich the metadata, making it more useable in AI. Komprise Smart Data Workflow Manager is the foundation for creating automated AI data workflows that enrich data and curate the right data sets for the right tools. Komprise also delivers systematic data workflow execution for RAG and inferencing. Read more about the data management requirements for AI inferencing.
• Govern data for AI workflows: When employees share organizational data with AI, IT needs a way to audit what was shared, ensure that sensitive information is restricted from AI and create data governance mechanisms. Komprise provides the framework for data orchestration and data governance with AI to protect sensitive data and avoid harmful outcomes.

If you have a lot of unstructured data, you need a solution like Komprise. Our flagship solution, Komprise Intelligent Data Managementer, it’s in your data center, at the edge or in the cloud.