Weka AI Integration With Deep Learning Pipeline Management Solution From Finish Valohai

WekaIO, Inc. integrates with the deep learning pipeline management solution from Valohai, a Weka Innovation Network (WIN) partner.

Click to enlarge

The announcement underpins Weka’s commitment to empower data scientists and chief data officers to manage and prioritize data science pipelines. The tools from Valohai are supported in an Amazon Virtual Private Cloud (Amazon VPC), and available in AWS Marketplace.

“New workloads are driving the need for modern foundational architectures, and the recently launched Weka AI offers a transformative solution framework for Accelerated DataOps,” said Shailesh Manjrekar, head, AI and strategic alliances, Weka. “Our partnership with Valohai and our integration with its Deep Learning Pipeline Management tools expand Weka AI’s capabilities to offer Explainable AI (XAI). This is a critical factor for use cases with a social impact, including autonomous driving, healthcare, and genomics.“

Integration with solutions from technology alliance partners such as Valohai enhances the power of Weka AI. Underpinned by the WekaFS, Weka AI provides a production-ready solution where the entire AI data pipeline workflow, from data ingestion to batch feature extraction, training, hyper-parameter optimization, and finally to inference and versioning, can run on the same storage platform, whether on-premises or on AWS. This is possible because of the mixed workload performance and the data management and governance abilities provided by WekaFS.

“ML gives businesses a competitive advantage, but while ML is hard, real-world ML is much harder,” said Eero Laaksonen, CEO, Valohai. “A real-world ML system is 95% enabling code, with only 5% actual ML code that creates business value, so the question becomes how to ensure that efforts are focused on that 5%. With Valohai, businesses can focus on data science, as it handles everything else. It is as simple as pointing to your code and data and hitting ‘run’. The seamless integration between our DLMS solution and Weka’s powerful snap-to-object capabilities offer a quick, zero-setup infrastructure for DataOps. This further helps businesses to build models 10x faster, regain 35% of lost cloud costs, and free their DataOps teams with automated ML orchestration, data management, and data mobility. It is a win-win.“

Valohai Build Machine Learning Pipelines screen
Click to enlarge

Weka AI with Valohai DLMS enables hybrid workflows, where data scientists can employ Jupyter Notebook or the Valohai GUI, either on-premise or on AWS, to do data transformation, model training, hyper-parameter optimization, and inference in a Kubernetes-orchestrated environment. Valohai DLMS integrates with WekaFS, which leverages i3.n Amazon Elastic Compute Cloud (Amazon EC2) instances with NVMe flash for the performance tier and extends the file namespace over Amazon S3 buckets for the capacity tier. DLMS also takes file namespace snapshots when running experiments and stores them for data versioning. These versions can then be rehydrated anytime to reproduce an experiment and provides the required explainability and transparency. The company’s solution provides security and governance with end-to-end encryption of the pipeline and also provides integration with management solutions such as HashiCorp Vault.

Manjrekar adds; “Weka AI, when deployed with Valohai on AWS, builds upon our success for accelerating Genomics, Fintech, and AI/ML/DL data pipelines. Customers use Weka on AWS for outstanding performance as we can showcase 100GB/s of throughput with 5 million 4KB IO/s for a 16xi3en cluster, all with less than 250μs latency.“

Resources:
Video: Weka and Valohai, a managed machine learning platform
Weka AI and Weka on AWS
Weka for AWS
DataOps with Weka AI

Read also:
WekaIO AI Accelerates Edge to Core to Cloud Data Pipelines
Expedites time-to-market and to-value while delivering agility and security at scale.
April 28, 2020 | Press Release
Shailesh Manjrekar Head of AI and Strategic Alliances, WekaIO
Comes from SwiftStack, Vexata, EMC, NetApp, Force10, HP and Arohi.
February 11, 2020 | Press Release