Partnership Between lakeFS and NetApp

This article was written on Tech Ontap Blogs by Ben-Houser, technical marketing engineer, NetApp, Inc.

In today’s data-driven world, managing and controlling vast amounts of data efficiently is crucial, especially when it comes to AI/ML use cases. That’s why we’re excited to introduce the partnership between lakeFS and NetApp. lakeFS brings the benefits of Git-like version control to the StorageGRID data lake, changing the way data is managed, organized, and utilized.

lakeFS, a ‘data version control, seamlessly integrates with StorageGRID, providing a robust set of features that enable commits, merges, rollbacks, and isolated branches for your data. This partnership addresses common pain points faced by AI/ML engineers and data scientists, making StorageGRID a solution for their demanding workloads.

One of the key advantages of lakeFS is its ability to create isolated environments for testing and validation. Developers can now make code changes and experiment with confidence, knowing that their actions won’t impact the production data. By leveraging de-dupe and copy-on-write techniques, lakeFS minimizes capacity usage.

Data reproducibility is another critical challenge in the AI/ML realm, and lakeFS simplifies this process significantly. With it, engineers can effortlessly track changes to their data over time, allowing them to pinpoint the exact state of their data at any given moment. This capability enhances data traceability and also provides the flexibility to roll back changes if necessary, ensuring data consistency and reliability.

lakeFS also provides continuous integration and continuous deployment (CI/CD) for data workflows. The platform offers hooks that can be integrated with commit and merge operations, allowing for automated file format validation, schema checks, and other custom operations. This ensures that data is thoroughly validated and prepared for production, streamlining the development process and reducing the risk of errors.

By combining the performance and scalability of StorageGRID with the version control capabilities of lakeFS, AI/ML practitioners can enjoy a simplified, efficient, and reliable data management experience.

Read the solution brief