De-Dupe in Open Source

De-dupe is a killer storage technology, but expensive and proprietary. That’s the reason some users finally prefer not to de-dupe at all, estimating that the price of hard disk drives is so low that there is no reason to invest in a technology supposed to diminish the price of their storage subsystems. It’s different on WAN where bandwidth is physically limited and can be notably ameliorate with this form of compression.

But it could change. ZFS has built-in de-dupe. There is also currently a project called Opendedup, developing SDFS, a de-dupe file system in open source for Linux and now in version 0.8.13. It has been tested and developed on ubuntu 9.1. It needs 2GB or RAM, Java 7 and Fuse 2.8+. And the results seem to be good as he can reduce storage utilization by up to 90%-95%. SDFS can de-dupe a petabyte and more at line speed from 89.9MB/s to 290MB/s – depending on the chunk size, 4K to
128K -, and it works with VMware, Xen and KVM supervisors.