> Correct, there are no data diffs. DVC just stores blobs and you can GC the old... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		cyphar on Feb 11, 2019 \| parent \| context \| favorite \| on: DVC – Open Source Machine Learning Version Control... > Correct, there are no data diffs. DVC just stores blobs and you can GC the old ones Have you looked into using content-defined chunking (a-la restic or borgbackup) so that you get deduplication without the need to send around diffs? This is related to a problem that I'm working on solving in OCI (container) images[1]. [1]: https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar

dmpetrov on Feb 11, 2019 [–]

Content-defined chunks - very interesting. I'd suggest you ask this question in DVC issue tracker or DVC channel https://dvc.org/chat

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact