pysync
Pysync has both a demonstration implementation of the rsync and related algorithms in pure Python, and a high speed librsync Python extension. The pure Python is not fast and is not optimized, however it does work and provides a simple implementation of the algorithm for reference and experimentation. It includes a combination of ideas taken from librsync, xdelta, and rsync. The librsync Python extension is less flexible and harder to understand, but is very fast.
| Tags | Communications File Sharing education Archiving Compression Software Development Libraries Python Modules |
|---|---|
| Licenses | LGPL |
| Operating Systems | OS Independent |
| Implementation | Python |
Recent releases
- 17 Oct 2003 11:00
Changes: Updated to use librsync 0.9.6. TODO and NEWS files have been added. Changed to use psyco if available, giving a 33% speedup. Changed to using a faster md4sum implementation based on the librsync implementation, and modified to use the RSA API. Rollin/rollout support has been added to historic adler32.py. There are minor cleanups to the rollsum code, and minor tweaks to the handling of block fragment matching.
- 24 Jun 2002 01:28
Changes: This version adds a fast rollsum extension module, and changes pysync.py to use it, and the README docs were updated.
- 02 May 2002 14:06
Changes: Be warned that the API and data format have been changed in ways that are not backwards compatible. A high speed librsync swig extension module was added. An MD4 extension module was added. The API was changed to be more like librsync/rdiff. distutils is now used to build releases for Linux and Windows.
- 12 Mar 2001 04:43
Changes: This release includes support for xdelta style deltas (optimized deltas calculated directly from the sourcefile, not a signature), simplified and optimized code to take advantage of python buffer() copy-less string slicing, and command-line delta calculation now reads files incrementaly.
- 01 Mar 2001 13:01
Changes: pysync.py was changed to use the new zlib-like API. A serious adler32 bug affecting rolling checksums was fixed. pysync-test.py was added for testing pysync.py.
- All comments
Recent commentsRe: Handles large archives?
> search on google.com for "python rsync".
> almost right at the top is someone
> implementing rsync in python, but he
> hasn't got round to doing the bits that
> this guy has.
I think you are refering to rsync.py. (http://www.vdesmedt.com/~vds2212/rsync.html) I just looked at it. It doesn't implement the rsync algorithm or any network transport. It only copies the walk/filter/copy functionality of rsync.
Another interesting project is zsync. (http://zsync.moria.org.uk/) This implements an inverse rsync algorithm and uses a normal http server for network transport. It has no walk/filter/copy functionality.
For those who want to add walk/filter functionality to pysync, I have bits and pieces that might be useful;
efnmatch.py (http://minkirri.apana.org.au/~abo/projects/pybackup/efnmatch.py) rsync style extended fnmatch.
dirscan.py (http://minkirri.apana.org.au/~abo/projects/pybackup/dirscan.py) rsync style include/exclude pattern directory scanning.
ddiffutils.py (http://minkirri.apana.org.au/~abo/projects/utils/ddiffutils.py) efficient directory comparison walk generators.
I haven't yet tied these together into a useful combination, but I should some day :-)
Re: Handles large archives?
search on google.com for "python rsync". almost right at the top is someone implementing rsync in python, but he hasn't got round to doing the bits that this guy has.
combine the two projects and you have a _complete_ implementation of rsync in python.
i aim to investigate this project because i want to be able to offer different files "merged" into one single repository, depending on who connects to the rsync server :)
i.e. i can back up several machines, but the config files will be different.... cool, huh? :)
Re: Handles large archives?
> I'm curious as to whether this tool has
> the same architectural limitation as
> rsync, that it must build a complete
> archive listing before it begins to
> transfer files.
...
pysync only implements the delta calculation and patch application, it does not include any directory walk or network transport stuff. So pysync doesn't have those limitations because it doesn't include that kind of functionality. Pysync could be used to implement something that does what you want without those limitations.
Have a look at librsync, rdiff-backup, unison etc for possible other alternatives that might be closer to what you want.
Handles large archives?
I'm curious as to whether this tool has the same architectural limitation as rsync, that it must build a complete archive listing before it begins to transfer files. I have a system with about 6-7 million files, and even though the vast majority of them are --exclude'd out, and even though the system has 2GB of RAM, rsync runs out of memory. Does this program work the same way?
Version 2.24 release
Version 2.24 has been released to update pysync for the new librsync 0.9.6. Also includes some minor tweaks, including psyco support which gives a 33% speedup.
If anyone wants windows binaries, let me know and I'll build them.