support for gocryptfs, ecryptfs, borg, duplicity, and rsync - #374

Open

bgemmill wants to merge 82 commits into

yadayada:masterfrom

bgemmill:master

Commits on Aug 7, 2016

support for ecryptfs; this requires sequential file writing using multiple file descriptors, as well as support for getting and setting xattrs on files.
bgemmill
committed
5024923

Commits on Aug 8, 2016

Merge remote-tracking branch 'upstream/master'
bgemmill
committed
1b36299
prevent binary blob spam in the logs when using set/getxattr
bgemmill
committed
6a1ee48

Commits on Aug 10, 2016

fix requirements
bgemmill
committed
15ac1e7
implement proper mtime handling so rsync can work over acd_cli.
bgemmill
committed
db9f9b0

Commits on Aug 11, 2016

bugfix: turns out reinterpret casting floats to bytes and back via structs leads to epsilon problems, causing rsync to think that mtime is different when it isn't.
benjamin.gemmill@gmail.com
committed
076dec0

Commits on Aug 13, 2016

workaround: due to ecryptfs' bug, reporting incorrect file sizes when using xattrs for crypto headers, we have to allow re-writing the first bytes of a file to make ecryptfs happy. once they fix th…
bgemmill
committed
9184a80

Commits on Aug 15, 2016

Commits on Aug 16, 2016

The corner cases around implementing truncate when we only have a (partial) buffer in the middle of writing is hard enough that we bail on it. We don't care about pre-allocating files since we have…
bgemmill
committed
8d87e06

Commits on Aug 17, 2016

Use memory as a backing for buffers only if the writing size is 1G (configurable) or smaller, disk otherwise.
bgemmill
committed
06efeca

Commits on Aug 18, 2016

expand the path -> node caching to solve large delays with readdir + many getattr calls for very full directories
bgemmill
committed
f1428f8

Commits on Aug 21, 2016

Commits on Aug 22, 2016

duplicity support: opportunistic reading and file size query if we're writing at the same time but have not yet flushed to amazon
bgemmill
committed
c25a2ea

Commits on Sep 11, 2016

Commits on Sep 12, 2016

retry fuse operations if there's a failure on amazon's side and the return code says to retry.
bgemmill
committed
d135b51

Commits on Sep 13, 2016

clean up codes when we retry
bgemmill
committed
d867f53
Amazon has any number of ways to make file operations fail.
bgemmill
committed
d7bc917

Commits on Sep 14, 2016

turns out the 1000 response is thrown rather then returned
bgemmill
committed
f4121e8

Commits on Sep 24, 2016

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
ed4b5a5

Commits on Oct 13, 2016

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
94b9d9f

Commits on Nov 6, 2016

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
1c537c0

Commits on Nov 7, 2016

we can get 1000 error codes on writing too.
bgemmill
committed
18671ea

Commits on Nov 9, 2016

Commits on Nov 22, 2016

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
282aff2

Commits on Dec 2, 2016

Commits on Dec 25, 2016

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
1de45d9

Commits on Dec 26, 2016

verify that file renames and moves happen since amazon can drop some despite returning success
bgemmill
committed
5124699

Commits on Jan 8, 2017

Commits on Jan 12, 2017

mode, uid, and gid
bgemmill
committed
6843dc1
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
b36aeda
blksize, blocks
bgemmill
committed
cf3cbb7

Commits on Jan 16, 2017

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
b14b36e

Commits on Jan 19, 2017

symlink, readlink
bgemmill
committed
be9a0a2
caching of symlink targts
bgemmill
committed
b2b6d4b

Commits on Jan 21, 2017

invalidate the path resolving cache less to help with rsync --partial file moves
bgemmill
committed
879d652
clean log spam
bgemmill
committed
bb1bec8

Commits on Jan 22, 2017

add elements to the resolve cache on creation
bgemmill
committed
1ef5657

Commits on Jan 24, 2017

The entirety of resolve caching can be removed if we apply gerph's PR. Thanks gerph!
bgemmill
committed
067a887

Commits on Jan 25, 2017

turns out it's faster with both
bgemmill
committed
cc47811

Commits on Jan 29, 2017

cache node ids instead of nodes to fix 0 file size issues
bgemmill
committed
2b75256

Commits on Feb 4, 2017

cache write buffer length so long uploads won't hold up getattr calls
bgemmill
committed
4c19184

Commits on Feb 6, 2017

hit sqlite less
bgemmill
committed
3b5044b

Commits on Feb 10, 2017

tidy and make flush a noop again
bgemmill
committed
8c0e506
credit
bgemmill
committed
c5401ac

Commits on Feb 14, 2017

Commits on Feb 15, 2017

store symlinks targets in file contents to ultimately allow for longer targets than amazon's max xattr size
bgemmill
committed
885b4d0

Commits on Feb 19, 2017

cache files' content, starting with small files and symlinks
bgemmill
committed
dc47fe2

Commits on Feb 21, 2017

retry on 429 rate throttling errors
bgemmill
committed
d428544

Commits on Feb 23, 2017

release file handle lock during file writes to amazon
bgemmill
committed
3a88b05

Commits on Feb 26, 2017

rely on rsync's later chmod to save an amazon call per file/folder create.
bgemmill
committed
f760f46
the folders table no longer exists
bgemmill
committed
6098ebc

Commits on Feb 27, 2017

turns out rsync needs this for proper change detection in some cases.
bgemmill
committed
a3e291e

Commits on Mar 1, 2017

releasing the fh lock can lead to 409 concurrent modification errors in huge directory trees with rsync
bgemmill
committed
ac80bec

Commits on Mar 3, 2017

sparse file support
bgemmill
committed
c9c0719

Commits on Mar 4, 2017

sparse file support at ends of files too
bgemmill
committed
3d2c26a

Commits on Mar 6, 2017

put flush back to try to solve plex issues
bgemmill
committed
96735d5

Commits on Mar 7, 2017

Commits on Mar 15, 2017

tidy
bgemmill
committed
b084226

Commits on Mar 16, 2017

speed up smaller syncs by not using the disk
bgemmill
committed
1a4b91e
make resolve recursive to cache intermediate results
bgemmill
committed
1d68ecf
fix schema for properties table
bgemmill
committed
a5fa452

Commits on Mar 21, 2017

prevent xattr cache from falling out of sync on failed acd calls
bgemmill
committed
f60ba35

Commits on Mar 29, 2017

the fuse subdir module sometimes leaves trailing slashes on directory paths
bgemmill
committed
2583cb0

Commits on Apr 16, 2017

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
9c08d52

Commits on May 13, 2017

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
c381f5d

Commits on May 26, 2017

Commits on May 27, 2017

Commits on May 29, 2017

Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
e2dcdf7

Commits on May 30, 2017

docs
bgemmill
committed
e2554a0
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill
committed
6b014d6