diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 85 |
1 files changed, 85 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..3b8c72f --- /dev/null +++ b/README.md @@ -0,0 +1,85 @@ +# ssync + +ssync is a suite of utilities that facilitate syncing a remote device with a local one regardless of file structure. For example if your remote storage's structure and design differs from your local - rsync wouldn't be able to easily 1:1 fetch everything neatly; so ssync works by indexing all of the files it hasn't yet - then queues them for download to single local target dir; then you just have to do whatever you want locally, separately, to reorganize things. I generally use [automv](https://git.senders.io/senders/automv) for common/repeat fetches. + +## setup + +ssync works via three main modules: + +- `ssync-index` - a script that indexes a directory (run separately from remote and local target) +- `ssync-queue` - generates the queue for `-fetch` to fetch +- `ssync-fetch` - fetches off of the queue + +With the optional `ssync` bundled as a single executable that runs in the typical aka "my desired" use-case. + +The typical process is: + +1) have a cron execute `ssync` on the local/target system +2) it will run `index` against the remote and local systems, refreshing the index for anything new +3) it will then run `queue` which generates the queue of files yet-to-be-downloaded from the index, appending any files to the queue +4) finally run `fetch` to process through the queue + +## logic flow + +- (s) establish a lock +- (i) Generate the remote and local indexes +- (q) Compare remote to local adding left side diff to the queue +- (f) iterate over the queue + - (f) check if the file exists + - (f) if not download +- (s) report queue diff, process duration, and status +- (s) release lock + +### Notes on queue maintenance + +Rather than maintain a queue, each process will generate its own queue from the index diffs. Each queue is placed in a specified directory, defaulting to `$XDG_CACHE_DIR/ssync/queue/` (if not set uses `/home/$USER/.config/` if exists, else creates and stores in `/tmp/ssync/queue/`) + +Since it runs off of process local indexes the queue can be reaped between processes and not incur any potential data loss. + +### Index window + +So long as files on the local system are expect to persist longer than it does on the remote you'll always be safe. But the index window helps set a maximum lookback - so that any older files may be removed from the local system without being resynced. + +In a previous implementation of this - not starting from scratch each run led to examples where a set of sequential files had a missing file in between. Like img-1, img-2, img-3, img-5, img-6: with img-4 missing. It was annoying and easy to not notice. + +## Configuration + +ssync.conf + +```config +remote_host=HOST +remote_root_dir=/path/to/sync/root/ +keyfile=/path/to/key +local_root_dir=/path/to/local/sync/root/ + +index_window_s=86400 # 24 hours +index_dir=/path/to/index/dir # optional +queue_dir=/path/to/queue/dir # optional +lock_file=/path/to/desired/file.lock # optional +``` + +## commands + +``` +ssync [options] + OPTIONS + -c [CONFIG_FILE] optional config file to use + default: ~/.config/ssync/ssync.conf + -l [LOCK_FILE] optional lock file + default: from config) + -q [QUEUE_DIR] optional queue dir + default: from config) + -k [KEY_FILE] optional key file + default: from config) +``` + +``` +ssync-index [options] -c [FILE] + REQUIRED + -c [CONFIG_FILE] config file to use + OPTIONS + -l local only (cannot be used in conjunction with -r) + -r remote only (cannot be used in conjunction with -l) + -o [OUTPUT_FILE] output file override + -k [KEY_FILE] key file override +``` |