1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
|
# ssync
ssync is a suite of utilities that facilitate syncing a remote device with a local one regardless of file structure. For example if your remote storage's structure and design differs from your local - rsync wouldn't be able to easily 1:1 fetch everything neatly; so ssync works by indexing all of the files it hasn't yet - then queues them for download to single local target dir; then you just have to do whatever you want locally, separately, to reorganize things. I generally use [automv](https://git.senders.io/senders/automv) for common/repeat fetches.
## setup
ssync works via three main modules:
- `ssync-index` - a script that indexes a directory (run separately from remote and local target)
- `ssync-queue` - generates the queue for `-fetch` to fetch
- `ssync-fetch` - fetches off of the queue
With the optional `ssync` bundled as a single executable that runs in the typical aka "my desired" use-case.
The typical process is:
1) have a cron execute `ssync` on the local/target system
2) it will run `index` against the remote and local systems, refreshing the index for anything new
3) it will then run `queue` which generates the queue of files yet-to-be-downloaded from the index, appending any files to the queue
4) finally run `fetch` to process through the queue
## logic flow
- (s) establish a lock
- (i) Generate the remote and local indexes
- (q) Compare remote to local adding left side diff to the queue
- (f) iterate over the queue
- (f) check if the file exists
- (f) if not download
- (s) report queue diff, process duration, and status
- (s) release lock
### Notes on queue maintenance
Rather than maintain a queue, each process will generate its own queue from the index diffs. Each queue is placed in a specified directory, defaulting to `$XDG_CACHE_DIR/ssync/queue/` (if not set uses `/home/$USER/.config/` if exists, else creates and stores in `/tmp/ssync/queue/`)
Since it runs off of process local indexes the queue can be reaped between processes and not incur any potential data loss.
### Index window
So long as files on the local system are expect to persist longer than it does on the remote you'll always be safe. But the index window helps set a maximum lookback - so that any older files may be removed from the local system without being resynced.
In a previous implementation of this - not starting from scratch each run led to examples where a set of sequential files had a missing file in between. Like img-1, img-2, img-3, img-5, img-6: with img-4 missing. It was annoying and easy to not notice.
## Configuration
See `man 5 ssync` for configuration details.
## Usage
There are man pages for each sub-process with details and `-h` flags for each printing the options and configs.
### ssync
```
ssync [options] CONFIG_FILE
OPTIONS
-b BACKEND
-v verbose logging
-h print this message
```
### ssync-index
```
ssync-index [options] ROOT_DIR
OPTIONS
-b index basename only
-k KEY_FILE (optional: if indexing local)
ssh-key file to use (needs to be non-interactive)
optional: will use default session key
or key set in ssh_config for HOST
-o OUTPUT_FILE
-r REMOTE_HOST (optional)
remote host to index from such as user@hostname
username can be omitted if identical to $USER
or if set in ssh_config
-v verbose logging
-w NUM_SECONDS
index all files newer than NOW - NUM_SECONDS
-h print this message
```
### ssync-queue
```
ssync-queue [options] -l LOCAL_INDEX_FILE -r REMOTE_INDEX_FILE -o QUEUE_OUTPUT_FILE
OPTIONS
-l LOCAL_INDEX_FILE
target local index file
-r REMOTE_INDEX_FILE
target remote index file
-o QUEUE_OUTPUT_FILE
queue output file
-v verbose logging
-h print this message
```
### ssync-fetch
```
ssync-fetch [options] QUEUE_FILE DEST_DIR
OPTIONS
-r REMOTE_HOST
remote host to download from such as user@hostname
username can be omitted if identical to $USER
or if set in ssh_config
-k KEY_FILE
ssh-key file to use (needs to be non-interactive)
optional: will use default session key
or key set in ssh_config for REMOTE_HOST
-b BACKEND
sftp (default) | rsync
-v verbose logging
-h print this message
```
|