Girish Ramakrishnan
0835a6d690
caas: remove storage backend
2019-05-08 15:33:23 -07:00
Girish Ramakrishnan
3ded411c15
caas: migrate from fqdn to boxId in backup api
2019-04-26 10:55:50 -07:00
Girish Ramakrishnan
706c970314
caas: fixup routes
2019-04-26 10:19:37 -07:00
Girish Ramakrishnan
d1582a435d
readdirp args have changed
2019-04-24 10:47:04 -07:00
Girish Ramakrishnan
1119a62576
fs: emit progress messages for copy and removeDir
2019-04-03 11:54:46 -07:00
Girish Ramakrishnan
9a3603fea3
Add progress messages for multi-part copy
2019-03-26 11:58:32 -07:00
Girish Ramakrishnan
a88893b10a
remove/inject backups secret fields
...
follow same pattern as dns code
fixes #615
2019-02-09 20:44:05 -08:00
Girish Ramakrishnan
ee65089eb7
s3: make copying and uploading significantly faster
2019-01-14 13:47:07 -08:00
Girish Ramakrishnan
fd4057df94
shell.exec -> shell.spawn
2018-11-17 19:26:19 -08:00
Girish Ramakrishnan
8efd496579
storage: add access denied function (unused)
2018-09-10 10:41:08 -07:00
Girish Ramakrishnan
d302dbc634
Fix tests
2018-08-02 14:59:50 -07:00
Girish Ramakrishnan
ef5dc7311f
Do not encrypt empty path
2018-07-31 19:41:03 -07:00
Girish Ramakrishnan
a1070b7da3
storage: fix file not found message
2018-07-30 07:47:52 -07:00
Girish Ramakrishnan
99ad3e499d
s3: do not rely on entry.s3
2018-07-29 09:00:57 -07:00
Girish Ramakrishnan
0b10e2b332
s3: add listDir
...
Fixes #440
2018-07-28 09:05:44 -07:00
Girish Ramakrishnan
f546d53ca2
gcs: add listDir
...
part of #440
2018-07-28 03:51:32 -07:00
Girish Ramakrishnan
2bcc0eef96
fs: downloadDir is unused
2018-07-27 16:56:30 -07:00
Girish Ramakrishnan
ba8a549235
fs: open the source stream using download()
2018-07-27 15:23:48 -07:00
Girish Ramakrishnan
737541f707
storage: standardize the listDir API
2018-07-27 15:23:26 -07:00
Girish Ramakrishnan
94cb222869
filesystem: Use listDir
...
We can now refactor all the iterating/fs write logic into backups.js
(required for restoring encrypted backups)
2018-07-27 14:09:52 -07:00
Girish Ramakrishnan
df98847535
Remove backupDone from storage interface
...
This is only used by caas
2018-07-27 12:08:19 -07:00
Girish Ramakrishnan
7fd1bb8597
backup: Add externalDisk option to fs backend
...
This merely confirms from the user understands that backups have to
stored on an external disk.
2018-06-07 11:14:13 -07:00
Girish Ramakrishnan
2a18070016
do-spaces: Force retry of 4xx error codes when copying
2018-03-21 15:41:21 -07:00
Girish Ramakrishnan
e0ece06b26
s3: improved copy logging
2018-03-21 14:22:41 -07:00
Girish Ramakrishnan
6adf88a6e5
Make uploads work with very slow upload speeds
...
chunk uploads get a timeout of 2mins (derived from http.timeout).
On servers like kimsufi, uploads takes forever (100 MB/sec limit).
Currently, our upload code does not dynamically adapt itself to
changing the concurrency when network is slow.
2018-03-20 19:37:45 -07:00
Girish Ramakrishnan
ce33681c37
Dump etag info
2018-03-20 18:19:14 -07:00
Girish Ramakrishnan
565eed015f
Add better backup logs
2018-03-20 16:41:45 -07:00
Girish Ramakrishnan
98e19e6df5
fix upload errors causing double callback
2018-02-27 19:16:03 -08:00
Girish Ramakrishnan
b6fb49956f
s3: better debug output when copying
2018-02-22 12:41:18 -08:00
Girish Ramakrishnan
0bba985ff1
storage: Add implementation note
2018-02-22 12:30:55 -08:00
Girish Ramakrishnan
3c8c15db01
s3: use a constant backoff since it takes forever to fail otherwise
2018-02-22 12:30:44 -08:00
Girish Ramakrishnan
c8a6294772
lint
2018-02-22 12:24:16 -08:00
Girish Ramakrishnan
cea83889ec
s3: Fix issue where it takes forever to timeout if the backend is down
2018-02-22 12:19:23 -08:00
Girish Ramakrishnan
2ecb66afd7
s3: cleanup code
2018-02-22 12:16:01 -08:00
Girish Ramakrishnan
e6c07fc148
merge the done callback into the main code
2018-02-22 11:06:28 -08:00
Girish Ramakrishnan
1f30a4f3ea
Make s3.deleteObjects return error
2018-02-22 11:05:29 -08:00
Girish Ramakrishnan
0bfdaeb2fb
rename to chunkSize
2018-02-22 11:01:04 -08:00
Girish Ramakrishnan
e022dbf8a6
Revert "merge the done callback into the main code"
...
This reverts commit c39bec8cc1
2018-02-22 10:58:56 -08:00
Girish Ramakrishnan
6075a7a890
typo
2018-02-22 10:34:48 -08:00
Girish Ramakrishnan
28b864c346
sos: Copy in 96M chunks as recommended by exoscale
2018-02-22 10:31:56 -08:00
Girish Ramakrishnan
c39bec8cc1
merge the done callback into the main code
2018-02-21 20:17:58 -08:00
Girish Ramakrishnan
727a25f491
DO Spaces: multipart copy now works
2018-02-20 14:48:03 -08:00
Girish Ramakrishnan
77fe595970
lint
2018-02-04 00:15:41 -08:00
Girish Ramakrishnan
14529d313a
sos: CopySource requires fancier encoding rules
2018-02-04 00:12:28 -08:00
Girish Ramakrishnan
f607010396
sos: remove exoscale specific hacks which are obsolete now
2018-02-02 20:13:51 -08:00
Girish Ramakrishnan
e4b12f0c4e
gcs: make testfile deletion work
...
It seems there is a race where the delete gets triggered even before the
file upload is complete. as a result, the delete succeeds but the file
is left on gcs.
2017-12-16 21:36:33 +05:30
Girish Ramakrishnan
61b56d4679
gcs: keyFilename is not used
...
also, some linter fixes
2017-12-16 21:36:29 +05:30
Girish Ramakrishnan
5c4ae6066d
gcs: lint
2017-12-15 17:34:09 +05:30
Girish Ramakrishnan
a35e048665
gcs: oldFilePath is not defined
2017-12-15 17:31:05 +05:30
Girish Ramakrishnan
48f6c39ae5
gcs: Make requires alphabetical
2017-12-15 17:28:45 +05:30
