Girish Ramakrishnan
555f914537
remove lodash.chunk
2022-04-15 08:07:46 -05:00
Girish Ramakrishnan
185d5d66ad
even more constness
2022-04-14 20:30:00 -05:00
Girish Ramakrishnan
685bda35b9
storage: make remove and removeDir async
2022-04-14 16:16:20 -05:00
Girish Ramakrishnan
d54c03f0a0
storage: make exists async
2022-04-14 12:24:34 -05:00
Girish Ramakrishnan
11f7be2065
storage: verifyConfig is now async
2022-04-14 12:24:30 -05:00
Girish Ramakrishnan
a39e0ab934
storage: make remount async
2022-04-14 09:57:31 -05:00
Girish Ramakrishnan
b51082f7e4
storage: checkPreconditions is now async
2022-04-14 07:40:19 -05:00
Girish Ramakrishnan
9ec76c69ec
s3: make callback of getS3Config
2022-04-14 07:35:41 -05:00
Girish Ramakrishnan
452a4d9a75
backups: add remotePath
...
the main motivation is that id can be used in REST API routes. previously,
the id was a path and this had a "/" in it. This made /api/v1/backups/:backupId
not work.
2022-04-04 20:40:40 -07:00
Girish Ramakrishnan
54934c41a7
storage: rename getBackupPath to getBasePath
2022-04-04 14:08:24 -07:00
Girish Ramakrishnan
964c1a5f5a
remove field from errors
...
we have standardized on indexOf in error.message by now
2022-02-07 13:44:29 -08:00
Johannes Zellner
d1a7172895
Add remount route for mountlike backup storages
2021-10-11 18:12:11 +02:00
Girish Ramakrishnan
5129465e59
aws: const correctness
2021-06-23 14:30:00 -07:00
Girish Ramakrishnan
f2e0ee12a2
vultr: object storage
2021-06-16 22:36:01 -07:00
Girish Ramakrishnan
1e96606110
error.code is a number which causes crash at times in BoxError
2021-03-12 10:10:49 -08:00
Girish Ramakrishnan
91a229305d
missing backups: check if the s3 end point is valid
...
s3 api never return NotFound or ENOENT - https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
Sadly, DO/OVH etc just return NotFound instead of NoSuchKey. And we cannot
distinguish easily if we are talking to some s3 server or some random server.
This is applicable for things like say minio where maybe there is something
apache now just giving out 404 / NotFound.
2021-03-05 01:24:16 -08:00
Girish Ramakrishnan
4275114d28
s3: remove retry options for exists check
2021-03-04 23:40:23 -08:00
Girish Ramakrishnan
c4dffa393b
backups: remove entries from database that don't exist in storage
...
fixes #772
2021-02-19 11:34:22 -08:00
Girish Ramakrishnan
382ae7424d
async 3: the whilst and doWhilst test funcs are async
2021-02-04 16:39:47 -08:00
Girish Ramakrishnan
282040ed1b
gcs: use delete concurrency
2021-02-01 14:23:15 -08:00
Girish Ramakrishnan
f55300eba5
reduce DO spaces copy part size
2020-12-15 14:37:18 -08:00
Girish Ramakrishnan
a68ddcbbc4
Fix progress message
2020-12-14 19:58:44 -08:00
Girish Ramakrishnan
0723b7d672
reduce copy concurrency to keep most providers happy
2020-12-14 17:26:44 -08:00
Girish Ramakrishnan
3d5599cdd9
b2: reduce copy part size
...
'Error copying snapshot/app_8b22dc8f-3e15-4314-8108-bcf1908a24df.tar.gz.enc (482405284 bytes): InternalError InternalError: too busy to complete copy - please try again' }
2020-11-29 11:16:05 -08:00
Girish Ramakrishnan
bedcd6fccf
Disable the timeout altogether for chunk to upload
2020-11-06 14:47:14 -08:00
Girish Ramakrishnan
df8a71cd8b
Each chunk can take up to 30 mins to upload
2020-11-06 00:05:53 -08:00
Girish Ramakrishnan
166c06c628
log the partSize
2020-09-10 00:09:54 -07:00
Girish Ramakrishnan
fc08f9823e
s3: copy parts in parallel
2020-09-03 14:31:56 -07:00
Girish Ramakrishnan
e25ccc5e9a
Double the timeout for upload now that chunks can have custom sizes
2020-08-20 16:50:58 -07:00
Girish Ramakrishnan
0e00492f54
backups: make part size configurable
2020-08-19 14:39:20 -07:00
Girish Ramakrishnan
df6e3eb1e6
Add deleteConcurrency setting
2020-08-11 09:14:09 -07:00
Girish Ramakrishnan
91288c96b1
s3: set queue size to 3
...
fixes #691
2020-08-07 00:28:00 -07:00
Johannes Zellner
8624e2260d
add storage api to make preflight checks
...
Currently there is only disk space checking but sshfs and cifs need
mount point checking as well
2020-06-08 16:25:05 +02:00
Johannes Zellner
330b4a613c
Retrieve the backupPath from the storage provider itself
2020-06-05 13:27:18 +02:00
Girish Ramakrishnan
4a0a934a76
start using vhost style for accessing s3 style storage
...
if bucket name has a '.', accept self-signed
fixes #680
2020-05-27 17:50:37 -07:00
Girish Ramakrishnan
f7c406bec9
s3: bucket name cannot contain _ or capitals or .
...
we can make it more elaborate, but not sure if it's needed
https://blogs.easydynamics.com/2016/10/24/aws-s3-bucket-name-validation-regex/
2020-05-27 17:01:42 -07:00
Johannes Zellner
9bdeff0a39
Always use constants.SECRET_PLACEHOLDER
2020-05-14 23:02:02 +02:00
Girish Ramakrishnan
101dc3a93c
s3: do not retry when testing config
2020-05-12 22:45:01 -07:00
Girish Ramakrishnan
4d050725b7
storage: done events must be called next tick
...
It seems that listDir() returns synchronously (!), not sure how.
This results in the done event getting called with an error but
the EE event handlers are not setup yet.
2020-02-11 11:48:49 -08:00
Girish Ramakrishnan
57597bd103
s3: bucket name cannot contain /
2020-02-11 11:19:47 -08:00
Girish Ramakrishnan
905bb92bad
s3: ensure BoxError return
2019-12-05 21:50:44 -08:00
Girish Ramakrishnan
85e17b570b
Use whilst instead of forever
...
this gets rid of the Error object
2019-12-04 11:17:44 -08:00
Girish Ramakrishnan
30eccfb54b
Use BoxError instead of Error in all places
...
This moves everything other than the addon code and some 'done' logic
2019-12-04 11:02:54 -08:00
Girish Ramakrishnan
37c6a96a3a
s3: if etag is not present, flag as error
2019-11-20 12:53:36 -08:00
Girish Ramakrishnan
8c9ce30d29
Move BackupsError to BoxError
2019-10-22 21:03:47 -07:00
Girish Ramakrishnan
4fba216af9
scaleway: try to keep part numbers low
2019-09-30 20:42:37 -07:00
Girish Ramakrishnan
800e25a7a7
Fix crash because params was undefined
2019-05-10 13:07:29 -07:00
Girish Ramakrishnan
0835a6d690
caas: remove storage backend
2019-05-08 15:33:23 -07:00
Girish Ramakrishnan
3ded411c15
caas: migrate from fqdn to boxId in backup api
2019-04-26 10:55:50 -07:00
Girish Ramakrishnan
706c970314
caas: fixup routes
2019-04-26 10:19:37 -07:00
