Girish Ramakrishnan
20a30b303b
s3: fix encoding of downloaded stream
2025-02-13 19:29:07 +01:00
Girish Ramakrishnan
59721a3f1a
s3: in v3, Body is a stream
...
https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/migrate-s3.html
2025-02-13 15:17:48 +01:00
Girish Ramakrishnan
cb6d531300
storage: automatically abort old multipart uploads in s3
2025-02-13 11:09:15 +01:00
Girish Ramakrishnan
a138425298
storage: start migration of s3 api
2025-02-12 23:04:37 +01:00
Girish Ramakrishnan
9888aa8c08
storage: rework listDir api to be a generator (like) function
2025-02-12 19:34:57 +01:00
Girish Ramakrishnan
b4d58f0609
aws: add a 20min timeout
...
in some services like b2, the multi-part copy just hangs. this allows
us to retry
2024-11-20 07:13:43 +05:30
Girish Ramakrishnan
ea72cef7f9
storage: remove getProviderStatus
2024-09-09 17:36:51 +02:00
Girish Ramakrishnan
4a29371907
s3: sometimes message is null and only code is valid
2024-08-13 07:08:33 +02:00
Girish Ramakrishnan
e9b6002f63
s3: fix exists check
2024-07-14 22:04:12 +02:00
Girish Ramakrishnan
1dc6b40a68
tgz: extract using tar-stream directly
...
we used have a fork of tar-fs. using tar-stream directly gives us
more control
2024-07-08 13:06:56 +02:00
Girish Ramakrishnan
e481606d0e
lint
2024-07-02 16:09:36 +02:00
Girish Ramakrishnan
934c701be2
vultr: fix copy of large objects
...
https://forum.cloudron.io/topic/10266/backups-are-failing
2023-10-26 09:51:07 +02:00
Girish Ramakrishnan
aa8c23c8b3
rework backup root
...
notes:
* backup root cannot come from backend. for dynamic mounts backend cannot know where it is mounted
* backupConfig is 3 parts - format / mount / password . there is also this rootPath (which should not be in db)
* password should be stored separately in settings at some point
* format has to be passed along everywhere because we allow restore from same backupConfig but different format. we do this by saving the format in the backups table
fixes #819
2023-08-15 22:51:45 +05:30
Girish Ramakrishnan
e6b85c2df7
remount does not need a backend hook
2023-08-15 08:55:38 +05:30
Girish Ramakrishnan
3d5c21d9ca
backups: encrypted backups must have .enc extension
2023-07-24 22:25:06 +05:30
Girish Ramakrishnan
050a82039a
getBackupProviderStatus -> getProviderStatus
2023-07-15 11:00:45 +05:30
Girish Ramakrishnan
c86059e070
backups: move limits into a sub object
...
fixes #817
2023-07-13 12:17:57 +05:30
Girish Ramakrishnan
4a1cdd4ef1
Update aws-sdk and suppress maintenance mode message
...
https://github.com/aws/aws-sdk-js/issues/4354
2023-05-11 22:18:00 +02:00
Girish Ramakrishnan
c4f4f3e914
logs: use %o to format error
...
otherwise, they are printed as multi-line and this messes up tail+date formatting
2023-04-16 10:49:59 +02:00
Girish Ramakrishnan
093fc98ae5
s3: add listing check
...
This is needed for situations like in cloudflare where the endpoint can
be mistakenly configured with the bucket name like https://xx.r2.cloudflarestorage.com/cloudron-backups .
The upload and del calls work but list and copy does not.
2023-01-17 11:13:03 +01:00
Girish Ramakrishnan
40bcfdba76
s3: throw any copy errors
2023-01-17 11:00:03 +01:00
Girish Ramakrishnan
c3793da5bb
split checkPrecondition so it can be used in cleaner as well
2022-10-02 17:41:21 +02:00
Girish Ramakrishnan
074e9cfd93
rename getRootPath to getBackupRootPath
2022-10-02 16:26:27 +02:00
Girish Ramakrishnan
9d17c6606b
rename to checkBackupPreconditions
...
since this is called only by the backup logic
2022-10-02 16:20:14 +02:00
Girish Ramakrishnan
ea01586b52
storage: make copy async
2022-04-30 16:24:45 -07:00
Girish Ramakrishnan
2280b7eaf5
Add S3MultipartDownloadStream
...
This extends the modern Readable class
2022-04-29 18:23:56 -07:00
Girish Ramakrishnan
801dddc269
inline S3ReadStream
2022-04-15 11:25:15 -05:00
Girish Ramakrishnan
9a886111ad
inline chunk.js
2022-04-15 09:25:54 -05:00
Girish Ramakrishnan
555f914537
remove lodash.chunk
2022-04-15 08:07:46 -05:00
Girish Ramakrishnan
185d5d66ad
even more constness
2022-04-14 20:30:00 -05:00
Girish Ramakrishnan
685bda35b9
storage: make remove and removeDir async
2022-04-14 16:16:20 -05:00
Girish Ramakrishnan
d54c03f0a0
storage: make exists async
2022-04-14 12:24:34 -05:00
Girish Ramakrishnan
11f7be2065
storage: verifyConfig is now async
2022-04-14 12:24:30 -05:00
Girish Ramakrishnan
a39e0ab934
storage: make remount async
2022-04-14 09:57:31 -05:00
Girish Ramakrishnan
b51082f7e4
storage: checkPreconditions is now async
2022-04-14 07:40:19 -05:00
Girish Ramakrishnan
9ec76c69ec
s3: make callback of getS3Config
2022-04-14 07:35:41 -05:00
Girish Ramakrishnan
452a4d9a75
backups: add remotePath
...
the main motivation is that id can be used in REST API routes. previously,
the id was a path and this had a "/" in it. This made /api/v1/backups/:backupId
not work.
2022-04-04 20:40:40 -07:00
Girish Ramakrishnan
54934c41a7
storage: rename getBackupPath to getBasePath
2022-04-04 14:08:24 -07:00
Girish Ramakrishnan
964c1a5f5a
remove field from errors
...
we have standardized on indexOf in error.message by now
2022-02-07 13:44:29 -08:00
Johannes Zellner
d1a7172895
Add remount route for mountlike backup storages
2021-10-11 18:12:11 +02:00
Girish Ramakrishnan
5129465e59
aws: const correctness
2021-06-23 14:30:00 -07:00
Girish Ramakrishnan
f2e0ee12a2
vultr: object storage
2021-06-16 22:36:01 -07:00
Girish Ramakrishnan
1e96606110
error.code is a number which causes crash at times in BoxError
2021-03-12 10:10:49 -08:00
Girish Ramakrishnan
91a229305d
missing backups: check if the s3 end point is valid
...
s3 api never return NotFound or ENOENT - https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
Sadly, DO/OVH etc just return NotFound instead of NoSuchKey. And we cannot
distinguish easily if we are talking to some s3 server or some random server.
This is applicable for things like say minio where maybe there is something
apache now just giving out 404 / NotFound.
2021-03-05 01:24:16 -08:00
Girish Ramakrishnan
4275114d28
s3: remove retry options for exists check
2021-03-04 23:40:23 -08:00
Girish Ramakrishnan
c4dffa393b
backups: remove entries from database that don't exist in storage
...
fixes #772
2021-02-19 11:34:22 -08:00
Girish Ramakrishnan
382ae7424d
async 3: the whilst and doWhilst test funcs are async
2021-02-04 16:39:47 -08:00
Girish Ramakrishnan
282040ed1b
gcs: use delete concurrency
2021-02-01 14:23:15 -08:00
Girish Ramakrishnan
f55300eba5
reduce DO spaces copy part size
2020-12-15 14:37:18 -08:00
Girish Ramakrishnan
a68ddcbbc4
Fix progress message
2020-12-14 19:58:44 -08:00
