Girish Ramakrishnan
53e9925880
backups: make id, provider a backend specific setting
...
the backend can stash whatever values it wants in the config.
just like the DNS backends, we make verifyConfig return a sanitized config.
added benefit is that extra user fields (via API) are also not dumped into the db.
2025-08-01 18:55:04 +02:00
Girish Ramakrishnan
ea419509f1
backups: add setup/teardown
...
1. add setup, teardown hooks
2. move the managed mount setup and teardown to filesystem backend
3. remove this vague storage.js
we should convert storageApi into a real object, so we don't have to
keep passing apiConfig around
2025-08-01 15:36:25 +02:00
Johannes Zellner
2386124089
Upcloud S3 does not return a Contents property if no keys are found
2025-07-22 18:55:44 +02:00
Girish Ramakrishnan
5548ed1b57
s3: fix malformed endpoint URL from older versions
...
this is specifically for linode, but may apply for other storages too
2025-07-16 09:12:57 +02:00
Girish Ramakrishnan
bb257f2bf2
s3: make delete objects work again again
2025-07-15 01:11:23 +02:00
Girish Ramakrishnan
76f365f7e8
s3: make delete objects work again
2025-07-15 00:13:40 +02:00
Girish Ramakrishnan
499336e752
s3: fix debug
2025-07-14 22:24:23 +02:00
Girish Ramakrishnan
e53ec3e43c
s3: add debugging code
2025-07-14 21:59:56 +02:00
Girish Ramakrishnan
15ac81c1cd
s3: format errors with .Code
2025-07-14 15:54:31 +02:00
Girish Ramakrishnan
241053e1a8
s3: error handling has changed in v3 api
2025-06-20 16:07:21 +02:00
Girish Ramakrishnan
48bf21a537
s3: use listObjectsV2 API
...
in some minio instances, listObjectV1 is not returning all the objects
2025-05-26 14:17:13 +02:00
Girish Ramakrishnan
20a30b303b
s3: fix encoding of downloaded stream
2025-02-13 19:29:07 +01:00
Girish Ramakrishnan
59721a3f1a
s3: in v3, Body is a stream
...
https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/migrate-s3.html
2025-02-13 15:17:48 +01:00
Girish Ramakrishnan
cb6d531300
storage: automatically abort old multipart uploads in s3
2025-02-13 11:09:15 +01:00
Girish Ramakrishnan
a138425298
storage: start migration of s3 api
2025-02-12 23:04:37 +01:00
Girish Ramakrishnan
9888aa8c08
storage: rework listDir api to be a generator (like) function
2025-02-12 19:34:57 +01:00
Girish Ramakrishnan
b4d58f0609
aws: add a 20min timeout
...
in some services like b2, the multi-part copy just hangs. this allows
us to retry
2024-11-20 07:13:43 +05:30
Girish Ramakrishnan
ea72cef7f9
storage: remove getProviderStatus
2024-09-09 17:36:51 +02:00
Girish Ramakrishnan
4a29371907
s3: sometimes message is null and only code is valid
2024-08-13 07:08:33 +02:00
Girish Ramakrishnan
e9b6002f63
s3: fix exists check
2024-07-14 22:04:12 +02:00
Girish Ramakrishnan
1dc6b40a68
tgz: extract using tar-stream directly
...
we used have a fork of tar-fs. using tar-stream directly gives us
more control
2024-07-08 13:06:56 +02:00
Girish Ramakrishnan
e481606d0e
lint
2024-07-02 16:09:36 +02:00
Girish Ramakrishnan
934c701be2
vultr: fix copy of large objects
...
https://forum.cloudron.io/topic/10266/backups-are-failing
2023-10-26 09:51:07 +02:00
Girish Ramakrishnan
aa8c23c8b3
rework backup root
...
notes:
* backup root cannot come from backend. for dynamic mounts backend cannot know where it is mounted
* backupConfig is 3 parts - format / mount / password . there is also this rootPath (which should not be in db)
* password should be stored separately in settings at some point
* format has to be passed along everywhere because we allow restore from same backupConfig but different format. we do this by saving the format in the backups table
fixes #819
2023-08-15 22:51:45 +05:30
Girish Ramakrishnan
e6b85c2df7
remount does not need a backend hook
2023-08-15 08:55:38 +05:30
Girish Ramakrishnan
3d5c21d9ca
backups: encrypted backups must have .enc extension
2023-07-24 22:25:06 +05:30
Girish Ramakrishnan
050a82039a
getBackupProviderStatus -> getProviderStatus
2023-07-15 11:00:45 +05:30
Girish Ramakrishnan
c86059e070
backups: move limits into a sub object
...
fixes #817
2023-07-13 12:17:57 +05:30
Girish Ramakrishnan
4a1cdd4ef1
Update aws-sdk and suppress maintenance mode message
...
https://github.com/aws/aws-sdk-js/issues/4354
2023-05-11 22:18:00 +02:00
Girish Ramakrishnan
c4f4f3e914
logs: use %o to format error
...
otherwise, they are printed as multi-line and this messes up tail+date formatting
2023-04-16 10:49:59 +02:00
Girish Ramakrishnan
093fc98ae5
s3: add listing check
...
This is needed for situations like in cloudflare where the endpoint can
be mistakenly configured with the bucket name like https://xx.r2.cloudflarestorage.com/cloudron-backups .
The upload and del calls work but list and copy does not.
2023-01-17 11:13:03 +01:00
Girish Ramakrishnan
40bcfdba76
s3: throw any copy errors
2023-01-17 11:00:03 +01:00
Girish Ramakrishnan
c3793da5bb
split checkPrecondition so it can be used in cleaner as well
2022-10-02 17:41:21 +02:00
Girish Ramakrishnan
074e9cfd93
rename getRootPath to getBackupRootPath
2022-10-02 16:26:27 +02:00
Girish Ramakrishnan
9d17c6606b
rename to checkBackupPreconditions
...
since this is called only by the backup logic
2022-10-02 16:20:14 +02:00
Girish Ramakrishnan
ea01586b52
storage: make copy async
2022-04-30 16:24:45 -07:00
Girish Ramakrishnan
2280b7eaf5
Add S3MultipartDownloadStream
...
This extends the modern Readable class
2022-04-29 18:23:56 -07:00
Girish Ramakrishnan
801dddc269
inline S3ReadStream
2022-04-15 11:25:15 -05:00
Girish Ramakrishnan
9a886111ad
inline chunk.js
2022-04-15 09:25:54 -05:00
Girish Ramakrishnan
555f914537
remove lodash.chunk
2022-04-15 08:07:46 -05:00
Girish Ramakrishnan
185d5d66ad
even more constness
2022-04-14 20:30:00 -05:00
Girish Ramakrishnan
685bda35b9
storage: make remove and removeDir async
2022-04-14 16:16:20 -05:00
Girish Ramakrishnan
d54c03f0a0
storage: make exists async
2022-04-14 12:24:34 -05:00
Girish Ramakrishnan
11f7be2065
storage: verifyConfig is now async
2022-04-14 12:24:30 -05:00
Girish Ramakrishnan
a39e0ab934
storage: make remount async
2022-04-14 09:57:31 -05:00
Girish Ramakrishnan
b51082f7e4
storage: checkPreconditions is now async
2022-04-14 07:40:19 -05:00
Girish Ramakrishnan
9ec76c69ec
s3: make callback of getS3Config
2022-04-14 07:35:41 -05:00
Girish Ramakrishnan
452a4d9a75
backups: add remotePath
...
the main motivation is that id can be used in REST API routes. previously,
the id was a path and this had a "/" in it. This made /api/v1/backups/:backupId
not work.
2022-04-04 20:40:40 -07:00
Girish Ramakrishnan
54934c41a7
storage: rename getBackupPath to getBasePath
2022-04-04 14:08:24 -07:00
Girish Ramakrishnan
964c1a5f5a
remove field from errors
...
we have standardized on indexOf in error.message by now
2022-02-07 13:44:29 -08:00
