enable cgroup memory accounting at least for the pi

http://sambal.org/2014/02/passing-options-node-shebang-line/ was a cool trick but not needed at all.

https://bitbucket.org/chromiumembedded/cef/issues/483/dont-always-add-the-expose-gc-v8-flag
says it will change behavior in ways we don't want.

CHANGES

@@ -1899,3 +1899,237 @@
 * graphs: sort disk contents by usage
 * backups: show apps that are not automatically backed up in backup view
 * turn: deny local address peers https://www.rtcsec.com/2020/04/01-slack-webrtc-turn-compromise/
+
+[5.1.3]
+* Fix crash with misconfigured reverse proxy
+* Fix issue where invitation links are not working anymore
+
+[5.1.4]
+* Add support for custom .well-known documents to be served
+* Add ECDHE-RSA-AES128-SHA256 to cipher list
+* Fix GPG signature verification
+
+[5.1.5]
+* Check for .well-known routes upstream as fallback. This broke nextcloud's caldav/carddav
+
+[5.2.0]
+* acme: request ECC certs
+* less-strict DKIM check to allow users to set a stronger DKIM key
+* Add members only flag to mailing list
+* oauth: add backward compat layer for backup and uninstall
+* fix bug in disk usage sorting
+* mail: aliases can be across domains
+* mail: allow an external MX to be set
+* Add UI to download backup config as JSON (and import it)
+* Ensure stopped apps are getting backed up
+* Add OVH Object Storage backend
+* Add per-app redis status and configuration to Services
+* spam: large emails were not scanned
+* mail relay: fix delivery event log
+* manual update check always gets the latest updates
+* graphs: fix issue where large number of apps would crash the box code (query param limit exceeded)
+* backups: fix various security issues in encypted backups (thanks @mehdi)
+* graphs: add app graphs
+* older encrypted backups cannot be used in this version
+* Add backup listing UI
+* stopping an app will stop dependent services
+* Add new wasabi s3 storage region us-east-2
+* mail: Fix bug where SRS translation was done on the main domain instead of mailing list domain
+* backups: add retention policy
+* Drop `NET_RAW` caps from container preventing sniffing of network traffic
+
+[5.2.1]
+* Fix app disk graphs
+* restart apps on addon container change
+
+[5.2.2]
+* regression: import UI
+* Mbps -> MBps
+* Remove verbose logs
+* Set dmode in tar extract
+* mail: fix crash in audit logs
+* import: fix crash because encryption is unset
+* create redis with the correct label
+
+[5.2.3]
+* Do not restart stopped apps
+
+[5.2.4]
+* mail: enable/disable incoming mail was showing an error
+* Do not trigger backup of stopped apps. Instead, we will just retain it's existing backups
+  based on retention policy
+* remove broken disk graphs
+* fix OVH backups
+
+[5.3.0]
+* better nginx config for higher loads
+* backups: add CIFS storage provider
+* backups: add SSHFS storage provider
+* backups: add NFS storage provider
+* s3: use vhost style
+* Fix crash when redis config was set
+* Update schedule was unselected in the UI
+* cloudron-setup: --provider is now optional
+* show warning for unstable updates
+* add forumUrl to app manifest
+* postgresql: add unaccent extension for peertube
+* mail: Add Auto-Submitted header to NDRs
+* backups: ensure that the latest backup of installed apps is always preserved
+* add nginx logs
+* mail: make authentication case insensitive
+* Fix timeout issues in postgresql and mysql addon
+* Do not count stopped apps for memory use
+* LDAP group synchronization
+
+[5.3.1]
+* better nginx config for higher loads
+* backups: add CIFS storage provider
+* backups: add SSHFS storage provider
+* backups: add NFS storage provider
+* s3: use vhost style
+* Fix crash when redis config was set
+* Update schedule was unselected in the UI
+* cloudron-setup: --provider is now optional
+* show warning for unstable updates
+* add forumUrl to app manifest
+* postgresql: add unaccent extension for peertube
+* mail: Add Auto-Submitted header to NDRs
+* backups: ensure that the latest backup of installed apps is always preserved
+* add nginx logs
+* mail: make authentication case insensitive
+* Fix timeout issues in postgresql and mysql addon
+* Do not count stopped apps for memory use
+* LDAP group synchronization
+
+[5.3.2]
+* Do not install sshfs package
+* 'provider' is not required anymore in various API calls
+* redis: Set maxmemory and maxmemory-policy
+* Add mlock capability to manifest (for vault app)
+
+[5.3.3]
+* Fix issue where some postinstall messages where causing angular to infinite loop
+
+[5.3.4]
+* Fix issue in database error handling
+
+[5.4.0]
+* Update nginx to 1.18 for various security fixes
+* Add ping capability (for statping app)
+* Fix bug where aliases were displayed incorrectly in SOGo
+* Add univention as LDAP provider
+* Bump max_connection for postgres addon to 200
+* mail: Add pagination to mailing list API
+* Allow admin to lock email and display name of users
+* Allow admin to ensure all users have 2FA setup
+* ami: fix regression where we didn't send provider as part of get status call
+* nginx: hide version
+* backups: add b2 provider
+* Add filemanager webinterface
+* Add darkmode
+* Add note that password reset and invite links expire in 24 hours
+
+[5.4.1]
+* Update nginx to 1.18 for various security fixes
+* Add ping capability (for statping app)
+* Fix bug where aliases were displayed incorrectly in SOGo
+* Add univention as LDAP provider
+* Bump max_connection for postgres addon to 200
+* mail: Add pagination to mailing list API
+* Allow admin to lock email and display name of users
+* Allow admin to ensure all users have 2FA setup
+* ami: fix regression where we didn't send provider as part of get status call
+* nginx: hide version
+* backups: add b2 provider
+* Add filemanager webinterface
+* Add darkmode
+* Add note that password reset and invite links expire in 24 hours
+
+[5.5.0]
+* postgresql: update to PostgreSQL 11
+* postgresql: add citext extension to whitelist for loomio
+* postgresql: add btree_gist,postgres_fdw,pg_stat_statements,plpgsql extensions for gitlab
+* SFTP/Filebrowser: fix access of external data directories
+* Fix contrast issues in dark mode
+* Add option to delete mailbox data when mailbox is delete
+* Allow days/hours of backups and updates to be configurable
+* backup cleaner: fix issue where referenced backups where not counted against time periods
+* route53: fix issue where verification failed if user had more than 100 zones
+* rework task workers to run them in a separate cgroup
+* backups: now much faster thanks to reworking of task worker
+* When custom fallback cert is set, make sure it's used over LE certs
+* mongodb: update to MongoDB 4.0.19
+* List groups ordered by name
+* Invite links are now valid for a week
+* Update release GPG key
+* Add pre-defined variables ($CLOUDRON_APPID) for better post install messages
+* filemanager: show folder first
+
+[5.6.0]
+* Remove IP nginx configuration that redirects to dashboard after activation
+* dashboard: looks for search string in app title as well
+* Add vaapi caps for transcoding
+* Fix issue where the long mongodb database names where causing app indices of rocket.chat to overflow (> 127)
+* Do not resize swap if swap file exists. This means that users can now control how swap is allocated on their own.
+* SFTP: fix issue where parallel rebuilds would cause an error
+* backups: make part size configurable
+* mail: set max email size
+* mail: allow mail server location to be set
+* spamassassin: custom configs and wl/bl
+* Do not automatically update to unstable release
+* scheduler: reduce container churn
+* mail: add API to set banner
+* Fix bug where systemd 237 ignores --nice value in systemd-run
+* postgresql: enable uuid-ossp extension
+* firewall: add blocklist
+* HTTP URLs now redirect directly to the HTTPS of the final domain
+* linode: Add singapore region
+* ovh: add sydney region
+* s3: makes multi-part copies in parallel
+
+[5.6.1]
+* Blocklists are now stored in a text file instead of json
+* regenerate nginx configs
+
+[5.6.2]
+* Update docker to 19.03.12
+* Fix sorting of user listing in the UI
+* namecheap: fix crash when server returns invalid response
+* unlink ghost file automatically on successful login
+* Bump mysql addon connection limit to 200
+* Fix install issue where `/dev/dri` may not be present
+* import: when importing filesystem backups, the input box is a path
+* firewall: fix race condition where blocklist was not added in correct position in the FORWARD chain
+* services: fix issue where services where scaled up/down too fast
+* turn: realm variable was not updated properly on dashboard change
+* nginx: add splash pages for IP based browser access
+* Give services panel a separate top-level view
+* Add app state filter
+* gcs: copy concurrency was not used
+* Mention why an app update cannot be applied and provide shortcut to start the app if stopped
+* Remove version from footer into the setting view
+* Give services panel a separate top-level view
+* postgresql: set collation order explicity when creating database to C.UTF-8 (for confluence)
+* rsync: fix error while goes missing when syncing
+* Pre-select app domain by default in the redirection drop down
+* robots: preseve leading and trailing whitespaces/newlines
+
+[5.6.3]
+* Fix postgres locale issue
+
+[6.0.0]
+* Focal support
+* Reduce duration of self-signed certs to 800 days
+* Better backup config filename when downloading
+* branding: footer can have template variables like %YEAR% and %VERSION%
+* sftp: secure the API with a token
+* filemanager: Add extract context menu item
+* Do not download docker images if present locally
+* sftp: disable access to non-admins by default
+* postgresql: whitelist pgcrypto extension for loomio
+* filemanager: Add new file creation action and collapse new and upload actions
+* rsync: add warning to remove lifecycle rules
+* Add volume management
+* backups: adjust node's heap size based on memory limit
+* s3: diasble per-chunk timeout
+

README.md

@@ -29,9 +29,9 @@ anyone to effortlessly host web applications on their server on their own terms.
 * Trivially migrate to another server keeping your apps and data (for example, switch your
   infrastructure provider or move to a bigger server).
 
-* Comprehensive [REST API](https://cloudron.io/documentation/developer/api/).
+* Comprehensive [REST API](https://docs.cloudron.io/api/).
 
-* [CLI](https://cloudron.io/documentation/cli/) to configure apps.
+* [CLI](https://docs.cloudron.io/custom-apps/cli/) to configure apps.
 
 * Alerts, audit logs, graphs, dns management ... and much more
 
@@ -41,25 +41,37 @@ Try our demo at https://my.demo.cloudron.io (username: cloudron password: cloudr
 
 ## Installing
 
-[Install script](https://cloudron.io/documentation/installation/) - [Pricing](https://cloudron.io/pricing.html)
+[Install script](https://docs.cloudron.io/installation/) - [Pricing](https://cloudron.io/pricing.html)
 
 **Note:** This repo is a small part of what gets installed on your server - there is
 the dashboard, database addons, graph container, base image etc. Cloudron also relies
 on external services such as the App Store for apps to be installed. As such, don't
 clone this repo and npm install and expect something to work.
 
-## Documentation
+## Development
 
-* [Documentation](https://cloudron.io/documentation/)
+This is the backend code of Cloudron. The frontend code is [here](https://git.cloudron.io/cloudron/dashboard).
 
-## Related repos
+The way to develop is to first install a full instance of Cloudron in a VM. Then you can use the [hotfix](https://git.cloudron.io/cloudron/cloudron-machine)
+tool to patch the VM with the latest code.
 
-The [base image repo](https://git.cloudron.io/cloudron/docker-base-image) is the parent image of all
-the containers in the Cloudron.
+```
+SSH_PASSPHRASE=sshkeypassword cloudron-machine hotfix --cloudron my.example.com --release 6.0.0 --ssh-key keyname
+```
 
-## Community
+## License
 
-* [Chat](https://chat.cloudron.io)
+Please note that the Cloudron code is under a source-available license. This is not the same as an
+open source license but ensures the code is available for introspection (and hacking!).
+
+## Contributions
+
+Just to give some heads up, we are a bit restrictive in merging changes. We are a small team and
+would like to keep our maintenance burden low. It might be best to discuss features first in the [forum](https://forum.cloudron.io),
+to also figure out how many other people will use it to justify maintenance for a feature.
+
+## Support
+
+* [Documentation](https://docs.cloudron.io/)
 * [Forum](https://forum.cloudron.io/)
-* [Support](mailto:support@cloudron.io)
 

baseimage/initializeBaseUbuntuImage.sh

@@ -4,8 +4,7 @@ set -euv -o pipefail
 
 readonly SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
-readonly arg_provider="${1:-generic}"
-readonly arg_infraversionpath="${SOURCE_DIR}/${2:-}"
+readonly arg_infraversionpath="${SOURCE_DIR}/../src"
 
 function die {
     echo $1
@@ -14,6 +13,12 @@ function die {
 
 export DEBIAN_FRONTEND=noninteractive
 
+readonly ubuntu_codename=$(lsb_release -cs)
+readonly ubuntu_version=$(lsb_release -rs)
+
+# readonly arch="amd64"
+readonly arch="arm64"
+
 # hold grub since updating it breaks on some VPS providers. also, dist-upgrade will trigger it
 apt-mark hold grub* >/dev/null
 apt-get -o Dpkg::Options::="--force-confdef" update -y
@@ -27,9 +32,8 @@ debconf-set-selections <<< 'mysql-server mysql-server/root_password_again passwo
 
 # this enables automatic security upgrades (https://help.ubuntu.com/community/AutomaticSecurityUpdates)
 # resolvconf is needed for unbound to work property after disabling systemd-resolved in 18.04
-ubuntu_version=$(lsb_release -rs)
-ubuntu_codename=$(lsb_release -cs)
 gpg_package=$([[ "${ubuntu_version}" == "16.04" ]] && echo "gnupg" || echo "gpg")
+mysql_package=$([[ "${ubuntu_version}" == "20.04" ]] && echo "mysql-server-8.0" || echo "mysql-server-5.7")
 apt-get -y install \
     acl \
     build-essential \
@@ -39,11 +43,11 @@ apt-get -y install \
     debconf-utils \
     dmsetup \
     $gpg_package \
+    ipset \
     iptables \
     libpython2.7 \
-    linux-generic \
     logrotate \
-    mysql-server-5.7 \
+    $mysql_package \
     openssh-server \
     pwgen \
     resolvconf \
@@ -53,17 +57,19 @@ apt-get -y install \
     unbound \
     xfsprogs
 
-if [[ "${ubuntu_version}" == "16.04" ]]; then
-    echo "==> installing nginx for xenial for TLSv3 support"
-
-    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.14.0-1~xenial_amd64.deb -o /tmp/nginx.deb
-    # apt install with install deps (as opposed to dpkg -i)
-    apt install -y /tmp/nginx.deb
-    rm /tmp/nginx.deb
+# TODO make it more generic for arm
+if [[ "${arch}" == "arm64" ]]; then
+    apt-get install -y linux-raspi
 else
-    apt install -y nginx-full
+    apt-get isntall -y linux-generic
 fi
 
+echo "==> installing nginx for ${ubuntu_codename} for TLSv3 support"
+curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_${arch}.deb -o /tmp/nginx.deb
+# apt install with install deps (as opposed to dpkg -i)
+apt install -y /tmp/nginx.deb
+rm /tmp/nginx.deb
+
 # on some providers like scaleway the sudo file is changed and we want to keep the old one
 apt-get -o Dpkg::Options::="--force-confold" install -y sudo
 
@@ -73,7 +79,11 @@ cp /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/20auto-upg
 
 echo "==> Installing node.js"
 mkdir -p /usr/local/node-10.18.1
-curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-10.18.1
+if [[ "${arch}" == "arm64" ]]; then
+    curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-arm64.tar.gz | tar zxf - --strip-components=1 -C /usr/local/node-10.18.1
+else
+    curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxf - --strip-components=1 -C /usr/local/node-10.18.1
+fi
 ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
 ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
 apt-get install -y python   # Install python which is required for npm rebuild
@@ -87,9 +97,9 @@ mkdir -p /etc/systemd/system/docker.service.d
 echo -e "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2" > /etc/systemd/system/docker.service.d/cloudron.conf
 
 # there are 3 packages for docker - containerd, CLI and the daemon
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.2-3_amd64.deb" -o /tmp/containerd.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/${arch}/containerd.io_1.2.13-2_${arch}.deb" -o /tmp/containerd.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/${arch}/docker-ce-cli_19.03.12~3-0~ubuntu-${ubuntu_codename}_${arch}.deb" -o /tmp/docker-ce-cli.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/${arch}/docker-ce_19.03.12~3-0~ubuntu-${ubuntu_codename}_${arch}.deb" -o /tmp/docker.deb
 # apt install with install deps (as opposed to dpkg -i)
 apt install -y /tmp/containerd.deb  /tmp/docker-ce-cli.deb /tmp/docker.deb
 rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
@@ -100,11 +110,15 @@ if [[ "${storage_driver}" != "overlay2" ]]; then
     exit 1
 fi
 
-# do not upgrade grub because it might prompt user and break this script
-echo "==> Enable memory accounting"
-apt-get -y --no-upgrade install grub2-common
-sed -e 's/^GRUB_CMDLINE_LINUX="\(.*\)"$/GRUB_CMDLINE_LINUX="\1 cgroup_enable=memory swapaccount=1 panic_on_oops=1 panic=5"/' -i /etc/default/grub
-update-grub
+if [[ "${arch}" == "arm64" ]]; then
+    echo -n " cgroup_enable=memory swapaccount=1 panic_on_oops=1 panic=5"  >> /boot/firmware/cmdline.txt
+else
+    # do not upgrade grub because it might prompt user and break this script
+    echo "==> Enable memory accounting"
+    apt-get -y --no-upgrade install grub2-common
+    sed -e 's/^GRUB_CMDLINE_LINUX="\(.*\)"$/GRUB_CMDLINE_LINUX="\1 cgroup_enable=memory swapaccount=1 panic_on_oops=1 panic=5"/' -i /etc/default/grub
+    update-grub
+fi
 
 echo "==> Downloading docker images"
 if [ ! -f "${arg_infraversionpath}/infra_version.js" ]; then
@@ -126,6 +140,10 @@ if ! apt-get install -y libcurl3-gnutls collectd collectd-utils; then
     echo "Failed to install collectd. Presumably because of http://mailman.verplant.org/pipermail/collectd/2015-March/006491.html"
     sed -e 's/^FQDNLookup true/FQDNLookup false/' -i /etc/collectd/collectd.conf
 fi
+# https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1872281
+
+[[ "${ubuntu_version}" == "20.04" && "${arch}" == "amd64" ]] && echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
+[[ "${ubuntu_version}" == "20.04" && "${arch}" == "arm64" ]] && echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-aarch64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
 
 echo "==> Configuring host"
 sed -e 's/^#NTP=/NTP=0.ubuntu.pool.ntp.org 1.ubuntu.pool.ntp.org 2.ubuntu.pool.ntp.org 3.ubuntu.pool.ntp.org/' -i /etc/systemd/timesyncd.conf
@@ -134,11 +152,13 @@ timedatectl set-ntp 1
 timedatectl set-timezone UTC
 
 echo "==> Adding sshd configuration warning"
-sed -e '/Port 22/ i # NOTE: Cloudron only supports moving SSH to port 202. See https://cloudron.io/documentation/security/#securing-ssh-access' -i /etc/ssh/sshd_config
+sed -e '/Port 22/ i # NOTE: Cloudron only supports moving SSH to port 202. See https://docs.cloudron.io/security/#securing-ssh-access' -i /etc/ssh/sshd_config
 
 # https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1701068
 echo "==> Disabling motd news"
-sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
+if [ -f "/etc/default/motd-news" ]; then
+    sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
+fi
 
 # Disable bind for good measure (on online.net, kimsufi servers these are pre-installed and conflicts with unbound)
 systemctl stop bind9 || true

box.js

@@ -2,57 +2,61 @@
 
 'use strict';
 
-// prefix all output with a timestamp
-// debug() already prefixes and uses process.stderr NOT console.*
-['log', 'info', 'warn', 'debug', 'error'].forEach(function (log) {
-    var orig = console[log];
-    console[log] = function () {
-        orig.apply(console, [new Date().toISOString()].concat(Array.prototype.slice.call(arguments)));
-    };
-});
-
-require('supererror')({ splatchError: true });
-
 let async = require('async'),
-    constants = require('./src/constants.js'),
     dockerProxy = require('./src/dockerproxy.js'),
+    fs = require('fs'),
     ldap = require('./src/ldap.js'),
+    paths = require('./src/paths.js'),
     server = require('./src/server.js');
 
-console.log();
-console.log('==========================================');
-console.log(`           Cloudron ${constants.VERSION}  `);
-console.log('==========================================');
-console.log();
+const NOOP_CALLBACK = function () { };
+
+function setupLogging(callback) {
+    if (process.env.BOX_ENV === 'test') return callback();
+
+    var logfileStream = fs.createWriteStream(paths.BOX_LOG_FILE, { flags:'a' });
+    process.stdout.write = process.stderr.write = logfileStream.write.bind(logfileStream);
+
+    callback();
+}
 
 async.series([
+    setupLogging,
     server.start,
     ldap.start,
     dockerProxy.start
 ], function (error) {
     if (error) {
-        console.error('Error starting server', error);
+        console.log('Error starting server', error);
         process.exit(1);
     }
-    console.log('Cloudron is up and running');
-});
-
-var NOOP_CALLBACK = function () { };
-
-process.on('SIGINT', function () {
-    console.log('Received SIGINT. Shutting down.');
-
-    server.stop(NOOP_CALLBACK);
-    ldap.stop(NOOP_CALLBACK);
-    dockerProxy.stop(NOOP_CALLBACK);
-    setTimeout(process.exit.bind(process), 3000);
-});
-
-process.on('SIGTERM', function () {
-    console.log('Received SIGTERM. Shutting down.');
-
-    server.stop(NOOP_CALLBACK);
-    ldap.stop(NOOP_CALLBACK);
-    dockerProxy.stop(NOOP_CALLBACK);
-    setTimeout(process.exit.bind(process), 3000);
+
+    // require those here so that logging handler is already setup
+    require('supererror');
+    const debug = require('debug')('box:box');
+
+    process.on('SIGINT', function () {
+        debug('Received SIGINT. Shutting down.');
+
+        server.stop(NOOP_CALLBACK);
+        ldap.stop(NOOP_CALLBACK);
+        dockerProxy.stop(NOOP_CALLBACK);
+        setTimeout(process.exit.bind(process), 3000);
+    });
+
+    process.on('SIGTERM', function () {
+        debug('Received SIGTERM. Shutting down.');
+
+        server.stop(NOOP_CALLBACK);
+        ldap.stop(NOOP_CALLBACK);
+        dockerProxy.stop(NOOP_CALLBACK);
+        setTimeout(process.exit.bind(process), 3000);
+    });
+
+    process.on('uncaughtException', function (error) {
+        console.error((error && error.stack) ? error.stack : error);
+        setTimeout(process.exit.bind(process, 1), 3000);
+    });
+
+    console.log(`Cloudron is up and running. Logs are at ${paths.BOX_LOG_FILE}`); // this goes to journalctl
 });

migrations/20160208164735-groups-add-admin.js

@@ -12,8 +12,6 @@ exports.up = function(db, callback) {
 			db.all('SELECT * FROM users WHERE admin=1', function (error, results) {
 				if (error) return done(error);
 
-				console.dir(results);
-
 				async.eachSeries(results, function (r, next) {
 					db.runSql('INSERT INTO groupMembers (groupId, userId) VALUES (?, ?)', [ ADMIN_GROUP_ID, r.id ], next);
 				}, done);

migrations/20200417235435-mailboxes-add-membersOnly.js

@@ -0,0 +1,17 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes ADD COLUMN membersOnly BOOLEAN DEFAULT 0', function (error) {
+        if (error) return callback(error);
+
+        callback();
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE users DROP COLUMN membersOnly', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20200420013715-mailboxes-add-aliasDomain.js

@@ -0,0 +1,28 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD COLUMN aliasDomain VARCHAR(128)'),
+        function setAliasDomain(done) {
+            db.all('SELECT * FROM mailboxes', function (error, mailboxes) {
+                async.eachSeries(mailboxes, function (mailbox, iteratorDone) {
+                    if (!mailbox.aliasTarget) return iteratorDone();
+
+                    db.runSql('UPDATE mailboxes SET aliasDomain=? WHERE name=? AND domain=?', [ mailbox.domain, mailbox.name, mailbox.domain ], iteratorDone);
+                }, done);
+            });
+        },
+        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD CONSTRAINT mailboxes_aliasDomain_constraint FOREIGN KEY(aliasDomain) REFERENCES mail(domain)'),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes CHANGE aliasTarget aliasName VARCHAR(128)')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE mailboxes DROP FOREIGN KEY mailboxes_aliasDomain_constraint'),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes DROP COLUMN aliasDomain'),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes CHANGE aliasName aliasTarget VARCHAR(128)')
+    ], callback);
+};

migrations/20200427044800-apps-add-servicesConfigJson.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN servicesConfigJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN servicesConfigJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200430035310-apps-add-bindsJson.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN bindsJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN bindsJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200512172301-settings-backup-encryption.js

@@ -0,0 +1,35 @@
+'use strict';
+
+const backups = require('../src/backups.js'),
+    fs = require('fs');
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        if (backupConfig.key) {
+            backupConfig.encryption = backups.generateEncryptionKeysSync(backupConfig.key);
+            backups.cleanupCacheFilesSync();
+
+            fs.writeFileSync('/home/yellowtent/platformdata/BACKUP_PASSWORD',
+                'This file contains your Cloudron backup password.\nBefore Cloudron v5.2, this was saved in the database.' +
+                'From Cloudron 5.2, this password is not required anymore. We generate strong keys based off this password and use those keys to encrypt the backups.\n' +
+                'This means that the password is only required at decryption/restore time.\n\n' +
+                'This file can be safely removed and only exists for the off-chance that you do not remember your backup password.\n\n' +
+                `Password: ${backupConfig.key}\n`,
+                'utf8');
+
+        } else {
+            backupConfig.encryption = null;
+        }
+
+        delete backupConfig.key;
+
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200514043113-backups-rename-version-to-packageVersion.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE backups CHANGE version packageVersion VARCHAR(128) NOT NULL', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE backups CHANGE packageVersion version VARCHAR(128) NOT NULL', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200514045746-backups-add-encryptionVersion.js

@@ -0,0 +1,24 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE backups ADD COLUMN encryptionVersion INTEGER', function (error) {
+        if (error) return callback(error);
+
+        db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+            if (error || results.length === 0) return callback(error);
+
+            var backupConfig = JSON.parse(results[0].value);
+            if (!backupConfig.encryption) return callback(null);
+
+            // mark old encrypted backups as v1
+            db.runSql('UPDATE backups SET encryptionVersion=1', callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE backups DROP COLUMN encryptionVersion', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200514231149-backups-retentionSecs-to-retentionPolicy.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        backupConfig.retentionPolicy = { keepWithinSecs: backupConfig.retentionSecs };
+        delete backupConfig.retentionSecs;
+
+        // mark old encrypted backups as v1
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200528003052-settings-backup-s3-path-style.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        if (backupConfig.provider !== 'minio' && backupConfig.provider !== 's3-v4-compat') return callback();
+
+        backupConfig.s3ForcePathStyle = true; // usually minio is self-hosted. s3 v4 compat, we don't know
+
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200530194049-appPasswords-change-unique-constraint.js

@@ -0,0 +1,17 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+
+    // http://stackoverflow.com/questions/386294/what-is-the-maximum-length-of-a-valid-email-address
+
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE appPasswords DROP INDEX name'),
+        db.runSql.bind(db, 'ALTER TABLE appPasswords ADD CONSTRAINT appPasswords_name_userId_identifier UNIQUE (name, userId, identifier)'),
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200604111448-userGroups-add-source.js

@@ -0,0 +1,17 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE userGroups ADD COLUMN source VARCHAR(128) DEFAULT ""', function (error) {
+        if (error) return callback(error);
+
+        callback();
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE userGroups DROP COLUMN source', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20200614181936-backups-add-identifier.js

@@ -0,0 +1,38 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE backups ADD COLUMN identifier VARCHAR(128)', function (error) {
+        if (error) return callback(error);
+
+
+        db.all('SELECT * FROM backups', function (error, backups) {
+            if (error) return callback(error);
+
+            async.eachSeries(backups, function (backup, next) {
+                let identifier = 'unknown';
+
+                if (backup.type === 'box') {
+                    identifier = 'box';
+                } else {
+                    const match = backup.id.match(/app_(.+?)_.+/);
+                    if (match) identifier = match[1];
+                }
+
+                db.runSql('UPDATE backups SET identifier=? WHERE id=?', [ identifier, backup.id ], next);
+            }, function (error) {
+                if (error) return callback(error);
+
+                db.runSql('ALTER TABLE backups MODIFY COLUMN identifier VARCHAR(128) NOT NULL', callback);
+            });
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE backups DROP COLUMN identifier', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200709224148-users-rename-modifiedAt-to-ts.js

@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE users ADD COLUMN ts TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP', function (error) {
+        if (error) console.error(error);
+
+        db.runSql('ALTER TABLE users DROP COLUMN modifiedAt', callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE users DROP COLUMN ts', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200728232132-settings-set-backups-schedulePattern.js

@@ -0,0 +1,29 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        if (backupConfig.intervalSecs === 6 * 60 * 60) { // every 6 hours
+            backupConfig.schedulePattern = '00 00 5,11,17,23 * * *';
+        } else if (backupConfig.intervalSecs === 12 * 60 * 60) { // every 12 hours
+            backupConfig.schedulePattern = '00 00 5,17 * * *';
+        } else if (backupConfig.intervalSecs === 24 * 60 * 60) { // every day
+            backupConfig.schedulePattern = '00 00 23 * * *';
+        } else if (backupConfig.intervalSecs === 3 * 24 * 60 * 60) { // every 3 days (based on day)
+            backupConfig.schedulePattern = '00 00 23 * * 1,3,5';
+        } else if (backupConfig.intervalSecs === 7 * 24 * 60 * 60) { // every week (saturday)
+            backupConfig.schedulePattern = '00 00 23 * * 6';
+        } else { // default to everyday
+            backupConfig.schedulePattern = '00 00 23 * * *';
+        }
+
+        delete backupConfig.intervalSecs;
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200816002703-settings-add-mail-domain-and-fqdn.js

@@ -0,0 +1,23 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="admin_domain"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        const adminDomain = results[0].value;
+
+        async.series([
+            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_domain', adminDomain ]),
+            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_fqdn', `my.${adminDomain}` ])
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_domain"'),
+        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_fqdn"'),
+    ], callback);
+};

migrations/20200820050259-settings-merge-autoupdate-pattern.js

@@ -0,0 +1,22 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    db.runSql('SELECT * FROM settings WHERE name=?', ['app_autoupdate_pattern'], function (error, results) {
+        if (error || results.length === 0) return callback(error); // will use defaults from box code
+
+        var updatePattern = results[0].value; // use app auto update patter for the box as well
+
+        async.series([
+            db.runSql.bind(db, 'START TRANSACTION;'),
+            db.runSql.bind(db, 'DELETE FROM settings WHERE name=? OR name=?', ['app_autoupdate_pattern', 'box_autoupdate_pattern']),
+            db.runSql.bind(db, 'INSERT settings (name, value) VALUES(?, ?)', ['autoupdate_pattern', updatePattern]),
+            db.runSql.bind(db, 'COMMIT')
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200824004859-mail-add-bannerJson.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mail ADD COLUMN bannerJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mail DROP COLUMN bannerJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200914172345-split-firewall-config.js

@@ -0,0 +1,27 @@
+'use strict';
+
+const OLD_FIREWALL_CONFIG_JSON = '/home/yellowtent/boxdata/firewall-config.json';
+const PORTS_FILE = '/home/yellowtent/boxdata/firewall/ports.json';
+const BLOCKLIST_FILE = '/home/yellowtent/boxdata/firewall/blocklist.txt';
+
+const fs = require('fs');
+
+exports.up = function (db, callback) {
+    if (!fs.existsSync(OLD_FIREWALL_CONFIG_JSON)) return callback();
+
+    try {
+        const dataJson = fs.readFileSync(OLD_FIREWALL_CONFIG_JSON, 'utf8');
+        const data = JSON.parse(dataJson);
+        fs.writeFileSync(BLOCKLIST_FILE, data.blocklist.join('\n') + '\n', 'utf8');
+        fs.writeFileSync(PORTS_FILE, JSON.stringify({ allowed_tcp_ports: data.allowed_tcp_ports }, null, 4), 'utf8');
+        fs.unlinkSync(OLD_FIREWALL_CONFIG_JSON);
+    } catch (error) {
+        console.log('Error migrating old firewall config', error);
+    }
+
+    callback();
+};
+
+exports.down = function (db, callback) {
+    callback();
+};

migrations/20201028052525-volumes-create-table.js

@@ -0,0 +1,40 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    var cmd1 = 'CREATE TABLE volumes(' +
+                'id VARCHAR(128) NOT NULL UNIQUE,' +
+                'name VARCHAR(256) NOT NULL UNIQUE,' +
+                'hostPath VARCHAR(1024) NOT NULL UNIQUE,' +
+                'creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,' +
+                'PRIMARY KEY (id)) CHARACTER SET utf8 COLLATE utf8_bin';
+
+    var cmd2 = 'CREATE TABLE appMounts(' +
+                 'appId VARCHAR(128) NOT NULL,' +
+                 'volumeId VARCHAR(128) NOT NULL,' +
+                 'readOnly BOOLEAN DEFAULT 1,' +
+                 'UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),' +
+                 'FOREIGN KEY(appId) REFERENCES apps(id),' +
+                 'FOREIGN KEY(volumeId) REFERENCES volumes(id)) CHARACTER SET utf8 COLLATE utf8_bin;';
+
+    db.runSql(cmd1, function (error) {
+        if (error) console.error(error);
+
+        db.runSql(cmd2, function (error) {
+            if (error) console.error(error);
+
+            db.runSql('ALTER TABLE apps DROP COLUMN bindsJson', callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('DROP TABLE appMounts', function (error) {
+        if (error) console.error(error);
+
+        db.runSql('DROP TABLE volumes', function (error) {
+            if (error) console.error(error);
+            callback(error);
+        });
+    });
+};
+

migrations/schema.sql

@@ -21,7 +21,7 @@ CREATE TABLE IF NOT EXISTS users(
     password VARCHAR(1024) NOT NULL,
     salt VARCHAR(512) NOT NULL,
     createdAt VARCHAR(512) NOT NULL,
-    modifiedAt VARCHAR(512) NOT NULL,
+    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
     displayName VARCHAR(512) DEFAULT "",
     fallbackEmail VARCHAR(512) DEFAULT "",
     twoFactorAuthenticationSecret VARCHAR(128) DEFAULT "",
@@ -37,6 +37,7 @@ CREATE TABLE IF NOT EXISTS users(
 CREATE TABLE IF NOT EXISTS userGroups(
     id VARCHAR(128) NOT NULL UNIQUE,
     name VARCHAR(254) NOT NULL UNIQUE,
+    source VARCHAR(128) DEFAULT "",
     PRIMARY KEY(id));
 
 CREATE TABLE IF NOT EXISTS groupMembers(
@@ -65,8 +66,6 @@ CREATE TABLE IF NOT EXISTS apps(
     containerId VARCHAR(128),
     manifestJson TEXT,
     httpPort INTEGER,                        // this is the nginx proxy port and not manifest.httpPort
-    location VARCHAR(128) NOT NULL,
-    domain VARCHAR(128) NOT NULL,
     accessRestrictionJson TEXT, // { users: [ ], groups: [ ] }
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app was installed
     updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,   // when the last app update was done
@@ -86,6 +85,7 @@ CREATE TABLE IF NOT EXISTS apps(
     dataDir VARCHAR(256) UNIQUE,
     taskId INTEGER, // current task
     errorJson TEXT,
+    servicesConfigJson TEXT, // app services configuration
 
     FOREIGN KEY(mailboxDomain) REFERENCES domains(domain),
     FOREIGN KEY(taskId) REFERENCES tasks(id),
@@ -120,8 +120,10 @@ CREATE TABLE IF NOT EXISTS appEnvVars(
 CREATE TABLE IF NOT EXISTS backups(
     id VARCHAR(128) NOT NULL,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    version VARCHAR(128) NOT NULL, /* app version or box version */
+    packageVersion VARCHAR(128) NOT NULL, /* app version or box version */
+    encryptionVersion INTEGER, /* when null, unencrypted backup */
     type VARCHAR(16) NOT NULL, /* 'box' or 'app' */
+    identifier VARCHAR(128) NOT NULL, /* 'box' or the app id */
     dependsOn TEXT, /* comma separate list of objects this backup depends on */
     state VARCHAR(16) NOT NULL,
     manifestJson TEXT, /* to validate if the app can be installed in this version of box */
@@ -158,6 +160,7 @@ CREATE TABLE IF NOT EXISTS mail(
     mailFromValidation BOOLEAN DEFAULT 1,
     catchAllJson TEXT,
     relayJson TEXT,
+    bannerJson TEXT,
 
     dkimSelector VARCHAR(128) NOT NULL DEFAULT "cloudron",
 
@@ -177,12 +180,15 @@ CREATE TABLE IF NOT EXISTS mailboxes(
     name VARCHAR(128) NOT NULL,
     type VARCHAR(16) NOT NULL, /* 'mailbox', 'alias', 'list' */
     ownerId VARCHAR(128) NOT NULL, /* user id */
-    aliasTarget VARCHAR(128), /* the target name type is an alias */
+    aliasName VARCHAR(128), /* the target name type is an alias */
+    aliasDomain VARCHAR(128), /* the target domain */
     membersJson TEXT, /* members of a group. fully qualified */
+    membersOnly BOOLEAN DEFAULT false,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     domain VARCHAR(128),
 
     FOREIGN KEY(domain) REFERENCES mail(domain),
+    FOREIGN KEY(aliasDomain) REFERENCES mail(domain),
     UNIQUE (name, domain));
 
 CREATE TABLE IF NOT EXISTS subdomains(
@@ -214,7 +220,7 @@ CREATE TABLE IF NOT EXISTS notifications(
     message TEXT,
     acknowledged BOOLEAN DEFAULT false,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-
+    UNIQUE KEY appPasswords_name_appId_identifier (name, userId, identifier),
     PRIMARY KEY (id)
 );
 
@@ -226,8 +232,24 @@ CREATE TABLE IF NOT EXISTS appPasswords(
     hashedPassword VARCHAR(1024) NOT NULL,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     FOREIGN KEY(userId) REFERENCES users(id),
-    UNIQUE (name, userId),
+
     PRIMARY KEY (id)
 );
 
+CREATE TABLE IF NOT EXISTS volumes(
+    id VARCHAR(128) NOT NULL UNIQUE,
+    name VARCHAR(256) NOT NULL UNIQUE,
+    hostPath VARCHAR(1024) NOT NULL UNIQUE,
+    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (id)
+);
+
+CREATE TABLE IF NOT EXISTS appMounts(
+    appId VARCHAR(128) NOT NULL,
+    volumeId VARCHAR(128) NOT NULL,
+    readOnly BOOLEAN DEFAULT 1,
+    UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),
+    FOREIGN KEY(appId) REFERENCES apps(id),
+    FOREIGN KEY(volumeId) REFERENCES volumes(id));
+
 CHARACTER SET utf8 COLLATE utf8_bin;

package.json

@@ -14,71 +14,72 @@
     "node": ">=4.0.0 <=4.1.1"
   },
   "dependencies": {
-    "@google-cloud/dns": "^1.1.0",
+    "@google-cloud/dns": "^1.2.9",
     "@google-cloud/storage": "^2.5.0",
     "@sindresorhus/df": "git+https://github.com/cloudron-io/df.git#type",
     "async": "^2.6.3",
-    "aws-sdk": "^2.610.0",
+    "aws-sdk": "^2.759.0",
     "body-parser": "^1.19.0",
-    "cloudron-manifestformat": "^5.1.1",
+    "cloudron-manifestformat": "^5.6.0",
     "connect": "^3.7.0",
-    "connect-lastmile": "^1.2.2",
+    "connect-lastmile": "^2.0.0",
     "connect-timeout": "^1.9.0",
     "cookie-session": "^1.4.0",
     "cron": "^1.8.2",
-    "db-migrate": "^0.11.6",
-    "db-migrate-mysql": "^1.1.10",
-    "debug": "^4.1.1",
+    "db-migrate": "^0.11.11",
+    "db-migrate-mysql": "^2.1.1",
+    "debug": "^4.2.0",
     "dockerode": "^2.5.8",
     "ejs": "^2.6.1",
-    "ejs-cli": "^2.1.1",
+    "ejs-cli": "^2.2.1",
     "express": "^4.17.1",
-    "js-yaml": "^3.13.1",
+    "ipaddr.js": "^2.0.0",
+    "js-yaml": "^3.14.0",
     "json": "^9.0.6",
-    "ldapjs": "^1.0.2",
-    "lodash": "^4.17.15",
+    "ldapjs": "^2.2.0",
+    "lodash": "^4.17.20",
     "lodash.chunk": "^4.2.0",
-    "mime": "^2.4.4",
-    "moment-timezone": "^0.5.27",
-    "morgan": "^1.9.1",
-    "multiparty": "^4.2.1",
+    "mime": "^2.4.6",
+    "moment": "^2.29.0",
+    "moment-timezone": "^0.5.31",
+    "morgan": "^1.10.0",
+    "multiparty": "^4.2.2",
     "mysql": "^2.18.1",
-    "nodemailer": "^6.4.2",
+    "nodemailer": "^6.4.11",
     "nodemailer-smtp-transport": "^2.7.4",
     "once": "^1.4.0",
-    "parse-links": "^0.1.0",
-    "pretty-bytes": "^5.3.0",
+    "pretty-bytes": "^5.4.1",
     "progress-stream": "^2.0.0",
     "proxy-middleware": "^0.15.0",
     "qrcode": "^1.4.4",
-    "readdirp": "^3.3.0",
-    "request": "^2.88.0",
+    "readdirp": "^3.4.0",
+    "request": "^2.88.2",
     "rimraf": "^2.6.3",
     "s3-block-read-stream": "^0.5.0",
-    "safetydance": "^1.0.0",
+    "safetydance": "^1.1.1",
     "semver": "^6.1.1",
     "showdown": "^1.9.1",
     "speakeasy": "^2.0.0",
     "split": "^1.0.1",
-    "superagent": "^5.2.1",
+    "superagent": "^5.3.1",
     "supererror": "^0.7.2",
     "tar-fs": "github:cloudron-io/tar-fs#ignore_stat_error",
-    "tar-stream": "^2.1.0",
+    "tar-stream": "^2.1.4",
     "tldjs": "^2.3.1",
-    "underscore": "^1.9.2",
+    "underscore": "^1.11.0",
     "uuid": "^3.4.0",
     "validator": "^11.0.0",
-    "ws": "^7.2.1",
+    "ws": "^7.3.1",
     "xml2js": "^0.4.23"
   },
   "devDependencies": {
     "expect.js": "*",
-    "hock": "^1.3.3",
-    "js2xmlparser": "^4.0.0",
-    "mocha": "^6.1.4",
+    "hock": "^1.4.1",
+    "js2xmlparser": "^4.0.1",
+    "mocha": "^6.2.3",
     "mock-aws-s3": "git+https://github.com/cloudron-io/mock-aws-s3.git",
     "nock": "^10.0.6",
-    "node-sass": "^4.12.0",
+    "node-sass": "^4.14.1",
     "recursive-readdir": "^2.2.2"
   },
   "scripts": {

scripts/cloudron-setup

@@ -41,22 +41,21 @@ if systemctl -q is-active box; then
 fi
 
 initBaseImage="true"
-# provisioning data
-provider=""
+provider="generic"
 requestedVersion=""
 apiServerOrigin="https://api.cloudron.io"
 webServerOrigin="https://cloudron.io"
 sourceTarballUrl=""
 rebootServer="true"
-license=""
 
-args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot,license:" -n "$0" -- "$@")
+args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,source-tarball-url:,version:,env:,skip-reboot" -n "$0" -- "$@")
 eval set -- "${args}"
 
 while true; do
     case "$1" in
-    --help) echo "See https://cloudron.io/documentation/installation/ on how to install Cloudron"; exit 0;;
+    --help) echo "See https://docs.cloudron.io/installation/ on how to install Cloudron"; exit 0;;
     --provider) provider="$2"; shift 2;;
+    --source-tarball-url) sourceTarballUrl="$2"; shift 2;;
     --version) requestedVersion="$2"; shift 2;;
     --env)
         if [[ "$2" == "dev" ]]; then
@@ -67,7 +66,6 @@ while true; do
             webServerOrigin="https://staging.cloudron.io"
         fi
         shift 2;;
-    --license) license="$2"; shift 2;;
     --skip-baseimage-init) initBaseImage="false"; shift;;
     --skip-reboot) rebootServer="false"; shift;;
     --) break;;
@@ -83,56 +81,14 @@ fi
 
 # Only --help works with mismatched ubuntu
 ubuntu_version=$(lsb_release -rs)
-if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" ]]; then
-    echo "Cloudron requires Ubuntu 16.04 or 18.04" > /dev/stderr
+if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" && "${ubuntu_version}" != "20.04" ]]; then
+    echo "Cloudron requires Ubuntu 16.04, 18.04 or 20.04" > /dev/stderr
     exit 1
 fi
 
 # Can only write after we have confirmed script has root access
 echo "Running cloudron-setup with args : $@" > "${LOG_FILE}"
 
-# validate arguments in the absence of data
-readonly AVAILABLE_PROVIDERS="azure, caas, cloudscale, contabo, digitalocean, ec2, exoscale, gce, hetzner, interox, lightsail, linode, netcup, ovh, rosehosting, scaleway, skysilk, time4vps, upcloud, vultr or generic"
-if [[ -z "${provider}" ]]; then
-    echo "--provider is required ($AVAILABLE_PROVIDERS)"
-    exit 1
-elif [[  \
-            "${provider}" != "ami" && \
-            "${provider}" != "azure" && \
-            "${provider}" != "azure-image" && \
-            "${provider}" != "caas" && \
-            "${provider}" != "cloudscale" && \
-            "${provider}" != "contabo" && \
-            "${provider}" != "digitalocean" && \
-            "${provider}" != "digitalocean-mp" && \
-            "${provider}" != "ec2" && \
-            "${provider}" != "exoscale" && \
-            "${provider}" != "gce" && \
-            "${provider}" != "hetzner" && \
-            "${provider}" != "interox" && \
-            "${provider}" != "interox-image" && \
-            "${provider}" != "lightsail" && \
-            "${provider}" != "linode" && \
-            "${provider}" != "linode-oneclick" && \
-            "${provider}" != "linode-stackscript" && \
-            "${provider}" != "netcup" && \
-            "${provider}" != "netcup-image" && \
-            "${provider}" != "ovh" && \
-            "${provider}" != "rosehosting" && \
-            "${provider}" != "scaleway" && \
-            "${provider}" != "skysilk" && \
-            "${provider}" != "skysilk-image" && \
-            "${provider}" != "time4vps" && \
-            "${provider}" != "time4vps-image" && \
-            "${provider}" != "upcloud" && \
-            "${provider}" != "upcloud-image" && \
-            "${provider}" != "vultr" && \
-            "${provider}" != "generic" \
-        ]]; then
-    echo "--provider must be one of: $AVAILABLE_PROVIDERS"
-    exit 1
-fi
-
 echo ""
 echo "##############################################"
 echo "         Cloudron Setup (${requestedVersion:-latest})"
@@ -151,12 +107,6 @@ if [[ "${initBaseImage}" == "true" ]]; then
         exit 1
     fi
 
-    echo "=> Ensure required apt sources"
-    if ! add-apt-repository universe &>> "${LOG_FILE}"; then
-        echo "Could not add required apt sources (for nginx-full). See ${LOG_FILE}"
-        exit 1
-    fi
-
     echo "=> Updating apt and installing script dependencies"
     if ! apt-get update &>> "${LOG_FILE}"; then
         echo "Could not update package repositories. See ${LOG_FILE}"
@@ -169,21 +119,25 @@ if [[ "${initBaseImage}" == "true" ]]; then
     fi
 fi
 
-echo "=> Checking version"
-if ! releaseJson=$($curl -s "${apiServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
-    echo "Failed to get release information"
-    exit 1
-fi
+if [[ "$sourceTarballUrl" == "" ]]; then
+    echo "=> Checking version"
+    if ! releaseJson=$($curl -s "${apiServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
+        echo "Failed to get release information"
+        exit 1
+    fi
 
-if [[ "$requestedVersion" == "" ]]; then
-    version=$(echo "${releaseJson}" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["version"])')
+    if [[ "$requestedVersion" == "" ]]; then
+        version=$(echo "${releaseJson}" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["version"])')
+    else
+        version="${requestedVersion}"
+    fi
+
+    if ! sourceTarballUrl=$(echo "${releaseJson}" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["info"]["sourceTarballUrl"])'); then
+        echo "No source code for version '${requestedVersion:-latest}'"
+        exit 1
+    fi
 else
-    version="${requestedVersion}"
-fi
-
-if ! sourceTarballUrl=$(echo "${releaseJson}" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["info"]["sourceTarballUrl"])'); then
-    echo "No source code for version '${requestedVersion:-latest}'"
-    exit 1
+    version=${requestedVersion}
 fi
 
 echo "=> Downloading version ${version} ..."
@@ -196,20 +150,19 @@ fi
 
 if [[ "${initBaseImage}" == "true" ]]; then
     echo -n "=> Installing base dependencies and downloading docker images (this takes some time) ..."
-    if ! /bin/bash "${box_src_tmp_dir}/baseimage/initializeBaseUbuntuImage.sh" "${provider}" "../src" &>> "${LOG_FILE}"; then
+    # initializeBaseUbuntuImage.sh args (provider, infraversion path) are only to support installation of pre 5.3 Cloudrons
+    if ! /bin/bash "${box_src_tmp_dir}/baseimage/initializeBaseUbuntuImage.sh" "generic" "../src" &>> "${LOG_FILE}"; then
         echo "Init script failed. See ${LOG_FILE} for details"
         exit 1
     fi
     echo ""
 fi
 
-# NOTE: this install script only supports 4.2 and above
+# The provider flag is still used for marketplace images
 echo "=> Installing version ${version} (this takes some time) ..."
 mkdir -p /etc/cloudron
 echo "${provider}" > /etc/cloudron/PROVIDER
 
-[[ -n "${license}" ]] && echo -n "$license" > /etc/cloudron/LICENSE
-
 if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" &>> "${LOG_FILE}"; then
     echo "Failed to install cloudron. See ${LOG_FILE} for details"
     exit 1
@@ -221,16 +174,16 @@ mysql -uroot -ppassword -e "REPLACE INTO box.settings (name, value) VALUES ('web
 echo -n "=> Waiting for cloudron to be ready (this takes some time) ..."
 while true; do
     echo -n "."
-    if status=$($curl -q -f "http://localhost:3000/api/v1/cloudron/status" 2>/dev/null); then
+    if status=$($curl -s -f "http://localhost:3000/api/v1/cloudron/status" 2>/dev/null); then
         break # we are up and running
     fi
     sleep 10
 done
 
-if ! ip=$(curl --fail --connect-timeout 2 --max-time 2 -q https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
+if ! ip=$(curl -s --fail --connect-timeout 2 --max-time 2 https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
     ip='<IP>'
 fi
-echo -e "\n\n${GREEN}Visit https://${ip} and accept the self-signed certificate to finish setup.${DONE}\n"
+echo -e "\n\n${GREEN}After reboot, visit https://${ip} and accept the self-signed certificate to finish setup.${DONE}\n"
 
 if [[ "${rebootServer}" == "true" ]]; then
     systemctl stop box mysql # sometimes mysql ends up having corrupt privilege tables

scripts/cloudron-support

@@ -37,12 +37,12 @@ while true; do
         # fall through
         ;&
     --owner-login)
-        admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' LIMIT 1" 2>/dev/null)
+        admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' AND username IS NOT NULL ORDER BY createdAt LIMIT 1" 2>/dev/null)
         admin_password=$(pwgen -1s 12)
         ghost_file=/home/yellowtent/platformdata/cloudron_ghost.json
         printf '{"%s":"%s"}\n' "${admin_username}" "${admin_password}" > "${ghost_file}"
         chown yellowtent:yellowtent "${ghost_file}" && chmod o-r,g-r "${ghost_file}"
-        echo "Login as ${admin_username} / ${admin_password} . Remove ${ghost_file} when done."
+        echo "Login as ${admin_username} / ${admin_password} . This password may only be used once. ${ghost_file} will be automatically removed after use."
         exit 0
         ;;
     --) break;;
@@ -57,7 +57,7 @@ if [[ "`df --output="avail" / | sed -n 2p`" -lt "10240" ]]; then
     echo ""
     df -h
     echo ""
-    echo "To recover from a full disk, follow the guide at https://cloudron.io/documentation/troubleshooting/#recovery-after-disk-full"
+    echo "To recover from a full disk, follow the guide at https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full"
     exit 1
 fi
 
@@ -94,7 +94,7 @@ echo -e $LINE"Backup stats (possibly misleading)"$LINE >> $OUT
 du -hcsL /var/backups/* &>> $OUT || true
 
 echo -e $LINE"System daemon status"$LINE >> $OUT
-systemctl status --lines=100 cloudron.target box mysql unbound cloudron-syslog nginx collectd docker &>> $OUT
+systemctl status --lines=100 box mysql unbound cloudron-syslog nginx collectd docker &>> $OUT
 
 echo -e $LINE"Box logs"$LINE >> $OUT
 tail -n 100 /home/yellowtent/platformdata/logs/box.log &>> $OUT
@@ -112,7 +112,7 @@ if [[ "${enableSSH}" == "true" ]]; then
     permit_root_login=$(grep -q ^PermitRootLogin.*yes /etc/ssh/sshd_config && echo "yes" || echo "no")
 
     # support.js uses similar logic
-    if $(grep -q "ec2\|lightsail\|ami" /etc/cloudron/PROVIDER); then
+    if [[ -d /home/ubuntu ]]; then
         ssh_user="ubuntu"
         keys_file="/home/ubuntu/.ssh/authorized_keys"
     else

scripts/installer.sh

@@ -11,9 +11,8 @@ if [[ ${EUID} -ne 0 ]]; then
     exit 1
 fi
 
-readonly USER=yellowtent
-readonly BOX_SRC_DIR=/home/${USER}/box
-readonly BASE_DATA_DIR=/home/${USER}
+readonly user=yellowtent
+readonly box_src_dir=/home/${user}/box
 
 readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400"
 readonly script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -24,13 +23,15 @@ readonly ubuntu_codename=$(lsb_release -cs)
 
 readonly is_update=$(systemctl is-active box && echo "yes" || echo "no")
 
+echo "==> installer: Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src_tmp_dir/VERSION) <=="
+
 echo "==> installer: updating docker"
 
-if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
+if [[ $(docker version --format {{.Client.Version}}) != "19.03.12" ]]; then
     # there are 3 packages for docker - containerd, CLI and the daemon
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.2-3_amd64.deb" -o /tmp/containerd.deb
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.13-2_amd64.deb" -o /tmp/containerd.deb
+    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
+    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
 
     echo "==> installer: Waiting for all dpkg tasks to finish..."
     while fuser /var/lib/dpkg/lock; do
@@ -56,15 +57,20 @@ if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
     rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
 fi
 
-readonly nginx_version=$(nginx -v)
-if [[ "${nginx_version}" != *"1.14."* && "${ubuntu_version}" == "16.04" ]]; then
-    echo "==> installer: installing nginx for xenial for TLSv3 support"
-    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.14.0-1~xenial_amd64.deb -o /tmp/nginx.deb
+readonly nginx_version=$(nginx -v 2>&1)
+if [[ "${nginx_version}" != *"1.18."* ]]; then
+    echo "==> installer: installing nginx 1.18"
+    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
     # apt install with install deps (as opposed to dpkg -i)
     apt install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes /tmp/nginx.deb
     rm /tmp/nginx.deb
 fi
 
+if ! which ipset; then
+    echo "==> installer: installing ipset"
+    apt install -y ipset
+fi
+
 echo "==> installer: updating node"
 if [[ "$(node --version)" != "v10.18.1" ]]; then
     mkdir -p /usr/local/node-10.18.1
@@ -118,22 +124,22 @@ while [[ ! -f "${CLOUDRON_SYSLOG}" || "$(${CLOUDRON_SYSLOG} --version)" != ${CLO
     sleep 5
 done
 
-if ! id "${USER}" 2>/dev/null; then
-    useradd "${USER}" -m
+if ! id "${user}" 2>/dev/null; then
+    useradd "${user}" -m
 fi
 
 if [[ "${is_update}" == "yes" ]]; then
-    echo "==> installer: stop cloudron.target service for update"
-    ${BOX_SRC_DIR}/setup/stop.sh
+    echo "==> installer: stop box service for update"
+    ${box_src_dir}/setup/stop.sh
 fi
 
 # ensure we are not inside the source directory, which we will remove now
 cd /root
 
 echo "==> installer: switching the box code"
-rm -rf "${BOX_SRC_DIR}"
-mv "${box_src_tmp_dir}" "${BOX_SRC_DIR}"
-chown -R "${USER}:${USER}" "${BOX_SRC_DIR}"
+rm -rf "${box_src_dir}"
+mv "${box_src_tmp_dir}" "${box_src_dir}"
+chown -R "${user}:${user}" "${box_src_dir}"
 
 echo "==> installer: calling box setup script"
-"${BOX_SRC_DIR}/setup/start.sh"
+"${box_src_dir}/setup/start.sh"

setup/start.sh

@@ -20,6 +20,11 @@ readonly ubuntu_version=$(lsb_release -rs)
 
 cp -f "${script_dir}/../scripts/cloudron-support" /usr/bin/cloudron-support
 
+# this needs to match the cloudron/base:2.0.0 gid
+if ! getent group media; then
+    addgroup --gid 500 --system media
+fi
+
 echo "==> Configuring docker"
 cp "${script_dir}/start/docker-cloudron-app.apparmor" /etc/apparmor.d/docker-cloudron-app
 systemctl enable apparmor
@@ -39,7 +44,7 @@ mkdir -p "${PLATFORM_DATA_DIR}/mysql"
 mkdir -p "${PLATFORM_DATA_DIR}/postgresql"
 mkdir -p "${PLATFORM_DATA_DIR}/mongodb"
 mkdir -p "${PLATFORM_DATA_DIR}/redis"
-mkdir -p "${PLATFORM_DATA_DIR}/addons/mail"
+mkdir -p "${PLATFORM_DATA_DIR}/addons/mail/banner"
 mkdir -p "${PLATFORM_DATA_DIR}/collectd/collectd.conf.d"
 mkdir -p "${PLATFORM_DATA_DIR}/logrotate.d"
 mkdir -p "${PLATFORM_DATA_DIR}/acme"
@@ -52,10 +57,12 @@ mkdir -p "${PLATFORM_DATA_DIR}/logs/backup" \
 mkdir -p "${PLATFORM_DATA_DIR}/update"
 
 mkdir -p "${BOX_DATA_DIR}/appicons"
+mkdir -p "${BOX_DATA_DIR}/firewall"
 mkdir -p "${BOX_DATA_DIR}/profileicons"
 mkdir -p "${BOX_DATA_DIR}/certs"
 mkdir -p "${BOX_DATA_DIR}/acme" # acme keys
 mkdir -p "${BOX_DATA_DIR}/mail/dkim"
+mkdir -p "${BOX_DATA_DIR}/well-known" # .well-known documents
 
 # ensure backups folder exists and is writeable
 mkdir -p /var/backups
@@ -79,6 +86,9 @@ systemctl daemon-reload
 systemctl restart systemd-journald
 setfacl -n -m u:${USER}:r /var/log/journal/*/system.journal
 
+# Give user access to nginx logs (uses adm group)
+usermod -a -G adm ${USER}
+
 echo "==> Setting up unbound"
 # DO uses Google nameservers by default. This causes RBL queries to fail (host 2.0.0.127.zen.spamhaus.org)
 # We do not use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!)
@@ -91,11 +101,13 @@ unbound-anchor -a /var/lib/unbound/root.key
 
 echo "==> Adding systemd services"
 cp -r "${script_dir}/start/systemd/." /etc/systemd/system/
+systemctl disable cloudron.target || true
+rm -f /etc/systemd/system/cloudron.target
 [[ "${ubuntu_version}" == "16.04" ]] && sed -e 's/MemoryMax/MemoryLimit/g' -i /etc/systemd/system/box.service
 systemctl daemon-reload
 systemctl enable unbound
 systemctl enable cloudron-syslog
-systemctl enable cloudron.target
+systemctl enable box
 systemctl enable cloudron-firewall
 
 # update firewall rules
@@ -118,6 +130,12 @@ echo "==> Configuring collectd"
 rm -rf /etc/collectd /var/log/collectd.log
 ln -sfF "${PLATFORM_DATA_DIR}/collectd" /etc/collectd
 cp "${script_dir}/start/collectd/collectd.conf" "${PLATFORM_DATA_DIR}/collectd/collectd.conf"
+if [[ "${ubuntu_version}" == "20.04" ]]; then
+    # https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1872281
+    if ! grep -q LD_PRELOAD /etc/default/collectd; then
+        echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
+    fi
+fi
 systemctl restart collectd
 
 echo "==> Configuring logrotate"
@@ -144,8 +162,15 @@ cp "${script_dir}/start/nginx/mime.types" "${PLATFORM_DATA_DIR}/nginx/mime.types
 if ! grep -q "^Restart=" /etc/systemd/system/multi-user.target.wants/nginx.service; then
     # default nginx service file does not restart on crash
     echo -e "\n[Service]\nRestart=always\n" >> /etc/systemd/system/multi-user.target.wants/nginx.service
-    systemctl daemon-reload
 fi
+
+# worker_rlimit_nofile in nginx config can be max this number
+mkdir -p /etc/systemd/system/nginx.service.d
+if ! grep -q "^LimitNOFILE=" /etc/systemd/system/nginx.service.d/cloudron.conf; then
+    echo -e "[Service]\nLimitNOFILE=16384\n" > /etc/systemd/system/nginx.service.d/cloudron.conf
+fi
+
+systemctl daemon-reload
 systemctl start nginx
 
 # restart mysql to make sure it has latest config
@@ -168,11 +193,17 @@ fi
 
 readonly mysql_root_password="password"
 mysqladmin -u root -ppassword password password # reset default root password
+if [[ "${ubuntu_version}" == "20.04" ]]; then
+    # mysql 8 added a new caching_sha2_password scheme which mysqljs does not support
+    mysql -u root -p${mysql_root_password} -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '${mysql_root_password}';"
+fi
 mysql -u root -p${mysql_root_password} -e 'CREATE DATABASE IF NOT EXISTS box'
 
+# set HOME explicity, because it's not set when the installer calls it. this is done because
+# paths.js uses this env var and some of the migrate code requires box code
 echo "==> Migrating data"
 cd "${BOX_SRC_DIR}"
-if ! BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up; then
+if ! HOME=${HOME_DIR} BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up; then
     echo "DB migration failed"
     exit 1
 fi
@@ -191,6 +222,9 @@ fi
 echo "==> Cleaning up stale redis directories"
 find "${APPS_DATA_DIR}" -maxdepth 2 -type d -name redis -exec rm -rf {} +
 
+echo "==> Cleaning up old logs"
+rm -f /home/yellowtent/platformdata/logs/*/*.log.* || true
+
 echo "==> Changing ownership"
 # be careful of what is chown'ed here. subdirs like mysql,redis etc are owned by the containers and will stop working if perms change
 chown -R "${USER}" /etc/cloudron
@@ -206,7 +240,7 @@ chown "${USER}:${USER}" "${BOX_DATA_DIR}/mail"
 chown "${USER}:${USER}" -R "${BOX_DATA_DIR}/mail/dkim" # this is owned by box currently since it generates the keys
 
 echo "==> Starting Cloudron"
-systemctl start cloudron.target
+systemctl start box
 
 sleep 2 # give systemd sometime to start the processes
 

setup/start/cloudron-firewall.sh

@@ -6,11 +6,25 @@ echo "==> Setting up firewall"
 iptables -t filter -N CLOUDRON || true
 iptables -t filter -F CLOUDRON # empty any existing rules
 
-# NOTE: keep these in sync with src/apps.js validatePortBindings
-# allow ssh, http, https, ping, dns
-iptables -t filter -I CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
-# ssh is allowed alternately on port 202
-iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443,587,993,4190 -j ACCEPT
+# first setup any user IP block lists
+ipset create cloudron_blocklist hash:net || true
+/home/yellowtent/box/src/scripts/setblocklist.sh
+
+iptables -t filter -A CLOUDRON -m set --match-set cloudron_blocklist src -j DROP
+# the DOCKER-USER chain is not cleared on docker restart
+if ! iptables -t filter -C DOCKER-USER -m set --match-set cloudron_blocklist src -j DROP; then
+    iptables -t filter -I DOCKER-USER 1 -m set --match-set cloudron_blocklist src -j DROP
+fi
+
+# allow related and establisted connections
+iptables -t filter -A CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
+iptables -t filter -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443 -j ACCEPT # 202 is the alternate ssh port
+
+# whitelist any user ports
+ports_json="/home/yellowtent/boxdata/firewall/ports.json"
+if allowed_tcp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_tcp_ports.join(','))" 2>/dev/null); then
+    [[ -n "${allowed_tcp_ports}" ]] && iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports "${allowed_tcp_ports}" -j ACCEPT
+fi
 
 # turn and stun service
 iptables -t filter -A CLOUDRON -p tcp -m multiport --dports 3478,5349 -j ACCEPT
@@ -52,8 +66,6 @@ for port in 22 202; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --dport ${port} -m state --state NEW -m recent --update --name "public-${port}" --seconds 10 --hitcount 5 -j CLOUDRON_RATELIMIT_LOG
 done
 
-# TODO: move docker platform rules to platform.js so it can be specialized to rate limit only when destination is the mail container
-
 # docker translates (dnat) 25, 587, 993, 4190 in the PREROUTING step
 for port in 2525 4190 9993; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --syn ! -s 172.18.0.0/16 -d 172.18.0.0/16 --dport ${port} -m connlimit --connlimit-above 50 -j CLOUDRON_RATELIMIT_LOG
@@ -69,12 +81,10 @@ for port in 3306 5432 6379 27017; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --syn -s 172.18.0.0/16 -d 172.18.0.0/16 --dport ${port} -m connlimit --connlimit-above 5000 -j CLOUDRON_RATELIMIT_LOG
 done
 
-# For ssh, http, https
 if ! iptables -t filter -C INPUT -j CLOUDRON_RATELIMIT 2>/dev/null; then
     iptables -t filter -I INPUT 1 -j CLOUDRON_RATELIMIT
 fi
 
-# For smtp, imap etc routed via docker/nat
-# Workaroud issue where Docker insists on adding itself first in FORWARD table
+# Workaround issue where Docker insists on adding itself first in FORWARD table
 iptables -D FORWARD -j CLOUDRON_RATELIMIT || true
 iptables -I FORWARD 1 -j CLOUDRON_RATELIMIT

setup/start/cloudron-motd

@@ -14,8 +14,8 @@ if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
     printf "\t\t\t-------------------\n"
 
     printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://${ip} on your browser and accept the self-signed certificate to finish setup."
-    printf "Cloudron overview - https://cloudron.io/documentation/ \n"
-    printf "Cloudron setup - https://cloudron.io/documentation/installation/#setup \n"
+    printf "Cloudron overview - https://docs.cloudron.io/ \n"
+    printf "Cloudron setup - https://docs.cloudron.io/installation/#setup \n"
 else
     printf "\t\t\tNOTE TO CLOUDRON ADMINS\n"
     printf "\t\t\t-----------------------\n"
@@ -23,7 +23,7 @@ else
     printf "Cloudron relies on and may break your installation. Ubuntu security updates\n"
     printf "are automatically installed on this server every night.\n"
     printf "\n"
-    printf "Read more at https://cloudron.io/documentation/security/#os-updates\n"
+    printf "Read more at https://docs.cloudron.io/security/#os-updates\n"
 fi
 
 printf "\nFor help and more information, visit https://forum.cloudron.io\n\n"

setup/start/cloudron-resize-fs.sh

@@ -4,6 +4,11 @@ set -eu -o pipefail
 
 readonly APPS_SWAP_FILE="/apps.swap"
 
+if [[ -f "${APPS_SWAP_FILE}" ]]; then
+    echo "Swap file already exists at /apps.swap . Skipping"
+    exit
+fi
+
 # all sizes are in mb
 readonly physical_memory=$(LC_ALL=C free -m | awk '/Mem:/ { print $2 }')
 readonly swap_size=$((${physical_memory} > 4096 ? 4096 : ${physical_memory})) # min(RAM, 4GB) if you change this, fix enoughResourcesAvailable() in client.js

setup/start/collectd/collectd.conf

@@ -121,7 +121,7 @@ LoadPlugin memory
 #LoadPlugin netlink
 #LoadPlugin network
 #LoadPlugin nfs
-LoadPlugin nginx
+#LoadPlugin nginx
 #LoadPlugin notify_desktop
 #LoadPlugin notify_email
 #LoadPlugin ntpd
@@ -149,7 +149,7 @@ LoadPlugin nginx
 #LoadPlugin statsd
 LoadPlugin swap
 #LoadPlugin table
-LoadPlugin tail
+#LoadPlugin tail
 #LoadPlugin tail_csv
 #LoadPlugin tcpconns
 #LoadPlugin teamspeak2
@@ -197,42 +197,11 @@ LoadPlugin write_graphite
    IgnoreSelected false
 </Plugin>
 
-<Plugin nginx>
-   URL "http://127.0.0.1/nginx_status"
-</Plugin>
-
 <Plugin swap>
    ReportByDevice false
    ReportBytes true
 </Plugin>
 
-<Plugin "tail">
-   <File "/var/log/nginx/error.log">
-     Instance "nginx"
-     <Match>
-       Regex ".*"
-       DSType "CounterInc"
-       Type counter
-       Instance "errors"
-     </Match>
-   </File>
-   <File "/var/log/nginx/access.log">
-     Instance "nginx"
-     <Match>
-       Regex ".*"
-       DSType "CounterInc"
-       Type counter
-       Instance "requests"
-     </Match>
-    <Match>
-      Regex " \".*\" [0-9]+ [0-9]+ ([0-9]+)"
-      DSType GaugeAverage
-      Type delay
-      Instance "response"
-    </Match>
-   </File>
-</Plugin>
-
 <Plugin python>
     # https://blog.dbrgn.ch/2017/3/10/write-a-collectd-python-plugin/
     ModulePath "/home/yellowtent/box/setup/start/collectd/"

setup/start/collectd/du.py

@@ -3,6 +3,7 @@ import collectd,os,subprocess,sys,re,time
 # https://www.programcreek.com/python/example/106897/collectd.register_read
 
 PATHS = [] # { name, dir, exclude }
+# there is a pattern in carbon/storage-schemas.conf which stores values every 12h for a year
 INTERVAL = 60 * 60 * 12 # twice a day. change values in docker-graphite if you change this
 
 def du(pathinfo):

setup/start/logrotate/platform

@@ -10,6 +10,7 @@
 /home/yellowtent/platformdata/logs/redis-*/*.log
 /home/yellowtent/platformdata/logs/crash/*.log
 /home/yellowtent/platformdata/logs/collectd/*.log
+/home/yellowtent/platformdata/logs/turn/*.log
 /home/yellowtent/platformdata/logs/updater/*.log {
     # only keep one rotated file, we currently do not send that over the api
     rotate 1
@@ -17,6 +18,7 @@
     missingok
     # we never compress so we can simply tail the files
     nocompress
+    # this truncates the original log file and not the rotated one
     copytruncate
 }
 

setup/start/nginx/nginx.conf

@@ -1,11 +1,18 @@
 user www-data;
 
-worker_processes  1;
+# detect based on available CPU cores
+worker_processes  auto;
+
+# this is 4096 by default. See /proc/<PID>/limits and /etc/security/limits.conf
+# usually twice the worker_connections (one for uptsream, one for downstream)
+# see also LimitNOFILE=16384 in systemd drop-in
+worker_rlimit_nofile 8192;
 
 pid /run/nginx.pid;
 
 events {
-    worker_connections  1024;
+    # a single worker has these many simultaneous connections max
+    worker_connections  4096;
 }
 
 http {
@@ -36,23 +43,5 @@ http {
     # zones for rate limiting
     limit_req_zone $binary_remote_addr zone=admin_login:10m rate=10r/s; # 10 request a second
 
-
-    # default http server that returns 404 for any domain we are not listening on
-    server {
-        listen      80 default_server;
-        listen      [::]:80 default_server;
-        server_name does_not_match_anything;
-
-        # acme challenges (for app installation and re-configure when the vhost config does not exist)
-        location /.well-known/acme-challenge/ {
-            default_type text/plain;
-            alias /home/yellowtent/platformdata/acme/;
-        }
-
-        location / {
-            return 404;
-        }
-    }
-
     include applications/*.conf;
 }

setup/start/sudoers

@@ -50,3 +50,15 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartdocker.s
 Defaults!/home/yellowtent/box/src/scripts/restartunbound.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartunbound.sh
 
+Defaults!/home/yellowtent/box/src/scripts/rmmailbox.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmmailbox.sh
+
+Defaults!/home/yellowtent/box/src/scripts/starttask.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/scripts/starttask.sh
+
+Defaults!/home/yellowtent/box/src/scripts/stoptask.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/stoptask.sh
+
+Defaults!/home/yellowtent/box/src/scripts/setblocklist.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/setblocklist.sh
+

setup/start/systemd/box.service

@@ -1,20 +1,21 @@
 [Unit]
 Description=Cloudron Admin
 OnFailure=crashnotifier@%n.service
-StopWhenUnneeded=true
 ; journald crashes result in a EPIPE in node. Cannot ignore it as it results in loss of logs.
 BindsTo=systemd-journald.service
 After=mysql.service nginx.service
 ; As cloudron-resize-fs is a one-shot, the Wants= automatically ensures that the service *finishes*
 Wants=cloudron-resize-fs.service
 
+[Install]
+WantedBy=multi-user.target
+
 [Service]
 Type=idle
 WorkingDirectory=/home/yellowtent/box
 Restart=always
-; Systemd does not append logs when logging to files, we spawn a shell first and exec to replace it after setting up the pipes
-ExecStart=/bin/sh -c 'echo "Logging to /home/yellowtent/platformdata/logs/box.log"; exec /usr/bin/node --max_old_space_size=150 /home/yellowtent/box/box.js >> /home/yellowtent/platformdata/logs/box.log 2>&1'
-Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box*,connect-lastmile" "BOX_ENV=cloudron" "NODE_ENV=production"
+ExecStart=/home/yellowtent/box/box.js
+Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box:*,connect-lastmile,-box:ldap" "BOX_ENV=cloudron" "NODE_ENV=production"
 ; kill apptask processes as well
 KillMode=control-group
 ; Do not kill this process on OOM. Children inherit this score. Do not set it to -1000 so that MemoryMax can keep working

setup/start/systemd/cloudron.target

@@ -1,10 +0,0 @@
-[Unit]
-Description=Cloudron Smartserver
-Documentation=https://cloudron.io/documentation.html
-StopWhenUnneeded=true
-Requires=box.service
-After=box.service
-# AllowIsolate=yes
-
-[Install]
-WantedBy=multi-user.target

setup/stop.sh

@@ -4,4 +4,4 @@ set -eu -o pipefail
 
 echo "Stopping cloudron"
 
-systemctl stop cloudron.target
+systemctl stop box

src/appdb.js

@@ -41,7 +41,7 @@ var assert = require('assert'),
 var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
     'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'subdomains.subdomain AS location', 'subdomains.domain',
     'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
-    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson',
+    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson',
     'apps.sso', 'apps.debugModeJson', 'apps.enableBackup',
     'apps.creationTime', 'apps.updateTime', 'apps.mailboxName', 'apps.mailboxDomain', 'apps.enableAutomaticUpdate',
     'apps.dataDir', 'apps.ts', 'apps.healthTime' ].join(',');
@@ -94,6 +94,10 @@ function postProcess(result) {
     result.debugMode = safe.JSON.parse(result.debugModeJson);
     delete result.debugModeJson;
 
+    assert(result.servicesConfigJson === null || typeof result.servicesConfigJson === 'string');
+    result.servicesConfig = safe.JSON.parse(result.servicesConfigJson) || {};
+    delete result.servicesConfigJson;
+
     result.alternateDomains = result.alternateDomains || [];
     result.alternateDomains.forEach(function (d) {
         delete d.appId;
@@ -108,6 +112,13 @@ function postProcess(result) {
         if (envNames[i]) result.env[envNames[i]] = envValues[i];
     }
 
+    let volumeIds = JSON.parse(result.volumeIds);
+    delete result.volumeIds;
+    let volumeReadOnlys = JSON.parse(result.volumeReadOnlys);
+    delete result.volumeReadOnlys;
+
+    result.mounts = volumeIds[0] === null ? [] : volumeIds.map((v, idx) => { return { volumeId: v, readOnly: !!volumeReadOnlys[idx] }; }); // NOTE: volumeIds is [null] when volumes of an app is empty
+
     result.error = safe.JSON.parse(result.errorJson);
     delete result.errorJson;
 
@@ -120,11 +131,13 @@ function get(id, callback) {
 
     database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
         + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes, '
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues,'
+        + 'JSON_ARRAYAGG(appMounts.volumeId) AS volumeIds, JSON_ARRAYAGG(appMounts.readOnly) AS volumeReadOnlys '
         + ' FROM apps'
         + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
         + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
         + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
+        + '  LEFT OUTER JOIN appMounts ON apps.id = appMounts.appId'
         + ' WHERE apps.id = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, id ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
@@ -147,11 +160,13 @@ function getByHttpPort(httpPort, callback) {
 
     database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
     + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-    + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+    + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues,'
+    + 'JSON_ARRAYAGG(appMounts.volumeId) AS volumeIds, JSON_ARRAYAGG(appMounts.readOnly) AS volumeReadOnlys '
         + ' FROM apps'
         + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
         + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
         + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
+        + '  LEFT OUTER JOIN appMounts ON apps.id = appMounts.appId'
         + ' WHERE httpPort = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, httpPort ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
@@ -173,11 +188,13 @@ function getByContainerId(containerId, callback) {
 
     database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
         + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues,'
+        + 'JSON_ARRAYAGG(appMounts.volumeId) AS volumeIds, JSON_ARRAYAGG(appMounts.readOnly) AS volumeReadOnlys '
         + ' FROM apps'
         + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
         + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
         + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
+        + '  LEFT OUTER JOIN appMounts ON apps.id = appMounts.appId'
         + ' WHERE containerId = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, containerId ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
@@ -198,11 +215,13 @@ function getAll(callback) {
 
     database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
         + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues,'
+        + 'JSON_ARRAYAGG(appMounts.volumeId) AS volumeIds, JSON_ARRAYAGG(appMounts.readOnly) AS volumeReadOnlys '
         + ' FROM apps'
         + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
         + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
         + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
+        + '  LEFT OUTER JOIN appMounts ON apps.id = appMounts.appId'
         + ' GROUP BY apps.id ORDER BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY ], function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -350,12 +369,13 @@ function del(id, callback) {
         { query: 'DELETE FROM appPortBindings WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appEnvVars WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appPasswords WHERE identifier = ?', args: [ id ] },
+        { query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM apps WHERE id = ?', args: [ id ] }
     ];
 
     database.transaction(queries, function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results[4].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (results[5].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
         callback(null);
     });
@@ -415,22 +435,29 @@ function updateWithConstraints(id, app, constraints, callback) {
     }
 
     if ('location' in app && 'domain' in app) { // must be updated together as they are unique together
-        queries.push({ query: 'UPDATE subdomains SET subdomain = ?, domain = ? WHERE appId = ? AND type = ?', args: [ app.location, app.domain, id, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ?', args: [ id ]}); // all locations of an app must be updated together
+        queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, app.domain, app.location, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+
+        if ('alternateDomains' in app) {
+            app.alternateDomains.forEach(function (d) {
+                queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+            });
+        }
     }
 
-    if ('alternateDomains' in app) {
-        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ? AND type = ?', args: [ id, exports.SUBDOMAIN_TYPE_REDIRECT ]});
-        app.alternateDomains.forEach(function (d) {
-            queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+    if ('mounts' in app) {
+        queries.push({ query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ]});
+        app.mounts.forEach(function (m) {
+            queries.push({ query: 'INSERT INTO appMounts (appId, volumeId, readOnly) VALUES (?, ?, ?)', args: [ id, m.volumeId, m.readOnly ]});
         });
     }
 
     var fields = [ ], values = [ ];
     for (var p in app) {
-        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig') {
+        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig') {
             fields.push(`${p}Json = ?`);
             values.push(JSON.stringify(app[p]));
-        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env') {
+        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env' && p !== 'mounts') {
             fields.push(p + ' = ?');
             values.push(app[p]);
         }

src/apphealthmonitor.js

@@ -14,7 +14,7 @@ var appdb = require('./appdb.js'),
     util = require('util');
 
 exports = module.exports = {
-    run: run
+    run
 };
 
 const HEALTHCHECK_INTERVAL = 10 * 1000; // every 10 seconds. this needs to be small since the UI makes only healthy apps clickable
@@ -73,22 +73,14 @@ function checkAppHealth(app, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     if (app.installationState !== apps.ISTATE_INSTALLED || app.runState !== apps.RSTATE_RUNNING) {
-        debugApp(app, 'skipped. istate:%s rstate:%s', app.installationState, app.runState);
         return callback(null);
     }
 
     const manifest = app.manifest;
 
     docker.inspect(app.containerId, function (error, data) {
-        if (error || !data || !data.State) {
-            debugApp(app, 'Error inspecting container');
-            return setHealth(app, apps.HEALTH_ERROR, callback);
-        }
-
-        if (data.State.Running !== true) {
-            debugApp(app, 'exited');
-            return setHealth(app, apps.HEALTH_DEAD, callback);
-        }
+        if (error || !data || !data.State) return setHealth(app, apps.HEALTH_ERROR, callback);
+        if (data.State.Running !== true) return setHealth(app, apps.HEALTH_DEAD, callback);
 
         // non-appstore apps may not have healthCheckPath
         if (!manifest.healthCheckPath) return setHealth(app, apps.HEALTH_HEALTHY, callback);
@@ -103,10 +95,8 @@ function checkAppHealth(app, callback) {
             .timeout(HEALTHCHECK_INTERVAL)
             .end(function (error, res) {
                 if (error && !error.response) {
-                    debugApp(app, 'not alive (network error): %s', error.message);
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
                 } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
-                    debugApp(app, 'not alive : %s', error || res.status);
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
                 } else {
                     setHealth(app, apps.HEALTH_HEALTHY, callback);
@@ -180,18 +170,14 @@ function processDockerEvents(intervalSecs, callback) {
 function processApp(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    apps.getAll(function (error, result) {
+    apps.getAll(function (error, allApps) {
         if (error) return callback(error);
 
-        async.each(result, checkAppHealth, function (error) {
-            if (error) console.error(error);
+        async.each(allApps, checkAppHealth, function (error) {
+            const alive = allApps
+                .filter(function (a) { return a.installationState === apps.ISTATE_INSTALLED && a.runState === apps.RSTATE_RUNNING && a.health === apps.HEALTH_HEALTHY; });
 
-            const alive = result
-                .filter(function (a) { return a.installationState === apps.ISTATE_INSTALLED && a.runState === apps.RSTATE_RUNNING && a.health === apps.HEALTH_HEALTHY; })
-                .map(a => a.fqdn)
-                .join(', ');
-
-            debug('apps alive: [%s]', alive);
+            debug(`app health: ${alive.length} alive / ${allApps.length - alive.length} dead.` + (error ? ` ${error.reason}` : ''));
 
             callback(null);
         });
@@ -206,7 +192,7 @@ function run(intervalSecs, callback) {
         processApp, // this is first because docker.getEvents seems to get 'stuck' sometimes
         processDockerEvents.bind(null, intervalSecs)
     ], function (error) {
-        if (error) debug(error);
+        if (error) debug(`run: could not check app health. ${error.message}`);
 
         callback();
     });

src/apps.js

@@ -1,68 +1,71 @@
 'use strict';
 
 exports = module.exports = {
-    hasAccessTo: hasAccessTo,
-    removeInternalFields: removeInternalFields,
-    removeRestrictedFields: removeRestrictedFields,
+    hasAccessTo,
+    removeInternalFields,
+    removeRestrictedFields,
 
-    get: get,
-    getByContainerId: getByContainerId,
-    getByIpAddress: getByIpAddress,
-    getByFqdn: getByFqdn,
-    getAll: getAll,
-    getAllByUser: getAllByUser,
-    install: install,
-    uninstall: uninstall,
+    get,
+    getByContainerId,
+    getByIpAddress,
+    getByFqdn,
+    getAll,
+    getAllByUser,
+    install,
+    uninstall,
 
-    setAccessRestriction: setAccessRestriction,
-    setLabel: setLabel,
-    setIcon: setIcon,
-    setTags: setTags,
-    setMemoryLimit: setMemoryLimit,
-    setCpuShares: setCpuShares,
-    setAutomaticBackup: setAutomaticBackup,
-    setAutomaticUpdate: setAutomaticUpdate,
-    setReverseProxyConfig: setReverseProxyConfig,
-    setCertificate: setCertificate,
-    setDebugMode: setDebugMode,
-    setEnvironment: setEnvironment,
-    setMailbox: setMailbox,
-    setLocation: setLocation,
-    setDataDir: setDataDir,
-    repair: repair,
+    setAccessRestriction,
+    setLabel,
+    setIcon,
+    setTags,
+    setMemoryLimit,
+    setCpuShares,
+    setMounts,
+    setAutomaticBackup,
+    setAutomaticUpdate,
+    setReverseProxyConfig,
+    setCertificate,
+    setDebugMode,
+    setEnvironment,
+    setMailbox,
+    setLocation,
+    setDataDir,
+    repair,
 
-    restore: restore,
-    importApp: importApp,
-    clone: clone,
+    restore,
+    importApp,
+    clone,
 
-    update: update,
+    update,
 
-    backup: backup,
-    listBackups: listBackups,
+    backup,
+    listBackups,
 
-    getLogs: getLogs,
+    getLocalLogfilePaths,
+    getLogs,
 
-    start: start,
-    stop: stop,
-    restart: restart,
+    start,
+    stop,
+    restart,
 
-    exec: exec,
+    exec,
 
-    checkManifestConstraints: checkManifestConstraints,
-    downloadManifest: downloadManifest,
+    checkManifestConstraints,
+    downloadManifest,
 
-    canAutoupdateApp: canAutoupdateApp,
-    autoupdateApps: autoupdateApps,
+    canAutoupdateApp,
+    autoupdateApps,
 
-    restoreInstalledApps: restoreInstalledApps,
-    configureInstalledApps: configureInstalledApps,
-    schedulePendingTasks: schedulePendingTasks,
+    restoreInstalledApps,
+    configureInstalledApps,
+    schedulePendingTasks,
+    restartAppsUsingAddons,
 
-    getDataDir: getDataDir,
-    getIconPath: getIconPath,
+    getDataDir,
+    getIconPath,
 
-    downloadFile: downloadFile,
-    uploadFile: uploadFile,
+    downloadFile,
+    uploadFile,
 
     PORT_TYPE_TCP: 'tcp',
     PORT_TYPE_UDP: 'udp',
@@ -404,14 +407,14 @@ function removeInternalFields(app) {
         'location', 'domain', 'fqdn', 'mailboxName', 'mailboxDomain',
         'accessRestriction', 'manifest', 'portBindings', 'iconUrl', 'memoryLimit', 'cpuShares',
         'sso', 'debugMode', 'reverseProxyConfig', 'enableBackup', 'creationTime', 'updateTime', 'ts', 'tags',
-        'label', 'alternateDomains', 'env', 'enableAutomaticUpdate', 'dataDir');
+        'label', 'alternateDomains', 'env', 'enableAutomaticUpdate', 'dataDir', 'mounts');
 }
 
 // non-admins can only see these
 function removeRestrictedFields(app) {
     return _.pick(app,
-        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId',
-        'location', 'domain', 'fqdn', 'manifest', 'portBindings', 'iconUrl', 'creationTime', 'ts', 'tags', 'label');
+        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId', 'alternateDomains', 'sso',
+        'location', 'domain', 'fqdn', 'manifest', 'portBindings', 'iconUrl', 'creationTime', 'ts', 'tags', 'label', 'enableBackup');
 }
 
 function getIconUrlSync(app) {
@@ -605,6 +608,8 @@ function downloadManifest(appStoreId, manifest, callback) {
 
         if (result.statusCode !== 200) return callback(new BoxError(BoxError.NOT_FOUND, util.format('Failed to get app info from store.', result.statusCode, result.text)));
 
+        if (!result.body.manifest || typeof result.body.manifest !== 'object') return callback(new BoxError(BoxError.NOT_FOUND, util.format('Missing manifest. Failed to get app info from store.', result.statusCode, result.text)));
+
         callback(null, parts[0], result.body.manifest);
     });
 }
@@ -622,7 +627,7 @@ function scheduleTask(appId, installationState, taskId, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     appTaskManager.scheduleTask(appId, taskId, function (error) {
-        debug(`scheduleTask: task ${taskId} of $${appId} completed`);
+        debug(`scheduleTask: task ${taskId} of ${appId} completed`);
         if (error && (error.code === tasks.ECRASHED || error.code === tasks.ESTOPPED)) { // if task crashed, update the error
             debug(`Apptask crashed/stopped: ${error.message}`);
             let boxError = new BoxError(BoxError.TASK_ERROR, error.message);
@@ -678,6 +683,11 @@ function checkAppState(app, state) {
         if (state !== exports.ISTATE_PENDING_UNINSTALL && state !== exports.ISTATE_PENDING_RESTORE) return new BoxError(BoxError.BAD_STATE, 'Not allowed in error state');
     }
 
+    if (app.runState === exports.RSTATE_STOPPED) {
+        // can't backup or restore since app addons are down. can't update because migration scripts won't run
+        if (state === exports.ISTATE_PENDING_UPDATE || state === exports.ISTATE_PENDING_BACKUP || state === exports.ISTATE_PENDING_RESTORE)  return new BoxError(BoxError.BAD_STATE, 'Not allowed in stopped state');
+    }
+
     return null;
 }
 
@@ -761,6 +771,8 @@ function install(data, auditSource, callback) {
     error = validateEnv(env);
     if (error) return callback(error);
 
+    if (settings.isDemo() && constants.DEMO_BLACKLISTED_APPS.includes(appStoreId)) return callback(new BoxError(BoxError.BAD_FIELD, 'This app is blacklisted in the demo'));
+
     const mailboxName = hasMailAddon(manifest) ? mailboxNameForLocation(location, manifest) : null;
     const mailboxDomain = hasMailAddon(manifest) ? domain : null;
     const appId = uuid.v4();
@@ -968,6 +980,30 @@ function setCpuShares(app, cpuShares, auditSource, callback) {
     });
 }
 
+function setMounts(app, mounts, auditSource, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert(Array.isArray(mounts));
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const appId = app.id;
+    let error = checkAppState(app, exports.ISTATE_PENDING_RECREATE_CONTAINER);
+    if (error) return callback(error);
+
+    const task = {
+        args: {},
+        values: { mounts }
+    };
+    addTask(appId, exports.ISTATE_PENDING_RECREATE_CONTAINER, task, function (error, result) {
+        if (error && error.reason === BoxError.ALREADY_EXISTS) return callback(new BoxError(BoxError.CONFLICT, 'Duplicate mount points'));
+        if (error) return callback(error);
+
+        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId, app, mounts, taskId: result.taskId });
+
+        callback(null, { taskId: result.taskId });
+    });
+}
+
 function setEnvironment(app, env, auditSource, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof env, 'object');
@@ -1314,6 +1350,19 @@ function update(app, data, auditSource, callback) {
     });
 }
 
+function getLocalLogfilePaths(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    const appId = app.id;
+
+    var filePaths = [];
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'app.log'));
+    if (app.manifest.addons && app.manifest.addons.redis) filePaths.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
+
+    return filePaths;
+}
+
 function getLogs(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
     assert(options && typeof options === 'object');
@@ -1333,11 +1382,8 @@ function getLogs(app, options, callback) {
 
     var args = [ '--lines=' + lines ];
     if (follow) args.push('--follow', '--retry', '--quiet'); // same as -F. to make it work if file doesn't exist, --quiet to not output file headers, which are no logs
-    args.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
-    args.push(path.join(paths.LOG_DIR, appId, 'app.log'));
-    if (app.manifest.addons && app.manifest.addons.redis) args.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
 
-    var cp = spawn('/usr/bin/tail', args);
+    var cp = spawn('/usr/bin/tail', args.concat(getLocalLogfilePaths(app)));
 
     var transformStream = split(function mapper(line) {
         if (format !== 'json') return line + '\n';
@@ -1435,7 +1481,8 @@ function restore(app, backupId, auditSource, callback) {
     func(function (error, backupInfo) {
         if (error) return callback(error);
 
-        if (!backupInfo.manifest) callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Could not get restore manifest'));
+        if (!backupInfo.manifest) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Could not get restore manifest'));
+        if (backupInfo.encryptionVersion === 1) return callback(new BoxError(BoxError.BAD_FIELD, 'This encrypted backup was created with an older Cloudron version and has to be restored using the CLI tool'));
 
         // re-validate because this new box version may not accept old configs
         error = checkManifestConstraints(backupInfo.manifest);
@@ -1495,6 +1542,15 @@ function importApp(app, data, auditSource, callback) {
     testBackupConfig(function (error) {
         if (error) return callback(error);
 
+        if (backupConfig) {
+            if ('password' in backupConfig) {
+                backupConfig.encryption = backups.generateEncryptionKeysSync(backupConfig.password);
+                delete backupConfig.password;
+            } else {
+                backupConfig.encryption = null;
+            }
+        }
+
         const restoreConfig = { backupId, backupFormat, backupConfig };
 
         const task = {
@@ -1553,7 +1609,8 @@ function clone(app, data, user, auditSource, callback) {
     backups.get(backupId, function (error, backupInfo) {
         if (error) return callback(error);
 
-        if (!backupInfo.manifest) callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Could not get restore config'));
+        if (!backupInfo.manifest) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Could not get restore config'));
+        if (backupInfo.encryptionVersion === 1) return callback(new BoxError(BoxError.BAD_FIELD, 'This encrypted backup was created with an older Cloudron version and cannot be cloned'));
 
         const manifest = backupInfo.manifest, appStoreId = app.appStoreId;
 
@@ -1775,14 +1832,25 @@ function exec(app, options, callback) {
     });
 }
 
-function canAutoupdateApp(app, newManifest) {
+function canAutoupdateApp(app, updateInfo) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof updateInfo, 'object');
+
+    const manifest = updateInfo.manifest;
+
     if (!app.enableAutomaticUpdate) return false;
-    if ((semver.major(app.manifest.version) !== 0) && (semver.major(app.manifest.version) !== semver.major(newManifest.version))) return false; // major changes are blocking
+
+    // for invalid subscriptions the appstore does not return a dockerImage
+    if (!manifest.dockerImage) return false;
+
+    if (updateInfo.unstable) return false; // only manual update allowed for unstable updates
+
+    if ((semver.major(app.manifest.version) !== 0) && (semver.major(app.manifest.version) !== semver.major(manifest.version))) return false; // major changes are blocking
 
     if (app.runState === exports.RSTATE_STOPPED) return false; // stopped apps won't run migration scripts and shouldn't be updated
 
-    const newTcpPorts = newManifest.tcpPorts || { };
-    const newUdpPorts = newManifest.udpPorts || { };
+    const newTcpPorts = manifest.tcpPorts || { };
+    const newUdpPorts = manifest.udpPorts || { };
     const portBindings = app.portBindings; // this is never null
 
     for (let portName in portBindings) {
@@ -1807,7 +1875,7 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
                 return iteratorDone();
             }
 
-            if (!canAutoupdateApp(app, updateInfo[appId].manifest)) {
+            if (!canAutoupdateApp(app, updateInfo[appId])) {
                 debug(`app ${app.fqdn} requires manual update`);
                 return iteratorDone();
             }
@@ -1852,7 +1920,7 @@ function listBackups(app, page, perPage, callback) {
     assert(typeof perPage === 'number' && perPage > 0);
     assert.strictEqual(typeof callback, 'function');
 
-    backups.getByAppIdPaged(page, perPage, app.id, function (error, results) {
+    backups.getByIdentifierAndStatePaged(app.id, backups.BACKUP_STATE_NORMAL, page, perPage, function (error, results) {
         if (error) return callback(error);
 
         callback(null, results);
@@ -1869,7 +1937,7 @@ function restoreInstalledApps(callback) {
         apps = apps.filter(app => app.installationState !== exports.ISTATE_PENDING_RESTORE); // safeguard against tasks being created non-stop if we crash on startup
 
         async.eachSeries(apps, function (app, iteratorDone) {
-            backups.getByAppIdPaged(1, 1, app.id, function (error, results) {
+            backups.getByIdentifierAndStatePaged(app.id, backups.BACKUP_STATE_NORMAL, 1, 1, function (error, results) {
                 let installationState, restoreConfig, oldManifest;
                 if (!error && results.length) {
                     installationState = exports.ISTATE_PENDING_RESTORE;
@@ -1930,6 +1998,41 @@ function configureInstalledApps(callback) {
     });
 }
 
+function restartAppsUsingAddons(changedAddons, callback) {
+    assert(Array.isArray(changedAddons));
+    assert.strictEqual(typeof callback, 'function');
+
+    getAll(function (error, apps) {
+        if (error) return callback(error);
+
+        apps = apps.filter(app => app.manifest.addons && _.intersection(Object.keys(app.manifest.addons), changedAddons).length !== 0);
+        apps = apps.filter(app => app.installationState !== exports.ISTATE_ERROR); // remove errored apps. let them be 'repaired' by hand
+        apps = apps.filter(app => app.installationState !== exports.ISTATE_PENDING_RESTART); // safeguard against tasks being created non-stop restart if we crash on startup
+        apps = apps.filter(app => app.runState !== exports.RSTATE_STOPPED); // don't start stopped apps
+
+        async.eachSeries(apps, function (app, iteratorDone) {
+            debug(`restartAppsUsingAddons: marking ${app.fqdn} for restart`);
+
+            const task = {
+                args: {},
+                values: { runState: exports.RSTATE_RUNNING }
+            };
+
+            // stop apps before updating the databases because postgres will "lock" them preventing import
+            docker.stopContainers(app.id, function (error) {
+                if (error) debug(`restartAppsUsingAddons: error stopping ${app.fqdn}`, error);
+
+                addTask(app.id, exports.ISTATE_PENDING_RESTART, task, function (error, result) {
+                    if (error) debug(`restartAppsUsingAddons: error marking ${app.fqdn} for restart: ${JSON.stringify(error)}`);
+                    else debug(`restartAppsUsingAddons: marked ${app.id} for restart with taskId ${result.taskId}`);
+
+                    iteratorDone(); // ignore error
+                });
+            });
+        }, callback);
+    });
+}
+
 // auto-restart app tasks after a crash
 function schedulePendingTasks(callback) {
     assert.strictEqual(typeof callback, 'function');

src/appstore.js

@@ -11,7 +11,6 @@ exports = module.exports = {
     trackFinishedSetup: trackFinishedSetup,
 
     registerWithLoginCredentials: registerWithLoginCredentials,
-    registerWithLicense: registerWithLicense,
 
     purchaseApp: purchaseApp,
     unpurchaseApp: unpurchaseApp,
@@ -20,8 +19,6 @@ exports = module.exports = {
     getSubscription: getSubscription,
     isFreePlan: isFreePlan,
 
-    sendAliveStatus: sendAliveStatus,
-
     getAppUpdate: getAppUpdate,
     getBoxUpdate: getBoxUpdate,
 
@@ -34,32 +31,28 @@ var apps = require('./apps.js'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:appstore'),
-    domains = require('./domains.js'),
     eventlog = require('./eventlog.js'),
-    groups = require('./groups.js'),
-    mail = require('./mail.js'),
-    os = require('os'),
+    path = require('path'),
     paths = require('./paths.js'),
     safe = require('safetydance'),
     semver = require('semver'),
     settings = require('./settings.js'),
     superagent = require('superagent'),
-    users = require('./users.js'),
+    support = require('./support.js'),
     util = require('util');
 
-const NOOP_CALLBACK = function (error) { if (error) debug(error); };
-
 // These are the default options and will be adjusted once a subscription state is obtained
 // Keep in sync with appstore/routes/cloudrons.js
 let gFeatures = {
-    userMaxCount: null,
-    externalLdap: true,
-    eventLog: true,
-    privateDockerRegistry: true,
-    branding: true,
-    userManager: true,
-    multiAdmin: true,
-    support: true
+    userMaxCount: 5,
+    domainMaxCount: 1,
+    externalLdap: false,
+    privateDockerRegistry: false,
+    branding: false,
+    support: false,
+    directoryConfig: false,
+    mailboxMaxCount: 5,
+    emailPremium: false
 };
 
 // attempt to load feature cache in case appstore would be down
@@ -127,7 +120,7 @@ function registerUser(email, password, callback) {
     const url = settings.apiServerOrigin() + '/api/v1/register_user';
     superagent.post(url).send(data).timeout(30 * 1000).end(function (error, result) {
         if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-        if (result.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS));
+        if (result.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
         if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `register status code: ${result.statusCode}`));
 
         callback(null);
@@ -235,112 +228,8 @@ function unpurchaseApp(appId, data, callback) {
     });
 }
 
-function sendAliveStatus(callback) {
-    callback = callback || NOOP_CALLBACK;
-
-    let allSettings, allDomains, mailDomains, loginEvents, userCount, groupCount;
-
-    async.series([
-        function (callback) {
-            settings.getAll(function (error, result) {
-                if (error) return callback(error);
-                allSettings = result;
-                callback();
-            });
-        },
-        function (callback) {
-            domains.getAll(function (error, result) {
-                if (error) return callback(error);
-                allDomains = result;
-                callback();
-            });
-        },
-        function (callback) {
-            mail.getDomains(function (error, result) {
-                if (error) return callback(error);
-                mailDomains = result;
-                callback();
-            });
-        },
-        function (callback) {
-            eventlog.getAllPaged([ eventlog.ACTION_USER_LOGIN ], null, 1, 1, function (error, result) {
-                if (error) return callback(error);
-                loginEvents = result;
-                callback();
-            });
-        },
-        function (callback) {
-            users.count(function (error, result) {
-                if (error) return callback(error);
-                userCount = result;
-                callback();
-            });
-        },
-        function (callback) {
-            groups.count(function (error, result) {
-                if (error) return callback(error);
-                groupCount = result;
-                callback();
-            });
-        }
-    ], function (error) {
-        if (error) return callback(error);
-
-        var backendSettings = {
-            backupConfig: {
-                provider: allSettings[settings.BACKUP_CONFIG_KEY].provider,
-                hardlinks: !allSettings[settings.BACKUP_CONFIG_KEY].noHardlinks
-            },
-            domainConfig: {
-                count: allDomains.length,
-                domains: Array.from(new Set(allDomains.map(function (d) { return { domain: d.domain, provider: d.provider }; })))
-            },
-            mailConfig: {
-                outboundCount: mailDomains.length,
-                inboundCount: mailDomains.filter(function (d) { return d.enabled; }).length,
-                catchAllCount: mailDomains.filter(function (d) { return d.catchAll.length !== 0; }).length,
-                relayProviders: Array.from(new Set(mailDomains.map(function (d) { return d.relay.provider; })))
-            },
-            userCount: userCount,
-            groupCount: groupCount,
-            appAutoupdatePattern: allSettings[settings.APP_AUTOUPDATE_PATTERN_KEY],
-            boxAutoupdatePattern: allSettings[settings.BOX_AUTOUPDATE_PATTERN_KEY],
-            timeZone: allSettings[settings.TIME_ZONE_KEY],
-            sysinfoProvider: allSettings[settings.SYSINFO_CONFIG_KEY].provider
-        };
-
-        var data = {
-            version: constants.VERSION,
-            adminFqdn: settings.adminFqdn(),
-            provider: settings.provider(),
-            backendSettings: backendSettings,
-            machine: {
-                cpus: os.cpus(),
-                totalmem: os.totalmem()
-            },
-            events: {
-                lastLogin: loginEvents[0] ? (new Date(loginEvents[0].creationTime).getTime()) : 0
-            }
-        };
-
-        getCloudronToken(function (error, token) {
-            if (error) return callback(error);
-
-            const url = `${settings.apiServerOrigin()}/api/v1/alive`;
-            superagent.post(url).send(data).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
-                if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Sending alive status failed. %s %j', result.status, result.body)));
-
-                callback(null);
-            });
-        });
-    });
-}
-
-function getBoxUpdate(callback) {
+function getBoxUpdate(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     getCloudronToken(function (error, token) {
@@ -348,7 +237,13 @@ function getBoxUpdate(callback) {
 
         const url = `${settings.apiServerOrigin()}/api/v1/boxupdate`;
 
-        superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION }).timeout(10 * 1000).end(function (error, result) {
+        const query = {
+            accessToken: token,
+            boxVersion: constants.VERSION,
+            automatic: options.automatic
+        };
+
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -374,16 +269,24 @@ function getBoxUpdate(callback) {
     });
 }
 
-function getAppUpdate(app, callback) {
+function getAppUpdate(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     getCloudronToken(function (error, token) {
         if (error) return callback(error);
 
         const url = `${settings.apiServerOrigin()}/api/v1/appupdate`;
+        const query = {
+            accessToken: token,
+            boxVersion: constants.VERSION,
+            appId: app.appStoreId,
+            appVersion: app.manifest.version,
+            automatic: options.automatic
+        };
 
-        superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, appId: app.appStoreId, appVersion: app.manifest.version }).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -401,7 +304,9 @@ function getAppUpdate(app, callback) {
                 return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Malformed update: %s %s', result.statusCode, result.text)));
             }
 
-            // { id, creationDate, manifest }
+            updateInfo.unstable = !!updateInfo.unstable;
+
+            // { id, creationDate, manifest, unstable }
             callback(null, updateInfo);
         });
     });
@@ -415,7 +320,7 @@ function registerCloudron(data, callback) {
 
     superagent.post(url).send(data).timeout(30 * 1000).end(function (error, result) {
         if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Unable to register cloudron: ${error.message}`));
+        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Unable to register cloudron: ${result.statusCode} ${error.message}`));
 
         // cloudronId, token, licenseKey
         if (!result.body.cloudronId) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response - no cloudron id'));
@@ -438,18 +343,16 @@ function registerCloudron(data, callback) {
 
 // This works without a Cloudron token as this Cloudron was not yet registered
 let gBeginSetupAlreadyTracked = false;
-function trackBeginSetup(provider) {
-    assert.strictEqual(typeof provider, 'string');
-
+function trackBeginSetup() {
     // avoid browser reload double tracking, not perfect since box might restart, but covers most cases and is simple
     if (gBeginSetupAlreadyTracked) return;
     gBeginSetupAlreadyTracked = true;
 
     const url = `${settings.apiServerOrigin()}/api/v1/helper/setup_begin`;
 
-    superagent.post(url).send({ provider }).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return console.error(error.message);
-        if (result.statusCode !== 200) return console.error(error.message);
+    superagent.post(url).send({}).timeout(30 * 1000).end(function (error, result) {
+        if (error && !error.response) return debug(`trackBeginSetup: ${error.message}`);
+        if (result.statusCode !== 200) return debug(`trackBeginSetup: ${result.statusCode} ${error.message}`);
     });
 }
 
@@ -460,23 +363,8 @@ function trackFinishedSetup(domain) {
     const url = `${settings.apiServerOrigin()}/api/v1/helper/setup_finished`;
 
     superagent.post(url).send({ domain }).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return console.error(error.message);
-        if (result.statusCode !== 200) return console.error(error.message);
-    });
-}
-
-function registerWithLicense(license, domain, callback) {
-    assert.strictEqual(typeof license, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    getCloudronToken(function (error, token) {
-        if (token) return callback(new BoxError(BoxError.CONFLICT));
-
-        const provider = settings.provider();
-        const version = constants.VERSION;
-
-        registerCloudron({ license, domain, provider, version }, callback);
+        if (error && !error.response) return debug(`trackFinishedSetup: ${error.message}`);
+        if (result.statusCode !== 200) return debug(`trackFinishedSetup: ${result.statusCode} ${error.message}`);
     });
 }
 
@@ -491,7 +379,7 @@ function registerWithLoginCredentials(options, callback) {
     }
 
     getCloudronToken(function (error, token) {
-        if (token) return callback(new BoxError(BoxError.CONFLICT));
+        if (token) return callback(new BoxError(BoxError.CONFLICT, 'Cloudron is already registered'));
 
         maybeSignup(function (error) {
             if (error) return callback(error);
@@ -499,7 +387,7 @@ function registerWithLoginCredentials(options, callback) {
             login(options.email, options.password, options.totpToken || '', function (error, result) {
                 if (error) return callback(error);
 
-                registerCloudron({ domain: settings.adminDomain(), accessToken: result.accessToken, provider: settings.provider(), version: constants.VERSION, purpose: options.purpose || '' }, callback);
+                registerCloudron({ domain: settings.adminDomain(), accessToken: result.accessToken, version: constants.VERSION, purpose: options.purpose || '' }, callback);
             });
         });
     });
@@ -520,26 +408,55 @@ function createTicket(info, auditSource, callback) {
         apps.get(info.appId, callback);
     }
 
+    function enableSshIfNeeded(callback) {
+        if (!info.enableSshSupport) return callback();
+
+        support.enableRemoteSupport(true, auditSource, function (error) {
+            // ensure we can at least get the ticket through
+            if (error) debug('Unable to enable SSH support.', error);
+
+            callback();
+        });
+    }
+
     getCloudronToken(function (error, token) {
         if (error) return callback(error);
 
-        collectAppInfoIfNeeded(function (error, result) {
-            if (error) console.error('Unable to get app info', error);
-            if (result) info.app = result;
+        enableSshIfNeeded(function (error) {
+            if (error) return callback(error);
 
-            let url = settings.apiServerOrigin() + '/api/v1/ticket';
+            collectAppInfoIfNeeded(function (error, app) {
+                if (error) return callback(error);
+                if (app) info.app = app;
 
-            info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
+                info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
 
-            superagent.post(url).query({ accessToken: token }).send(info).timeout(10 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+                var req = superagent.post(`${settings.apiServerOrigin()}/api/v1/ticket`)
+                    .query({ accessToken: token })
+                    .timeout(30 * 1000);
 
-                eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+                // either send as JSON through body or as multipart, depending on attachments
+                if (info.app) {
+                    req.field('infoJSON', JSON.stringify(info));
 
-                callback(null, { message: `An email for sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                    apps.getLocalLogfilePaths(info.app).forEach(function (filePath) {
+                        var logs = safe.child_process.execSync(`tail --lines=1000 ${filePath}`);
+                        if (logs) req.attach(path.basename(filePath), logs, path.basename(filePath));
+                    });
+                } else {
+                    req.send(info);
+                }
+
+                req.end(function (error, result) {
+                    if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                    if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+                    if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
+                    if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+
+                    eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+
+                    callback(null, { message: `An email was sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                });
             });
         });
     });
@@ -555,7 +472,7 @@ function getApps(callback) {
             if (error) return callback(error);
 
             const url = `${settings.apiServerOrigin()}/api/v1/apps`;
-            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -590,7 +507,7 @@ function getAppVersion(appId, version, callback) {
             let url = `${settings.apiServerOrigin()}/api/v1/apps/${appId}`;
             if (version !== 'latest') url += `/versions/${version}`;
 
-            superagent.get(url).query({ accessToken: token }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));

src/apptask.js

@@ -17,8 +17,6 @@ exports = module.exports = {
     _waitForDnsPropagation: waitForDnsPropagation
 };
 
-require('supererror')({ splatchError: true });
-
 var addons = require('./addons.js'),
     appdb = require('./appdb.js'),
     apps = require('./apps.js'),
@@ -37,7 +35,6 @@ var addons = require('./addons.js'),
     eventlog = require('./eventlog.js'),
     fs = require('fs'),
     manifestFormat = require('cloudron-manifestformat'),
-    mkdirp = require('mkdirp'),
     net = require('net'),
     os = require('os'),
     path = require('path'),
@@ -176,7 +173,7 @@ function createAppDir(app, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     const appDir = path.join(paths.APPS_DATA_DIR, app.id);
-    mkdirp(appDir, function (error) {
+    fs.mkdir(appDir, { recursive: true }, function (error) {
         if (error) return callback(new BoxError(BoxError.FS_ERROR, `Error creating directory: ${error.message}`, { appDir }));
 
         callback(null);
@@ -741,7 +738,8 @@ function migrateDataDir(app, args, progressCallback, callback) {
             debugApp(app, 'error migrating data dir : %s', error);
             return updateApp(app, { installationState: apps.ISTATE_ERROR, error: makeTaskError(error, app) }, callback.bind(null, error));
         }
-        callback(null);
+
+        callback();
     });
 }
 
@@ -786,7 +784,8 @@ function configure(app, args, progressCallback, callback) {
             debugApp(app, 'error reconfiguring : %s', error);
             return updateApp(app, { installationState: apps.ISTATE_ERROR, error: makeTaskError(error, app) }, callback.bind(null, error));
         }
-        callback(null);
+
+        callback();
     });
 }
 
@@ -853,7 +852,7 @@ function update(app, args, progressCallback, callback) {
                 if (newTcpPorts[portName] || newUdpPorts[portName]) return callback(null); // port still in use
 
                 appdb.delPortBinding(currentPorts[portName], apps.PORT_TYPE_TCP, function (error) {
-                    if (error && error.reason === BoxError.NOT_FOUND) console.error('Portbinding does not exist in database.');
+                    if (error && error.reason === BoxError.NOT_FOUND) debug('update: portbinding does not exist in database', error);
                     else if (error) return next(error);
 
                     // also delete from app object for further processing (the db is updated in the next step)
@@ -869,10 +868,10 @@ function update(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 45, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
 
-        progressCallback.bind(null, { percent: 70, message: 'Updating addons' }),
+        progressCallback.bind(null, { percent: 60, message: 'Updating addons' }),
         addons.setupAddons.bind(null, app, updateConfig.manifest.addons),
 
-        progressCallback.bind(null, { percent: 80, message: 'Creating container' }),
+        progressCallback.bind(null, { percent: 70, message: 'Creating container' }),
         createContainer.bind(null, app),
 
         startApp.bind(null, app),
@@ -899,7 +898,10 @@ function start(app, args, progressCallback, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     async.series([
-        progressCallback.bind(null, { percent: 20, message: 'Starting container' }),
+        progressCallback.bind(null, { percent: 10, message: 'Starting app services' }),
+        addons.startAppServices.bind(null, app),
+
+        progressCallback.bind(null, { percent: 35, message: 'Starting container' }),
         docker.startContainer.bind(null, app.id),
 
         // stopped apps do not renew certs. currently, we don't do DNS to not overwrite existing user settings
@@ -927,6 +929,9 @@ function stop(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 20, message: 'Stopping container' }),
         docker.stopContainers.bind(null, app.id),
 
+        progressCallback.bind(null, { percent: 50, message: 'Stopping app services' }),
+        addons.stopAppServices.bind(null, app),
+
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null })
     ], function seriesDone(error) {
@@ -973,16 +978,18 @@ function uninstall(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 30, message: 'Teardown addons' }),
         addons.teardownAddons.bind(null, app, app.manifest.addons),
 
-        progressCallback.bind(null, { percent: 40, message: 'Deleting app data directory' }),
+        progressCallback.bind(null, { percent: 40, message: 'Cleanup file manager' }),
+
+        progressCallback.bind(null, { percent: 50, message: 'Deleting app data directory' }),
         deleteAppDir.bind(null, app, { removeDirectory: true }),
 
-        progressCallback.bind(null, { percent: 50, message: 'Deleting image' }),
+        progressCallback.bind(null, { percent: 60, message: 'Deleting image' }),
         docker.deleteImage.bind(null, app.manifest),
 
-        progressCallback.bind(null, { percent: 60, message: 'Unregistering domains' }),
+        progressCallback.bind(null, { percent: 70, message: 'Unregistering domains' }),
         unregisterSubdomains.bind(null, app, [ { subdomain: app.location, domain: app.domain } ].concat(app.alternateDomains)),
 
-        progressCallback.bind(null, { percent: 70, message: 'Cleanup icon' }),
+        progressCallback.bind(null, { percent: 80, message: 'Cleanup icon' }),
         removeIcon.bind(null, app),
 
         progressCallback.bind(null, { percent: 90, message: 'Cleanup logs' }),
@@ -1038,6 +1045,8 @@ function run(appId, args, progressCallback, callback) {
             return stop(app, args, progressCallback, callback);
         case apps.ISTATE_PENDING_RESTART:
             return restart(app, args, progressCallback, callback);
+        case apps.ISTATE_INSTALLED: // can only happen when we have a bug in our code while testing/development
+            return updateApp(app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null }, callback);
         default:
             debugApp(app, 'apptask launched with invalid command');
             return callback(new BoxError(BoxError.INTERNAL_ERROR, 'Unknown install command in apptask:' + app.installationState));

src/apptaskmanager.js

@@ -12,6 +12,7 @@ let assert = require('assert'),
     safe = require('safetydance'),
     path = require('path'),
     paths = require('./paths.js'),
+    sftp = require('./sftp.js'),
     tasks = require('./tasks.js');
 
 let gActiveTasks = { }; // indexed by app id
@@ -68,11 +69,17 @@ function scheduleTask(appId, taskId, callback) {
 
     if (!fs.existsSync(path.dirname(logFile))) safe.fs.mkdirSync(path.dirname(logFile)); // ensure directory
 
-    tasks.startTask(taskId, { logFile, timeout: 20 * 60 * 60 * 1000 /* 20 hours */ }, function (error, result) {
+    // TODO: set memory limit for app backup task
+    tasks.startTask(taskId, { logFile, timeout: 20 * 60 * 60 * 1000 /* 20 hours */, nice: 15 }, function (error, result) {
         callback(error, result);
 
         delete gActiveTasks[appId];
         locker.unlock(locker.OP_APPTASK); // unlock event will trigger next task
+
+        // post app task hooks
+        sftp.rebuild(function (error) {
+            if (error) debug('Unable to rebuild sftp:', error);
+        });
     });
 }
 

src/backupdb.js

@@ -6,27 +6,20 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var BACKUPS_FIELDS = [ 'id', 'creationTime', 'version', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs' ];
+var BACKUPS_FIELDS = [ 'id', 'identifier', 'creationTime', 'packageVersion', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs', 'encryptionVersion' ];
 
 exports = module.exports = {
-    add: add,
+    add,
 
-    getByTypeAndStatePaged: getByTypeAndStatePaged,
-    getByTypePaged: getByTypePaged,
+    getByTypePaged,
+    getByIdentifierPaged,
+    getByIdentifierAndStatePaged,
 
-    get: get,
-    del: del,
-    update: update,
-    getByAppIdPaged: getByAppIdPaged,
+    get,
+    del,
+    update,
 
-    _clear: clear,
-
-    BACKUP_TYPE_APP: 'app',
-    BACKUP_TYPE_BOX: 'box',
-
-    BACKUP_STATE_NORMAL: 'normal', // should rename to created to avoid listing in UI?
-    BACKUP_STATE_CREATING: 'creating',
-    BACKUP_STATE_ERROR: 'error'
+    _clear: clear
 };
 
 function postProcess(result) {
@@ -38,15 +31,15 @@ function postProcess(result) {
     delete result.manifestJson;
 }
 
-function getByTypeAndStatePaged(type, state, page, perPage, callback) {
-    assert(type === exports.BACKUP_TYPE_APP || type === exports.BACKUP_TYPE_BOX);
+function getByIdentifierAndStatePaged(identifier, state, page, perPage, callback) {
+    assert.strictEqual(typeof identifier, 'string');
     assert.strictEqual(typeof state, 'string');
     assert(typeof page === 'number' && page > 0);
     assert(typeof perPage === 'number' && perPage > 0);
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? ORDER BY creationTime DESC LIMIT ?,?',
-        [ type, state, (page-1)*perPage, perPage ], function (error, results) {
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE identifier = ? AND state = ? ORDER BY creationTime DESC LIMIT ?,?',
+        [ identifier, state, (page-1)*perPage, perPage ], function (error, results) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
             results.forEach(function (result) { postProcess(result); });
@@ -56,7 +49,7 @@ function getByTypeAndStatePaged(type, state, page, perPage, callback) {
 }
 
 function getByTypePaged(type, page, perPage, callback) {
-    assert(type === exports.BACKUP_TYPE_APP || type === exports.BACKUP_TYPE_BOX);
+    assert.strictEqual(typeof type, 'string');
     assert(typeof page === 'number' && page > 0);
     assert(typeof perPage === 'number' && perPage > 0);
     assert.strictEqual(typeof callback, 'function');
@@ -71,15 +64,14 @@ function getByTypePaged(type, page, perPage, callback) {
         });
 }
 
-function getByAppIdPaged(page, perPage, appId, callback) {
+function getByIdentifierPaged(identifier, page, perPage, callback) {
+    assert.strictEqual(typeof identifier, 'string');
     assert(typeof page === 'number' && page > 0);
     assert(typeof perPage === 'number' && perPage > 0);
-    assert.strictEqual(typeof appId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    // box versions (0.93.x and below) used to use appbackup_ prefix
-    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? AND id LIKE ? ORDER BY creationTime DESC LIMIT ?,?',
-        [ exports.BACKUP_TYPE_APP, exports.BACKUP_STATE_NORMAL, '%app%\\_' + appId + '\\_%', (page-1)*perPage, perPage ], function (error, results) {
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE identifier = ? ORDER BY creationTime DESC LIMIT ?,?',
+        [ identifier, (page-1)*perPage, perPage ], function (error, results) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
             results.forEach(function (result) { postProcess(result); });
@@ -106,8 +98,11 @@ function get(id, callback) {
 function add(id, data, callback) {
     assert(data && typeof data === 'object');
     assert.strictEqual(typeof id, 'string');
-    assert.strictEqual(typeof data.version, 'string');
-    assert(data.type === exports.BACKUP_TYPE_APP || data.type === exports.BACKUP_TYPE_BOX);
+    assert(data.encryptionVersion === null || typeof data.encryptionVersion === 'number');
+    assert.strictEqual(typeof data.packageVersion, 'string');
+    assert.strictEqual(typeof data.type, 'string');
+    assert.strictEqual(typeof data.identifier, 'string');
+    assert.strictEqual(typeof data.state, 'string');
     assert(util.isArray(data.dependsOn));
     assert.strictEqual(typeof data.manifest, 'object');
     assert.strictEqual(typeof data.format, 'string');
@@ -116,8 +111,8 @@ function add(id, data, callback) {
     var creationTime = data.creationTime || new Date(); // allow tests to set the time
     var manifestJson = JSON.stringify(data.manifest);
 
-    database.query('INSERT INTO backups (id, version, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
-        [ id, data.version, data.type, creationTime, exports.BACKUP_STATE_NORMAL, data.dependsOn.join(','), manifestJson, data.format ],
+    database.query('INSERT INTO backups (id, identifier, encryptionVersion, packageVersion, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
+        [ id, data.identifier, data.encryptionVersion, data.packageVersion, data.type, creationTime, data.state, data.dependsOn.join(','), manifestJson, data.format ],
         function (error) {
             if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS));
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));

src/boxerror.js

@@ -50,6 +50,7 @@ BoxError.FS_ERROR = 'FileSystem Error';
 BoxError.INACTIVE = 'Inactive';
 BoxError.INTERNAL_ERROR = 'Internal Error';
 BoxError.INVALID_CREDENTIALS = 'Invalid Credentials';
+BoxError.IPTABLES_ERROR = 'IPTables Error';
 BoxError.LICENSE_ERROR = 'License Error';
 BoxError.LOGROTATE_ERROR = 'Logrotate Error';
 BoxError.MAIL_ERROR = 'Mail Error';
@@ -92,6 +93,7 @@ BoxError.toHttpError = function (error) {
     case BoxError.MAIL_ERROR:
     case BoxError.DOCKER_ERROR:
     case BoxError.ADDONS_ERROR:
+    case BoxError.IPTABLES_ERROR:
         return new HttpError(424, error);
     case BoxError.DATABASE_ERROR:
     case BoxError.INTERNAL_ERROR:

src/branding.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports = module.exports = {
+    renderFooter
+};
+
+const assert = require('assert'),
+    constants = require('./constants.js');
+
+function renderFooter(footer) {
+    assert.strictEqual(typeof footer, 'string');
+
+    const year = new Date().getFullYear();
+
+    return footer.replace(/%YEAR%/g, year)
+        .replace(/%VERSION%/g, constants.VERSION);
+}
+

src/cert/acme2.js

@@ -332,7 +332,7 @@ Acme2.prototype.createKeyAndCsr = function (hostname, callback) {
         // in some old releases, csr file was corrupt. so always regenerate it
         debug('createKeyAndCsr: reuse the key for renewal at %s', privateKeyFile);
     } else {
-        var key = safe.child_process.execSync('openssl genrsa 4096');
+        var key = safe.child_process.execSync('openssl ecparam -genkey -name secp384r1'); // openssl ecparam -list_curves
         if (!key) return callback(new BoxError(BoxError.OPENSSL_ERROR, safe.error));
         if (!safe.fs.writeFileSync(privateKeyFile, key)) return callback(new BoxError(BoxError.FS_ERROR, safe.error));
 

src/cert/caas.js

@@ -1,22 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    getCertificate: getCertificate,
-
-    // testing
-    _name: 'caas'
-};
-
-var assert = require('assert'),
-    debug = require('debug')('box:cert/caas.js');
-
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('getCertificate: using fallback certificate', hostname);
-
-    return callback(null, '', '');
-}

src/cert/fallback.js

@@ -1,22 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    getCertificate: getCertificate,
-
-    // testing
-    _name: 'fallback'
-};
-
-var assert = require('assert'),
-    debug = require('debug')('box:cert/fallback.js');
-
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('getCertificate: using fallback certificate', hostname);
-
-    return callback(null, '', '');
-}

src/cert/interface.js

@@ -1,24 +0,0 @@
-'use strict';
-
-// -------------------------------------------
-//  This file just describes the interface
-//
-//  New backends can start from here
-// -------------------------------------------
-
-exports = module.exports = {
-    getCertificate: getCertificate
-};
-
-var assert = require('assert'),
-    BoxError = require('../boxerror.js');
-
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    return callback(new BoxError(BoxError.NOT_IMPLEMENTED, 'getCertificate is not implemented'));
-}
-

src/cloudron.js

@@ -1,24 +1,24 @@
 'use strict';
 
 exports = module.exports = {
-    initialize: initialize,
-    uninitialize: uninitialize,
-    getConfig: getConfig,
-    getLogs: getLogs,
+    initialize,
+    uninitialize,
+    getConfig,
+    getLogs,
 
-    reboot: reboot,
-    isRebootRequired: isRebootRequired,
+    reboot,
+    isRebootRequired,
 
-    onActivated: onActivated,
+    onActivated,
 
-    prepareDashboardDomain: prepareDashboardDomain,
-    setDashboardDomain: setDashboardDomain,
-    setDashboardAndMailDomain: setDashboardAndMailDomain,
-    renewCerts: renewCerts,
+    setupDnsAndCert,
 
-    setupDashboard: setupDashboard,
+    prepareDashboardDomain,
+    setDashboardDomain,
+    updateDashboardDomain,
+    renewCerts,
 
-    runSystemChecks: runSystemChecks,
+    runSystemChecks
 };
 
 var addons = require('./addons.js'),
@@ -29,6 +29,7 @@ var addons = require('./addons.js'),
     auditSource = require('./auditsource.js'),
     backups = require('./backups.js'),
     BoxError = require('./boxerror.js'),
+    branding = require('./branding.js'),
     constants = require('./constants.js'),
     cron = require('./cron.js'),
     debug = require('debug')('box:cloudron'),
@@ -46,6 +47,7 @@ var addons = require('./addons.js'),
     shell = require('./shell.js'),
     spawn = require('child_process').spawn,
     split = require('split'),
+    sysinfo = require('./sysinfo.js'),
     tasks = require('./tasks.js'),
     users = require('./users.js');
 
@@ -66,7 +68,7 @@ function uninitialize(callback) {
 
     async.series([
         cron.stopJobs,
-        platform.stop
+        platform.stopAllTasks
     ], callback);
 }
 
@@ -78,7 +80,16 @@ function onActivated(callback) {
     // 2. the restore code path can run without sudo (since mail/ is non-root)
     async.series([
         platform.start,
-        cron.startJobs
+        cron.startJobs,
+        function checkBackupConfiguration(done) {
+            backups.checkConfiguration(function (error, message) {
+                if (error) return done(error);
+                notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, done);
+            });
+        },
+        // disable responding to api calls via IP to not leak domain info. this is carefully placed as the last item, so it buys
+        // the UI some time to query the dashboard domain in the restore code path
+        (done) => setTimeout(() => reverseProxy.writeDefaultConfig({ activated :true }, done), 30000)
     ], callback);
 }
 
@@ -103,24 +114,49 @@ function notifyUpdate(callback) {
 
 // each of these tasks can fail. we will add some routes to fix/re-run them
 function runStartupTasks() {
-    // configure nginx to be reachable by IP
-    reverseProxy.writeDefaultConfig(NOOP_CALLBACK);
+    const tasks = [
+        // stop all the systemd tasks
+        platform.stopAllTasks,
 
-    // this configures collectd to collect backup storage metrics if filesystem is used. This is also triggerd when the settings change with the rest api
-    settings.getBackupConfig(function (error, backupConfig) {
-        if (error) return console.error('Failed to read backup config.', error);
-        backups.configureCollectd(backupConfig, NOOP_CALLBACK);
-    });
+        // this configures collectd to collect backup storage metrics if filesystem is used. This is also triggerd when the settings change with the rest api
+        function (callback) {
+            settings.getBackupConfig(function (error, backupConfig) {
+                if (error) return callback(error);
 
-    // always generate webadmin config since we have no versioning mechanism for the ejs
-    if (settings.adminDomain()) reverseProxy.writeAdminConfig(settings.adminDomain(), NOOP_CALLBACK);
+                backups.configureCollectd(backupConfig, callback);
+            });
+        },
 
-    // check activation state and start the platform
-    users.isActivated(function (error, activated) {
-        if (error) return debug(error);
-        if (!activated) return debug('initialize: not activated yet'); // not activated
+        // always generate webadmin config since we have no versioning mechanism for the ejs
+        function (callback) {
+            if (!settings.adminDomain()) return callback();
 
-        onActivated(NOOP_CALLBACK);
+            reverseProxy.writeDashboardConfig(settings.adminDomain(), callback);
+        },
+
+        // check activation state and start the platform
+        function (callback) {
+            users.isActivated(function (error, activated) {
+                if (error) return callback(error);
+
+                // configure nginx to be reachable by IP when not activated. for the moment, the IP based redirect exists even after domain is setup
+                // just in case user forgot or some network error happenned in the middle (then browser refresh takes you to activation page)
+                // we remove the config as a simple security measure to not expose IP <-> domain
+                if (!activated) {
+                    debug('runStartupTasks: not activated. generating IP based redirection config');
+                    return reverseProxy.writeDefaultConfig({ activated: false }, callback);
+                }
+
+                onActivated(callback);
+            });
+        }
+    ];
+
+    // we used to run tasks in parallel but simultaneous nginx reloads was causing issues
+    async.series(async.reflectAll(tasks), function (error, results) {
+        results.forEach((result, idx) => {
+            if (result.error) debug(`Startup task at index ${idx} failed: ${result.error.message}`);
+        });
     });
 }
 
@@ -139,17 +175,18 @@ function getConfig(callback) {
             mailFqdn: settings.mailFqdn(),
             version: constants.VERSION,
             isDemo: settings.isDemo(),
-            provider: settings.provider(),
             cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-            footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
-            features: appstore.getFeatures()
+            footer: branding.renderFooter(allSettings[settings.FOOTER_KEY] || constants.FOOTER),
+            features: appstore.getFeatures(),
+            profileLocked: allSettings[settings.DIRECTORY_CONFIG_KEY].lockUserProfiles,
+            mandatory2FA: allSettings[settings.DIRECTORY_CONFIG_KEY].mandatory2FA
         });
     });
 }
 
 function reboot(callback) {
     notifications.alert(notifications.ALERT_REBOOT, 'Reboot Required', '', function (error) {
-        if (error) console.error('Failed to clear reboot notification.', error);
+        if (error) debug('reboot: failed to clear reboot notification.', error);
 
         shell.sudo('reboot', [ REBOOT_CMD ], {}, callback);
     });
@@ -167,24 +204,11 @@ function runSystemChecks(callback) {
     assert.strictEqual(typeof callback, 'function');
 
     async.parallel([
-        checkBackupConfiguration,
         checkMailStatus,
         checkRebootRequired
     ], callback);
 }
 
-function checkBackupConfiguration(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('checking backup configuration');
-
-    backups.checkConfiguration(function (error, message) {
-        if (error) return callback(error);
-
-        notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, callback);
-    });
-}
-
 function checkMailStatus(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -275,7 +299,7 @@ function prepareDashboardDomain(domain, auditSource, callback) {
             const conflict = result.filter(app => app.fqdn === fqdn);
             if (conflict.length) return callback(new BoxError(BoxError.BAD_STATE, 'Dashboard location conflicts with an existing app'));
 
-            tasks.add(tasks.TASK_PREPARE_DASHBOARD_DOMAIN, [ domain, auditSource ], function (error, taskId) {
+            tasks.add(tasks.TASK_SETUP_DNS_AND_CERT, [ constants.ADMIN_LOCATION, domain, auditSource ], function (error, taskId) {
                 if (error) return callback(error);
 
                 tasks.startTask(taskId, {}, NOOP_CALLBACK);
@@ -297,12 +321,12 @@ function setDashboardDomain(domain, auditSource, callback) {
     domains.get(domain, function (error, domainObject) {
         if (error) return callback(error);
 
-        reverseProxy.writeAdminConfig(domain, function (error) {
+        reverseProxy.writeDashboardConfig(domain, function (error) {
             if (error) return callback(error);
 
             const fqdn = domains.fqdn(constants.ADMIN_LOCATION, domainObject);
 
-            settings.setAdmin(domain, fqdn, function (error) {
+            settings.setAdminLocation(domain, fqdn, function (error) {
                 if (error) return callback(error);
 
                 eventlog.add(eventlog.ACTION_DASHBOARD_DOMAIN_UPDATE, auditSource, { domain: domain, fqdn: fqdn });
@@ -314,36 +338,24 @@ function setDashboardDomain(domain, auditSource, callback) {
 }
 
 // call this only post activation because it will restart mail server
-function setDashboardAndMailDomain(domain, auditSource, callback) {
+function updateDashboardDomain(domain, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`setDashboardAndMailDomain: ${domain}`);
+    debug(`updateDashboardDomain: ${domain}`);
 
     if (settings.isDemo()) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
 
     setDashboardDomain(domain, auditSource, function (error) {
         if (error) return callback(error);
 
-        mail.onMailFqdnChanged(NOOP_CALLBACK); // this will update dns and re-configure mail server
-        addons.restartService('turn', NOOP_CALLBACK); // to update the realm variable
+        addons.rebuildService('turn', NOOP_CALLBACK); // to update the realm variable
 
         callback(null);
     });
 }
 
-function setupDashboard(auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof callback, 'function');
-
-    async.series([
-        domains.prepareDashboardDomain.bind(null, settings.adminDomain(), auditSource, progressCallback),
-        setDashboardDomain.bind(null, settings.adminDomain(), auditSource)
-    ], callback);
-}
-
 function renewCerts(options, auditSource, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof auditSource, 'object');
@@ -357,3 +369,34 @@ function renewCerts(options, auditSource, callback) {
         callback(null, taskId);
     });
 }
+
+function setupDnsAndCert(subdomain, domain, auditSource, progressCallback, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    domains.get(domain, function (error, domainObject) {
+        if (error) return callback(error);
+
+        const adminFqdn = domains.fqdn(subdomain, domainObject);
+
+        sysinfo.getServerIp(function (error, ip) {
+            if (error) return callback(error);
+
+            async.series([
+                (done) => { progressCallback({ message: `Updating DNS of ${adminFqdn}` }); done(); },
+                domains.upsertDnsRecords.bind(null, subdomain, domain, 'A', [ ip ]),
+                (done) => { progressCallback({ message: `Waiting for DNS of ${adminFqdn}` }); done(); },
+                domains.waitForDnsRecord.bind(null, subdomain, domain, 'A', ip, { interval: 30000, times: 50000 }),
+                (done) => { progressCallback({ message: `Getting certificate of ${adminFqdn}` }); done(); },
+                reverseProxy.ensureCertificate.bind(null, domains.fqdn(subdomain, domainObject), domain, auditSource)
+            ], function (error) {
+                if (error) return callback(error);
+
+                callback(null);
+            });
+        });
+    });
+}

src/constants.js

@@ -37,18 +37,19 @@ exports = module.exports = {
     DEFAULT_MEMORY_LIMIT: (256 * 1024 * 1024), // see also client.js
 
     DEMO_USERNAME: 'cloudron',
+    DEMO_BLACKLISTED_APPS: [ 'com.github.cloudtorrent', 'net.alltubedownload.cloudronapp' ],
 
     AUTOUPDATE_PATTERN_NEVER: 'never',
 
-    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8),
+    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8), // also used in dashboard client.js
 
     CLOUDRON: CLOUDRON,
     TEST: TEST,
 
     SUPPORT_EMAIL: 'support@cloudron.io',
 
-    FOOTER: '&copy; 2020  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
+    FOOTER: '&copy; %YEAR%  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
 
-    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '4.2.0-test'
+    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '5.1.1-test'
 };
 

src/cron.js

@@ -1,16 +1,23 @@
 'use strict';
 
+// IMPORTANT: These patterns are together because they spin tasks which acquire a lock
+// If the patterns overlap all the time, then the task may not ever get a chance to run!
+// If you change this change dashboard patterns in settings.html
+const DEFAULT_CLEANUP_BACKUPS_PATTERN = '00 30 1,3,5,23 * * *',
+    DEFAULT_AUTOUPDATE_PATTERN = '00 00 1,3,5,23 * * *';
+
 exports = module.exports = {
-    startJobs: startJobs,
+    startJobs,
 
-    stopJobs: stopJobs,
+    stopJobs,
 
-    handleSettingsChanged: handleSettingsChanged
+    handleSettingsChanged,
+
+    DEFAULT_AUTOUPDATE_PATTERN,
 };
 
 var appHealthMonitor = require('./apphealthmonitor.js'),
     apps = require('./apps.js'),
-    appstore = require('./appstore.js'),
     assert = require('assert'),
     async = require('async'),
     auditSource = require('./auditsource.js'),
@@ -29,12 +36,9 @@ var appHealthMonitor = require('./apphealthmonitor.js'),
     updateChecker = require('./updatechecker.js');
 
 var gJobs = {
-    alive: null, // send periodic stats
-    appAutoUpdater: null,
-    boxAutoUpdater: null,
-    appUpdateChecker: null,
+    autoUpdater: null,
     backup: null,
-    boxUpdateChecker: null,
+    updateChecker: null,
     systemChecks: null,
     diskSpaceChecker: null,
     certificateRenew: null,
@@ -60,15 +64,11 @@ var NOOP_CALLBACK = function (error) { if (error) debug(error); };
 function startJobs(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    const randomMinute = Math.floor(60*Math.random());
-    gJobs.alive = new CronJob({
-        cronTime: '00 ' + randomMinute + ' * * * *', // every hour on a random minute
-        onTick: appstore.sendAliveStatus,
-        start: true
-    });
+    debug('startJobs: starting cron jobs');
 
+    const randomTick = Math.floor(60*Math.random());
     gJobs.systemChecks = new CronJob({
-        cronTime: '00 30 * * * *', // every 30 minutes. if you change this interval, change the notification messages with correct duration
+        cronTime: '00 30 2 * * *', // once a day. if you change this interval, change the notification messages with correct duration
         onTick: () => cloudron.runSystemChecks(NOOP_CALLBACK),
         start: true
     });
@@ -79,15 +79,10 @@ function startJobs(callback) {
         start: true
     });
 
-    gJobs.boxUpdateCheckerJob = new CronJob({
-        cronTime: '00 ' + randomMinute + ' * * * *', // once an hour
-        onTick: () => updateChecker.checkBoxUpdates(NOOP_CALLBACK),
-        start: true
-    });
-
-    gJobs.appUpdateChecker = new CronJob({
-        cronTime: '00 ' + randomMinute + ' * * * *', // once an hour
-        onTick: () => updateChecker.checkAppUpdates(NOOP_CALLBACK),
+    // this is run separately from the update itself so that the user can disable automatic updates but can still get a notification
+    gJobs.updateCheckerJob = new CronJob({
+        cronTime: `${randomTick} ${randomTick} 1,5,9,13,17,21,23 * * *`,
+        onTick: () => updateChecker.checkForUpdates({ automatic: true }, NOOP_CALLBACK),
         start: true
     });
 
@@ -98,7 +93,7 @@ function startJobs(callback) {
     });
 
     gJobs.cleanupBackups = new CronJob({
-        cronTime: '00 45 1,3,5,23 * * *', // every 6 hours. try not to overlap with ensureBackup job
+        cronTime: DEFAULT_CLEANUP_BACKUPS_PATTERN,
         onTick: backups.startCleanupTask.bind(null, auditSource.CRON, NOOP_CALLBACK),
         start: true
     });
@@ -138,8 +133,7 @@ function startJobs(callback) {
 
         const tz = allSettings[settings.TIME_ZONE_KEY];
         backupConfigChanged(allSettings[settings.BACKUP_CONFIG_KEY], tz);
-        appAutoupdatePatternChanged(allSettings[settings.APP_AUTOUPDATE_PATTERN_KEY], tz);
-        boxAutoupdatePatternChanged(allSettings[settings.BOX_AUTOUPDATE_PATTERN_KEY], tz);
+        autoupdatePatternChanged(allSettings[settings.AUTOUPDATE_PATTERN_KEY], tz);
         dynamicDnsChanged(allSettings[settings.DYNAMIC_DNS_KEY]);
 
         callback();
@@ -154,8 +148,7 @@ function handleSettingsChanged(key, value) {
     switch (key) {
     case settings.TIME_ZONE_KEY:
     case settings.BACKUP_CONFIG_KEY:
-    case settings.APP_AUTOUPDATE_PATTERN_KEY:
-    case settings.BOX_AUTOUPDATE_PATTERN_KEY:
+    case settings.AUTOUPDATE_PATTERN_KEY:
     case settings.DYNAMIC_DNS_KEY:
         debug('handleSettingsChanged: recreating all jobs');
         async.series([
@@ -172,71 +165,47 @@ function backupConfigChanged(value, tz) {
     assert.strictEqual(typeof value, 'object');
     assert.strictEqual(typeof tz, 'string');
 
-    debug(`backupConfigChanged: interval ${value.intervalSecs} (${tz})`);
+    debug(`backupConfigChanged: schedule ${value.schedulePattern} (${tz})`);
 
     if (gJobs.backup) gJobs.backup.stop();
-    let pattern;
-    if (value.intervalSecs <= 6 * 60 * 60) {
-        pattern = '00 00 1,7,13,19 * * *'; // no option but to backup in the middle of the day
-    } else {
-        pattern = '00 00 1,3,5,23 * * *'; // avoid middle of the day backups
-    }
 
     gJobs.backup = new CronJob({
-        cronTime: pattern,
-        onTick: backups.ensureBackup.bind(null, auditSource.CRON, NOOP_CALLBACK),
+        cronTime: value.schedulePattern,
+        onTick: backups.startBackupTask.bind(null, auditSource.CRON, NOOP_CALLBACK),
         start: true,
         timeZone: tz
     });
 }
 
-function boxAutoupdatePatternChanged(pattern, tz) {
+function autoupdatePatternChanged(pattern, tz) {
     assert.strictEqual(typeof pattern, 'string');
     assert.strictEqual(typeof tz, 'string');
 
-    debug(`boxAutoupdatePatternChanged: pattern - ${pattern} (${tz})`);
+    debug(`autoupdatePatternChanged: pattern - ${pattern} (${tz})`);
 
-    if (gJobs.boxAutoUpdater) gJobs.boxAutoUpdater.stop();
+    if (gJobs.autoUpdater) gJobs.autoUpdater.stop();
 
     if (pattern === constants.AUTOUPDATE_PATTERN_NEVER) return;
 
-    gJobs.boxAutoUpdater = new CronJob({
+    gJobs.autoUpdater = new CronJob({
         cronTime: pattern,
         onTick: function() {
-            var updateInfo = updateChecker.getUpdateInfo();
-            if (updateInfo.box) {
-                debug('Starting autoupdate to %j', updateInfo.box);
+            const updateInfo = updateChecker.getUpdateInfo();
+            // do box before app updates. for the off chance that the box logic fixes some app update logic issue
+            if (updateInfo.box && !updateInfo.box.unstable) {
+                debug('Starting box autoupdate to %j', updateInfo.box);
                 updater.updateToLatest({ skipBackup: false }, auditSource.CRON, NOOP_CALLBACK);
-            } else {
-                debug('No box auto updates available');
+                return;
             }
-        },
-        start: true,
-        timeZone: tz
-    });
-}
 
-function appAutoupdatePatternChanged(pattern, tz) {
-    assert.strictEqual(typeof pattern, 'string');
-    assert.strictEqual(typeof tz, 'string');
-
-    debug(`appAutoupdatePatternChanged: pattern ${pattern} (${tz})`);
-
-    if (gJobs.appAutoUpdater) gJobs.appAutoUpdater.stop();
-
-    if (pattern === constants.AUTOUPDATE_PATTERN_NEVER) return;
-
-    gJobs.appAutoUpdater = new CronJob({
-        cronTime: pattern,
-        onTick: function() {
-            var updateInfo = updateChecker.getUpdateInfo();
-            if (updateInfo.apps) {
+            if (updateInfo.apps && Object.keys(updateInfo.apps).length > 0) {
                 debug('Starting app update to %j', updateInfo.apps);
                 apps.autoupdateApps(updateInfo.apps, auditSource.CRON, NOOP_CALLBACK);
             } else {
                 debug('No app auto updates available');
             }
         },
+
         start: true,
         timeZone: tz
     });

src/database.js

@@ -17,12 +17,12 @@ var assert = require('assert'),
     BoxError = require('./boxerror.js'),
     child_process = require('child_process'),
     constants = require('./constants.js'),
+    debug = require('debug')('box:database'),
     mysql = require('mysql'),
     once = require('once'),
     util = require('util');
 
-var gConnectionPool = null,
-    gDefaultConnection = null;
+var gConnectionPool = null;
 
 const gDatabase = {
     hostname: '127.0.0.1',
@@ -42,59 +42,37 @@ function initialize(callback) {
         gDatabase.hostname = require('child_process').execSync('docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" mysql-server').toString().trim();
     }
 
+    // https://github.com/mysqljs/mysql#pool-options
     gConnectionPool  = mysql.createPool({
-        connectionLimit: 5, // this has to be > 1 since we store one connection as 'default'. the rest for transactions
+        connectionLimit: 5,
         host: gDatabase.hostname,
         user: gDatabase.username,
         password: gDatabase.password,
         port: gDatabase.port,
         database: gDatabase.name,
         multipleStatements: false,
+        waitForConnections: true, // getConnection() will wait until a connection is avaiable
         ssl: false,
         timezone: 'Z' // mysql follows the SYSTEM timezone. on Cloudron, this is UTC
     });
 
     gConnectionPool.on('connection', function (connection) {
+        // connection objects are re-used. so we have to attach to the event here (once) to prevent crash
+        // note the pool also has an 'acquire' event but that is called whenever we do a getConnection()
+        connection.on('error', (error) => debug(`Connection ${connection.threadId} error: ${error.message} ${error.code}`));
+
         connection.query('USE ' + gDatabase.name);
         connection.query('SET SESSION sql_mode = \'strict_all_tables\'');
     });
 
-    reconnect(callback);
+    callback(null);
 }
 
 function uninitialize(callback) {
-    if (gConnectionPool) {
-        gConnectionPool.end(callback);
-        gConnectionPool = null;
-    } else {
-        callback(null);
-    }
-}
+    if (!gConnectionPool) return callback(null);
 
-function reconnect(callback) {
-    callback = callback ? once(callback) : function () {};
-
-    gConnectionPool.getConnection(function (error, connection) {
-        if (error) {
-            console.error('Unable to reestablish connection to database. Try again in a bit.', error.message);
-            return setTimeout(reconnect.bind(null, callback), 1000);
-        }
-
-        connection.on('error', function (error) {
-            // by design, we catch all normal errors by providing callbacks.
-            // this function should be invoked only when we have no callbacks pending and we have a fatal error
-            assert(error.fatal, 'Non-fatal error on connection object');
-
-            console.error('Unhandled mysql connection error.', error);
-
-            // This is most likely an issue an can cause double callbacks from reconnect()
-            setTimeout(reconnect.bind(null, callback), 1000);
-        });
-
-        gDefaultConnection = connection;
-
-        callback(null);
-    });
+    gConnectionPool.end(callback);
+    gConnectionPool = null;
 }
 
 function clear(callback) {
@@ -107,80 +85,43 @@ function clear(callback) {
     child_process.exec(cmd, callback);
 }
 
-function beginTransaction(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    if (gConnectionPool === null) return callback(new BoxError(BoxError.DATABASE_ERROR, 'No database connection pool.'));
-
-    gConnectionPool.getConnection(function (error, connection) {
-        if (error) {
-            console.error('Unable to get connection to database. Try again in a bit.', error.message);
-            return setTimeout(beginTransaction.bind(null, callback), 1000);
-        }
-
-        connection.beginTransaction(function (error) {
-            if (error) return callback(error);
-
-            return callback(null, connection);
-        });
-    });
-}
-
-function rollback(connection, callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    connection.rollback(function (error) {
-        if (error) console.error(error); // can this happen?
-
-        connection.release();
-        callback(null);
-    });
-}
-
-// FIXME: if commit fails, is it supposed to return an error ?
-function commit(connection, callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    connection.commit(function (error) {
-        if (error) return rollback(connection, callback);
-
-        connection.release();
-        return callback(null);
-    });
-}
-
 function query() {
-    var args = Array.prototype.slice.call(arguments);
-    var callback = args[args.length - 1];
+    const args = Array.prototype.slice.call(arguments);
+    const callback = args[args.length - 1];
     assert.strictEqual(typeof callback, 'function');
 
-    if (gDefaultConnection === null) return callback(new BoxError(BoxError.DATABASE_ERROR, 'No connection to database'));
+    if (constants.TEST && !gConnectionPool) return callback(new BoxError(BoxError.DATABASE_ERROR, 'database.js not initialized'));
 
-    args[args.length -1 ] = function (error, result) {
-        if (error && error.fatal) {
-            gDefaultConnection = null;
-            setTimeout(reconnect, 1000);
-        }
-
-        callback(error, result);
-    };
-
-    gDefaultConnection.query.apply(gDefaultConnection, args);
+    gConnectionPool.query.apply(gConnectionPool, args); // this is same as getConnection/query/release
 }
 
 function transaction(queries, callback) {
     assert(util.isArray(queries));
     assert.strictEqual(typeof callback, 'function');
 
-    beginTransaction(function (error, conn) {
+    callback = once(callback);
+
+    gConnectionPool.getConnection(function (error, connection) {
         if (error) return callback(error);
 
-        async.mapSeries(queries, function iterator(query, done) {
-            conn.query(query.query, query.args, done);
-        }, function seriesDone(error, results) {
-            if (error) return rollback(conn, callback.bind(null, error));
+        const releaseConnection = (error) => { connection.release(); callback(error); };
 
-            commit(conn, callback.bind(null, null, results));
+        connection.beginTransaction(function (error) {
+            if (error) return releaseConnection(error);
+
+            async.mapSeries(queries, function iterator(query, done) {
+                connection.query(query.query, query.args, done);
+            }, function seriesDone(error, results) {
+                if (error) return connection.rollback(() => releaseConnection(error));
+
+                connection.commit(function (error) {
+                    if (error) return connection.rollback(() => releaseConnection(error));
+
+                    connection.release();
+
+                    callback(null, results);
+                });
+            });
         });
     });
 }
@@ -203,7 +144,7 @@ function exportToFile(file, callback) {
 
     // latest mysqldump enables column stats by default which is not present in MySQL 5.7 server
     // this option must not be set in production cloudrons which still use the old mysqldump
-    const disableColStats = (constants.TEST && process.env.DESKTOP_SESSION !== 'ubuntu') ? '--column-statistics=0' : '';
+    const disableColStats = (constants.TEST && require('fs').readFileSync('/etc/lsb-release', 'utf-8').includes('20.04')) ? '--column-statistics=0' : '';
 
     var cmd = `/usr/bin/mysqldump -h "${gDatabase.hostname}" -u root -p${gDatabase.password} ${disableColStats} --single-transaction --routines --triggers ${gDatabase.name} > "${file}"`;
 

src/dns/caas.js

@@ -1,176 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    removePrivateFields: removePrivateFields,
-    injectPrivateFields: injectPrivateFields,
-    upsert: upsert,
-    get: get,
-    del: del,
-    wait: wait,
-    verifyDnsConfig: verifyDnsConfig
-};
-
-var assert = require('assert'),
-    BoxError = require('../boxerror.js'),
-    debug = require('debug')('box:dns/caas'),
-    domains = require('../domains.js'),
-    settings = require('../settings.js'),
-    superagent = require('superagent'),
-    util = require('util'),
-    waitForDns = require('./waitfordns.js');
-
-function formatError(response) {
-    return util.format('Caas DNS error [%s] %j', response.statusCode, response.body);
-}
-
-function getFqdn(location, domain) {
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof domain, 'string');
-
-    return (location === '') ? domain : location + '-' + domain;
-}
-
-function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
-
-    // do not return the 'key'. in caas, this is private
-    delete domainObject.fallbackCertificate.key;
-
-    return domainObject;
-}
-
-function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
-}
-
-function upsert(domainObject, location, type, values, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert(util.isArray(values));
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-
-    let fqdn = location !== '' && type === 'TXT' ? location + '.' + domainObject.domain : getFqdn(location, domainObject.domain);
-
-    debug('add: %s for zone %s of type %s with values %j', location, domainObject.domain, type, values);
-
-    var data = {
-        type: type,
-        values: values
-    };
-
-    superagent
-        .post(settings.apiServerOrigin() + '/api/v1/caas/domains/' + fqdn)
-        .query({ token: dnsConfig.token })
-        .send(data)
-        .timeout(30 * 1000)
-        .end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode === 400) return callback(new BoxError(BoxError.BAD_FIELD, result.body.message));
-            if (result.statusCode === 420) return callback(new BoxError(BoxError.BUSY));
-            if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
-
-            return callback(null);
-        });
-}
-
-function get(domainObject, location, type, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-    const fqdn = location !== '' && type === 'TXT' ? location + '.' + domainObject.domain : getFqdn(location, domainObject.domain);
-
-    debug('get: zoneName: %s subdomain: %s type: %s fqdn: %s', domainObject.domain, location, type, fqdn);
-
-    superagent
-        .get(settings.apiServerOrigin() + '/api/v1/caas/domains/' + fqdn)
-        .query({ token: dnsConfig.token, type: type })
-        .timeout(30 * 1000)
-        .end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
-
-            return callback(null, result.body.values);
-        });
-}
-
-function del(domainObject, location, type, values, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert(util.isArray(values));
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-    debug('del: %s for zone %s of type %s with values %j', location, domainObject.domain, type, values);
-
-    var data = {
-        type: type,
-        values: values
-    };
-
-    superagent
-        .del(settings.apiServerOrigin() + '/api/v1/caas/domains/' + getFqdn(location, domainObject.domain))
-        .query({ token: dnsConfig.token })
-        .send(data)
-        .timeout(30 * 1000)
-        .end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode === 400) return callback(new BoxError(BoxError.BAD_FIELD, result.body.message));
-            if (result.statusCode === 420) return callback(new BoxError(BoxError.BUSY));
-            if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
-            if (result.statusCode !== 204) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
-
-            return callback(null);
-        });
-}
-
-function wait(domainObject, location, type, value, options, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert.strictEqual(typeof value, 'string');
-    assert(options && typeof options === 'object'); // { interval: 5000, times: 50000 }
-    assert.strictEqual(typeof callback, 'function');
-
-    const fqdn = domains.fqdn(location, domainObject);
-
-    waitForDns(fqdn, domainObject.zoneName, type, value, options, callback);
-}
-
-function verifyDnsConfig(domainObject, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-
-    if (!dnsConfig.token || typeof dnsConfig.token !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'token must be a non-empty string', { field: 'token' }));
-
-    const ip = '127.0.0.1';
-
-    var credentials = {
-        token: dnsConfig.token,
-        hyphenatedSubdomains: true  // this will ensure we always use them, regardless of passed-in configs
-    };
-
-    const location = 'cloudrontestdns';
-
-    upsert(domainObject, location, 'A', [ ip ], function (error) {
-        if (error) return callback(error);
-
-        debug('verifyDnsConfig: Test A record added');
-
-        del(domainObject, location, 'A', [ ip ], function (error) {
-            if (error) return callback(error);
-
-            debug('verifyDnsConfig: Test A record removed again');
-
-            callback(null, credentials);
-        });
-    });
-}

src/dns/cloudflare.js

@@ -13,6 +13,7 @@ exports = module.exports = {
 var assert = require('assert'),
     async = require('async'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/cloudflare'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -25,12 +26,12 @@ var assert = require('assert'),
 var CLOUDFLARE_ENDPOINT = 'https://api.cloudflare.com/client/v4';
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function translateRequestError(result, callback) {
@@ -39,9 +40,14 @@ function translateRequestError(result, callback) {
 
     if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND, util.format('%s %j', result.statusCode, 'API does not exist')));
     if (result.statusCode === 422) return callback(new BoxError(BoxError.BAD_FIELD, result.body.message));
-    if ((result.statusCode === 400 || result.statusCode === 401 || result.statusCode === 403) && result.body.errors.length > 0) {
-        let error = result.body.errors[0];
-        let message = `message: ${error.message} statusCode: ${result.statusCode} code:${error.code}`;
+    if (result.statusCode === 400 || result.statusCode === 401 || result.statusCode === 403) {
+        let message = 'Unknown error';
+        if (typeof result.body.error === 'string') {
+            message = `message: ${result.body.error} statusCode: ${result.statusCode}`;
+        } else if (Array.isArray(result.body.errors) && result.body.errors.length > 0) {
+            let error = result.body.errors[0];
+            message = `message: ${error.message} statusCode: ${result.statusCode} code:${error.code}`;
+        }
         return callback(new BoxError(BoxError.ACCESS_DENIED, message));
     }
 
@@ -284,7 +290,7 @@ function verifyDnsConfig(domainObject, callback) {
     if (dnsConfig.tokenType !== 'GlobalApiKey' && dnsConfig.tokenType !== 'ApiToken')  return callback(new BoxError(BoxError.BAD_FIELD, 'tokenType is required', { field: 'tokenType' }));
 
     if (dnsConfig.tokenType === 'GlobalApiKey') {
-        if ('email' in dnsConfig && typeof dnsConfig.email !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'email must be a non-empty string', { field: 'email' }));
+        if (typeof dnsConfig.email !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'email must be a non-empty string', { field: 'email' }));
     }
 
     const ip = '127.0.0.1';

src/dns/digitalocean.js

@@ -13,6 +13,7 @@ exports = module.exports = {
 var assert = require('assert'),
     async = require('async'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/digitalocean'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -28,12 +29,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function getInternal(dnsConfig, zoneName, name, type, callback) {

src/dns/gandi.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/gandi'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -26,12 +27,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function upsert(domainObject, location, type, values, callback) {

src/dns/gcdns.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/gcdns'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -21,12 +22,12 @@ var assert = require('assert'),
     _ = require('underscore');
 
 function removePrivateFields(domainObject) {
-    domainObject.config.credentials.private_key = domains.SECRET_PLACEHOLDER;
+    domainObject.config.credentials.private_key = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.credentials.private_key === domains.SECRET_PLACEHOLDER && currentConfig.credentials) newConfig.credentials.private_key = currentConfig.credentials.private_key;
+    if (newConfig.credentials.private_key === constants.SECRET_PLACEHOLDER && currentConfig.credentials) newConfig.credentials.private_key = currentConfig.credentials.private_key;
 }
 
 function getDnsCredentials(dnsConfig) {

src/dns/godaddy.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/godaddy'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -32,12 +33,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.apiSecret = domains.SECRET_PLACEHOLDER;
+    domainObject.config.apiSecret = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.apiSecret === domains.SECRET_PLACEHOLDER) newConfig.apiSecret = currentConfig.apiSecret;
+    if (newConfig.apiSecret === constants.SECRET_PLACEHOLDER) newConfig.apiSecret = currentConfig.apiSecret;
 }
 
 function upsert(domainObject, location, type, values, callback) {

src/dns/interface.js

@@ -21,13 +21,13 @@ var assert = require('assert'),
     util = require('util');
 
 function removePrivateFields(domainObject) {
-    // in-place removal of tokens and api keys with domains.SECRET_PLACEHOLDER
+    // in-place removal of tokens and api keys with constants.SECRET_PLACEHOLDER
     return domainObject;
 }
 
 // eslint-disable-next-line no-unused-vars
 function injectPrivateFields(newConfig, currentConfig) {
-    // in-place injection of tokens and api keys which came in with domains.SECRET_PLACEHOLDER
+    // in-place injection of tokens and api keys which came in with constants.SECRET_PLACEHOLDER
 }
 
 function upsert(domainObject, location, type, values, callback) {

src/dns/linode.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 let async = require('async'),
     assert = require('assert'),
+    constants = require('../constants.js'),
     BoxError = require('../boxerror.js'),
     debug = require('debug')('box:dns/linode'),
     dns = require('../native-dns.js'),
@@ -27,12 +28,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function getZoneId(dnsConfig, zoneName, callback) {

src/dns/namecheap.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/namecheap'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -25,12 +26,12 @@ var assert = require('assert'),
 const ENDPOINT = 'https://api.namecheap.com/xml.response';
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function getQuery(dnsConfig, callback) {
@@ -77,13 +78,11 @@ function getInternal(dnsConfig, zoneName, subdomain, type, callback) {
 
                     return callback(new BoxError(BoxError.EXTERNAL_ERROR, errorMessage));
                 }
-                if (!tmp.CommandResponse[0]) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response'));
-                if (!tmp.CommandResponse[0].DomainDNSGetHostsResult[0]) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response'));
-
-                var hosts = result.ApiResponse.CommandResponse[0].DomainDNSGetHostsResult[0].host.map(function (h) {
-                    return h['$'];
-                });
+                const host = safe.query(tmp, 'CommandResponse[0].DomainDNSGetHostsResult[0].host');
+                if (!host) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Invalid response: ${JSON.stringify(tmp)}`));
+                if (!Array.isArray(host)) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `host is not an array: ${JSON.stringify(tmp)}`));
 
+                const hosts = host.map(h => h['$']);
                 callback(null, hosts);
             });
         });

src/dns/namecom.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/namecom'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -27,12 +28,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function addRecord(dnsConfig, zoneName, name, type, values, callback) {
@@ -54,6 +55,10 @@ function addRecord(dnsConfig, zoneName, name, type, values, callback) {
     if (type === 'MX') {
         data.priority = parseInt(values[0].split(' ')[0], 10);
         data.answer = values[0].split(' ')[1];
+    } else if (type === 'TXT') {
+        // we have to strip the quoting for some odd reason for name.com! If you change that also change updateRecord
+        let tmp = values[0];
+        data.answer = tmp.indexOf('"') === 0 && tmp.lastIndexOf('"') === tmp.length-1 ? tmp.slice(1, tmp.length-1) : tmp;
     } else {
         data.answer = values[0];
     }
@@ -91,6 +96,10 @@ function updateRecord(dnsConfig, zoneName, recordId, name, type, values, callbac
     if (type === 'MX') {
         data.priority = parseInt(values[0].split(' ')[0], 10);
         data.answer = values[0].split(' ')[1];
+    } else if (type === 'TXT') {
+        // we have to strip the quoting for some odd reason for name.com! If you change that also change addRecord
+        let tmp = values[0];
+        data.answer = tmp.indexOf('"') === 0 && tmp.lastIndexOf('"') === tmp.length-1 ? tmp.slice(1, tmp.length-1) : tmp;
     } else {
         data.answer = values[0];
     }

src/dns/route53.js

@@ -13,6 +13,7 @@ exports = module.exports = {
 var assert = require('assert'),
     AWS = require('aws-sdk'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/route53'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -21,12 +22,12 @@ var assert = require('assert'),
     _ = require('underscore');
 
 function removePrivateFields(domainObject) {
-    domainObject.config.secretAccessKey = domains.SECRET_PLACEHOLDER;
+    domainObject.config.secretAccessKey = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.secretAccessKey === domains.SECRET_PLACEHOLDER) newConfig.secretAccessKey = currentConfig.secretAccessKey;
+    if (newConfig.secretAccessKey === constants.SECRET_PLACEHOLDER) newConfig.secretAccessKey = currentConfig.secretAccessKey;
 }
 
 function getDnsCredentials(dnsConfig) {
@@ -280,13 +281,14 @@ function verifyDnsConfig(domainObject, callback) {
             }
 
             const location = 'cloudrontestdns';
+            const newDomainObject = Object.assign({ }, domainObject, { config: credentials });
 
-            upsert(domainObject, location, 'A', [ ip ], function (error) {
+            upsert(newDomainObject, location, 'A', [ ip ], function (error) {
                 if (error) return callback(error);
 
                 debug('verifyDnsConfig: Test A record added');
 
-                del(domainObject, location, 'A', [ ip ], function (error) {
+                del(newDomainObject, location, 'A', [ ip ], function (error) {
                     if (error) return callback(error);
 
                     debug('verifyDnsConfig: Test A record removed again');

src/docker.js

@@ -6,8 +6,6 @@ exports = module.exports = {
     injectPrivateFields: injectPrivateFields,
     removePrivateFields: removePrivateFields,
 
-    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8),
-
     ping: ping,
 
     info: info,
@@ -19,7 +17,6 @@ exports = module.exports = {
     stopContainerByName: stopContainer,
     stopContainers: stopContainers,
     deleteContainer: deleteContainer,
-    deleteContainerByName: deleteContainer,
     deleteImage: deleteImage,
     deleteContainers: deleteContainers,
     createSubcontainer: createSubcontainer,
@@ -47,6 +44,7 @@ var addons = require('./addons.js'),
     shell = require('./shell.js'),
     safe = require('safetydance'),
     util = require('util'),
+    volumes = require('./volumes.js'),
     _ = require('underscore');
 
 const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
@@ -55,12 +53,6 @@ const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
 const DOCKER_SOCKET_PATH = '/var/run/docker.sock';
 const gConnection = new Docker({ socketPath: DOCKER_SOCKET_PATH });
 
-function debugApp(app) {
-    assert(typeof app === 'object');
-
-    debug(app.fqdn + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)));
-}
-
 function testRegistryConfig(auth, callback) {
     assert.strictEqual(typeof auth, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -73,13 +65,13 @@ function testRegistryConfig(auth, callback) {
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.password === exports.SECRET_PLACEHOLDER) newConfig.password = currentConfig.password;
+    if (newConfig.password === constants.SECRET_PLACEHOLDER) newConfig.password = currentConfig.password;
 }
 
 function removePrivateFields(registryConfig) {
     assert.strictEqual(typeof registryConfig, 'object');
 
-    if (registryConfig.password) registryConfig.password = exports.SECRET_PLACEHOLDER;
+    if (registryConfig.password) registryConfig.password = constants.SECRET_PLACEHOLDER;
 
     return registryConfig;
 }
@@ -179,13 +171,41 @@ function downloadImage(manifest, callback) {
 
     debug('downloadImage %s', manifest.dockerImage);
 
-    var attempt = 1;
+    const image = gConnection.getImage(manifest.dockerImage);
 
-    async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
-        debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+    image.inspect(function (error, result) {
+        if (!error && result) return callback(null); // image is already present locally
 
-        pullImage(manifest, retryCallback);
-    }, callback);
+        let attempt = 1;
+
+        async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
+            debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+
+            pullImage(manifest, retryCallback);
+        }, callback);
+    });
+}
+
+function getBinds(app, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (app.mounts.length === 0) return callback(null);
+
+    let binds = [];
+
+    volumes.list(function (error, result) {
+        if (error) return callback(error);
+        let volumesById = {};
+        result.forEach(r => volumesById[r.id] = r);
+
+        for (const mount of app.mounts) {
+            const volume = volumesById[mount.volumeId];
+            binds.push(`${volume.hostPath}:/media/${volume.name}:${mount.readOnly ? 'ro' : 'rw'}`);
+        }
+
+        callback(null, binds);
+    });
 }
 
 function createSubcontainer(app, name, cmd, options, callback) {
@@ -195,12 +215,11 @@ function createSubcontainer(app, name, cmd, options, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let isAppContainer = !cmd; // non app-containers are like scheduler and exec (terminal) containers
+    let isAppContainer = !cmd; // non app-containers are like scheduler
 
     var manifest = app.manifest;
     var exposedPorts = {}, dockerPortBindings = { };
     var domain = app.fqdn;
-    const hostname = isAppContainer ? app.id : name;
 
     const envPrefix = manifest.manifestVersion <= 1 ? '' : 'CLOUDRON_';
 
@@ -252,79 +271,98 @@ function createSubcontainer(app, name, cmd, options, callback) {
     addons.getEnvironment(app, function (error, addonEnv) {
         if (error) return callback(error);
 
-        // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
-        // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
-        // name to look up the internal docker ip. this makes curl from within container fail
-        // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
-        // Hostname cannot be set with container NetworkMode
-        var containerOptions = {
-            name: name, // for referencing containers
-            Tty: isAppContainer,
-            Hostname: hostname,
-            Image: app.manifest.dockerImage,
-            Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
-            Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
-            ExposedPorts: isAppContainer ? exposedPorts : { },
-            Volumes: { // see also ReadonlyRootfs
-                '/tmp': {},
-                '/run': {}
-            },
-            Labels: {
-                'fqdn': app.fqdn,
-                'appId': app.id,
-                'isSubcontainer': String(!isAppContainer),
-                'isCloudronManaged': String(true)
-            },
-            HostConfig: {
-                Mounts: addons.getMountsSync(app, app.manifest.addons),
-                LogConfig: {
-                    Type: 'syslog',
-                    Config: {
-                        'tag': app.id,
-                        'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
-                        'syslog-format': 'rfc5424'
-                    }
+        getBinds(app, function (error, binds) {
+            if (error) return callback(error);
+
+            let containerOptions = {
+                name: name, // for referencing containers
+                Tty: isAppContainer,
+                Image: app.manifest.dockerImage,
+                Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
+                Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
+                ExposedPorts: isAppContainer ? exposedPorts : { },
+                Volumes: { // see also ReadonlyRootfs
+                    '/tmp': {},
+                    '/run': {}
                 },
-                Memory: memoryLimit / 2,
-                MemorySwap: memoryLimit, // Memory + Swap
-                PortBindings: isAppContainer ? dockerPortBindings : { },
-                PublishAllPorts: false,
-                ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
-                RestartPolicy: {
-                    'Name': isAppContainer ? 'unless-stopped' : 'no',
-                    'MaximumRetryCount': 0
+                Labels: {
+                    'fqdn': app.fqdn,
+                    'appId': app.id,
+                    'isSubcontainer': String(!isAppContainer),
+                    'isCloudronManaged': String(true)
                 },
-                CpuShares: app.cpuShares,
-                VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
-                NetworkMode: 'cloudron', // user defined bridge network
-                Dns: ['172.18.0.1'], // use internal dns
-                DnsSearch: ['.'], // use internal dns
-                SecurityOpt: [ 'apparmor=docker-cloudron-app' ]
-            },
-            NetworkingConfig: {
-                EndpointsConfig: {
-                    cloudron: {
-                        Aliases: [ name ] // this allows sub-containers reach app containers by name
-                    }
+                HostConfig: {
+                    Mounts: addons.getMountsSync(app, app.manifest.addons),
+                    Binds: binds, // ideally, we have to use 'Mounts' but we have to create volumes then
+                    LogConfig: {
+                        Type: 'syslog',
+                        Config: {
+                            'tag': app.id,
+                            'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
+                            'syslog-format': 'rfc5424'
+                        }
+                    },
+                    Memory: memoryLimit / 2,
+                    MemorySwap: memoryLimit, // Memory + Swap
+                    PortBindings: isAppContainer ? dockerPortBindings : { },
+                    PublishAllPorts: false,
+                    ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
+                    RestartPolicy: {
+                        'Name': isAppContainer ? 'unless-stopped' : 'no',
+                        'MaximumRetryCount': 0
+                    },
+                    CpuShares: app.cpuShares,
+                    VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
+                    SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
+                    CapAdd: [],
+                    CapDrop: []
                 }
+            };
+
+            // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
+            // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
+            // name to look up the internal docker ip. this makes curl from within container fail
+            // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
+            // Hostname cannot be set with container NetworkMode. Subcontainers run is the network space of the app container
+            // This is done to prevent lots of up/down events and iptables locking
+            if (isAppContainer) {
+                containerOptions.Hostname = app.id;
+                containerOptions.HostConfig.NetworkMode = 'cloudron'; // user defined bridge network
+                containerOptions.HostConfig.Dns = ['172.18.0.1']; // use internal dns
+                containerOptions.HostConfig.DnsSearch = ['.']; // use internal dns
+
+                containerOptions.NetworkingConfig = {
+                    EndpointsConfig: {
+                        cloudron: {
+                            Aliases: [ name ] // adds hostname entry with container name
+                        }
+                    }
+                };
+            } else {
+                containerOptions.HostConfig.NetworkMode = `container:${app.containerId}`;
             }
-        };
 
-        var capabilities = manifest.capabilities || [];
-        if (capabilities.includes('net_admin')) {
-            containerOptions.HostConfig.CapAdd = [
-                'NET_ADMIN'
-            ];
-        }
+            var capabilities = manifest.capabilities || [];
 
-        containerOptions = _.extend(containerOptions, options);
+            // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
+            if (capabilities.includes('net_admin')) containerOptions.HostConfig.CapAdd.push('NET_ADMIN', 'NET_RAW');
+            if (capabilities.includes('mlock')) containerOptions.HostConfig.CapAdd.push('IPC_LOCK'); // mlock prevents swapping
+            if (!capabilities.includes('ping')) containerOptions.HostConfig.CapDrop.push('NET_RAW'); // NET_RAW is included by default by Docker
 
-        debugApp(app, 'Creating container for %s', app.manifest.dockerImage);
+            if (capabilities.includes('vaapi') && safe.fs.existsSync('/dev/dri')) {
+                containerOptions.HostConfig.Devices = [
+                    { PathOnHost: '/dev/dri', PathInContainer: '/dev/dri', CgroupPermissions: 'rwm' }
+                ];
+            }
 
-        gConnection.createContainer(containerOptions, function (error, container) {
-            if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+            containerOptions = _.extend(containerOptions, options);
 
-            callback(null, container);
+            gConnection.createContainer(containerOptions, function (error, container) {
+                if (error && error.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
+                if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+
+                callback(null, container);
+            });
         });
     });
 }
@@ -338,7 +376,6 @@ function startContainer(containerId, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     var container = gConnection.getContainer(containerId);
-    debug('Starting container %s', containerId);
 
     container.start(function (error) {
         if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
@@ -354,7 +391,6 @@ function restartContainer(containerId, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     var container = gConnection.getContainer(containerId);
-    debug('Restarting container %s', containerId);
 
     container.restart(function (error) {
         if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
@@ -375,7 +411,6 @@ function stopContainer(containerId, callback) {
     }
 
     var container = gConnection.getContainer(containerId);
-    debug('Stopping container %s', containerId);
 
     var options = {
         t: 10 // wait for 10 seconds before killing it
@@ -384,24 +419,18 @@ function stopContainer(containerId, callback) {
     container.stop(options, function (error) {
         if (error && (error.statusCode !== 304 && error.statusCode !== 404)) return callback(new BoxError(BoxError.DOCKER_ERROR, 'Error stopping container:' + error.message));
 
-        debug('Waiting for container ' + containerId);
-
-        container.wait(function (error, data) {
+        container.wait(function (error/*, data */) {
             if (error && (error.statusCode !== 304 && error.statusCode !== 404)) return callback(new BoxError(BoxError.DOCKER_ERROR, 'Error waiting on container:' + error.message));
 
-            debug('Container %s stopped with status code [%s]', containerId, data ? String(data.StatusCode) : '');
-
             return callback(null);
         });
     });
 }
 
-function deleteContainer(containerId, callback) {
+function deleteContainer(containerId, callback) { // id can also be name
     assert(!containerId || typeof containerId === 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug('deleting container %s', containerId);
-
     if (containerId === null) return callback(null);
 
     var container = gConnection.getContainer(containerId);
@@ -428,8 +457,6 @@ function deleteContainers(appId, options, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debug('deleting containers of %s', appId);
-
     let labels = [ 'appId=' + appId ];
     if (options.managedOnly) labels.push('isCloudronManaged=true');
 
@@ -446,8 +473,6 @@ function stopContainers(appId, callback) {
     assert.strictEqual(typeof appId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug('Stopping containers of %s', appId);
-
     gConnection.listContainers({ all: 1, filters: JSON.stringify({ label: [ 'appId=' + appId ] }) }, function (error, containers) {
         if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
 
@@ -514,7 +539,7 @@ function inspect(containerId, callback) {
     var container = gConnection.getContainer(containerId);
 
     container.inspect(function (error, result) {
-        if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
+        if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND, `Unable to find container ${containerId}`));
         if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
 
         callback(null, result);
@@ -573,10 +598,10 @@ function memoryUsage(containerId, callback) {
     });
 }
 
-function createVolume(app, name, volumeDataDir, callback) {
-    assert.strictEqual(typeof app, 'object');
+function createVolume(name, volumeDataDir, labels, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof volumeDataDir, 'string');
+    assert.strictEqual(typeof labels, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     const volumeOptions = {
@@ -587,10 +612,7 @@ function createVolume(app, name, volumeDataDir, callback) {
             device: volumeDataDir,
             o: 'bind'
         },
-        Labels: {
-            'fqdn': app.fqdn,
-            'appId': app.id
-        },
+        Labels: labels
     };
 
     // requires sudo because the path can be outside appsdata
@@ -605,8 +627,7 @@ function createVolume(app, name, volumeDataDir, callback) {
     });
 }
 
-function clearVolume(app, name, options, callback) {
-    assert.strictEqual(typeof app, 'object');
+function clearVolume(name, options, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -626,14 +647,13 @@ function clearVolume(app, name, options, callback) {
 }
 
 // this only removes the volume and not the data
-function removeVolume(app, name, callback) {
-    assert.strictEqual(typeof app, 'object');
+function removeVolume(name, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof callback, 'function');
 
     let volume = gConnection.getVolume(name);
     volume.remove(function (error) {
-        if (error && error.statusCode !== 404) return callback(new BoxError(BoxError.DOCKER_ERROR, `removeVolume: Error removing volume of ${app.id} ${error.message}`));
+        if (error && error.statusCode !== 404) return callback(new BoxError(BoxError.DOCKER_ERROR, `removeVolume: Error removing volume: ${error.message}`));
 
         callback();
     });

src/dockerproxy.js

@@ -55,7 +55,7 @@ function attachDockerRequest(req, res, next) {
         // Force node to send out the headers, this is required for the /container/wait api to make the docker cli proceed
         res.write(' ');
 
-        dockerResponse.on('error', function (error) { console.error('dockerResponse error:', error); });
+        dockerResponse.on('error', function (error) { debug('dockerResponse error:', error); });
         dockerResponse.pipe(res, { end: true });
     });
 

src/domaindb.js

@@ -66,7 +66,7 @@ function add(name, data, callback) {
     ];
 
     database.transaction(queries, function (error) {
-        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'Domain already exists'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(null);

src/domains.js

@@ -26,15 +26,10 @@ module.exports = exports = {
 
     parentDomain: parentDomain,
 
-    checkDnsRecords: checkDnsRecords,
-
-    prepareDashboardDomain: prepareDashboardDomain,
-
-    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8)
+    checkDnsRecords: checkDnsRecords
 };
 
 var assert = require('assert'),
-    async = require('async'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:domains'),
@@ -56,7 +51,6 @@ function api(provider) {
     assert.strictEqual(typeof provider, 'string');
 
     switch (provider) {
-    case 'caas': return require('./dns/caas.js');
     case 'cloudflare': return require('./dns/cloudflare.js');
     case 'route53': return require('./dns/route53.js');
     case 'gcdns': return require('./dns/gcdns.js');
@@ -95,14 +89,12 @@ function verifyDnsConfig(dnsConfig, domain, zoneName, provider, callback) {
         if (error && error.reason === BoxError.EXTERNAL_ERROR) return callback(new BoxError(BoxError.BAD_FIELD, `Configuration error: ${error.message}`));
         if (error) return callback(error);
 
-        result.hyphenatedSubdomains = !!dnsConfig.hyphenatedSubdomains;
-
         callback(null, result);
     });
 }
 
 function fqdn(location, domainObject) {
-    return location + (location ? (domainObject.config.hyphenatedSubdomains ? '-' : '.') : '') + domainObject.domain;
+    return location + (location ? '.' : '') + domainObject.domain;
 }
 
 // Hostname validation comes from RFC 1123 (section 2.1)
@@ -136,10 +128,6 @@ function validateHostname(location, domainObject) {
         if (/^[-.]/.test(location)) return new BoxError(BoxError.BAD_FIELD, 'Subdomain cannot start or end with hyphen or dot', { field: 'location' });
     }
 
-    if (domainObject.config.hyphenatedSubdomains) {
-        if (location.indexOf('.') !== -1) return new BoxError(BoxError.BAD_FIELD, 'Subdomain cannot contain a dot', { field: 'location' });
-    }
-
     return null;
 }
 
@@ -151,10 +139,9 @@ function validateTlsConfig(tlsConfig, dnsProvider) {
     case 'letsencrypt-prod':
     case 'letsencrypt-staging':
     case 'fallback':
-    case 'caas':
         break;
     default:
-        return new BoxError(BoxError.BAD_FIELD, 'tlsConfig.provider must be caas, fallback, letsencrypt-prod/staging', { field: 'tlsProvider' });
+        return new BoxError(BoxError.BAD_FIELD, 'tlsConfig.provider must be fallback, letsencrypt-prod/staging', { field: 'tlsProvider' });
     }
 
     if (tlsConfig.wildcard) {
@@ -315,6 +302,7 @@ function del(domain, auditSource, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     if (domain === settings.adminDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove admin domain'));
+    if (domain === settings.mailDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove mail domain'));
 
     domaindb.del(domain, function (error) {
         if (error) return callback(error);
@@ -343,19 +331,7 @@ function getName(domain, location, type) {
 
     if (location === '') return part;
 
-    if (!domain.config.hyphenatedSubdomains) return part ? `${location}.${part}` : location;
-
-    // hyphenatedSubdomains
-    if (type !== 'TXT') return `${location}-${part}`;
-
-    if (location.startsWith('_acme-challenge.')) {
-        return `${location}-${part}`;
-    } else if (location === '_acme-challenge') {
-        const up = part.replace(/^[^.]*\.?/, ''); // this gets the domain one level up
-        return up ? `${location}.${up}` : location;
-    } else {
-        return `${location}.${part}`;
-    }
+    return part ? `${location}.${part}` : location;
 }
 
 function getDnsRecords(location, domain, type, callback) {
@@ -463,8 +439,7 @@ function removePrivateFields(domain) {
 function removeRestrictedFields(domain) {
     var result = _.pick(domain, 'domain', 'zoneName', 'provider');
 
-    // always ensure config object
-    result.config = { hyphenatedSubdomains: !!domain.config.hyphenatedSubdomains };
+    result.config = {}; // always ensure config object
 
     return result;
 }
@@ -476,33 +451,3 @@ function makeWildcard(hostname) {
     parts[0] = '*';
     return parts.join('.');
 }
-
-function prepareDashboardDomain(domain, auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof callback, 'function');
-
-    get(domain, function (error, domainObject) {
-        if (error) return callback(error);
-
-        const adminFqdn = fqdn(constants.ADMIN_LOCATION, domainObject);
-
-        sysinfo.getServerIp(function (error, ip) {
-            if (error) return callback(error);
-
-            async.series([
-                (done) => { progressCallback({ percent: 10, message: `Updating DNS of ${adminFqdn}` }); done(); },
-                upsertDnsRecords.bind(null, constants.ADMIN_LOCATION, domain, 'A', [ ip ]),
-                (done) => { progressCallback({ percent: 40, message: `Waiting for DNS of ${adminFqdn}` }); done(); },
-                waitForDnsRecord.bind(null, constants.ADMIN_LOCATION, domain, 'A', ip, { interval: 30000, times: 50000 }),
-                (done) => { progressCallback({ percent: 70, message: `Getting certificate of ${adminFqdn}` }); done(); },
-                reverseProxy.ensureCertificate.bind(null, fqdn(constants.ADMIN_LOCATION, domainObject), domain, auditSource)
-            ], function (error) {
-                if (error) return callback(error);
-
-                callback(null);
-            });
-        });
-    });
-}

src/eventlog.js

@@ -39,6 +39,7 @@ exports = module.exports = {
     ACTION_DOMAIN_UPDATE: 'domain.update',
     ACTION_DOMAIN_REMOVE: 'domain.remove',
 
+    ACTION_MAIL_LOCATION: 'mail.location',
     ACTION_MAIL_ENABLED: 'mail.enabled',
     ACTION_MAIL_DISABLED: 'mail.disabled',
     ACTION_MAIL_MAILBOX_ADD: 'mail.box.add',
@@ -60,6 +61,10 @@ exports = module.exports = {
     ACTION_USER_UPDATE: 'user.update',
     ACTION_USER_TRANSFER: 'user.transfer',
 
+    ACTION_VOLUME_ADD: 'volume.add',
+    ACTION_VOLUME_UPDATE: 'volume.update',
+    ACTION_VOLUME_REMOVE: 'volume.remove',
+
     ACTION_DYNDNS_UPDATE: 'dyndns.update',
 
     ACTION_SUPPORT_TICKET: 'support.ticket',

src/externalldap.js

@@ -20,7 +20,9 @@ var assert = require('assert'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:externalldap'),
+    groups = require('./groups.js'),
     ldap = require('ldapjs'),
+    once = require('once'),
     settings = require('./settings.js'),
     tasks = require('./tasks.js'),
     users = require('./users.js');
@@ -40,14 +42,14 @@ function translateUser(ldapConfig, ldapUser) {
 
     return {
         username: ldapUser[ldapConfig.usernameField],
-        email: ldapUser.mail,
+        email: ldapUser.mail || ldapUser.mailPrimaryAddress,
         displayName: ldapUser.cn // user.giveName + ' ' + user.sn
     };
 }
 
 function validUserRequirements(user) {
     if (!user.username || !user.email || !user.displayName) {
-        debug(`[LDAP user empty username/email/displayName] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+        debug(`[Invalid LDAP user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
         return false;
     } else {
         return true;
@@ -55,40 +57,95 @@ function validUserRequirements(user) {
 }
 
 // performs service bind if required
-function getClient(externalLdapConfig, callback) {
+function getClient(externalLdapConfig, doBindAuth, callback) {
     assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof doBindAuth, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
+    // ensure we only callback once since we also have to listen to client.error events
+    callback = once(callback);
+
     // basic validation to not crash
     try { ldap.parseDN(externalLdapConfig.baseDn); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid baseDn')); }
     try { ldap.parseFilter(externalLdapConfig.filter); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid filter')); }
 
+    var config = {
+        url: externalLdapConfig.url,
+        tlsOptions: {
+            rejectUnauthorized: externalLdapConfig.acceptSelfSignedCerts ? false : true
+        }
+    };
+
     var client;
     try {
-        client = ldap.createClient({ url: externalLdapConfig.url });
+        client = ldap.createClient(config);
     } catch (e) {
         if (e instanceof ldap.ProtocolError) return callback(new BoxError(BoxError.BAD_FIELD, 'url protocol is invalid'));
         return callback(new BoxError(BoxError.INTERNAL_ERROR, e));
     }
 
-    if (!externalLdapConfig.bindDn) return callback(null, client);
+    // ensure we don't just crash
+    client.on('error', function (error) {
+        callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+    });
+
+    // skip bind auth if none exist or if not wanted
+    if (!externalLdapConfig.bindDn || !doBindAuth) return callback(null, client);
 
     client.bind(externalLdapConfig.bindDn, externalLdapConfig.bindPassword, function (error) {
         if (error instanceof ldap.InvalidCredentialsError) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
         if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
 
-        callback(null, client, externalLdapConfig);
+        callback(null, client);
     });
 }
 
+function ldapGetByDN(externalLdapConfig, dn, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof dn, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    getClient(externalLdapConfig, true, function (error, client) {
+        if (error) return callback(error);
+
+        let searchOptions = {
+            paged: true,
+            scope: 'sub'    // We may have to make this configurable
+        };
+
+        debug(`Get object at ${dn}`);
+
+        // basic validation to not crash
+        try { ldap.parseDN(dn); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid DN')); }
+
+        client.search(dn, searchOptions, function (error, result) {
+            if (error instanceof ldap.NoSuchObjectError) return callback(new BoxError(BoxError.NOT_FOUND));
+            if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+
+            let ldapObjects = [];
+
+            result.on('searchEntry', entry => ldapObjects.push(entry.object));
+            result.on('error', error => callback(new BoxError(BoxError.EXTERNAL_ERROR, error)));
+
+            result.on('end', function (result) {
+                client.unbind();
+
+                if (result.status !== 0) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Server returned status ' + result.status));
+                if (ldapObjects.length === 0) return callback(new BoxError(BoxError.NOT_FOUND));
+
+                callback(null, ldapObjects[0]);
+            });
+        });
+    });
+}
 
 // TODO support search by email
-function ldapSearch(externalLdapConfig, options, callback) {
+function ldapUserSearch(externalLdapConfig, options, callback) {
     assert.strictEqual(typeof externalLdapConfig, 'object');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    getClient(externalLdapConfig, function (error, client) {
+    getClient(externalLdapConfig, true, function (error, client) {
         if (error) return callback(error);
 
         let searchOptions = {
@@ -124,6 +181,48 @@ function ldapSearch(externalLdapConfig, options, callback) {
     });
 }
 
+function ldapGroupSearch(externalLdapConfig, options, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    getClient(externalLdapConfig, true, function (error, client) {
+        if (error) return callback(error);
+
+        let searchOptions = {
+            paged: true,
+            scope: 'sub'    // We may have to make this configurable
+        };
+
+        if (externalLdapConfig.groupFilter) searchOptions.filter = ldap.parseFilter(externalLdapConfig.groupFilter);
+
+        if (options.filter) { // https://github.com/ldapjs/node-ldapjs/blob/master/docs/filters.md
+            let extraFilter = ldap.parseFilter(options.filter);
+            searchOptions.filter = new ldap.AndFilter({ filters: [ extraFilter, searchOptions.filter ] });
+        }
+
+        debug(`Listing groups at ${externalLdapConfig.groupBaseDn} with filter ${searchOptions.filter.toString()}`);
+
+        client.search(externalLdapConfig.groupBaseDn, searchOptions, function (error, result) {
+            if (error instanceof ldap.NoSuchObjectError) return callback(new BoxError(BoxError.NOT_FOUND));
+            if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+
+            let ldapGroups = [];
+
+            result.on('searchEntry', entry => ldapGroups.push(entry.object));
+            result.on('error', error => callback(new BoxError(BoxError.EXTERNAL_ERROR, error)));
+
+            result.on('end', function (result) {
+                client.unbind();
+
+                if (result.status !== 0) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Server returned status ' + result.status));
+
+                callback(null, ldapGroups);
+            });
+        });
+    });
+}
+
 function testConfig(config, callback) {
     assert.strictEqual(typeof config, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -141,7 +240,20 @@ function testConfig(config, callback) {
     if (!config.filter) return callback(new BoxError(BoxError.BAD_FIELD, 'filter must not be empty'));
     try { ldap.parseFilter(config.filter); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid filter')); }
 
-    getClient(config, function (error, client) {
+    if ('syncGroups' in config && typeof config.syncGroups !== 'boolean') return callback(new BoxError(BoxError.BAD_FIELD, 'syncGroups must be a boolean'));
+    if ('acceptSelfSignedCerts' in config && typeof config.acceptSelfSignedCerts !== 'boolean') return callback(new BoxError(BoxError.BAD_FIELD, 'acceptSelfSignedCerts must be a boolean'));
+
+    if (config.syncGroups) {
+        if (!config.groupBaseDn) return callback(new BoxError(BoxError.BAD_FIELD, 'groupBaseDn must not be empty'));
+        try { ldap.parseDN(config.groupBaseDn); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid groupBaseDn')); }
+
+        if (!config.groupFilter) return callback(new BoxError(BoxError.BAD_FIELD, 'groupFilter must not be empty'));
+        try { ldap.parseFilter(config.groupFilter); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid groupFilter')); }
+
+        if (!config.groupnameField || typeof config.groupnameField !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'groupFilter must not be empty'));
+    }
+
+    getClient(config, true, function (error, client) {
         if (error) return callback(error);
 
         var opts = {
@@ -167,7 +279,7 @@ function search(identifier, callback) {
         if (error) return callback(error);
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
 
-        ldapSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
+        ldapUserSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
             if (error) return callback(error);
 
             // translate ldap properties to ours
@@ -188,7 +300,7 @@ function createAndVerifyUserIfNotExist(identifier, password, callback) {
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
         if (!externalLdapConfig.autoCreate) return callback(new BoxError(BoxError.BAD_STATE, 'auto create not enabled'));
 
-        ldapSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
+        ldapUserSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
             if (error) return callback(error);
             if (ldapUsers.length === 0) return callback(new BoxError(BoxError.NOT_FOUND));
             if (ldapUsers.length > 1) return callback(new BoxError(BoxError.CONFLICT));
@@ -198,7 +310,7 @@ function createAndVerifyUserIfNotExist(identifier, password, callback) {
 
             users.create(user.username, null /* password */, user.email, user.displayName, { source: 'ldap' }, auditSource.EXTERNAL_LDAP_AUTO_CREATE, function (error, user) {
                 if (error) {
-                    console.error('Failed to auto create user', user.username, error);
+                    debug(`createAndVerifyUserIfNotExist: Failed to auto create user ${user.username}`, error);
                     return callback(new BoxError(BoxError.INTERNAL_ERROR));
                 }
 
@@ -220,17 +332,20 @@ function verifyPassword(user, password, callback) {
         if (error) return callback(error);
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
 
-        ldapSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${user.username}` }, function (error, ldapUsers) {
+        ldapUserSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${user.username}` }, function (error, ldapUsers) {
             if (error) return callback(error);
             if (ldapUsers.length === 0) return callback(new BoxError(BoxError.NOT_FOUND));
             if (ldapUsers.length > 1) return callback(new BoxError(BoxError.CONFLICT));
 
-            let client = ldap.createClient({ url: externalLdapConfig.url });
-            client.bind(ldapUsers[0].dn, password, function (error) {
-                if (error instanceof ldap.InvalidCredentialsError) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+            getClient(externalLdapConfig, false, function (error, client) {
+                if (error) return callback(error);
 
-                callback(null, translateUser(externalLdapConfig, ldapUsers[0]));
+                client.bind(ldapUsers[0].dn, password, function (error) {
+                    if (error instanceof ldap.InvalidCredentialsError) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+                    if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+
+                    callback(null, translateUser(externalLdapConfig, ldapUsers[0]));
+                });
             });
         });
     });
@@ -255,6 +370,197 @@ function startSyncer(callback) {
     });
 }
 
+function syncUsers(externalLdapConfig, progressCallback, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    ldapUserSearch(externalLdapConfig, {}, function (error, ldapUsers) {
+        if (error) return callback(error);
+
+        debug(`Found ${ldapUsers.length} users`);
+
+        let percent = 10;
+        let step = 30/(ldapUsers.length+1); // ensure no divide by 0
+
+        // we ignore all errors here and just log them for now
+        async.eachSeries(ldapUsers, function (user, iteratorCallback) {
+            user = translateUser(externalLdapConfig, user);
+
+            if (!validUserRequirements(user)) return iteratorCallback();
+
+            percent += step;
+            progressCallback({ percent, message: `Syncing... ${user.username}` });
+
+            users.getByUsername(user.username, function (error, result) {
+                if (error && error.reason !== BoxError.NOT_FOUND) return iteratorCallback(error);
+
+                if (!result) {
+                    debug(`[adding user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    users.create(user.username, null /* password */, user.email, user.displayName, { source: 'ldap' }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
+                        if (error) debug('syncUsers: Failed to create user', user, error.message);
+                        iteratorCallback();
+                    });
+                } else if (result.source !== 'ldap') {
+                    debug(`[conflicting user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    iteratorCallback();
+                } else if (result.email !== user.email || result.displayName !== user.displayName) {
+                    debug(`[updating user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    users.update(result, { email: user.email, fallbackEmail: user.email, displayName: user.displayName }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
+                        if (error) debug('Failed to update user', user, error);
+
+                        iteratorCallback();
+                    });
+                } else {
+                    // user known and up-to-date
+                    debug(`[up-to-date user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    iteratorCallback();
+                }
+            });
+        }, callback);
+    });
+}
+
+function syncGroups(externalLdapConfig, progressCallback, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!externalLdapConfig.syncGroups) {
+        debug('Group sync is disabled');
+        progressCallback({ percent: 70, message: 'Skipping group sync...' });
+        return callback(null, []);
+    }
+
+    ldapGroupSearch(externalLdapConfig, {}, function (error, ldapGroups) {
+        if (error) return callback(error);
+
+        debug(`Found ${ldapGroups.length} groups`);
+
+        let percent = 40;
+        let step = 30/(ldapGroups.length+1); // ensure no divide by 0
+
+        // we ignore all non internal errors here and just log them for now
+        async.eachSeries(ldapGroups, function (ldapGroup, iteratorCallback) {
+            var groupName = ldapGroup[externalLdapConfig.groupnameField];
+            if (!groupName) return iteratorCallback();
+            // some servers return empty array for unknown properties :-/
+            if (typeof groupName !== 'string') return iteratorCallback();
+
+            // groups are lowercase
+            groupName = groupName.toLowerCase();
+
+            percent += step;
+            progressCallback({ percent, message: `Syncing... ${groupName}` });
+
+            groups.getByName(groupName, function (error, result) {
+                if (error && error.reason !== BoxError.NOT_FOUND) return iteratorCallback(error);
+
+                if (!result) {
+                    debug(`[adding group] groupname=${groupName}`);
+
+                    groups.create(groupName, 'ldap', function (error) {
+                        if (error) debug('syncGroups: Failed to create group', groupName, error);
+                        iteratorCallback();
+                    });
+                } else {
+                    debug(`[up-to-date group] groupname=${groupName}`);
+
+                    iteratorCallback();
+                }
+            });
+        }, function (error) {
+            if (error) return callback(error);
+
+            debug('sync: ldap sync is done', error);
+
+            callback(error);
+        });
+    });
+}
+
+function syncGroupUsers(externalLdapConfig, progressCallback, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!externalLdapConfig.syncGroups) {
+        debug('Group users sync is disabled');
+        progressCallback({ percent: 99, message: 'Skipping group users sync...' });
+        return callback(null, []);
+    }
+
+    groups.getAll(function (error, result) {
+        if (error) return callback(error);
+
+        var ldapGroups = result.filter(function (g) { return g.source === 'ldap'; });
+        debug(`Found ${ldapGroups.length} groups to sync users`);
+
+        async.eachSeries(ldapGroups, function (group, iteratorCallback) {
+            debug(`Sync users for group ${group.name}`);
+
+            ldapGroupSearch(externalLdapConfig, {}, function (error, result) {
+                if (error) return callback(error);
+                if (!result || result.length === 0) {
+                    debug(`syncGroupUsers: Unable to find group ${group.name} ignoring for now.`);
+                    return callback();
+                }
+
+                // since our group names are lowercase we cannot use potentially case matching ldap filters
+                let found = result.find(function (r) {
+                    if (!r[externalLdapConfig.groupnameField]) return false;
+                    return r[externalLdapConfig.groupnameField].toLowerCase() === group.name;
+                });
+
+                if (!found) {
+                    debug(`syncGroupUsers: Unable to find group ${group.name} ignoring for now.`);
+                    return callback();
+                }
+
+                var ldapGroupMembers = found.member || found.uniqueMember || [];
+
+                // if only one entry is in the group ldap returns a string, not an array!
+                if (typeof ldapGroupMembers === 'string') ldapGroupMembers = [ ldapGroupMembers ];
+
+                debug(`Group ${group.name} has ${ldapGroupMembers.length} members.`);
+
+                async.eachSeries(ldapGroupMembers, function (memberDn, iteratorCallback) {
+                    ldapGetByDN(externalLdapConfig, memberDn, function (error, result) {
+                        if (error) {
+                            debug(`Failed to get ${memberDn}:`, error);
+                            return iteratorCallback();
+                        }
+
+                        debug(`Found member object at ${memberDn} adding to group ${group.name}`);
+
+                        const username = result[externalLdapConfig.usernameField];
+                        if (!username) return iteratorCallback();
+
+                        users.getByUsername(username, function (error, result) {
+                            if (error) {
+                                debug(`syncGroupUsers: Failed to get user by username ${username}`, error);
+                                return iteratorCallback();
+                            }
+
+                            groups.addMember(group.id, result.id, function (error) {
+                                if (error && error.reason !== BoxError.ALREADY_EXISTS) debug('syncGroupUsers: Failed to add member', error);
+                                iteratorCallback();
+                            });
+                        });
+                    });
+                }, function (error) {
+                    if (error) debug('syncGroupUsers: ', error);
+                    iteratorCallback();
+                });
+            });
+        }, callback);
+    });
+}
+
 function sync(progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
@@ -265,58 +571,18 @@ function sync(progressCallback, callback) {
         if (error) return callback(error);
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
 
-        ldapSearch(externalLdapConfig, {}, function (error, ldapUsers) {
+        async.series([
+            syncUsers.bind(null, externalLdapConfig, progressCallback),
+            syncGroups.bind(null, externalLdapConfig, progressCallback),
+            syncGroupUsers.bind(null, externalLdapConfig, progressCallback)
+        ], function (error) {
             if (error) return callback(error);
 
-            debug(`Found ${ldapUsers.length} users`);
-            let percent = 10;
-            let step = 90/(ldapUsers.length+1); // ensure no divide by 0
+            progressCallback({ percent: 100, message: 'Done' });
 
-            // we ignore all errors here and just log them for now
-            async.eachSeries(ldapUsers, function (user, iteratorCallback) {
-                user = translateUser(externalLdapConfig, user);
+            debug('sync: ldap sync is done', error);
 
-                if (!validUserRequirements(user)) return iteratorCallback();
-
-                percent += step;
-                progressCallback({ percent, message: `Syncing... ${user.username}` });
-
-                users.getByUsername(user.username, function (error, result) {
-                    if (error && error.reason !== BoxError.NOT_FOUND) {
-                        debug(`Could not find user with username ${user.username}: ${error.message}`);
-                        return iteratorCallback();
-                    }
-
-                    if (error) {
-                        debug(`[adding user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        users.create(user.username, null /* password */, user.email, user.displayName, { source: 'ldap' }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
-                            if (error) console.error('Failed to create user', user, error);
-                            iteratorCallback();
-                        });
-                    } else if (result.source !== 'ldap') {
-                        debug(`[conflicting user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        iteratorCallback();
-                    } else if (result.email !== user.email || result.displayName !== user.displayName) {
-                        debug(`[updating user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        users.update(result, { email: user.email, fallbackEmail: user.email, displayName: user.displayName }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
-                            if (error) debug('Failed to update user', user, error);
-
-                            iteratorCallback();
-                        });
-                    } else {
-                        // user known and up-to-date
-                        debug(`[up-to-date user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        iteratorCallback();
-                    }
-                });
-            }, function (error) {
-                debug('sync: ldap sync is done', error);
-                callback(error);
-            });
+            callback(error);
         });
     });
 }

src/graphs.js

@@ -5,6 +5,7 @@ exports = module.exports = {
 };
 
 var assert = require('assert'),
+    async = require('async'),
     infra = require('./infra_version.js'),
     paths = require('./paths.js'),
     shell = require('./shell.js');
@@ -26,7 +27,7 @@ function startGraphite(existingInfra, callback) {
                 --log-opt syslog-address=udp://127.0.0.1:2514 \
                 --log-opt syslog-format=rfc5424 \
                 --log-opt tag=graphite \
-                -m 75m \
+                -m 150m \
                 --memory-swap 150m \
                 --dns 172.18.0.1 \
                 --dns-search=. \
@@ -37,5 +38,9 @@ function startGraphite(existingInfra, callback) {
                 --label isCloudronManaged=true \
                 --read-only -v /tmp -v /run "${tag}"`;
 
-    shell.exec('startGraphite', cmd, callback);
+    async.series([
+        shell.exec.bind(null, 'stopGraphite', 'docker stop graphite || true'),
+        shell.exec.bind(null, 'removeGraphite', 'docker rm -f graphite || true'),
+        shell.exec.bind(null, 'startGraphite', cmd)
+    ], callback);
 }

src/groupdb.js

@@ -2,6 +2,7 @@
 
 exports = module.exports = {
     get: get,
+    getByName: getByName,
     getWithMembers: getWithMembers,
     getAll: getAll,
     getAllWithMembers: getAllWithMembers,
@@ -19,8 +20,6 @@ exports = module.exports = {
     getMembership: getMembership,
     setMembership: setMembership,
 
-    getGroups: getGroups,
-
     _clear: clear
 };
 
@@ -28,7 +27,7 @@ var assert = require('assert'),
     BoxError = require('./boxerror.js'),
     database = require('./database.js');
 
-var GROUPS_FIELDS = [ 'id', 'name' ].join(',');
+var GROUPS_FIELDS = [ 'id', 'name', 'source' ].join(',');
 
 function get(groupId, callback) {
     assert.strictEqual(typeof groupId, 'string');
@@ -42,6 +41,18 @@ function get(groupId, callback) {
     });
 }
 
+function getByName(name, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups WHERE name = ?', [ name ], function (error, result) {
+        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Group not found'));
+
+        callback(null, result[0]);
+    });
+}
+
 function getWithMembers(groupId, callback) {
     assert.strictEqual(typeof groupId, 'string');
     assert.strictEqual(typeof callback, 'function');
@@ -63,7 +74,7 @@ function getWithMembers(groupId, callback) {
 function getAll(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups', function (error, results) {
+    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups ORDER BY name', function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(null, results);
@@ -73,9 +84,8 @@ function getAll(callback) {
 function getAllWithMembers(callback) {
     database.query('SELECT ' + GROUPS_FIELDS + ',GROUP_CONCAT(groupMembers.userId) AS userIds ' +
                     ' FROM userGroups LEFT OUTER JOIN groupMembers ON userGroups.id = groupMembers.groupId ' +
-                    ' GROUP BY userGroups.id', function (error, results) {
+                    ' GROUP BY userGroups.id ORDER BY name', function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Group not found'));
 
         results.forEach(function (result) { result.userIds = result.userIds ? result.userIds.split(',') : [ ]; });
 
@@ -83,12 +93,13 @@ function getAllWithMembers(callback) {
     });
 }
 
-function add(id, name, callback) {
+function add(id, name, source, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof source, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO userGroups (id, name) VALUES (?, ?)', [ id, name ], function (error, result) {
+    database.query('INSERT INTO userGroups (id, name, source) VALUES (?, ?, ?)', [ id, name, source ], function (error, result) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
         if (error || result.affectedRows !== 1) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -260,15 +271,3 @@ function isMember(groupId, userId, callback) {
         callback(null, result.length !== 0);
     });
 }
-
-function getGroups(userId, callback) {
-    assert.strictEqual(typeof userId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query('SELECT ' + GROUPS_FIELDS + ' ' +
-        ' FROM userGroups INNER JOIN groupMembers ON userGroups.id = groupMembers.groupId AND groupMembers.userId = ?', [ userId ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-        callback(null, results);
-    });
-}

src/groups.js

@@ -4,6 +4,7 @@ exports = module.exports = {
     create: create,
     remove: remove,
     get: get,
+    getByName: getByName,
     update: update,
     getWithMembers: getWithMembers,
     getAll: getAll,
@@ -15,8 +16,6 @@ exports = module.exports = {
     removeMember: removeMember,
     isMember: isMember,
 
-    getGroups: getGroups,
-
     setMembership: setMembership,
     getMembership: getMembership,
 
@@ -45,8 +44,17 @@ function validateGroupname(name) {
     return null;
 }
 
-function create(name, callback) {
+function validateGroupSource(source) {
+    assert.strictEqual(typeof source, 'string');
+
+    if (source !== '' && source !== 'ldap') return new BoxError(BoxError.BAD_FIELD, 'source must be "" or "ldap"', { field: source });
+
+    return null;
+}
+
+function create(name, source, callback) {
     assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof source, 'string');
     assert.strictEqual(typeof callback, 'function');
 
     // we store names in lowercase
@@ -55,8 +63,11 @@ function create(name, callback) {
     var error = validateGroupname(name);
     if (error) return callback(error);
 
+    error = validateGroupSource(source);
+    if (error) return callback(error);
+
     var id = 'gid-' + uuid.v4();
-    groupdb.add(id, name, function (error) {
+    groupdb.add(id, name, source, function (error) {
         if (error) return callback(error);
 
         callback(null, { id: id, name: name });
@@ -85,6 +96,17 @@ function get(id, callback) {
     });
 }
 
+function getByName(name, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    groupdb.getByName(name, function (error, result) {
+        if (error) return callback(error);
+
+        return callback(null, result);
+    });
+}
+
 function getWithMembers(id, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');
@@ -217,17 +239,6 @@ function update(groupId, data, callback) {
     });
 }
 
-function getGroups(userId, callback) {
-    assert.strictEqual(typeof userId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    groupdb.getGroups(userId, function (error, results) {
-        if (error) return callback(error);
-
-        callback(null, results);
-    });
-}
-
 function count(callback) {
     assert.strictEqual(typeof callback, 'function');
 

src/infra_version.js

@@ -6,22 +6,22 @@
 
 exports = module.exports = {
     // a version change recreates all containers with latest docker config
-    'version': '48.17.0',
+    'version': '48.17.2',
 
     'baseImages': [
-        { repo: 'cloudron/base', tag: 'cloudron/base:1.0.0@sha256:147a648a068a2e746644746bbfb42eb7a50d682437cead3c67c933c546357617' }
+        { repo: 'cloudron/base-arm64', tag: 'cloudron/base-arm64:2.0.0@sha256:cc336184d5968636804951a0ab44f8d2c8cdd19b94f045753d312d81705b5806' }
     ],
 
     // a major version bump in the db containers will trigger the restore logic that uses the db dumps
     // docker inspect --format='{{index .RepoDigests 0}}' $IMAGE to get the sha256
     'images': {
-        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.0.2@sha256:2643b73fe371154e37647957cc7103cacb34c50737f2954abd7d70f167a1f33a' },
-        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.2.0@sha256:440c8a9ca4d2958d51a375359f8158ef702b83395aa9ac4f450c51825ec09239' },
-        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:2.0.2@sha256:6dcee0731dfb9b013ed94d56205eee219040ee806c7e251db3b3886eaa4947ff' },
-        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:2.1.0@sha256:6d1bf221cfe6124957e2c58b57c0a47214353496009296acb16adf56df1da9d5' },
-        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.1.0@sha256:f2cda21bd15c21bbf44432df412525369ef831a2d53860b5c5b1675e6f384de2' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.7.2@sha256:f20d112ff9a97e052a9187063eabbd8d484ce369114d44186e344169a1b3ef6b' },
-        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.2.0@sha256:fc9ca69d16e6ebdbd98ed53143d4a0d2212eef60cb638dc71219234e6f427a2c' },
-        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:1.0.0@sha256:3b70aac36700225945a4a39b5a400c28e010e980879d0dcca76e4a37b04a16ed' }
+        'turn': { repo: 'cloudron/turn-arm64', tag: 'cloudron/turn-arm64:1.1.0@sha256:198577e2105fbc5f69588f84eda9ce27420e62c9fc957faddb037cd5708edf01' },
+        'mysql': { repo: 'cloudron/mysql-arm64', tag: 'cloudron/mysql-arm64:2.3.2@sha256:695670fa5438b0c7b44784995014a570165b6f6c990d9778c44e833b511a7bdc' },
+        'postgresql': { repo: 'cloudron/postgresql-arm64', tag: 'cloudron/postgresql-arm64:3.3.0@sha256:76b79a99dc968bc2de6e6b20f889e41584b7c671d0953ade9e13ebcea9a5b80b' },
+        'mongodb': { repo: 'cloudron/mongodb-arm64', tag: 'cloudron/mongodb-arm64:3.0.0@sha256:a57769b6d8f94c26548a019712edb261edf34c2bee68cb929287501d2cc6a10d' },
+        'redis': { repo: 'cloudron/redis-arm64', tag: 'cloudron/redis-arm64:2.3.0@sha256:674ba9135c2bc3b7ecd6791f4fa04cb7196c5b73997acf1f31df644978042d69' },
+        'mail': { repo: 'cloudron/mail-arm64', tag: 'cloudron/mail-arm64:2.10.0@sha256:acb67fbd1ad0346fd1634fa2673633a2e9f6088df2f854f3336390c7ceefd04d' },
+        'graphite': { repo: 'cloudron/graphite-arm64', tag: 'cloudron/graphite-arm64:2.3.0@sha256:181c8c7ac0e9cd35cc93013e4e44b41de1b5c9f4454b1c0f8a8acde7080558e1' },
+        'sftp': { repo: 'cloudron/sftp-arm64', tag: 'cloudron/sftp-arm64:3.0.0@sha256:a77554569a039495b0fbdb1ef0265ade53a9f8be48d9346ed64291d72b9be6ac' }
     }
 };

src/janitor.js

@@ -16,21 +16,15 @@ const NOOP_CALLBACK = function () { };
 
 const gConnection = new Docker({ socketPath: '/var/run/docker.sock' });
 
-function ignoreError(func) {
-    return function (callback) {
-        func(function (error) {
-            if (error) console.error('Ignored error:', error);
+function cleanupTokens(callback) {
+    assert(!callback || typeof callback === 'function'); // callback is null when called from cronjob
 
-            callback();
-        });
-    };
-}
+    callback = callback || NOOP_CALLBACK;
 
-function cleanupExpiredTokens(callback) {
-    assert.strictEqual(typeof callback, 'function');
+    debug('Cleaning up expired tokens');
 
     tokendb.delExpired(function (error, result) {
-        if (error) return callback(error);
+        if (error) return debug('cleanupTokens: error removing expired tokens', error);
 
         debug('Cleaned up %s expired tokens.', result);
 
@@ -38,16 +32,6 @@ function cleanupExpiredTokens(callback) {
     });
 }
 
-function cleanupTokens(callback) {
-    assert(!callback || typeof callback === 'function'); // callback is null when called from cronjob
-
-    debug('Cleaning up expired tokens');
-
-    async.series([
-        ignoreError(cleanupExpiredTokens)
-    ], callback);
-}
-
 function cleanupTmpVolume(containerInfo, callback) {
     assert.strictEqual(typeof containerInfo, 'object');
     assert.strictEqual(typeof callback, 'function');

src/ldap.js

@@ -5,7 +5,8 @@ exports = module.exports = {
     stop: stop
 };
 
-var assert = require('assert'),
+var addons = require('./addons.js'),
+    assert = require('assert'),
     appdb = require('./appdb.js'),
     apps = require('./apps.js'),
     async = require('async'),
@@ -154,7 +155,6 @@ function userSearch(req, res, next) {
                     givenName: firstName,
                     username: user.username,
                     samaccountname: user.username,      // to support ActiveDirectory clients
-                    isadmin: users.compareRoles(user.role, users.ROLE_ADMIN) >= 0,
                     memberof: groups
                 }
             };
@@ -347,7 +347,7 @@ function mailboxSearch(req, res, next) {
                         if (error) return callback(error);
 
                         aliases.forEach(function (a, idx) {
-                            obj.attributes['mail' + idx] = `${a}@${mailbox.domain}`;
+                            obj.attributes['mail' + idx] = `${a.name}@${a.domain}`;
                         });
 
                         // ensure all filter values are also lowercase
@@ -392,7 +392,7 @@ function mailAliasSearch(req, res, next) {
                 objectclass: ['nisMailAlias'],
                 objectcategory: 'nisMailAlias',
                 cn: `${alias.name}@${alias.domain}`,
-                rfc822MailMember: `${alias.aliasTarget}@${alias.domain}`
+                rfc822MailMember: `${alias.aliasName}@${alias.aliasDomain}`
             }
         };
 
@@ -418,7 +418,7 @@ function mailingListSearch(req, res, next) {
     if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));
     const name = parts[0], domain = parts[1];
 
-    mail.resolveList(parts[0], parts[1], function (error, resolvedMembers) {
+    mail.resolveList(parts[0], parts[1], function (error, resolvedMembers, list) {
         if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
         if (error) return next(new ldap.OperationsError(error.toString()));
 
@@ -431,6 +431,7 @@ function mailingListSearch(req, res, next) {
                 objectcategory: 'mailGroup',
                 cn: `${name}@${domain}`, // fully qualified
                 mail: `${name}@${domain}`,
+                membersOnly: list.membersOnly, // ldapjs only supports strings and string array. so this is not a bool!
                 mgrpRFC822MailMember: resolvedMembers // fully qualified
             }
         };
@@ -547,6 +548,17 @@ function authenticateSftp(req, res, next) {
     });
 }
 
+function loadSftpConfig(req, res, next) {
+    addons.getServicesConfig('sftp', function (error, service, servicesConfig) {
+        if (error) return next(new ldap.OperationsError(error.toString()));
+
+        const serviceConfig = servicesConfig['sftp'];
+        req.requireAdmin = 'requireAdmin' in serviceConfig ? serviceConfig.requireAdmin : true;
+
+        next();
+    });
+}
+
 function userSearchSftp(req, res, next) {
     debug('sftp user search: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);
 
@@ -570,6 +582,8 @@ function userSearchSftp(req, res, next) {
         users.getByUsername(username, function (error, user) {
             if (error) return next(new ldap.OperationsError(error.toString()));
 
+            if (req.requireAdmin && users.compareRoles(user.role, users.ROLE_ADMIN) < 0) return next(new ldap.InsufficientAccessRightsError('Insufficient previleges'));
+
             apps.hasAccessTo(app, user, function (error, hasAccess) {
                 if (error) return next(new ldap.OperationsError(error.toString()));
                 if (!hasAccess) return next(new ldap.InsufficientAccessRightsError('Not authorized'));
@@ -577,7 +591,7 @@ function userSearchSftp(req, res, next) {
                 var obj = {
                     dn: ldap.parseDN(`cn=${username}@${appFqdn},ou=sftp,dc=cloudron`).toString(),
                     attributes: {
-                        homeDirectory: path.join('/app/data', app.id, 'data'),
+                        homeDirectory: path.join('/app/data', app.id),
                         objectclass: ['user'],
                         objectcategory: 'person',
                         cn: user.id,
@@ -618,10 +632,7 @@ function authenticateMailAddon(req, res, next) {
         // note: with sendmail addon, apps can send mail without a mailbox (unlike users)
         appdb.getAppIdByAddonConfigValue(addonId, namePattern, req.credentials || '', function (error, appId) {
             if (error && error.reason !== BoxError.NOT_FOUND) return next(new ldap.OperationsError(error.message));
-            if (appId) { // matched app password
-                eventlog.add(eventlog.ACTION_APP_LOGIN, { authType: 'ldap', mailboxId: email }, { appId: appId, addonId: addonId });
-                return res.end();
-            }
+            if (appId) return res.end();
 
             mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
@@ -648,14 +659,14 @@ function start(callback) {
         debug: NOOP,
         info: debug,
         warn: debug,
-        error: console.error,
-        fatal: console.error
+        error: debug,
+        fatal: debug
     };
 
     gServer = ldap.createServer({ log: logger });
 
     gServer.on('error', function (error) {
-        console.error('LDAP:', error);
+        debug('start: server error ', error);
     });
 
     gServer.search('ou=users,dc=cloudron', authenticateApp, userSearch);
@@ -672,7 +683,7 @@ function start(callback) {
     gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka
 
     gServer.bind('ou=sftp,dc=cloudron', authenticateSftp);    // sftp
-    gServer.search('ou=sftp,dc=cloudron', userSearchSftp);
+    gServer.search('ou=sftp,dc=cloudron', loadSftpConfig, userSearchSftp);
 
     gServer.compare('cn=users,ou=groups,dc=cloudron', authenticateApp, groupUsersCompare);
     gServer.compare('cn=admins,ou=groups,dc=cloudron', authenticateApp, groupAdminsCompare);

src/logrotate.ejs

@@ -8,6 +8,7 @@
     maxsize 1M
     missingok
     delaycompress
+    # this truncates the original log file and not the rotated one
     copytruncate
 }
 
@@ -18,6 +19,7 @@
     missingok
     # we never compress so we can simply tail the files
     nocompress
+    # this truncates the original log file and not the rotated one
     copytruncate
 }
 

src/mail.js

@@ -1,60 +1,65 @@
 'use strict';
 
 exports = module.exports = {
-    getStatus: getStatus,
-    checkConfiguration: checkConfiguration,
+    getStatus,
+    checkConfiguration,
 
-    getDomains: getDomains,
+    getLocation,
+    setLocation, // triggers the change task
+    changeLocation, // does the actual changing
 
-    getDomain: getDomain,
-    clearDomains: clearDomains,
+    getDomains,
 
-    onDomainAdded: onDomainAdded,
-    onDomainRemoved: onDomainRemoved,
+    getDomain,
+    clearDomains,
 
-    removePrivateFields: removePrivateFields,
+    onDomainAdded,
+    onDomainRemoved,
 
-    setDnsRecords: setDnsRecords,
-    onMailFqdnChanged: onMailFqdnChanged,
+    removePrivateFields,
 
-    validateName: validateName,
+    setDnsRecords,
 
-    setMailFromValidation: setMailFromValidation,
-    setCatchAllAddress: setCatchAllAddress,
-    setMailRelay: setMailRelay,
-    setMailEnabled: setMailEnabled,
+    validateName,
+
+    setMailFromValidation,
+    setCatchAllAddress,
+    setMailRelay,
+    setMailEnabled,
+    setBanner,
 
     startMail: restartMail,
-    restartMail: restartMail,
-    handleCertChanged: handleCertChanged,
-    getMailAuth: getMailAuth,
+    restartMail,
+    handleCertChanged,
+    getMailAuth,
 
-    sendTestMail: sendTestMail,
+    sendTestMail,
 
-    listMailboxes: listMailboxes,
-    removeMailboxes: removeMailboxes,
-    getMailbox: getMailbox,
-    addMailbox: addMailbox,
-    updateMailboxOwner: updateMailboxOwner,
-    removeMailbox: removeMailbox,
+    getMailboxCount,
+    listMailboxes,
+    getMailbox,
+    addMailbox,
+    updateMailboxOwner,
+    removeMailbox,
 
-    listAliases: listAliases,
-    getAliases: getAliases,
-    setAliases: setAliases,
+    getAliases,
+    setAliases,
 
-    getLists: getLists,
-    getList: getList,
-    addList: addList,
-    updateList: updateList,
-    removeList: removeList,
-    resolveList: resolveList,
+    getLists,
+    getList,
+    addList,
+    updateList,
+    removeList,
+    resolveList,
 
+    _removeMailboxes: removeMailboxes,
     _readDkimPublicKeySync: readDkimPublicKeySync
 };
 
 var assert = require('assert'),
     async = require('async'),
     BoxError = require('./boxerror.js'),
+    cloudron = require('./cloudron.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:mail'),
     dns = require('./native-dns.js'),
@@ -76,12 +81,14 @@ var assert = require('assert'),
     shell = require('./shell.js'),
     smtpTransport = require('nodemailer-smtp-transport'),
     sysinfo = require('./sysinfo.js'),
+    tasks = require('./tasks.js'),
     users = require('./users.js'),
     validator = require('validator'),
     _ = require('underscore');
 
 const DNS_OPTIONS = { timeout: 5000 };
 var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+const REMOVE_MAILBOX = path.join(__dirname, 'scripts/rmmailbox.sh');
 
 function validateName(name) {
     assert.strictEqual(typeof name, 'string');
@@ -101,7 +108,6 @@ function checkOutboundPort25(callback) {
     var smtpServer = _.sample([
         'smtp.gmail.com',
         'smtp.live.com',
-        'smtp.mail.yahoo.com',
         'smtp.1und1.de',
     ]);
 
@@ -208,7 +214,8 @@ function checkDkim(mailDomain, callback) {
 
         if (txtRecords.length !== 0) {
             dkim.value = txtRecords[0].join('');
-            dkim.status = (dkim.value === dkim.expected);
+            const actual = txtToDict(dkim.value);
+            dkim.status = actual.p === dkimKey;
         }
 
         callback(null, dkim);
@@ -237,14 +244,14 @@ function checkSpf(domain, mailFqdn, callback) {
             let txtRecord = txtRecords[i].join(''); // https://agari.zendesk.com/hc/en-us/articles/202952749-How-long-can-my-SPF-record-be-
             if (txtRecord.indexOf('v=spf1 ') !== 0) continue; // not SPF
             spf.value = txtRecord;
-            spf.status = spf.value.indexOf(' a:' + settings.adminFqdn()) !== -1;
+            spf.status = spf.value.indexOf(' a:' + settings.mailFqdn()) !== -1;
             break;
         }
 
         if (spf.status) {
             spf.expected = spf.value;
         } else if (i !== txtRecords.length) {
-            spf.expected = 'v=spf1 a:' + settings.adminFqdn() + ' ' + spf.value.slice('v=spf1 '.length);
+            spf.expected = 'v=spf1 a:' + settings.mailFqdn() + ' ' + spf.value.slice('v=spf1 '.length);
         }
 
         callback(null, spf);
@@ -269,7 +276,7 @@ function checkMx(domain, mailFqdn, callback) {
         if (error) return callback(error, mx);
         if (mxRecords.length === 0) return callback(null, mx);
 
-        mx.status = mxRecords.length == 1 && mxRecords[0].exchange === mailFqdn;
+        mx.status = mxRecords.some(mx => mx.exchange === mailFqdn); // this lets use change priority and/or setup backup MX
         mx.value = mxRecords.map(function (r) { return r.priority + ' ' + r.exchange + '.'; }).join(' ');
 
         if (mx.status) return callback(null, mx); // MX record is "my."
@@ -543,7 +550,7 @@ function checkConfiguration(callback) {
                 markdownMessage += '\n\n';
             });
 
-            if (markdownMessage) markdownMessage += 'Email Status is checked every 30 minutes.\n See the [troubleshooting docs](https://cloudron.io/documentation/troubleshooting/#mail-dns) for more information.\n';
+            if (markdownMessage) markdownMessage += 'Email Status is checked every 30 minutes.\n See the [troubleshooting docs](https://docs.cloudron.io/troubleshooting/#mail-dns) for more information.\n';
 
             callback(null, markdownMessage); // empty message means all status checks succeeded
         });
@@ -575,15 +582,18 @@ function createMailConfig(mailFqdn, mailDomain, callback) {
         }
 
         // create sections for per-domain configuration
-        mailDomains.forEach(function (domain) {
+        async.eachSeries(mailDomains, function (domain, iteratorDone) {
             const catchAll = domain.catchAll.map(function (c) { return `${c}@${domain.domain}`; }).join(',');
             const mailFromValidation = domain.mailFromValidation;
 
             if (!safe.fs.appendFileSync(path.join(paths.ADDON_CONFIG_DIR, 'mail/mail.ini'),
                 `[${domain.domain}]\ncatch_all=${catchAll}\nmail_from_validation=${mailFromValidation}\n\n`, 'utf8')) {
-                return callback(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
+                return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
             }
 
+            if (!safe.fs.writeFileSync(`${paths.ADDON_CONFIG_DIR}/mail/banner/${domain.domain}.text`, domain.banner.text || '')) return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create text banner file:' + safe.error.message));
+            if (!safe.fs.writeFileSync(`${paths.ADDON_CONFIG_DIR}/mail/banner/${domain.domain}.html`, domain.banner.html || '')) return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create html banner file:' + safe.error.message));
+
             const relay = domain.relay;
 
             const enableRelay = relay.provider !== 'cloudron-smtp' && relay.provider !== 'noop',
@@ -593,15 +603,19 @@ function createMailConfig(mailFqdn, mailDomain, callback) {
                 username = relay.username || '',
                 password = relay.password || '';
 
-            if (!enableRelay) return;
+            if (!enableRelay) return iteratorDone();
 
             if (!safe.fs.appendFileSync(paths.ADDON_CONFIG_DIR + '/mail/smtp_forward.ini',
                 `[${domain.domain}]\nenable_outbound=true\nhost=${host}\nport=${port}\nenable_tls=true\nauth_type=${authType}\nauth_user=${username}\nauth_pass=${password}\n\n`, 'utf8')) {
-                return callback(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
+                return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
             }
-        });
 
-        callback(null, mailInDomains.length !== 0 /* allowInbound */);
+            iteratorDone();
+        }, function (error) {
+            if (error) return callback(error);
+
+            callback(null, mailInDomains.length !== 0 /* allowInbound */);
+        });
     });
 }
 
@@ -629,7 +643,10 @@ function configureMail(mailFqdn, mailDomain, callback) {
         if (!safe.child_process.execSync(`cp ${bundle.certFilePath} ${mailCertFilePath}`)) return callback(new BoxError(BoxError.FS_ERROR, 'Could not create cert file:' + safe.error.message));
         if (!safe.child_process.execSync(`cp ${bundle.keyFilePath} ${mailKeyFilePath}`)) return callback(new BoxError(BoxError.FS_ERROR, 'Could not create key file:' + safe.error.message));
 
-        shell.exec('startMail', 'docker rm -f mail || true', function (error) {
+        async.series([
+            shell.exec.bind(null, 'stopMail', 'docker stop mail || true'),
+            shell.exec.bind(null, 'removeMail', 'docker rm -f mail || true'),
+        ], function (error) {
             if (error) return callback(error);
 
             createMailConfig(mailFqdn, mailDomain, function (error, allowInbound) {
@@ -798,6 +815,7 @@ function ensureDkimKeySync(mailDomain) {
         return new BoxError(BoxError.FS_ERROR, safe.error);
     }
 
+    // https://www.unlocktheinbox.com/dkim-key-length-statistics/ and https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-authentication-dkim-easy.html for key size
     if (!safe.child_process.execSync('openssl genrsa -out ' + dkimPrivateKeyFile + ' 1024')) return new BoxError(BoxError.OPENSSL_ERROR, safe.error);
     if (!safe.child_process.execSync('openssl rsa -in ' + dkimPrivateKeyFile + ' -out ' + dkimPublicKeyFile + ' -pubout -outform PEM')) return new BoxError(BoxError.OPENSSL_ERROR, safe.error);
 
@@ -886,21 +904,70 @@ function setDnsRecords(domain, callback) {
     upsertDnsRecords(domain, settings.mailFqdn(), callback);
 }
 
-function onMailFqdnChanged(callback) {
+function getLocation(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    const mailFqdn = settings.mailFqdn(),
-        mailDomain = settings.adminDomain();
+    const domain = settings.mailDomain(), fqdn = settings.mailFqdn();
+    const subdomain = fqdn.substr(0, fqdn.length - domain.length - 1);
 
-    domains.getAll(function (error, allDomains) {
+    callback(null, { domain, subdomain });
+}
+
+function changeLocation(auditSource, progressCallback, callback) {
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    const fqdn = settings.mailFqdn(), domain = settings.mailDomain();
+    const subdomain = fqdn.substr(0, fqdn.length - domain.length - 1);
+
+    let progress = 20;
+    progressCallback({ percent: progress, message: `Setting up DNS of certs of mail server ${fqdn}` });
+
+    cloudron.setupDnsAndCert(subdomain, domain, auditSource, progressCallback, function (error) {
         if (error) return callback(error);
 
-        async.eachOfSeries(allDomains, function (domainObject, idx, iteratorDone) {
-            upsertDnsRecords(domainObject.domain, mailFqdn, iteratorDone);
-        }, function (error) {
+        domains.getAll(function (error, allDomains) {
             if (error) return callback(error);
 
-            configureMail(mailFqdn, mailDomain, callback);
+            async.eachOfSeries(allDomains, function (domainObject, idx, iteratorDone) {
+                progressCallback({ percent: progress, message: `Updating DNS of ${domainObject.domain}` });
+                progress += Math.round(70/allDomains.length);
+
+                upsertDnsRecords(domainObject.domain, fqdn, iteratorDone);
+            }, function (error) {
+                if (error) return callback(error);
+
+                progressCallback({ percent: 90, message: 'Restarting mail server' });
+
+                restartMailIfActivated(callback);
+            });
+        });
+    });
+}
+
+function setLocation(subdomain, domain, auditSource, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    domains.get(domain, function (error, domainObject) {
+        if (error) return callback(error);
+
+        const fqdn = domains.fqdn(subdomain, domainObject);
+
+        settings.setMailLocation(domain, fqdn, function (error) {
+            if (error) return callback(error);
+
+            tasks.add(tasks.TASK_CHANGE_MAIL_LOCATION, [ auditSource ], function (error, taskId) {
+                if (error) return callback(error);
+
+                tasks.startTask(taskId, {}, NOOP_CALLBACK);
+                eventlog.add(eventlog.ACTION_MAIL_LOCATION, auditSource, { subdomain, domain, taskId });
+
+                callback(null, taskId);
+            });
         });
     });
 }
@@ -909,6 +976,8 @@ function onDomainAdded(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
+    if (!settings.mailFqdn()) return callback(); // mail domain is not set yet (when provisioning)
+
     async.series([
         upsertDnsRecords.bind(null, domain, settings.mailFqdn()), // do this first to ensure DKIM keys
         restartMailIfActivated
@@ -934,7 +1003,7 @@ function clearDomains(callback) {
 
 // remove all fields that should never be sent out via REST API
 function removePrivateFields(domain) {
-    let result = _.pick(domain, 'domain', 'enabled', 'mailFromValidation', 'catchAll', 'relay');
+    let result = _.pick(domain, 'domain', 'enabled', 'mailFromValidation', 'catchAll', 'relay', 'banner');
     if (result.relay.provider !== 'cloudron-smtp') {
         if (result.relay.username === result.relay.password) result.relay.username = constants.SECRET_PLACEHOLDER;
         result.relay.password = constants.SECRET_PLACEHOLDER;
@@ -956,6 +1025,20 @@ function setMailFromValidation(domain, enabled, callback) {
     });
 }
 
+function setBanner(domain, banner, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof banner, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    maildb.update(domain, { banner }, function (error) {
+        if (error) return callback(error);
+
+        restartMail(NOOP_CALLBACK);
+
+        callback(null);
+    });
+}
+
 function setCatchAllAddress(domain, addresses, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(addresses));
@@ -1031,13 +1114,25 @@ function sendTestMail(domain, to, callback) {
     });
 }
 
-function listMailboxes(domain, page, perPage, callback) {
+function listMailboxes(domain, search, page, perPage, callback) {
     assert.strictEqual(typeof domain, 'string');
+    assert(typeof search === 'string' || search === null);
     assert.strictEqual(typeof page, 'number');
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    mailboxdb.listMailboxes(domain, page, perPage, function (error, result) {
+    mailboxdb.listMailboxes(domain, search, page, perPage, function (error, result) {
+        if (error) return callback(error);
+
+        callback(null, result);
+    });
+}
+
+function getMailboxCount(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    mailboxdb.getMailboxCount(domain, function (error, result) {
         if (error) return callback(error);
 
         callback(null, result);
@@ -1110,31 +1205,25 @@ function updateMailboxOwner(name, domain, userId, auditSource, callback) {
     });
 }
 
-function removeMailbox(name, domain, auditSource, callback) {
+function removeMailbox(name, domain, options, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    mailboxdb.del(name, domain, function (error) {
-        if (error) return callback(error);
+    const deleteMailFunc = options.deleteMails ? shell.sudo.bind(null, 'removeMailbox', [ REMOVE_MAILBOX, `${name}@${domain}` ], {}) : (next) => next();
 
-        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_REMOVE, auditSource, { name, domain });
+    deleteMailFunc(function (error) {
+        if (error) return callback(new BoxError(BoxError.FS_ERROR, `Error removing mailbox: ${error.message}`));
 
-        callback(null);
-    });
-}
+        mailboxdb.del(name, domain, function (error) {
+            if (error) return callback(error);
 
-function listAliases(domain, page, perPage, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof page, 'number');
-    assert.strictEqual(typeof perPage, 'number');
-    assert.strictEqual(typeof callback, 'function');
+            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_REMOVE, auditSource, { name, domain });
 
-    mailboxdb.listAliases(domain, page, perPage, function (error, result) {
-        if (error) return callback(error);
-
-        callback(null, result);
+            callback();
+        });
     });
 }
 
@@ -1161,12 +1250,15 @@ function setAliases(name, domain, aliases, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     for (var i = 0; i < aliases.length; i++) {
-        aliases[i] = aliases[i].toLowerCase();
+        let name = aliases[i].name.toLowerCase();
+        let domain = aliases[i].domain.toLowerCase();
 
-        var error = validateName(aliases[i]);
+        let error = validateName(name);
         if (error) return callback(error);
-    }
 
+        if (!validator.isEmail(`${name}@${domain}`)) return callback(new BoxError(BoxError.BAD_FIELD, `Invalid email: ${name}@${domain}`));
+        aliases[i] = { name, domain };
+    }
     mailboxdb.setAliasesForName(name, domain, aliases, function (error) {
         if (error) return callback(error);
 
@@ -1174,11 +1266,14 @@ function setAliases(name, domain, aliases, callback) {
     });
 }
 
-function getLists(domain, callback) {
+function getLists(domain, search, page, perPage, callback) {
     assert.strictEqual(typeof domain, 'string');
+    assert(typeof search === 'string' || search === null);
+    assert.strictEqual(typeof page, 'number');
+    assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    mailboxdb.getLists(domain, function (error, result) {
+    mailboxdb.getLists(domain, search, page, perPage, function (error, result) {
         if (error) return callback(error);
 
         callback(null, result);
@@ -1197,10 +1292,11 @@ function getList(name, domain, callback) {
     });
 }
 
-function addList(name, domain, members, auditSource, callback) {
+function addList(name, domain, members, membersOnly, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof name, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1213,19 +1309,20 @@ function addList(name, domain, members, auditSource, callback) {
         if (!validator.isEmail(members[i])) return callback(new BoxError(BoxError.BAD_FIELD, 'Invalid mail member: ' + members[i]));
     }
 
-    mailboxdb.addList(name, domain, members, function (error) {
+    mailboxdb.addList(name, domain, members, membersOnly, function (error) {
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_MAIL_LIST_ADD, auditSource, { name, domain, members });
+        eventlog.add(eventlog.ACTION_MAIL_LIST_ADD, auditSource, { name, domain, members, membersOnly });
 
         callback();
     });
 }
 
-function updateList(name, domain, members, auditSource, callback) {
+function updateList(name, domain, members, membersOnly, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1241,10 +1338,10 @@ function updateList(name, domain, members, auditSource, callback) {
     getList(name, domain, function (error, result) {
         if (error) return callback(error);
 
-        mailboxdb.updateList(name, domain, members, function (error) {
+        mailboxdb.updateList(name, domain, members, membersOnly, function (error) {
             if (error) return callback(error);
 
-            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldMembers: result.members, members });
+            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldMembers: result.members, members, membersOnly });
 
             callback(null);
         });
@@ -1266,6 +1363,7 @@ function removeList(name, domain, auditSource, callback) {
     });
 }
 
+// resolves the members of a list. i.e the lists and aliases
 function resolveList(listName, listDomain, callback) {
     assert.strictEqual(typeof listName, 'string');
     assert.strictEqual(typeof listDomain, 'string');
@@ -1296,18 +1394,21 @@ function resolveList(listName, listDomain, callback) {
                 visited.push(member);
 
                 mailboxdb.get(memberName, memberDomain, function (error, entry) {
-                    if (error && error.reason == BoxError.NOT_FOUND) { result.push(member); return iteratorCallback(); }
+                    if (error && error.reason == BoxError.NOT_FOUND) { result.push(member); return iteratorCallback(); } // let it bounce
                     if (error) return iteratorCallback(error);
 
-                    if (entry.type === mailboxdb.TYPE_MAILBOX) { result.push(member); return iteratorCallback(); }
-                    // no need to resolve alias because we only allow one level and within same domain
-                    if (entry.type === mailboxdb.TYPE_ALIAS) { result.push(`${entry.aliasTarget}@${entry.domain}`); return iteratorCallback(); }
+                    if (entry.type === mailboxdb.TYPE_MAILBOX) { // concrete mailbox
+                        result.push(member);
+                    } else if (entry.type === mailboxdb.TYPE_ALIAS) { // resolve aliases
+                        toResolve = toResolve.concat(`${entry.aliasName}@${entry.aliasDomain}`);
+                    } else { // resolve list members
+                        toResolve = toResolve.concat(entry.members);
+                    }
 
-                    toResolve = toResolve.concat(entry.members);
                     iteratorCallback();
                 });
             }, function (error) {
-                callback(error, result);
+                callback(error, result, list);
             });
         });
     });

src/mail_templates/app_down.ejs

@@ -7,8 +7,8 @@ The application '<%= title %>' installed at <%= appFqdn %> is not responding.
 This is most likely a problem in the application.
 
 To resolve this, you can try the following:
-* Restart the app by opening the app's web terminal - https://cloudron.io/documentation/apps/#web-terminal
-* Restore the app to the latest backup - https://cloudron.io/documentation/backups/#restoring-an-app
+* Restart the app by opening the app's web terminal - https://docs.cloudron.io/apps/#web-terminal
+* Restore the app to the latest backup - https://docs.cloudron.io/backups/#restoring-an-app
 * Contact us via <%= supportEmail %> or https://forum.cloudron.io
 
 

src/mail_templates/backup_failed.ejs

@@ -2,7 +2,7 @@
 
 Dear <%= cloudronName %> Admin,
 
-Cloudron failed to create a complete backup. Please see https://cloudron.io/documentation/troubleshooting/#backups
+Cloudron failed to create a complete backup. Please see https://docs.cloudron.io/troubleshooting/#backups
 for troubleshooting.
 
 Logs for this failure are available at <%= logUrl %>

src/mail_templates/certificate_renewal_error.ejs

@@ -8,7 +8,7 @@ The Cloudron will attempt to renew the certificate every 12 hours
 until the certificate expires (at which point it will switch to
 using the fallback certificate).
 
-See https://cloudron.io/documentation/troubleshooting/#certificates to
+See https://docs.cloudron.io/troubleshooting/#certificates to
 double check if your server is configured correctly to obtain certificates
 via Let's Encrypt.
 

src/mail_templates/oom_event.ejs

@@ -6,8 +6,8 @@ Dear <%= cloudronName %> Admin,
 
 If this message appears repeatedly, give the app more memory.
 
-* To increase an app's memory limit - https://cloudron.io/documentation/apps/#increasing-the-memory-limit-of-an-app
-* To increase a service's memory limit - https://cloudron.io/documentation/troubleshooting/#services
+* To increase an app's memory limit - https://docs.cloudron.io/apps/#memory-limit
+* To increase a service's memory limit - https://docs.cloudron.io/troubleshooting/#services
 
 Out of memory event:
 

src/mail_templates/password_reset.ejs

@@ -8,7 +8,7 @@ be reset. If you did not request this reset, please ignore this message.
 To reset your password, please visit the following page:
 <%- resetLink %>
 
-
+Please note that the password reset link will expire in 24 hours.
 
 Powered by https://cloudron.io
 
@@ -29,6 +29,10 @@ Powered by https://cloudron.io
     <a href="<%= resetLink %>">Click to reset your password</a>
 </p>
 
+<br/>
+
+Please note that the password reset link will expire in 24 hours.
+
 <br/>
 <br/>
 

src/mail_templates/welcome_user.ejs

@@ -11,6 +11,7 @@ Follow the link to get started.
 You are receiving this email because you were invited by <%= invitor.email %>.
 <% } %>
 
+Please note that the invite link will expire in 7 days.
 
 Powered by https://cloudron.io
 
@@ -36,6 +37,9 @@ Powered by https://cloudron.io
         You are receiving this email because you were invited by <%= invitor.email %>.
     <% } %>
 
+    <br/>
+
+        Please note that the invite link will expire in 7 days.
     <br/>
 
     Powered by <a href="https://cloudron.io">Cloudron</a>

src/mailboxdb.js

@@ -1,32 +1,32 @@
 'use strict';
 
 exports = module.exports = {
-    addMailbox: addMailbox,
-    addList: addList,
+    addMailbox,
+    addList,
 
-    updateMailboxOwner: updateMailboxOwner,
-    updateList: updateList,
-    del: del,
+    updateMailboxOwner,
+    updateList,
+    del,
 
-    listAliases: listAliases,
-    listMailboxes: listMailboxes,
-    getLists: getLists,
+    getMailboxCount,
+    listMailboxes,
+    getLists,
 
-    listAllMailboxes: listAllMailboxes,
+    listAllMailboxes,
 
-    get: get,
-    getMailbox: getMailbox,
-    getList: getList,
-    getAlias: getAlias,
+    get,
+    getMailbox,
+    getList,
+    getAlias,
 
-    getAliasesForName: getAliasesForName,
-    setAliasesForName: setAliasesForName,
+    getAliasesForName,
+    setAliasesForName,
 
-    getByOwnerId: getByOwnerId,
-    delByOwnerId: delByOwnerId,
-    delByDomain: delByDomain,
+    getByOwnerId,
+    delByOwnerId,
+    delByDomain,
 
-    updateName: updateName,
+    updateName,
 
     _clear: clear,
 
@@ -38,15 +38,18 @@ exports = module.exports = {
 var assert = require('assert'),
     BoxError = require('./boxerror.js'),
     database = require('./database.js'),
+    mysql = require('mysql'),
     safe = require('safetydance'),
     util = require('util');
 
-var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'aliasTarget', 'creationTime', 'membersJson', 'domain' ].join(',');
+var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'aliasName', 'aliasDomain', 'creationTime', 'membersJson', 'membersOnly', 'domain' ].join(',');
 
 function postProcess(data) {
     data.members = safe.JSON.parse(data.membersJson) || [ ];
     delete data.membersJson;
 
+    data.membersOnly = !!data.membersOnly;
+
     return data;
 }
 
@@ -78,14 +81,15 @@ function updateMailboxOwner(name, domain, ownerId, callback) {
     });
 }
 
-function addList(name, domain, members, callback) {
+function addList(name, domain, members, membersOnly, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, membersJson) VALUES (?, ?, ?, ?, ?)',
-        [ name, exports.TYPE_LIST, domain, 'admin', JSON.stringify(members) ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, membersJson, membersOnly) VALUES (?, ?, ?, ?, ?, ?)',
+        [ name, exports.TYPE_LIST, domain, 'admin', JSON.stringify(members), membersOnly ], function (error) {
             if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'mailbox already exists'));
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -93,14 +97,15 @@ function addList(name, domain, members, callback) {
         });
 }
 
-function updateList(name, domain, members, callback) {
+function updateList(name, domain, members, membersOnly, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('UPDATE mailboxes SET membersJson = ? WHERE name = ? AND domain = ?',
-        [ JSON.stringify(members), name, domain ], function (error, result) {
+    database.query('UPDATE mailboxes SET membersJson = ?, membersOnly = ? WHERE name = ? AND domain = ?',
+        [ JSON.stringify(members), membersOnly, name, domain ], function (error, result) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
             if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Mailbox not found'));
 
@@ -123,7 +128,7 @@ function del(name, domain, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     // deletes aliases as well
-    database.query('DELETE FROM mailboxes WHERE (name=? OR aliasTarget = ?) AND domain = ?', [ name, name, domain ], function (error, result) {
+    database.query('DELETE FROM mailboxes WHERE ((name=? AND domain=?) OR (aliasName = ? AND aliasDomain=?))', [ name, domain, name, domain ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Mailbox not found'));
 
@@ -200,20 +205,35 @@ function getMailbox(name, domain, callback) {
         });
 }
 
-function listMailboxes(domain, page, perPage, callback) {
+function getMailboxCount(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT COUNT(*) AS total FROM mailboxes WHERE type = ? AND domain = ?', [ exports.TYPE_MAILBOX, domain ], function (error, results) {
+        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+
+        callback(null, results[0].total);
+    });
+}
+
+function listMailboxes(domain, search, page, perPage, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert(typeof search === 'string' || search === null);
     assert.strictEqual(typeof page, 'number');
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query(`SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? AND domain = ? ORDER BY name LIMIT ${(page-1)*perPage},${perPage}`,
-        [ exports.TYPE_MAILBOX, domain ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+    let query = `SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? AND domain = ?`;
+    if (search) query += ' AND (name LIKE ' + mysql.escape('%' + search + '%') + ')';
+    query += 'ORDER BY name LIMIT ?,?';
 
-            results.forEach(function (result) { postProcess(result); });
+    database.query(query, [ exports.TYPE_MAILBOX, domain, (page-1)*perPage, perPage ], function (error, results) {
+        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-            callback(null, results);
-        });
+        results.forEach(function (result) { postProcess(result); });
+
+        callback(null, results);
+    });
 }
 
 function listAllMailboxes(page, perPage, callback) {
@@ -221,8 +241,8 @@ function listAllMailboxes(page, perPage, callback) {
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query(`SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? ORDER BY name LIMIT ${(page-1)*perPage},${perPage}`,
-        [ exports.TYPE_MAILBOX ], function (error, results) {
+    database.query(`SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? ORDER BY name LIMIT ?,?`,
+        [ exports.TYPE_MAILBOX, (page-1)*perPage, perPage ], function (error, results) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
             results.forEach(function (result) { postProcess(result); });
@@ -231,18 +251,25 @@ function listAllMailboxes(page, perPage, callback) {
         });
 }
 
-function getLists(domain, callback) {
+function getLists(domain, search, page, perPage, callback) {
     assert.strictEqual(typeof domain, 'string');
+    assert(typeof search === 'string' || search === null);
+    assert.strictEqual(typeof page, 'number');
+    assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE type = ? AND domain = ?',
-        [ exports.TYPE_LIST, domain ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+    let query = `SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? AND domain = ?`;
+    if (search) query += ' AND (name LIKE ' + mysql.escape('%' + search + '%') + ' OR membersJson LIKE ' + mysql.escape('%' + search + '%') + ')';
 
-            results.forEach(function (result) { postProcess(result); });
+    query += 'ORDER BY name LIMIT ?,?';
 
-            callback(null, results);
-        });
+    database.query(query, [ exports.TYPE_LIST, domain, (page-1)*perPage, perPage ], function (error, results) {
+        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+
+        results.forEach(function (result) { postProcess(result); });
+
+        callback(null, results);
+    });
 }
 
 function getList(name, domain, callback) {
@@ -285,10 +312,10 @@ function setAliasesForName(name, domain, aliases, callback) {
 
         var queries = [];
         // clear existing aliases
-        queries.push({ query: 'DELETE FROM mailboxes WHERE aliasTarget = ? AND domain = ? AND type = ?', args: [ name, domain, exports.TYPE_ALIAS ] });
+        queries.push({ query: 'DELETE FROM mailboxes WHERE aliasName = ? AND aliasDomain = ? AND type = ?', args: [ name, domain, exports.TYPE_ALIAS ] });
         aliases.forEach(function (alias) {
-            queries.push({ query: 'INSERT INTO mailboxes (name, type, domain, aliasTarget, ownerId) VALUES (?, ?, ?, ?, ?)',
-                args: [ alias, exports.TYPE_ALIAS, domain, name, results[0].ownerId ] });
+            queries.push({ query: 'INSERT INTO mailboxes (name, domain, type, aliasName, aliasDomain, ownerId) VALUES (?, ?, ?, ?, ?, ?)',
+                args: [ alias.name, alias.domain, exports.TYPE_ALIAS, name, domain, results[0].ownerId ] });
         });
 
         database.transaction(queries, function (error) {
@@ -311,27 +338,10 @@ function getAliasesForName(name, domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT name FROM mailboxes WHERE type = ? AND aliasTarget = ? AND domain = ? ORDER BY name',
+    database.query('SELECT name, domain FROM mailboxes WHERE type = ? AND aliasName = ? AND aliasDomain = ? ORDER BY name',
         [ exports.TYPE_ALIAS, name, domain ], function (error, results) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-            results = results.map(function (r) { return r.name; });
-            callback(null, results);
-        });
-}
-
-function listAliases(domain, page, perPage, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof page, 'number');
-    assert.strictEqual(typeof perPage, 'number');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query(`SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE domain = ? AND type = ? ORDER BY name LIMIT ${(page-1)*perPage},${perPage}`,
-        [ domain, exports.TYPE_ALIAS ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-            results.forEach(function (result) { postProcess(result); });
-
             callback(null, results);
         });
 }

src/maildb.js

@@ -17,7 +17,7 @@ var assert = require('assert'),
     database = require('./database.js'),
     safe = require('safetydance');
 
-var MAILDB_FIELDS = [ 'domain', 'enabled', 'mailFromValidation', 'catchAllJson', 'relayJson', 'dkimSelector' ].join(',');
+var MAILDB_FIELDS = [ 'domain', 'enabled', 'mailFromValidation', 'catchAllJson', 'relayJson', 'dkimSelector', 'bannerJson' ].join(',');
 
 function postProcess(data) {
     data.enabled = !!data.enabled; // int to boolean
@@ -29,6 +29,9 @@ function postProcess(data) {
     data.relay = safe.JSON.parse(data.relayJson) || { provider: 'cloudron-smtp' };
     delete data.relayJson;
 
+    data.banner = safe.JSON.parse(data.bannerJson) || { text: null, html: null };
+    delete data.bannerJson;
+
     return data;
 }
 
@@ -74,8 +77,8 @@ function update(domain, data, callback) {
     var args = [ ];
     var fields = [ ];
     for (var k in data) {
-        if (k === 'catchAll') {
-            fields.push('catchAllJson = ?');
+        if (k === 'catchAll' || k === 'banner') {
+            fields.push(`${k}Json = ?`);
             args.push(JSON.stringify(data[k]));
         } else if (k === 'relay') {
             fields.push('relayJson = ?');

src/network.js

@@ -0,0 +1,56 @@
+'use strict';
+
+exports = module.exports = {
+    getBlocklist,
+    setBlocklist
+};
+
+const assert = require('assert'),
+    BoxError = require('./boxerror.js'),
+    ipaddr = require('ipaddr.js'),
+    path = require('path'),
+    paths = require('./paths.js'),
+    safe = require('safetydance'),
+    settings = require('./settings.js'),
+    shell = require('./shell.js'),
+    validator = require('validator');
+
+const SET_BLOCKLIST_CMD = path.join(__dirname, 'scripts/setblocklist.sh');
+
+function getBlocklist(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    const data = safe.fs.readFileSync(paths.FIREWALL_BLOCKLIST_FILE, 'utf8');
+    callback(null, data);
+}
+
+function setBlocklist(blocklist, auditSource, callback) {
+    assert.strictEqual(typeof blocklist, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const parsedIp = ipaddr.process(auditSource.ip);
+
+    for (const line of blocklist.split('\n')) {
+        if (!line || line.startsWith('#')) continue;
+        const rangeOrIP = line.trim();
+        if (!validator.isIP(rangeOrIP) && !validator.isIPRange(rangeOrIP)) return callback(new BoxError(BoxError.BAD_FIELD, `${rangeOrIP} is not a valid IP or range`));
+
+        if (rangeOrIP.indexOf('/') === -1) {
+            if (auditSource.ip === rangeOrIP) return callback(new BoxError(BoxError.BAD_FIELD, `${rangeOrIP} includes client IP. Cannot block yourself`));
+        } else {
+            const parsedRange = ipaddr.parseCIDR(rangeOrIP);
+            if (parsedIp.match(parsedRange)) return callback(new BoxError(BoxError.BAD_FIELD, `${rangeOrIP} includes client IP. Cannot block yourself`));
+        }
+    }
+
+    if (settings.isDemo()) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
+
+    if (!safe.fs.writeFileSync(paths.FIREWALL_BLOCKLIST_FILE, blocklist + '\n', 'utf8')) return callback(new BoxError(BoxError.FS_ERROR, safe.error.message));
+
+    shell.sudo('setBlocklist', [ SET_BLOCKLIST_CMD ], {}, function (error) {
+        if (error) return callback(new BoxError(BoxError.IPTABLES_ERROR, `Error setting blocklist: ${error.message}`));
+
+        callback();
+    });
+}

src/nginxconfig.ejs

@@ -6,54 +6,61 @@ map $http_upgrade $connection_upgrade {
 
 # http server
 server {
-    listen       80;
+<% if (endpoint === 'ip' || endpoint === 'setup') { -%>
+    listen 80 default_server;
+    server_name _;
 <% if (hasIPv6) { -%>
-    listen       [::]:80;
+    listen [::]:80 default_server;
 <% } -%>
-
-<% if (vhost) { -%>
-    server_name  <%= vhost %>;
 <% } else { -%>
-    # IP based access from collectd or initial cloudron setup. TODO: match the IPv6 address
-    server_name  "~^\d+\.\d+\.\d+\.\d+$";
-
-    # collectd
-    location /nginx_status {
-        stub_status on;
-        access_log off;
-        allow 127.0.0.1;
-        deny all;
-    }
+    listen 80;
+    server_name  <%= vhost %>;
+<% if (hasIPv6) { -%>
+    listen [::]:80;
+<% } -%>
 <% } -%>
 
-    # acme challenges (for cert renewal where the vhost config exists)
+    server_tokens off; # hide version
+
+    # acme challenges
     location /.well-known/acme-challenge/ {
         default_type text/plain;
         alias /home/yellowtent/platformdata/acme/;
     }
 
+    # for default server, serve the splash page. for other endpoints, redirect to HTTPS
     location / {
-        # redirect everything to HTTPS
+<% if ( endpoint === 'admin' || endpoint === 'setup' ) { %>
         return 301 https://$host$request_uri;
+<% } else if ( endpoint === 'app' ) { %>
+        return 301 https://$host$request_uri;
+<% } else if ( endpoint === 'redirect' ) { %>
+        return 301 https://<%= redirectTo %>$request_uri;
+<% } else if ( endpoint === 'ip' ) { %>
+        root <%= sourceDir %>/dashboard/dist;
+        try_files /splash.html =404;
+<% } %>
     }
 }
 
 # https server
 server {
-<% if (vhost) { -%>
-    server_name  <%= vhost %>;
-    listen       443 http2;
+<% if (endpoint === 'ip' || endpoint === 'setup') { -%>
+    listen       443 ssl http2 default_server;
+    server_name _;
 <% if (hasIPv6) { -%>
-    listen       [::]:443 http2;
+    listen       [::]:443 ssl http2 default_server;
 <% } -%>
 <% } else { -%>
-    listen       443 http2 default_server;
+    listen       443 ssl http2;
+    server_name  <%= vhost %>;
 <% if (hasIPv6) { -%>
-    listen       [::]:443 http2 default_server;
+    listen       [::]:443 ssl http2;
 <% } -%>
 <% } -%>
 
-    ssl                  on;
+    server_tokens off; # hide version
+
     # paths are relative to prefix and not to this file
     ssl_certificate      <%= certFilePath %>;
     ssl_certificate_key  <%= keyFilePath %>;
@@ -65,7 +72,7 @@ server {
     # https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices#25-use-forward-secrecy
     # ciphers according to https://ssl-config.mozilla.org/#server=nginx&version=1.14.0&config=intermediate&openssl=1.1.1&guideline=5.4
     ssl_protocols TLSv1.2 TLSv1.3;
-    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256;
     ssl_prefer_server_ciphers off;
 
     ssl_dhparam /home/yellowtent/boxdata/dhparams.pem;
@@ -93,7 +100,7 @@ server {
     # enable for proxied requests as well
     gzip_proxied any;
 
-<% if ( endpoint === 'admin' ) { -%>
+<% if ( endpoint === 'admin' || endpoint === 'ip' || endpoint === 'setup' ) { -%>
     # CSP headers for the admin/dashboard resources
     add_header Content-Security-Policy "default-src 'none'; frame-src 'self' cloudron.io *.cloudron.io; connect-src wss: https: 'self' *.cloudron.io; script-src https: 'self' 'unsafe-inline' 'unsafe-eval'; img-src * data:; style-src https: 'unsafe-inline'; object-src 'none'; font-src https: 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'self';";
 <% } else { %>
@@ -135,8 +142,21 @@ server {
         # internal means this is for internal routing and cannot be accessed as URL from browser
         internal;
     }
-    location /appstatus.html {
-        internal;
+
+    location @wellknown-upstream {
+<% if ( endpoint === 'admin' ) { %>
+        proxy_pass http://127.0.0.1:3000;
+<% } else if ( endpoint === 'app' ) { %>
+        proxy_pass http://127.0.0.1:<%= port %>;
+<% } else if ( endpoint === 'redirect' ) { %>
+        return 302 https://<%= redirectTo %>$request_uri;
+<% } %>
+    }
+
+    # user defined .well-known resources
+    location ~ ^/.well-known/(.*)$ {
+        root /home/yellowtent/boxdata/well-known/$host;
+        try_files /$1 @wellknown-upstream;
     }
 
     location / {
@@ -157,7 +177,7 @@ server {
         }
 <% } %>
 
-<% if ( endpoint === 'admin' ) { %>
+<% if ( endpoint === 'admin' || endpoint === 'setup' ) { %>
         location /api/ {
             proxy_pass   http://127.0.0.1:3000;
             client_max_body_size 1m;
@@ -180,6 +200,11 @@ server {
             client_max_body_size 0;
         }
 
+        location ~ ^/api/v1/apps/.*/files/ {
+            proxy_pass   http://127.0.0.1:3000;
+            client_max_body_size 0;
+        }
+
         # graphite paths (uncomment block below and visit /graphite-web/dashboard)
         # remember to comment out the CSP policy as well to access the graphite dashboard
         # location ~ ^/graphite-web/ {
@@ -197,6 +222,11 @@ server {
         # redirect everything to the app. this is temporary because there is no way
         # to clear a permanent redirect on the browser
         return 302 https://<%= redirectTo %>$request_uri;
+<% } else if ( endpoint === 'ip' ) { %>
+        location / {
+            root   <%= sourceDir %>/dashboard/dist;
+            try_files /splash.html =404;
+        }
 <% } %>
     }
 }

src/notifications.js

@@ -155,12 +155,12 @@ function oomEvent(eventId, app, addon, containerId, event, callback) {
     let title, message, program;
     if (app) {
         program = `App ${app.fqdn}`;
-        title = `The application ${app.fqdn} (${app.manifest.title}) ran out of memory.`;
-        message = 'The application has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://cloudron.io/documentation/apps/#increasing-the-memory-limit-of-an-app)';
+        title = `The application at ${app.fqdn} ran out of memory.`;
+        message = 'The application has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://docs.cloudron.io/apps/#memory-limit)';
     } else if (addon) {
         program = `${addon.name} service`;
         title = `The ${addon.name} service ran out of memory`;
-        message = 'The service has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://cloudron.io/documentation/troubleshooting/#services)';
+        message = 'The service has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://docs.cloudron.io/troubleshooting/#services)';
     } else { // this never happens currently
         program = `Container ${containerId}`;
         title = `The container ${containerId} ran out of memory`;
@@ -181,7 +181,7 @@ function appUp(eventId, app, callback) {
 
     actionForAllAdmins([], function (admin, done) {
         mailer.appUp(admin.email, app);
-        add(admin.id, eventId, `App ${app.fqdn} is back online`, `The application ${app.manifest.title} installed at ${app.fqdn} is back online.`, done);
+        add(admin.id, eventId, `App ${app.fqdn} is back online`, `The application installed at ${app.fqdn} is back online.`, done);
     }, callback);
 }
 
@@ -192,7 +192,7 @@ function appDied(eventId, app, callback) {
 
     actionForAllAdmins([], function (admin, callback) {
         mailer.appDied(admin.email, app);
-        add(admin.id, eventId, `App ${app.fqdn} is down`, `The application ${app.manifest.title} installed at ${app.fqdn} is not responding.`, callback);
+        add(admin.id, eventId, `App ${app.fqdn} is down`, `The application installed at ${app.fqdn} is not responding.`, callback);
     }, callback);
 }
 
@@ -201,17 +201,19 @@ function appUpdated(eventId, app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
+    if (!app.appStoreId) return callback(); // skip notification of dev apps
+
     const tmp = app.manifest.description.match(/<upstream>(.*)<\/upstream>/i);
     const upstreamVersion = (tmp && tmp[1]) ? tmp[1] : '';
     const title = upstreamVersion ? `${app.manifest.title} at ${app.fqdn} updated to ${upstreamVersion} (package version ${app.manifest.version})`
         : `${app.manifest.title} at ${app.fqdn} updated to package version ${app.manifest.version}`;
 
     actionForAllAdmins([], function (admin, done) {
-        add(admin.id, eventId, title, `The application ${app.manifest.title} installed at https://${app.fqdn} was updated.\n\nChangelog:\n${app.manifest.changelog}\n`, function (error) {
+        add(admin.id, eventId, title, `The application installed at https://${app.fqdn} was updated.\n\nChangelog:\n${app.manifest.changelog}\n`, function (error) {
             if (error) return callback(error);
 
             mailer.appUpdated(admin.email, app, function (error) {
-                if (error) console.error('Failed to send app updated email', error); // non fatal
+                if (error) debug('appUpdated: Failed to send app updated email', error); // non fatal
                 done();
             });
         });
@@ -239,7 +241,7 @@ function boxUpdateError(eventId, errorMessage, callback) {
 
     actionForAllAdmins([], function (admin, done) {
         mailer.boxUpdateError(admin.email, errorMessage);
-        add(admin.id, eventId, 'Cloudron update failed', `Failed to update Cloudron: ${errorMessage}. Update will be retried in 4 hours`, done);
+        add(admin.id, eventId, 'Cloudron update failed', `Failed to update Cloudron: ${errorMessage}.`, done);
     }, callback);
 }
 
@@ -263,7 +265,7 @@ function backupFailed(eventId, taskId, errorMessage, callback) {
 
     actionForAllAdmins([], function (admin, callback) {
         mailer.backupFailed(admin.email, errorMessage, `${settings.adminOrigin()}/logs.html?taskId=${taskId}`);
-        add(admin.id, eventId, 'Backup failed', `Backup failed: ${errorMessage}. Logs are available [here](/logs.html?taskId=${taskId}). Will be retried in 4 hours`, callback);
+        add(admin.id, eventId, 'Backup failed', `Backup failed: ${errorMessage}. Logs are available [here](/logs.html?taskId=${taskId}).`, callback);
     }, callback);
 }
 
@@ -273,7 +275,7 @@ function alert(id, title, message, callback) {
     assert.strictEqual(typeof message, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`alert: id=${id} title=${title} message=${message}`);
+    debug(`alert: id=${id} title=${title}`);
 
     const acknowledged = !message;
 
@@ -301,7 +303,7 @@ function alert(id, title, message, callback) {
             });
         });
     }, function (error) {
-        if (error) console.error(error);
+        if (error) debug('alert: error notifying', error);
 
         callback();
     });
@@ -338,7 +340,6 @@ function onEvent(id, action, source, data, callback) {
         return appUp(id, data.app, callback);
 
     case eventlog.ACTION_APP_UPDATE_FINISH:
-        if (!data.app.appStoreId) return callback(); // skip notification of dev apps
         return appUpdated(id, data.app, callback);
 
     case eventlog.ACTION_CERTIFICATE_RENEWAL:

src/paths.js

@@ -17,7 +17,6 @@ exports = module.exports = {
     CLOUDRON_DEFAULT_AVATAR_FILE: path.join(__dirname + '/../assets/avatar.png'),
     INFRA_VERSION_FILE: path.join(baseDir(), 'platformdata/INFRA_VERSION'),
 
-    LICENSE_FILE: '/etc/cloudron/LICENSE',
     PROVIDER_FILE: '/etc/cloudron/PROVIDER',
 
     PLATFORM_DATA_DIR: path.join(baseDir(), 'platformdata'),
@@ -47,10 +46,13 @@ exports = module.exports = {
     CLOUDRON_AVATAR_FILE: path.join(baseDir(), 'boxdata/avatar.png'),
     UPDATE_CHECKER_FILE: path.join(baseDir(), 'boxdata/updatechecker.json'),
     ADDON_TURN_SECRET_FILE: path.join(baseDir(), 'boxdata/addon-turn-secret'),
+    FIREWALL_BLOCKLIST_FILE: path.join(baseDir(), 'boxdata/firewall/blocklist.txt'),
+    FIREWALL_CONFIG_FILE: path.join(baseDir(), 'boxdata/firewall-config.json'),
 
     LOG_DIR: path.join(baseDir(), 'platformdata/logs'),
     TASKS_LOG_DIR: path.join(baseDir(), 'platformdata/logs/tasks'),
     CRASH_LOG_DIR: path.join(baseDir(), 'platformdata/logs/crash'),
+    BOX_LOG_FILE: path.join(baseDir(), 'platformdata/logs/box.log'),
 
     GHOST_USER_FILE: path.join(baseDir(), 'platformdata/cloudron_ghost.json'),
 

src/platform.js

@@ -2,7 +2,7 @@
 
 exports = module.exports = {
     start: start,
-    stop: stop,
+    stopAllTasks: stopAllTasks,
 
     // exported for testing
     _isReady: false
@@ -26,8 +26,6 @@ var addons = require('./addons.js'),
     tasks = require('./tasks.js'),
     _ = require('underscore');
 
-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
-
 function start(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -45,7 +43,7 @@ function start(callback) {
     if (_.isEqual(infra, existingInfra)) {
         debug('platform is uptodate at version %s', infra.version);
 
-        onPlatformReady();
+        onPlatformReady(false /* !infraChanged */);
 
         return callback();
     }
@@ -56,9 +54,8 @@ function start(callback) {
     if (error) return callback(error);
 
     async.series([
-        stopContainers.bind(null, existingInfra),
-        // mark app state before we start addons. this gives the db import logic a chance to mark an app as errored
-        startApps.bind(null, existingInfra),
+        (next) => { if (existingInfra.version !== infra.version) removeAllContainers(existingInfra, next); else next(); },
+        markApps.bind(null, existingInfra), // mark app state before we start addons. this gives the db import logic a chance to mark an app as errored
         graphs.startGraphite.bind(null, existingInfra),
         sftp.startSftp.bind(null, existingInfra),
         addons.startServices.bind(null, existingInfra),
@@ -68,41 +65,36 @@ function start(callback) {
 
         locker.unlock(locker.OP_PLATFORM_START);
 
-        onPlatformReady();
+        onPlatformReady(true /* infraChanged */);
 
         callback();
     });
 }
 
-function stop(callback) {
+function stopAllTasks(callback) {
     tasks.stopAllTasks(callback);
 }
 
-function onPlatformReady() {
-    debug('onPlatformReady: platform is ready');
+function onPlatformReady(infraChanged) {
+    debug(`onPlatformReady: platform is ready. infra changed: ${infraChanged}`);
     exports._isReady = true;
 
-    apps.schedulePendingTasks(NOOP_CALLBACK);
+    let tasks = [ apps.schedulePendingTasks ];
+    if (infraChanged) tasks.push(applyPlatformConfig, pruneInfraImages);
 
-    applyPlatformConfig(NOOP_CALLBACK);
-    pruneInfraImages(NOOP_CALLBACK);
+    async.series(async.reflectAll(tasks), function (error, results) {
+        results.forEach((result, idx) => {
+            if (result.error) debug(`Startup task at index ${idx} failed: ${result.error.message}`);
+        });
+    });
 }
 
 function applyPlatformConfig(callback) {
-    // scale back db containers, if possible. this is retried because updating memory constraints can fail
-    // with failed to write to memory.memsw.limit_in_bytes: write /sys/fs/cgroup/memory/docker/xx/memory.memsw.limit_in_bytes: device or resource busy
+    settings.getPlatformConfig(function (error, platformConfig) {
+        if (error) return callback(error);
 
-    async.retry({ times: 10, interval: 5 * 60 * 1000 }, function (retryCallback) {
-        settings.getPlatformConfig(function (error, platformConfig) {
-            if (error) return retryCallback(error);
-
-            addons.updateServiceConfig(platformConfig, function (error) {
-                if (error) debug('Error updating services. Will rety in 5 minutes', platformConfig, error);
-
-                retryCallback(error);
-            });
-        });
-    }, callback);
+        addons.updateServiceConfig(platformConfig, callback);
+    });
 }
 
 function pruneInfraImages(callback) {
@@ -130,41 +122,37 @@ function pruneInfraImages(callback) {
     }, callback);
 }
 
-function stopContainers(existingInfra, callback) {
-    // always stop addons to restart them on any infra change, regardless of minor or major update
-    if (existingInfra.version !== infra.version) {
-        debug('stopping all containers for infra upgrade');
-        async.series([
-            shell.exec.bind(null, 'stopContainers', 'docker ps -qa --filter \'label=isCloudronManaged\' | xargs --no-run-if-empty docker stop'),
-            shell.exec.bind(null, 'stopContainers', 'docker ps -qa --filter \'label=isCloudronManaged\' | xargs --no-run-if-empty docker rm -f')
-        ], callback);
-    } else {
-        assert(typeof infra.images, 'object');
-        var changedAddons = [ ];
-        for (var imageName in existingInfra.images) { // do not use infra.images because we can only stop things which are existing
-            if (infra.images[imageName].tag !== existingInfra.images[imageName].tag) changedAddons.push(imageName);
-        }
+function removeAllContainers(existingInfra, callback) {
+    debug('removeAllContainers: removing all containers for infra upgrade');
 
-        debug('stopContainer: stopping addons for incremental infra update: %j', changedAddons);
-        let filterArg = changedAddons.map(function (c) { return `--filter 'name=${c}'`; }).join(' '); // name=c matches *c*. required for redis-{appid}
-        // ignore error if container not found (and fail later) so that this code works across restarts
-        async.series([
-            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker stop || true`),
-            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker rm -f || true`)
-        ], callback);
-    }
+    async.series([
+        shell.exec.bind(null, 'removeAllContainers', 'docker ps -qa --filter \'label=isCloudronManaged\' | xargs --no-run-if-empty docker stop'),
+        shell.exec.bind(null, 'removeAllContainers', 'docker ps -qa --filter \'label=isCloudronManaged\' | xargs --no-run-if-empty docker rm -f')
+    ], callback);
 }
 
-function startApps(existingInfra, callback) {
+function markApps(existingInfra, callback) {
     if (existingInfra.version === 'none') { // cloudron is being restored from backup
-        debug('startApps: restoring installed apps');
+        debug('markApps: restoring installed apps');
         apps.restoreInstalledApps(callback);
     } else if (existingInfra.version !== infra.version) {
-        debug('startApps: reconfiguring installed apps');
+        debug('markApps: reconfiguring installed apps');
         reverseProxy.removeAppConfigs(); // should we change the cert location, nginx will not start
         apps.configureInstalledApps(callback);
     } else {
-        debug('startApps: apps are already uptodate');
-        callback();
+        let changedAddons = [];
+        if (infra.images.mysql.tag !== existingInfra.images.mysql.tag) changedAddons.push('mysql');
+        if (infra.images.postgresql.tag !== existingInfra.images.postgresql.tag) changedAddons.push('postgresql');
+        if (infra.images.mongodb.tag !== existingInfra.images.mongodb.tag) changedAddons.push('mongodb');
+        if (infra.images.redis.tag !== existingInfra.images.redis.tag) changedAddons.push('redis');
+
+        if (changedAddons.length) {
+            // restart apps if docker image changes since the IP changes and any "persistent" connections fail
+            debug(`markApps: changedAddons: ${JSON.stringify(changedAddons)}`);
+            apps.restartAppsUsingAddons(changedAddons, callback);
+        } else {
+            debug('markApps: apps are already uptodate');
+            callback();
+        }
     }
 }

src/provision.js

@@ -4,25 +4,20 @@ exports = module.exports = {
     setup: setup,
     restore: restore,
     activate: activate,
-    getStatus: getStatus,
-
-    autoRegister: autoRegister
+    getStatus: getStatus
 };
 
-var appstore = require('./appstore.js'),
-    assert = require('assert'),
+var assert = require('assert'),
     async = require('async'),
     backups = require('./backups.js'),
     BoxError = require('./boxerror.js'),
+    branding = require('./branding.js'),
     constants = require('./constants.js'),
     cloudron = require('./cloudron.js'),
     debug = require('debug')('box:provision'),
     domains = require('./domains.js'),
     eventlog = require('./eventlog.js'),
-    fs = require('fs'),
     mail = require('./mail.js'),
-    paths = require('./paths.js'),
-    safe = require('safetydance'),
     semver = require('semver'),
     settings = require('./settings.js'),
     sysinfo = require('./sysinfo.js'),
@@ -53,27 +48,6 @@ function setProgress(task, message, callback) {
     callback();
 }
 
-function autoRegister(domain, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    if (!fs.existsSync(paths.LICENSE_FILE)) return callback();
-
-    const license = safe.fs.readFileSync(paths.LICENSE_FILE, 'utf8');
-    if (!license) return callback(new BoxError(BoxError.LICENSE_ERROR, 'Cannot read license'));
-
-    debug('Auto-registering cloudron');
-
-    appstore.registerWithLicense(license.trim(), domain, function (error) {
-        if (error && error.reason !== BoxError.CONFLICT) { // not already registered
-            debug('Failed to auto-register cloudron', error);
-            return callback(new BoxError(BoxError.LICENSE_ERROR, 'Failed to auto-register Cloudron with license. Please contact support@cloudron.io'));
-        }
-
-        callback();
-    });
-}
-
 function unprovision(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -81,7 +55,7 @@ function unprovision(callback) {
 
     // TODO: also cancel any existing configureWebadmin task
     async.series([
-        settings.setAdmin.bind(null, '', ''),
+        settings.setAdminLocation.bind(null, '', ''),
         mail.clearDomains,
         domains.clear
     ], callback);
@@ -125,25 +99,24 @@ function setup(dnsConfig, sysinfoConfig, auditSource, callback) {
                 dkimSelector: 'cloudron'
             };
 
-            domains.add(domain, data, auditSource, function (error) {
+            async.series([
+                settings.setMailLocation.bind(null, domain, `${constants.ADMIN_LOCATION}.${domain}`), // default mail location. do this before we add the domain for upserting mail DNS
+                domains.add.bind(null, domain, data, auditSource),
+                sysinfo.testConfig.bind(null, sysinfoConfig)
+            ], function (error) {
                 if (error) return done(error);
 
-                sysinfo.testConfig(sysinfoConfig, function (error) {
-                    if (error) return done(error);
+                callback(); // now that args are validated run the task in the background
 
-                    callback(); // now that args are validated run the task in the background
-
-                    async.series([
-                        autoRegister.bind(null, domain),
-                        settings.setSysinfoConfig.bind(null, sysinfoConfig),
-                        domains.prepareDashboardDomain.bind(null, domain, auditSource, (progress) => setProgress('setup', progress.message, NOOP_CALLBACK)),
-                        cloudron.setDashboardDomain.bind(null, domain, auditSource),
-                        setProgress.bind(null, 'setup', 'Done'),
-                        eventlog.add.bind(null, eventlog.ACTION_PROVISION, auditSource, { })
-                    ], function (error) {
-                        gProvisionStatus.setup.active = false;
-                        gProvisionStatus.setup.errorMessage = error ? error.message : '';
-                    });
+                async.series([
+                    settings.setSysinfoConfig.bind(null, sysinfoConfig),
+                    cloudron.setupDnsAndCert.bind(null, constants.ADMIN_LOCATION, domain, auditSource, (progress) => setProgress('setup', progress.message, NOOP_CALLBACK)),
+                    cloudron.setDashboardDomain.bind(null, domain, auditSource),
+                    setProgress.bind(null, 'setup', 'Done'),
+                    eventlog.add.bind(null, eventlog.ACTION_PROVISION, auditSource, { })
+                ], function (error) {
+                    gProvisionStatus.setup.active = false;
+                    gProvisionStatus.setup.errorMessage = error ? error.message : '';
                 });
             });
         });
@@ -190,7 +163,7 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
     assert.strictEqual(typeof callback, 'function');
 
     if (!semver.valid(version)) return callback(new BoxError(BoxError.BAD_FIELD, 'version is not a valid semver', { field: 'version' }));
-    if (constants.VERSION !== version) return callback(new BoxError(BoxError.BAD_STATE, `Run cloudron-setup with --version ${version} to restore from this backup`));
+    if (constants.VERSION !== version) return callback(new BoxError(BoxError.BAD_STATE, `Run "cloudron-setup --version ${version}" on a fresh Ubuntu installation to restore from this backup`));
 
     if (gProvisionStatus.setup.active || gProvisionStatus.restore.active) return callback(new BoxError(BoxError.BAD_STATE, 'Already setting up or restoring'));
 
@@ -206,9 +179,16 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
         if (error) return done(error);
         if (activated) return done(new BoxError(BoxError.CONFLICT, 'Already activated. Restore with a fresh Cloudron installation.'));
 
-        backups.testConfig(backupConfig, function (error) {
+        backups.testProviderConfig(backupConfig, function (error) {
             if (error) return done(error);
 
+            if ('password' in backupConfig) {
+                backupConfig.encryption = backups.generateEncryptionKeysSync(backupConfig.password);
+                delete backupConfig.password;
+            } else {
+                backupConfig.encryption = null;
+            }
+
             sysinfo.testConfig(sysinfoConfig, function (error) {
                 if (error) return done(error);
 
@@ -220,7 +200,13 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
                     setProgress.bind(null, 'restore', 'Downloading backup'),
                     backups.restore.bind(null, backupConfig, backupId, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK)),
                     settings.setSysinfoConfig.bind(null, sysinfoConfig),
-                    cloudron.setupDashboard.bind(null, auditSource, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK)),
+                    (done) => {
+                        const adminDomain = settings.adminDomain(); // load this fresh from after the backup.restore
+                        async.series([
+                            cloudron.setupDnsAndCert.bind(null, constants.ADMIN_LOCATION, adminDomain, auditSource, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK)),
+                            cloudron.setDashboardDomain.bind(null, adminDomain, auditSource)
+                        ], done);
+                    },
                     settings.setBackupCredentials.bind(null, backupConfig), // update just the credentials and not the policy and flags
                     eventlog.add.bind(null, eventlog.ACTION_RESTORE, auditSource, { backupId }),
                 ], function (error) {
@@ -247,11 +233,11 @@ function getStatus(callback) {
                 version: constants.VERSION,
                 apiServerOrigin: settings.apiServerOrigin(), // used by CaaS tool
                 webServerOrigin: settings.webServerOrigin(), // used by CaaS tool
-                provider: settings.provider(),
                 cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-                footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
+                footer: branding.renderFooter(allSettings[settings.FOOTER_KEY] || constants.FOOTER),
                 adminFqdn: settings.adminDomain() ? settings.adminFqdn() : null,
                 activated: activated,
+                provider: settings.provider() // used by setup wizard of marketplace images
             }, gProvisionStatus));
         });
     });