gcs: use delete concurrency

listAllMailboxes query was mangled

CHANGES

@@ -2065,3 +2065,129 @@
 * Add pre-defined variables ($CLOUDRON_APPID) for better post install messages
 * filemanager: show folder first
 
+[5.6.0]
+* Remove IP nginx configuration that redirects to dashboard after activation
+* dashboard: looks for search string in app title as well
+* Add vaapi caps for transcoding
+* Fix issue where the long mongodb database names where causing app indices of rocket.chat to overflow (> 127)
+* Do not resize swap if swap file exists. This means that users can now control how swap is allocated on their own.
+* SFTP: fix issue where parallel rebuilds would cause an error
+* backups: make part size configurable
+* mail: set max email size
+* mail: allow mail server location to be set
+* spamassassin: custom configs and wl/bl
+* Do not automatically update to unstable release
+* scheduler: reduce container churn
+* mail: add API to set banner
+* Fix bug where systemd 237 ignores --nice value in systemd-run
+* postgresql: enable uuid-ossp extension
+* firewall: add blocklist
+* HTTP URLs now redirect directly to the HTTPS of the final domain
+* linode: Add singapore region
+* ovh: add sydney region
+* s3: makes multi-part copies in parallel
+
+[5.6.1]
+* Blocklists are now stored in a text file instead of json
+* regenerate nginx configs
+
+[5.6.2]
+* Update docker to 19.03.12
+* Fix sorting of user listing in the UI
+* namecheap: fix crash when server returns invalid response
+* unlink ghost file automatically on successful login
+* Bump mysql addon connection limit to 200
+* Fix install issue where `/dev/dri` may not be present
+* import: when importing filesystem backups, the input box is a path
+* firewall: fix race condition where blocklist was not added in correct position in the FORWARD chain
+* services: fix issue where services where scaled up/down too fast
+* turn: realm variable was not updated properly on dashboard change
+* nginx: add splash pages for IP based browser access
+* Give services panel a separate top-level view
+* Add app state filter
+* gcs: copy concurrency was not used
+* Mention why an app update cannot be applied and provide shortcut to start the app if stopped
+* Remove version from footer into the setting view
+* Give services panel a separate top-level view
+* postgresql: set collation order explicity when creating database to C.UTF-8 (for confluence)
+* rsync: fix error while goes missing when syncing
+* Pre-select app domain by default in the redirection drop down
+* robots: preseve leading and trailing whitespaces/newlines
+
+[5.6.3]
+* Fix postgres locale issue
+
+[6.0.0]
+* Focal support
+* Reduce duration of self-signed certs to 800 days
+* Better backup config filename when downloading
+* branding: footer can have template variables like %YEAR% and %VERSION%
+* sftp: secure the API with a token
+* filemanager: Add extract context menu item
+* Do not download docker images if present locally
+* sftp: disable access to non-admins by default
+* postgresql: whitelist pgcrypto extension for loomio
+* filemanager: Add new file creation action and collapse new and upload actions
+* rsync: add warning to remove lifecycle rules
+* Add volume management
+* backups: adjust node's heap size based on memory limit
+* s3: diasble per-chunk timeout
+* logs: more descriptive log file names on download
+* collectd: remove collectd config when app stopped (and add it back when started)
+* Apps can optionally request an authwall to be installed in front of them
+* mailbox can now owned by a group
+* linode: enable dns provider in setup view
+* dns: apps can now use the dns port
+* httpPaths: allow apps to specify forwarding from custom paths to container ports (for OLS)
+* add elasticemail smtp relay option
+* mail: add option to fts using solr
+* mail: change the namespace separator of new installations to /
+* mail: enable acl
+* Disable THP
+* filemanager: allow download dirs as zip files
+* aws: add china region
+* security: fix issue where apps could send with any username (but valid password)
+* i18n support
+
+[6.0.1]
+* app: add export route
+* mail: on location change, fix lock up when one or more domains have invalid credentials
+* mail: fix crash because of write after timeout closure
+* scaleway: fix installation issue where THP is not enabled in kernel
+
+[6.1.0]
+* mail: update haraka to 2.8.27. this fixes zero-length queue file crash
+* update: set/unset appStoreId from the update route
+* proxyauth: Do not follow redirects
+* proxyauth: add 2FA
+* appstore: add category translations
+* appstore: add media category
+* prepend the version to assets when sourcing to avoid cache hits on update
+* filemanger: list volumes of the app
+* Display upload size and size progress
+* nfs: chown the backups for hardlinks to work
+* remove user add/remove/role change email notifications
+* persist update indicator across restarts
+* cloudron-setup: add --generate-setup-token
+* dashboard: pass accessToken query param to automatically login
+* wellknown: add a way to set well known docs
+* oom: notification mails have links to dashboard
+* collectd: do not install xorg* packages
+* apptask: backup/restore tasks now use the backup memory limit configuration
+* eventlog: add logout event
+* mailbox: include alias in mailbox search
+* proxyAuth: add path exclusion
+* turn: fix for CVE-2020-26262
+* app password: fix regression where apps are not listed anymore in the UI
+* Support for multiDomain apps (domain aliases)
+* netcup: add dns provider
+* Container swap size is now dynamically determined based on system RAM/swap ratio
+
+[6.1.1]
+* Fix bug where platform does not start if memory limits could not be applied
+
+[6.1.2]
+* App disk usage was not shown in graphs
+* Email autoconfig
+* Fix SOGo login
+

README.md

@@ -29,9 +29,9 @@ anyone to effortlessly host web applications on their server on their own terms.
 * Trivially migrate to another server keeping your apps and data (for example, switch your
   infrastructure provider or move to a bigger server).
 
-* Comprehensive [REST API](https://cloudron.io/documentation/developer/api/).
+* Comprehensive [REST API](https://docs.cloudron.io/api/).
 
-* [CLI](https://cloudron.io/documentation/cli/) to configure apps.
+* [CLI](https://docs.cloudron.io/custom-apps/cli/) to configure apps.
 
 * Alerts, audit logs, graphs, dns management ... and much more
 
@@ -41,15 +41,37 @@ Try our demo at https://my.demo.cloudron.io (username: cloudron password: cloudr
 
 ## Installing
 
-[Install script](https://cloudron.io/documentation/installation/) - [Pricing](https://cloudron.io/pricing.html)
+[Install script](https://docs.cloudron.io/installation/) - [Pricing](https://cloudron.io/pricing.html)
 
 **Note:** This repo is a small part of what gets installed on your server - there is
 the dashboard, database addons, graph container, base image etc. Cloudron also relies
 on external services such as the App Store for apps to be installed. As such, don't
 clone this repo and npm install and expect something to work.
 
+## Development
+
+This is the backend code of Cloudron. The frontend code is [here](https://git.cloudron.io/cloudron/dashboard).
+
+The way to develop is to first install a full instance of Cloudron in a VM. Then you can use the [hotfix](https://git.cloudron.io/cloudron/cloudron-machine)
+tool to patch the VM with the latest code.
+
+```
+SSH_PASSPHRASE=sshkeypassword cloudron-machine hotfix --cloudron my.example.com --release 6.0.0 --ssh-key keyname
+```
+
+## License
+
+Please note that the Cloudron code is under a source-available license. This is not the same as an
+open source license but ensures the code is available for introspection (and hacking!).
+
+## Contributions
+
+Just to give some heads up, we are a bit restrictive in merging changes. We are a small team and
+would like to keep our maintenance burden low. It might be best to discuss features first in the [forum](https://forum.cloudron.io),
+to also figure out how many other people will use it to justify maintenance for a feature.
+
 ## Support
 
-* [Documentation](https://cloudron.io/documentation/)
+* [Documentation](https://docs.cloudron.io/)
 * [Forum](https://forum.cloudron.io/)
 

baseimage/initializeBaseUbuntuImage.sh

@@ -29,9 +29,12 @@ debconf-set-selections <<< 'mysql-server mysql-server/root_password_again passwo
 
 # this enables automatic security upgrades (https://help.ubuntu.com/community/AutomaticSecurityUpdates)
 # resolvconf is needed for unbound to work property after disabling systemd-resolved in 18.04
+
 gpg_package=$([[ "${ubuntu_version}" == "16.04" ]] && echo "gnupg" || echo "gpg")
-apt-get -y install \
+mysql_package=$([[ "${ubuntu_version}" == "20.04" ]] && echo "mysql-server-8.0" || echo "mysql-server-5.7")
+apt-get -y install --no-install-recommends \
     acl \
+    apparmor \
     build-essential \
     cifs-utils \
     cron \
@@ -39,11 +42,12 @@ apt-get -y install \
     debconf-utils \
     dmsetup \
     $gpg_package \
+    ipset \
     iptables \
     libpython2.7 \
     linux-generic \
     logrotate \
-    mysql-server-5.7 \
+    $mysql_package \
     openssh-server \
     pwgen \
     resolvconf \
@@ -51,16 +55,17 @@ apt-get -y install \
     tzdata \
     unattended-upgrades \
     unbound \
+    unzip \
     xfsprogs
 
 echo "==> installing nginx for xenial for TLSv3 support"
-curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
+curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-2~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
 # apt install with install deps (as opposed to dpkg -i)
 apt install -y /tmp/nginx.deb
 rm /tmp/nginx.deb
 
 # on some providers like scaleway the sudo file is changed and we want to keep the old one
-apt-get -o Dpkg::Options::="--force-confold" install -y sudo
+apt-get -o Dpkg::Options::="--force-confold" install -y --no-install-recommends sudo
 
 # this ensures that unattended upgades are enabled, if it was disabled during ubuntu install time (see #346)
 # debconf-set-selection of unattended-upgrades/enable_auto_updates + dpkg-reconfigure does not work
@@ -71,7 +76,7 @@ mkdir -p /usr/local/node-10.18.1
 curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxf - --strip-components=1 -C /usr/local/node-10.18.1
 ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
 ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
-apt-get install -y python   # Install python which is required for npm rebuild
+apt-get install -y --no-install-recommends python   # Install python which is required for npm rebuild
 [[ "$(python --version 2>&1)" == "Python 2.7."* ]] || die "Expecting python version to be 2.7.x"
 
 # https://docs.docker.com/engine/installation/linux/ubuntulinux/
@@ -82,9 +87,9 @@ mkdir -p /etc/systemd/system/docker.service.d
 echo -e "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2" > /etc/systemd/system/docker.service.d/cloudron.conf
 
 # there are 3 packages for docker - containerd, CLI and the daemon
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.2-3_amd64.deb" -o /tmp/containerd.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.13-2_amd64.deb" -o /tmp/containerd.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
 # apt install with install deps (as opposed to dpkg -i)
 apt install -y /tmp/containerd.deb  /tmp/docker-ce-cli.deb /tmp/docker.deb
 rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
@@ -97,7 +102,7 @@ fi
 
 # do not upgrade grub because it might prompt user and break this script
 echo "==> Enable memory accounting"
-apt-get -y --no-upgrade install grub2-common
+apt-get -y --no-upgrade --no-install-recommends install grub2-common
 sed -e 's/^GRUB_CMDLINE_LINUX="\(.*\)"$/GRUB_CMDLINE_LINUX="\1 cgroup_enable=memory swapaccount=1 panic_on_oops=1 panic=5"/' -i /etc/default/grub
 update-grub
 
@@ -116,26 +121,37 @@ for image in ${images}; do
 done
 
 echo "==> Install collectd"
-if ! apt-get install -y libcurl3-gnutls collectd collectd-utils; then
+# without this, libnotify4 will install gnome-shell
+apt-get install -y libnotify4 --no-install-recommends
+if ! apt-get install -y --no-install-recommends libcurl3-gnutls collectd collectd-utils; then
     # FQDNLookup is true in default debian config. The box code has a custom collectd.conf that fixes this
     echo "Failed to install collectd. Presumably because of http://mailman.verplant.org/pipermail/collectd/2015-March/006491.html"
     sed -e 's/^FQDNLookup true/FQDNLookup false/' -i /etc/collectd/collectd.conf
 fi
+# https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1872281
+[[ "${ubuntu_version}" == "20.04" ]] && echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
 
+# some hosts like atlantic install ntp which conflicts with timedatectl. https://serverfault.com/questions/1024770/ubuntu-20-04-time-sync-problems-and-possibly-incorrect-status-information
 echo "==> Configuring host"
 sed -e 's/^#NTP=/NTP=0.ubuntu.pool.ntp.org 1.ubuntu.pool.ntp.org 2.ubuntu.pool.ntp.org 3.ubuntu.pool.ntp.org/' -i /etc/systemd/timesyncd.conf
+if systemctl is-active ntp; then
+    systemctl stop ntp
+    apt purge -y ntp
+fi
 timedatectl set-ntp 1
 # mysql follows the system timezone
 timedatectl set-timezone UTC
 
 echo "==> Adding sshd configuration warning"
-sed -e '/Port 22/ i # NOTE: Cloudron only supports moving SSH to port 202. See https://cloudron.io/documentation/security/#securing-ssh-access' -i /etc/ssh/sshd_config
+sed -e '/Port 22/ i # NOTE: Cloudron only supports moving SSH to port 202. See https://docs.cloudron.io/security/#securing-ssh-access' -i /etc/ssh/sshd_config
 
 # https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1701068
 echo "==> Disabling motd news"
-sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
+if [ -f "/etc/default/motd-news" ]; then
+    sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
+fi
 
-# Disable bind for good measure (on online.net, kimsufi servers these are pre-installed and conflicts with unbound)
+# Disable bind for good measure (on online.net, kimsufi servers these are pre-installed)
 systemctl stop bind9 || true
 systemctl disable bind9 || true
 
@@ -147,7 +163,7 @@ systemctl disable dnsmasq || true
 systemctl stop postfix || true
 systemctl disable postfix || true
 
-# on ubuntu 18.04, this is the default. this requires resolvconf for DNS to work further after the disable
+# on ubuntu 18.04 and 20.04, this is the default. this requires resolvconf for DNS to work further after the disable
 systemctl stop systemd-resolved || true
 systemctl disable systemd-resolved || true
 
@@ -156,4 +172,3 @@ systemctl disable systemd-resolved || true
 ip6=$([[ -s /proc/net/if_inet6 ]] && echo "yes" || echo "no")
 echo -e "server:\n\tinterface: 127.0.0.1\n\tdo-ip6: ${ip6}" > /etc/unbound/unbound.conf.d/cloudron-network.conf
 systemctl restart unbound
-

box.js

@@ -7,28 +7,24 @@ let async = require('async'),
     fs = require('fs'),
     ldap = require('./src/ldap.js'),
     paths = require('./src/paths.js'),
-    server = require('./src/server.js'),
-    util = require('util');
+    proxyAuth = require('./src/proxyauth.js'),
+    server = require('./src/server.js');
 
 const NOOP_CALLBACK = function () { };
 
 function setupLogging(callback) {
     if (process.env.BOX_ENV === 'test') return callback();
 
-    fs.open(paths.BOX_LOG_FILE, 'a', function (error, fd) {
-        if (error) return callback(error);
+    var logfileStream = fs.createWriteStream(paths.BOX_LOG_FILE, { flags:'a' });
+    process.stdout.write = process.stderr.write = logfileStream.write.bind(logfileStream);
 
-        require('debug').log = function (...args) {
-            fs.appendFileSync(fd, util.format(...args) + '\n');
-        };
-
-        callback();
-    });
+    callback();
 }
 
 async.series([
     setupLogging,
-    server.start,
+    server.start, // do this first since it also inits the database
+    proxyAuth.start,
     ldap.start,
     dockerProxy.start
 ], function (error) {
@@ -37,11 +33,14 @@ async.series([
         process.exit(1);
     }
 
-    const debug = require('debug')('box:box'); // require this here so that logging handler is already setup
+    // require those here so that logging handler is already setup
+    require('supererror');
+    const debug = require('debug')('box:box');
 
     process.on('SIGINT', function () {
         debug('Received SIGINT. Shutting down.');
 
+        proxyAuth.stop(NOOP_CALLBACK);
         server.stop(NOOP_CALLBACK);
         ldap.stop(NOOP_CALLBACK);
         dockerProxy.stop(NOOP_CALLBACK);
@@ -51,11 +50,17 @@ async.series([
     process.on('SIGTERM', function () {
         debug('Received SIGTERM. Shutting down.');
 
+        proxyAuth.stop(NOOP_CALLBACK);
         server.stop(NOOP_CALLBACK);
         ldap.stop(NOOP_CALLBACK);
         dockerProxy.stop(NOOP_CALLBACK);
         setTimeout(process.exit.bind(process), 3000);
     });
 
+    process.on('uncaughtException', function (error) {
+        console.error((error && error.stack) ? error.stack : error);
+        setTimeout(process.exit.bind(process, 1), 3000);
+    });
+
     console.log(`Cloudron is up and running. Logs are at ${paths.BOX_LOG_FILE}`); // this goes to journalctl
 });

migrations/20200816002703-settings-add-mail-domain-and-fqdn.js

@@ -0,0 +1,23 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="admin_domain"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        const adminDomain = results[0].value;
+
+        async.series([
+            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_domain', adminDomain ]),
+            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_fqdn', `my.${adminDomain}` ])
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_domain"'),
+        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_fqdn"'),
+    ], callback);
+};

migrations/20200820050259-settings-merge-autoupdate-pattern.js

@@ -0,0 +1,22 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    db.runSql('SELECT * FROM settings WHERE name=?', ['app_autoupdate_pattern'], function (error, results) {
+        if (error || results.length === 0) return callback(error); // will use defaults from box code
+
+        var updatePattern = results[0].value; // use app auto update patter for the box as well
+
+        async.series([
+            db.runSql.bind(db, 'START TRANSACTION;'),
+            db.runSql.bind(db, 'DELETE FROM settings WHERE name=? OR name=?', ['app_autoupdate_pattern', 'box_autoupdate_pattern']),
+            db.runSql.bind(db, 'INSERT settings (name, value) VALUES(?, ?)', ['autoupdate_pattern', updatePattern]),
+            db.runSql.bind(db, 'COMMIT')
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200824004859-mail-add-bannerJson.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mail ADD COLUMN bannerJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mail DROP COLUMN bannerJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200914172345-split-firewall-config.js

@@ -0,0 +1,27 @@
+'use strict';
+
+const OLD_FIREWALL_CONFIG_JSON = '/home/yellowtent/boxdata/firewall-config.json';
+const PORTS_FILE = '/home/yellowtent/boxdata/firewall/ports.json';
+const BLOCKLIST_FILE = '/home/yellowtent/boxdata/firewall/blocklist.txt';
+
+const fs = require('fs');
+
+exports.up = function (db, callback) {
+    if (!fs.existsSync(OLD_FIREWALL_CONFIG_JSON)) return callback();
+
+    try {
+        const dataJson = fs.readFileSync(OLD_FIREWALL_CONFIG_JSON, 'utf8');
+        const data = JSON.parse(dataJson);
+        fs.writeFileSync(BLOCKLIST_FILE, data.blocklist.join('\n') + '\n', 'utf8');
+        fs.writeFileSync(PORTS_FILE, JSON.stringify({ allowed_tcp_ports: data.allowed_tcp_ports }, null, 4), 'utf8');
+        fs.unlinkSync(OLD_FIREWALL_CONFIG_JSON);
+    } catch (error) {
+        console.log('Error migrating old firewall config', error);
+    }
+
+    callback();
+};
+
+exports.down = function (db, callback) {
+    callback();
+};

migrations/20201028052525-volumes-create-table.js

@@ -0,0 +1,40 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    var cmd1 = 'CREATE TABLE volumes(' +
+                'id VARCHAR(128) NOT NULL UNIQUE,' +
+                'name VARCHAR(256) NOT NULL UNIQUE,' +
+                'hostPath VARCHAR(1024) NOT NULL UNIQUE,' +
+                'creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,' +
+                'PRIMARY KEY (id)) CHARACTER SET utf8 COLLATE utf8_bin';
+
+    var cmd2 = 'CREATE TABLE appMounts(' +
+                 'appId VARCHAR(128) NOT NULL,' +
+                 'volumeId VARCHAR(128) NOT NULL,' +
+                 'readOnly BOOLEAN DEFAULT 1,' +
+                 'UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),' +
+                 'FOREIGN KEY(appId) REFERENCES apps(id),' +
+                 'FOREIGN KEY(volumeId) REFERENCES volumes(id)) CHARACTER SET utf8 COLLATE utf8_bin;';
+
+    db.runSql(cmd1, function (error) {
+        if (error) console.error(error);
+
+        db.runSql(cmd2, function (error) {
+            if (error) console.error(error);
+
+            db.runSql('ALTER TABLE apps DROP COLUMN bindsJson', callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('DROP TABLE appMounts', function (error) {
+        if (error) console.error(error);
+
+        db.runSql('DROP TABLE volumes', function (error) {
+            if (error) console.error(error);
+            callback(error);
+        });
+    });
+};
+

migrations/20201110174504-apps-add-proxyAuth.js

@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN proxyAuth BOOLEAN DEFAULT 0', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN proxyAuth', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20201113071728-mailboxes-add-ownerType.js

@@ -0,0 +1,18 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD COLUMN ownerType VARCHAR(16)'),
+        db.runSql.bind(db, 'UPDATE mailboxes SET ownerType=?', [ 'user' ]),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes MODIFY ownerType VARCHAR(16) NOT NULL'),
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes DROP COLUMN ownerType', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20201119072309-apps-remove-httpPort.js

@@ -0,0 +1,13 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE apps DROP COLUMN httpPort')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20201120212726-apps-add-containerIp.js

@@ -0,0 +1,29 @@
+'use strict';
+
+const async = require('async'),
+    iputils = require('../src/iputils.js');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN containerIp VARCHAR(16) UNIQUE', function (error) {
+        if (error) console.error(error);
+
+        let baseIp = iputils.intFromIp('172.18.16.0');
+
+        db.all('SELECT * FROM apps', function (error, apps) {
+            if (error) return callback(error);
+
+            async.eachSeries(apps, function (app, iteratorDone) {
+                const nextIp = iputils.ipFromInt(++baseIp);
+                db.runSql('UPDATE apps SET containerIp=? WHERE id=?', [ nextIp, app.id ], iteratorDone);
+            }, callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN containerIp', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20201204081504-settings-sftp-requireAdmin.js

@@ -0,0 +1,21 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM settings WHERE name=?', ['platform_config'], function (error, results) {
+        let value;
+        if (error || results.length === 0) {
+            value = { sftp: { requireAdmin: true } };
+        } else {
+            value = JSON.parse(results[0].value);
+            if (!value.sftp) value.sftp = {};
+            value.sftp.requireAdmin = true;
+        }
+
+        // existing installations may not even have the key. so use REPLACE instead of UPDATE
+        db.runSql('REPLACE INTO settings (name, value) VALUES (?, ?)', [ 'platform_config', JSON.stringify(value) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20201222182945-groupMembers-unique-members.js

@@ -0,0 +1,18 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'CREATE TABLE groupMembers_copy(groupId VARCHAR(128) NOT NULL, userId VARCHAR(128) NOT NULL, FOREIGN KEY(groupId) REFERENCES userGroups(id), FOREIGN KEY(userId) REFERENCES users(id), UNIQUE (groupId, userId)) CHARACTER SET utf8 COLLATE utf8_bin'),    // In mysql CREATE TABLE.. LIKE does not copy indexes
+        db.runSql.bind(db, 'INSERT INTO groupMembers_copy SELECT * FROM groupMembers GROUP BY groupId, userId'),
+        db.runSql.bind(db, 'DROP TABLE groupMembers'),
+        db.runSql.bind(db, 'ALTER TABLE groupMembers_copy RENAME TO groupMembers')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE groupMembers DROP INDEX groupMembers_member'),
+    ], callback);
+};

migrations/20201223014453-domains-add-wellKnownJson.js

@@ -0,0 +1,51 @@
+'use strict';
+
+const async = require('async'),
+    safe = require('safetydance');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE domains ADD COLUMN wellKnownJson TEXT', function (error) {
+        if (error) return callback(error);
+
+        // keep the paths around, so that we don't need to trigger a re-configure. the old nginx config will use the paths
+        // the new one will proxy calls to the box code
+        const WELLKNOWN_DIR = '/home/yellowtent/boxdata/well-known';
+        const output = safe.child_process.execSync('find . -type f -printf "%P\n"', { cwd: WELLKNOWN_DIR, encoding: 'utf8' });
+        if (!output) return callback();
+        const paths = output.trim().split('\n');
+        if (paths.length === 0) return callback(); // user didn't configure any well-known
+
+        let wellKnown = {};
+        for (let path of paths) {
+            const fqdn = path.split('/', 1)[0];
+            const loc = path.slice(fqdn.length+1);
+            const doc = safe.fs.readFileSync(`${WELLKNOWN_DIR}/${path}`, { encoding: 'utf8' });
+            if (!doc) continue;
+
+            wellKnown[fqdn] = {};
+            wellKnown[fqdn][loc] = doc;
+        }
+
+        console.log('Migrating well-known', JSON.stringify(wellKnown, null, 4));
+
+        async.eachSeries(Object.keys(wellKnown), function (fqdn, iteratorDone) {
+            db.runSql('UPDATE domains SET wellKnownJson=? WHERE domain=?', [ JSON.stringify(wellKnown[fqdn]), fqdn ], function (error, result) {
+                if (error) {
+                    console.error(error); // maybe the domain does not exist anymore
+                } else if (result.affectedRows === 0) {
+                    console.log(`Could not migrate wellknown as domain ${fqdn} is missing`);
+                }
+                iteratorDone();
+            });
+        }, function (error) {
+            callback(error);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE domains DROP COLUMN wellKnownJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20210120025243-settings-platform-config-memoryLimit.js

@@ -0,0 +1,23 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM settings WHERE name=?', ['platform_config'], function (error, results) {
+        if (error || results.length === 0) return callback(null);
+
+        let value = JSON.parse(results[0].value);
+
+        for (const serviceName of Object.keys(value)) {
+            const service = value[serviceName];
+            if (!service.memorySwap) continue;
+            service.memoryLimit = service.memorySwap;
+            delete service.memorySwap;
+            delete service.memory;
+        }
+
+        db.runSql('UPDATE settings SET value=? WHERE name=?', [ JSON.stringify(value), 'platform_config' ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210120025750-apps-servicesConfig-memoryLimit.js

@@ -0,0 +1,28 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM apps', function (error, apps) {
+        if (error) return callback(error);
+
+        async.eachSeries(apps, function (app, iteratorDone) {
+            if (!app.servicesConfigJson) return iteratorDone();
+
+            let servicesConfig = JSON.parse(app.servicesConfigJson);
+            for (const serviceName of Object.keys(servicesConfig)) {
+                const service = servicesConfig[serviceName];
+                if (!service.memorySwap) continue;
+                service.memoryLimit = service.memorySwap;
+                delete service.memorySwap;
+                delete service.memory;
+            }
+
+            db.runSql('UPDATE apps SET servicesConfigJson=? WHERE id=?', [ JSON.stringify(servicesConfig), app.id ], iteratorDone);
+        }, callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210121201822-settings-rename-platform-config-to-services-config.js

@@ -0,0 +1,9 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('UPDATE settings SET name=? WHERE name=?', [ 'services_config', 'platform_config' ], callback);
+};
+
+exports.down = function(db, callback) {
+    db.runSql('UPDATE settings SET name=? WHERE name=?', [ 'platform_config', 'services_config' ], callback);
+};

migrations/schema.sql

@@ -44,7 +44,8 @@ CREATE TABLE IF NOT EXISTS groupMembers(
     groupId VARCHAR(128) NOT NULL,
     userId VARCHAR(128) NOT NULL,
     FOREIGN KEY(groupId) REFERENCES userGroups(id),
-    FOREIGN KEY(userId) REFERENCES users(id));
+    FOREIGN KEY(userId) REFERENCES users(id),
+    UNIQUE (groupId, userId));
 
 CREATE TABLE IF NOT EXISTS tokens(
     id VARCHAR(128) NOT NULL UNIQUE,
@@ -65,9 +66,6 @@ CREATE TABLE IF NOT EXISTS apps(
     healthTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app last responded
     containerId VARCHAR(128),
     manifestJson TEXT,
-    httpPort INTEGER,                        // this is the nginx proxy port and not manifest.httpPort
-    location VARCHAR(128) NOT NULL,
-    domain VARCHAR(128) NOT NULL,
     accessRestrictionJson TEXT, // { users: [ ], groups: [ ] }
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app was installed
     updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,   // when the last app update was done
@@ -87,8 +85,8 @@ CREATE TABLE IF NOT EXISTS apps(
     dataDir VARCHAR(256) UNIQUE,
     taskId INTEGER, // current task
     errorJson TEXT,
-    bindsJson TEXT, // bind mounts
     servicesConfigJson TEXT, // app services configuration
+    containerIp VARCHAR(16) UNIQUE, // this is not-null because of ip allocation fails, user can 'repair'
 
     FOREIGN KEY(mailboxDomain) REFERENCES domains(domain),
     FOREIGN KEY(taskId) REFERENCES tasks(id),
@@ -150,6 +148,7 @@ CREATE TABLE IF NOT EXISTS domains(
     provider VARCHAR(16) NOT NULL,
     configJson TEXT, /* JSON containing the dns backend provider config */
     tlsConfigJson TEXT, /* JSON containing the tls provider config */
+    wellKnownJson TEXT, /* JSON containing well known docs for this domain */
 
     PRIMARY KEY (domain))
 
@@ -163,6 +162,7 @@ CREATE TABLE IF NOT EXISTS mail(
     mailFromValidation BOOLEAN DEFAULT 1,
     catchAllJson TEXT,
     relayJson TEXT,
+    bannerJson TEXT,
 
     dkimSelector VARCHAR(128) NOT NULL DEFAULT "cloudron",
 
@@ -182,6 +182,7 @@ CREATE TABLE IF NOT EXISTS mailboxes(
     name VARCHAR(128) NOT NULL,
     type VARCHAR(16) NOT NULL, /* 'mailbox', 'alias', 'list' */
     ownerId VARCHAR(128) NOT NULL, /* user id */
+    ownerType VARCHAR(16) NOT NULL,
     aliasName VARCHAR(128), /* the target name type is an alias */
     aliasDomain VARCHAR(128), /* the target domain */
     membersJson TEXT, /* members of a group. fully qualified */
@@ -238,4 +239,20 @@ CREATE TABLE IF NOT EXISTS appPasswords(
     PRIMARY KEY (id)
 );
 
+CREATE TABLE IF NOT EXISTS volumes(
+    id VARCHAR(128) NOT NULL UNIQUE,
+    name VARCHAR(256) NOT NULL UNIQUE,
+    hostPath VARCHAR(1024) NOT NULL UNIQUE,
+    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (id)
+);
+
+CREATE TABLE IF NOT EXISTS appMounts(
+    appId VARCHAR(128) NOT NULL,
+    volumeId VARCHAR(128) NOT NULL,
+    readOnly BOOLEAN DEFAULT 1,
+    UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),
+    FOREIGN KEY(appId) REFERENCES apps(id),
+    FOREIGN KEY(volumeId) REFERENCES volumes(id));
+
 CHARACTER SET utf8 COLLATE utf8_bin;

package.json

@@ -14,45 +14,49 @@
     "node": ">=4.0.0 <=4.1.1"
   },
   "dependencies": {
-    "@google-cloud/dns": "^1.1.0",
+    "@google-cloud/dns": "^1.2.9",
     "@google-cloud/storage": "^2.5.0",
     "@sindresorhus/df": "git+https://github.com/cloudron-io/df.git#type",
     "async": "^2.6.3",
-    "aws-sdk": "^2.685.0",
+    "aws-sdk": "^2.828.0",
+    "basic-auth": "^2.0.1",
     "body-parser": "^1.19.0",
-    "cloudron-manifestformat": "^5.5.0",
+    "cloudron-manifestformat": "^5.10.1",
     "connect": "^3.7.0",
     "connect-lastmile": "^2.0.0",
     "connect-timeout": "^1.9.0",
+    "cookie-parser": "^1.4.5",
     "cookie-session": "^1.4.0",
     "cron": "^1.8.2",
-    "db-migrate": "^0.11.11",
-    "db-migrate-mysql": "^1.1.10",
-    "debug": "^4.1.1",
+    "db-migrate": "^0.11.12",
+    "db-migrate-mysql": "^2.1.2",
+    "debug": "^4.3.1",
     "dockerode": "^2.5.8",
     "ejs": "^2.6.1",
-    "ejs-cli": "^2.2.0",
+    "ejs-cli": "^2.2.1",
     "express": "^4.17.1",
+    "ipaddr.js": "^2.0.0",
     "js-yaml": "^3.14.0",
     "json": "^9.0.6",
-    "ldapjs": "^1.0.2",
-    "lodash": "^4.17.15",
+    "jsonwebtoken": "^8.5.1",
+    "ldapjs": "^2.2.3",
+    "lodash": "^4.17.20",
     "lodash.chunk": "^4.2.0",
-    "mime": "^2.4.6",
-    "moment": "^2.26.0",
-    "moment-timezone": "^0.5.31",
+    "mime": "^2.5.0",
+    "moment": "^2.29.1",
+    "moment-timezone": "^0.5.32",
     "morgan": "^1.10.0",
-    "multiparty": "^4.2.1",
+    "multiparty": "^4.2.2",
+    "mustache-express": "^1.3.0",
     "mysql": "^2.18.1",
-    "nodemailer": "^6.4.6",
+    "nodemailer": "^6.4.17",
     "nodemailer-smtp-transport": "^2.7.4",
     "once": "^1.4.0",
-    "parse-links": "^0.1.0",
-    "pretty-bytes": "^5.3.0",
+    "pretty-bytes": "^5.5.0",
     "progress-stream": "^2.0.0",
     "proxy-middleware": "^0.15.0",
     "qrcode": "^1.4.4",
-    "readdirp": "^3.4.0",
+    "readdirp": "^3.5.0",
     "request": "^2.88.2",
     "rimraf": "^2.6.3",
     "s3-block-read-stream": "^0.5.0",
@@ -61,21 +65,22 @@
     "showdown": "^1.9.1",
     "speakeasy": "^2.0.0",
     "split": "^1.0.1",
-    "superagent": "^5.2.2",
+    "superagent": "^5.3.1",
+    "supererror": "^0.7.2",
     "tar-fs": "github:cloudron-io/tar-fs#ignore_stat_error",
-    "tar-stream": "^2.1.2",
+    "tar-stream": "^2.2.0",
     "tldjs": "^2.3.1",
-    "underscore": "^1.10.2",
+    "underscore": "^1.12.0",
     "uuid": "^3.4.0",
     "validator": "^11.0.0",
-    "ws": "^7.3.0",
+    "ws": "^7.4.2",
     "xml2js": "^0.4.23"
   },
   "devDependencies": {
     "expect.js": "*",
     "hock": "^1.4.1",
     "js2xmlparser": "^4.0.1",
-    "mocha": "^6.1.4",
+    "mocha": "^6.2.3",
     "mock-aws-s3": "git+https://github.com/cloudron-io/mock-aws-s3.git",
     "nock": "^10.0.6",
     "node-sass": "^4.14.1",

runTests

@@ -2,11 +2,11 @@
 
 set -eu
 
-readonly SOURCE_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly source_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 readonly DATA_DIR="${HOME}/.cloudron_test"
 readonly DEFAULT_TESTS="./src/test/*-test.js ./src/routes/test/*-test.js"
 
-! "${SOURCE_dir}/src/test/checkInstall" && exit 1
+! "${source_dir}/src/test/checkInstall" && exit 1
 
 # cleanup old data dirs some of those docker container data requires sudo to be removed
 echo "=> Provide root password to purge any leftover data in ${DATA_DIR} and load apparmor profile:"
@@ -22,19 +22,27 @@ fi
 mkdir -p ${DATA_DIR}
 cd ${DATA_DIR}
 mkdir -p appsdata
-mkdir -p boxdata/profileicons boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com
-mkdir -p platformdata/addons/mail platformdata/nginx/cert platformdata/nginx/applications platformdata/collectd/collectd.conf.d platformdata/addons platformdata/logrotate.d platformdata/backup platformdata/logs/tasks
+mkdir -p boxdata/profileicons boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com boxdata/sftp/ssh
+mkdir -p platformdata/addons/mail/banner platformdata/nginx/cert platformdata/nginx/applications platformdata/collectd/collectd.conf.d platformdata/addons platformdata/logrotate.d platformdata/backup platformdata/logs/tasks
+sudo mkdir -p /mnt/cloudron-test-music /media/cloudron-test-music # volume test
+
+# translations
+mkdir -p box/dashboard/dist/translation
+cp -r ${source_dir}/../dashboard/dist/translation/* box/dashboard/dist/translation
 
 # put cert
 echo "=> Generating a localhost selfsigned cert"
 openssl req -x509 -newkey rsa:2048 -keyout platformdata/nginx/cert/host.key -out platformdata/nginx/cert/host.cert -days 3650 -subj '/CN=localhost' -nodes -config <(cat /etc/ssl/openssl.cnf <(printf "\n[SAN]\nsubjectAltName=DNS:*.localhost"))
 
+# generate legacy key format for sftp
+ssh-keygen -m PEM -t rsa -f boxdata/sftp/ssh/ssh_host_rsa_key -q -N ""
+
 # clear out any containers
 echo "=> Delete all docker containers first"
 docker ps -qa | xargs --no-run-if-empty docker rm -f
 
 # create docker network (while the infra code does this, most tests skip infra setup)
-docker network create --subnet=172.18.0.0/16 cloudron || true
+docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 cloudron || true
 
 # create the same mysql server version to test with
 OUT=`docker inspect mysql-server` || true
@@ -59,7 +67,7 @@ echo "=> Ensure database"
 mysql -h"${MYSQL_IP}" -uroot -ppassword -e 'CREATE DATABASE IF NOT EXISTS box'
 
 echo "=> Run database migrations"
-cd "${SOURCE_dir}"
+cd "${source_dir}"
 BOX_ENV=test DATABASE_URL=mysql://root:password@${MYSQL_IP}/box node_modules/.bin/db-migrate up
 
 echo "=> Run tests with mocha"

scripts/cloudron-setup

@@ -43,30 +43,37 @@ fi
 initBaseImage="true"
 provider="generic"
 requestedVersion=""
+installServerOrigin="https://api.cloudron.io"
 apiServerOrigin="https://api.cloudron.io"
 webServerOrigin="https://cloudron.io"
 sourceTarballUrl=""
 rebootServer="true"
+setupToken=""
 
-args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot" -n "$0" -- "$@")
+args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot,generate-setup-token" -n "$0" -- "$@")
 eval set -- "${args}"
 
 while true; do
     case "$1" in
-    --help) echo "See https://cloudron.io/documentation/installation/ on how to install Cloudron"; exit 0;;
+    --help) echo "See https://docs.cloudron.io/installation/ on how to install Cloudron"; exit 0;;
     --provider) provider="$2"; shift 2;;
     --version) requestedVersion="$2"; shift 2;;
     --env)
         if [[ "$2" == "dev" ]]; then
             apiServerOrigin="https://api.dev.cloudron.io"
             webServerOrigin="https://dev.cloudron.io"
+            installServerOrigin="https://api.dev.cloudron.io"
         elif [[ "$2" == "staging" ]]; then
             apiServerOrigin="https://api.staging.cloudron.io"
             webServerOrigin="https://staging.cloudron.io"
+            installServerOrigin="https://api.staging.cloudron.io"
+        elif [[ "$2" == "unstable" ]]; then
+            installServerOrigin="https://api.dev.cloudron.io"
         fi
         shift 2;;
     --skip-baseimage-init) initBaseImage="false"; shift;;
     --skip-reboot) rebootServer="false"; shift;;
+    --generate-setup-token) setupToken="$(openssl rand -hex 10)"; shift;;
     --) break;;
     *) echo "Unknown option $1"; exit 1;;
     esac
@@ -80,8 +87,8 @@ fi
 
 # Only --help works with mismatched ubuntu
 ubuntu_version=$(lsb_release -rs)
-if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" ]]; then
-    echo "Cloudron requires Ubuntu 16.04 or 18.04" > /dev/stderr
+if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" && "${ubuntu_version}" != "20.04" ]]; then
+    echo "Cloudron requires Ubuntu 16.04, 18.04 or 20.04" > /dev/stderr
     exit 1
 fi
 
@@ -100,26 +107,20 @@ echo " Join us at https://forum.cloudron.io for any questions."
 echo ""
 
 if [[ "${initBaseImage}" == "true" ]]; then
-    echo "=> Installing software-properties-common"
-    if ! apt-get install -y software-properties-common &>> "${LOG_FILE}"; then
-        echo "Could not install software-properties-common (for add-apt-repository below). See ${LOG_FILE}"
-        exit 1
-    fi
-
     echo "=> Updating apt and installing script dependencies"
     if ! apt-get update &>> "${LOG_FILE}"; then
         echo "Could not update package repositories. See ${LOG_FILE}"
         exit 1
     fi
 
-    if ! DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install curl python3 ubuntu-standard -y &>> "${LOG_FILE}"; then
+    if ! DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install --no-install-recommends curl python3 ubuntu-standard software-properties-common -y &>> "${LOG_FILE}"; then
         echo "Could not install setup dependencies (curl). See ${LOG_FILE}"
         exit 1
     fi
 fi
 
 echo "=> Checking version"
-if ! releaseJson=$($curl -s "${apiServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
+if ! releaseJson=$($curl -s "${installServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
     echo "Failed to get release information"
     exit 1
 fi
@@ -157,6 +158,7 @@ fi
 echo "=> Installing version ${version} (this takes some time) ..."
 mkdir -p /etc/cloudron
 echo "${provider}" > /etc/cloudron/PROVIDER
+[[ ! -z "${setupToken}" ]] && echo "${setupToken}" > /etc/cloudron/SETUP_TOKEN
 
 if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" &>> "${LOG_FILE}"; then
     echo "Failed to install cloudron. See ${LOG_FILE} for details"
@@ -178,7 +180,12 @@ done
 if ! ip=$(curl -s --fail --connect-timeout 2 --max-time 2 https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
     ip='<IP>'
 fi
-echo -e "\n\n${GREEN}Visit https://${ip} and accept the self-signed certificate to finish setup.${DONE}\n"
+if [[ -z "${setupToken}" ]]; then
+    url="https://${ip}"
+else
+    url="https://${ip}/?setupToken=${setupToken}"
+fi
+echo -e "\n\n${GREEN}After reboot, visit ${url} and accept the self-signed certificate to finish setup.${DONE}\n"
 
 if [[ "${rebootServer}" == "true" ]]; then
     systemctl stop box mysql # sometimes mysql ends up having corrupt privilege tables

scripts/cloudron-support

@@ -37,12 +37,12 @@ while true; do
         # fall through
         ;&
     --owner-login)
-        admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' LIMIT 1" 2>/dev/null)
+        admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' AND username IS NOT NULL ORDER BY createdAt LIMIT 1" 2>/dev/null)
         admin_password=$(pwgen -1s 12)
         ghost_file=/home/yellowtent/platformdata/cloudron_ghost.json
         printf '{"%s":"%s"}\n' "${admin_username}" "${admin_password}" > "${ghost_file}"
         chown yellowtent:yellowtent "${ghost_file}" && chmod o-r,g-r "${ghost_file}"
-        echo "Login as ${admin_username} / ${admin_password} . Remove ${ghost_file} when done."
+        echo "Login as ${admin_username} / ${admin_password} . This password may only be used once. ${ghost_file} will be automatically removed after use."
         exit 0
         ;;
     --) break;;
@@ -57,7 +57,7 @@ if [[ "`df --output="avail" / | sed -n 2p`" -lt "10240" ]]; then
     echo ""
     df -h
     echo ""
-    echo "To recover from a full disk, follow the guide at https://cloudron.io/documentation/troubleshooting/#recovery-after-disk-full"
+    echo "To recover from a full disk, follow the guide at https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full"
     exit 1
 fi
 

scripts/cloudron-translation-update

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+set -eu -o pipefail
+
+# This script downloads new translation data from weblate at https://translate.cloudron.io
+
+OUT="/home/yellowtent/box/dashboard/dist/translation"
+
+# We require root
+if [[ ${EUID} -ne 0 ]]; then
+    echo "This script should be run as root. Run with sudo"
+    exit 1
+fi
+
+echo "=> Downloading new translation files..."
+curl https://translate.cloudron.io/download/cloudron/dashboard/?format=zip -o /tmp/lang.zip
+
+echo "=> Unpacking..."
+unzip -jo /tmp/lang.zip -d $OUT
+chown -R yellowtent:yellowtent $OUT
+# unzip put very restrictive permissions
+chmod ua+r $OUT/*
+
+echo "=> Cleanup..."
+rm /tmp/lang.zip
+
+echo "=> Done"
+
+echo ""
+echo "Reload the dashboard to see the new translations"
+echo ""

scripts/installer.sh

@@ -27,11 +27,11 @@ echo "==> installer: Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src
 
 echo "==> installer: updating docker"
 
-if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
+if [[ $(docker version --format {{.Client.Version}}) != "19.03.12" ]]; then
     # there are 3 packages for docker - containerd, CLI and the daemon
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.2-3_amd64.deb" -o /tmp/containerd.deb
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.13-2_amd64.deb" -o /tmp/containerd.deb
+    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
+    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
 
     echo "==> installer: Waiting for all dpkg tasks to finish..."
     while fuser /var/lib/dpkg/lock; do
@@ -60,12 +60,16 @@ fi
 readonly nginx_version=$(nginx -v 2>&1)
 if [[ "${nginx_version}" != *"1.18."* ]]; then
     echo "==> installer: installing nginx 1.18"
-    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
+    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-2~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
     # apt install with install deps (as opposed to dpkg -i)
     apt install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes /tmp/nginx.deb
     rm /tmp/nginx.deb
 fi
 
+# Cloudron 6 on ubuntu 20 installed recommended packages of collectd -> libinotify -> gnome->shell
+apt remove -y gnome-shell || true
+apt -y autoremove || true
+
 echo "==> installer: updating node"
 if [[ "$(node --version)" != "v10.18.1" ]]; then
     mkdir -p /usr/local/node-10.18.1

setup/start.sh

@@ -19,6 +19,7 @@ readonly json="$(realpath ${script_dir}/../node_modules/.bin/json)"
 readonly ubuntu_version=$(lsb_release -rs)
 
 cp -f "${script_dir}/../scripts/cloudron-support" /usr/bin/cloudron-support
+cp -f "${script_dir}/../scripts/cloudron-translation-update" /usr/bin/cloudron-translation-update
 
 # this needs to match the cloudron/base:2.0.0 gid
 if ! getent group media; then
@@ -31,7 +32,8 @@ systemctl enable apparmor
 systemctl restart apparmor
 
 usermod ${USER} -a -G docker
-docker network create --subnet=172.18.0.0/16 cloudron || true
+# unbound (which starts after box code) relies on this interface to exist. dockerproxy also relies on this.
+docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 cloudron || true
 
 mkdir -p "${BOX_DATA_DIR}"
 mkdir -p "${APPS_DATA_DIR}"
@@ -44,7 +46,7 @@ mkdir -p "${PLATFORM_DATA_DIR}/mysql"
 mkdir -p "${PLATFORM_DATA_DIR}/postgresql"
 mkdir -p "${PLATFORM_DATA_DIR}/mongodb"
 mkdir -p "${PLATFORM_DATA_DIR}/redis"
-mkdir -p "${PLATFORM_DATA_DIR}/addons/mail"
+mkdir -p "${PLATFORM_DATA_DIR}/addons/mail/banner"
 mkdir -p "${PLATFORM_DATA_DIR}/collectd/collectd.conf.d"
 mkdir -p "${PLATFORM_DATA_DIR}/logrotate.d"
 mkdir -p "${PLATFORM_DATA_DIR}/acme"
@@ -57,11 +59,13 @@ mkdir -p "${PLATFORM_DATA_DIR}/logs/backup" \
 mkdir -p "${PLATFORM_DATA_DIR}/update"
 
 mkdir -p "${BOX_DATA_DIR}/appicons"
+mkdir -p "${BOX_DATA_DIR}/firewall"
 mkdir -p "${BOX_DATA_DIR}/profileicons"
 mkdir -p "${BOX_DATA_DIR}/certs"
 mkdir -p "${BOX_DATA_DIR}/acme" # acme keys
 mkdir -p "${BOX_DATA_DIR}/mail/dkim"
 mkdir -p "${BOX_DATA_DIR}/well-known" # .well-known documents
+mkdir -p "${BOX_DATA_DIR}/sftp/ssh" # sftp keys
 
 # ensure backups folder exists and is writeable
 mkdir -p /var/backups
@@ -100,14 +104,13 @@ unbound-anchor -a /var/lib/unbound/root.key
 
 echo "==> Adding systemd services"
 cp -r "${script_dir}/start/systemd/." /etc/systemd/system/
-systemctl disable cloudron.target || true
-rm -f /etc/systemd/system/cloudron.target
 [[ "${ubuntu_version}" == "16.04" ]] && sed -e 's/MemoryMax/MemoryLimit/g' -i /etc/systemd/system/box.service
 systemctl daemon-reload
+systemctl enable --now cloudron-syslog
 systemctl enable unbound
-systemctl enable cloudron-syslog
 systemctl enable box
 systemctl enable cloudron-firewall
+systemctl enable --now cloudron-disable-thp
 
 # update firewall rules
 systemctl restart cloudron-firewall
@@ -129,6 +132,12 @@ echo "==> Configuring collectd"
 rm -rf /etc/collectd /var/log/collectd.log
 ln -sfF "${PLATFORM_DATA_DIR}/collectd" /etc/collectd
 cp "${script_dir}/start/collectd/collectd.conf" "${PLATFORM_DATA_DIR}/collectd/collectd.conf"
+if [[ "${ubuntu_version}" == "20.04" ]]; then
+    # https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1872281
+    if ! grep -q LD_PRELOAD /etc/default/collectd; then
+        echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
+    fi
+fi
 systemctl restart collectd
 
 echo "==> Configuring logrotate"
@@ -186,6 +195,10 @@ fi
 
 readonly mysql_root_password="password"
 mysqladmin -u root -ppassword password password # reset default root password
+if [[ "${ubuntu_version}" == "20.04" ]]; then
+    # mysql 8 added a new caching_sha2_password scheme which mysqljs does not support
+    mysql -u root -p${mysql_root_password} -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '${mysql_root_password}';"
+fi
 mysql -u root -p${mysql_root_password} -e 'CREATE DATABASE IF NOT EXISTS box'
 
 # set HOME explicity, because it's not set when the installer calls it. this is done because
@@ -207,12 +220,15 @@ else
     cp "${BOX_DATA_DIR}/dhparams.pem" "${PLATFORM_DATA_DIR}/addons/mail/dhparams.pem"
 fi
 
-# old installations used to create appdata/<app>/redis which is now part of old backups and prevents restore
-echo "==> Cleaning up stale redis directories"
-find "${APPS_DATA_DIR}" -maxdepth 2 -type d -name redis -exec rm -rf {} +
-
-echo "==> Cleaning up old logs"
-rm -f /home/yellowtent/platformdata/logs/*/*.log.* || true
+if [[ ! -f "${BOX_DATA_DIR}/sftp/ssh/ssh_host_rsa_key" ]]; then
+    # the key format in Ubuntu 20 changed, so we create keys in legacy format. for older ubuntu, just re-use the host keys
+    # see https://github.com/proftpd/proftpd/issues/793
+    if [[ "${ubuntu_version}" == "20.04" ]]; then
+        ssh-keygen -m PEM -t rsa -f "${BOX_DATA_DIR}/sftp/ssh/ssh_host_rsa_key" -q -N ""
+    else
+        cp /etc/ssh/ssh_host_rsa_key* ${BOX_DATA_DIR}/sftp/ssh
+    fi
+fi
 
 echo "==> Changing ownership"
 # be careful of what is chown'ed here. subdirs like mysql,redis etc are owned by the containers and will stop working if perms change

setup/start/cloudron-disable-thp.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+set -eu
+
+echo "==> Disabling THP"
+
+# https://docs.couchbase.com/server/current/install/thp-disable.html
+if [[ -d /sys/kernel/mm/transparent_hugepage ]]; then
+    echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
+    echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
+else
+    echo "==> kernel does not have THP"
+fi
+

setup/start/cloudron-firewall.sh

@@ -6,11 +6,29 @@ echo "==> Setting up firewall"
 iptables -t filter -N CLOUDRON || true
 iptables -t filter -F CLOUDRON # empty any existing rules
 
-# NOTE: keep these in sync with src/apps.js validatePortBindings
-# allow ssh, http, https, ping, dns
-iptables -t filter -I CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
-# ssh is allowed alternately on port 202
-iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443,587,993,4190 -j ACCEPT
+# first setup any user IP block lists
+ipset create cloudron_blocklist hash:net || true
+/home/yellowtent/box/src/scripts/setblocklist.sh
+
+iptables -t filter -A CLOUDRON -m set --match-set cloudron_blocklist src -j DROP
+# the DOCKER-USER chain is not cleared on docker restart
+if ! iptables -t filter -C DOCKER-USER -m set --match-set cloudron_blocklist src -j DROP; then
+    iptables -t filter -I DOCKER-USER 1 -m set --match-set cloudron_blocklist src -j DROP
+fi
+
+# allow related and establisted connections
+iptables -t filter -A CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
+iptables -t filter -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443 -j ACCEPT # 202 is the alternate ssh port
+
+# whitelist any user ports
+ports_json="/home/yellowtent/boxdata/firewall/ports.json"
+if allowed_tcp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_tcp_ports.join(','))" 2>/dev/null); then
+    [[ -n "${allowed_tcp_ports}" ]] && iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports "${allowed_tcp_ports}" -j ACCEPT
+fi
+
+if allowed_udp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_udp_ports.join(','))" 2>/dev/null); then
+    [[ -n "${allowed_tcp_ports}" ]] && iptables -A CLOUDRON -p udp -m udp -m multiport --dports "${allowed_tcp_ports}" -j ACCEPT
+fi
 
 # turn and stun service
 iptables -t filter -A CLOUDRON -p tcp -m multiport --dports 3478,5349 -j ACCEPT
@@ -52,8 +70,6 @@ for port in 22 202; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --dport ${port} -m state --state NEW -m recent --update --name "public-${port}" --seconds 10 --hitcount 5 -j CLOUDRON_RATELIMIT_LOG
 done
 
-# TODO: move docker platform rules to platform.js so it can be specialized to rate limit only when destination is the mail container
-
 # docker translates (dnat) 25, 587, 993, 4190 in the PREROUTING step
 for port in 2525 4190 9993; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --syn ! -s 172.18.0.0/16 -d 172.18.0.0/16 --dport ${port} -m connlimit --connlimit-above 50 -j CLOUDRON_RATELIMIT_LOG
@@ -69,12 +85,10 @@ for port in 3306 5432 6379 27017; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --syn -s 172.18.0.0/16 -d 172.18.0.0/16 --dport ${port} -m connlimit --connlimit-above 5000 -j CLOUDRON_RATELIMIT_LOG
 done
 
-# For ssh, http, https
 if ! iptables -t filter -C INPUT -j CLOUDRON_RATELIMIT 2>/dev/null; then
     iptables -t filter -I INPUT 1 -j CLOUDRON_RATELIMIT
 fi
 
-# For smtp, imap etc routed via docker/nat
-# Workaroud issue where Docker insists on adding itself first in FORWARD table
+# Workaround issue where Docker insists on adding itself first in FORWARD table
 iptables -D FORWARD -j CLOUDRON_RATELIMIT || true
 iptables -I FORWARD 1 -j CLOUDRON_RATELIMIT

setup/start/cloudron-motd

@@ -10,12 +10,19 @@ if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
     fi
     echo "${ip}" > /tmp/.cloudron-motd-cache
 
+    if [[ ! -f /etc/cloudron/SETUP_TOKEN ]]; then
+        url="https://${ip}"
+    else
+        setupToken="$(cat /etc/cloudron/SETUP_TOKEN)"
+        url="https://${ip}/?setupToken=${setupToken}"
+    fi
+
     printf "\t\t\tWELCOME TO CLOUDRON\n"
     printf "\t\t\t-------------------\n"
 
-    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://${ip} on your browser and accept the self-signed certificate to finish setup."
-    printf "Cloudron overview - https://cloudron.io/documentation/ \n"
-    printf "Cloudron setup - https://cloudron.io/documentation/installation/#setup \n"
+    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit ${url} on your browser and accept the self-signed certificate to finish setup."
+    printf "Cloudron overview - https://docs.cloudron.io/ \n"
+    printf "Cloudron setup - https://docs.cloudron.io/installation/#setup \n"
 else
     printf "\t\t\tNOTE TO CLOUDRON ADMINS\n"
     printf "\t\t\t-----------------------\n"
@@ -23,7 +30,7 @@ else
     printf "Cloudron relies on and may break your installation. Ubuntu security updates\n"
     printf "are automatically installed on this server every night.\n"
     printf "\n"
-    printf "Read more at https://cloudron.io/documentation/security/#os-updates\n"
+    printf "Read more at https://docs.cloudron.io/security/#os-updates\n"
 fi
 
 printf "\nFor help and more information, visit https://forum.cloudron.io\n\n"

setup/start/cloudron-resize-fs.sh

@@ -4,6 +4,11 @@ set -eu -o pipefail
 
 readonly APPS_SWAP_FILE="/apps.swap"
 
+if [[ -f "${APPS_SWAP_FILE}" ]]; then
+    echo "Swap file already exists at /apps.swap . Skipping"
+    exit
+fi
+
 # all sizes are in mb
 readonly physical_memory=$(LC_ALL=C free -m | awk '/Mem:/ { print $2 }')
 readonly swap_size=$((${physical_memory} > 4096 ? 4096 : ${physical_memory})) # min(RAM, 4GB) if you change this, fix enoughResourcesAvailable() in client.js

setup/start/collectd/collectd.conf

@@ -121,7 +121,7 @@ LoadPlugin memory
 #LoadPlugin netlink
 #LoadPlugin network
 #LoadPlugin nfs
-LoadPlugin nginx
+#LoadPlugin nginx
 #LoadPlugin notify_desktop
 #LoadPlugin notify_email
 #LoadPlugin ntpd
@@ -149,7 +149,7 @@ LoadPlugin nginx
 #LoadPlugin statsd
 LoadPlugin swap
 #LoadPlugin table
-LoadPlugin tail
+#LoadPlugin tail
 #LoadPlugin tail_csv
 #LoadPlugin tcpconns
 #LoadPlugin teamspeak2
@@ -197,42 +197,11 @@ LoadPlugin write_graphite
    IgnoreSelected false
 </Plugin>
 
-<Plugin nginx>
-   URL "http://127.0.0.1/nginx_status"
-</Plugin>
-
 <Plugin swap>
    ReportByDevice false
    ReportBytes true
 </Plugin>
 
-<Plugin "tail">
-   <File "/var/log/nginx/error.log">
-     Instance "nginx"
-     <Match>
-       Regex ".*"
-       DSType "CounterInc"
-       Type counter
-       Instance "errors"
-     </Match>
-   </File>
-   <File "/var/log/nginx/access.log">
-     Instance "nginx"
-     <Match>
-       Regex ".*"
-       DSType "CounterInc"
-       Type counter
-       Instance "requests"
-     </Match>
-    <Match>
-      Regex " \".*\" [0-9]+ [0-9]+ ([0-9]+)"
-      DSType GaugeAverage
-      Type delay
-      Instance "response"
-    </Match>
-   </File>
-</Plugin>
-
 <Plugin python>
     # https://blog.dbrgn.ch/2017/3/10/write-a-collectd-python-plugin/
     ModulePath "/home/yellowtent/box/setup/start/collectd/"

setup/start/collectd/df.py

@@ -6,7 +6,7 @@ disks = []
 
 def init():
     global disks
-    lines = [s.split() for s in subprocess.check_output(["df", "--type=ext4", "--output=source,target,size,used,avail"]).splitlines()]
+    lines = [s.split() for s in subprocess.check_output(["df", "--type=ext4", "--output=source,target,size,used,avail"]).decode('utf-8').splitlines()]
     disks = lines[1:] # strip header
     collectd.info('custom df plugin initialized with %s' % disks)
 

setup/start/mysql.cnf

@@ -6,7 +6,7 @@
 performance_schema=OFF
 max_connections=50
 # on ec2, without this we get a sporadic connection drop when doing the initial migration
-max_allowed_packet=32M
+max_allowed_packet=64M
 
 # https://mathiasbynens.be/notes/mysql-utf8mb4
 character-set-server = utf8mb4

setup/start/nginx/nginx.conf

@@ -6,7 +6,7 @@ worker_processes  auto;
 # this is 4096 by default. See /proc/<PID>/limits and /etc/security/limits.conf
 # usually twice the worker_connections (one for uptsream, one for downstream)
 # see also LimitNOFILE=16384 in systemd drop-in
-worker_rlimit_nofile 8192; 
+worker_rlimit_nofile 8192;
 
 pid /run/nginx.pid;
 
@@ -43,23 +43,5 @@ http {
     # zones for rate limiting
     limit_req_zone $binary_remote_addr zone=admin_login:10m rate=10r/s; # 10 request a second
 
-
-    # default http server that returns 404 for any domain we are not listening on
-    server {
-        listen      80 default_server;
-        listen      [::]:80 default_server;
-        server_name does_not_match_anything;
-
-        # acme challenges (for app installation and re-configure when the vhost config does not exist)
-        location /.well-known/acme-challenge/ {
-            default_type text/plain;
-            alias /home/yellowtent/platformdata/acme/;
-        }
-
-        location / {
-            return 404;
-        }
-    }
-
     include applications/*.conf;
 }

setup/start/sudoers

@@ -59,3 +59,6 @@ yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/scripts/starttas
 Defaults!/home/yellowtent/box/src/scripts/stoptask.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/stoptask.sh
 
+Defaults!/home/yellowtent/box/src/scripts/setblocklist.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/setblocklist.sh
+

setup/start/systemd/cloudron-disable-thp.service

@@ -0,0 +1,15 @@
+# https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
+[Unit]
+Description=Disable Transparent Huge Pages (THP)
+DefaultDependencies=no
+After=sysinit.target local-fs.target
+Before=docker.service
+
+[Service]
+Type=oneshot
+ExecStart="/home/yellowtent/box/setup/start/cloudron-disable-thp.sh"
+RemainAfterExit=yes
+
+[Install]
+WantedBy=basic.target
+

setup/start/systemd/unbound.service

@@ -2,7 +2,7 @@
 
 [Unit]
 Description=Unbound DNS Resolver
-After=network.target
+After=network.target docker.service
 
 [Service]
 PIDFile=/run/unbound.pid

setup/start/unbound.conf

@@ -1,5 +1,7 @@
 server:
-        interface: 0.0.0.0
+        port: 53
+        interface: 127.0.0.1
+        interface: 172.18.0.1
         do-ip6: no
         access-control: 127.0.0.1 allow
         access-control: 172.18.0.1/16 allow

src/appdb.js

@@ -1,32 +1,32 @@
 'use strict';
 
 exports = module.exports = {
-    get: get,
-    getByHttpPort: getByHttpPort,
-    getByContainerId: getByContainerId,
-    add: add,
-    exists: exists,
-    del: del,
-    update: update,
-    getAll: getAll,
-    getPortBindings: getPortBindings,
-    delPortBinding: delPortBinding,
+    get,
+    add,
+    exists,
+    del,
+    update,
+    getAll,
+    getPortBindings,
+    delPortBinding,
 
-    setAddonConfig: setAddonConfig,
-    getAddonConfig: getAddonConfig,
-    getAddonConfigByAppId: getAddonConfigByAppId,
-    getAddonConfigByName: getAddonConfigByName,
-    unsetAddonConfig: unsetAddonConfig,
-    unsetAddonConfigByAppId: unsetAddonConfigByAppId,
-    getAppIdByAddonConfigValue: getAppIdByAddonConfigValue,
+    setAddonConfig,
+    getAddonConfig,
+    getAddonConfigByAppId,
+    getAddonConfigByName,
+    unsetAddonConfig,
+    unsetAddonConfigByAppId,
+    getAppIdByAddonConfigValue,
+    getByIpAddress,
 
-    setHealth: setHealth,
-    setTask: setTask,
-    getAppStoreIds: getAppStoreIds,
+    setHealth,
+    setTask,
+    getAppStoreIds,
 
     // subdomain table types
     SUBDOMAIN_TYPE_PRIMARY: 'primary',
     SUBDOMAIN_TYPE_REDIRECT: 'redirect',
+    SUBDOMAIN_TYPE_ALIAS: 'alias',
 
     _clear: clear
 };
@@ -38,17 +38,14 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
-    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'subdomains.subdomain AS location', 'subdomains.domain',
-    'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
-    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson', 'apps.bindsJson',
-    'apps.sso', 'apps.debugModeJson', 'apps.enableBackup',
+const APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
+    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
+    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson',
+    'apps.sso', 'apps.debugModeJson', 'apps.enableBackup', 'apps.proxyAuth', 'apps.containerIp',
     'apps.creationTime', 'apps.updateTime', 'apps.mailboxName', 'apps.mailboxDomain', 'apps.enableAutomaticUpdate',
     'apps.dataDir', 'apps.ts', 'apps.healthTime' ].join(',');
 
-var PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');
-
-const SUBDOMAIN_FIELDS = [ 'appId', 'domain', 'subdomain', 'type' ].join(',');
+const PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');
 
 function postProcess(result) {
     assert.strictEqual(typeof result, 'object');
@@ -89,6 +86,7 @@ function postProcess(result) {
     result.sso = !!result.sso; // make it bool
     result.enableBackup = !!result.enableBackup; // make it bool
     result.enableAutomaticUpdate = !!result.enableAutomaticUpdate; // make it bool
+    result.proxyAuth = !!result.proxyAuth;
 
     assert(result.debugModeJson === null || typeof result.debugModeJson === 'string');
     result.debugMode = safe.JSON.parse(result.debugModeJson);
@@ -98,15 +96,23 @@ function postProcess(result) {
     result.servicesConfig = safe.JSON.parse(result.servicesConfigJson) || {};
     delete result.servicesConfigJson;
 
-    assert(result.bindsJson === null || typeof result.bindsJson === 'string');
-    result.binds = safe.JSON.parse(result.bindsJson) || {};
-    delete result.bindsJson;
+    let subdomains = JSON.parse(result.subdomains), domains = JSON.parse(result.domains), subdomainTypes = JSON.parse(result.subdomainTypes);
+    delete result.subdomains;
+    delete result.domains;
+    delete result.subdomainTypes;
 
-    result.alternateDomains = result.alternateDomains || [];
-    result.alternateDomains.forEach(function (d) {
-        delete d.appId;
-        delete d.type;
-    });
+    result.alternateDomains = [];
+    result.aliasDomains = [];
+    for (let i = 0; i < subdomainTypes.length; i++) {
+        if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_PRIMARY) {
+            result.location = subdomains[i];
+            result.domain = domains[i];
+        } else if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_REDIRECT) {
+            result.alternateDomains.push({ domain: domains[i], subdomain: subdomains[i] });
+        } else if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_ALIAS) {
+            result.aliasDomains.push({ domain: domains[i], subdomain: subdomains[i] });
+        }
+    }
 
     let envNames = JSON.parse(result.envNames), envValues = JSON.parse(result.envValues);
     delete result.envNames;
@@ -116,119 +122,67 @@ function postProcess(result) {
         if (envNames[i]) result.env[envNames[i]] = envValues[i];
     }
 
+    let volumeIds = JSON.parse(result.volumeIds);
+    delete result.volumeIds;
+    let volumeReadOnlys = JSON.parse(result.volumeReadOnlys);
+    delete result.volumeReadOnlys;
+
+    result.mounts = volumeIds[0] === null ? [] : volumeIds.map((v, idx) => { return { volumeId: v, readOnly: !!volumeReadOnlys[idx] }; }); // NOTE: volumeIds is [null] when volumes of an app is empty
+
     result.error = safe.JSON.parse(result.errorJson);
     delete result.errorJson;
 
     result.taskId = result.taskId ? String(result.taskId) : null;
 }
 
+// each query simply join apps table with another table by id. we then join the full result together
+const PB_QUERY = 'SELECT id, GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes FROM apps LEFT JOIN appPortBindings ON apps.id = appPortBindings.appId GROUP BY apps.id';
+const ENV_QUERY = 'SELECT id, JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues FROM apps LEFT JOIN appEnvVars ON apps.id = appEnvVars.appId GROUP BY apps.id';
+const SUBDOMAIN_QUERY = 'SELECT id, JSON_ARRAYAGG(subdomains.subdomain) AS subdomains, JSON_ARRAYAGG(subdomains.domain) AS domains, JSON_ARRAYAGG(subdomains.type) AS subdomainTypes FROM apps LEFT JOIN subdomains ON apps.id = subdomains.appId GROUP BY apps.id';
+const MOUNTS_QUERY = 'SELECT id, JSON_ARRAYAGG(appMounts.volumeId) AS volumeIds, JSON_ARRAYAGG(appMounts.readOnly) AS volumeReadOnlys FROM apps LEFT JOIN appMounts ON apps.id = appMounts.appId GROUP BY apps.id';
+const APPS_QUERY = `SELECT ${APPS_FIELDS_PREFIXED}, hostPorts, environmentVariables, portTypes, envNames, envValues, subdomains, domains, subdomainTypes, volumeIds, volumeReadOnlys FROM apps`
+    + ` LEFT JOIN (${PB_QUERY}) AS q1 on q1.id = apps.id`
+    + ` LEFT JOIN (${ENV_QUERY}) AS q2 on q2.id = apps.id`
+    + ` LEFT JOIN (${SUBDOMAIN_QUERY}) AS q3 on q3.id = apps.id`
+    + ` LEFT JOIN (${MOUNTS_QUERY}) AS q4 on q4.id = apps.id`;
+
 function get(id, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes, '
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE apps.id = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, id ], function (error, result) {
+    database.query(`${APPS_QUERY} WHERE apps.id = ?`, [ id ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        postProcess(result[0]);
 
-            result[0].alternateDomains = alternateDomains;
-
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
+        callback(null, result[0]);
     });
 }
 
-function getByHttpPort(httpPort, callback) {
-    assert.strictEqual(typeof httpPort, 'number');
+function getByIpAddress(ip, callback) {
+    assert.strictEqual(typeof ip, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-    + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-    + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE httpPort = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, httpPort ], function (error, result) {
+    database.query(`${APPS_QUERY} WHERE apps.containerIp = ?`, [ ip ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        postProcess(result[0]);
 
-            result[0].alternateDomains = alternateDomains;
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
-    });
-}
-
-function getByContainerId(containerId, callback) {
-    assert.strictEqual(typeof containerId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE containerId = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, containerId ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
-
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-            result[0].alternateDomains = alternateDomains;
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
+        callback(null, result[0]);
     });
 }
 
 function getAll(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' GROUP BY apps.id ORDER BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY ], function (error, results) {
+    database.query(`${APPS_QUERY} ORDER BY apps.id`, [ ], function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE type = ?', [ exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        results.forEach(postProcess);
 
-            alternateDomains.forEach(function (d) {
-                var domain = results.find(function (a) { return d.appId === a.id; });
-                if (!domain) return;
-
-                domain.alternateDomains = domain.alternateDomains || [];
-                domain.alternateDomains.push(d);
-            });
-
-            results.forEach(postProcess);
-
-            callback(null, results);
-        });
+        callback(null, results);
     });
 }
 
@@ -300,6 +254,15 @@ function add(id, appStoreId, manifest, location, domain, portBindings, data, cal
         });
     }
 
+    if (data.aliasDomains) {
+        data.aliasDomains.forEach(function (d) {
+            queries.push({
+                query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)',
+                args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_ALIAS ]
+            });
+        });
+    }
+
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, 'no such domain'));
@@ -358,12 +321,13 @@ function del(id, callback) {
         { query: 'DELETE FROM appPortBindings WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appEnvVars WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appPasswords WHERE identifier = ?', args: [ id ] },
+        { query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM apps WHERE id = ?', args: [ id ] }
     ];
 
     database.transaction(queries, function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results[4].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (results[5].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
         callback(null);
     });
@@ -396,6 +360,7 @@ function updateWithConstraints(id, app, constraints, callback) {
     assert(!('portBindings' in app) || typeof app.portBindings === 'object');
     assert(!('accessRestriction' in app) || typeof app.accessRestriction === 'object' || app.accessRestriction === '');
     assert(!('alternateDomains' in app) || Array.isArray(app.alternateDomains));
+    assert(!('aliasDomains' in app) || Array.isArray(app.aliasDomains));
     assert(!('tags' in app) || Array.isArray(app.tags));
     assert(!('env' in app) || typeof app.env === 'object');
 
@@ -423,22 +388,35 @@ function updateWithConstraints(id, app, constraints, callback) {
     }
 
     if ('location' in app && 'domain' in app) { // must be updated together as they are unique together
-        queries.push({ query: 'UPDATE subdomains SET subdomain = ?, domain = ? WHERE appId = ? AND type = ?', args: [ app.location, app.domain, id, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ?', args: [ id ]}); // all locations of an app must be updated together
+        queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, app.domain, app.location, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+
+        if ('alternateDomains' in app) {
+            app.alternateDomains.forEach(function (d) {
+                queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+            });
+        }
+
+        if ('aliasDomains' in app) {
+            app.aliasDomains.forEach(function (d) {
+                queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_ALIAS ]});
+            });
+        }
     }
 
-    if ('alternateDomains' in app) {
-        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ? AND type = ?', args: [ id, exports.SUBDOMAIN_TYPE_REDIRECT ]});
-        app.alternateDomains.forEach(function (d) {
-            queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+    if ('mounts' in app) {
+        queries.push({ query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ]});
+        app.mounts.forEach(function (m) {
+            queries.push({ query: 'INSERT INTO appMounts (appId, volumeId, readOnly) VALUES (?, ?, ?)', args: [ id, m.volumeId, m.readOnly ]});
         });
     }
 
     var fields = [ ], values = [ ];
     for (var p in app) {
-        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig' || p === 'binds') {
+        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig') {
             fields.push(`${p}Json = ?`);
             values.push(JSON.stringify(app[p]));
-        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env') {
+        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'aliasDomains' && p !== 'env' && p !== 'mounts') {
             fields.push(p + ' = ?');
             values.push(app[p]);
         }

src/apphealthmonitor.js

@@ -14,13 +14,14 @@ var appdb = require('./appdb.js'),
     util = require('util');
 
 exports = module.exports = {
-    run: run
+    run
 };
 
 const HEALTHCHECK_INTERVAL = 10 * 1000; // every 10 seconds. this needs to be small since the UI makes only healthy apps clickable
 const UNHEALTHY_THRESHOLD = 10 * 60 * 1000; // 10 minutes
 
 const OOM_EVENT_LIMIT = 60 * 60 * 1000; // 60 minutes
+const gStartTime = new Date(); // time when apphealthmonitor was started
 let gLastOomMailTime = Date.now() - (5 * 60 * 1000); // pretend we sent email 5 minutes ago
 
 function debugApp(app) {
@@ -34,7 +35,8 @@ function setHealth(app, health, callback) {
     assert.strictEqual(typeof health, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    let now = new Date(), healthTime = app.healthTime, curHealth = app.health;
+    let now = new Date(), curHealth = app.health;
+    let healthTime = gStartTime > app.healthTime ? gStartTime : app.healthTime; // on box restart, clamp value to start time
 
     if (health === apps.HEALTH_HEALTHY) {
         healthTime = now;
@@ -79,21 +81,13 @@ function checkAppHealth(app, callback) {
     const manifest = app.manifest;
 
     docker.inspect(app.containerId, function (error, data) {
-        if (error || !data || !data.State) {
-            debugApp(app, 'Error inspecting container');
-            return setHealth(app, apps.HEALTH_ERROR, callback);
-        }
-
-        if (data.State.Running !== true) {
-            debugApp(app, 'exited');
-            return setHealth(app, apps.HEALTH_DEAD, callback);
-        }
+        if (error || !data || !data.State) return setHealth(app, apps.HEALTH_ERROR, callback);
+        if (data.State.Running !== true) return setHealth(app, apps.HEALTH_DEAD, callback);
 
         // non-appstore apps may not have healthCheckPath
         if (!manifest.healthCheckPath) return setHealth(app, apps.HEALTH_HEALTHY, callback);
 
-        // poll through docker network instead of nginx to bypass any potential oauth proxy
-        var healthCheckUrl = 'http://127.0.0.1:' + app.httpPort + manifest.healthCheckPath;
+        const healthCheckUrl = `http://${app.containerIp}:${manifest.httpPort}${manifest.healthCheckPath}`;
         superagent
             .get(healthCheckUrl)
             .set('Host', app.fqdn) // required for some apache configs with rewrite rules
@@ -103,7 +97,7 @@ function checkAppHealth(app, callback) {
             .end(function (error, res) {
                 if (error && !error.response) {
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
-                } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
+                } else if (res.statusCode >= 403) { // 2xx and 3xx are ok. even 401 and 403 are ok for now (for WP sites)
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
                 } else {
                     setHealth(app, apps.HEALTH_HEALTHY, callback);
@@ -126,7 +120,7 @@ function getContainerInfo(containerId, callback) {
 
 /*
     OOM can be tested using stress tool like so:
-        docker run -ti -m 100M cloudron/base:0.10.0 /bin/bash
+        docker run -ti -m 100M cloudron/base:2.0.0 /bin/bash
         apt-get update && apt-get install stress
         stress --vm 1 --vm-bytes 200M --vm-hang 0
 */

src/apps.js

@@ -1,70 +1,72 @@
 'use strict';
 
 exports = module.exports = {
-    hasAccessTo: hasAccessTo,
-    removeInternalFields: removeInternalFields,
-    removeRestrictedFields: removeRestrictedFields,
+    hasAccessTo,
+    removeInternalFields,
+    removeRestrictedFields,
 
-    get: get,
-    getByContainerId: getByContainerId,
-    getByIpAddress: getByIpAddress,
-    getByFqdn: getByFqdn,
-    getAll: getAll,
-    getAllByUser: getAllByUser,
-    install: install,
-    uninstall: uninstall,
+    get,
+    getByIpAddress,
+    getByFqdn,
+    getAll,
+    getAllByUser,
+    install,
+    uninstall,
 
-    setAccessRestriction: setAccessRestriction,
-    setLabel: setLabel,
-    setIcon: setIcon,
-    setTags: setTags,
-    setMemoryLimit: setMemoryLimit,
-    setCpuShares: setCpuShares,
-    setBinds: setBinds,
-    setAutomaticBackup: setAutomaticBackup,
-    setAutomaticUpdate: setAutomaticUpdate,
-    setReverseProxyConfig: setReverseProxyConfig,
-    setCertificate: setCertificate,
-    setDebugMode: setDebugMode,
-    setEnvironment: setEnvironment,
-    setMailbox: setMailbox,
-    setLocation: setLocation,
-    setDataDir: setDataDir,
-    repair: repair,
+    setAccessRestriction,
+    setLabel,
+    setIcon,
+    setTags,
+    setMemoryLimit,
+    setCpuShares,
+    setMounts,
+    setAutomaticBackup,
+    setAutomaticUpdate,
+    setReverseProxyConfig,
+    setCertificate,
+    setDebugMode,
+    setEnvironment,
+    setMailbox,
+    setLocation,
+    setDataDir,
+    repair,
 
-    restore: restore,
-    importApp: importApp,
-    clone: clone,
+    restore,
+    importApp,
+    exportApp,
+    clone,
 
-    update: update,
+    update,
 
-    backup: backup,
-    listBackups: listBackups,
+    backup,
+    listBackups,
 
-    getLogs: getLogs,
+    getLocalLogfilePaths,
+    getLogs,
 
-    start: start,
-    stop: stop,
-    restart: restart,
+    start,
+    stop,
+    restart,
 
-    exec: exec,
+    exec,
 
-    checkManifestConstraints: checkManifestConstraints,
-    downloadManifest: downloadManifest,
+    checkManifestConstraints,
+    downloadManifest,
 
-    canAutoupdateApp: canAutoupdateApp,
-    autoupdateApps: autoupdateApps,
+    canAutoupdateApp,
+    autoupdateApps,
 
-    restoreInstalledApps: restoreInstalledApps,
-    configureInstalledApps: configureInstalledApps,
-    schedulePendingTasks: schedulePendingTasks,
-    restartAppsUsingAddons: restartAppsUsingAddons,
+    restoreInstalledApps,
+    configureInstalledApps,
+    schedulePendingTasks,
+    restartAppsUsingAddons,
 
-    getDataDir: getDataDir,
-    getIconPath: getIconPath,
+    getDataDir,
+    getIconPath,
+    getMemoryLimit,
 
-    downloadFile: downloadFile,
-    uploadFile: uploadFile,
+    downloadFile,
+    uploadFile,
 
     PORT_TYPE_TCP: 'tcp',
     PORT_TYPE_UDP: 'udp',
@@ -134,7 +136,6 @@ var appdb = require('./appdb.js'),
     superagent = require('superagent'),
     tasks = require('./tasks.js'),
     TransformStream = require('stream').Transform,
-    updateChecker = require('./updatechecker.js'),
     users = require('./users.js'),
     util = require('util'),
     uuid = require('uuid'),
@@ -154,7 +155,6 @@ function validatePortBindings(portBindings, manifest) {
     const RESERVED_PORTS = [
         22, /* ssh */
         25, /* smtp */
-        53, /* dns */
         80, /* http */
         143, /* imap */
         202, /* alternate ssh */
@@ -167,7 +167,7 @@ function validatePortBindings(portBindings, manifest) {
         2004, /* graphite (lo) */
         2514, /* cloudron-syslog (lo) */
         constants.PORT, /* app server (lo) */
-        constants.SYSADMIN_PORT, /* sysadmin app server (lo) */
+        constants.AUTHWALL_PORT, /* protected sites */
         constants.INTERNAL_SMTP_PORT, /* internal smtp port (lo) */
         constants.LDAP_PORT,
         3306, /* mysql (lo) */
@@ -191,7 +191,7 @@ function validatePortBindings(portBindings, manifest) {
         if (!Number.isInteger(hostPort)) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not an integer`, { field: 'portBindings', portName: portName });
         if (RESERVED_PORTS.indexOf(hostPort) !== -1) return new BoxError(BoxError.BAD_FIELD, `Port ${hostPort} is reserved.`, { field: 'portBindings', portName: portName });
         if (RESERVED_PORT_RANGES.find(range => (hostPort >= range[0] && hostPort <= range[1]))) return new BoxError(BoxError.BAD_FIELD, `Port ${hostPort} is reserved.`, { field: 'portBindings', portName: portName });
-        if (hostPort <= 1023 || hostPort > 65535) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not in permitted range`, { field: 'portBindings', portName: portName });
+        if (hostPort !== 53 && (hostPort <= 1023 || hostPort > 65535)) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not in permitted range`, { field: 'portBindings', portName: portName }); // dns 53 is special and adblocker apps can use them
     }
 
     // it is OK if there is no 1-1 mapping between values in manifest.tcpPorts and portBindings. missing values implies
@@ -334,20 +334,6 @@ function validateEnv(env) {
     return null;
 }
 
-function validateBinds(binds) {
-    for (let name of Object.keys(binds)) {
-        // just have friendly characters under /media
-        if (!/^[-0-9a-zA-Z_@$=#.%+]+$/.test(name)) return new BoxError(BoxError.BAD_FIELD, `Invalid bind name: ${name}`);
-
-        const bind = binds[name];
-
-        if (!bind.hostPath.startsWith('/mnt') && !bind.hostPath.startsWith('/media')) return new BoxError(BoxError.BAD_FIELD, 'hostPath must be in /mnt or /media');
-        if (path.normalize(bind.hostPath) !== bind.hostPath) return new BoxError(BoxError.BAD_FIELD, 'hostPath is not normalized');
-    }
-
-    return null;
-}
-
 function validateDataDir(dataDir) {
     if (dataDir === null) return null;
 
@@ -398,7 +384,7 @@ function getDuplicateErrorDetails(errorMessage, locations, domainObjectMap, port
 
     // check if any of the port bindings conflict
     for (let portName in portBindings) {
-        if (portBindings[portName] === parseInt(match[1])) return new BoxError(BoxError.ALREADY_EXISTS, `Port ${match[1]} is reserved`, { portName });
+        if (portBindings[portName] === parseInt(match[1])) return new BoxError(BoxError.ALREADY_EXISTS, `Port ${match[1]} is in use`, { portName });
     }
 
     if (match[2] === 'dataDir') {
@@ -420,13 +406,13 @@ function removeInternalFields(app) {
         'location', 'domain', 'fqdn', 'mailboxName', 'mailboxDomain',
         'accessRestriction', 'manifest', 'portBindings', 'iconUrl', 'memoryLimit', 'cpuShares',
         'sso', 'debugMode', 'reverseProxyConfig', 'enableBackup', 'creationTime', 'updateTime', 'ts', 'tags',
-        'label', 'alternateDomains', 'env', 'enableAutomaticUpdate', 'dataDir', 'binds');
+        'label', 'alternateDomains', 'aliasDomains', 'env', 'enableAutomaticUpdate', 'dataDir', 'mounts');
 }
 
 // non-admins can only see these
 function removeRestrictedFields(app) {
     return _.pick(app,
-        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId', 'alternateDomains', 'sso',
+        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId', 'alternateDomains', 'aliasDomains', 'sso',
         'location', 'domain', 'fqdn', 'manifest', 'portBindings', 'iconUrl', 'creationTime', 'ts', 'tags', 'label', 'enableBackup');
 }
 
@@ -460,6 +446,20 @@ function getIconPath(app, options, callback) {
     callback(new BoxError(BoxError.NOT_FOUND, 'No icon'));
 }
 
+function getMemoryLimit(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    let memoryLimit = app.memoryLimit || app.manifest.memoryLimit || 0;
+
+    if (memoryLimit === -1) { // unrestricted
+        memoryLimit = 0;
+    } else if (memoryLimit === 0 || memoryLimit < constants.DEFAULT_MEMORY_LIMIT) { // ensure we never go below minimum (in case we change the default)
+        memoryLimit = constants.DEFAULT_MEMORY_LIMIT;
+    }
+
+    return memoryLimit;
+}
+
 function postProcess(app, domainObjectMap) {
     let result = {};
     for (let portName in app.portBindings) {
@@ -470,6 +470,7 @@ function postProcess(app, domainObjectMap) {
     app.iconUrl = getIconUrlSync(app);
     app.fqdn = domains.fqdn(app.location, domainObjectMap[app.domain]);
     app.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+    app.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 }
 
 function hasAccessTo(app, user, callback) {
@@ -512,25 +513,6 @@ function get(appId, callback) {
         if (error) return callback(error);
 
         appdb.get(appId, function (error, app) {
-            if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
-            if (error) return callback(error);
-
-            postProcess(app, domainObjectMap);
-
-            callback(null, app);
-        });
-    });
-}
-
-function getByContainerId(containerId, callback) {
-    assert.strictEqual(typeof containerId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    getDomainObjectMap(function (error, domainObjectMap) {
-        if (error) return callback(error);
-
-        appdb.getByContainerId(containerId, function (error, app) {
-            if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
             if (error) return callback(error);
 
             postProcess(app, domainObjectMap);
@@ -548,16 +530,15 @@ function getByIpAddress(ip, callback) {
     // this is only used by the ldap test. the apps tests still uses proper docker
     if (constants.TEST && exports._MOCK_GET_BY_IP_APP_ID) return get(exports._MOCK_GET_BY_IP_APP_ID, callback);
 
-    docker.getContainerIdByIp(ip, function (error, containerId) {
+    appdb.getByIpAddress(ip, function (error, app) {
         if (error) return callback(error);
 
-        docker.inspect(containerId, function (error, result) {
+        getDomainObjectMap(function (error, domainObjectMap) {
             if (error) return callback(error);
 
-            const appId = safe.query(result, 'Config.Labels.appId', null);
-            if (!appId) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
+            postProcess(app, domainObjectMap);
 
-            get(appId, callback);
+            callback(null, app);
         });
     });
 }
@@ -639,20 +620,31 @@ function scheduleTask(appId, installationState, taskId, callback) {
     assert.strictEqual(typeof taskId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    appTaskManager.scheduleTask(appId, taskId, function (error) {
-        debug(`scheduleTask: task ${taskId} of ${appId} completed`);
-        if (error && (error.code === tasks.ECRASHED || error.code === tasks.ESTOPPED)) { // if task crashed, update the error
-            debug(`Apptask crashed/stopped: ${error.message}`);
-            let boxError = new BoxError(BoxError.TASK_ERROR, error.message);
-            boxError.details.crashed = error.code === tasks.ECRASHED;
-            boxError.details.stopped = error.code === tasks.ESTOPPED;
-            // see also apptask makeTaskError
-            boxError.details.taskId = taskId;
-            boxError.details.installationState = installationState;
-            appdb.update(appId, { installationState: exports.ISTATE_ERROR, error: boxError.toPlainObject(), taskId: null }, callback);
-        } else if (!(installationState === exports.ISTATE_PENDING_UNINSTALL && !error)) { // clear out taskId except for successful uninstall
-            appdb.update(appId, { taskId: null }, callback);
+    settings.getBackupConfig(function (error, backupConfig) {
+        if (error) return callback(error);
+
+        let memoryLimit = 400;
+        if (installationState === exports.ISTATE_PENDING_BACKUP || installationState === exports.ISTATE_PENDING_CLONE || installationState === exports.ISTATE_PENDING_RESTORE) {
+            memoryLimit = 'memoryLimit' in backupConfig ? Math.max(backupConfig.memoryLimit/1024/1024, 400) : 400;
         }
+
+        const options = { timeout: 20 * 60 * 60 * 1000 /* 20 hours */, nice: 15, memoryLimit };
+
+        appTaskManager.scheduleTask(appId, taskId, options, function (error) {
+            debug(`scheduleTask: task ${taskId} of ${appId} completed`);
+            if (error && (error.code === tasks.ECRASHED || error.code === tasks.ESTOPPED)) { // if task crashed, update the error
+                debug(`Apptask crashed/stopped: ${error.message}`);
+                let boxError = new BoxError(BoxError.TASK_ERROR, error.message);
+                boxError.details.crashed = error.code === tasks.ECRASHED;
+                boxError.details.stopped = error.code === tasks.ESTOPPED;
+                // see also apptask makeTaskError
+                boxError.details.taskId = taskId;
+                boxError.details.installationState = installationState;
+                appdb.update(appId, { installationState: exports.ISTATE_ERROR, error: boxError.toPlainObject(), taskId: null }, callback);
+            } else if (!(installationState === exports.ISTATE_PENDING_UNINSTALL && !error)) { // clear out taskId except for successful uninstall
+                appdb.update(appId, { taskId: null }, callback);
+            }
+        });
     });
 }
 
@@ -714,7 +706,13 @@ function validateLocations(locations, callback) {
         for (let location of locations) {
             if (!(location.domain in domainObjectMap)) return callback(new BoxError(BoxError.BAD_FIELD, 'No such domain', { field: 'location', domain: location.domain, subdomain: location.subdomain }));
 
-            error = domains.validateHostname(location.subdomain, domainObjectMap[location.domain]);
+            let subdomain = location.subdomain;
+            if (location.type === 'alias' && subdomain.startsWith('*')) {
+                if (subdomain === '*') continue;
+                subdomain = subdomain.replace(/^\*\./, ''); // remove *.
+            }
+
+            error = domains.validateHostname(subdomain, domainObjectMap[location.domain]);
             if (error) return callback(new BoxError(BoxError.BAD_FIELD, 'Bad location: ' + error.message, { field: 'location', domain: location.domain, subdomain: location.subdomain }));
         }
 
@@ -746,6 +744,7 @@ function install(data, auditSource, callback) {
         enableBackup = 'enableBackup' in data ? data.enableBackup : true,
         enableAutomaticUpdate = 'enableAutomaticUpdate' in data ? data.enableAutomaticUpdate : true,
         alternateDomains = data.alternateDomains || [],
+        aliasDomains = data.aliasDomains || [],
         env = data.env || {},
         label = data.label || null,
         tags = data.tags || [],
@@ -779,7 +778,7 @@ function install(data, auditSource, callback) {
 
     if ('sso' in data && !('optionalSso' in manifest)) return callback(new BoxError(BoxError.BAD_FIELD, 'sso can only be specified for apps with optionalSso'));
     // if sso was unspecified, enable it by default if possible
-    if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['oauth'];
+    if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['proxyAuth'];
 
     error = validateEnv(env);
     if (error) return callback(error);
@@ -798,7 +797,10 @@ function install(data, auditSource, callback) {
         }
     }
 
-    const locations = [{subdomain: location, domain}].concat(alternateDomains);
+    const locations = [{ subdomain: location, domain, type: 'primary' }]
+        .concat(alternateDomains.map(ad => _.extend(ad, { type: 'redirect' })))
+        .concat(aliasDomains.map(ad => _.extend(ad, { type: 'alias' })));
+
     validateLocations(locations, function (error, domainObjectMap) {
         if (error) return callback(error);
 
@@ -810,18 +812,19 @@ function install(data, auditSource, callback) {
         debug('Will install app with id : ' + appId);
 
         var data = {
-            accessRestriction: accessRestriction,
-            memoryLimit: memoryLimit,
-            sso: sso,
-            debugMode: debugMode,
-            mailboxName: mailboxName,
-            mailboxDomain: mailboxDomain,
-            enableBackup: enableBackup,
-            enableAutomaticUpdate: enableAutomaticUpdate,
-            alternateDomains: alternateDomains,
-            env: env,
-            label: label,
-            tags: tags,
+            accessRestriction,
+            memoryLimit,
+            sso,
+            debugMode,
+            mailboxName,
+            mailboxDomain,
+            enableBackup,
+            enableAutomaticUpdate,
+            alternateDomains,
+            aliasDomains,
+            env,
+            label,
+            tags,
             runState: exports.RSTATE_RUNNING,
             installationState: exports.ISTATE_PENDING_INSTALL
         };
@@ -851,6 +854,7 @@ function install(data, auditSource, callback) {
                     const newApp = _.extend({}, data, { appStoreId, manifest, location, domain, portBindings });
                     newApp.fqdn = domains.fqdn(newApp.location, domainObjectMap[newApp.domain]);
                     newApp.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+                    newApp.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 
                     eventlog.add(eventlog.ACTION_APP_INSTALL, auditSource, { appId, app: newApp, taskId: result.taskId });
 
@@ -993,9 +997,9 @@ function setCpuShares(app, cpuShares, auditSource, callback) {
     });
 }
 
-function setBinds(app, binds, auditSource, callback) {
+function setMounts(app, mounts, auditSource, callback) {
     assert.strictEqual(typeof app, 'object');
-    assert(binds && typeof binds === 'object');
+    assert(Array.isArray(mounts));
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1003,17 +1007,15 @@ function setBinds(app, binds, auditSource, callback) {
     let error = checkAppState(app, exports.ISTATE_PENDING_RECREATE_CONTAINER);
     if (error) return callback(error);
 
-    error = validateBinds(binds);
-    if (error) return callback(error);
-
     const task = {
         args: {},
-        values: { binds }
+        values: { mounts }
     };
     addTask(appId, exports.ISTATE_PENDING_RECREATE_CONTAINER, task, function (error, result) {
+        if (error && error.reason === BoxError.ALREADY_EXISTS) return callback(new BoxError(BoxError.CONFLICT, 'Duplicate mount points'));
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId, app, binds, taskId: result.taskId });
+        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId, app, mounts, taskId: result.taskId });
 
         callback(null, { taskId: result.taskId });
     });
@@ -1208,7 +1210,8 @@ function setLocation(app, data, auditSource, callback) {
         domain: data.domain.toLowerCase(),
         // these are intentionally reset, if not set
         portBindings: null,
-        alternateDomains: []
+        alternateDomains: [],
+        aliasDomains: []
     };
 
     if ('portBindings' in data) {
@@ -1228,14 +1231,20 @@ function setLocation(app, data, auditSource, callback) {
         values.alternateDomains = data.alternateDomains;
     }
 
-    const locations = [{subdomain: values.location, domain: values.domain}].concat(values.alternateDomains);
+    if ('aliasDomains' in data) {
+        values.aliasDomains = data.aliasDomains;
+    }
+
+    const locations = [{ subdomain: values.location, domain: values.domain, type: 'primary' }]
+        .concat(values.alternateDomains.map(ad => _.extend(ad, { type: 'redirect' })))
+        .concat(values.aliasDomains.map(ad => _.extend(ad, { type: 'alias' })));
 
     validateLocations(locations, function (error, domainObjectMap) {
         if (error) return callback(error);
 
         const task = {
             args: {
-                oldConfig: _.pick(app, 'location', 'domain', 'fqdn', 'alternateDomains', 'portBindings'),
+                oldConfig: _.pick(app, 'location', 'domain', 'fqdn', 'alternateDomains', 'aliasDomains', 'portBindings'),
                 overwriteDns: !!data.overwriteDns
             },
             values
@@ -1246,6 +1255,7 @@ function setLocation(app, data, auditSource, callback) {
 
             values.fqdn = domains.fqdn(values.location, domainObjectMap[values.domain]);
             values.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+            values.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 
             eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, _.extend({ appId, app, taskId: result.taskId }, values));
 
@@ -1289,7 +1299,8 @@ function update(app, data, auditSource, callback) {
 
     const skipBackup = !!data.skipBackup,
         appId = app.id,
-        manifest = data.manifest;
+        manifest = data.manifest,
+        appStoreId = data.appStoreId;
 
     let values = {};
 
@@ -1304,14 +1315,12 @@ function update(app, data, auditSource, callback) {
     error = checkManifestConstraints(manifest);
     if (error) return callback(error);
 
-    var updateConfig = { skipBackup, manifest };
+    var updateConfig = { skipBackup, manifest, appStoreId }; // this will clear appStoreId when updating from a repo and set it if passed in for update route
 
     // prevent user from installing a app with different manifest id over an existing app
     // this allows cloudron install -f --app <appid> for an app installed from the appStore
     if (app.manifest.id !== updateConfig.manifest.id) {
         if (!data.force) return callback(new BoxError(BoxError.BAD_FIELD, 'manifest id does not match. force to override'));
-        // clear appStoreId so that this app does not get updates anymore
-        updateConfig.appStoreId = '';
     }
 
     // suffix '0' if prerelease is missing for semver.lte to work as expected
@@ -1358,13 +1367,23 @@ function update(app, data, auditSource, callback) {
 
         eventlog.add(eventlog.ACTION_APP_UPDATE, auditSource, { appId, app, skipBackup, toManifest: manifest, fromManifest: app.manifest, force: data.force, taskId: result.taskId });
 
-        // clear update indicator, if update fails, it will come back through the update checker
-        updateChecker.resetAppUpdateInfo(appId);
-
         callback(null, { taskId: result.taskId });
     });
 }
 
+function getLocalLogfilePaths(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    const appId = app.id;
+
+    var filePaths = [];
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'app.log'));
+    if (app.manifest.addons && app.manifest.addons.redis) filePaths.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
+
+    return filePaths;
+}
+
 function getLogs(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
     assert(options && typeof options === 'object');
@@ -1384,11 +1403,8 @@ function getLogs(app, options, callback) {
 
     var args = [ '--lines=' + lines ];
     if (follow) args.push('--follow', '--retry', '--quiet'); // same as -F. to make it work if file doesn't exist, --quiet to not output file headers, which are no logs
-    args.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
-    args.push(path.join(paths.LOG_DIR, appId, 'app.log'));
-    if (app.manifest.addons && app.manifest.addons.redis) args.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
 
-    var cp = spawn('/usr/bin/tail', args);
+    var cp = spawn('/usr/bin/tail', args.concat(getLocalLogfilePaths(app)));
 
     var transformStream = split(function mapper(line) {
         if (format !== 'json') return line + '\n';
@@ -1576,6 +1592,26 @@ function importApp(app, data, auditSource, callback) {
     });
 }
 
+function exportApp(app, data, auditSource, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const appId = app.id;
+
+    let error = checkAppState(app, exports.ISTATE_PENDING_BACKUP);
+    if (error) return callback(error);
+
+    const task = {
+        args: { snapshotOnly: true },
+        values: {}
+    };
+    addTask(appId, exports.ISTATE_PENDING_BACKUP, task, (error, result) => {
+        if (error) return callback(error);
+
+        callback(null, { taskId: result.taskId });
+    });
+}
+
 function purchaseApp(data, callback) {
     assert.strictEqual(typeof data, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -1630,7 +1666,7 @@ function clone(app, data, user, auditSource, callback) {
         let mailboxName = hasMailAddon(manifest) ? mailboxNameForLocation(location, manifest) : null;
         let mailboxDomain = hasMailAddon(manifest) ? domain : null;
 
-        const locations = [{subdomain: location, domain}];
+        const locations = [{ subdomain: location, domain, type: 'primary' }];
         validateLocations(locations, function (error, domainObjectMap) {
             if (error) return callback(error);
 
@@ -1647,7 +1683,8 @@ function clone(app, data, user, auditSource, callback) {
                 enableBackup: app.enableBackup,
                 reverseProxyConfig: app.reverseProxyConfig,
                 env: app.env,
-                alternateDomains: []
+                alternateDomains: [],
+                aliasDomains: []
             };
 
             appdb.add(newAppId, appStoreId, manifest, location, domain, translatePortBindings(portBindings, manifest), data, function (error) {
@@ -1669,6 +1706,8 @@ function clone(app, data, user, auditSource, callback) {
                         const newApp = _.extend({}, data, { appStoreId, manifest, location, domain, portBindings });
                         newApp.fqdn = domains.fqdn(newApp.location, domainObjectMap[newApp.domain]);
                         newApp.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+                        newApp.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+
                         eventlog.add(eventlog.ACTION_APP_CLONE, auditSource, { appId: newAppId, oldAppId: appId, backupId: backupId, oldApp: app, newApp: newApp, taskId: result.taskId });
 
                         callback(null, { id: newAppId, taskId: result.taskId });
@@ -1871,8 +1910,6 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!updateInfo) return callback(null);
-
     async.eachSeries(Object.keys(updateInfo), function iterator(appId, iteratorDone) {
         get(appId, function (error, app) {
             if (error) {
@@ -2022,11 +2059,17 @@ function restartAppsUsingAddons(changedAddons, callback) {
                 args: {},
                 values: { runState: exports.RSTATE_RUNNING }
             };
-            addTask(app.id, exports.ISTATE_PENDING_RESTART, task, function (error, result) {
-                if (error) debug(`restartAppsUsingAddons: error marking ${app.fqdn} for restart: ${JSON.stringify(error)}`);
-                else debug(`restartAppsUsingAddons: marked ${app.id} for restart with taskId ${result.taskId}`);
 
-                iteratorDone(); // ignore error
+            // stop apps before updating the databases because postgres will "lock" them preventing import
+            docker.stopContainers(app.id, function (error) {
+                if (error) debug(`restartAppsUsingAddons: error stopping ${app.fqdn}`, error);
+
+                addTask(app.id, exports.ISTATE_PENDING_RESTART, task, function (error, result) {
+                    if (error) debug(`restartAppsUsingAddons: error marking ${app.fqdn} for restart: ${JSON.stringify(error)}`);
+                    else debug(`restartAppsUsingAddons: marked ${app.id} for restart with taskId ${result.taskId}`);
+
+                    iteratorDone(); // ignore error
+                });
             });
         }, callback);
     });

src/appstore.js

@@ -1,28 +1,28 @@
 'use strict';
 
 exports = module.exports = {
-    getFeatures: getFeatures,
+    getFeatures,
 
-    getApps: getApps,
-    getApp: getApp,
-    getAppVersion: getAppVersion,
+    getApps,
+    getApp,
+    getAppVersion,
 
-    trackBeginSetup: trackBeginSetup,
-    trackFinishedSetup: trackFinishedSetup,
+    trackBeginSetup,
+    trackFinishedSetup,
 
-    registerWithLoginCredentials: registerWithLoginCredentials,
+    registerWithLoginCredentials,
 
-    purchaseApp: purchaseApp,
-    unpurchaseApp: unpurchaseApp,
+    purchaseApp,
+    unpurchaseApp,
 
-    getUserToken: getUserToken,
-    getSubscription: getSubscription,
-    isFreePlan: isFreePlan,
+    getUserToken,
+    getSubscription,
+    isFreePlan,
 
-    getAppUpdate: getAppUpdate,
-    getBoxUpdate: getBoxUpdate,
+    getAppUpdate,
+    getBoxUpdate,
 
-    createTicket: createTicket
+    createTicket
 };
 
 var apps = require('./apps.js'),
@@ -32,17 +32,21 @@ var apps = require('./apps.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:appstore'),
     eventlog = require('./eventlog.js'),
+    path = require('path'),
     paths = require('./paths.js'),
     safe = require('safetydance'),
     semver = require('semver'),
     settings = require('./settings.js'),
     superagent = require('superagent'),
+    support = require('./support.js'),
     util = require('util');
 
 // These are the default options and will be adjusted once a subscription state is obtained
 // Keep in sync with appstore/routes/cloudrons.js
 let gFeatures = {
     userMaxCount: 5,
+    userGroups: false,
+    userRoles: false,
     domainMaxCount: 1,
     externalLdap: false,
     privateDockerRegistry: false,
@@ -241,17 +245,17 @@ function getBoxUpdate(options, callback) {
             automatic: options.automatic
         };
 
-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-            if (result.statusCode === 204) return callback(null); // no update
+            if (result.statusCode === 204) return callback(null, null); // no update
             if (result.statusCode !== 200 || !result.body) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
 
             var updateInfo = result.body;
 
             if (!semver.valid(updateInfo.version) || semver.gt(constants.VERSION, updateInfo.version)) {
-                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Invalid update version: %s %s', result.statusCode, result.text)));
+                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Update version invalid or is a downgrade: %s %s', result.statusCode, result.text)));
             }
 
             // updateInfo: { version, changelog, sourceTarballUrl, sourceTarballSigUrl, boxVersionsUrl, boxVersionsSigUrl }
@@ -284,7 +288,7 @@ function getAppUpdate(app, options, callback) {
             automatic: options.automatic
         };
 
-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -406,26 +410,55 @@ function createTicket(info, auditSource, callback) {
         apps.get(info.appId, callback);
     }
 
+    function enableSshIfNeeded(callback) {
+        if (!info.enableSshSupport) return callback();
+
+        support.enableRemoteSupport(true, auditSource, function (error) {
+            // ensure we can at least get the ticket through
+            if (error) debug('Unable to enable SSH support.', error);
+
+            callback();
+        });
+    }
+
     getCloudronToken(function (error, token) {
         if (error) return callback(error);
 
-        collectAppInfoIfNeeded(function (error, result) {
+        enableSshIfNeeded(function (error) {
             if (error) return callback(error);
-            if (result) info.app = result;
 
-            let url = settings.apiServerOrigin() + '/api/v1/ticket';
+            collectAppInfoIfNeeded(function (error, app) {
+                if (error) return callback(error);
+                if (app) info.app = app;
 
-            info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
+                info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
 
-            superagent.post(url).query({ accessToken: token }).send(info).timeout(10 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+                var req = superagent.post(`${settings.apiServerOrigin()}/api/v1/ticket`)
+                    .query({ accessToken: token })
+                    .timeout(30 * 1000);
 
-                eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+                // either send as JSON through body or as multipart, depending on attachments
+                if (info.app) {
+                    req.field('infoJSON', JSON.stringify(info));
 
-                callback(null, { message: `An email for sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                    apps.getLocalLogfilePaths(info.app).forEach(function (filePath) {
+                        var logs = safe.child_process.execSync(`tail --lines=1000 ${filePath}`);
+                        if (logs) req.attach(path.basename(filePath), logs, path.basename(filePath));
+                    });
+                } else {
+                    req.send(info);
+                }
+
+                req.end(function (error, result) {
+                    if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                    if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+                    if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
+                    if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+
+                    eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+
+                    callback(null, { message: `An email was sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                });
             });
         });
     });
@@ -441,7 +474,7 @@ function getApps(callback) {
             if (error) return callback(error);
 
             const url = `${settings.apiServerOrigin()}/api/v1/apps`;
-            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -476,7 +509,7 @@ function getAppVersion(appId, version, callback) {
             let url = `${settings.apiServerOrigin()}/api/v1/apps/${appId}`;
             if (version !== 'latest') url += `/versions/${version}`;
 
-            superagent.get(url).query({ accessToken: token }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));

src/apptask.js

@@ -6,7 +6,6 @@ exports = module.exports = {
     run: run,
 
     // exported for testing
-    _reserveHttpPort: reserveHttpPort,
     _configureReverseProxy: configureReverseProxy,
     _unconfigureReverseProxy: unconfigureReverseProxy,
     _createAppDir: createAppDir,
@@ -17,8 +16,7 @@ exports = module.exports = {
     _waitForDnsPropagation: waitForDnsPropagation
 };
 
-var addons = require('./addons.js'),
-    appdb = require('./appdb.js'),
+const appdb = require('./appdb.js'),
     apps = require('./apps.js'),
     assert = require('assert'),
     async = require('async'),
@@ -34,16 +32,16 @@ var addons = require('./addons.js'),
     ejs = require('ejs'),
     eventlog = require('./eventlog.js'),
     fs = require('fs'),
+    iputils = require('./iputils.js'),
     manifestFormat = require('cloudron-manifestformat'),
-    net = require('net'),
     os = require('os'),
     path = require('path'),
     paths = require('./paths.js'),
     reverseProxy = require('./reverseproxy.js'),
     rimraf = require('rimraf'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
-    sftp = require('./sftp.js'),
     shell = require('./shell.js'),
     superagent = require('superagent'),
     sysinfo = require('./sysinfo.js'),
@@ -90,22 +88,16 @@ function updateApp(app, values, callback) {
     });
 }
 
-function reserveHttpPort(app, callback) {
+function allocateContainerIp(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let server = net.createServer();
-    server.listen(0, function () {
-        let port = server.address().port;
-
-        updateApp(app, { httpPort: port }, function (error) {
-            server.close(function (/* closeError */) {
-                if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Failed to allocate http port ${port}: ${error.message}`));
-
-                callback(null);
-            });
-        });
-    });
+    async.retry({ times: 10 }, function (retryCallback) {
+        const iprange = iputils.intFromIp('172.18.20.255') - iputils.intFromIp('172.18.16.1');
+        let rnd = Math.floor(Math.random() * iprange);
+        const containerIp = iputils.ipFromInt(iputils.intFromIp('172.18.16.1') + rnd);
+        updateApp(app, { containerIp }, retryCallback);
+    }, callback);
 }
 
 function configureReverseProxy(app, callback) {
@@ -349,9 +341,9 @@ function registerSubdomains(app, overwrite, callback) {
     sysinfo.getServerIp(function (error, ip) {
         if (error) return callback(error);
 
-        const allDomains = [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains);
+        const allDomains = [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains).concat(app.aliasDomains);
 
-        debug(`registerSubdomain: Will register ${JSON.stringify(allDomains)}`);
+        debugApp(app, `registerSubdomain: Will register ${JSON.stringify(allDomains)}`);
 
         async.eachSeries(allDomains, function (domain, iteratorDone) {
             async.retry({ times: 200, interval: 5000 }, function (retryCallback) {
@@ -371,7 +363,7 @@ function registerSubdomains(app, overwrite, callback) {
 
                     domains.upsertDnsRecords(domain.subdomain, domain.domain, 'A', [ ip ], function (error) {
                         if (error && (error.reason === BoxError.BUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
-                            debug('registerSubdomains: Upsert error. Will retry.', error.message);
+                            debugApp(app, 'registerSubdomains: Upsert error. Will retry.', error.message);
                             return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
                         }
 
@@ -402,7 +394,7 @@ function unregisterSubdomains(app, allDomains, callback) {
                 domains.removeDnsRecords(domain.subdomain, domain.domain, 'A', [ ip ], function (error) {
                     if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, null);
                     if (error && (error.reason === BoxError.SBUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
-                        debug('registerSubdomains: Remove error. Will retry.', error.message);
+                        debugApp(app, 'registerSubdomains: Remove error. Will retry.', error.message);
                         return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
                     }
 
@@ -432,8 +424,8 @@ function waitForDnsPropagation(app, callback) {
         domains.waitForDnsRecord(app.location, app.domain, 'A', ip, { times: 240 }, function (error) {
             if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: app.location, domain: app.domain }));
 
-            // now wait for alternateDomains, if any
-            async.eachSeries(app.alternateDomains, function (domain, iteratorCallback) {
+            // now wait for alternateDomains and aliasDomains, if any
+            async.eachSeries(app.alternateDomains.concat(app.aliasDomains), function (domain, iteratorCallback) {
                 domains.waitForDnsRecord(domain.subdomain, domain.domain, 'A', ip, { times: 240 }, function (error) {
                     if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: domain.subdomain, domain: domain.domain }));
 
@@ -452,7 +444,7 @@ function moveDataDir(app, targetDir, callback) {
     let resolvedSourceDir = apps.getDataDir(app, app.dataDir);
     let resolvedTargetDir = apps.getDataDir(app, targetDir);
 
-    debug(`moveDataDir: migrating data from ${resolvedSourceDir} to ${resolvedTargetDir}`);
+    debugApp(app, `moveDataDir: migrating data from ${resolvedSourceDir} to ${resolvedTargetDir}`);
 
     if (resolvedSourceDir === resolvedTargetDir) return callback();
 
@@ -485,6 +477,8 @@ function downloadImage(manifest, callback) {
 }
 
 function startApp(app, callback){
+    debugApp(app, 'startApp: starting container');
+
     if (app.runState === apps.RSTATE_STOPPED) return callback();
 
     docker.startContainer(app.id, callback);
@@ -518,7 +512,7 @@ function install(app, args, progressCallback, callback) {
                 addonsToRemove = app.manifest.addons;
             }
 
-            addons.teardownAddons(app, addonsToRemove, next);
+            services.teardownAddons(app, addonsToRemove, next);
         },
 
         function deleteAppDirIfNeeded(done) {
@@ -533,7 +527,8 @@ function install(app, args, progressCallback, callback) {
             docker.deleteImage(oldManifest, done);
         },
 
-        reserveHttpPort.bind(null, app),
+        // allocating container ip here, lets the users "repair" an app if allocation fails at appdb.add time
+        allocateContainerIp.bind(null, app),
 
         progressCallback.bind(null, { percent: 20, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
@@ -551,24 +546,24 @@ function install(app, args, progressCallback, callback) {
             if (!restoreConfig) {
                 async.series([
                     progressCallback.bind(null, { percent: 60, message: 'Setting up addons' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
                 ], next);
             } else if (!restoreConfig.backupId) { // in-place import
                 async.series([
                     progressCallback.bind(null, { percent: 60, message: 'Importing addons in-place' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
-                    addons.clearAddons.bind(null, app, _.omit(app.manifest.addons, 'localstorage')),
-                    addons.restoreAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
+                    services.clearAddons.bind(null, app, _.omit(app.manifest.addons, 'localstorage')),
+                    services.restoreAddons.bind(null, app, app.manifest.addons),
                 ], next);
             } else {
                 async.series([
                     progressCallback.bind(null, { percent: 65, message: 'Download backup and restoring addons' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
-                    addons.clearAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
+                    services.clearAddons.bind(null, app, app.manifest.addons),
                     backups.downloadApp.bind(null, app, restoreConfig, (progress) => {
                         progressCallback({ percent: 65, message: progress.message });
                     }),
-                    addons.restoreAddons.bind(null, app, app.manifest.addons)
+                    services.restoreAddons.bind(null, app, app.manifest.addons)
                 ], next);
             }
         },
@@ -578,9 +573,6 @@ function install(app, args, progressCallback, callback) {
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 80, message: 'Configuring file manager' }),
-        sftp.rebuild.bind(null, {}),
-
         progressCallback.bind(null, { percent: 85, message: 'Waiting for DNS propagation' }),
         exports._waitForDnsPropagation.bind(null, app),
 
@@ -606,7 +598,7 @@ function backup(app, args, progressCallback, callback) {
 
     async.series([
         progressCallback.bind(null, { percent: 10, message: 'Backing up' }),
-        backups.backupApp.bind(null, app, { /* options */ }, (progress) => {
+        backups.backupApp.bind(null, app, { snapshotOnly: !!args.snapshotOnly }, (progress) => {
             progressCallback({ percent: 30, message: progress.message });
         }),
 
@@ -634,7 +626,7 @@ function create(app, args, progressCallback, callback) {
 
         // FIXME: re-setup addons only because sendmail addon to re-inject env vars on mailboxName change
         progressCallback.bind(null, { percent: 30, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
@@ -671,6 +663,12 @@ function changeLocation(app, args, progressCallback, callback) {
                 return !app.alternateDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; });
             });
 
+            if (oldConfig.aliasDomains) {
+                obsoleteDomains = obsoleteDomains.concat(oldConfig.aliasDomains.filter(function (o) {
+                    return !app.aliasDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; });
+                }));
+            }
+
             if (locationChanged) obsoleteDomains.push({ subdomain: oldConfig.location, domain: oldConfig.domain });
 
             if (obsoleteDomains.length === 0) return next();
@@ -683,7 +681,7 @@ function changeLocation(app, args, progressCallback, callback) {
 
         // re-setup addons since they rely on the app's fqdn (e.g oauth)
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
@@ -725,7 +723,7 @@ function migrateDataDir(app, args, progressCallback, callback) {
 
         // re-setup addons since this creates the localStorage volume
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, _.extend({}, app, { dataDir: newDataDir }), app.manifest.addons),
+        services.setupAddons.bind(null, _.extend({}, app, { dataDir: newDataDir }), app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Moving data dir' }),
         moveDataDir.bind(null, app, newDataDir),
@@ -743,11 +741,7 @@ function migrateDataDir(app, args, progressCallback, callback) {
             return updateApp(app, { installationState: apps.ISTATE_ERROR, error: makeTaskError(error, app) }, callback.bind(null, error));
         }
 
-        // We do this after the app has the new data commited to the database
-        sftp.rebuild({}, function (error) {
-            if (error) debug('migrateDataDir: failed to rebuild sftp addon:', error);
-            callback();
-        });
+        callback();
     });
 }
 
@@ -762,7 +756,6 @@ function configure(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 10, message: 'Cleaning up old install' }),
         unconfigureReverseProxy.bind(null, app),
         deleteContainers.bind(null, app, { managedOnly: true }),
-        reserveHttpPort.bind(null, app),
 
         progressCallback.bind(null, { percent: 20, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
@@ -775,16 +768,13 @@ function configure(app, args, progressCallback, callback) {
 
         // re-setup addons since they rely on the app's fqdn (e.g oauth)
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 80, message: 'Configuring file manager' }),
-        sftp.rebuild.bind(null, {}),
-
         progressCallback.bind(null, { percent: 90, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
 
@@ -800,7 +790,6 @@ function configure(app, args, progressCallback, callback) {
     });
 }
 
-// nginx configuration is skipped because app.httpPort is expected to be available
 function update(app, args, progressCallback, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof args, 'object');
@@ -812,7 +801,10 @@ function update(app, args, progressCallback, callback) {
 
     // app does not want these addons anymore
     // FIXME: this does not handle option changes (like multipleDatabases)
-    var unusedAddons = _.omit(app.manifest.addons, Object.keys(updateConfig.manifest.addons));
+    const unusedAddons = _.omit(app.manifest.addons, Object.keys(updateConfig.manifest.addons));
+    const httpPathsChanged = app.manifest.httpPaths !== updateConfig.manifest.httpPaths;
+    const httpPortChanged = app.manifest.httpPort !== updateConfig.manifest.httpPort;
+    const proxyAuthChanged = !_.isEqual(safe.query(app.manifest, 'addons.proxyAuth'), safe.query(updateConfig.manifest, 'addons.proxyAuth'));
 
     async.series([
         // this protects against the theoretical possibility of an app being marked for update from
@@ -851,7 +843,7 @@ function update(app, args, progressCallback, callback) {
         },
 
         // only delete unused addons after backup
-        addons.teardownAddons.bind(null, app, unusedAddons),
+        services.teardownAddons.bind(null, app, unusedAddons),
 
         // free unused ports
         function (next) {
@@ -863,7 +855,7 @@ function update(app, args, progressCallback, callback) {
                 if (newTcpPorts[portName] || newUdpPorts[portName]) return callback(null); // port still in use
 
                 appdb.delPortBinding(currentPorts[portName], apps.PORT_TYPE_TCP, function (error) {
-                    if (error && error.reason === BoxError.NOT_FOUND) debug('update: portbinding does not exist in database', error);
+                    if (error && error.reason === BoxError.NOT_FOUND) debugApp(app, 'update: portbinding does not exist in database', error);
                     else if (error) return next(error);
 
                     // also delete from app object for further processing (the db is updated in the next step)
@@ -880,15 +872,19 @@ function update(app, args, progressCallback, callback) {
         downloadIcon.bind(null, app),
 
         progressCallback.bind(null, { percent: 60, message: 'Updating addons' }),
-        addons.setupAddons.bind(null, app, updateConfig.manifest.addons),
+        services.setupAddons.bind(null, app, updateConfig.manifest.addons),
 
         progressCallback.bind(null, { percent: 70, message: 'Creating container' }),
         createContainer.bind(null, app),
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 80, message: 'Configuring file manager' }),
-        sftp.rebuild.bind(null, {}),
+        progressCallback.bind(null, { percent: 90, message: 'Configuring reverse proxy' }),
+        function (next) {
+            if (!httpPathsChanged && !proxyAuthChanged && !httpPortChanged) return next();
+
+            configureReverseProxy(app, next);
+        },
 
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null, updateTime: new Date() })
@@ -913,13 +909,16 @@ function start(app, args, progressCallback, callback) {
 
     async.series([
         progressCallback.bind(null, { percent: 10, message: 'Starting app services' }),
-        addons.startAppServices.bind(null, app),
+        services.startAppServices.bind(null, app),
 
         progressCallback.bind(null, { percent: 35, message: 'Starting container' }),
         docker.startContainer.bind(null, app.id),
 
+        progressCallback.bind(null, { percent: 60, message: 'Adding collectd profile' }),
+        addCollectdProfile.bind(null, app),
+
         // stopped apps do not renew certs. currently, we don't do DNS to not overwrite existing user settings
-        progressCallback.bind(null, { percent: 60, message: 'Configuring reverse proxy' }),
+        progressCallback.bind(null, { percent: 80, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
 
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
@@ -944,7 +943,10 @@ function stop(app, args, progressCallback, callback) {
         docker.stopContainers.bind(null, app.id),
 
         progressCallback.bind(null, { percent: 50, message: 'Stopping app services' }),
-        addons.stopAppServices.bind(null, app),
+        services.stopAppServices.bind(null, app),
+
+        progressCallback.bind(null, { percent: 80, message: 'Removing collectd profile' }),
+        removeCollectdProfile.bind(null, app),
 
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null })
@@ -990,13 +992,9 @@ function uninstall(app, args, progressCallback, callback) {
         deleteContainers.bind(null, app, {}),
 
         progressCallback.bind(null, { percent: 30, message: 'Teardown addons' }),
-        addons.teardownAddons.bind(null, app, app.manifest.addons),
+        services.teardownAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 40, message: 'Cleanup file manager' }),
-        function (callback) {
-            if (!app.dataDir) return callback();
-            sftp.rebuild({ ignoredApps: [ app.id ] }, callback);
-        },
 
         progressCallback.bind(null, { percent: 50, message: 'Deleting app data directory' }),
         deleteAppDir.bind(null, app, { removeDirectory: true }),
@@ -1005,7 +1003,7 @@ function uninstall(app, args, progressCallback, callback) {
         docker.deleteImage.bind(null, app.manifest),
 
         progressCallback.bind(null, { percent: 70, message: 'Unregistering domains' }),
-        unregisterSubdomains.bind(null, app, [ { subdomain: app.location, domain: app.domain } ].concat(app.alternateDomains)),
+        unregisterSubdomains.bind(null, app, [ { subdomain: app.location, domain: app.domain } ].concat(app.alternateDomains).concat(app.aliasDomains)),
 
         progressCallback.bind(null, { percent: 80, message: 'Cleanup icon' }),
         removeIcon.bind(null, app),

src/apptaskmanager.js

@@ -1,7 +1,7 @@
 'use strict';
 
 exports = module.exports = {
-    scheduleTask: scheduleTask
+    scheduleTask
 };
 
 let assert = require('assert'),
@@ -12,6 +12,8 @@ let assert = require('assert'),
     safe = require('safetydance'),
     path = require('path'),
     paths = require('./paths.js'),
+    scheduler = require('./scheduler.js'),
+    services = require('./services.js'),
     tasks = require('./tasks.js');
 
 let gActiveTasks = { }; // indexed by app id
@@ -35,9 +37,10 @@ function initializeSync() {
 }
 
 // callback is called when task is finished
-function scheduleTask(appId, taskId, callback) {
+function scheduleTask(appId, taskId, options, callback) {
     assert.strictEqual(typeof appId, 'string');
     assert.strictEqual(typeof taskId, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     if (!gInitialized) initializeSync();
@@ -49,7 +52,7 @@ function scheduleTask(appId, taskId, callback) {
     if (Object.keys(gActiveTasks).length >= TASK_CONCURRENCY) {
         debug(`Reached concurrency limit, queueing task id ${taskId}`);
         tasks.update(taskId, { percent: 1, message: 'Waiting for other app tasks to complete' }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
+        gPendingTasks.push({ appId, taskId, options, callback });
         return;
     }
 
@@ -58,7 +61,7 @@ function scheduleTask(appId, taskId, callback) {
     if (lockError) {
         debug(`Could not get lock. ${lockError.message}, queueing task id ${taskId}`);
         tasks.update(taskId, { percent: 1, message: waitText(lockError.operation) }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
+        gPendingTasks.push({ appId, taskId, options, callback });
         return;
     }
 
@@ -68,12 +71,17 @@ function scheduleTask(appId, taskId, callback) {
 
     if (!fs.existsSync(path.dirname(logFile))) safe.fs.mkdirSync(path.dirname(logFile)); // ensure directory
 
-    // TODO: set memory limit for app backup task
-    tasks.startTask(taskId, { logFile, timeout: 20 * 60 * 60 * 1000 /* 20 hours */, nice: 15 }, function (error, result) {
+    scheduler.suspendJobs(appId);
+
+    tasks.startTask(taskId, Object.assign(options, { logFile }), function (error, result) {
         callback(error, result);
 
         delete gActiveTasks[appId];
         locker.unlock(locker.OP_APPTASK); // unlock event will trigger next task
+
+        // post app task hooks
+        services.rebuildService('sftp', error => { if (error) debug('Unable to rebuild sftp:', error); });
+        scheduler.resumeJobs(appId);
     });
 }
 
@@ -83,6 +91,6 @@ function startNextTask() {
     assert(Object.keys(gActiveTasks).length < TASK_CONCURRENCY);
 
     const t = gPendingTasks.shift();
-    scheduleTask(t.appId, t.taskId, t.callback);
+    scheduleTask(t.appId, t.taskId, t.options, t.callback);
 }
 

src/autoconfig.xml.ejs

@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<clientConfig version="1.1">
+  <emailProvider id="<%= domain %>">
+    <domain><%= domain %></domain>
+    <displayName>Cloudron Mail</displayName>
+    <displayShortName>Cloudron</displayShortName>
+    <incomingServer type="imap">
+      <hostname><%= mailFqdn %></hostname>
+      <port>993</port>
+      <socketType>SSL</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </incomingServer>
+    <outgoingServer type="smtp">
+      <hostname><%= mailFqdn %></hostname>
+      <port>587</port>
+      <socketType>STARTTLS</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+      <addThisServer>true</addThisServer>
+    </outgoingServer>
+
+    <documentation url="http://cloudron.io/email/#autodiscover">
+       <descr lang="en">Cloudron Email</descr>
+    </documentation>
+
+  </emailProvider>
+</clientConfig>
+

src/backups.js

@@ -1,36 +1,36 @@
 'use strict';
 
 exports = module.exports = {
-    testConfig: testConfig,
-    testProviderConfig: testProviderConfig,
+    testConfig,
+    testProviderConfig,
 
     getByIdentifierAndStatePaged,
 
-    get: get,
+    get,
 
-    startBackupTask: startBackupTask,
+    startBackupTask,
 
-    restore: restore,
+    restore,
 
-    backupApp: backupApp,
-    downloadApp: downloadApp,
+    backupApp,
+    downloadApp,
 
-    backupBoxAndApps: backupBoxAndApps,
+    backupBoxAndApps,
 
-    upload: upload,
+    upload,
 
-    startCleanupTask: startCleanupTask,
-    cleanup: cleanup,
-    cleanupCacheFilesSync: cleanupCacheFilesSync,
+    startCleanupTask,
+    cleanup,
+    cleanupCacheFilesSync,
 
-    injectPrivateFields: injectPrivateFields,
-    removePrivateFields: removePrivateFields,
+    injectPrivateFields,
+    removePrivateFields,
 
-    checkConfiguration: checkConfiguration,
+    checkConfiguration,
 
-    configureCollectd: configureCollectd,
+    configureCollectd,
 
-    generateEncryptionKeysSync: generateEncryptionKeysSync,
+    generateEncryptionKeysSync,
 
     BACKUP_IDENTIFIER_BOX: 'box',
 
@@ -48,8 +48,7 @@ exports = module.exports = {
     _applyBackupRetentionPolicy: applyBackupRetentionPolicy
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     async = require('async'),
     assert = require('assert'),
     backupdb = require('./backupdb.js'),
@@ -72,6 +71,7 @@ var addons = require('./addons.js'),
     progressStream = require('progress-stream'),
     safe = require('safetydance'),
     shell = require('./shell.js'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     syncer = require('./syncer.js'),
     tar = require('tar-fs'),
@@ -380,9 +380,11 @@ function createReadStream(sourceFile, encryption) {
 
     stream.on('error', function (error) {
         debug(`createReadStream: read stream error at ${sourceFile}`, error);
-        ps.emit('error', new BoxError(BoxError.FS_ERROR, `Error reading ${sourceFile}: ${error.message}`));
+        ps.emit('error', new BoxError(BoxError.FS_ERROR, `Error reading ${sourceFile}: ${error.message} ${error.code}`));
     });
 
+    stream.on('open', () => ps.emit('open'));
+
     if (encryption) {
         let encryptStream = new EncryptStream(encryption);
 
@@ -515,18 +517,19 @@ function sync(backupConfig, backupId, dataLayout, progressCallback, callback) {
                 progressCallback({ message: `Adding ${task.path}` + (retryCount > 1 ?  ` (Try ${retryCount})` : '') });
                 debug(`Adding ${task.path} position ${task.position} try ${retryCount}`);
                 var stream = createReadStream(dataLayout.toLocalPath('./' + task.path), backupConfig.encryption);
-                stream.on('error', function (error) {
-                    debug(`read stream error for ${task.path}: ${error.message}`);
-                    retryCallback();
-                }); // ignore error if file disappears
+                stream.on('error', (error) => retryCallback(error.message.includes('ENOENT') ? null : error)); // ignore error if file disappears
                 stream.on('progress', function (progress) {
                     const transferred = Math.round(progress.transferred/1024/1024), speed = Math.round(progress.speed/1024/1024);
                     if (!transferred && !speed) return progressCallback({ message: `Uploading ${task.path}` }); // 0M@0MBps looks wrong
                     progressCallback({ message: `Uploading ${task.path}: ${transferred}M@${speed}MBps` }); // 0M@0MBps looks wrong
                 });
-                api(backupConfig.provider).upload(backupConfig, backupFilePath, stream, function (error) {
-                    debug(error ? `Error uploading ${task.path} try ${retryCount}: ${error.message}` : `Uploaded ${task.path}`);
-                    retryCallback(error);
+                // only create the destination path when we have confirmation that the source is available. otherwise, we end up with
+                // files owned as 'root' and the cp later will fail
+                stream.on('open', function () {
+                    api(backupConfig.provider).upload(backupConfig, backupFilePath, stream, function (error) {
+                        debug(error ? `Error uploading ${task.path} try ${retryCount}: ${error.message}` : `Uploaded ${task.path}`);
+                        retryCallback(error);
+                    });
                 });
             }
         }, iteratorCallback);
@@ -848,15 +851,23 @@ function runBackupUpload(uploadConfig, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    const { backupId, format, dataLayout, progressTag } = uploadConfig;
+    const { backupId, backupConfig, dataLayout, progressTag } = uploadConfig;
     assert.strictEqual(typeof backupId, 'string');
-    assert.strictEqual(typeof format, 'string');
+    assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof progressTag, 'string');
     assert(dataLayout instanceof DataLayout, 'dataLayout must be a DataLayout');
 
     let result = ''; // the script communicates error result as a string
 
-    shell.sudo(`backup-${backupId}`, [ BACKUP_UPLOAD_CMD, backupId, format, dataLayout.toString() ], { preserveEnv: true, ipc: true }, function (error) {
+    // https://stackoverflow.com/questions/48387040/node-js-recommended-max-old-space-size
+    const envCopy = Object.assign({}, process.env);
+    if (backupConfig.memoryLimit && backupConfig.memoryLimit >= 2*1024*1024*1024) {
+        const heapSize = Math.min((backupConfig.memoryLimit/1024/1024) - 256, 8192);
+        debug(`runBackupUpload: adjusting heap size to ${heapSize}M`);
+        envCopy.NODE_OPTIONS = `--max-old-space-size=${heapSize}`;
+    }
+
+    shell.sudo(`backup-${backupId}`, [ BACKUP_UPLOAD_CMD, backupId, backupConfig.format, dataLayout.toString() ], { env: envCopy, preserveEnv: true, ipc: true }, function (error) {
         if (error && (error.code === null /* signal */ || (error.code !== 0 && error.code !== 50))) { // backuptask crashed
             return callback(new BoxError(BoxError.INTERNAL_ERROR, 'Backuptask crashed'));
         } else if (error && error.code === 50) { // exited with error
@@ -925,7 +936,7 @@ function uploadBoxSnapshot(backupConfig, progressCallback, callback) {
 
         const uploadConfig = {
             backupId: 'snapshot/box',
-            format: backupConfig.format,
+            backupConfig,
             dataLayout: new DataLayout(boxDataDir, []),
             progressTag: 'box'
         };
@@ -944,9 +955,10 @@ function uploadBoxSnapshot(backupConfig, progressCallback, callback) {
     });
 }
 
-function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, callback) {
+function rotateBoxBackup(backupConfig, tag, options, appBackupIds, progressCallback, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof tag, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert(Array.isArray(appBackupIds));
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
@@ -977,7 +989,7 @@ function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, call
         copy.on('done', function (copyBackupError) {
             const state = copyBackupError ? exports.BACKUP_STATE_ERROR : exports.BACKUP_STATE_NORMAL;
 
-            backupdb.update(backupId, { state }, function (error) {
+            backupdb.update(backupId, { preserveSecs: options.preserveSecs || 0, state }, function (error) {
                 if (copyBackupError) return callback(copyBackupError);
                 if (error) return callback(error);
 
@@ -989,9 +1001,10 @@ function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, call
     });
 }
 
-function backupBoxWithAppBackupIds(appBackupIds, tag, progressCallback, callback) {
+function backupBoxWithAppBackupIds(appBackupIds, tag, options, progressCallback, callback) {
     assert(Array.isArray(appBackupIds));
     assert.strictEqual(typeof tag, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1001,7 +1014,7 @@ function backupBoxWithAppBackupIds(appBackupIds, tag, progressCallback, callback
         uploadBoxSnapshot(backupConfig, progressCallback, function (error) {
             if (error) return callback(error);
 
-            rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, callback);
+            rotateBoxBackup(backupConfig, tag, options, appBackupIds, progressCallback, callback);
         });
     });
 }
@@ -1032,7 +1045,7 @@ function snapshotApp(app, progressCallback, callback) {
         return callback(new BoxError(BoxError.FS_ERROR, 'Error creating config.json: ' + safe.error.message));
     }
 
-    addons.backupAddons(app, app.manifest.addons, function (error) {
+    services.backupAddons(app, app.manifest.addons, function (error) {
         if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error.message));
 
         debugApp(app, `snapshotApp: took ${(new Date() - startTime)/1000} seconds`);
@@ -1110,7 +1123,7 @@ function uploadAppSnapshot(backupConfig, app, progressCallback, callback) {
 
         const uploadConfig = {
             backupId,
-            format: backupConfig.format,
+            backupConfig,
             dataLayout,
             progressTag: app.fqdn
         };
@@ -1162,6 +1175,8 @@ function backupApp(app, options, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
+    if (options.snapshotOnly) return snapshotApp(app, progressCallback, callback);
+
     const tag = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
 
     debug(`backupApp - Backing up ${app.fqdn} with tag ${tag}`);
@@ -1170,7 +1185,8 @@ function backupApp(app, options, progressCallback, callback) {
 }
 
 // this function expects you to have a lock. Unlike other progressCallback this also has a progress field
-function backupBoxAndApps(progressCallback, callback) {
+function backupBoxAndApps(options, progressCallback, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1192,7 +1208,7 @@ function backupBoxAndApps(progressCallback, callback) {
             }
 
             const startTime = new Date();
-            backupAppWithTag(app, tag, { /* options */ }, (progress) => progressCallback({ percent: percent, message: progress.message }), function (error, backupId) {
+            backupAppWithTag(app, tag, options, (progress) => progressCallback({ percent: percent, message: progress.message }), function (error, backupId) {
                 if (error) {
                     debugApp(app, 'Unable to backup', error);
                     return iteratorCallback(error);
@@ -1210,7 +1226,7 @@ function backupBoxAndApps(progressCallback, callback) {
             progressCallback({ percent: percent, message: 'Backing up system data' });
             percent += step;
 
-            backupBoxWithAppBackupIds(backupIds, tag, (progress) => progressCallback({ percent: percent, message: progress.message }), callback);
+            backupBoxWithAppBackupIds(backupIds, tag, options, (progress) => progressCallback({ percent: percent, message: progress.message }), callback);
         });
     });
 }
@@ -1224,12 +1240,12 @@ function startBackupTask(auditSource, callback) {
 
         const memoryLimit = 'memoryLimit' in backupConfig ? Math.max(backupConfig.memoryLimit/1024/1024, 400) : 400;
 
-        tasks.add(tasks.TASK_BACKUP, [ ], function (error, taskId) {
+        tasks.add(tasks.TASK_BACKUP, [ { /* options */ } ], function (error, taskId) {
             if (error) return callback(error);
 
             eventlog.add(eventlog.ACTION_BACKUP_START, auditSource, { taskId });
 
-            tasks.startTask(taskId, { timeout: 12 * 60 * 60 * 1000 /* 12 hours */, nice: 15, memoryLimit }, function (error, backupId) {
+            tasks.startTask(taskId, { timeout: 24 * 60 * 60 * 1000 /* 24 hours */, nice: 15, memoryLimit }, function (error, backupId) {
                 locker.unlock(locker.OP_FULL_BACKUP);
 
                 const errorMessage = error ? error.message : '';
@@ -1520,9 +1536,9 @@ function checkConfiguration(callback) {
 
         let message = '';
         if (backupConfig.provider === 'noop') {
-            message = 'Cloudron backups are disabled. Please ensure this server is backed up using alternate means. See https://cloudron.io/documentation/backups/#storage-providers for more information.';
+            message = 'Cloudron backups are disabled. Please ensure this server is backed up using alternate means. See https://docs.cloudron.io/backups/#storage-providers for more information.';
         } else if (backupConfig.provider === 'filesystem' && !backupConfig.externalDisk) {
-            message = 'Cloudron backups are currently on the same disk as the Cloudron server instance. This is dangerous and can lead to complete data loss if the disk fails. See https://cloudron.io/documentation/backups/#storage-providers for storing backups in an external location.';
+            message = 'Cloudron backups are currently on the same disk as the Cloudron server instance. This is dangerous and can lead to complete data loss if the disk fails. See https://docs.cloudron.io/backups/#storage-providers for storing backups in an external location.';
         }
 
         callback(null, message);

src/boxerror.js

@@ -50,6 +50,7 @@ BoxError.FS_ERROR = 'FileSystem Error';
 BoxError.INACTIVE = 'Inactive';
 BoxError.INTERNAL_ERROR = 'Internal Error';
 BoxError.INVALID_CREDENTIALS = 'Invalid Credentials';
+BoxError.IPTABLES_ERROR = 'IPTables Error';
 BoxError.LICENSE_ERROR = 'License Error';
 BoxError.LOGROTATE_ERROR = 'Logrotate Error';
 BoxError.MAIL_ERROR = 'Mail Error';
@@ -92,6 +93,7 @@ BoxError.toHttpError = function (error) {
     case BoxError.MAIL_ERROR:
     case BoxError.DOCKER_ERROR:
     case BoxError.ADDONS_ERROR:
+    case BoxError.IPTABLES_ERROR:
         return new HttpError(424, error);
     case BoxError.DATABASE_ERROR:
     case BoxError.INTERNAL_ERROR:

src/branding.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports = module.exports = {
+    renderFooter
+};
+
+const assert = require('assert'),
+    constants = require('./constants.js');
+
+function renderFooter(footer) {
+    assert.strictEqual(typeof footer, 'string');
+
+    const year = new Date().getFullYear();
+
+    return footer.replace(/%YEAR%/g, year)
+        .replace(/%VERSION%/g, constants.VERSION);
+}
+

src/cert/acme2.js

@@ -139,7 +139,7 @@ Acme2.prototype.updateContact = function (registrationUri, callback) {
     const that = this;
     this.sendSignedRequest(registrationUri, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to update contact. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to update contact. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug(`updateContact: contact of user updated to ${that.email}`);
 
@@ -160,7 +160,7 @@ Acme2.prototype.registerUser = function (callback) {
     this.sendSignedRequest(this.directory.newAccount, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
         // 200 if already exists. 201 for new accounts
-        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register new account. Expecting 200 or 201, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to register new account. Expecting 200 or 201, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug(`registerUser: user registered keyid: ${result.headers.location}`);
 
@@ -185,8 +185,8 @@ Acme2.prototype.newOrder = function (domain, callback) {
 
     this.sendSignedRequest(this.directory.newOrder, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode === 403) return callback(new BoxError(BoxError.ACCESS_DENIED, `Forbidden sending signed request: ${result.body.detail}`));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register user. Expecting 201, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode === 403) return callback(new BoxError(BoxError.ACCESS_DENIED, `Forbidden sending new order: ${result.body.detail}`));
+        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to send new order. Expecting 201, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug('newOrder: created order %s %j', domain, result.body);
 
@@ -259,7 +259,7 @@ Acme2.prototype.notifyChallengeReady = function (challenge, callback) {
 
     this.sendSignedRequest(challenge.url, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to notify challenge. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to notify challenge. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         callback();
     });
@@ -313,7 +313,7 @@ Acme2.prototype.signCertificate = function (domain, finalizationUrl, csrDer, cal
     this.sendSignedRequest(finalizationUrl, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
         // 429 means we reached the cert limit for this domain
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to sign certificate. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to sign certificate. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         return callback(null);
     });
@@ -362,7 +362,7 @@ Acme2.prototype.downloadCertificate = function (hostname, certUrl, callback) {
         that.postAsGet(certUrl, function (error, result) {
             if (error) return retryCallback(new BoxError(BoxError.NETWORK_ERROR, `Network error when downloading certificate: ${error.message}`));
             if (result.statusCode === 202) return retryCallback(new BoxError(BoxError.TRY_AGAIN, 'Retry downloading certificate'));
-            if (result.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to get cert. Expecting 200, got %s %s', result.statusCode, result.text)));
+            if (result.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to get cert. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
             const fullChainPem = result.body; // buffer
 
@@ -586,34 +586,34 @@ Acme2.prototype.getDirectory = function (callback) {
     });
 };
 
-Acme2.prototype.getCertificate = function (hostname, domain, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+Acme2.prototype.getCertificate = function (vhost, domain, callback) {
+    assert.strictEqual(typeof vhost, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`getCertificate: start acme flow for ${hostname} from ${this.caDirectory}`);
+    debug(`getCertificate: start acme flow for ${vhost} from ${this.caDirectory}`);
 
-    if (hostname !== domain && this.wildcard) { // bare domain is not part of wildcard SAN
-        hostname = domains.makeWildcard(hostname);
-        debug(`getCertificate: will get wildcard cert for ${hostname}`);
+    if (vhost !== domain && this.wildcard) { // bare domain is not part of wildcard SAN
+        vhost = domains.makeWildcard(vhost);
+        debug(`getCertificate: will get wildcard cert for ${vhost}`);
     }
 
     const that = this;
     this.getDirectory(function (error) {
         if (error) return callback(error);
 
-        that.acmeFlow(hostname, domain, function (error) {
+        that.acmeFlow(vhost, domain, function (error) {
             if (error) return callback(error);
 
             var outdir = paths.APP_CERTS_DIR;
-            const certName = hostname.replace('*.', '_.');
+            const certName = vhost.replace('*.', '_.');
             callback(null, path.join(outdir, `${certName}.cert`), path.join(outdir, `${certName}.key`));
         });
     });
 };
 
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+function getCertificate(vhost, domain, options, callback) {
+    assert.strictEqual(typeof vhost, 'string'); // this can also be a wildcard domain (for alias domains)
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -623,6 +623,6 @@ function getCertificate(hostname, domain, options, callback) {
         debug(`getCertificate: attempt ${attempt++}`);
 
         let acme = new Acme2(options || { });
-        acme.getCertificate(hostname, domain, retryCallback);
+        acme.getCertificate(vhost, domain, retryCallback);
     }, callback);
 }

src/cloudron.js

@@ -1,34 +1,34 @@
 'use strict';
 
 exports = module.exports = {
-    initialize: initialize,
-    uninitialize: uninitialize,
-    getConfig: getConfig,
-    getLogs: getLogs,
+    initialize,
+    uninitialize,
+    getConfig,
+    getLogs,
 
-    reboot: reboot,
-    isRebootRequired: isRebootRequired,
+    reboot,
+    isRebootRequired,
 
-    onActivated: onActivated,
+    onActivated,
 
-    prepareDashboardDomain: prepareDashboardDomain,
-    setDashboardDomain: setDashboardDomain,
-    setDashboardAndMailDomain: setDashboardAndMailDomain,
-    renewCerts: renewCerts,
+    setupDnsAndCert,
 
-    setupDashboard: setupDashboard,
+    prepareDashboardDomain,
+    setDashboardDomain,
+    updateDashboardDomain,
+    renewCerts,
 
-    runSystemChecks: runSystemChecks
+    runSystemChecks
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     appstore = require('./appstore.js'),
     assert = require('assert'),
     async = require('async'),
     auditSource = require('./auditsource.js'),
     backups = require('./backups.js'),
     BoxError = require('./boxerror.js'),
+    branding = require('./branding.js'),
     constants = require('./constants.js'),
     cron = require('./cron.js'),
     debug = require('debug')('box:cloudron'),
@@ -42,10 +42,12 @@ var addons = require('./addons.js'),
     platform = require('./platform.js'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     spawn = require('child_process').spawn,
     split = require('split'),
+    sysinfo = require('./sysinfo.js'),
     tasks = require('./tasks.js'),
     users = require('./users.js');
 
@@ -79,12 +81,15 @@ function onActivated(callback) {
     async.series([
         platform.start,
         cron.startJobs,
-        function checkBackupConfiguration(callback) {
+        function checkBackupConfiguration(done) {
             backups.checkConfiguration(function (error, message) {
-                if (error) return callback(error);
-                notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, callback);
+                if (error) return done(error);
+                notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, done);
             });
-        }
+        },
+        // disable responding to api calls via IP to not leak domain info. this is carefully placed as the last item, so it buys
+        // the UI some time to query the dashboard domain in the restore code path
+        (done) => setTimeout(() => reverseProxy.writeDefaultConfig({ activated :true }, done), 30000)
     ], callback);
 }
 
@@ -109,27 +114,49 @@ function notifyUpdate(callback) {
 
 // each of these tasks can fail. we will add some routes to fix/re-run them
 function runStartupTasks() {
-    // stop all the systemd tasks
-    platform.stopAllTasks(NOOP_CALLBACK);
+    const tasks = [
+        // stop all the systemd tasks
+        platform.stopAllTasks,
 
-    // configure nginx to be reachable by IP
-    reverseProxy.writeDefaultConfig(NOOP_CALLBACK);
+        // this configures collectd to collect backup storage metrics if filesystem is used. This is also triggerd when the settings change with the rest api
+        function (callback) {
+            settings.getBackupConfig(function (error, backupConfig) {
+                if (error) return callback(error);
 
-    // this configures collectd to collect backup storage metrics if filesystem is used. This is also triggerd when the settings change with the rest api
-    settings.getBackupConfig(function (error, backupConfig) {
-        if (error) return debug('runStartupTasks: failed to get backup config.', error);
-        backups.configureCollectd(backupConfig, NOOP_CALLBACK);
-    });
+                backups.configureCollectd(backupConfig, callback);
+            });
+        },
 
-    // always generate webadmin config since we have no versioning mechanism for the ejs
-    if (settings.adminDomain()) reverseProxy.writeAdminConfig(settings.adminDomain(), NOOP_CALLBACK);
+        // always generate webadmin config since we have no versioning mechanism for the ejs
+        function (callback) {
+            if (!settings.adminDomain()) return callback();
 
-    // check activation state and start the platform
-    users.isActivated(function (error, activated) {
-        if (error) return debug(error);
-        if (!activated) return debug('initialize: not activated yet'); // not activated
+            reverseProxy.writeDashboardConfig(settings.adminDomain(), callback);
+        },
 
-        onActivated(NOOP_CALLBACK);
+        // check activation state and start the platform
+        function (callback) {
+            users.isActivated(function (error, activated) {
+                if (error) return callback(error);
+
+                // configure nginx to be reachable by IP when not activated. for the moment, the IP based redirect exists even after domain is setup
+                // just in case user forgot or some network error happenned in the middle (then browser refresh takes you to activation page)
+                // we remove the config as a simple security measure to not expose IP <-> domain
+                if (!activated) {
+                    debug('runStartupTasks: not activated. generating IP based redirection config');
+                    return reverseProxy.writeDefaultConfig({ activated: false }, callback);
+                }
+
+                onActivated(callback);
+            });
+        }
+    ];
+
+    // we used to run tasks in parallel but simultaneous nginx reloads was causing issues
+    async.series(async.reflectAll(tasks), function (error, results) {
+        results.forEach((result, idx) => {
+            if (result.error) debug(`Startup task at index ${idx} failed: ${result.error.message}`);
+        });
     });
 }
 
@@ -149,7 +176,7 @@ function getConfig(callback) {
             version: constants.VERSION,
             isDemo: settings.isDemo(),
             cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-            footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
+            footer: branding.renderFooter(allSettings[settings.FOOTER_KEY] || constants.FOOTER),
             features: appstore.getFeatures(),
             profileLocked: allSettings[settings.DIRECTORY_CONFIG_KEY].lockUserProfiles,
             mandatory2FA: allSettings[settings.DIRECTORY_CONFIG_KEY].mandatory2FA
@@ -272,7 +299,7 @@ function prepareDashboardDomain(domain, auditSource, callback) {
             const conflict = result.filter(app => app.fqdn === fqdn);
             if (conflict.length) return callback(new BoxError(BoxError.BAD_STATE, 'Dashboard location conflicts with an existing app'));
 
-            tasks.add(tasks.TASK_PREPARE_DASHBOARD_DOMAIN, [ domain, auditSource ], function (error, taskId) {
+            tasks.add(tasks.TASK_SETUP_DNS_AND_CERT, [ constants.ADMIN_LOCATION, domain, auditSource ], function (error, taskId) {
                 if (error) return callback(error);
 
                 tasks.startTask(taskId, {}, NOOP_CALLBACK);
@@ -294,12 +321,12 @@ function setDashboardDomain(domain, auditSource, callback) {
     domains.get(domain, function (error, domainObject) {
         if (error) return callback(error);
 
-        reverseProxy.writeAdminConfig(domain, function (error) {
+        reverseProxy.writeDashboardConfig(domain, function (error) {
             if (error) return callback(error);
 
             const fqdn = domains.fqdn(constants.ADMIN_LOCATION, domainObject);
 
-            settings.setAdmin(domain, fqdn, function (error) {
+            settings.setAdminLocation(domain, fqdn, function (error) {
                 if (error) return callback(error);
 
                 eventlog.add(eventlog.ACTION_DASHBOARD_DOMAIN_UPDATE, auditSource, { domain: domain, fqdn: fqdn });
@@ -311,36 +338,24 @@ function setDashboardDomain(domain, auditSource, callback) {
 }
 
 // call this only post activation because it will restart mail server
-function setDashboardAndMailDomain(domain, auditSource, callback) {
+function updateDashboardDomain(domain, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`setDashboardAndMailDomain: ${domain}`);
+    debug(`updateDashboardDomain: ${domain}`);
 
     if (settings.isDemo()) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
 
     setDashboardDomain(domain, auditSource, function (error) {
         if (error) return callback(error);
 
-        mail.onMailFqdnChanged(NOOP_CALLBACK); // this will update dns and re-configure mail server
-        addons.restartService('turn', NOOP_CALLBACK); // to update the realm variable
+        services.rebuildService('turn', NOOP_CALLBACK); // to update the realm variable
 
         callback(null);
     });
 }
 
-function setupDashboard(auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof callback, 'function');
-
-    async.series([
-        domains.prepareDashboardDomain.bind(null, settings.adminDomain(), auditSource, progressCallback),
-        setDashboardDomain.bind(null, settings.adminDomain(), auditSource)
-    ], callback);
-}
-
 function renewCerts(options, auditSource, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof auditSource, 'object');
@@ -354,3 +369,34 @@ function renewCerts(options, auditSource, callback) {
         callback(null, taskId);
     });
 }
+
+function setupDnsAndCert(subdomain, domain, auditSource, progressCallback, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    domains.get(domain, function (error, domainObject) {
+        if (error) return callback(error);
+
+        const adminFqdn = domains.fqdn(subdomain, domainObject);
+
+        sysinfo.getServerIp(function (error, ip) {
+            if (error) return callback(error);
+
+            async.series([
+                (done) => { progressCallback({ message: `Updating DNS of ${adminFqdn}` }); done(); },
+                domains.upsertDnsRecords.bind(null, subdomain, domain, 'A', [ ip ]),
+                (done) => { progressCallback({ message: `Waiting for DNS of ${adminFqdn}` }); done(); },
+                domains.waitForDnsRecord.bind(null, subdomain, domain, 'A', ip, { interval: 30000, times: 50000 }),
+                (done) => { progressCallback({ message: `Getting certificate of ${adminFqdn}` }); done(); },
+                reverseProxy.ensureCertificate.bind(null, domains.fqdn(subdomain, domainObject), domain, auditSource)
+            ], function (error) {
+                if (error) return callback(error);
+
+                callback(null);
+            });
+        });
+    });
+}

src/collectd/volume.ejs

@@ -0,0 +1,9 @@
+<Plugin python>
+  <Module du>
+    <Path>
+        Instance "<%= volumeId %>"
+        Dir "<%= hostPath %>"
+    </Path>
+  </Module>
+</Plugin>
+

src/constants.js

@@ -26,7 +26,7 @@ exports = module.exports = {
 
     PORT: CLOUDRON ? 3000 : 5454,
     INTERNAL_SMTP_PORT: 2525, // this value comes from the mail container
-    SYSADMIN_PORT: 3001, // unused
+    AUTHWALL_PORT: 3001,
     LDAP_PORT: 3002,
     DOCKER_PROXY_PORT: 3003,
 
@@ -37,7 +37,14 @@ exports = module.exports = {
     DEFAULT_MEMORY_LIMIT: (256 * 1024 * 1024), // see also client.js
 
     DEMO_USERNAME: 'cloudron',
-    DEMO_BLACKLISTED_APPS: [ 'com.github.cloudtorrent' ],
+    DEMO_BLACKLISTED_APPS: [
+        'com.github.cloudtorrent',
+        'net.alltubedownload.cloudronapp',
+        'com.adguard.home.cloudronapp',
+        'com.transmissionbt.cloudronapp',
+        'io.github.sickchill.cloudronapp',
+        'to.couchpota.cloudronapp'
+    ],
 
     AUTOUPDATE_PATTERN_NEVER: 'never',
 
@@ -48,8 +55,8 @@ exports = module.exports = {
 
     SUPPORT_EMAIL: 'support@cloudron.io',
 
-    FOOTER: '&copy; 2020  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
+    FOOTER: '&copy; %YEAR%  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
 
-    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '5.1.1-test'
+    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '6.0.1-test'
 };
 

src/cron.js

@@ -4,8 +4,7 @@
 // If the patterns overlap all the time, then the task may not ever get a chance to run!
 // If you change this change dashboard patterns in settings.html
 const DEFAULT_CLEANUP_BACKUPS_PATTERN = '00 30 1,3,5,23 * * *',
-    DEFAULT_BOX_AUTOUPDATE_PATTERN = '00 00 1,3,5,23 * * *',
-    DEFAULT_APP_AUTOUPDATE_PATTERN = '00 15 1,3,5,23 * * *';
+    DEFAULT_AUTOUPDATE_PATTERN = '00 00 1,3,5,23 * * *';
 
 exports = module.exports = {
     startJobs,
@@ -14,8 +13,7 @@ exports = module.exports = {
 
     handleSettingsChanged,
 
-    DEFAULT_BOX_AUTOUPDATE_PATTERN,
-    DEFAULT_APP_AUTOUPDATE_PATTERN
+    DEFAULT_AUTOUPDATE_PATTERN,
 };
 
 var appHealthMonitor = require('./apphealthmonitor.js'),
@@ -35,14 +33,13 @@ var appHealthMonitor = require('./apphealthmonitor.js'),
     settings = require('./settings.js'),
     system = require('./system.js'),
     updater = require('./updater.js'),
-    updateChecker = require('./updatechecker.js');
+    updateChecker = require('./updatechecker.js'),
+    _ = require('underscore');
 
-var gJobs = {
-    appAutoUpdater: null,
-    boxAutoUpdater: null,
-    appUpdateChecker: null,
+const gJobs = {
+    autoUpdater: null,
     backup: null,
-    boxUpdateChecker: null,
+    updateChecker: null,
     systemChecks: null,
     diskSpaceChecker: null,
     certificateRenew: null,
@@ -55,7 +52,7 @@ var gJobs = {
     appHealthMonitor: null
 };
 
-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+const NOOP_CALLBACK = function (error) { if (error) debug(error); };
 
 // cron format
 // Seconds: 0-59
@@ -68,7 +65,9 @@ var NOOP_CALLBACK = function (error) { if (error) debug(error); };
 function startJobs(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    const randomMinute = Math.floor(60*Math.random());
+    debug('startJobs: starting cron jobs');
+
+    const randomTick = Math.floor(60*Math.random());
     gJobs.systemChecks = new CronJob({
         cronTime: '00 30 2 * * *', // once a day. if you change this interval, change the notification messages with correct duration
         onTick: () => cloudron.runSystemChecks(NOOP_CALLBACK),
@@ -81,16 +80,10 @@ function startJobs(callback) {
         start: true
     });
 
-    gJobs.boxUpdateCheckerJob = new CronJob({
-        cronTime: '00 ' + randomMinute + ' 1,3,5,21,23 * * *', // 5 times
-        onTick: () => updateChecker.checkBoxUpdates({ automatic: true }, NOOP_CALLBACK),
-        start: true
-    });
-
     // this is run separately from the update itself so that the user can disable automatic updates but can still get a notification
-    gJobs.appUpdateChecker = new CronJob({
-        cronTime: '00 ' + randomMinute + ' 2,4,6,20,22 * * *', // 5 times
-        onTick: () => updateChecker.checkAppUpdates({ automatic: true }, NOOP_CALLBACK),
+    gJobs.updateCheckerJob = new CronJob({
+        cronTime: `${randomTick} ${randomTick} 1,5,9,13,17,21,23 * * *`,
+        onTick: () => updateChecker.checkForUpdates({ automatic: true }, NOOP_CALLBACK),
         start: true
     });
 
@@ -141,8 +134,7 @@ function startJobs(callback) {
 
         const tz = allSettings[settings.TIME_ZONE_KEY];
         backupConfigChanged(allSettings[settings.BACKUP_CONFIG_KEY], tz);
-        appAutoupdatePatternChanged(allSettings[settings.APP_AUTOUPDATE_PATTERN_KEY], tz);
-        boxAutoupdatePatternChanged(allSettings[settings.BOX_AUTOUPDATE_PATTERN_KEY], tz);
+        autoupdatePatternChanged(allSettings[settings.AUTOUPDATE_PATTERN_KEY], tz);
         dynamicDnsChanged(allSettings[settings.DYNAMIC_DNS_KEY]);
 
         callback();
@@ -157,8 +149,7 @@ function handleSettingsChanged(key, value) {
     switch (key) {
     case settings.TIME_ZONE_KEY:
     case settings.BACKUP_CONFIG_KEY:
-    case settings.APP_AUTOUPDATE_PATTERN_KEY:
-    case settings.BOX_AUTOUPDATE_PATTERN_KEY:
+    case settings.AUTOUPDATE_PATTERN_KEY:
     case settings.DYNAMIC_DNS_KEY:
         debug('handleSettingsChanged: recreating all jobs');
         async.series([
@@ -187,53 +178,36 @@ function backupConfigChanged(value, tz) {
     });
 }
 
-function boxAutoupdatePatternChanged(pattern, tz) {
+function autoupdatePatternChanged(pattern, tz) {
     assert.strictEqual(typeof pattern, 'string');
     assert.strictEqual(typeof tz, 'string');
 
-    debug(`boxAutoupdatePatternChanged: pattern - ${pattern} (${tz})`);
+    debug(`autoupdatePatternChanged: pattern - ${pattern} (${tz})`);
 
-    if (gJobs.boxAutoUpdater) gJobs.boxAutoUpdater.stop();
+    if (gJobs.autoUpdater) gJobs.autoUpdater.stop();
 
     if (pattern === constants.AUTOUPDATE_PATTERN_NEVER) return;
 
-    gJobs.boxAutoUpdater = new CronJob({
+    gJobs.autoUpdater = new CronJob({
         cronTime: pattern,
         onTick: function() {
-            var updateInfo = updateChecker.getUpdateInfo();
-            if (updateInfo.box) {
-                debug('Starting autoupdate to %j', updateInfo.box);
+            const updateInfo = updateChecker.getUpdateInfo();
+            // do box before app updates. for the off chance that the box logic fixes some app update logic issue
+            if (updateInfo.box && !updateInfo.box.unstable) {
+                debug('Starting box autoupdate to %j', updateInfo.box);
                 updater.updateToLatest({ skipBackup: false }, auditSource.CRON, NOOP_CALLBACK);
-            } else {
-                debug('No box auto updates available');
+                return;
             }
-        },
-        start: true,
-        timeZone: tz
-    });
-}
 
-function appAutoupdatePatternChanged(pattern, tz) {
-    assert.strictEqual(typeof pattern, 'string');
-    assert.strictEqual(typeof tz, 'string');
-
-    debug(`appAutoupdatePatternChanged: pattern ${pattern} (${tz})`);
-
-    if (gJobs.appAutoUpdater) gJobs.appAutoUpdater.stop();
-
-    if (pattern === constants.AUTOUPDATE_PATTERN_NEVER) return;
-
-    gJobs.appAutoUpdater = new CronJob({
-        cronTime: pattern,
-        onTick: function() {
-            var updateInfo = updateChecker.getUpdateInfo();
-            if (updateInfo.apps) {
-                debug('Starting app update to %j', updateInfo.apps);
-                apps.autoupdateApps(updateInfo.apps, auditSource.CRON, NOOP_CALLBACK);
+            const appUpdateInfo = _.omit(updateInfo, 'box');
+            if (Object.keys(appUpdateInfo).length > 0) {
+                debug('Starting app update to %j', appUpdateInfo);
+                apps.autoupdateApps(appUpdateInfo, auditSource.CRON, NOOP_CALLBACK);
             } else {
                 debug('No app auto updates available');
             }
         },
+
         start: true,
         timeZone: tz
     });

src/dns/linode.js

@@ -119,9 +119,10 @@ function get(domainObject, location, type, callback) {
         zoneName = domainObject.zoneName,
         name = domains.getName(domainObject, location, type) || '';
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { records } = result;
         var tmp = records.map(function (record) { return record.target; });
 
         debug('get: %j', tmp);
@@ -143,9 +144,10 @@ function upsert(domainObject, location, type, values, callback) {
 
     debug('upsert: %s for zone %s of type %s with values %j', name, zoneName, type, values);
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { zoneId, records } = result;
         let i = 0, recordIds = []; // used to track available records to update instead of create
 
         async.eachSeries(values, function (value, iteratorCallback) {
@@ -222,9 +224,10 @@ function del(domainObject, location, type, values, callback) {
         zoneName = domainObject.zoneName,
         name = domains.getName(domainObject, location, type) || '';
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { zoneId, records } = result;
         if (records.length === 0) return callback(null);
 
         var tmp = records.filter(function (record) { return values.some(function (value) { return value === record.target; }); });
@@ -287,7 +290,7 @@ function verifyDnsConfig(domainObject, callback) {
         if (error || !nameservers) return callback(new BoxError(BoxError.BAD_FIELD, error ? error.message : 'Unable to get nameservers', { field: 'nameservers' }));
 
         if (nameservers.map(function (n) { return n.toLowerCase(); }).indexOf('ns1.linode.com') === -1) {
-            debug('verifyDnsConfig: %j does not contains DO NS', nameservers);
+            debug('verifyDnsConfig: %j does not contains linode NS', nameservers);
             return callback(new BoxError(BoxError.BAD_FIELD, 'Domain nameservers are not set to Linode', { field: 'nameservers' }));
         }
 

src/dns/namecheap.js

@@ -78,13 +78,11 @@ function getInternal(dnsConfig, zoneName, subdomain, type, callback) {
 
                     return callback(new BoxError(BoxError.EXTERNAL_ERROR, errorMessage));
                 }
-                if (!tmp.CommandResponse[0]) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response'));
-                if (!tmp.CommandResponse[0].DomainDNSGetHostsResult[0]) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response'));
-
-                var hosts = result.ApiResponse.CommandResponse[0].DomainDNSGetHostsResult[0].host.map(function (h) {
-                    return h['$'];
-                });
+                const host = safe.query(tmp, 'CommandResponse[0].DomainDNSGetHostsResult[0].host');
+                if (!host) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Invalid response: ${JSON.stringify(tmp)}`));
+                if (!Array.isArray(host)) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `host is not an array: ${JSON.stringify(tmp)}`));
 
+                const hosts = host.map(h => h['$']);
                 callback(null, hosts);
             });
         });

src/dns/netcup.js

@@ -0,0 +1,303 @@
+'use strict';
+
+exports = module.exports = {
+    removePrivateFields: removePrivateFields,
+    injectPrivateFields: injectPrivateFields,
+    upsert: upsert,
+    get: get,
+    del: del,
+    wait: wait,
+    verifyDnsConfig: verifyDnsConfig
+};
+
+var assert = require('assert'),
+    BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
+    debug = require('debug')('box:dns/netcup'),
+    dns = require('../native-dns.js'),
+    domains = require('../domains.js'),
+    superagent = require('superagent'),
+    util = require('util'),
+    waitForDns = require('./waitfordns.js');
+
+var API_ENDPOINT = 'https://ccp.netcup.net/run/webservice/servers/endpoint.php?JSON';
+
+function formatError(response) {
+    if (response.body) return util.format('Netcup DNS error [%s] %s', response.body.statuscode, response.body.longmessage);
+    else return util.format('Netcup DNS error [%s] %s', response.statusCode, response.text);
+}
+
+function removePrivateFields(domainObject) {
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
+    return domainObject;
+}
+
+function injectPrivateFields(newConfig, currentConfig) {
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+}
+
+// returns a api session id
+function login(dnsConfig, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const data = {
+        action: 'login',
+        param:{
+            apikey: dnsConfig.apiKey,
+            apipassword: dnsConfig.apiPassword,
+            customernumber: dnsConfig.customerNumber
+        }
+    };
+
+    superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+        callback(null, result.body.responsedata.apisessionid);
+    });
+}
+
+function getAllRecords(dnsConfig, apiSessionId, zoneName, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof apiSessionId, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug(`getAllRecords: getting dns records of ${zoneName}`);
+
+    const data = {
+        action: 'infoDnsRecords',
+        param:{
+            apikey: dnsConfig.apiKey,
+            apisessionid: apiSessionId,
+            customernumber: dnsConfig.customerNumber,
+            domainname: zoneName,
+        }
+    };
+
+    superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+        debug('getAllRecords:', JSON.stringify(result.body.responsedata.dnsrecords || []));
+
+        callback(null, result.body.responsedata.dnsrecords || []);
+    });
+}
+
+function upsert(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('upsert: %s for zone %s of type %s with values %j', name, zoneName, type, values);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            let records = [];
+
+            values.forEach(function (value) {
+                // remove possible quotation
+                if (value.charAt(0) === '"') value = value.slice(1);
+                if (value.charAt(value.length -1) === '"') value = value.slice(0, -1);
+
+                let priority = null;
+                if (type === 'MX') {
+                    priority = parseInt(value.split(' ')[0], 10);
+                    value = value.split(' ')[1];
+                }
+
+                let record = result.find(function (r) { return r.hostname === name && r.type === type; });
+                if (!record) record = { hostname: name, type: type, destination: value, deleterecord: false };
+                else record.destination = value;
+
+                if (priority !== null) record.priority = priority;
+
+                records.push(record);
+            });
+
+            const data = {
+                action: 'updateDnsRecords',
+                param:{
+                    apikey: dnsConfig.apiKey,
+                    apisessionid: apiSessionId,
+                    customernumber: dnsConfig.customerNumber,
+                    domainname: zoneName,
+                    dnsrecordset: {
+                        dnsrecords: records
+                    }
+                }
+            };
+
+            debug('upserting', JSON.stringify(data));
+
+            superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+                if (result.body.statuscode !== 2000) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                debug('upsert:', result.body);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function get(domainObject, location, type, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('get: %s for zone %s of type %s', name, zoneName, type);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            // We only return the value string
+            callback(null, result.filter(function (r) { return r.hostname === name && r.type === type; }).map(function (r) { return r.destination; }));
+        });
+    });
+}
+
+function del(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('del: %s for zone %s of type %s with values %j', name, zoneName, type, values);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            let records = [];
+
+            values.forEach(function (value) {
+                // remove possible quotation
+                if (value.charAt(0) === '"') value = value.slice(1);
+                if (value.charAt(value.length -1) === '"') value = value.slice(0, -1);
+
+                let record = result.find(function (r) { return r.hostname === name && r.type === type && r.destination === value; });
+                if (!record) return;
+
+                record.deleterecord = true;
+
+                records.push(record);
+            });
+
+            if (records.length === 0) return callback(null);
+
+            const data = {
+                action: 'updateDnsRecords',
+                param:{
+                    apikey: dnsConfig.apiKey,
+                    apisessionid: apiSessionId,
+                    customernumber: dnsConfig.customerNumber,
+                    domainname: zoneName,
+                    dnsrecordset: {
+                        dnsrecords: records
+                    }
+                }
+            };
+
+            superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+                if (result.body.statuscode !== 2000) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                debug('del:', result.body.responsedata);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function wait(domainObject, location, type, value, options, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof value, 'string');
+    assert(options && typeof options === 'object'); // { interval: 5000, times: 50000 }
+    assert.strictEqual(typeof callback, 'function');
+
+    const fqdn = domains.fqdn(location, domainObject);
+
+    waitForDns(fqdn, domainObject.zoneName, type, value, options, callback);
+}
+
+function verifyDnsConfig(domainObject, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName;
+
+    if (!dnsConfig.customerNumber || typeof dnsConfig.customerNumber !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'customerNumber must be a non-empty string', { field: 'customerNumber' }));
+    if (!dnsConfig.apiKey || typeof dnsConfig.apiKey !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'apiKey must be a non-empty string', { field: 'apiKey' }));
+    if (!dnsConfig.apiPassword || typeof dnsConfig.apiPassword !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'apiPassword must be a non-empty string', { field: 'apiPassword' }));
+
+    const ip = '127.0.0.1';
+
+    var credentials = {
+        customerNumber: dnsConfig.customerNumber,
+        apiKey: dnsConfig.apiKey,
+        apiPassword: dnsConfig.apiPassword,
+    };
+
+    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here
+
+    dns.resolve(zoneName, 'NS', { timeout: 5000 }, function (error, nameservers) {
+        if (error && error.code === 'ENOTFOUND') return callback(new BoxError(BoxError.BAD_FIELD, 'Unable to resolve nameservers for this domain', { field: 'nameservers' }));
+        if (error || !nameservers) return callback(new BoxError(BoxError.BAD_FIELD, error ? error.message : 'Unable to get nameservers', { field: 'nameservers' }));
+
+        if (!nameservers.every(function (n) { return n.toLowerCase().indexOf('dns.netcup.net') !== -1; })) {
+            debug('verifyDnsConfig: %j does not contains Netcup NS', nameservers);
+            return callback(new BoxError(BoxError.BAD_FIELD, 'Domain nameservers are not set to Netcup', { field: 'nameservers' }));
+        }
+
+        const location = 'cloudrontestdns';
+
+        upsert(domainObject, location, 'A', [ ip ], function (error) {
+            if (error) return callback(error);
+
+            debug('verifyDnsConfig: Test A record added');
+
+            del(domainObject, location, 'A', [ ip ], function (error) {
+                if (error) return callback(error);
+
+                debug('verifyDnsConfig: Test A record removed again');
+
+                callback(null, credentials);
+            });
+        });
+    });
+}

src/docker.js

@@ -1,38 +1,39 @@
 'use strict';
 
 exports = module.exports = {
-    testRegistryConfig: testRegistryConfig,
-    setRegistryConfig: setRegistryConfig,
-    injectPrivateFields: injectPrivateFields,
-    removePrivateFields: removePrivateFields,
+    testRegistryConfig,
+    setRegistryConfig,
+    injectPrivateFields,
+    removePrivateFields,
 
-    ping: ping,
+    ping,
 
-    info: info,
-    downloadImage: downloadImage,
-    createContainer: createContainer,
-    startContainer: startContainer,
-    restartContainer: restartContainer,
-    stopContainer: stopContainer,
+    info,
+    downloadImage,
+    createContainer,
+    startContainer,
+    restartContainer,
+    stopContainer,
     stopContainerByName: stopContainer,
-    stopContainers: stopContainers,
-    deleteContainer: deleteContainer,
-    deleteContainerByName: deleteContainer,
-    deleteImage: deleteImage,
-    deleteContainers: deleteContainers,
-    createSubcontainer: createSubcontainer,
-    getContainerIdByIp: getContainerIdByIp,
-    inspect: inspect,
+    stopContainers,
+    deleteContainer,
+    deleteImage,
+    deleteContainers,
+    createSubcontainer,
+    getContainerIdByIp,
+    inspect,
+    getContainerIp,
     inspectByName: inspect,
-    execContainer: execContainer,
-    getEvents: getEvents,
-    memoryUsage: memoryUsage,
-    createVolume: createVolume,
-    removeVolume: removeVolume,
-    clearVolume: clearVolume
+    execContainer,
+    getEvents,
+    memoryUsage,
+    createVolume,
+    removeVolume,
+    clearVolume,
+    update
 };
 
-var addons = require('./addons.js'),
+const apps = require('./apps.js'),
     async = require('async'),
     assert = require('assert'),
     BoxError = require('./boxerror.js'),
@@ -40,11 +41,15 @@ var addons = require('./addons.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:docker'),
     Docker = require('dockerode'),
+    os = require('os'),
     path = require('path'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     safe = require('safetydance'),
+    system = require('./system.js'),
     util = require('util'),
+    volumes = require('./volumes.js'),
     _ = require('underscore');
 
 const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
@@ -171,26 +176,53 @@ function downloadImage(manifest, callback) {
 
     debug('downloadImage %s', manifest.dockerImage);
 
-    var attempt = 1;
+    const image = gConnection.getImage(manifest.dockerImage);
 
-    async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
-        debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+    image.inspect(function (error, result) {
+        if (!error && result) return callback(null); // image is already present locally
 
-        pullImage(manifest, retryCallback);
-    }, callback);
+        let attempt = 1;
+
+        async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
+            debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+
+            pullImage(manifest, retryCallback);
+        }, callback);
+    });
 }
 
-function getBindsSync(app) {
+function getBinds(app, callback) {
     assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (app.mounts.length === 0) return callback(null);
 
     let binds = [];
 
-    for (let name of Object.keys(app.binds)) {
-        const bind = app.binds[name];
-        binds.push(`${bind.hostPath}:/media/${name}:${bind.readOnly ? 'ro' : 'rw'}`);
+    volumes.list(function (error, result) {
+        if (error) return callback(error);
+        let volumesById = {};
+        result.forEach(r => volumesById[r.id] = r);
+
+        for (const mount of app.mounts) {
+            const volume = volumesById[mount.volumeId];
+            binds.push(`${volume.hostPath}:/media/${volume.name}:${mount.readOnly ? 'ro' : 'rw'}`);
+        }
+
+        callback(null, binds);
+    });
+}
+
+function getLowerUpIp() { // see getifaddrs and IFF_LOWER_UP and netdevice
+    const ni = os.networkInterfaces(); // { lo: [], eth0: [] }
+    for (const iname of Object.keys(ni)) {
+        if (iname === 'lo') continue;
+        for (const address of ni[iname]) {
+            if (!address.internal && address.family === 'IPv4') return address.address;
+        }
     }
 
-    return binds;
+    return null;
 }
 
 function createSubcontainer(app, name, cmd, options, callback) {
@@ -200,12 +232,11 @@ function createSubcontainer(app, name, cmd, options, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let isAppContainer = !cmd; // non app-containers are like scheduler and exec (terminal) containers
+    let isAppContainer = !cmd; // non app-containers are like scheduler
 
     var manifest = app.manifest;
     var exposedPorts = {}, dockerPortBindings = { };
     var domain = app.fqdn;
-    const hostname = isAppContainer ? app.id : name;
 
     const envPrefix = manifest.manifestVersion <= 1 ? '' : 'CLOUDRON_';
 
@@ -219,11 +250,6 @@ function createSubcontainer(app, name, cmd, options, callback) {
         `${envPrefix}APP_DOMAIN=${domain}`
     ];
 
-    // docker portBindings requires ports to be exposed
-    exposedPorts[manifest.httpPort + '/tcp'] = {};
-
-    dockerPortBindings[manifest.httpPort + '/tcp'] = [ { HostIp: '127.0.0.1', HostPort: app.httpPort + '' } ];
-
     var portEnv = [];
     for (let portName in app.portBindings) {
         const hostPort = app.portBindings[portName];
@@ -232,105 +258,121 @@ function createSubcontainer(app, name, cmd, options, callback) {
 
         var containerPort = ports[portName].containerPort || hostPort;
 
+        // docker portBindings requires ports to be exposed
         exposedPorts[`${containerPort}/${portType}`] = {};
         portEnv.push(`${portName}=${hostPort}`);
 
-        dockerPortBindings[`${containerPort}/${portType}`] = [ { HostIp: '0.0.0.0', HostPort: hostPort + '' } ];
+        const hostIp = hostPort === 53 ? getLowerUpIp() : '0.0.0.0'; // port 53 is special because it is possibly taken by systemd-resolved
+        dockerPortBindings[`${containerPort}/${portType}`] = [ { HostIp: hostIp, HostPort: hostPort + '' } ];
     }
 
     let appEnv = [];
     Object.keys(app.env).forEach(function (name) { appEnv.push(`${name}=${app.env[name]}`); });
 
-    // first check db record, then manifest
-    var memoryLimit = app.memoryLimit || manifest.memoryLimit || 0;
-
-    if (memoryLimit === -1) { // unrestricted
-        memoryLimit = 0;
-    } else if (memoryLimit === 0 || memoryLimit < constants.DEFAULT_MEMORY_LIMIT) { // ensure we never go below minimum (in case we change the default)
-        memoryLimit = constants.DEFAULT_MEMORY_LIMIT;
-    }
+    let memoryLimit = apps.getMemoryLimit(app);
 
     // give scheduler tasks twice the memory limit since background jobs take more memory
     // if required, we can make this a manifest and runtime argument later
     if (!isAppContainer) memoryLimit *= 2;
 
-    addons.getEnvironment(app, function (error, addonEnv) {
+    services.getEnvironment(app, function (error, addonEnv) {
         if (error) return callback(error);
 
-        // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
-        // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
-        // name to look up the internal docker ip. this makes curl from within container fail
-        // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
-        // Hostname cannot be set with container NetworkMode
-        var containerOptions = {
-            name: name, // for referencing containers
-            Tty: isAppContainer,
-            Hostname: hostname,
-            Image: app.manifest.dockerImage,
-            Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
-            Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
-            ExposedPorts: isAppContainer ? exposedPorts : { },
-            Volumes: { // see also ReadonlyRootfs
-                '/tmp': {},
-                '/run': {}
-            },
-            Labels: {
-                'fqdn': app.fqdn,
-                'appId': app.id,
-                'isSubcontainer': String(!isAppContainer),
-                'isCloudronManaged': String(true)
-            },
-            HostConfig: {
-                Mounts: addons.getMountsSync(app, app.manifest.addons),
-                Binds: getBindsSync(app), // ideally, we have to use 'Mounts' but we have to create volumes then
-                LogConfig: {
-                    Type: 'syslog',
-                    Config: {
-                        'tag': app.id,
-                        'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
-                        'syslog-format': 'rfc5424'
-                    }
+        getBinds(app, function (error, binds) {
+            if (error) return callback(error);
+
+            let containerOptions = {
+                name: name, // for referencing containers
+                Tty: isAppContainer,
+                Image: app.manifest.dockerImage,
+                Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
+                Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
+                ExposedPorts: isAppContainer ? exposedPorts : { },
+                Volumes: { // see also ReadonlyRootfs
+                    '/tmp': {},
+                    '/run': {}
                 },
-                Memory: memoryLimit / 2,
-                MemorySwap: memoryLimit, // Memory + Swap
-                PortBindings: isAppContainer ? dockerPortBindings : { },
-                PublishAllPorts: false,
-                ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
-                RestartPolicy: {
-                    'Name': isAppContainer ? 'unless-stopped' : 'no',
-                    'MaximumRetryCount': 0
+                Labels: {
+                    'fqdn': app.fqdn,
+                    'appId': app.id,
+                    'isSubcontainer': String(!isAppContainer),
+                    'isCloudronManaged': String(true)
                 },
-                CpuShares: app.cpuShares,
-                VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
-                NetworkMode: 'cloudron', // user defined bridge network
-                Dns: ['172.18.0.1'], // use internal dns
-                DnsSearch: ['.'], // use internal dns
-                SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
-                CapAdd: [],
-                CapDrop: []
-            },
-            NetworkingConfig: {
-                EndpointsConfig: {
-                    cloudron: {
-                        Aliases: [ name ] // this allows sub-containers reach app containers by name
-                    }
+                HostConfig: {
+                    Mounts: services.getMountsSync(app, app.manifest.addons),
+                    Binds: binds, // ideally, we have to use 'Mounts' but we have to create volumes then
+                    LogConfig: {
+                        Type: 'syslog',
+                        Config: {
+                            'tag': app.id,
+                            'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
+                            'syslog-format': 'rfc5424'
+                        }
+                    },
+                    Memory: system.getMemoryAllocation(memoryLimit),
+                    MemorySwap: memoryLimit, // Memory + Swap
+                    PortBindings: isAppContainer ? dockerPortBindings : { },
+                    PublishAllPorts: false,
+                    ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
+                    RestartPolicy: {
+                        'Name': isAppContainer ? 'unless-stopped' : 'no',
+                        'MaximumRetryCount': 0
+                    },
+                    CpuShares: app.cpuShares,
+                    VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
+                    SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
+                    CapAdd: [],
+                    CapDrop: []
                 }
+            };
+
+            // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
+            // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
+            // name to look up the internal docker ip. this makes curl from within container fail
+            // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
+            // Hostname cannot be set with container NetworkMode. Subcontainers run is the network space of the app container
+            // This is done to prevent lots of up/down events and iptables locking
+            if (isAppContainer) {
+                containerOptions.Hostname = app.id;
+                containerOptions.HostConfig.NetworkMode = 'cloudron'; // user defined bridge network
+                containerOptions.HostConfig.Dns = ['172.18.0.1']; // use internal dns
+                containerOptions.HostConfig.DnsSearch = ['.']; // use internal dns
+
+                containerOptions.NetworkingConfig = {
+                    EndpointsConfig: {
+                        cloudron: {
+                            IPAMConfig: {
+                                IPv4Address: app.containerIp
+                            },
+                            Aliases: [ name ] // adds hostname entry with container name
+                        }
+                    }
+                };
+            } else {
+                containerOptions.HostConfig.NetworkMode = `container:${app.containerId}`; // scheduler containers must have same IP as app for various addon auth
             }
-        };
 
-        var capabilities = manifest.capabilities || [];
+            var capabilities = manifest.capabilities || [];
 
-        // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
-        if (capabilities.includes('net_admin')) containerOptions.HostConfig.CapAdd.push('NET_ADMIN', 'NET_RAW');
-        if (capabilities.includes('mlock')) containerOptions.HostConfig.CapAdd.push('IPC_LOCK'); // mlock prevents swapping
-        if (!capabilities.includes('ping')) containerOptions.HostConfig.CapDrop.push('NET_RAW'); // NET_RAW is included by default by Docker
+            // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
+            if (capabilities.includes('net_admin')) containerOptions.HostConfig.CapAdd.push('NET_ADMIN', 'NET_RAW');
+            if (capabilities.includes('mlock')) containerOptions.HostConfig.CapAdd.push('IPC_LOCK'); // mlock prevents swapping
+            if (!capabilities.includes('ping')) containerOptions.HostConfig.CapDrop.push('NET_RAW'); // NET_RAW is included by default by Docker
 
-        containerOptions = _.extend(containerOptions, options);
+            if (capabilities.includes('vaapi') && safe.fs.existsSync('/dev/dri')) {
+                containerOptions.HostConfig.Devices = [
+                    { PathOnHost: '/dev/dri', PathInContainer: '/dev/dri', CgroupPermissions: 'rwm' }
+                ];
+            }
 
-        gConnection.createContainer(containerOptions, function (error, container) {
-            if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+            containerOptions = _.extend(containerOptions, options);
 
-            callback(null, container);
+            gConnection.createContainer(containerOptions, function (error, container) {
+                if (error && error.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
+                if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+
+                callback(null, container);
+            });
         });
     });
 }
@@ -395,7 +437,7 @@ function stopContainer(containerId, callback) {
     });
 }
 
-function deleteContainer(containerId, callback) {
+function deleteContainer(containerId, callback) { // id can also be name
     assert(!containerId || typeof containerId === 'string');
     assert.strictEqual(typeof callback, 'function');
 
@@ -514,6 +556,22 @@ function inspect(containerId, callback) {
     });
 }
 
+function getContainerIp(containerId, callback) {
+    assert.strictEqual(typeof containerId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (constants.TEST) return callback(null, '127.0.5.5');
+
+    inspect(containerId, function (error, result) {
+        if (error) return callback(error);
+
+        const ip = safe.query(result, 'NetworkSettings.Networks.cloudron.IPAddress', null);
+        if (!ip) return callback(new BoxError(BoxError.DOCKER_ERROR, 'Error getting container IP'));
+
+        callback(null, ip);
+    });
+}
+
 function execContainer(containerId, options, callback) {
     assert.strictEqual(typeof containerId, 'string');
     assert.strictEqual(typeof options, 'object');
@@ -636,3 +694,17 @@ function info(callback) {
         callback(null, result);
     });
 }
+
+function update(name, memory, memorySwap, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof memory, 'number');
+    assert.strictEqual(typeof memorySwap, 'number');
+    assert.strictEqual(typeof callback, 'function');
+
+    const args = `update --memory ${memory} --memory-swap ${memorySwap} ${name}`.split(' ');
+    // scale back db containers, if possible. this is retried because updating memory constraints can fail
+    // with failed to write to memory.memsw.limit_in_bytes: write /sys/fs/cgroup/memory/docker/xx/memory.memsw.limit_in_bytes: device or resource busy
+    async.retry({ times: 10, interval: 60 * 1000 }, function (retryCallback) {
+        shell.spawn(`update(${name})`, '/usr/bin/docker', args, { }, retryCallback);
+    }, callback);
+}

src/domaindb.js

@@ -16,14 +16,18 @@ var assert = require('assert'),
     database = require('./database.js'),
     safe = require('safetydance');
 
-var DOMAINS_FIELDS = [ 'domain', 'zoneName', 'provider', 'configJson', 'tlsConfigJson' ].join(',');
+var DOMAINS_FIELDS = [ 'domain', 'zoneName', 'provider', 'configJson', 'tlsConfigJson', 'wellKnownJson' ].join(',');
 
 function postProcess(data) {
     data.config = safe.JSON.parse(data.configJson);
-    data.tlsConfig = safe.JSON.parse(data.tlsConfigJson);
     delete data.configJson;
+
+    data.tlsConfig = safe.JSON.parse(data.tlsConfigJson);
     delete data.tlsConfigJson;
 
+    data.wellKnown = safe.JSON.parse(data.wellKnownJson);
+    delete data.wellKnownJson;
+
     return data;
 }
 
@@ -86,6 +90,9 @@ function update(name, domain, callback) {
         } else if (k === 'tlsConfig') {
             fields.push('tlsConfigJson = ?');
             args.push(JSON.stringify(domain[k]));
+        } else if (k === 'wellKnown') {
+            fields.push('wellKnownJson = ?');
+            args.push(JSON.stringify(domain[k]));
         } else {
             fields.push(k + ' = ?');
             args.push(domain[k]);

src/domains.js

@@ -1,38 +1,35 @@
 'use strict';
 
 module.exports = exports = {
-    add: add,
-    get: get,
-    getAll: getAll,
-    update: update,
-    del: del,
-    clear: clear,
+    add,
+    get,
+    getAll,
+    update,
+    del,
+    clear,
 
-    fqdn: fqdn,
-    getName: getName,
+    fqdn,
+    getName,
 
-    getDnsRecords: getDnsRecords,
-    upsertDnsRecords: upsertDnsRecords,
-    removeDnsRecords: removeDnsRecords,
+    getDnsRecords,
+    upsertDnsRecords,
+    removeDnsRecords,
 
-    waitForDnsRecord: waitForDnsRecord,
+    waitForDnsRecord,
 
-    removePrivateFields: removePrivateFields,
-    removeRestrictedFields: removeRestrictedFields,
+    removePrivateFields,
+    removeRestrictedFields,
 
-    validateHostname: validateHostname,
+    validateHostname,
 
-    makeWildcard: makeWildcard,
+    makeWildcard,
 
-    parentDomain: parentDomain,
+    parentDomain,
 
-    checkDnsRecords: checkDnsRecords,
-
-    prepareDashboardDomain: prepareDashboardDomain
+    checkDnsRecords
 };
 
 var assert = require('assert'),
-    async = require('async'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:domains'),
@@ -63,6 +60,7 @@ function api(provider) {
     case 'linode': return require('./dns/linode.js');
     case 'namecom': return require('./dns/namecom.js');
     case 'namecheap': return require('./dns/namecheap.js');
+    case 'netcup': return require('./dns/netcup.js');
     case 'noop': return require('./dns/noop.js');
     case 'manual': return require('./dns/manual.js');
     case 'wildcard': return require('./dns/wildcard.js');
@@ -92,14 +90,12 @@ function verifyDnsConfig(dnsConfig, domain, zoneName, provider, callback) {
         if (error && error.reason === BoxError.EXTERNAL_ERROR) return callback(new BoxError(BoxError.BAD_FIELD, `Configuration error: ${error.message}`));
         if (error) return callback(error);
 
-        result.hyphenatedSubdomains = !!dnsConfig.hyphenatedSubdomains;
-
         callback(null, result);
     });
 }
 
 function fqdn(location, domainObject) {
-    return location + (location ? (domainObject.config.hyphenatedSubdomains ? '-' : '.') : '') + domainObject.domain;
+    return location + (location ? '.' : '') + domainObject.domain;
 }
 
 // Hostname validation comes from RFC 1123 (section 2.1)
@@ -133,10 +129,6 @@ function validateHostname(location, domainObject) {
         if (/^[-.]/.test(location)) return new BoxError(BoxError.BAD_FIELD, 'Subdomain cannot start or end with hyphen or dot', { field: 'location' });
     }
 
-    if (domainObject.config.hyphenatedSubdomains) {
-        if (location.indexOf('.') !== -1) return new BoxError(BoxError.BAD_FIELD, 'Subdomain cannot contain a dot', { field: 'location' });
-    }
-
     return null;
 }
 
@@ -161,6 +153,12 @@ function validateTlsConfig(tlsConfig, dnsProvider) {
     return null;
 }
 
+function validateWellKnown(wellKnown) {
+    assert.strictEqual(typeof wellKnown, 'object');
+
+    return null;
+}
+
 function add(domain, data, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof data.zoneName, 'string');
@@ -187,7 +185,7 @@ function add(domain, data, auditSource, callback) {
         if (error) return callback(error);
     } else {
         fallbackCertificate = reverseProxy.generateFallbackCertificateSync({ domain, config });
-        if (fallbackCertificate.error) return callback(error);
+        if (fallbackCertificate.error) return callback(fallbackCertificate.error);
     }
 
     let error = validateTlsConfig(tlsConfig, provider);
@@ -255,7 +253,7 @@ function update(domain, data, auditSource, callback) {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let { zoneName, provider, config, fallbackCertificate, tlsConfig } = data;
+    let { zoneName, provider, config, fallbackCertificate, tlsConfig, wellKnown } = data;
 
     if (settings.isDemo() && (domain === settings.adminDomain())) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
 
@@ -276,6 +274,9 @@ function update(domain, data, auditSource, callback) {
         error = validateTlsConfig(tlsConfig, provider);
         if (error) return callback(error);
 
+        error = validateWellKnown(wellKnown, provider);
+        if (error) return callback(error);
+
         if (provider === domainObject.provider) api(provider).injectPrivateFields(config, domainObject.config);
 
         verifyDnsConfig(config, domain, zoneName, provider, function (error, sanitizedConfig) {
@@ -283,9 +284,10 @@ function update(domain, data, auditSource, callback) {
 
             let newData = {
                 config: sanitizedConfig,
-                zoneName: zoneName,
-                provider: provider,
-                tlsConfig: tlsConfig
+                zoneName,
+                provider,
+                tlsConfig,
+                wellKnown
             };
 
             domaindb.update(domain, newData, function (error) {
@@ -311,6 +313,7 @@ function del(domain, auditSource, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     if (domain === settings.adminDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove admin domain'));
+    if (domain === settings.mailDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove mail domain'));
 
     domaindb.del(domain, function (error) {
         if (error) return callback(error);
@@ -339,19 +342,7 @@ function getName(domain, location, type) {
 
     if (location === '') return part;
 
-    if (!domain.config.hyphenatedSubdomains) return part ? `${location}.${part}` : location;
-
-    // hyphenatedSubdomains
-    if (type !== 'TXT') return `${location}-${part}`;
-
-    if (location.startsWith('_acme-challenge.')) {
-        return `${location}-${part}`;
-    } else if (location === '_acme-challenge') {
-        const up = part.replace(/^[^.]*\.?/, ''); // this gets the domain one level up
-        return up ? `${location}.${up}` : location;
-    } else {
-        return `${location}.${part}`;
-    }
+    return part ? `${location}.${part}` : location;
 }
 
 function getDnsRecords(location, domain, type, callback) {
@@ -451,7 +442,7 @@ function waitForDnsRecord(location, domain, type, value, options, callback) {
 
 // removes all fields that are strictly private and should never be returned by API calls
 function removePrivateFields(domain) {
-    var result = _.pick(domain, 'domain', 'zoneName', 'provider', 'config', 'tlsConfig', 'fallbackCertificate');
+    var result = _.pick(domain, 'domain', 'zoneName', 'provider', 'config', 'tlsConfig', 'fallbackCertificate', 'wellKnown');
     return api(result.provider).removePrivateFields(result);
 }
 
@@ -459,46 +450,16 @@ function removePrivateFields(domain) {
 function removeRestrictedFields(domain) {
     var result = _.pick(domain, 'domain', 'zoneName', 'provider');
 
-    // always ensure config object
-    result.config = { hyphenatedSubdomains: !!domain.config.hyphenatedSubdomains };
+    result.config = {}; // always ensure config object
 
     return result;
 }
 
-function makeWildcard(hostname) {
-    assert.strictEqual(typeof hostname, 'string');
+function makeWildcard(vhost) {
+    assert.strictEqual(typeof vhost, 'string');
 
-    let parts = hostname.split('.');
+    // if the vhost is like *.example.com, this function will do nothing
+    let parts = vhost.split('.');
     parts[0] = '*';
     return parts.join('.');
 }
-
-function prepareDashboardDomain(domain, auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof callback, 'function');
-
-    get(domain, function (error, domainObject) {
-        if (error) return callback(error);
-
-        const adminFqdn = fqdn(constants.ADMIN_LOCATION, domainObject);
-
-        sysinfo.getServerIp(function (error, ip) {
-            if (error) return callback(error);
-
-            async.series([
-                (done) => { progressCallback({ percent: 10, message: `Updating DNS of ${adminFqdn}` }); done(); },
-                upsertDnsRecords.bind(null, constants.ADMIN_LOCATION, domain, 'A', [ ip ]),
-                (done) => { progressCallback({ percent: 40, message: `Waiting for DNS of ${adminFqdn}` }); done(); },
-                waitForDnsRecord.bind(null, constants.ADMIN_LOCATION, domain, 'A', ip, { interval: 30000, times: 50000 }),
-                (done) => { progressCallback({ percent: 70, message: `Getting certificate of ${adminFqdn}` }); done(); },
-                reverseProxy.ensureCertificate.bind(null, fqdn(constants.ADMIN_LOCATION, domainObject), domain, auditSource)
-            ], function (error) {
-                if (error) return callback(error);
-
-                callback(null);
-            });
-        });
-    });
-}

src/eventlog.js

@@ -1,11 +1,12 @@
 'use strict';
 
 exports = module.exports = {
-    add: add,
-    get: get,
-    getAllPaged: getAllPaged,
-    getByCreationTime: getByCreationTime,
-    cleanup: cleanup,
+    add,
+    upsert,
+    get,
+    getAllPaged,
+    getByCreationTime,
+    cleanup,
 
     // keep in sync with webadmin index.js filter
     ACTION_ACTIVATE: 'cloudron.activate',
@@ -39,6 +40,7 @@ exports = module.exports = {
     ACTION_DOMAIN_UPDATE: 'domain.update',
     ACTION_DOMAIN_REMOVE: 'domain.remove',
 
+    ACTION_MAIL_LOCATION: 'mail.location',
     ACTION_MAIL_ENABLED: 'mail.enabled',
     ACTION_MAIL_DISABLED: 'mail.disabled',
     ACTION_MAIL_MAILBOX_ADD: 'mail.box.add',
@@ -56,10 +58,15 @@ exports = module.exports = {
 
     ACTION_USER_ADD: 'user.add',
     ACTION_USER_LOGIN: 'user.login',
+    ACTION_USER_LOGOUT: 'user.logout',
     ACTION_USER_REMOVE: 'user.remove',
     ACTION_USER_UPDATE: 'user.update',
     ACTION_USER_TRANSFER: 'user.transfer',
 
+    ACTION_VOLUME_ADD: 'volume.add',
+    ACTION_VOLUME_UPDATE: 'volume.update',
+    ACTION_VOLUME_REMOVE: 'volume.remove',
+
     ACTION_DYNDNS_UPDATE: 'dyndns.update',
 
     ACTION_SUPPORT_TICKET: 'support.ticket',
@@ -85,9 +92,24 @@ function add(action, source, data, callback) {
 
     callback = callback || NOOP_CALLBACK;
 
-    // we do only daily upserts for login actions, so they don't spam the db
-    var api = action === exports.ACTION_USER_LOGIN ? eventlogdb.upsert : eventlogdb.add;
-    api(uuid.v4(), action, source, data, function (error, id) {
+    eventlogdb.add(uuid.v4(), action, source, data, function (error, id) {
+        if (error) return callback(error);
+
+        callback(null, { id: id });
+
+        notifications.onEvent(id, action, source, data, NOOP_CALLBACK);
+    });
+}
+
+function upsert(action, source, data, callback) {
+    assert.strictEqual(typeof action, 'string');
+    assert.strictEqual(typeof source, 'object');
+    assert.strictEqual(typeof data, 'object');
+    assert(!callback || typeof callback === 'function');
+
+    callback = callback || NOOP_CALLBACK;
+
+    eventlogdb.upsert(uuid.v4(), action, source, data, function (error, id) {
         if (error) return callback(error);
 
         callback(null, { id: id });

src/graphite.js

@@ -1,23 +1,27 @@
 'use strict';
 
 exports = module.exports = {
-    startGraphite: startGraphite
+    start,
+
+    DEFAULT_MEMORY_LIMIT: 256 * 1024 * 1024
 };
 
 var assert = require('assert'),
     async = require('async'),
     infra = require('./infra_version.js'),
     paths = require('./paths.js'),
-    shell = require('./shell.js');
+    shell = require('./shell.js'),
+    system = require('./system.js');
 
-function startGraphite(existingInfra, callback) {
+function start(existingInfra, serviceConfig, callback) {
     assert.strictEqual(typeof existingInfra, 'object');
+    assert.strictEqual(typeof serviceConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     const tag = infra.images.graphite.tag;
     const dataDir = paths.PLATFORM_DATA_DIR;
-
-    if (existingInfra.version === infra.version && infra.images.graphite.tag === existingInfra.images.graphite.tag) return callback();
+    const memoryLimit = serviceConfig.memoryLimit || exports.DEFAULT_MEMORY_LIMIT;
+    const memory = system.getMemoryAllocation(memoryLimit);
 
     const cmd = `docker run --restart=always -d --name="graphite" \
                 --hostname graphite \
@@ -27,8 +31,8 @@ function startGraphite(existingInfra, callback) {
                 --log-opt syslog-address=udp://127.0.0.1:2514 \
                 --log-opt syslog-format=rfc5424 \
                 --log-opt tag=graphite \
-                -m 150m \
-                --memory-swap 150m \
+                -m ${memory} \
+                --memory-swap ${memoryLimit} \
                 --dns 172.18.0.1 \
                 --dns-search=. \
                 -p 127.0.0.1:2003:2003 \

src/groupdb.js

@@ -196,6 +196,7 @@ function setMembers(groupId, userIds, callback) {
 
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, 'Group not found'));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.CONFLICT, 'Duplicate member in list'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(error);
@@ -227,6 +228,7 @@ function setMembership(userId, groupIds, callback) {
 
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, error.message));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.CONFLICT, 'Already member'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(null);

src/groups.js

@@ -1,25 +1,25 @@
 'use strict';
 
 exports = module.exports = {
-    create: create,
-    remove: remove,
-    get: get,
-    getByName: getByName,
-    update: update,
-    getWithMembers: getWithMembers,
-    getAll: getAll,
-    getAllWithMembers: getAllWithMembers,
+    create,
+    remove,
+    get,
+    getByName,
+    update,
+    getWithMembers,
+    getAll,
+    getAllWithMembers,
 
-    getMembers: getMembers,
-    addMember: addMember,
-    setMembers: setMembers,
-    removeMember: removeMember,
-    isMember: isMember,
+    getMembers,
+    addMember,
+    setMembers,
+    removeMember,
+    isMember,
 
-    setMembership: setMembership,
-    getMembership: getMembership,
+    setMembership,
+    getMembership,
 
-    count: count
+    count
 };
 
 var assert = require('assert'),

src/infra_version.js

@@ -6,7 +6,7 @@
 
 exports = module.exports = {
     // a version change recreates all containers with latest docker config
-    'version': '48.17.0',
+    'version': '48.18.0',
 
     'baseImages': [
         { repo: 'cloudron/base', tag: 'cloudron/base:2.0.0@sha256:f9fea80513aa7c92fe2e7bf3978b54c8ac5222f47a9a32a7f8833edf0eb5a4f4' }
@@ -15,13 +15,13 @@ exports = module.exports = {
     // a major version bump in the db containers will trigger the restore logic that uses the db dumps
     // docker inspect --format='{{index .RepoDigests 0}}' $IMAGE to get the sha256
     'images': {
-        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.1.0@sha256:e1dd22aa6eef5beb7339834b200a8bb787ffc2264ce11139857a054108fefb4f' },
-        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.3.1@sha256:c1145d43c8a912fe6f5a5629a4052454a4aa6f23391c1efbffeec9d12d72a256' },
-        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:3.0.0@sha256:b00e5118a8f829c422234117bf113803be79a1d5102c51497c6d3005b041ce37' },
+        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.2.0@sha256:4359aae80050a92bae3be30600fb93ef4dbaec6dc9254bda353c0b131a36f969' },
+        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.3.2@sha256:dd624870c7f8ba9b2759f93ce740d1e092a1ac4b2d6af5007a01b30ad6b316d0' },
+        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:3.3.0@sha256:0daf1be5320c095077392bf21d247b93ceaddca46c866c17259a335c80d2f357' },
         'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:3.0.0@sha256:59e50b1f55e433ffdf6d678f8c658812b4119f631db8325572a52ee40d3bc562' },
         'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.3.0@sha256:0e31ec817e235b1814c04af97b1e7cf0053384aca2569570ce92bef0d95e94d2' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.9.4@sha256:0e169b97a0584a76197d2bbc039d8698bf93f815588b3b43c251bd83dd545465' },
+        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:3.1.1@sha256:e6d7c840719843c3dda26a26bb4bd97b46f4ce0f281634550136971d228be68a' },
         'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.3.0@sha256:b7bc1ca4f4d0603a01369a689129aa273a938ce195fe43d00d42f4f2d5212f50' },
-        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:2.0.2@sha256:cbd604eaa970c99ba5c4c2e7984929668e05de824172f880e8c576b2fb7c976d' }
+        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:3.1.0@sha256:329f03f2789fbb5d19678950177153246f1853ac71230fbc6718f6ca1c1eddb3' }
     }
 };

src/iputils.js

@@ -0,0 +1,35 @@
+'use strict';
+
+exports = module.exports = {
+    ipFromInt,
+    intFromIp
+};
+
+const assert = require('assert');
+
+function intFromIp(address) {
+    assert.strictEqual(typeof address, 'string');
+
+    const parts = address.split('.');
+
+    if (parts.length !== 4) return null;
+
+    return (parseInt(parts[0], 10) << (8*3)) & 0xFF000000 |
+           (parseInt(parts[1], 10) << (8*2)) & 0x00FF0000 |
+           (parseInt(parts[2], 10) << (8*1)) & 0x0000FF00 |
+           (parseInt(parts[3], 10) << (8*0)) & 0x000000FF;
+}
+
+function ipFromInt(input) {
+    assert.strictEqual(typeof input, 'number');
+
+    let output = [];
+
+    for (let i = 3; i >= 0; --i) {
+        const octet = (input >> (i*8)) & 0x000000FF;
+        output.push(octet);
+    }
+
+    return output.join('.');
+}
+

src/ldap.js

@@ -1,11 +1,11 @@
 'use strict';
 
 exports = module.exports = {
-    start: start,
-    stop: stop
+    start,
+    stop
 };
 
-var assert = require('assert'),
+const assert = require('assert'),
     appdb = require('./appdb.js'),
     apps = require('./apps.js'),
     async = require('async'),
@@ -13,11 +13,13 @@ var assert = require('assert'),
     constants = require('./constants.js'),
     debug = require('debug')('box:ldap'),
     eventlog = require('./eventlog.js'),
+    groups = require('./groups.js'),
     ldap = require('ldapjs'),
     mail = require('./mail.js'),
     mailboxdb = require('./mailboxdb.js'),
     path = require('path'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     users = require('./users.js');
 
 var gServer = null;
@@ -132,8 +134,8 @@ function userSearch(req, res, next) {
 
             var dn = ldap.parseDN('cn=' + user.id + ',ou=users,dc=cloudron');
 
-            var groups = [ GROUP_USERS_DN ];
-            if (users.compareRoles(user.role, users.ROLE_ADMIN) >= 0) groups.push(GROUP_ADMINS_DN);
+            var memberof = [ GROUP_USERS_DN ];
+            if (users.compareRoles(user.role, users.ROLE_ADMIN) >= 0) memberof.push(GROUP_ADMINS_DN);
 
             var displayName = user.displayName || user.username || ''; // displayName can be empty and username can be null
             var nameParts = displayName.split(' ');
@@ -154,7 +156,7 @@ function userSearch(req, res, next) {
                     givenName: firstName,
                     username: user.username,
                     samaccountname: user.username,      // to support ActiveDirectory clients
-                    memberof: groups
+                    memberof: memberof
                 }
             };
 
@@ -286,14 +288,14 @@ function mailboxSearch(req, res, next) {
     } else if (req.dn.rdns[0].attrs.domain) { // legacy ldap mailbox search for old sogo
         var domain = req.dn.rdns[0].attrs.domain.value.toLowerCase();
 
-        mailboxdb.listMailboxes(domain, 1, 1000, function (error, result) {
+        mailboxdb.listMailboxes(domain, 1, 1000, function (error, mailboxes) {
 
             if (error) return next(new ldap.OperationsError(error.toString()));
 
             var results = [];
 
             // send mailbox objects
-            result.forEach(function (mailbox) {
+            mailboxes.forEach(function (mailbox) {
                 var dn = ldap.parseDN(`cn=${mailbox.name}@${domain},domain=${domain},ou=mailboxes,dc=cloudron`);
 
                 var obj = {
@@ -327,7 +329,9 @@ function mailboxSearch(req, res, next) {
             async.eachSeries(mailboxes, function (mailbox, callback) {
                 var dn = ldap.parseDN(`cn=${mailbox.name}@${mailbox.domain},ou=mailboxes,dc=cloudron`);
 
-                users.get(mailbox.ownerId, function (error, userObject) {
+                let getFunc = mailbox.ownerType === mail.OWNERTYPE_USER ? users.get : groups.get;
+
+                getFunc(mailbox.ownerId, function (error, ownerObject) {
                     if (error) return callback(); // skip mailboxes with unknown owner
 
                     var obj = {
@@ -335,30 +339,26 @@ function mailboxSearch(req, res, next) {
                         attributes: {
                             objectclass: ['mailbox'],
                             objectcategory: 'mailbox',
-                            displayname: userObject.displayName,
+                            displayname: mailbox.ownerType === mail.OWNERTYPE_USER ? ownerObject.displayName : ownerObject.name,
                             cn: `${mailbox.name}@${mailbox.domain}`,
                             uid: `${mailbox.name}@${mailbox.domain}`,
                             mail: `${mailbox.name}@${mailbox.domain}`
                         }
                     };
 
-                    mailboxdb.getAliasesForName(mailbox.name, mailbox.domain, function (error, aliases) {
-                        if (error) return callback(error);
-
-                        aliases.forEach(function (a, idx) {
-                            obj.attributes['mail' + idx] = `${a.name}@${a.domain}`;
-                        });
-
-                        // ensure all filter values are also lowercase
-                        var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
-                        if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));
-
-                        if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
-                            results.push(obj);
-                        }
-
-                        callback();
+                    mailbox.aliases.forEach(function (a, idx) {
+                        obj.attributes['mail' + idx] = `${a.name}@${a.domain}`;
                     });
+
+                    // ensure all filter values are also lowercase
+                    var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
+                    if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));
+
+                    if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
+                        results.push(obj);
+                    }
+
+                    callback();
                 });
             }, function (error) {
                 if (error) return next(new ldap.OperationsError(error.toString()));
@@ -482,18 +482,42 @@ function authorizeUserForApp(req, res, next) {
     assert.strictEqual(typeof req.user, 'object');
     assert.strictEqual(typeof req.app, 'object');
 
-    apps.hasAccessTo(req.app, req.user, function (error, result) {
+    apps.hasAccessTo(req.app, req.user, function (error, hasAccess) {
         if (error) return next(new ldap.OperationsError(error.toString()));
 
         // we return no such object, to avoid leakage of a users existence
-        if (!result) return next(new ldap.NoSuchObjectError(req.dn.toString()));
+        if (!hasAccess) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-        eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', appId: req.app.id }, { userId: req.user.id, user: users.removePrivateFields(req.user) });
+        eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', appId: req.app.id }, { userId: req.user.id, user: users.removePrivateFields(req.user) });
 
         res.end();
     });
 }
 
+function verifyMailboxPassword(mailbox, password, callback) {
+    assert.strictEqual(typeof mailbox, 'object');
+    assert.strictEqual(typeof password, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (mailbox.ownerType === mail.OWNERTYPE_USER) return users.verify(mailbox.ownerId, password, users.AP_MAIL /* identifier */, callback);
+
+    groups.getMembers(mailbox.ownerId, function (error, userIds) {
+        if (error) return callback(error);
+
+        let verifiedUser = null;
+        async.someSeries(userIds, function iterator(userId, iteratorDone) {
+            users.verify(userId, password, users.AP_MAIL /* identifier */, function (error, result) {
+                if (error) return iteratorDone(null, false);
+                verifiedUser = result;
+                iteratorDone(null, true);
+            });
+        }, function (error, result) {
+            if (!result) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+            callback(null, verifiedUser);
+        });
+    });
+}
+
 function authenticateUserMailbox(req, res, next) {
     debug('user mailbox auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
 
@@ -513,12 +537,12 @@ function authenticateUserMailbox(req, res, next) {
             if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
             if (error) return next(new ldap.OperationsError(error.message));
 
-            users.verify(mailbox.ownerId, req.credentials || '', users.AP_MAIL, function (error, result) {
+            verifyMailboxPassword(mailbox, req.credentials || '', function (error, result) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                 if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
                 if (error) return next(new ldap.OperationsError(error.message));
 
-                eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
+                eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
                 res.end();
             });
         });
@@ -547,6 +571,16 @@ function authenticateSftp(req, res, next) {
     });
 }
 
+function loadSftpConfig(req, res, next) {
+    services.getServiceConfig('sftp', function (error, serviceConfig) {
+        if (error) return next(new ldap.OperationsError(error.toString()));
+
+        req.requireAdmin = serviceConfig.requireAdmin;
+
+        next();
+    });
+}
+
 function userSearchSftp(req, res, next) {
     debug('sftp user search: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);
 
@@ -570,6 +604,8 @@ function userSearchSftp(req, res, next) {
         users.getByUsername(username, function (error, user) {
             if (error) return next(new ldap.OperationsError(error.toString()));
 
+            if (req.requireAdmin && users.compareRoles(user.role, users.ROLE_ADMIN) < 0) return next(new ldap.InsufficientAccessRightsError('Insufficient previleges'));
+
             apps.hasAccessTo(app, user, function (error, hasAccess) {
                 if (error) return next(new ldap.OperationsError(error.toString()));
                 if (!hasAccess) return next(new ldap.InsufficientAccessRightsError('Not authorized'));
@@ -593,16 +629,37 @@ function userSearchSftp(req, res, next) {
     });
 }
 
+function verifyAppMailboxPassword(addonId, username, password, callback) {
+    assert.strictEqual(typeof addonId, 'string');
+    assert.strictEqual(typeof username, 'string');
+    assert.strictEqual(typeof password, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const pattern = addonId === 'sendmail' ? 'MAIL_SMTP' : 'MAIL_IMAP';
+    appdb.getAppIdByAddonConfigValue(addonId, `%${pattern}_PASSWORD`, password, function (error, appId) { // search by password because this is unique for each app
+        if (error) return callback(error);
+
+        appdb.getAddonConfig(appId, addonId, function (error, result) {
+            if (error) return callback(error);
+
+            if (!result.some(r => r.name.endsWith(`${pattern}_USERNAME`) && r.value === username)) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+
+            callback(null);
+        });
+    });
+}
+
 function authenticateMailAddon(req, res, next) {
     debug('mail addon auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
 
     if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-    var email = req.dn.rdns[0].attrs.cn.value.toLowerCase();
-    var parts = email.split('@');
+    const email = req.dn.rdns[0].attrs.cn.value.toLowerCase();
+    const parts = email.split('@');
     if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
     const addonId = req.dn.rdns[1].attrs.ou.value.toLowerCase(); // 'sendmail' or 'recvmail'
+    if (addonId !== 'sendmail' && addonId !== 'recvmail') return next(new ldap.OperationsError('Invalid DN'));
 
     mail.getDomain(parts[1], function (error, domain) {
         if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
@@ -610,26 +667,22 @@ function authenticateMailAddon(req, res, next) {
 
         if (addonId === 'recvmail' && !domain.enabled) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-        let namePattern; // manifest v2 has a CLOUDRON_ prefix for names
-        if (addonId === 'sendmail') namePattern = '%MAIL_SMTP_PASSWORD';
-        else if (addonId === 'recvmail') namePattern = '%MAIL_IMAP_PASSWORD';
-        else return next(new ldap.OperationsError('Invalid DN'));
+        verifyAppMailboxPassword(addonId, email, req.credentials || '', function (error) {
+            if (!error) return res.end(); // validated as app
 
-        // note: with sendmail addon, apps can send mail without a mailbox (unlike users)
-        appdb.getAppIdByAddonConfigValue(addonId, namePattern, req.credentials || '', function (error, appId) {
+            if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
             if (error && error.reason !== BoxError.NOT_FOUND) return next(new ldap.OperationsError(error.message));
-            if (appId) return res.end();
 
             mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                 if (error) return next(new ldap.OperationsError(error.message));
 
-                users.verify(mailbox.ownerId, req.credentials || '', users.AP_MAIL, function (error, result) {
+                verifyMailboxPassword(mailbox, req.credentials || '', function (error, result) {
                     if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                     if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
                     if (error) return next(new ldap.OperationsError(error.message));
 
-                    eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
+                    eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
                     res.end();
                 });
             });
@@ -660,16 +713,16 @@ function start(callback) {
     gServer.bind('ou=users,dc=cloudron', authenticateApp, authenticateUser, authorizeUserForApp);
 
     // http://www.ietf.org/proceedings/43/I-D/draft-srivastava-ldap-mail-00.txt
-    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch); // haraka, dovecot
+    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch); // haraka (address translation), dovecot (LMTP), sogo (mailbox search)
     gServer.bind('ou=mailboxes,dc=cloudron', authenticateUserMailbox); // apps like sogo can use domain=${domain} to authenticate a mailbox
     gServer.search('ou=mailaliases,dc=cloudron', mailAliasSearch); // haraka
     gServer.search('ou=mailinglists,dc=cloudron', mailingListSearch); // haraka
 
-    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailAddon); // dovecot
-    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka
+    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailAddon); // dovecot (IMAP auth)
+    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka (MSA auth)
 
     gServer.bind('ou=sftp,dc=cloudron', authenticateSftp);    // sftp
-    gServer.search('ou=sftp,dc=cloudron', userSearchSftp);
+    gServer.search('ou=sftp,dc=cloudron', loadSftpConfig, userSearchSftp);
 
     gServer.compare('cn=users,ou=groups,dc=cloudron', authenticateApp, groupUsersCompare);
     gServer.compare('cn=admins,ou=groups,dc=cloudron', authenticateApp, groupAdminsCompare);

src/mail.js

@@ -4,6 +4,10 @@ exports = module.exports = {
     getStatus,
     checkConfiguration,
 
+    getLocation,
+    setLocation, // triggers the change task
+    changeLocation, // does the actual changing
+
     getDomains,
 
     getDomain,
@@ -11,7 +15,6 @@ exports = module.exports = {
 
     onDomainAdded,
     onDomainRemoved,
-    onMailFqdnChanged,
 
     removePrivateFields,
 
@@ -23,6 +26,7 @@ exports = module.exports = {
     setCatchAllAddress,
     setMailRelay,
     setMailEnabled,
+    setBanner,
 
     startMail: restartMail,
     restartMail,
@@ -48,13 +52,19 @@ exports = module.exports = {
     removeList,
     resolveList,
 
+    OWNERTYPE_USER: 'user',
+    OWNERTYPE_GROUP: 'group',
+
+    DEFAULT_MEMORY_LIMIT: 512 * 1024 * 1024,
+
     _removeMailboxes: removeMailboxes,
     _readDkimPublicKeySync: readDkimPublicKeySync
 };
 
-var assert = require('assert'),
+const assert = require('assert'),
     async = require('async'),
     BoxError = require('./boxerror.js'),
+    cloudron = require('./cloudron.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:mail'),
     dns = require('./native-dns.js'),
@@ -70,12 +80,16 @@ var assert = require('assert'),
     nodemailer = require('nodemailer'),
     path = require('path'),
     paths = require('./paths.js'),
+    request = require('request'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     smtpTransport = require('nodemailer-smtp-transport'),
     sysinfo = require('./sysinfo.js'),
+    system = require('./system.js'),
+    tasks = require('./tasks.js'),
     users = require('./users.js'),
     validator = require('validator'),
     _ = require('underscore');
@@ -238,14 +252,14 @@ function checkSpf(domain, mailFqdn, callback) {
             let txtRecord = txtRecords[i].join(''); // https://agari.zendesk.com/hc/en-us/articles/202952749-How-long-can-my-SPF-record-be-
             if (txtRecord.indexOf('v=spf1 ') !== 0) continue; // not SPF
             spf.value = txtRecord;
-            spf.status = spf.value.indexOf(' a:' + settings.adminFqdn()) !== -1;
+            spf.status = spf.value.indexOf(' a:' + settings.mailFqdn()) !== -1;
             break;
         }
 
         if (spf.status) {
             spf.expected = spf.value;
         } else if (i !== txtRecords.length) {
-            spf.expected = 'v=spf1 a:' + settings.adminFqdn() + ' ' + spf.value.slice('v=spf1 '.length);
+            spf.expected = 'v=spf1 a:' + settings.mailFqdn() + ' ' + spf.value.slice('v=spf1 '.length);
         }
 
         callback(null, spf);
@@ -544,7 +558,7 @@ function checkConfiguration(callback) {
                 markdownMessage += '\n\n';
             });
 
-            if (markdownMessage) markdownMessage += 'Email Status is checked every 30 minutes.\n See the [troubleshooting docs](https://cloudron.io/documentation/troubleshooting/#mail-dns) for more information.\n';
+            if (markdownMessage) markdownMessage += 'Email Status is checked every 30 minutes.\n See the [troubleshooting docs](https://docs.cloudron.io/troubleshooting/#mail-dns) for more information.\n';
 
             callback(null, markdownMessage); // empty message means all status checks succeeded
         });
@@ -576,15 +590,18 @@ function createMailConfig(mailFqdn, mailDomain, callback) {
         }
 
         // create sections for per-domain configuration
-        mailDomains.forEach(function (domain) {
+        async.eachSeries(mailDomains, function (domain, iteratorDone) {
             const catchAll = domain.catchAll.map(function (c) { return `${c}@${domain.domain}`; }).join(',');
             const mailFromValidation = domain.mailFromValidation;
 
             if (!safe.fs.appendFileSync(path.join(paths.ADDON_CONFIG_DIR, 'mail/mail.ini'),
                 `[${domain.domain}]\ncatch_all=${catchAll}\nmail_from_validation=${mailFromValidation}\n\n`, 'utf8')) {
-                return callback(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
+                return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
             }
 
+            if (!safe.fs.writeFileSync(`${paths.ADDON_CONFIG_DIR}/mail/banner/${domain.domain}.text`, domain.banner.text || '')) return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create text banner file:' + safe.error.message));
+            if (!safe.fs.writeFileSync(`${paths.ADDON_CONFIG_DIR}/mail/banner/${domain.domain}.html`, domain.banner.html || '')) return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create html banner file:' + safe.error.message));
+
             const relay = domain.relay;
 
             const enableRelay = relay.provider !== 'cloudron-smtp' && relay.provider !== 'noop',
@@ -594,21 +611,26 @@ function createMailConfig(mailFqdn, mailDomain, callback) {
                 username = relay.username || '',
                 password = relay.password || '';
 
-            if (!enableRelay) return;
+            if (!enableRelay) return iteratorDone();
 
             if (!safe.fs.appendFileSync(paths.ADDON_CONFIG_DIR + '/mail/smtp_forward.ini',
                 `[${domain.domain}]\nenable_outbound=true\nhost=${host}\nport=${port}\nenable_tls=true\nauth_type=${authType}\nauth_user=${username}\nauth_pass=${password}\n\n`, 'utf8')) {
-                return callback(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
+                return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
             }
-        });
 
-        callback(null, mailInDomains.length !== 0 /* allowInbound */);
+            iteratorDone();
+        }, function (error) {
+            if (error) return callback(error);
+
+            callback(null, mailInDomains.length !== 0 /* allowInbound */);
+        });
     });
 }
 
-function configureMail(mailFqdn, mailDomain, callback) {
+function configureMail(mailFqdn, mailDomain, serviceConfig, callback) {
     assert.strictEqual(typeof mailFqdn, 'string');
     assert.strictEqual(typeof mailDomain, 'string');
+    assert.strictEqual(typeof serviceConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     // mail (note: 2525 is hardcoded in mail container and app use this port)
@@ -617,7 +639,8 @@ function configureMail(mailFqdn, mailDomain, callback) {
     // mail container uses /app/data for backed up data and /run for restart-able data
 
     const tag = infra.images.mail.tag;
-    const memoryLimit = 4 * 256;
+    const memoryLimit = serviceConfig.memoryLimit || exports.DEFAULT_MEMORY_LIMIT;
+    const memory = system.getMemoryAllocation(memoryLimit);
     const cloudronToken = hat(8 * 128), relayToken = hat(8 * 128);
 
     reverseProxy.getCertificate(mailFqdn, mailDomain, function (error, bundle) {
@@ -648,8 +671,8 @@ function configureMail(mailFqdn, mailDomain, callback) {
                             --log-opt syslog-address=udp://127.0.0.1:2514 \
                             --log-opt syslog-format=rfc5424 \
                             --log-opt tag=mail \
-                            -m ${memoryLimit}m \
-                            --memory-swap ${memoryLimit * 2}m \
+                            -m ${memory} \
+                            --memory-swap ${memoryLimit} \
                             --dns 172.18.0.1 \
                             --dns-search=. \
                             -e CLOUDRON_MAIL_TOKEN="${cloudronToken}" \
@@ -696,8 +719,12 @@ function restartMail(callback) {
 
     if (process.env.BOX_ENV === 'test' && !process.env.TEST_CREATE_INFRA) return callback();
 
-    debug(`restartMail: restarting mail container with ${settings.mailFqdn()} ${settings.adminDomain()}`);
-    configureMail(settings.mailFqdn(), settings.adminDomain(), callback);
+    services.getServiceConfig('mail', function (error, serviceConfig) {
+        if (error) return callback(error);
+
+        debug(`restartMail: restarting mail container with ${settings.mailFqdn()} ${settings.adminDomain()}`);
+        configureMail(settings.mailFqdn(), settings.adminDomain(), serviceConfig, callback);
+    });
 }
 
 function restartMailIfActivated(callback) {
@@ -750,7 +777,7 @@ function txtRecordsWithSpf(domain, mailFqdn, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     domains.getDnsRecords('', domain, 'TXT', function (error, txtRecords) {
-        if (error) return error;
+        if (error) return callback(error);
 
         debug('txtRecordsWithSpf: current txt records - %j', txtRecords);
 
@@ -891,21 +918,73 @@ function setDnsRecords(domain, callback) {
     upsertDnsRecords(domain, settings.mailFqdn(), callback);
 }
 
-function onMailFqdnChanged(callback) {
+function getLocation(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    const mailFqdn = settings.mailFqdn(),
-        mailDomain = settings.adminDomain();
+    const domain = settings.mailDomain(), fqdn = settings.mailFqdn();
+    const subdomain = fqdn.substr(0, fqdn.length - domain.length - 1);
 
-    domains.getAll(function (error, allDomains) {
+    callback(null, { domain, subdomain });
+}
+
+function changeLocation(auditSource, progressCallback, callback) {
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    const fqdn = settings.mailFqdn(), domain = settings.mailDomain();
+    const subdomain = fqdn.substr(0, fqdn.length - domain.length - 1);
+
+    let progress = 20;
+    progressCallback({ percent: progress, message: `Setting up DNS of certs of mail server ${fqdn}` });
+
+    cloudron.setupDnsAndCert(subdomain, domain, auditSource, progressCallback, function (error) {
         if (error) return callback(error);
 
-        async.eachOfSeries(allDomains, function (domainObject, idx, iteratorDone) {
-            upsertDnsRecords(domainObject.domain, mailFqdn, iteratorDone);
-        }, function (error) {
+        domains.getAll(function (error, allDomains) {
             if (error) return callback(error);
 
-            configureMail(mailFqdn, mailDomain, callback);
+            async.eachOfSeries(allDomains, function (domainObject, idx, iteratorDone) {
+                progressCallback({ percent: progress, message: `Updating DNS of ${domainObject.domain}` });
+                progress += Math.round(70/allDomains.length);
+
+                upsertDnsRecords(domainObject.domain, fqdn, function (error) { // ignore any errors. we anyway report dns errors in status tab
+                    progressCallback({ percent: progress, message: `Updated DNS of ${domainObject.domain}: ${error ? error.message : 'success'}` });
+                    iteratorDone();
+                });
+            }, function (error) {
+                if (error) return callback(error);
+
+                progressCallback({ percent: 90, message: 'Restarting mail server' });
+
+                restartMailIfActivated(callback);
+            });
+        });
+    });
+}
+
+function setLocation(subdomain, domain, auditSource, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    domains.get(domain, function (error, domainObject) {
+        if (error) return callback(error);
+
+        const fqdn = domains.fqdn(subdomain, domainObject);
+
+        settings.setMailLocation(domain, fqdn, function (error) {
+            if (error) return callback(error);
+
+            tasks.add(tasks.TASK_CHANGE_MAIL_LOCATION, [ auditSource ], function (error, taskId) {
+                if (error) return callback(error);
+
+                tasks.startTask(taskId, {}, NOOP_CALLBACK);
+                eventlog.add(eventlog.ACTION_MAIL_LOCATION, auditSource, { subdomain, domain, taskId });
+
+                callback(null, taskId);
+            });
         });
     });
 }
@@ -914,6 +993,8 @@ function onDomainAdded(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
+    if (!settings.mailFqdn()) return callback(); // mail domain is not set yet (when provisioning)
+
     async.series([
         upsertDnsRecords.bind(null, domain, settings.mailFqdn()), // do this first to ensure DKIM keys
         restartMailIfActivated
@@ -939,7 +1020,7 @@ function clearDomains(callback) {
 
 // remove all fields that should never be sent out via REST API
 function removePrivateFields(domain) {
-    let result = _.pick(domain, 'domain', 'enabled', 'mailFromValidation', 'catchAll', 'relay');
+    let result = _.pick(domain, 'domain', 'enabled', 'mailFromValidation', 'catchAll', 'relay', 'banner');
     if (result.relay.provider !== 'cloudron-smtp') {
         if (result.relay.username === result.relay.password) result.relay.username = constants.SECRET_PLACEHOLDER;
         result.relay.password = constants.SECRET_PLACEHOLDER;
@@ -961,6 +1042,20 @@ function setMailFromValidation(domain, enabled, callback) {
     });
 }
 
+function setBanner(domain, banner, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof banner, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    maildb.update(domain, { banner }, function (error) {
+        if (error) return callback(error);
+
+        restartMail(NOOP_CALLBACK);
+
+        callback(null);
+    });
+}
+
 function setCatchAllAddress(domain, addresses, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(addresses));
@@ -1084,10 +1179,11 @@ function getMailbox(name, domain, callback) {
     });
 }
 
-function addMailbox(name, domain, userId, auditSource, callback) {
+function addMailbox(name, domain, ownerId, ownerType, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1096,31 +1192,53 @@ function addMailbox(name, domain, userId, auditSource, callback) {
     var error = validateName(name);
     if (error) return callback(error);
 
-    mailboxdb.addMailbox(name, domain, userId, function (error) {
+    if (ownerType !== exports.OWNERTYPE_USER && ownerType !== exports.OWNERTYPE_GROUP) return callback(new BoxError(BoxError.BAD_FIELD, 'bad owner type'));
+
+    mailboxdb.addMailbox(name, domain, ownerId, ownerType, function (error) {
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_ADD, auditSource, { name, domain, userId });
+        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_ADD, auditSource, { name, domain, ownerId, ownerType });
 
         callback(null);
     });
 }
 
-function updateMailboxOwner(name, domain, userId, auditSource, callback) {
+function updateMailboxOwner(name, domain, ownerId, ownerType, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     name = name.toLowerCase();
 
+    if (ownerType !== exports.OWNERTYPE_USER && ownerType !== exports.OWNERTYPE_GROUP) return callback(new BoxError(BoxError.BAD_FIELD, 'bad owner type'));
+
     getMailbox(name, domain, function (error, result) {
         if (error) return callback(error);
 
-        mailboxdb.updateMailboxOwner(name, domain, userId, function (error) {
+        mailboxdb.updateMailboxOwner(name, domain, ownerId, ownerType, function (error) {
             if (error) return callback(error);
 
-            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldUserId: result.userId, userId });
+            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldUserId: result.userId, ownerId, ownerType });
+
+            callback(null);
+        });
+    });
+}
+
+function removeSolrIndex(mailbox, callback) {
+    assert.strictEqual(typeof mailbox, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    services.getContainerDetails('mail', 'CLOUDRON_MAIL_TOKEN', function (error, addonDetails) {
+        if (error) return callback(error);
+
+        request.post(`https://${addonDetails.ip}:3000/solr_delete_index?access_token=${addonDetails.token}`, { timeout: 2000, rejectUnauthorized: false, json: { mailbox } }, function (error, response) {
+            if (error) return callback(error);
+
+            if (response.statusCode !== 200) return callback(new Error(`Error removing solr index - ${response.statusCode} ${JSON.stringify(response.body)}`));
 
             callback(null);
         });
@@ -1134,7 +1252,8 @@ function removeMailbox(name, domain, options, auditSource, callback) {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    const deleteMailFunc = options.deleteMails ? shell.sudo.bind(null, 'removeMailbox', [ REMOVE_MAILBOX, `${name}@${domain}` ], {}) : (next) => next();
+    const mailbox =`${name}@${domain}`;
+    const deleteMailFunc = options.deleteMails ? shell.sudo.bind(null, 'removeMailbox', [ REMOVE_MAILBOX, mailbox ], {}) : (next) => next();
 
     deleteMailFunc(function (error) {
         if (error) return callback(new BoxError(BoxError.FS_ERROR, `Error removing mailbox: ${error.message}`));
@@ -1142,6 +1261,7 @@ function removeMailbox(name, domain, options, auditSource, callback) {
         mailboxdb.del(name, domain, function (error) {
             if (error) return callback(error);
 
+            removeSolrIndex(mailbox, NOOP_CALLBACK);
             eventlog.add(eventlog.ACTION_MAIL_MAILBOX_REMOVE, auditSource, { name, domain });
 
             callback();

src/mail_templates/app_down.ejs

@@ -7,8 +7,8 @@ The application '<%= title %>' installed at <%= appFqdn %> is not responding.
 This is most likely a problem in the application.
 
 To resolve this, you can try the following:
-* Restart the app by opening the app's web terminal - https://cloudron.io/documentation/apps/#web-terminal
-* Restore the app to the latest backup - https://cloudron.io/documentation/backups/#restoring-an-app
+* Restart the app by opening the app's web terminal - https://docs.cloudron.io/apps/#web-terminal
+* Restore the app to the latest backup - https://docs.cloudron.io/backups/#restoring-an-app
 * Contact us via <%= supportEmail %> or https://forum.cloudron.io
 
 

src/mail_templates/app_updates_available.ejs

@@ -3,18 +3,14 @@
 Dear Cloudron Admin,
 
 <% for (var i = 0; i < apps.length; i++) { -%>
-A new version <%= apps[i].updateInfo.manifest.version %> of the app '<%= apps[i].app.manifest.title %>' installed at <%= apps[i].app.fqdn %> is available.
+The app '<%= apps[i].app.manifest.title %>' installed at <%= apps[i].app.fqdn %> has an update available.
 
-Changes:
+<%= apps[i].app.manifest.title %> v<%= apps[i].updateInfo.manifest.version %> changes:
 <%= apps[i].updateInfo.manifest.changelog %>
 
 <% } -%>
 
-<% if (!hasSubscription) { -%>
-*Keep your Cloudron automatically up-to-date and secure by upgrading to a paid plan at* <%= webadminUrl %>/#/settings
-<% } else { -%>
 Update now at <%= webadminUrl %>
-<% } -%>
 
 Powered by https://cloudron.io
 
@@ -33,24 +29,20 @@ Sent at: <%= new Date().toUTCString() %>
     <div style="width: 650px; text-align: left;">
 <% for (var i = 0; i < apps.length; i++) { -%>
         <p>
-            A new version <%= apps[i].updateInfo.manifest.version %> of the app '<%= apps[i].app.manifest.title %>' installed at <a href="https://<%= apps[i].app.fqdn %>"><%= apps[i].app.fqdn %></a> is available.
+            The app '<%= apps[i].app.manifest.title %>' installed at <a href="https://<%= apps[i].app.fqdn %>"><%= apps[i].app.fqdn %></a> has an update available.
         </p>
 
-        <h5>Changelog:</h5>
+        <h5><%= apps[i].app.manifest.title %> v<%= apps[i].updateInfo.manifest.version %> changes:</h5>
         <%- apps[i].changelogHTML %>
 
         <br/>
 <% } -%>
 
-<% if (!hasSubscription) { -%>
-        <p>Keep your Cloudron automatically up-to-date and secure by upgrading to a <a href="<%= webadminUrl %>/#/settings">paid plan</a>.</p>
-<% } else { -%>
         <p>
             <br/>
             <center><a href="<%= webadminUrl %>">Update now</a></center>
             <br/>
         </p>
-<% } -%>
     </div>
 
     <div style="font-size: 10px; color: #333333; background: #ffffff;">

src/mail_templates/backup_failed.ejs

@@ -2,7 +2,7 @@
 
 Dear <%= cloudronName %> Admin,
 
-Cloudron failed to create a complete backup. Please see https://cloudron.io/documentation/troubleshooting/#backups
+Cloudron failed to create a complete backup. Please see https://docs.cloudron.io/troubleshooting/#backups
 for troubleshooting.
 
 Logs for this failure are available at <%= logUrl %>

src/mail_templates/box_update_available.ejs

@@ -0,0 +1,45 @@
+<%if (format === 'text') { %>
+
+Dear <%= cloudronName %> Admin,
+
+Cloudron v<%= newBoxVersion %> is now available!
+
+Changes:
+<% for (var i = 0; i < changelog.length; i++) { %>
+  * <%- changelog[i] %>
+<% } %>
+
+Powered by https://cloudron.io
+
+Sent at: <%= new Date().toUTCString() %>
+
+<% } else { %>
+
+<center>
+
+<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
+
+<h3>Dear <%= cloudronName %> Admin,</h3>
+
+<div style="width: 650px; text-align: left;">
+    <p>
+        Cloudron v<%= newBoxVersion %> is now available!
+    </p>
+
+    <h5>Changes:</h5>
+    <ul>
+    <% for (var i = 0; i < changelogHTML.length; i++) { %>
+        <li><%- changelogHTML[i] %></li>
+    <% } %>
+    </ul>
+
+    <br/>
+</div>
+
+<div style="font-size: 10px; color: #333333; background: #ffffff;">
+    Powered by <a href="https://cloudron.io">Cloudron</a>.
+</div>
+
+</center>
+
+<% } %>

src/mail_templates/certificate_renewal_error.ejs

@@ -8,7 +8,7 @@ The Cloudron will attempt to renew the certificate every 12 hours
 until the certificate expires (at which point it will switch to
 using the fallback certificate).
 
-See https://cloudron.io/documentation/troubleshooting/#certificates to
+See https://docs.cloudron.io/troubleshooting/#certificates to
 double check if your server is configured correctly to obtain certificates
 via Let's Encrypt.
 

src/mail_templates/oom_event.ejs

@@ -2,12 +2,13 @@
 
 Dear <%= cloudronName %> Admin,
 
-<%= program %> was restarted now as it ran out of memory.
-
-If this message appears repeatedly, give the app more memory.
-
-* To increase an app's memory limit - https://cloudron.io/documentation/apps/#memory-limit
-* To increase a service's memory limit - https://cloudron.io/documentation/troubleshooting/#services
+<%if (app) { %>
+The application at <%= app.fqdn %> ran out of memory. The application has been restarted automatically. If you see this notification often,
+consider increasing the memory limit - <%= webadminUrl %>/#/app/<%= app.id %>/resources .
+<% } else { %>
+The addon <%= addon.name %> service ran out of memory. The service has been restarted automatically. If you see this notification often,
+consider increasing the memory limit - <%= webadminUrl %>/#/services .
+<% } %>
 
 Out of memory event:
 

src/mail_templates/password_reset-html.ejs

@@ -0,0 +1,24 @@
+<center>
+
+<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
+
+<h3>{{ passwordResetEmail.salutation }}</h3>
+
+<p>{{ passwordResetEmail.description }}</p>
+
+<p>
+  <a href="<%= resetLink %>">{{ passwordResetEmail.resetAction }}</a>
+</p>
+
+<br/>
+
+{{ passwordResetEmail.expireNote }}
+
+<br/>
+<br/>
+
+<div style="font-size: 10px; color: #333333; background: #ffffff;">
+  Powered by <a href="https://cloudron.io">Cloudron</a>
+</div>
+
+</center>

src/mail_templates/password_reset-text.ejs

@@ -0,0 +1,9 @@
+{{ passwordResetEmail.salutation }}
+
+{{ passwordResetEmail.description }}
+
+{{ passwordResetEmail.resetActionText }}
+
+{{ passwordResetEmail.expireNote }}
+
+Powered by https://cloudron.io

src/mail_templates/password_reset.ejs

@@ -1,45 +0,0 @@
-<%if (format === 'text') { %>
-
-Hi <%= user.displayName || user.username || user.email %>,
-
-Someone, hopefully you, has requested your account's password
-be reset. If you did not request this reset, please ignore this message.
-
-To reset your password, please visit the following page:
-<%- resetLink %>
-
-Please note that the password reset link will expire in 24 hours.
-
-Powered by https://cloudron.io
-
-<% } else { %>
-
-<center>
-
-<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
-
-<h3>Hi <%= user.displayName || user.username || user.email %>,</h3>
-
-<p>
-    Someone, hopefully you, has requested your account's password be reset.<br/>
-    If you did not request this reset, please ignore this message.
-</p>
-
-<p>
-    <a href="<%= resetLink %>">Click to reset your password</a>
-</p>
-
-<br/>
-
-Please note that the password reset link will expire in 24 hours.
-
-<br/>
-<br/>
-
-<div style="font-size: 10px; color: #333333; background: #ffffff;">
-    Powered by <a href="https://cloudron.io">Cloudron</a>
-</div>
-
-</center>
-
-<% } %>

src/mail_templates/user_added.ejs

@@ -1,30 +0,0 @@
-<%if (format === 'text') { %>
-
-Dear <%= cloudronName %> Admin,
-
-A new user with email <%= user.email %> was added to <%= cloudronName %>.
-
-Powered by https://cloudron.io
-
-<% } else { %>
-
-<center>
-
-<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
-
-<h3>Dear <%= cloudronName %> Admin,</h3>
-
-<p>
-    A new user with email <%= user.email %> was added to <%= cloudronName %>.
-</p>
-
-<br/>
-<br/>
-
-<div style="font-size: 10px; color: #333333; background: #ffffff;">
-    Powered by <a href="https://cloudron.io">Cloudron</a>.
-</div>
-
-</center>
-
-<% } %>

src/mail_templates/user_event.ejs

@@ -1,14 +0,0 @@
-<%if (format === 'text') { %>
-
-Dear Cloudron Admin,
-
-User <%= user.username || user.email %> <%= event %>.
-
-
-Powered by https://cloudron.io
-
-Sent at: <%= new Date().toUTCString() %>
-
-<% } else { %>
-
-<% } %>

src/mail_templates/welcome_user-html.ejs

@@ -0,0 +1,28 @@
+<center>
+
+<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
+
+<h3>{{ welcomeEmail.salutation }}</h3>
+<h2>{{ welcomeEmail.welcomeTo }}</h2>
+
+<p>
+  <a href="<%= inviteLink %>">{{ welcomeEmail.inviteLinkAction }}</a>
+</p>
+
+<br/>
+<br/>
+
+<div style="font-size: 10px; color: #333333; background: #ffffff;">
+<% if (invitor) { -%>
+  {{ welcomeEmail.invitor }}
+<% } -%>
+
+  <br/>
+
+  {{ welcomeEmail.expireNote }}
+  <br/>
+
+  Powered by <a href="https://cloudron.io">Cloudron</a>
+</div>
+
+</center>

src/mail_templates/welcome_user-text.ejs

@@ -0,0 +1,13 @@
+{{ welcomeEmail.salutation }}
+
+{{ welcomeEmail.welcomeTo }}
+
+{{ welcomeEmail.inviteLinkActionText }}
+
+<% if (invitor) { %>
+{{ welcomeEmail.invitor }}
+<% } %>
+
+{{ welcomeEmail.expireNote }}
+
+Powered by https://cloudron.io

src/mail_templates/welcome_user.ejs

@@ -1,50 +0,0 @@
-<%if (format === 'text') { %>
-
-Dear <%= user.displayName || user.username || user.email %>,
-
-Welcome to <%= cloudronName %>!
-
-Follow the link to get started.
-<%- inviteLink %>
-
-<% if (invitor && invitor.email) { %>
-You are receiving this email because you were invited by <%= invitor.email %>.
-<% } %>
-
-Please note that the invite link will expire in 7 days.
-
-Powered by https://cloudron.io
-
-<% } else { %>
-
-<center>
-
-<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
-
-<h3>Hi <%= user.displayName || user.username || user.email %>,</h3>
-
-<h2>Welcome to <%= cloudronName %>!</h2>
-
-<p>
-    <a href="<%= inviteLink %>">Get started</a>.
-</p>
-
-<br/>
-<br/>
-
-<div style="font-size: 10px; color: #333333; background: #ffffff;">
-    <% if (invitor && invitor.email) { %>
-        You are receiving this email because you were invited by <%= invitor.email %>.
-    <% } %>
-
-    <br/>
-
-        Please note that the invite link will expire in 7 days.
-    <br/>
-
-    Powered by <a href="https://cloudron.io">Cloudron</a>
-</div>
-
-</center>
-
-<% } %>

src/mailboxdb.js

@@ -42,7 +42,7 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'aliasName', 'aliasDomain', 'creationTime', 'membersJson', 'membersOnly', 'domain' ].join(',');
+var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'ownerType', 'aliasName', 'aliasDomain', 'creationTime', 'membersJson', 'membersOnly', 'domain' ].join(',');
 
 function postProcess(data) {
     data.members = safe.JSON.parse(data.membersJson) || [ ];
@@ -53,13 +53,24 @@ function postProcess(data) {
     return data;
 }
 
-function addMailbox(name, domain, ownerId, callback) {
+function postProcessAliases(data) {
+    const aliasNames = JSON.parse(data.aliasNames), aliasDomains = JSON.parse(data.aliasDomains);
+    delete data.aliasNames;
+    delete data.aliasDomains;
+    data.aliases = [];
+    for (let i = 0; i < aliasNames.length; i++) { // NOTE: aliasNames is [ null ] when no aliases
+        if (aliasNames[i]) data.aliases[i] = { name: aliasNames[i], domain: aliasDomains[i] };
+    }
+}
+
+function addMailbox(name, domain, ownerId, ownerType, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId) VALUES (?, ?, ?, ?)', [ name, exports.TYPE_MAILBOX, domain, ownerId ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, ownerType) VALUES (?, ?, ?, ?, ?)', [ name, exports.TYPE_MAILBOX, domain, ownerId, ownerType ], function (error) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'mailbox already exists'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -67,13 +78,14 @@ function addMailbox(name, domain, ownerId, callback) {
     });
 }
 
-function updateMailboxOwner(name, domain, ownerId, callback) {
+function updateMailboxOwner(name, domain, ownerId, ownerType, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('UPDATE mailboxes SET ownerId = ? WHERE name = ? AND domain = ?', [ ownerId, name, domain ], function (error, result) {
+    database.query('UPDATE mailboxes SET ownerId = ?, ownerType = ? WHERE name = ? AND domain = ?', [ ownerId, ownerType, name, domain ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Mailbox not found'));
 
@@ -88,8 +100,8 @@ function addList(name, domain, members, membersOnly, callback) {
     assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, membersJson, membersOnly) VALUES (?, ?, ?, ?, ?, ?)',
-        [ name, exports.TYPE_LIST, domain, 'admin', JSON.stringify(members), membersOnly ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, ownerType, membersJson, membersOnly) VALUES (?, ?, ?, ?, ?, ?, ?)',
+        [ name, exports.TYPE_LIST, domain, 'admin', 'user', JSON.stringify(members), membersOnly ], function (error) {
             if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'mailbox already exists'));
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -223,14 +235,22 @@ function listMailboxes(domain, search, page, perPage, callback) {
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    let query = `SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? AND domain = ?`;
-    if (search) query += ' AND (name LIKE ' + mysql.escape('%' + search + '%') + ')';
-    query += 'ORDER BY name LIMIT ?,?';
+    const escapedSearch = mysql.escape('%' + search + '%'); // this also quotes the string
+    const searchQuery = search ? ` HAVING (name LIKE ${escapedSearch} OR aliasNames LIKE ${escapedSearch} OR aliasDomains LIKE ${escapedSearch})` : ''; // having instead of where because of aggregated columns use
 
-    database.query(query, [ exports.TYPE_MAILBOX, domain, (page-1)*perPage, perPage ], function (error, results) {
+    const query = 'SELECT m1.name AS name, m1.domain AS domain, m1.ownerId AS ownerId, m1.ownerType as ownerType, JSON_ARRAYAGG(m2.name) AS aliasNames, JSON_ARRAYAGG(m2.domain) AS aliasDomains '
+        + ` FROM (SELECT * FROM mailboxes WHERE type='${exports.TYPE_MAILBOX}') AS m1`
+        + ` LEFT JOIN (SELECT * FROM mailboxes WHERE type='${exports.TYPE_ALIAS}') AS m2`
+        + '     ON m1.name=m2.aliasName AND m1.domain=m2.aliasDomain AND m1.ownerId=m2.ownerId'
+        + ' WHERE m1.domain = ?'
+        + ' GROUP BY m1.name, m1.domain, m1.ownerId'
+        + searchQuery
+        + ' ORDER BY name LIMIT ?,?';
+
+    database.query(query, [ domain, (page-1)*perPage, perPage ], function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-        results.forEach(function (result) { postProcess(result); });
+        results.forEach(postProcessAliases);
 
         callback(null, results);
     });
@@ -241,14 +261,20 @@ function listAllMailboxes(page, perPage, callback) {
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query(`SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? ORDER BY name LIMIT ?,?`,
-        [ exports.TYPE_MAILBOX, (page-1)*perPage, perPage ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+    const query = 'SELECT m1.name AS name, m1.domain AS domain, m1.ownerId AS ownerId, m1.ownerType as ownerType, JSON_ARRAYAGG(m2.name) AS aliasNames, JSON_ARRAYAGG(m2.domain) AS aliasDomains '
+        + ` FROM (SELECT * FROM mailboxes WHERE type='${exports.TYPE_MAILBOX}') AS m1`
+        + ` LEFT JOIN (SELECT * FROM mailboxes WHERE type='${exports.TYPE_ALIAS}') AS m2`
+        + '     ON m1.name=m2.aliasName AND m1.domain=m2.aliasDomain AND m1.ownerId=m2.ownerId'
+        + ' GROUP BY m1.name, m1.domain, m1.ownerId'
+        + ' ORDER BY name LIMIT ?,?';
 
-            results.forEach(function (result) { postProcess(result); });
+    database.query(query, [ (page-1)*perPage, perPage ], function (error, results) {
+        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-            callback(null, results);
-        });
+        results.forEach(postProcessAliases);
+
+        callback(null, results);
+    });
 }
 
 function getLists(domain, search, page, perPage, callback) {
@@ -314,8 +340,8 @@ function setAliasesForName(name, domain, aliases, callback) {
         // clear existing aliases
         queries.push({ query: 'DELETE FROM mailboxes WHERE aliasName = ? AND aliasDomain = ? AND type = ?', args: [ name, domain, exports.TYPE_ALIAS ] });
         aliases.forEach(function (alias) {
-            queries.push({ query: 'INSERT INTO mailboxes (name, domain, type, aliasName, aliasDomain, ownerId) VALUES (?, ?, ?, ?, ?, ?)',
-                args: [ alias.name, alias.domain, exports.TYPE_ALIAS, name, domain, results[0].ownerId ] });
+            queries.push({ query: 'INSERT INTO mailboxes (name, domain, type, aliasName, aliasDomain, ownerId, ownerType) VALUES (?, ?, ?, ?, ?, ?, ?)',
+                args: [ alias.name, alias.domain, exports.TYPE_ALIAS, name, domain, results[0].ownerId, results[0].ownerType ] });
         });
 
         database.transaction(queries, function (error) {

src/maildb.js

@@ -17,7 +17,7 @@ var assert = require('assert'),
     database = require('./database.js'),
     safe = require('safetydance');
 
-var MAILDB_FIELDS = [ 'domain', 'enabled', 'mailFromValidation', 'catchAllJson', 'relayJson', 'dkimSelector' ].join(',');
+var MAILDB_FIELDS = [ 'domain', 'enabled', 'mailFromValidation', 'catchAllJson', 'relayJson', 'dkimSelector', 'bannerJson' ].join(',');
 
 function postProcess(data) {
     data.enabled = !!data.enabled; // int to boolean
@@ -29,6 +29,9 @@ function postProcess(data) {
     data.relay = safe.JSON.parse(data.relayJson) || { provider: 'cloudron-smtp' };
     delete data.relayJson;
 
+    data.banner = safe.JSON.parse(data.bannerJson) || { text: null, html: null };
+    delete data.bannerJson;
+
     return data;
 }
 
@@ -74,8 +77,8 @@ function update(domain, data, callback) {
     var args = [ ];
     var fields = [ ];
     for (var k in data) {
-        if (k === 'catchAll') {
-            fields.push('catchAllJson = ?');
+        if (k === 'catchAll' || k === 'banner') {
+            fields.push(`${k}Json = ?`);
             args.push(JSON.stringify(data[k]));
         } else if (k === 'relay') {
             fields.push('relayJson = ?');

src/mailer.js

@@ -1,25 +1,23 @@
 'use strict';
 
 exports = module.exports = {
-    userAdded: userAdded,
-    userRemoved: userRemoved,
-    roleChanged: roleChanged,
-    passwordReset: passwordReset,
-    appUpdatesAvailable: appUpdatesAvailable,
+    passwordReset,
+    boxUpdateAvailable,
+    appUpdatesAvailable,
 
-    sendInvite: sendInvite,
+    sendInvite,
 
-    appUp: appUp,
-    appDied: appDied,
-    appUpdated: appUpdated,
-    oomEvent: oomEvent,
+    appUp,
+    appDied,
+    appUpdated,
+    oomEvent,
 
-    backupFailed: backupFailed,
+    backupFailed,
 
-    certificateRenewalError: certificateRenewalError,
-    boxUpdateError: boxUpdateError,
+    certificateRenewalError,
+    boxUpdateError,
 
-    sendTestMail: sendTestMail,
+    sendTestMail,
 
     _mailQueue: [] // accumulate mails in test mode
 };
@@ -34,8 +32,8 @@ var assert = require('assert'),
     safe = require('safetydance'),
     settings = require('./settings.js'),
     showdown = require('showdown'),
-    smtpTransport = require('nodemailer-smtp-transport'),
-    util = require('util');
+    translation = require('./translation.js'),
+    smtpTransport = require('nodemailer-smtp-transport');
 
 var NOOP_CALLBACK = function (error) { if (error) debug(error); };
 
@@ -91,14 +89,21 @@ function sendMail(mailOptions, callback) {
     });
 }
 
-function render(templateFile, params) {
+function render(templateFile, params, translationAssets) {
     assert.strictEqual(typeof templateFile, 'string');
     assert.strictEqual(typeof params, 'object');
 
     var content = null;
+    var raw = safe.fs.readFileSync(path.join(MAIL_TEMPLATES_DIR, templateFile), 'utf8');
+    if (raw === null) {
+        debug(`Error loading ${templateFile}`);
+        return '';
+    }
+
+    if (typeof translationAssets === 'object') raw = translation.translate(raw, translationAssets.translations || {}, translationAssets.fallback || {});
 
     try {
-        content = ejs.render(safe.fs.readFileSync(path.join(MAIL_TEMPLATES_DIR, templateFile), 'utf8'), params);
+        content = ejs.render(raw, params);
     } catch (e) {
         debug(`Error rendering ${templateFile}`, e);
     }
@@ -106,25 +111,6 @@ function render(templateFile, params) {
     return content;
 }
 
-function mailUserEvent(mailTo, user, event) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-    assert.strictEqual(typeof event, 'string');
-
-    getMailConfig(function (error, mailConfig) {
-        if (error) return debug('Error getting mail details:', error);
-
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: mailTo,
-            subject: util.format('[%s] %s %s', mailConfig.cloudronName, user.username || user.fallbackEmail || user.email, event),
-            text: render('user_event.ejs', { user: user, event: event, format: 'text' }),
-        };
-
-        sendMail(mailOptions);
-    });
-}
-
 function sendInvite(user, invitor, inviteLink) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof invitor, 'object');
@@ -135,84 +121,31 @@ function sendInvite(user, invitor, inviteLink) {
     getMailConfig(function (error, mailConfig) {
         if (error) return debug('Error getting mail details:', error);
 
-        var templateData = {
-            user: user,
-            webadminUrl: settings.adminOrigin(),
-            inviteLink: inviteLink,
-            invitor: invitor,
-            cloudronName: mailConfig.cloudronName,
-            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
-        };
+        translation.getTranslations(function (error, translationAssets) {
+            if (error) return debug('Error getting translations:', error);
 
-        var templateDataText = JSON.parse(JSON.stringify(templateData));
-        templateDataText.format = 'text';
+            var templateData = {
+                user: user.displayName || user.username || user.email,
+                webadminUrl: settings.adminOrigin(),
+                inviteLink: inviteLink,
+                invitor: invitor ? invitor.email : null,
+                cloudronName: mailConfig.cloudronName,
+                cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
+            };
 
-        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
-        templateDataHTML.format = 'html';
+            var mailOptions = {
+                from: mailConfig.notificationFrom,
+                to: user.fallbackEmail,
+                subject: ejs.render(translation.translate('{{ welcomeEmail.subject }}', translationAssets.translations || {}, translationAssets.fallback || {}), { cloudron: mailConfig.cloudronName }),
+                text: render('welcome_user-text.ejs', templateData, translationAssets),
+                html: render('welcome_user-html.ejs', templateData, translationAssets)
+            };
 
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: user.fallbackEmail,
-            subject: util.format('Welcome to %s', mailConfig.cloudronName),
-            text: render('welcome_user.ejs', templateDataText),
-            html: render('welcome_user.ejs', templateDataHTML)
-        };
-
-        sendMail(mailOptions);
+            sendMail(mailOptions);
+        });
     });
 }
 
-function userAdded(mailTo, user) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-
-    debug(`userAdded: Sending mail for added users ${user.fallbackEmail} to ${mailTo}`);
-
-    getMailConfig(function (error, mailConfig) {
-        if (error) return debug('Error getting mail details:', error);
-
-        var templateData = {
-            user: user,
-            cloudronName: mailConfig.cloudronName,
-            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
-        };
-
-        var templateDataText = JSON.parse(JSON.stringify(templateData));
-        templateDataText.format = 'text';
-
-        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
-        templateDataHTML.format = 'html';
-
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: mailTo,
-            subject: util.format('[%s] User %s added', mailConfig.cloudronName, user.fallbackEmail),
-            text: render('user_added.ejs', templateDataText),
-            html: render('user_added.ejs', templateDataHTML)
-        };
-
-        sendMail(mailOptions);
-    });
-}
-
-function userRemoved(mailTo, user) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-
-    debug('Sending mail for userRemoved.', user.id, user.username, user.email);
-
-    mailUserEvent(mailTo, user, 'was removed');
-}
-
-function roleChanged(mailTo, user) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-
-    debug('Sending mail for roleChanged');
-
-    mailUserEvent(mailTo, user, `now has the role '${user.role}'`);
-}
-
 function passwordReset(user) {
     assert.strictEqual(typeof user, 'object');
 
@@ -221,28 +154,26 @@ function passwordReset(user) {
     getMailConfig(function (error, mailConfig) {
         if (error) return debug('Error getting mail details:', error);
 
-        var templateData = {
-            user: user,
-            resetLink: `${settings.adminOrigin()}/login.html?resetToken=${user.resetToken}`,
-            cloudronName: mailConfig.cloudronName,
-            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
-        };
+        translation.getTranslations(function (error, translationAssets) {
+            if (error) return debug('Error getting translations:', error);
 
-        var templateDataText = JSON.parse(JSON.stringify(templateData));
-        templateDataText.format = 'text';
+            var templateData = {
+                user: user.displayName || user.username || user.email,
+                resetLink: `${settings.adminOrigin()}/login.html?resetToken=${user.resetToken}`,
+                cloudronName: mailConfig.cloudronName,
+                cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
+            };
 
-        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
-        templateDataHTML.format = 'html';
+            var mailOptions = {
+                from: mailConfig.notificationFrom,
+                to: user.fallbackEmail,
+                subject: ejs.render(translation.translate('{{ passwordResetEmail.subject }}', translationAssets.translations || {}, translationAssets.fallback || {}), { cloudron: mailConfig.cloudronName }),
+                text: render('password_reset-text.ejs', templateData, translationAssets),
+                html: render('password_reset-html.ejs', templateData, translationAssets)
+            };
 
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: user.fallbackEmail,
-            subject: util.format('[%s] Password Reset', mailConfig.cloudronName),
-            text: render('password_reset.ejs', templateDataText),
-            html: render('password_reset.ejs', templateDataHTML)
-        };
-
-        sendMail(mailOptions);
+            sendMail(mailOptions);
+        });
     });
 }
 
@@ -258,7 +189,7 @@ function appUp(mailTo, app) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] App %s is back online', mailConfig.cloudronName, app.fqdn),
+            subject: `[${mailConfig.cloudronName}] App ${app.fqdn} is back online`,
             text: render('app_up.ejs', { title: app.manifest.title, appFqdn: app.fqdn, format: 'text' })
         };
 
@@ -278,7 +209,7 @@ function appDied(mailTo, app) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] App %s is down', mailConfig.cloudronName, app.fqdn),
+            subject: `[${mailConfig.cloudronName}] App ${app.fqdn} is down`,
             text: render('app_down.ejs', { title: app.manifest.title, appFqdn: app.fqdn, supportEmail: mailConfig.supportEmail, format: 'text' })
         };
 
@@ -325,10 +256,46 @@ function appUpdated(mailTo, app, callback) {
     });
 }
 
-function appUpdatesAvailable(mailTo, apps, hasSubscription, callback) {
+function boxUpdateAvailable(mailTo, updateInfo, callback) {
+    assert.strictEqual(typeof mailTo, 'string');
+    assert.strictEqual(typeof updateInfo, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    getMailConfig(function (error, mailConfig) {
+        if (error) return debug('Error getting mail details:', error);
+
+        var converter = new showdown.Converter();
+
+        var templateData = {
+            webadminUrl: settings.adminOrigin(),
+            newBoxVersion: updateInfo.version,
+            changelog: updateInfo.changelog,
+            changelogHTML: updateInfo.changelog.map(function (e) { return converter.makeHtml(e); }),
+            cloudronName: mailConfig.cloudronName,
+            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
+        };
+
+        var templateDataText = JSON.parse(JSON.stringify(templateData));
+        templateDataText.format = 'text';
+
+        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
+        templateDataHTML.format = 'html';
+
+        var mailOptions = {
+            from: mailConfig.notificationFrom,
+            to: mailTo,
+            subject: `[${mailConfig.cloudronName}] Cloudron update available`,
+            text: render('box_update_available.ejs', templateDataText),
+            html: render('box_update_available.ejs', templateDataHTML)
+        };
+
+        sendMail(mailOptions, callback);
+    });
+}
+
+function appUpdatesAvailable(mailTo, apps, callback) {
     assert.strictEqual(typeof mailTo, 'string');
     assert.strictEqual(typeof apps, 'object');
-    assert.strictEqual(typeof hasSubscription, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
     getMailConfig(function (error, mailConfig) {
@@ -341,7 +308,6 @@ function appUpdatesAvailable(mailTo, apps, hasSubscription, callback) {
 
         var templateData = {
             webadminUrl: settings.adminOrigin(),
-            hasSubscription: hasSubscription,
             apps: apps,
             cloudronName: mailConfig.cloudronName,
             cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
@@ -356,7 +322,7 @@ function appUpdatesAvailable(mailTo, apps, hasSubscription, callback) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: `New app updates available for ${mailConfig.cloudronName}`,
+            subject: `[${mailConfig.cloudronName}] App update available`,
             text: render('app_updates_available.ejs', templateDataText),
             html: render('app_updates_available.ejs', templateDataHTML)
         };
@@ -374,7 +340,7 @@ function backupFailed(mailTo, errorMessage, logUrl) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] Failed to backup', mailConfig.cloudronName),
+            subject: `[${mailConfig.cloudronName}] Failed to backup`,
             text: render('backup_failed.ejs', { cloudronName: mailConfig.cloudronName, message: errorMessage, logUrl: logUrl, format: 'text' })
         };
 
@@ -393,7 +359,7 @@ function certificateRenewalError(mailTo, domain, message) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] Certificate renewal error', domain),
+            subject: `[${mailConfig.cloudronName}] Certificate renewal error`,
             text: render('certificate_renewal_error.ejs', { domain: domain, message: message, format: 'text' })
         };
 
@@ -411,7 +377,7 @@ function boxUpdateError(mailTo, message) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] Cloudron update error', mailConfig.cloudronName),
+            subject: `[${mailConfig.cloudronName}] Cloudron update error`,
             text: render('box_update_error.ejs', { message: message, format: 'text' })
         };
 
@@ -419,19 +385,30 @@ function boxUpdateError(mailTo, message) {
     });
 }
 
-function oomEvent(mailTo, program, event) {
+function oomEvent(mailTo, app, addon, containerId, event) {
     assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof program, 'string');
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof addon, 'object');
+    assert.strictEqual(typeof containerId, 'string');
     assert.strictEqual(typeof event, 'object');
 
     getMailConfig(function (error, mailConfig) {
         if (error) return debug('Error getting mail details:', error);
 
+        const templateData = {
+            webadminUrl: settings.adminOrigin(),
+            cloudronName: mailConfig.cloudronName,
+            app,
+            addon,
+            event: JSON.stringify(event),
+            format: 'text'
+        };
+
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] %s was restarted (OOM)', mailConfig.cloudronName, program),
-            text: render('oom_event.ejs', { cloudronName: mailConfig.cloudronName, program: program, event: JSON.stringify(event), format: 'text' })
+            subject: `[${mailConfig.cloudronName}] ${app ? app.fqdn : addon.name} was restarted (OOM)`,
+            text: render('oom_event.ejs', templateData)
         };
 
         sendMail(mailOptions);
@@ -450,7 +427,7 @@ function sendTestMail(domain, email, callback) {
             authUser: `no-reply@${domain}`,
             from: `"${mailConfig.cloudronName}" <no-reply@${domain}>`,
             to: email,
-            subject: util.format('Test Email from %s', mailConfig.cloudronName),
+            subject: `[${mailConfig.cloudronName}] Test Email`,
             text: render('test.ejs', { cloudronName: mailConfig.cloudronName, format: 'text'})
         };
 

src/middleware/index.js

@@ -1,10 +1,11 @@
 'use strict';
 
 exports = module.exports = {
+    cookieParser: require('cookie-parser'),
     cors: require('./cors'),
     json: require('body-parser').json,
     morgan: require('morgan'),
-    proxy: require('proxy-middleware'),
+    proxy: require('./proxy-middleware.js'),
     lastMile: require('connect-lastmile'),
     multipart: require('./multipart.js'),
     timeout: require('connect-timeout'),

src/middleware/proxy-middleware.js

@@ -0,0 +1,149 @@
+// https://github.com/cloudron-io/node-proxy-middleware
+// MIT license
+// contains https://github.com/gonzalocasas/node-proxy-middleware/pull/59
+
+var os = require('os');
+var http = require('http');
+var https = require('https');
+var owns = {}.hasOwnProperty;
+
+module.exports = function proxyMiddleware(options) {
+  //enable ability to quickly pass a url for shorthand setup
+  if(typeof options === 'string'){
+      options = require('url').parse(options);
+  }
+
+  var httpLib = options.protocol === 'https:' ? https : http;
+  var request = httpLib.request;
+
+  options = options || {};
+  options.hostname = options.hostname;
+  options.port = options.port;
+  options.pathname = options.pathname || '/';
+
+  return function (req, resp, next) {
+    var url = req.url;
+    // You can pass the route within the options, as well
+    if (typeof options.route === 'string') {
+      if (url === options.route) {
+        url = '';
+      } else if (url.slice(0, options.route.length) === options.route) {
+        url = url.slice(options.route.length);
+      } else {
+        return next();
+      }
+    }
+
+    //options for this request
+    var opts = extend({}, options);
+    if (url && url.charAt(0) === '?') { // prevent /api/resource/?offset=0
+      if (options.pathname.length > 1 && options.pathname.charAt(options.pathname.length - 1) === '/') {
+        opts.path = options.pathname.substring(0, options.pathname.length - 1) + url;
+      } else {
+        opts.path = options.pathname + url;
+      }
+    } else if (url) {
+      opts.path = slashJoin(options.pathname, url);
+    } else {
+      opts.path = options.pathname;
+    }
+    opts.method = req.method;
+    opts.headers = options.headers ? merge(req.headers, options.headers) : req.headers;
+
+    applyViaHeader(req.headers, opts, opts.headers);
+
+    if (!options.preserveHost) {
+      // Forwarding the host breaks dotcloud
+      delete opts.headers.host;
+    }
+
+    var myReq = request(opts, function (myRes) {
+      var statusCode = myRes.statusCode
+        , headers = myRes.headers
+        , location = headers.location;
+      // Fix the location
+      if (((statusCode > 300 && statusCode < 304) || statusCode === 201) && location && location.indexOf(options.href) > -1) {
+        // absoulte path
+        headers.location = location.replace(options.href, slashJoin('/', slashJoin((options.route || ''), '')));
+      }
+      applyViaHeader(myRes.headers, opts, myRes.headers);
+      rewriteCookieHosts(myRes.headers, opts, myRes.headers, req);
+      resp.writeHead(myRes.statusCode, myRes.headers);
+      myRes.on('error', function (err) {
+        next(err);
+      });
+      myRes.on('end', function (err) {
+        next();
+      });
+      myRes.pipe(resp);
+    });
+    myReq.on('error', function (err) {
+      next(err);
+    });
+    if (!req.readable) {
+      myReq.end();
+    } else {
+      req.pipe(myReq);
+    }
+  };
+};
+
+function applyViaHeader(existingHeaders, opts, applyTo) {
+  if (!opts.via) return;
+
+  var viaName = (true === opts.via) ?  os.hostname() : opts.via;
+  var viaHeader = '1.1 ' + viaName;
+  if(existingHeaders.via) {
+    viaHeader = existingHeaders.via + ', ' + viaHeader;
+  }
+
+  applyTo.via = viaHeader;
+}
+
+function rewriteCookieHosts(existingHeaders, opts, applyTo, req) {
+  if (!opts.cookieRewrite || !owns.call(existingHeaders, 'set-cookie')) {
+    return;
+  }
+
+  var existingCookies = existingHeaders['set-cookie'],
+      rewrittenCookies = [],
+      rewriteHostname = (true === opts.cookieRewrite) ? os.hostname() : opts.cookieRewrite;
+
+  if (!Array.isArray(existingCookies)) {
+    existingCookies = [ existingCookies ];
+  }
+
+  for (var i = 0; i < existingCookies.length; i++) {
+    var rewrittenCookie = existingCookies[i].replace(/(Domain)=[a-z\.-_]*?(;|$)/gi, '$1=' + rewriteHostname + '$2');
+
+    if (!req.connection.encrypted) {
+      rewrittenCookie = rewrittenCookie.replace(/;\s*?(Secure)/i, '');
+    }
+    rewrittenCookies.push(rewrittenCookie);
+  }
+
+  applyTo['set-cookie'] = rewrittenCookies;
+}
+
+function slashJoin(p1, p2) {
+  var trailing_slash = false;
+
+  if (p1.length && p1[p1.length - 1] === '/') { trailing_slash = true; }
+  if (trailing_slash && p2.length && p2[0] === '/') {p2 = p2.substring(1); }
+
+  return p1 + p2;
+}
+
+function extend(obj, src) {
+  for (var key in src) if (owns.call(src, key)) obj[key] = src[key];
+  return obj;
+}
+
+//merges data without changing state in either argument
+function merge(src1, src2) {
+    var merged = {};
+    extend(merged, src1);
+    extend(merged, src2);
+    return merged;
+}
+

src/network.js

@@ -0,0 +1,56 @@
+'use strict';
+
+exports = module.exports = {
+    getBlocklist,
+    setBlocklist
+};
+
+const assert = require('assert'),
+    BoxError = require('./boxerror.js'),
+    ipaddr = require('ipaddr.js'),
+    path = require('path'),
+    paths = require('./paths.js'),
+    safe = require('safetydance'),
+    settings = require('./settings.js'),
+    shell = require('./shell.js'),
+    validator = require('validator');
+
+const SET_BLOCKLIST_CMD = path.join(__dirname, 'scripts/setblocklist.sh');
+
+function getBlocklist(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    const data = safe.fs.readFileSync(paths.FIREWALL_BLOCKLIST_FILE, 'utf8');
+    callback(null, data);
+}
+
+function setBlocklist(blocklist, auditSource, callback) {
+    assert.strictEqual(typeof blocklist, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const parsedIp = ipaddr.process(auditSource.ip);
+
+    for (const line of blocklist.split('\n')) {
+        if (!line || line.startsWith('#')) continue;
+        const rangeOrIP = line.trim();
+        if (!validator.isIP(rangeOrIP) && !validator.isIPRange(rangeOrIP)) return callback(new BoxError(BoxError.BAD_FIELD, `${rangeOrIP} is not a valid IP or range`));
+
+        if (rangeOrIP.indexOf('/') === -1) {
+            if (auditSource.ip === rangeOrIP) return callback(new BoxError(BoxError.BAD_FIELD, `${rangeOrIP} includes client IP. Cannot block yourself`));
+        } else {
+            const parsedRange = ipaddr.parseCIDR(rangeOrIP);
+            if (parsedIp.match(parsedRange)) return callback(new BoxError(BoxError.BAD_FIELD, `${rangeOrIP} includes client IP. Cannot block yourself`));
+        }
+    }
+
+    if (settings.isDemo()) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
+
+    if (!safe.fs.writeFileSync(paths.FIREWALL_BLOCKLIST_FILE, blocklist + '\n', 'utf8')) return callback(new BoxError(BoxError.FS_ERROR, safe.error.message));
+
+    shell.sudo('setBlocklist', [ SET_BLOCKLIST_CMD ], {}, function (error) {
+        if (error) return callback(new BoxError(BoxError.IPTABLES_ERROR, `Error setting blocklist: ${error.message}`));
+
+        callback();
+    });
+}

src/nginxconfig.ejs

@@ -6,52 +6,63 @@ map $http_upgrade $connection_upgrade {
 
 # http server
 server {
-    listen       80;
-    server_tokens off; # hide version
+<% if (endpoint === 'ip' || endpoint === 'setup') { -%>
+    listen 80 default_server;
+    server_name _;
 <% if (hasIPv6) { -%>
-    listen       [::]:80;
+    listen [::]:80 default_server;
 <% } -%>
-
-<% if (vhost) { -%>
-    server_name  <%= vhost %>;
 <% } else { -%>
-    # IP based access from collectd or initial cloudron setup. TODO: match the IPv6 address
-    server_name  "~^\d+\.\d+\.\d+\.\d+$";
-
-    # collectd
-    location /nginx_status {
-        stub_status on;
-        access_log off;
-        allow 127.0.0.1;
-        deny all;
-    }
+    listen 80;
+    server_name  <%= vhost %>;
+<% if (hasIPv6) { -%>
+    listen [::]:80;
+<% } -%>
 <% } -%>
 
-    # acme challenges (for cert renewal where the vhost config exists)
+    server_tokens off; # hide version
+
+    # acme challenges
     location /.well-known/acme-challenge/ {
         default_type text/plain;
         alias /home/yellowtent/platformdata/acme/;
     }
 
+    location /notfound.html {
+        root <%= sourceDir %>/dashboard/dist;
+        try_files /notfound.html =404;
+        internal;
+    }
+
+    # for default server, serve the notfound page. for other endpoints, redirect to HTTPS
     location / {
-        # redirect everything to HTTPS
+<% if ( endpoint === 'admin' || endpoint === 'setup' ) { %>
         return 301 https://$host$request_uri;
+<% } else if ( endpoint === 'app' ) { %>
+        return 301 https://$host$request_uri;
+<% } else if ( endpoint === 'redirect' ) { %>
+        return 301 https://<%= redirectTo %>$request_uri;
+<% } else if ( endpoint === 'ip' ) { %>
+        root /home/yellowtent/boxdata;
+        try_files /custom_pages/notfound.html /notfound.html;
+<% } %>
     }
 }
 
 # https server
 server {
-<% if (vhost) { -%>
-    server_name  <%= vhost %>;
-    listen       443 ssl http2;
-<% if (hasIPv6) { -%>
-    listen       [::]:443 ssl http2;
-<% } -%>
-<% } else { -%>
+<% if (endpoint === 'ip' || endpoint === 'setup') { -%>
     listen       443 ssl http2 default_server;
+    server_name _;
 <% if (hasIPv6) { -%>
     listen       [::]:443 ssl http2 default_server;
 <% } -%>
+<% } else { -%>
+    listen       443 ssl http2;
+    server_name  <%= vhost %>;
+<% if (hasIPv6) { -%>
+    listen       [::]:443 ssl http2;
+<% } -%>
 <% } -%>
 
     server_tokens off; # hide version
@@ -87,6 +98,14 @@ server {
     add_header Referrer-Policy "no-referrer-when-downgrade";
     proxy_hide_header Referrer-Policy;
 
+    # workaround caching issue after /logout. if max-age is set, browser uses cache and user thinks they have not logged out
+    # have to keep all the add_header here to avoid repeating all add_header in location block
+    <% if (proxyAuth.enabled) { %>
+    proxy_hide_header Cache-Control;
+    add_header Cache-Control no-cache;
+    add_header Set-Cookie $auth_cookie;
+    <% } %>
+
     # gzip responses that are > 50k and not images
     gzip on;
     gzip_min_length 50k;
@@ -95,7 +114,7 @@ server {
     # enable for proxied requests as well
     gzip_proxied any;
 
-<% if ( endpoint === 'admin' ) { -%>
+<% if ( endpoint === 'admin' || endpoint === 'ip' || endpoint === 'setup' ) { -%>
     # CSP headers for the admin/dashboard resources
     add_header Content-Security-Policy "default-src 'none'; frame-src 'self' cloudron.io *.cloudron.io; connect-src wss: https: 'self' *.cloudron.io; script-src https: 'self' 'unsafe-inline' 'unsafe-eval'; img-src * data:; style-src https: 'unsafe-inline'; object-src 'none'; font-src https: 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'self';";
 <% } else { %>
@@ -142,81 +161,150 @@ server {
 <% if ( endpoint === 'admin' ) { %>
         proxy_pass http://127.0.0.1:3000;
 <% } else if ( endpoint === 'app' ) { %>
-        proxy_pass http://127.0.0.1:<%= port %>;
+        proxy_pass http://<%= ip %>:<%= port %>;
 <% } else if ( endpoint === 'redirect' ) { %>
         return 302 https://<%= redirectTo %>$request_uri;
 <% } %>
     }
 
     # user defined .well-known resources
-    location ~ ^/.well-known/(.*)$ {
-        root /home/yellowtent/boxdata/well-known/$host;
-        try_files /$1 @wellknown-upstream;
+    location /.well-known/ {
+        error_page 404 = @wellknown-upstream;
+        proxy_pass http://127.0.0.1:3000/well-known-handler/;
     }
 
-    location / {
-        # increase the proxy buffer sizes to not run into buffer issues (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers)
-        proxy_buffer_size       128k;
-        proxy_buffers           4 256k;
-        proxy_busy_buffers_size 256k;
-
-        # No buffering to temp files, it fails for large downloads
-        proxy_max_temp_file_size 0;
-
-        # Disable check to allow unlimited body sizes. this allows apps to accept whatever size they want
-        client_max_body_size 0;
-
-<% if (robotsTxtQuoted) { %>
-        location = /robots.txt {
-            return 200 <%- robotsTxtQuoted %>;
-        }
+<% if (proxyAuth.enabled) { %>
+    proxy_set_header X-App-ID "<%= proxyAuth.id %>";
 <% } %>
 
-<% if ( endpoint === 'admin' ) { %>
-        location /api/ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 1m;
-        }
+    # increase the proxy buffer sizes to not run into buffer issues (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers)
+    proxy_buffer_size       128k;
+    proxy_buffers           4 256k;
+    proxy_busy_buffers_size 256k;
 
-        location ~ ^/api/v1/(developer|session)/login$ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 1m;
-            limit_req zone=admin_login burst=5;
-        }
+    # No buffering to temp files, it fails for large downloads
+    proxy_max_temp_file_size 0;
 
-        # the read timeout is between successive reads and not the whole connection
-        location ~ ^/api/v1/apps/.*/exec$ {
-            proxy_pass   http://127.0.0.1:3000;
-            proxy_read_timeout 30m;
-        }
+    # Disable check to allow unlimited body sizes. this allows apps to accept whatever size they want
+    client_max_body_size 0;
 
-        location ~ ^/api/v1/apps/.*/upload$ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 0;
-        }
+<% if (robotsTxtQuoted) { %>
+    location = /robots.txt {
+        return 200 <%- robotsTxtQuoted %>;
+    }
+<% } %>
 
-        location ~ ^/api/v1/apps/.*/files/ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 0;
-        }
+<% if ( endpoint === 'admin' || endpoint === 'setup' ) { %>
+    location /api/ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 1m;
+    }
 
-        # graphite paths (uncomment block below and visit /graphite-web/dashboard)
-        # remember to comment out the CSP policy as well to access the graphite dashboard
-        # location ~ ^/graphite-web/ {
-        #     proxy_pass   http://127.0.0.1:8417;
-        #     client_max_body_size 1m;
-        # }
+    location ~ ^/api/v1/(developer|session)/login$ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 1m;
+        limit_req zone=admin_login burst=5;
+    }
 
-        location / {
-            root   <%= sourceDir %>/dashboard/dist;
-            index  index.html index.htm;
-        }
+    # the read timeout is between successive reads and not the whole connection
+    location ~ ^/api/v1/apps/.*/exec$ {
+        proxy_pass   http://127.0.0.1:3000;
+        proxy_read_timeout 30m;
+    }
+
+    location ~ ^/api/v1/apps/.*/upload$ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 0;
+    }
+
+    location ~ ^/api/v1/apps/.*/files/ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 0;
+    }
+
+    location ~ ^/api/v1/volumes/.*/files/ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 0;
+    }
+
+    # graphite paths (uncomment block below and visit /graphite-web/dashboard)
+    # remember to comment out the CSP policy as well to access the graphite dashboard
+    # location ~ ^/graphite-web/ {
+    #     proxy_pass   http://127.0.0.1:8417;
+    #     client_max_body_size 1m;
+    # }
+
+    location / {
+        root   <%= sourceDir %>/dashboard/dist;
+        index  index.html index.htm;
+    }
 <% } else if ( endpoint === 'app' ) { %>
-        proxy_pass http://127.0.0.1:<%= port %>;
+    location = /appstatus.html {
+        root /home/yellowtent/box/dashboard/dist;
+    }
+
+    <% if (proxyAuth.enabled) { %>
+    location = /proxy-auth {
+        internal;
+        proxy_pass http://127.0.0.1:3001/auth;
+        proxy_pass_request_body off;
+        # repeat proxy headers since we addded proxy_set_header at this location level
+        proxy_set_header X-App-ID "<%= proxyAuth.id %>";
+        proxy_set_header Content-Length "";
+    }
+
+    location ~ ^/(login|logout)$ {
+        proxy_pass http://127.0.0.1:3001;
+    }
+
+    location @proxy-auth-login {
+        if ($http_user_agent ~* "docker-client") {
+            return 401;
+        }
+        return 302 /login?redirect=$request_uri;
+    }
+
+    location <%= proxyAuth.location %> {
+        auth_request /proxy-auth;
+        auth_request_set $auth_cookie $upstream_http_set_cookie;
+        error_page 401 = @proxy-auth-login;
+
+        proxy_pass http://<%= ip %>:<%= port %>;
+    }
+
+    <% if (proxyAuth.location !== '/') { %>
+    location / {
+        proxy_pass http://<%= ip %>:<%= port %>;
+    }
+    <% } %>
+
+    <% } else { %>
+    location / {
+        proxy_pass http://<%= ip %>:<%= port %>;
+    }
+    <% } %>
+
+    <% Object.keys(httpPaths).forEach(function (path) { -%>
+    location "<%= path %>" {
+        # the trailing / will replace part of the original URI matched by the location.
+        proxy_pass http://<%= ip %>:<%= httpPaths[path] %>/;
+    }
+    <% }); %>
+
 <% } else if ( endpoint === 'redirect' ) { %>
         # redirect everything to the app. this is temporary because there is no way
         # to clear a permanent redirect on the browser
         return 302 https://<%= redirectTo %>$request_uri;
-<% } %>
+<% } else if ( endpoint === 'ip' ) { %>
+    location /notfound.html {
+        root <%= sourceDir %>/dashboard/dist;
+        try_files /notfound.html =404;
+        internal;
     }
+
+    location / {
+        root /home/yellowtent/boxdata;
+        try_files /custom_pages/notfound.html /notfound.html;
+    }
+<% } %>
 }

src/notifications.js

@@ -1,11 +1,14 @@
 'use strict';
 
 exports = module.exports = {
-    get: get,
-    ack: ack,
-    getAllPaged: getAllPaged,
+    get,
+    ack,
+    getAllPaged,
 
-    onEvent: onEvent,
+    onEvent,
+
+    appUpdatesAvailable,
+    boxUpdateAvailable,
 
     // NOTE: if you add an alert, be sure to add title below
     ALERT_BACKUP_CONFIG: 'backupConfig',
@@ -20,11 +23,13 @@ exports = module.exports = {
     _add: add
 };
 
-let assert = require('assert'),
+let apps = require('./apps.js'),
+    assert = require('assert'),
     async = require('async'),
     auditSource = require('./auditsource.js'),
     BoxError = require('./boxerror.js'),
     changelog = require('./changelog.js'),
+    constants = require('./constants.js'),
     debug = require('debug')('box:notifications'),
     eventlog = require('./eventlog.js'),
     mailer = require('./mailer.js'),
@@ -94,8 +99,8 @@ function getAllPaged(userId, acknowledged, page, perPage, callback) {
 }
 
 // Calls iterator with (admin, callback)
-function actionForAllAdmins(skippingUserIds, iterator, callback) {
-    assert(Array.isArray(skippingUserIds));
+function forEachAdmin(options, iterator, callback) {
+    assert(Array.isArray(options.skip));
     assert.strictEqual(typeof iterator, 'function');
     assert.strictEqual(typeof callback, 'function');
 
@@ -104,7 +109,7 @@ function actionForAllAdmins(skippingUserIds, iterator, callback) {
         if (error) return callback(error);
 
         // filter out users we want to skip (like the user who did the action or the user the action was performed on)
-        result = result.filter(function (r) { return skippingUserIds.indexOf(r.id) === -1; });
+        result = result.filter(function (r) { return options.skip.indexOf(r.id) === -1; });
 
         async.each(result, iterator, callback);
     });
@@ -116,8 +121,7 @@ function userAdded(performedBy, eventId, user, callback) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([ performedBy, user.id ], function (admin, done) {
-        mailer.userAdded(admin.email, user);
+    forEachAdmin({ skip: [ performedBy, user.id ] }, function (admin, done) {
         add(admin.id, eventId, `User '${user.displayName}' added`, `User '${user.username || user.email || user.fallbackEmail}' was added.`, done);
     }, callback);
 }
@@ -128,8 +132,7 @@ function userRemoved(performedBy, eventId, user, callback) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([ performedBy, user.id ], function (admin, done) {
-        mailer.userRemoved(admin.email, user);
+    forEachAdmin({ skip: [ performedBy, user.id ] }, function (admin, done) {
         add(admin.id, eventId, `User '${user.displayName}' removed`, `User '${user.username || user.email || user.fallbackEmail}' was removed.`, done);
     }, callback);
 }
@@ -139,8 +142,7 @@ function roleChanged(performedBy, eventId, user, callback) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([ performedBy, user.id ], function (admin, done) {
-        mailer.roleChanged(admin.email, user);
+    forEachAdmin({ skip: [ performedBy, user.id ] }, function (admin, done) {
         add(admin.id, eventId, `User '${user.displayName}'s role changed`, `User '${user.username || user.email || user.fallbackEmail}' now has the role ${user.role}.`, done);
     }, callback);
 }
@@ -152,23 +154,19 @@ function oomEvent(eventId, app, addon, containerId, event, callback) {
     assert.strictEqual(typeof containerId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    let title, message, program;
+    assert(app || addon);
+
+    let title, message;
     if (app) {
-        program = `App ${app.fqdn}`;
-        title = `The application ${app.fqdn} (${app.manifest.title}) ran out of memory.`;
-        message = 'The application has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://cloudron.io/documentation/apps/#memory-limit)';
+        title = `The application at ${app.fqdn} ran out of memory.`;
+        message = `The application has been restarted automatically. If you see this notification often, consider increasing the [memory limit](${settings.adminOrigin()}/#/app/${app.id}/resources)`;
     } else if (addon) {
-        program = `${addon.name} service`;
         title = `The ${addon.name} service ran out of memory`;
-        message = 'The service has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://cloudron.io/documentation/troubleshooting/#services)';
-    } else { // this never happens currently
-        program = `Container ${containerId}`;
-        title = `The container ${containerId} ran out of memory`;
-        message = 'The container has been restarted automatically. Consider increasing the [memory limit](https://docs.docker.com/v17.09/edge/engine/reference/commandline/update/#update-a-containers-kernel-memory-constraints)';
+        message = `The service has been restarted automatically. If you see this notification often, consider increasing the [memory limit](${settings.adminOrigin()}/#/services)`;
     }
 
-    actionForAllAdmins([], function (admin, done) {
-        mailer.oomEvent(admin.email, program, event);
+    forEachAdmin({ skip: [] }, function (admin, done) {
+        mailer.oomEvent(admin.email, app, addon, containerId, event);
 
         add(admin.id, eventId, title, message, done);
     }, callback);
@@ -179,9 +177,9 @@ function appUp(eventId, app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, done) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
         mailer.appUp(admin.email, app);
-        add(admin.id, eventId, `App ${app.fqdn} is back online`, `The application ${app.manifest.title} installed at ${app.fqdn} is back online.`, done);
+        add(admin.id, eventId, `App ${app.fqdn} is back online`, `The application installed at ${app.fqdn} is back online.`, done);
     }, callback);
 }
 
@@ -190,9 +188,9 @@ function appDied(eventId, app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         mailer.appDied(admin.email, app);
-        add(admin.id, eventId, `App ${app.fqdn} is down`, `The application ${app.manifest.title} installed at ${app.fqdn} is not responding.`, callback);
+        add(admin.id, eventId, `App ${app.fqdn} is down`, `The application installed at ${app.fqdn} is not responding.`, callback);
     }, callback);
 }
 
@@ -201,13 +199,15 @@ function appUpdated(eventId, app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
+    if (!app.appStoreId) return callback(); // skip notification of dev apps
+
     const tmp = app.manifest.description.match(/<upstream>(.*)<\/upstream>/i);
     const upstreamVersion = (tmp && tmp[1]) ? tmp[1] : '';
     const title = upstreamVersion ? `${app.manifest.title} at ${app.fqdn} updated to ${upstreamVersion} (package version ${app.manifest.version})`
         : `${app.manifest.title} at ${app.fqdn} updated to package version ${app.manifest.version}`;
 
-    actionForAllAdmins([], function (admin, done) {
-        add(admin.id, eventId, title, `The application ${app.manifest.title} installed at https://${app.fqdn} was updated.\n\nChangelog:\n${app.manifest.changelog}\n`, function (error) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
+        add(admin.id, eventId, title, `The application installed at https://${app.fqdn} was updated.\n\nChangelog:\n${app.manifest.changelog}\n`, function (error) {
             if (error) return callback(error);
 
             mailer.appUpdated(admin.email, app, function (error) {
@@ -218,6 +218,39 @@ function appUpdated(eventId, app, callback) {
     }, callback);
 }
 
+function boxUpdateAvailable(updateInfo, callback) {
+    assert.strictEqual(typeof updateInfo, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    settings.getAutoupdatePattern(function (error, result) {
+        if (error) return callback(error);
+
+        if (result !== constants.AUTOUPDATE_PATTERN_NEVER) return callback();
+
+        forEachAdmin({ skip: [] }, function (admin, done) {
+            mailer.boxUpdateAvailable(admin.email, updateInfo, done);
+        }, callback);
+    });
+}
+
+function appUpdatesAvailable(appUpdates, callback) {
+    assert.strictEqual(typeof appUpdates, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    settings.getAutoupdatePattern(function (error, result) {
+        if (error) return callback(error);
+
+        // if we are auto updating, then just consider apps that cannot be auto updated
+        if (result !== constants.AUTOUPDATE_PATTERN_NEVER) appUpdates = appUpdates.filter(update => !apps.canAutoupdateApp(update.app, update.updateInfo));
+
+        if (appUpdates.length === 0) return callback();
+
+        forEachAdmin({ skip: [] }, function (admin, done) {
+            mailer.appUpdatesAvailable(admin.email, appUpdates, done);
+        }, callback);
+    });
+}
+
 function boxUpdated(eventId, oldVersion, newVersion, callback) {
     assert.strictEqual(typeof eventId, 'string');
     assert.strictEqual(typeof oldVersion, 'string');
@@ -227,7 +260,7 @@ function boxUpdated(eventId, oldVersion, newVersion, callback) {
     const changes = changelog.getChanges(newVersion);
     const changelogMarkdown = changes.map((m) => `* ${m}\n`).join('');
 
-    actionForAllAdmins([], function (admin, done) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
         add(admin.id, eventId, `Cloudron updated to v${newVersion}`, `Cloudron was updated from v${oldVersion} to v${newVersion}.\n\nChangelog:\n${changelogMarkdown}\n`, done);
     }, callback);
 }
@@ -237,9 +270,9 @@ function boxUpdateError(eventId, errorMessage, callback) {
     assert.strictEqual(typeof errorMessage, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, done) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
         mailer.boxUpdateError(admin.email, errorMessage);
-        add(admin.id, eventId, 'Cloudron update failed', `Failed to update Cloudron: ${errorMessage}. Update will be retried in 4 hours`, done);
+        add(admin.id, eventId, 'Cloudron update failed', `Failed to update Cloudron: ${errorMessage}.`, done);
     }, callback);
 }
 
@@ -249,7 +282,7 @@ function certificateRenewalError(eventId, vhost, errorMessage, callback) {
     assert.strictEqual(typeof errorMessage, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         mailer.certificateRenewalError(admin.email, vhost, errorMessage);
         add(admin.id, eventId, `Certificate renewal of ${vhost} failed`, `Failed to new certs of ${vhost}: ${errorMessage}. Renewal will be retried in 12 hours`, callback);
     }, callback);
@@ -261,9 +294,9 @@ function backupFailed(eventId, taskId, errorMessage, callback) {
     assert.strictEqual(typeof errorMessage, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         mailer.backupFailed(admin.email, errorMessage, `${settings.adminOrigin()}/logs.html?taskId=${taskId}`);
-        add(admin.id, eventId, 'Backup failed', `Backup failed: ${errorMessage}. Logs are available [here](/logs.html?taskId=${taskId}). Will be retried in 4 hours`, callback);
+        add(admin.id, eventId, 'Backup failed', `Backup failed: ${errorMessage}. Logs are available [here](/logs.html?taskId=${taskId}).`, callback);
     }, callback);
 }
 
@@ -273,11 +306,10 @@ function alert(id, title, message, callback) {
     assert.strictEqual(typeof message, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`alert: id=${id} title=${title}`);
-
     const acknowledged = !message;
+    debug(`alert: id=${id} title=${title} ack=${acknowledged}`);
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         const data = {
             userId: admin.id,
             eventId: null,
@@ -338,7 +370,6 @@ function onEvent(id, action, source, data, callback) {
         return appUp(id, data.app, callback);
 
     case eventlog.ACTION_APP_UPDATE_FINISH:
-        if (!data.app.appStoreId) return callback(); // skip notification of dev apps
         return appUpdated(id, data.app, callback);
 
     case eventlog.ACTION_CERTIFICATE_RENEWAL:

src/paths.js

@@ -16,8 +16,10 @@ exports = module.exports = {
 
     CLOUDRON_DEFAULT_AVATAR_FILE: path.join(__dirname + '/../assets/avatar.png'),
     INFRA_VERSION_FILE: path.join(baseDir(), 'platformdata/INFRA_VERSION'),
+    DASHBOARD_DIR: constants.TEST ? path.join(__dirname, '../../dashboard/src') : path.join(baseDir(), 'box/dashboard/dist'),
 
     PROVIDER_FILE: '/etc/cloudron/PROVIDER',
+    SETUP_TOKEN_FILE: '/etc/cloudron/SETUP_TOKEN',
 
     PLATFORM_DATA_DIR: path.join(baseDir(), 'platformdata'),
     APPS_DATA_DIR: path.join(baseDir(), 'appsdata'),
@@ -35,17 +37,21 @@ exports = module.exports = {
     SNAPSHOT_INFO_FILE: path.join(baseDir(), 'platformdata/backup/snapshot-info.json'),
     DYNDNS_INFO_FILE: path.join(baseDir(), 'platformdata/dyndns-info.json'),
     FEATURES_INFO_FILE: path.join(baseDir(), 'platformdata/features-info.json'),
+    PROXY_AUTH_TOKEN_SECRET_FILE: path.join(baseDir(), 'platformdata/proxy-auth-token-secret'),
     VERSION_FILE: path.join(baseDir(), 'platformdata/VERSION'),
 
     // this is not part of appdata because an icon may be set before install
     APP_ICONS_DIR: path.join(baseDir(), 'boxdata/appicons'),
     PROFILE_ICONS_DIR: path.join(baseDir(), 'boxdata/profileicons'),
     MAIL_DATA_DIR: path.join(baseDir(), 'boxdata/mail'),
+    SFTP_KEYS_DIR: path.join(baseDir(), 'boxdata/sftp/ssh'),
     ACME_ACCOUNT_KEY_FILE: path.join(baseDir(), 'boxdata/acme/acme.key'),
     APP_CERTS_DIR: path.join(baseDir(), 'boxdata/certs'),
     CLOUDRON_AVATAR_FILE: path.join(baseDir(), 'boxdata/avatar.png'),
     UPDATE_CHECKER_FILE: path.join(baseDir(), 'boxdata/updatechecker.json'),
     ADDON_TURN_SECRET_FILE: path.join(baseDir(), 'boxdata/addon-turn-secret'),
+    FIREWALL_BLOCKLIST_FILE: path.join(baseDir(), 'boxdata/firewall/blocklist.txt'),
+    FIREWALL_CONFIG_FILE: path.join(baseDir(), 'boxdata/firewall-config.json'),
 
     LOG_DIR: path.join(baseDir(), 'platformdata/logs'),
     TASKS_LOG_DIR: path.join(baseDir(), 'platformdata/logs/tasks'),
@@ -54,7 +60,9 @@ exports = module.exports = {
 
     GHOST_USER_FILE: path.join(baseDir(), 'platformdata/cloudron_ghost.json'),
 
+    SWAP_RATIO_FILE: path.join(baseDir(), 'platformdata/swap-ratio'),
+
     // this pattern is for the cloudron logs API route to work
     BACKUP_LOG_FILE: path.join(baseDir(), 'platformdata/logs/backup/app.log'),
-    UPDATER_LOG_FILE: path.join(baseDir(), 'platformdata/logs/updater/app.log')
+    UPDATER_LOG_FILE: path.join(baseDir(), 'platformdata/logs/updater/app.log'),
 };

src/platform.js

@@ -1,33 +1,28 @@
 'use strict';
 
 exports = module.exports = {
-    start: start,
-    stopAllTasks: stopAllTasks,
+    start,
+    stopAllTasks,
 
     // exported for testing
     _isReady: false
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     assert = require('assert'),
     async = require('async'),
     debug = require('debug')('box:platform'),
     fs = require('fs'),
-    graphs = require('./graphs.js'),
     infra = require('./infra_version.js'),
     locker = require('./locker.js'),
     paths = require('./paths.js'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
-    settings = require('./settings.js'),
-    sftp = require('./sftp.js'),
+    services = require('./services.js'),
     shell = require('./shell.js'),
     tasks = require('./tasks.js'),
     _ = require('underscore');
 
-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
-
 function start(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -45,7 +40,7 @@ function start(callback) {
     if (_.isEqual(infra, existingInfra)) {
         debug('platform is uptodate at version %s', infra.version);
 
-        onPlatformReady();
+        onPlatformReady(false /* !infraChanged */);
 
         return callback();
     }
@@ -56,18 +51,16 @@ function start(callback) {
     if (error) return callback(error);
 
     async.series([
-        (next) => { if (existingInfra.version !== infra.version) removeAllContainers(existingInfra, next); else next(); },
+        (next) => { if (existingInfra.version !== infra.version) removeAllContainers(next); else next(); },
         markApps.bind(null, existingInfra), // mark app state before we start addons. this gives the db import logic a chance to mark an app as errored
-        graphs.startGraphite.bind(null, existingInfra),
-        sftp.startSftp.bind(null, existingInfra),
-        addons.startServices.bind(null, existingInfra),
+        services.startServices.bind(null, existingInfra),
         fs.writeFile.bind(fs, paths.INFRA_VERSION_FILE, JSON.stringify(infra, null, 4))
     ], function (error) {
         if (error) return callback(error);
 
         locker.unlock(locker.OP_PLATFORM_START);
 
-        onPlatformReady();
+        onPlatformReady(true /* infraChanged */);
 
         callback();
     });
@@ -77,31 +70,18 @@ function stopAllTasks(callback) {
     tasks.stopAllTasks(callback);
 }
 
-function onPlatformReady() {
-    debug('onPlatformReady: platform is ready');
+function onPlatformReady(infraChanged) {
+    debug(`onPlatformReady: platform is ready. infra changed: ${infraChanged}`);
     exports._isReady = true;
 
-    apps.schedulePendingTasks(NOOP_CALLBACK);
+    let tasks = [ apps.schedulePendingTasks ];
+    if (infraChanged) tasks.push(pruneInfraImages);
 
-    applyPlatformConfig(NOOP_CALLBACK);
-    pruneInfraImages(NOOP_CALLBACK);
-}
-
-function applyPlatformConfig(callback) {
-    // scale back db containers, if possible. this is retried because updating memory constraints can fail
-    // with failed to write to memory.memsw.limit_in_bytes: write /sys/fs/cgroup/memory/docker/xx/memory.memsw.limit_in_bytes: device or resource busy
-
-    async.retry({ times: 10, interval: 5 * 60 * 1000 }, function (retryCallback) {
-        settings.getPlatformConfig(function (error, platformConfig) {
-            if (error) return retryCallback(error);
-
-            addons.updateServiceConfig(platformConfig, function (error) {
-                if (error) debug('Error updating services. Will rety in 5 minutes', platformConfig, error);
-
-                retryCallback(error);
-            });
+    async.series(async.reflectAll(tasks), function (error, results) {
+        results.forEach((result, idx) => {
+            if (result.error) debug(`Startup task at index ${idx} failed: ${result.error.message}`);
         });
-    }, callback);
+    });
 }
 
 function pruneInfraImages(callback) {
@@ -122,14 +102,14 @@ function pruneInfraImages(callback) {
             debug(`pruneInfraImages: removing unused image of ${image.repo}: tag: ${parts[1]} id: ${parts[0]}`);
 
             let result = safe.child_process.execSync(`docker rmi ${parts[0]}`, { encoding: 'utf8' });
-            if (result === null) debug(`Erroring removing image ${parts[0]}: ${safe.error.mesage}`);
+            if (result === null) debug(`Error removing image ${parts[0]}: ${safe.error.mesage}`);
         }
 
         iteratorCallback();
     }, callback);
 }
 
-function removeAllContainers(existingInfra, callback) {
+function removeAllContainers(callback) {
     debug('removeAllContainers: removing all containers for infra upgrade');
 
     async.series([

src/provision.js

@@ -11,6 +11,7 @@ var assert = require('assert'),
     async = require('async'),
     backups = require('./backups.js'),
     BoxError = require('./boxerror.js'),
+    branding = require('./branding.js'),
     constants = require('./constants.js'),
     cloudron = require('./cloudron.js'),
     debug = require('debug')('box:provision'),
@@ -54,7 +55,7 @@ function unprovision(callback) {
 
     // TODO: also cancel any existing configureWebadmin task
     async.series([
-        settings.setAdmin.bind(null, '', ''),
+        settings.setAdminLocation.bind(null, '', ''),
         mail.clearDomains,
         domains.clear
     ], callback);
@@ -98,24 +99,24 @@ function setup(dnsConfig, sysinfoConfig, auditSource, callback) {
                 dkimSelector: 'cloudron'
             };
 
-            domains.add(domain, data, auditSource, function (error) {
+            async.series([
+                settings.setMailLocation.bind(null, domain, `${constants.ADMIN_LOCATION}.${domain}`), // default mail location. do this before we add the domain for upserting mail DNS
+                domains.add.bind(null, domain, data, auditSource),
+                sysinfo.testConfig.bind(null, sysinfoConfig)
+            ], function (error) {
                 if (error) return done(error);
 
-                sysinfo.testConfig(sysinfoConfig, function (error) {
-                    if (error) return done(error);
+                callback(); // now that args are validated run the task in the background
 
-                    callback(); // now that args are validated run the task in the background
-
-                    async.series([
-                        settings.setSysinfoConfig.bind(null, sysinfoConfig),
-                        domains.prepareDashboardDomain.bind(null, domain, auditSource, (progress) => setProgress('setup', progress.message, NOOP_CALLBACK)),
-                        cloudron.setDashboardDomain.bind(null, domain, auditSource),
-                        setProgress.bind(null, 'setup', 'Done'),
-                        eventlog.add.bind(null, eventlog.ACTION_PROVISION, auditSource, { })
-                    ], function (error) {
-                        gProvisionStatus.setup.active = false;
-                        gProvisionStatus.setup.errorMessage = error ? error.message : '';
-                    });
+                async.series([
+                    settings.setSysinfoConfig.bind(null, sysinfoConfig),
+                    cloudron.setupDnsAndCert.bind(null, constants.ADMIN_LOCATION, domain, auditSource, (progress) => setProgress('setup', progress.message, NOOP_CALLBACK)),
+                    cloudron.setDashboardDomain.bind(null, domain, auditSource),
+                    setProgress.bind(null, 'setup', 'Done'),
+                    eventlog.add.bind(null, eventlog.ACTION_PROVISION, auditSource, { })
+                ], function (error) {
+                    gProvisionStatus.setup.active = false;
+                    gProvisionStatus.setup.errorMessage = error ? error.message : '';
                 });
             });
         });
@@ -162,7 +163,7 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
     assert.strictEqual(typeof callback, 'function');
 
     if (!semver.valid(version)) return callback(new BoxError(BoxError.BAD_FIELD, 'version is not a valid semver', { field: 'version' }));
-    if (constants.VERSION !== version) return callback(new BoxError(BoxError.BAD_STATE, `Run cloudron-setup with --version ${version} to restore from this backup`));
+    if (constants.VERSION !== version) return callback(new BoxError(BoxError.BAD_STATE, `Run "cloudron-setup --version ${version}" on a fresh Ubuntu installation to restore from this backup`));
 
     if (gProvisionStatus.setup.active || gProvisionStatus.restore.active) return callback(new BoxError(BoxError.BAD_STATE, 'Already setting up or restoring'));
 
@@ -199,7 +200,13 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
                     setProgress.bind(null, 'restore', 'Downloading backup'),
                     backups.restore.bind(null, backupConfig, backupId, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK)),
                     settings.setSysinfoConfig.bind(null, sysinfoConfig),
-                    cloudron.setupDashboard.bind(null, auditSource, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK)),
+                    (done) => {
+                        const adminDomain = settings.adminDomain(); // load this fresh from after the backup.restore
+                        async.series([
+                            cloudron.setupDnsAndCert.bind(null, constants.ADMIN_LOCATION, adminDomain, auditSource, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK)),
+                            cloudron.setDashboardDomain.bind(null, adminDomain, auditSource)
+                        ], done);
+                    },
                     settings.setBackupCredentials.bind(null, backupConfig), // update just the credentials and not the policy and flags
                     eventlog.add.bind(null, eventlog.ACTION_RESTORE, auditSource, { backupId }),
                 ], function (error) {
@@ -227,8 +234,9 @@ function getStatus(callback) {
                 apiServerOrigin: settings.apiServerOrigin(), // used by CaaS tool
                 webServerOrigin: settings.webServerOrigin(), // used by CaaS tool
                 cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-                footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
+                footer: branding.renderFooter(allSettings[settings.FOOTER_KEY] || constants.FOOTER),
                 adminFqdn: settings.adminDomain() ? settings.adminFqdn() : null,
+                language: allSettings[settings.LANGUAGE_KEY],
                 activated: activated,
                 provider: settings.provider() // used by setup wizard of marketplace images
             }, gProvisionStatus));

src/proxyauth.js

@@ -0,0 +1,259 @@
+'use strict';
+
+// heavily inspired from https://gock.net/blog/2020/nginx-subrequest-authentication-server/ and https://github.com/andygock/auth-server
+
+exports = module.exports = {
+    start,
+    stop
+};
+
+const apps = require('./apps.js'),
+    assert = require('assert'),
+    basicAuth = require('basic-auth'),
+    constants = require('./constants.js'),
+    debug = require('debug')('box:proxyAuth'),
+    ejs = require('ejs'),
+    express = require('express'),
+    fs = require('fs'),
+    hat = require('./hat.js'),
+    http = require('http'),
+    HttpError = require('connect-lastmile').HttpError,
+    HttpSuccess = require('connect-lastmile').HttpSuccess,
+    jwt = require('jsonwebtoken'),
+    middleware = require('./middleware'),
+    path = require('path'),
+    paths = require('./paths.js'),
+    safe = require('safetydance'),
+    speakeasy = require('speakeasy'),
+    translation = require('./translation.js'),
+    users = require('./users.js');
+
+let gHttpServer = null;
+let TOKEN_SECRET = null;
+const EXPIRY_DAYS = 7;
+
+function jwtVerify(req, res, next) {
+    const token = req.cookies.authToken;
+
+    if (!token) return next();
+
+    jwt.verify(token, TOKEN_SECRET, function (error, decoded) {
+        if (error) {
+            debug('clearing token', error);
+            res.clearCookie('authToken');
+            return next(new HttpError(403, 'Malformed token or bad signature'));
+        }
+
+        req.user = decoded.user || null;
+        next();
+    });
+}
+
+function basicAuthVerify(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    const credentials = basicAuth(req);
+    if (!appId || !credentials) return next();
+
+    const api = credentials.name.indexOf('@') !== -1 ? users.verifyWithEmail : users.verifyWithUsername;
+
+    api(credentials.name, credentials.pass, appId, function (error, user) {
+        if (error) return next(new HttpError(403, 'Invalid username or password' ));
+
+        req.user = user;
+        next();
+    });
+}
+
+function loginPage(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    translation.getTranslations(function (error, translationAssets) {
+        if (error) return next(new HttpError(500, 'No translation found'));
+
+        const raw = safe.fs.readFileSync(path.join(paths.DASHBOARD_DIR, 'templates/proxyauth-login.ejs'), 'utf8');
+        if (raw === null) return next(new HttpError(500, 'Login template not found'));
+
+        const translatedContent = translation.translate(raw, translationAssets.translations || {}, translationAssets.fallback || {});
+        var finalContent = '';
+
+        apps.get(appId, function (error, app) {
+            if (error) return next(new HttpError(503, error.message));
+
+            const title = app.label || app.manifest.title;
+
+            apps.getIconPath(app, {}, function (error, iconPath) {
+                const icon = 'data:image/png;base64,' + safe.fs.readFileSync(iconPath || '', 'base64');
+
+                try {
+                    finalContent = ejs.render(translatedContent, { title, icon });
+                } catch (e) {
+                    debug('Error rendering proxyauth-login.ejs', e);
+                    return next(new HttpError(500, 'Login template error'));
+                }
+
+                res.set('Content-Type', 'text/html');
+                return res.send(finalContent);
+            });
+        });
+    });
+}
+
+// someday this can be more sophisticated and check for a real browser
+function isBrowser(req) {
+    const userAgent = req.get('user-agent');
+    if (!userAgent) return false;
+
+    return !userAgent.toLowerCase().includes('docker-client');
+}
+
+// called by nginx to authorize any protected route. this route must return only 2xx or 401/403 (http://nginx.org/en/docs/http/ngx_http_auth_request_module.html)
+function auth(req, res, next) {
+    if (!req.user) {
+        if (isBrowser(req)) return next(new HttpError(401, 'Unauthorized'));
+
+        // the header has to be generated here and cannot be set in nginx config - https://forum.nginx.org/read.php?2,171461,171469#msg-171469
+        res.set('www-authenticate', 'Basic realm="Cloudron"');
+        return next(new HttpError(401, 'Unauthorized'));
+    }
+
+    // user is already authenticated, refresh cookie
+    const token = jwt.sign({ user: req.user }, TOKEN_SECRET, { expiresIn: `${EXPIRY_DAYS}d` });
+
+    res.cookie('authToken', token, {
+        httpOnly: true,
+        maxAge: EXPIRY_DAYS * 86400 * 1000, // milliseconds
+        secure: true
+    });
+
+    return next(new HttpSuccess(200, {}));
+}
+
+// endpoint called by login page, username and password posted as JSON body
+function passwordAuth(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    if (typeof req.body.username !== 'string') return next(new HttpError(400, 'username must be non empty string' ));
+    if (typeof req.body.password !== 'string') return next(new HttpError(400, 'password must be non empty string' ));
+    if ('totpToken' in req.body && typeof req.body.totpToken !== 'string') return next(new HttpError(400, 'totpToken must be a string' ));
+
+    const { username, password, totpToken } = req.body;
+
+    const api = username.indexOf('@') !== -1 ? users.verifyWithEmail : users.verifyWithUsername;
+
+    api(username, password, appId, function (error, user) {
+        if (error) return next(new HttpError(403, 'Invalid username or password' ));
+
+        if (!user.ghost && !user.appPassword && user.twoFactorAuthenticationEnabled) {
+            if (!totpToken) return next(new HttpError(403, 'A totpToken must be provided'));
+
+            let verified = speakeasy.totp.verify({ secret: user.twoFactorAuthenticationSecret, encoding: 'base32', token: req.body.totpToken, window: 2 });
+            if (!verified) return next(new HttpError(403, 'Invalid totpToken'));
+        }
+
+        req.user = user;
+        next();
+    });
+}
+
+function authorize(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    apps.get(appId, function (error, app) {
+        if (error) return next(new HttpError(403, 'No such app' ));
+
+        apps.hasAccessTo(app, req.user, function (error, hasAccess) {
+            if (error) return next(new HttpError(403, 'Forbidden' ));
+            if (!hasAccess) return next(new HttpError(403, 'Forbidden' ));
+
+            const token = jwt.sign({ user: users.removePrivateFields(req.user) }, TOKEN_SECRET, { expiresIn: `${EXPIRY_DAYS}d` });
+
+            res.cookie('authToken', token, {
+                httpOnly: true,
+                maxAge: EXPIRY_DAYS * 86400 * 1000, // milliseconds
+                secure: true
+            });
+
+            res.redirect(302, '/');
+        });
+    });
+}
+
+function logoutPage(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    apps.get(appId, function (error, app) {
+        if (error) return next(new HttpError(503, error.message));
+
+        res.clearCookie('authToken');
+
+        // when we have no path, redirect to the login page. we cannot redirect to '/' because browsers will immediately serve up the cached page
+        // if a path is set, we can assume '/' is a public page
+        res.redirect(302, app.manifest.addons.proxyAuth.path ? '/' : '/login');
+    });
+}
+
+function logout(req, res, next) {
+    res.clearCookie('authToken');
+    next(new HttpSuccess(200, {}));
+}
+
+// provides webhooks for the auth wall
+function initializeAuthwallExpressSync() {
+    let app = express();
+    let httpServer = http.createServer(app);
+
+    let QUERY_LIMIT = '1mb'; // max size for json and urlencoded queries
+    let REQUEST_TIMEOUT = 10000; // timeout for all requests
+
+    let json = middleware.json({ strict: true, limit: QUERY_LIMIT }); // application/json
+
+    if (process.env.BOX_ENV !== 'test') app.use(middleware.morgan('proxyauth :method :url :status :response-time ms - :res[content-length]', { immediate: false }));
+
+    var router = new express.Router();
+    router.del = router.delete; // amend router.del for readability further on
+
+    app
+        .use(middleware.timeout(REQUEST_TIMEOUT))
+        .use(middleware.cookieParser())
+        .use(router)
+        .use(middleware.lastMile());
+
+    router.get ('/login',     loginPage);
+    router.get ('/auth',      jwtVerify, basicAuthVerify, auth);
+    router.post('/login',     json, passwordAuth, authorize);
+    router.get ('/logout',    logoutPage);
+    router.post('/logout',    json, logout);
+
+    return httpServer;
+}
+
+function start(callback) {
+    assert.strictEqual(typeof callback, 'function');
+    assert.strictEqual(gHttpServer, null, 'Authwall is already up and running.');
+
+    if (!fs.existsSync(paths.PROXY_AUTH_TOKEN_SECRET_FILE)) {
+        TOKEN_SECRET = hat(64);
+        fs.writeFileSync(paths.PROXY_AUTH_TOKEN_SECRET_FILE, TOKEN_SECRET, 'utf8');
+    } else {
+        TOKEN_SECRET = fs.readFileSync(paths.PROXY_AUTH_TOKEN_SECRET_FILE, 'utf8').trim();
+    }
+
+    gHttpServer = initializeAuthwallExpressSync();
+
+    gHttpServer.listen(constants.AUTHWALL_PORT, '127.0.0.1', callback);
+}
+
+function stop(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!gHttpServer) return callback(null);
+
+    gHttpServer.close(callback);
+    gHttpServer = null;
+}

src/reverseproxy.js

@@ -1,30 +1,30 @@
 'use strict';
 
 exports = module.exports = {
-    setFallbackCertificate: setFallbackCertificate,
-    getFallbackCertificate: getFallbackCertificate,
+    setFallbackCertificate,
+    getFallbackCertificate,
 
-    generateFallbackCertificateSync: generateFallbackCertificateSync,
-    setAppCertificateSync: setAppCertificateSync,
+    generateFallbackCertificateSync,
+    setAppCertificateSync,
 
-    validateCertificate: validateCertificate,
+    validateCertificate,
 
-    getCertificate: getCertificate,
-    ensureCertificate: ensureCertificate,
+    getCertificate,
+    ensureCertificate,
 
-    renewCerts: renewCerts,
+    renewCerts,
 
     // the 'configure' ensure a certificate and generate nginx config
-    configureAdmin: configureAdmin,
-    configureApp: configureApp,
-    unconfigureApp: unconfigureApp,
+    configureAdmin,
+    configureApp,
+    unconfigureApp,
 
     // these only generate nginx config
-    writeDefaultConfig: writeDefaultConfig,
-    writeAdminConfig: writeAdminConfig,
-    writeAppConfig: writeAppConfig,
+    writeDefaultConfig,
+    writeDashboardConfig,
+    writeAppConfig,
 
-    removeAppConfigs: removeAppConfigs,
+    removeAppConfigs,
 
     // exported for testing
     _getAcmeApi: getAcmeApi
@@ -57,6 +57,14 @@ var acme2 = require('./cert/acme2.js'),
 var NGINX_APPCONFIG_EJS = fs.readFileSync(__dirname + '/nginxconfig.ejs', { encoding: 'utf8' }),
     RELOAD_NGINX_CMD = path.join(__dirname, 'scripts/reloadnginx.sh');
 
+function nginxLocation(s) {
+    if (!s.startsWith('!')) return s;
+
+    let re = s.replace(/[\^$\\.*+?()[\]{}|]/g, '\\$&'); // https://github.com/es-shims/regexp.escape/blob/master/implementation.js
+
+    return `~ ^(?!(${re.slice(1)}))`; // negative regex assertion - https://stackoverflow.com/questions/16302897/nginx-location-not-equal-to-regex
+}
+
 function getAcmeApi(domainObject, callback) {
     assert.strictEqual(typeof domainObject, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -109,7 +117,7 @@ function providerMatchesSync(domainObject, certFilePath, apiOptions) {
     const domain = subject.substr(subject.indexOf('=') + 1).trim(); // subject can be /CN=, CN=, CN = and other forms
     const issuer = subjectAndIssuer.match(/^issuer=(.*)$/m)[1];
     const isWildcardCert = domain.includes('*');
-    const isLetsEncryptProd = issuer.includes('Let\'s Encrypt Authority');
+    const isLetsEncryptProd = issuer.includes('Let\'s Encrypt');
 
     const issuerMismatch = (apiOptions.prod && !isLetsEncryptProd) || (!apiOptions.prod && isLetsEncryptProd);
     // bare domain is not part of wildcard SAN
@@ -117,7 +125,9 @@ function providerMatchesSync(domainObject, certFilePath, apiOptions) {
 
     const mismatch = issuerMismatch || wildcardMismatch;
 
-    debug(`providerMatchesSync: ${certFilePath} subject=${subject} domain=${domain} issuer=${issuer} wildcard=${isWildcardCert}/${apiOptions.wildcard} prod=${isLetsEncryptProd}/${apiOptions.prod} match=${!mismatch}`);
+    debug(`providerMatchesSync: ${certFilePath} subject=${subject} domain=${domain} issuer=${issuer} `
+        + `wildcard=${isWildcardCert}/${apiOptions.wildcard} prod=${isLetsEncryptProd}/${apiOptions.prod} `
+        + `issuerMismatch=${issuerMismatch} wildcardMismatch=${wildcardMismatch} match=${!mismatch}`);
 
     return !mismatch;
 }
@@ -160,7 +170,7 @@ function validateCertificate(location, domainObject, certificate) {
 }
 
 function reload(callback) {
-    if (process.env.BOX_ENV === 'test') return callback();
+    if (constants.TEST) return callback();
 
     shell.sudo('reload', [ RELOAD_NGINX_CMD ], {}, function (error) {
         if (error) return callback(new BoxError(BoxError.NGINX_ERROR, `Error reloading nginx: ${error.message}`));
@@ -179,14 +189,15 @@ function generateFallbackCertificateSync(domainObject) {
     let opensslConf = safe.fs.readFileSync('/etc/ssl/openssl.cnf', 'utf8');
     // SAN must contain all the domains since CN check is based on implementation if SAN is found. -checkhost also checks only SAN if present!
     let opensslConfWithSan;
-    let cn = domainObject.config.hyphenatedSubdomains ? domains.parentDomain(domain) : domain;
+    let cn = domain;
 
-    debug(`generateFallbackCertificateSync: domain=${domainObject.domain} cn=${cn} hyphenated=${domainObject.config.hyphenatedSubdomains}`);
+    debug(`generateFallbackCertificateSync: domain=${domainObject.domain} cn=${cn}`);
 
     opensslConfWithSan = `${opensslConf}\n[SAN]\nsubjectAltName=DNS:${domain},DNS:*.${cn}\n`;
     let configFile = path.join(os.tmpdir(), 'openssl-' + crypto.randomBytes(4).readUInt32LE(0) + '.conf');
     safe.fs.writeFileSync(configFile, opensslConfWithSan, 'utf8');
-    let certCommand = util.format(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 3650 -subj /CN=*.${cn} -extensions SAN -config ${configFile} -nodes`);
+    // the days field is chosen to be less than 825 days per apple requirement (https://support.apple.com/en-us/HT210176)
+    let certCommand = util.format(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 800 -subj /CN=*.${cn} -extensions SAN -config ${configFile} -nodes`);
     if (!safe.child_process.execSync(certCommand)) return { error: new BoxError(BoxError.OPENSSL_ERROR, safe.error.message) };
     safe.fs.unlinkSync(configFile);
 
@@ -246,22 +257,22 @@ function setAppCertificateSync(location, domainObject, certificate) {
     return null;
 }
 
-function getAcmeCertificate(hostname, domainObject, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+function getAcmeCertificate(vhost, domainObject, callback) {
+    assert.strictEqual(typeof vhost, 'string'); // this can contain wildcard domain (for alias domains)
     assert.strictEqual(typeof domainObject, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     let certFilePath, keyFilePath;
 
-    if (hostname !== domainObject.domain && domainObject.tlsConfig.wildcard) { // bare domain is not part of wildcard SAN
-        let certName = domains.makeWildcard(hostname).replace('*.', '_.');
+    if (vhost !== domainObject.domain && domainObject.tlsConfig.wildcard) { // bare domain is not part of wildcard SAN
+        let certName = domains.makeWildcard(vhost).replace('*.', '_.');
         certFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.cert`);
         keyFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.key`);
 
         if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
     } else {
-        certFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.cert`);
-        keyFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.key`);
+        certFilePath = path.join(paths.APP_CERTS_DIR, `${vhost}.cert`);
+        keyFilePath = path.join(paths.APP_CERTS_DIR, `${vhost}.key`);
 
         if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
     }
@@ -297,17 +308,6 @@ function getCertificate(fqdn, domain, callback) {
     });
 }
 
-function notifyCertChanged(vhost, callback) {
-    assert.strictEqual(typeof vhost, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug(`notifyCertChanged: vhost: ${vhost} mailFqdn: ${settings.mailFqdn()}`);
-
-    if (vhost !== settings.mailFqdn()) return callback();
-
-    mail.handleCertChanged(callback);
-}
-
 function ensureCertificate(vhost, domain, auditSource, callback) {
     assert.strictEqual(typeof vhost, 'string');
     assert.strictEqual(typeof domain, 'string');
@@ -344,7 +344,7 @@ function ensureCertificate(vhost, domain, auditSource, callback) {
                     debug(`ensureCertificate: ${vhost} certificate already exists at ${currentBundle.keyFilePath}`);
 
                     if (!isExpiringSync(currentBundle.certFilePath, 24 * 30) && providerMatchesSync(domainObject, currentBundle.certFilePath, apiOptions)) return callback(null, currentBundle, { renewed: false });
-                    debug(`ensureCertificate: ${vhost} cert require renewal`);
+                    debug(`ensureCertificate: ${vhost} cert requires renewal`);
                 } else {
                     debug(`ensureCertificate: ${vhost} cert does not exist`);
                 }
@@ -361,18 +361,14 @@ function ensureCertificate(vhost, domain, auditSource, callback) {
                         return callback(null, currentBundle, { renewed: false });
                     }
 
-                    notifyCertChanged(vhost, function (error) {
+                    if (certFilePath && keyFilePath) return callback(null, { certFilePath, keyFilePath }, { renewed: true });
+
+                    debug(`ensureCertificate: renewal of ${vhost} failed. using fallback certificates for ${domain}`);
+
+                    getFallbackCertificate(domain, function (error, bundle) {
                         if (error) return callback(error);
 
-                        if (certFilePath && keyFilePath) return callback(null, { certFilePath, keyFilePath }, { renewed: true });
-
-                        debug(`ensureCertificate: renewal of ${vhost} failed. using fallback certificates for ${domain}`);
-
-                        getFallbackCertificate(domain, function (error, bundle) {
-                            if (error) return callback(error);
-
-                            callback(null, bundle, { renewed: false });
-                        });
+                        callback(null, bundle, { renewed: false });
                     });
                 });
             });
@@ -380,7 +376,7 @@ function ensureCertificate(vhost, domain, auditSource, callback) {
     });
 }
 
-function writeAdminNginxConfig(bundle, configFileName, vhost, callback) {
+function writeDashboardNginxConfig(bundle, configFileName, vhost, callback) {
     assert.strictEqual(typeof bundle, 'object');
     assert.strictEqual(typeof configFileName, 'string');
     assert.strictEqual(typeof vhost, 'string');
@@ -389,12 +385,13 @@ function writeAdminNginxConfig(bundle, configFileName, vhost, callback) {
     var data = {
         sourceDir: path.resolve(__dirname, '..'),
         adminOrigin: settings.adminOrigin(),
-        vhost: vhost, // if vhost is empty it will become the default_server
+        vhost: vhost,
         hasIPv6: sysinfo.hasIPv6(),
         endpoint: 'admin',
         certFilePath: bundle.certFilePath,
         keyFilePath: bundle.keyFilePath,
-        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n')
+        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n'),
+        proxyAuth: { enabled: false, id: null, location: nginxLocation('/') }
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
     var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, configFileName);
@@ -417,16 +414,16 @@ function configureAdmin(domain, auditSource, callback) {
         ensureCertificate(adminFqdn, domainObject.domain, auditSource, function (error, bundle) {
             if (error) return callback(error);
 
-            writeAdminNginxConfig(bundle, `${adminFqdn}.conf`, adminFqdn, callback);
+            writeDashboardNginxConfig(bundle, `${adminFqdn}.conf`, adminFqdn, callback);
         });
     });
 }
 
-function writeAdminConfig(domain, callback) {
+function writeDashboardConfig(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`writeAdminConfig: writing admin config for ${domain}`);
+    debug(`writeDashboardConfig: writing admin config for ${domain}`);
 
     domains.get(domain, function (error, domainObject) {
         if (error) return callback(error);
@@ -436,13 +433,14 @@ function writeAdminConfig(domain, callback) {
         getCertificate(adminFqdn, domainObject.domain, function (error, bundle) {
             if (error) return callback(error);
 
-            writeAdminNginxConfig(bundle, `${adminFqdn}.conf`, adminFqdn, callback);
+            writeDashboardNginxConfig(bundle, `${adminFqdn}.conf`, adminFqdn, callback);
         });
     });
 }
 
-function writeAppNginxConfig(app, bundle, callback) {
+function writeAppNginxConfig(app, fqdn, bundle, callback) {
     assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof fqdn, 'string');
     assert.strictEqual(typeof bundle, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -461,20 +459,28 @@ function writeAppNginxConfig(app, bundle, callback) {
     var data = {
         sourceDir: sourceDir,
         adminOrigin: settings.adminOrigin(),
-        vhost: app.fqdn,
+        vhost: fqdn,
         hasIPv6: sysinfo.hasIPv6(),
-        port: app.httpPort,
+        ip: app.containerIp,
+        port: app.manifest.httpPort,
         endpoint: endpoint,
         certFilePath: bundle.certFilePath,
         keyFilePath: bundle.keyFilePath,
         robotsTxtQuoted,
         cspQuoted,
-        hideHeaders
+        hideHeaders,
+        proxyAuth: {
+            enabled: app.sso && app.manifest.addons && app.manifest.addons.proxyAuth,
+            id: app.id,
+            location: nginxLocation(safe.query(app.manifest, 'addons.proxyAuth.path') || '/')
+        },
+        httpPaths: app.manifest.httpPaths || {}
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
 
-    var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, app.id + '.conf');
-    debug('writeAppNginxConfig: writing config for "%s" to %s with options %j', app.fqdn, nginxConfigFilename, data);
+    const aliasSuffix = app.fqdn === fqdn ? '' : `-alias-${fqdn.replace('*', '_')}`;
+    var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, `${app.id}${aliasSuffix}.conf`);
+    debug('writeAppNginxConfig: writing config for "%s" to %s with options %j', fqdn, nginxConfigFilename, data);
 
     if (!safe.fs.writeFileSync(nginxConfigFilename, nginxConf)) {
         debug('Error creating nginx config for "%s" : %s', app.fqdn, safe.error.message);
@@ -500,7 +506,8 @@ function writeAppRedirectNginxConfig(app, fqdn, bundle, callback) {
         keyFilePath: bundle.keyFilePath,
         robotsTxtQuoted: null,
         cspQuoted: null,
-        hideHeaders: []
+        hideHeaders: [],
+        proxyAuth: { enabled: false, id: app.id, location: nginxLocation('/') }
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
 
@@ -520,21 +527,30 @@ function writeAppConfig(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    getCertificate(app.fqdn, app.domain, function (error, bundle) {
-        if (error) return callback(error);
+    let appDomains = [];
+    appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'primary' });
 
-        writeAppNginxConfig(app, bundle, function (error) {
-            if (error) return callback(error);
-
-            async.eachSeries(app.alternateDomains, function (alternateDomain, iteratorDone) {
-                getCertificate(alternateDomain.fqdn, alternateDomain.domain, function (error, bundle) {
-                    if (error) return iteratorDone(error);
-
-                    writeAppRedirectNginxConfig(app, alternateDomain.fqdn, bundle, iteratorDone);
-                });
-            }, callback);
-        });
+    app.alternateDomains.forEach(function (alternateDomain) {
+        appDomains.push({ domain: alternateDomain.domain, fqdn: alternateDomain.fqdn, type: 'alternate' });
     });
+
+    app.aliasDomains.forEach(function (aliasDomain) {
+        appDomains.push({ domain: aliasDomain.domain, fqdn: aliasDomain.fqdn, type: 'alias' });
+    });
+
+    async.eachSeries(appDomains, function (appDomain, iteratorDone) {
+        getCertificate(appDomain.fqdn, appDomain.domain, function (error, bundle) {
+            if (error) return iteratorDone(error);
+
+            if (appDomain.type === 'primary') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alternate') {
+                writeAppRedirectNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alias') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            }
+        });
+    }, callback);
 }
 
 function configureApp(app, auditSource, callback) {
@@ -542,21 +558,30 @@ function configureApp(app, auditSource, callback) {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    ensureCertificate(app.fqdn, app.domain, auditSource, function (error, bundle) {
-        if (error) return callback(error);
+    let appDomains = [];
+    appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'primary' });
 
-        writeAppNginxConfig(app, bundle, function (error) {
-            if (error) return callback(error);
-
-            async.eachSeries(app.alternateDomains, function (alternateDomain, iteratorDone) {
-                ensureCertificate(alternateDomain.fqdn, alternateDomain.domain, auditSource, function (error, bundle) {
-                    if (error) return iteratorDone(error);
-
-                    writeAppRedirectNginxConfig(app, alternateDomain.fqdn, bundle, iteratorDone);
-                });
-            }, callback);
-        });
+    app.alternateDomains.forEach(function (alternateDomain) {
+        appDomains.push({ domain: alternateDomain.domain, fqdn: alternateDomain.fqdn, type: 'alternate' });
     });
+
+    app.aliasDomains.forEach(function (aliasDomain) {
+        appDomains.push({ domain: aliasDomain.domain, fqdn: aliasDomain.fqdn, type: 'alias' });
+    });
+
+    async.eachSeries(appDomains, function (appDomain, iteratorDone) {
+        ensureCertificate(appDomain.fqdn, appDomain.domain, auditSource, function (error, bundle) {
+            if (error) return iteratorDone(error);
+
+            if (appDomain.type === 'primary') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alternate') {
+                writeAppRedirectNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alias') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            }
+        });
+    }, callback);
 }
 
 function unconfigureApp(app, callback) {
@@ -580,20 +605,29 @@ function renewCerts(options, auditSource, progressCallback, callback) {
     apps.getAll(function (error, allApps) {
         if (error) return callback(error);
 
-        var appDomains = [];
+        let appDomains = [];
 
-        // add webadmin domain
-        appDomains.push({ domain: settings.adminDomain(), fqdn: settings.adminFqdn(), type: 'webadmin', nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, `${settings.adminFqdn()}.conf`) });
+        // add webadmin and mail domain
+        if (settings.mailFqdn() === settings.adminFqdn()) {
+            appDomains.push({ domain: settings.adminDomain(), fqdn: settings.adminFqdn(), type: 'webadmin+mail', nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, `${settings.adminFqdn()}.conf`) });
+        } else {
+            appDomains.push({ domain: settings.adminDomain(), fqdn: settings.adminFqdn(), type: 'webadmin', nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, `${settings.adminFqdn()}.conf`) });
+            appDomains.push({ domain: settings.mailDomain(), fqdn: settings.mailFqdn(), type: 'mail' });
+        }
 
-        // add app main
         allApps.forEach(function (app) {
             if (app.runState === apps.RSTATE_STOPPED) return; // do not renew certs of stopped apps
 
-            appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'main', app: app, nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, app.id + '.conf') });
+            appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'primary', app: app, nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, app.id + '.conf') });
 
             app.alternateDomains.forEach(function (alternateDomain) {
-                let nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, `${app.id}-redirect-${alternateDomain.fqdn}.conf`);
-                appDomains.push({ domain: alternateDomain.domain, fqdn: alternateDomain.fqdn, type: 'alternate', app: app, nginxConfigFilename: nginxConfigFilename });
+                const nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, `${app.id}-redirect-${alternateDomain.fqdn}.conf`);
+                appDomains.push({ domain: alternateDomain.domain, fqdn: alternateDomain.fqdn, type: 'alternate', app: app, nginxConfigFilename });
+            });
+
+            app.aliasDomains.forEach(function (aliasDomain) {
+                const nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, `${app.id}-alias-${aliasDomain.fqdn.replace('*', '_')}.conf`);
+                appDomains.push({ domain: aliasDomain.domain, fqdn: aliasDomain.fqdn, type: 'alias', app: app, nginxConfigFilename });
             });
         });
 
@@ -610,6 +644,8 @@ function renewCerts(options, auditSource, progressCallback, callback) {
 
                 if (state.renewed) renewed.push(appDomain.fqdn);
 
+                if (appDomain.type === 'mail') return iteratorCallback(); // mail has no nginx config to check current cert
+
                 // hack to check if the app's cert changed or not. this doesn't handle prod/staging le change since they use same file name
                 let currentNginxConfig = safe.fs.readFileSync(appDomain.nginxConfigFilename, 'utf8') || '';
                 if (currentNginxConfig.includes(bundle.certFilePath)) return iteratorCallback();
@@ -617,13 +653,22 @@ function renewCerts(options, auditSource, progressCallback, callback) {
                 debug(`renewCerts: creating new nginx config since ${appDomain.nginxConfigFilename} does not have ${bundle.certFilePath}`);
 
                 // reconfigure since the cert changed
-                var configureFunc;
-                if (appDomain.type === 'webadmin') configureFunc = writeAdminNginxConfig.bind(null, bundle, `${settings.adminFqdn()}.conf`, settings.adminFqdn());
-                else if (appDomain.type === 'main') configureFunc = writeAppNginxConfig.bind(null, appDomain.app, bundle);
-                else if (appDomain.type === 'alternate') configureFunc = writeAppRedirectNginxConfig.bind(null, appDomain.app, appDomain.fqdn, bundle);
-                else return iteratorCallback(new BoxError(BoxError.INTERNAL_ERROR, `Unknown domain type for ${appDomain.fqdn}. This should never happen`));
+                if (appDomain.type === 'webadmin') {
+                    return writeDashboardNginxConfig(bundle, `${settings.adminFqdn()}.conf`, settings.adminFqdn(), iteratorCallback);
+                } else if (appDomain.type === 'webadmin+mail') {
+                    return async.series([
+                        mail.handleCertChanged,
+                        writeDashboardNginxConfig.bind(null, bundle, `${settings.adminFqdn()}.conf`, settings.adminFqdn())
+                    ], iteratorCallback);
+                } else if (appDomain.type === 'primary') {
+                    return writeAppNginxConfig(appDomain.app, appDomain.fqdn, bundle, iteratorCallback);
+                } else if (appDomain.type === 'alternate') {
+                    return writeAppRedirectNginxConfig(appDomain.app, appDomain.fqdn, bundle, iteratorCallback);
+                } else if (appDomain.type === 'alias') {
+                    return writeAppNginxConfig(appDomain.app, appDomain.fqdn, bundle, iteratorCallback);
+                }
 
-                configureFunc(iteratorCallback);
+                iteratorCallback(new BoxError(BoxError.INTERNAL_ERROR, `Unknown domain type for ${appDomain.fqdn}. This should never happen`));
             });
         }, function (error) {
             if (error) return callback(error);
@@ -631,8 +676,10 @@ function renewCerts(options, auditSource, progressCallback, callback) {
             debug(`renewCerts: Renewed certs of ${JSON.stringify(renewed)}`);
             if (renewed.length === 0) return callback(null);
 
-            // reload nginx if any certs were updated but the config was not rewritten
-            reload(callback);
+            async.series([
+                (next) => { return renewed.includes(settings.mailFqdn()) ? mail.handleCertChanged(next) : next(); },// mail cert renewed
+                reload // reload nginx if any certs were updated but the config was not rewritten
+            ], callback);
         });
     });
 }
@@ -645,27 +692,41 @@ function removeAppConfigs() {
     }
 }
 
-function writeDefaultConfig(callback) {
+function writeDefaultConfig(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    var certFilePath = path.join(paths.NGINX_CERT_DIR,  'default.cert');
-    var keyFilePath = path.join(paths.NGINX_CERT_DIR, 'default.key');
+    const certFilePath = path.join(paths.NGINX_CERT_DIR,  'default.cert');
+    const keyFilePath = path.join(paths.NGINX_CERT_DIR, 'default.key');
 
     if (!fs.existsSync(certFilePath) || !fs.existsSync(keyFilePath)) {
         debug('writeDefaultConfig: create new cert');
 
-        var cn = 'cloudron-' + (new Date()).toISOString(); // randomize date a bit to keep firefox happy
-        if (!safe.child_process.execSync(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 3650 -subj /CN=${cn} -nodes`)) {
+        const cn = 'cloudron-' + (new Date()).toISOString(); // randomize date a bit to keep firefox happy
+        // the days field is chosen to be less than 825 days per apple requirement (https://support.apple.com/en-us/HT210176)
+        if (!safe.child_process.execSync(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 800 -subj /CN=${cn} -nodes`)) {
             debug(`writeDefaultConfig: could not generate certificate: ${safe.error.message}`);
             return callback(new BoxError(BoxError.OPENSSL_ERROR, safe.error));
         }
     }
 
-    writeAdminNginxConfig({ certFilePath, keyFilePath }, constants.NGINX_DEFAULT_CONFIG_FILE_NAME, '', function (error) {
-        if (error) return callback(error);
+    const data = {
+        sourceDir: path.resolve(__dirname, '..'),
+        adminOrigin: settings.adminOrigin(),
+        vhost: '',
+        hasIPv6: sysinfo.hasIPv6(),
+        endpoint: options.activated ? 'ip' : 'setup',
+        certFilePath,
+        keyFilePath,
+        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n'),
+        proxyAuth: { enabled: false, id: null, location: nginxLocation('/') }
+    };
+    const nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
+    const nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, constants.NGINX_DEFAULT_CONFIG_FILE_NAME);
 
-        debug('writeDefaultConfig: done');
+    debug(`writeDefaultConfig: writing configs for endpoint "${data.endpoint}"`);
 
-        callback(null);
-    });
+    if (!safe.fs.writeFileSync(nginxConfigFilename, nginxConf)) return callback(new BoxError(BoxError.FS_ERROR, safe.error));
+
+    reload(callback);
 }

src/routes/accesscontrol.js

@@ -1,11 +1,11 @@
 'use strict';
 
 exports = module.exports = {
-    passwordAuth: passwordAuth,
-    tokenAuth: tokenAuth,
+    passwordAuth,
+    tokenAuth,
 
-    authorize: authorize,
-    websocketAuth: websocketAuth
+    authorize,
+    websocketAuth
 };
 
 var accesscontrol = require('../accesscontrol.js'),
@@ -21,17 +21,17 @@ function passwordAuth(req, res, next) {
 
     if (!req.body.username || typeof req.body.username !== 'string') return next(new HttpError(400, 'A username must be non-empty string'));
     if (!req.body.password || typeof req.body.password !== 'string') return next(new HttpError(400, 'A password must be non-empty string'));
+    if ('totpToken' in req.body && typeof req.body.totpToken !== 'string') return next(new HttpError(400, 'totpToken must be a string' ));
 
-    const username = req.body.username;
-    const password = req.body.password;
+    const { username, password, totpToken } = req.body;
 
     function check2FA(user) {
         assert.strictEqual(typeof user, 'object');
 
         if (!user.ghost && !user.appPassword && user.twoFactorAuthenticationEnabled) {
-            if (!req.body.totpToken) return next(new HttpError(401, 'A totpToken must be provided'));
+            if (!totpToken) return next(new HttpError(401, 'A totpToken must be provided'));
 
-            let verified = speakeasy.totp.verify({ secret: user.twoFactorAuthenticationSecret, encoding: 'base32', token: req.body.totpToken, window: 2 });
+            let verified = speakeasy.totp.verify({ secret: user.twoFactorAuthenticationSecret, encoding: 'base32', token: totpToken, window: 2 });
             if (!verified) return next(new HttpError(401, 'Invalid totpToken'));
         }
 
@@ -99,6 +99,7 @@ function tokenAuth(req, res, next) {
         if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new HttpError(401, 'Unauthorized'));
         if (error) return next(new HttpError(500, error.message));
 
+        req.access_token = token; // used in logout route
         req.user = user;
 
         next();

src/routes/apps.js

@@ -1,49 +1,51 @@
 'use strict';
 
 exports = module.exports = {
-    getApp: getApp,
-    getApps: getApps,
-    getAppIcon: getAppIcon,
-    install: install,
-    uninstall: uninstall,
-    restore: restore,
-    importApp: importApp,
-    backup: backup,
-    update: update,
-    getLogs: getLogs,
-    getLogStream: getLogStream,
-    listBackups: listBackups,
-    repair: repair,
+    getApp,
+    getApps,
+    getAppIcon,
+    install,
+    uninstall,
+    restore,
+    importApp,
+    exportApp,
+    backup,
+    update,
+    getLogs,
+    getLogStream,
+    listBackups,
+    repair,
 
-    setAccessRestriction: setAccessRestriction,
-    setLabel: setLabel,
-    setTags: setTags,
-    setIcon: setIcon,
-    setMemoryLimit: setMemoryLimit,
-    setCpuShares: setCpuShares,
-    setAutomaticBackup: setAutomaticBackup,
-    setAutomaticUpdate: setAutomaticUpdate,
-    setReverseProxyConfig: setReverseProxyConfig,
-    setCertificate: setCertificate,
-    setDebugMode: setDebugMode,
-    setEnvironment: setEnvironment,
-    setMailbox: setMailbox,
-    setLocation: setLocation,
-    setDataDir: setDataDir,
-    setBinds: setBinds,
+    setAccessRestriction,
+    setLabel,
+    setTags,
+    setIcon,
+    setMemoryLimit,
+    setCpuShares,
+    setAutomaticBackup,
+    setAutomaticUpdate,
+    setReverseProxyConfig,
+    setCertificate,
+    setDebugMode,
+    setEnvironment,
+    setMailbox,
+    setLocation,
+    setDataDir,
+    setMounts,
 
-    stop: stop,
-    start: start,
-    restart: restart,
-    exec: exec,
-    execWebSocket: execWebSocket,
+    stop,
+    start,
+    restart,
+    exec,
+    execWebSocket,
 
-    clone: clone,
+    clone,
 
-    uploadFile: uploadFile,
-    downloadFile: downloadFile,
+    uploadFile,
+    downloadFile,
 
-    load: load
+
+    load
 };
 
 var apps = require('../apps.js'),
@@ -138,6 +140,11 @@ function install(req, res, next) {
         if (data.alternateDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'alternateDomains array must contain objects with domain and subdomain strings'));
     }
 
+    if ('aliasDomains' in data) {
+        if (!Array.isArray(data.aliasDomains)) return next(new HttpError(400, 'aliasDomains must be an array'));
+        if (data.aliasDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'aliasDomains array must contain objects with domain and subdomain strings'));
+    }
+
     if ('env' in data) {
         if (!data.env || typeof data.env !== 'object') return next(new HttpError(400, 'env must be an object'));
         if (Object.keys(data.env).some(function (key) { return typeof data.env[key] !== 'string'; })) return next(new HttpError(400, 'env must contain values as strings'));
@@ -352,6 +359,11 @@ function setLocation(req, res, next) {
         if (req.body.alternateDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'alternateDomains array must contain objects with domain and subdomain strings'));
     }
 
+    if ('aliasDomains' in req.body) {
+        if (!Array.isArray(req.body.aliasDomains)) return next(new HttpError(400, 'aliasDomains must be an array'));
+        if (req.body.aliasDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'aliasDomains array must contain objects with domain and subdomain strings'));
+    }
+
     if ('overwriteDns' in req.body && typeof req.body.overwriteDns !== 'boolean') return next(new HttpError(400, 'overwriteDns must be boolean'));
 
     apps.setLocation(req.resource, req.body, auditSource.fromRequest(req), function (error, result) {
@@ -443,6 +455,17 @@ function importApp(req, res, next) {
     });
 }
 
+function exportApp(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+    assert.strictEqual(typeof req.resource, 'object');
+
+    apps.exportApp(req.resource, {}, auditSource.fromRequest(req), function (error, result) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(202, { taskId: result.taskId }));
+    });
+}
+
 function clone(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
     assert.strictEqual(typeof req.resource, 'object');
@@ -597,7 +620,7 @@ function getLogs(req, res, next) {
 
         res.writeHead(200, {
             'Content-Type': 'application/x-logs',
-            'Content-Disposition': 'attachment; filename="log.txt"',
+            'Content-Disposition': `attachment; filename="${req.resource.id}.log"`,
             'Cache-Control': 'no-cache',
             'X-Accel-Buffering': 'no' // disable nginx buffering
         });
@@ -766,22 +789,20 @@ function downloadFile(req, res, next) {
     });
 }
 
-function setBinds(req, res, next) {
+function setMounts(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
     assert.strictEqual(typeof req.resource, 'object');
 
-    if (!req.body.binds || typeof req.body.binds !== 'object') return next(new HttpError(400, 'binds should be an object'));
-
-    for (let name of Object.keys(req.body.binds)) {
-        if (!req.body.binds[name] || typeof req.body.binds[name] !== 'object') return next(new HttpError(400, 'each bind should be an object'));
-        if (typeof req.body.binds[name].hostPath !== 'string') return next(new HttpError(400, 'hostPath must be a string'));
-        if (typeof req.body.binds[name].readOnly !== 'boolean') return next(new HttpError(400, 'readOnly must be a boolean'));
+    if (!Array.isArray(req.body.mounts)) return next(new HttpError(400, 'mounts should be an array'));
+    for (let m of req.body.mounts) {
+        if (!m || typeof m !== 'object') return next(new HttpError(400, 'mounts must be an object'));
+        if (typeof m.volumeId !== 'string') return next(new HttpError(400, 'volumeId must be a string'));
+        if (typeof m.readOnly !== 'boolean') return next(new HttpError(400, 'readOnly must be a boolean'));
     }
 
-    apps.setBinds(req.resource, req.body.binds, auditSource.fromRequest(req), function (error, result) {
+    apps.setMounts(req.resource, req.body.mounts, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
-

src/routes/cloudron.js

@@ -1,30 +1,30 @@
 'use strict';
 
 exports = module.exports = {
-    login: login,
-    logout: logout,
-    passwordResetRequest: passwordResetRequest,
-    passwordReset: passwordReset,
-    setupAccount: setupAccount,
-    reboot: reboot,
-    isRebootRequired: isRebootRequired,
-    getConfig: getConfig,
-    getDisks: getDisks,
-    getMemory: getMemory,
-    getUpdateInfo: getUpdateInfo,
-    update: update,
-    checkForUpdates: checkForUpdates,
-    getLogs: getLogs,
-    getLogStream: getLogStream,
-    setDashboardAndMailDomain: setDashboardAndMailDomain,
-    prepareDashboardDomain: prepareDashboardDomain,
-    renewCerts: renewCerts,
-    getServerIp: getServerIp,
-    syncExternalLdap: syncExternalLdap
+    login,
+    logout,
+    passwordResetRequest,
+    passwordReset,
+    setupAccount,
+    reboot,
+    isRebootRequired,
+    getConfig,
+    getDisks,
+    getMemory,
+    getUpdateInfo,
+    update,
+    checkForUpdates,
+    getLogs,
+    getLogStream,
+    updateDashboardDomain,
+    prepareDashboardDomain,
+    renewCerts,
+    getServerIp,
+    getLanguages,
+    syncExternalLdap
 };
 
 let assert = require('assert'),
-    async = require('async'),
     auditSource = require('../auditsource.js'),
     BoxError = require('../boxerror.js'),
     cloudron = require('../cloudron.js'),
@@ -37,6 +37,7 @@ let assert = require('assert'),
     system = require('../system.js'),
     tokendb = require('../tokendb.js'),
     tokens = require('../tokens.js'),
+    translation = require('../translation.js'),
     updater = require('../updater.js'),
     users = require('../users.js'),
     updateChecker = require('../updatechecker.js');
@@ -63,24 +64,11 @@ function login(req, res, next) {
 }
 
 function logout(req, res) {
-    var token;
+    assert.strictEqual(typeof req.access_token, 'string');
 
-    // this determines the priority
-    if (req.body && req.body.access_token) token = req.body.access_token;
-    if (req.query && req.query.access_token) token = req.query.access_token;
-    if (req.headers && req.headers.authorization) {
-        var parts = req.headers.authorization.split(' ');
-        if (parts.length == 2) {
-            var scheme = parts[0];
-            var credentials = parts[1];
+    eventlog.add(eventlog.ACTION_USER_LOGOUT, auditSource.fromRequest(req), { userId: req.user.id, user: users.removePrivateFields(req.user) });
 
-            if (/^Bearer$/i.test(scheme)) token = credentials;
-        }
-    }
-
-    if (!token) return res.redirect('/login.html');
-
-    tokendb.delByAccessToken(token, function () { res.redirect('/login.html'); });
+    tokendb.delByAccessToken(req.access_token, function () { res.redirect('/login.html'); });
 }
 
 function passwordResetRequest(req, res, next) {
@@ -204,10 +192,7 @@ function checkForUpdates(req, res, next) {
     // it can take a while sometimes to get all the app updates one by one
     req.clearTimeout();
 
-    async.series([
-        (done) => updateChecker.checkAppUpdates({ automatic: false }, done),
-        (done) => updateChecker.checkBoxUpdates({ automatic: false }, done),
-    ], function () {
+    updateChecker.checkForUpdates({ automatic: false }, function () {
         next(new HttpSuccess(200, { update: updateChecker.getUpdateInfo() }));
     });
 }
@@ -229,7 +214,7 @@ function getLogs(req, res, next) {
 
         res.writeHead(200, {
             'Content-Type': 'application/x-logs',
-            'Content-Disposition': 'attachment; filename="log.txt"',
+            'Content-Disposition': `attachment; filename="${req.params.unit}.log"`,
             'Cache-Control': 'no-cache',
             'X-Accel-Buffering': 'no' // disable nginx buffering
         });
@@ -274,10 +259,10 @@ function getLogStream(req, res, next) {
     });
 }
 
-function setDashboardAndMailDomain(req, res, next) {
+function updateDashboardDomain(req, res, next) {
     if (!req.body.domain || typeof req.body.domain !== 'string') return next(new HttpError(400, 'domain must be a string'));
 
-    cloudron.setDashboardAndMailDomain(req.body.domain, auditSource.fromRequest(req), function (error) {
+    cloudron.updateDashboardDomain(req.body.domain, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(204, {}));
@@ -317,3 +302,11 @@ function getServerIp(req, res, next) {
         next(new HttpSuccess(200, { ip }));
     });
 }
+
+function getLanguages(req, res, next) {
+    translation.getLanguages(function (error, languages) {
+        if (error) return next(new BoxError.toHttpError(error));
+
+        next(new HttpSuccess(200, { languages }));
+    });
+}

src/routes/domains.js

@@ -24,7 +24,6 @@ function add(req, res, next) {
     if (typeof req.body.provider !== 'string') return next(new HttpError(400, 'provider must be a string'));
 
     if (!req.body.config || typeof req.body.config !== 'object') return next(new HttpError(400, 'config must be an object'));
-    if ('hyphenatedSubdomains' in req.body.config && typeof req.body.config.hyphenatedSubdomains !== 'boolean') return next(new HttpError(400, 'hyphenatedSubdomains must be a boolean'));
     if ('wildcard' in req.body.config && typeof req.body.config.wildcard !== 'boolean') return next(new HttpError(400, 'wildcard must be a boolean'));
 
     if ('zoneName' in req.body && typeof req.body.zoneName !== 'string') return next(new HttpError(400, 'zoneName must be a string'));
@@ -85,7 +84,6 @@ function update(req, res, next) {
     if (typeof req.body.provider !== 'string') return next(new HttpError(400, 'provider must be an object'));
 
     if (!req.body.config || typeof req.body.config !== 'object') return next(new HttpError(400, 'config must be an object'));
-    if ('hyphenatedSubdomains' in req.body.config && typeof req.body.config.hyphenatedSubdomains !== 'boolean') return next(new HttpError(400, 'hyphenatedSubdomains must be a boolean'));
     if ('wildcard' in req.body.config && typeof req.body.config.wildcard !== 'boolean') return next(new HttpError(400, 'wildcard must be a boolean'));
 
     if ('zoneName' in req.body && typeof req.body.zoneName !== 'string') return next(new HttpError(400, 'zoneName must be a string'));
@@ -101,6 +99,13 @@ function update(req, res, next) {
         if (!req.body.tlsConfig.provider || typeof req.body.tlsConfig.provider !== 'string') return next(new HttpError(400, 'tlsConfig.provider must be a string'));
     }
 
+    if ('wellKnown' in req.body) {
+        if (typeof req.body.wellKnown !== 'object') return next(new HttpError(400, 'wellKnown must be an object'));
+        if (req.body.wellKnown) {
+            if (Object.keys(req.body.wellKnown).some(k => typeof req.body.wellKnown[k] !== 'string')) return next(new HttpError(400, 'wellKnown is a map of strings'));
+        }
+    }
+
     // some DNS providers like DigitalOcean take a really long time to verify credentials (https://github.com/expressjs/timeout/issues/26)
     req.clearTimeout();
 
@@ -109,7 +114,8 @@ function update(req, res, next) {
         provider: req.body.provider,
         config: req.body.config,
         fallbackCertificate: req.body.fallbackCertificate || null,
-        tlsConfig: req.body.tlsConfig || { provider: 'letsencrypt-prod' }
+        tlsConfig: req.body.tlsConfig || { provider: 'letsencrypt-prod' },
+        wellKnown: req.body.wellKnown || null
     };
 
     domains.update(req.params.domain, data, auditSource.fromRequest(req), function (error) {