graphite: carbon crash fix

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923464
https://forum.cloudron.io/topic/4797/graphite-keeps-crashing-oom/34
(cherry picked from commit cd300bb6e2)

CHANGES

@@ -1935,6 +1935,8 @@
 * stopping an app will stop dependent services
 * Add new wasabi s3 storage region us-east-2
 * mail: Fix bug where SRS translation was done on the main domain instead of mailing list domain
+* backups: add retention policy
+* Drop `NET_RAW` caps from container preventing sniffing of network traffic
 
 [5.2.1]
 * Fix app disk graphs
@@ -1949,3 +1951,305 @@
 * import: fix crash because encryption is unset
 * create redis with the correct label
 
+[5.2.3]
+* Do not restart stopped apps
+
+[5.2.4]
+* mail: enable/disable incoming mail was showing an error
+* Do not trigger backup of stopped apps. Instead, we will just retain it's existing backups
+  based on retention policy
+* remove broken disk graphs
+* fix OVH backups
+
+[5.3.0]
+* better nginx config for higher loads
+* backups: add CIFS storage provider
+* backups: add SSHFS storage provider
+* backups: add NFS storage provider
+* s3: use vhost style
+* Fix crash when redis config was set
+* Update schedule was unselected in the UI
+* cloudron-setup: --provider is now optional
+* show warning for unstable updates
+* add forumUrl to app manifest
+* postgresql: add unaccent extension for peertube
+* mail: Add Auto-Submitted header to NDRs
+* backups: ensure that the latest backup of installed apps is always preserved
+* add nginx logs
+* mail: make authentication case insensitive
+* Fix timeout issues in postgresql and mysql addon
+* Do not count stopped apps for memory use
+* LDAP group synchronization
+
+[5.3.1]
+* better nginx config for higher loads
+* backups: add CIFS storage provider
+* backups: add SSHFS storage provider
+* backups: add NFS storage provider
+* s3: use vhost style
+* Fix crash when redis config was set
+* Update schedule was unselected in the UI
+* cloudron-setup: --provider is now optional
+* show warning for unstable updates
+* add forumUrl to app manifest
+* postgresql: add unaccent extension for peertube
+* mail: Add Auto-Submitted header to NDRs
+* backups: ensure that the latest backup of installed apps is always preserved
+* add nginx logs
+* mail: make authentication case insensitive
+* Fix timeout issues in postgresql and mysql addon
+* Do not count stopped apps for memory use
+* LDAP group synchronization
+
+[5.3.2]
+* Do not install sshfs package
+* 'provider' is not required anymore in various API calls
+* redis: Set maxmemory and maxmemory-policy
+* Add mlock capability to manifest (for vault app)
+
+[5.3.3]
+* Fix issue where some postinstall messages where causing angular to infinite loop
+
+[5.3.4]
+* Fix issue in database error handling
+
+[5.4.0]
+* Update nginx to 1.18 for various security fixes
+* Add ping capability (for statping app)
+* Fix bug where aliases were displayed incorrectly in SOGo
+* Add univention as LDAP provider
+* Bump max_connection for postgres addon to 200
+* mail: Add pagination to mailing list API
+* Allow admin to lock email and display name of users
+* Allow admin to ensure all users have 2FA setup
+* ami: fix regression where we didn't send provider as part of get status call
+* nginx: hide version
+* backups: add b2 provider
+* Add filemanager webinterface
+* Add darkmode
+* Add note that password reset and invite links expire in 24 hours
+
+[5.4.1]
+* Update nginx to 1.18 for various security fixes
+* Add ping capability (for statping app)
+* Fix bug where aliases were displayed incorrectly in SOGo
+* Add univention as LDAP provider
+* Bump max_connection for postgres addon to 200
+* mail: Add pagination to mailing list API
+* Allow admin to lock email and display name of users
+* Allow admin to ensure all users have 2FA setup
+* ami: fix regression where we didn't send provider as part of get status call
+* nginx: hide version
+* backups: add b2 provider
+* Add filemanager webinterface
+* Add darkmode
+* Add note that password reset and invite links expire in 24 hours
+
+[5.5.0]
+* postgresql: update to PostgreSQL 11
+* postgresql: add citext extension to whitelist for loomio
+* postgresql: add btree_gist,postgres_fdw,pg_stat_statements,plpgsql extensions for gitlab
+* SFTP/Filebrowser: fix access of external data directories
+* Fix contrast issues in dark mode
+* Add option to delete mailbox data when mailbox is delete
+* Allow days/hours of backups and updates to be configurable
+* backup cleaner: fix issue where referenced backups where not counted against time periods
+* route53: fix issue where verification failed if user had more than 100 zones
+* rework task workers to run them in a separate cgroup
+* backups: now much faster thanks to reworking of task worker
+* When custom fallback cert is set, make sure it's used over LE certs
+* mongodb: update to MongoDB 4.0.19
+* List groups ordered by name
+* Invite links are now valid for a week
+* Update release GPG key
+* Add pre-defined variables ($CLOUDRON_APPID) for better post install messages
+* filemanager: show folder first
+
+[5.6.0]
+* Remove IP nginx configuration that redirects to dashboard after activation
+* dashboard: looks for search string in app title as well
+* Add vaapi caps for transcoding
+* Fix issue where the long mongodb database names where causing app indices of rocket.chat to overflow (> 127)
+* Do not resize swap if swap file exists. This means that users can now control how swap is allocated on their own.
+* SFTP: fix issue where parallel rebuilds would cause an error
+* backups: make part size configurable
+* mail: set max email size
+* mail: allow mail server location to be set
+* spamassassin: custom configs and wl/bl
+* Do not automatically update to unstable release
+* scheduler: reduce container churn
+* mail: add API to set banner
+* Fix bug where systemd 237 ignores --nice value in systemd-run
+* postgresql: enable uuid-ossp extension
+* firewall: add blocklist
+* HTTP URLs now redirect directly to the HTTPS of the final domain
+* linode: Add singapore region
+* ovh: add sydney region
+* s3: makes multi-part copies in parallel
+
+[5.6.1]
+* Blocklists are now stored in a text file instead of json
+* regenerate nginx configs
+
+[5.6.2]
+* Update docker to 19.03.12
+* Fix sorting of user listing in the UI
+* namecheap: fix crash when server returns invalid response
+* unlink ghost file automatically on successful login
+* Bump mysql addon connection limit to 200
+* Fix install issue where `/dev/dri` may not be present
+* import: when importing filesystem backups, the input box is a path
+* firewall: fix race condition where blocklist was not added in correct position in the FORWARD chain
+* services: fix issue where services where scaled up/down too fast
+* turn: realm variable was not updated properly on dashboard change
+* nginx: add splash pages for IP based browser access
+* Give services panel a separate top-level view
+* Add app state filter
+* gcs: copy concurrency was not used
+* Mention why an app update cannot be applied and provide shortcut to start the app if stopped
+* Remove version from footer into the setting view
+* Give services panel a separate top-level view
+* postgresql: set collation order explicity when creating database to C.UTF-8 (for confluence)
+* rsync: fix error while goes missing when syncing
+* Pre-select app domain by default in the redirection drop down
+* robots: preseve leading and trailing whitespaces/newlines
+
+[5.6.3]
+* Fix postgres locale issue
+
+[6.0.0]
+* Focal support
+* Reduce duration of self-signed certs to 800 days
+* Better backup config filename when downloading
+* branding: footer can have template variables like %YEAR% and %VERSION%
+* sftp: secure the API with a token
+* filemanager: Add extract context menu item
+* Do not download docker images if present locally
+* sftp: disable access to non-admins by default
+* postgresql: whitelist pgcrypto extension for loomio
+* filemanager: Add new file creation action and collapse new and upload actions
+* rsync: add warning to remove lifecycle rules
+* Add volume management
+* backups: adjust node's heap size based on memory limit
+* s3: diasble per-chunk timeout
+* logs: more descriptive log file names on download
+* collectd: remove collectd config when app stopped (and add it back when started)
+* Apps can optionally request an authwall to be installed in front of them
+* mailbox can now owned by a group
+* linode: enable dns provider in setup view
+* dns: apps can now use the dns port
+* httpPaths: allow apps to specify forwarding from custom paths to container ports (for OLS)
+* add elasticemail smtp relay option
+* mail: add option to fts using solr
+* mail: change the namespace separator of new installations to /
+* mail: enable acl
+* Disable THP
+* filemanager: allow download dirs as zip files
+* aws: add china region
+* security: fix issue where apps could send with any username (but valid password)
+* i18n support
+
+[6.0.1]
+* app: add export route
+* mail: on location change, fix lock up when one or more domains have invalid credentials
+* mail: fix crash because of write after timeout closure
+* scaleway: fix installation issue where THP is not enabled in kernel
+
+[6.1.0]
+* mail: update haraka to 2.8.27. this fixes zero-length queue file crash
+* update: set/unset appStoreId from the update route
+* proxyauth: Do not follow redirects
+* proxyauth: add 2FA
+* appstore: add category translations
+* appstore: add media category
+* prepend the version to assets when sourcing to avoid cache hits on update
+* filemanger: list volumes of the app
+* Display upload size and size progress
+* nfs: chown the backups for hardlinks to work
+* remove user add/remove/role change email notifications
+* persist update indicator across restarts
+* cloudron-setup: add --generate-setup-token
+* dashboard: pass accessToken query param to automatically login
+* wellknown: add a way to set well known docs
+* oom: notification mails have links to dashboard
+* collectd: do not install xorg* packages
+* apptask: backup/restore tasks now use the backup memory limit configuration
+* eventlog: add logout event
+* mailbox: include alias in mailbox search
+* proxyAuth: add path exclusion
+* turn: fix for CVE-2020-26262
+* app password: fix regression where apps are not listed anymore in the UI
+* Support for multiDomain apps (domain aliases)
+* netcup: add dns provider
+* Container swap size is now dynamically determined based on system RAM/swap ratio
+
+[6.1.1]
+* Fix bug where platform does not start if memory limits could not be applied
+
+[6.1.2]
+* App disk usage was not shown in graphs
+* Email autoconfig
+* Fix SOGo login
+
+[6.2.0]
+* ovh: object storage URL has changed from s3 to storage subdomain
+* ionos: add profit bricks object storage
+* update node to 14.15.4
+* update docker to 20.10.3
+* new base image 3.0.0
+* postgresql updated to 12.5
+* redis updated to 5.0.7
+* dovecot updated to 2.3.7
+* proxyAuth: fix docker UA detection
+* registry config: add UI to disable it
+* update solr to 8.8.1
+* firewall: fix issue where script errored when having more than 15 wl/bl ports
+* If groups are used, do not allow app installation without choosing the access settings
+* tls addon
+* Do not overwrite existing DMARC record
+* Sync dns records
+* Dry run restore
+* linode: show cloudron is installing when user SSHs
+* mysql: disable bin logs
+* Show cancel task button if task is still running after 2 minutes
+* filemanager: fix various bugs involving file names with spaces
+* Change Referrer-policy default to 'same-origin'
+* rsync: preserve and restore symlinks
+* Clean up backups function now removes missing backups
+
+[6.2.1]
+* Avoid updown notifications on full restore
+* Add retries to downloader logic in installer
+
+[6.2.2]
+* Fix ENOBUFS issue with backups when collecting fs metadata
+
+[6.2.3]
+* Fix addon crashes with missing databases
+* Update mail container for LMTP cert fix
+* Fix services view showing yellow icon
+
+[6.2.4]
+* Another addon crash fix
+
+[6.2.5]
+* update: set memory limit properly
+* Fix bug where renew certs button did not work
+* sftp: fix rebuild condition
+* Fix display of user management/dashboard visiblity for email apps
+* graphite: disable tagdb and reduce log noise
+
+[6.2.6]
+* Fix issue where collectd is restarted too quickly before graphite
+
+[6.2.7]
+* collectd: restart on graphite upgrade
+
+[6.2.8]
+* linode object storage: update aws sdk to make it work again
+* Fix crash in blocklist setting when source and list have mixed ip versions
+* mysql: bump connection limit to 200
+* namecheap: fix issue where DNS updates and del were not working
+* turn: turn off verbose logging
+* Fix crash when parsing df output (set LC_ALL for box service)
+

README.md

@@ -29,9 +29,9 @@ anyone to effortlessly host web applications on their server on their own terms.
 * Trivially migrate to another server keeping your apps and data (for example, switch your
   infrastructure provider or move to a bigger server).
 
-* Comprehensive [REST API](https://cloudron.io/documentation/developer/api/).
+* Comprehensive [REST API](https://docs.cloudron.io/api/).
 
-* [CLI](https://cloudron.io/documentation/cli/) to configure apps.
+* [CLI](https://docs.cloudron.io/custom-apps/cli/) to configure apps.
 
 * Alerts, audit logs, graphs, dns management ... and much more
 
@@ -41,15 +41,37 @@ Try our demo at https://my.demo.cloudron.io (username: cloudron password: cloudr
 
 ## Installing
 
-[Install script](https://cloudron.io/documentation/installation/) - [Pricing](https://cloudron.io/pricing.html)
+[Install script](https://docs.cloudron.io/installation/) - [Pricing](https://cloudron.io/pricing.html)
 
 **Note:** This repo is a small part of what gets installed on your server - there is
 the dashboard, database addons, graph container, base image etc. Cloudron also relies
 on external services such as the App Store for apps to be installed. As such, don't
 clone this repo and npm install and expect something to work.
 
+## Development
+
+This is the backend code of Cloudron. The frontend code is [here](https://git.cloudron.io/cloudron/dashboard).
+
+The way to develop is to first install a full instance of Cloudron in a VM. Then you can use the [hotfix](https://git.cloudron.io/cloudron/cloudron-machine)
+tool to patch the VM with the latest code.
+
+```
+SSH_PASSPHRASE=sshkeypassword cloudron-machine hotfix --cloudron my.example.com --release 6.0.0 --ssh-key keyname
+```
+
+## License
+
+Please note that the Cloudron code is under a source-available license. This is not the same as an
+open source license but ensures the code is available for introspection (and hacking!).
+
+## Contributions
+
+Just to give some heads up, we are a bit restrictive in merging changes. We are a small team and
+would like to keep our maintenance burden low. It might be best to discuss features first in the [forum](https://forum.cloudron.io),
+to also figure out how many other people will use it to justify maintenance for a feature.
+
 ## Support
 
-* [Documentation](https://cloudron.io/documentation/)
+* [Documentation](https://docs.cloudron.io/)
 * [Forum](https://forum.cloudron.io/)
 

baseimage/createAMI

@@ -1,193 +0,0 @@
-#!/bin/bash
-
-set -eu -o pipefail
-
-assertNotEmpty() {
-    : "${!1:? "$1 is not set."}"
-}
-
-readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-readonly SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")"/.. && pwd)"
-export JSON="${SOURCE_DIR}/node_modules/.bin/json"
-
-INSTANCE_TYPE="t2.micro"
-BLOCK_DEVICE="DeviceName=/dev/sda1,Ebs={VolumeSize=20,DeleteOnTermination=true,VolumeType=gp2}"
-SSH_KEY_NAME="id_rsa_yellowtent"
-
-revision=$(git rev-parse HEAD)
-ami_name=""
-server_id=""
-server_ip=""
-destroy_server="yes"
-deploy_env="prod"
-image_id=""
-
-args=$(getopt -o "" -l "revision:,name:,no-destroy,env:,region:" -n "$0" -- "$@")
-eval set -- "${args}"
-
-while true; do
-    case "$1" in
-    --env) deploy_env="$2"; shift 2;;
-    --revision) revision="$2"; shift 2;;
-    --name) ami_name="$2"; shift 2;;
-    --no-destroy) destroy_server="no"; shift 2;;
-    --region)
-        case "$2" in
-        "us-east-1")
-            image_id="ami-6edd3078"
-            security_group="sg-a5e17fd9"
-            subnet_id="subnet-b8fbc0f1"
-            ;;
-        "eu-central-1")
-            image_id="ami-5aee2235"
-            security_group="sg-19f5a770" # everything open on eu-central-1
-            subnet_id=""
-            ;;
-        *)
-            echo "Unknown aws region $2"
-            exit 1
-            ;;
-        esac
-        export AWS_DEFAULT_REGION="$2"    # used by the aws cli tool
-        shift 2
-        ;;
-    --) break;;
-    *) echo "Unknown option $1"; exit 1;;
-    esac
-done
-
-# TODO fix this
-export AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY}"
-export AWS_SECRET_ACCESS_KEY="${AWS_ACCESS_SECRET}"
-
-readonly ssh_keys="${HOME}/.ssh/id_rsa_yellowtent"
-readonly SSH="ssh -o IdentitiesOnly=yes -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${ssh_keys}"
-
-if [[ ! -f "${ssh_keys}" ]]; then
-    echo "caas ssh key is missing at ${ssh_keys} (pick it up from secrets repo)"
-    exit 1
-fi
-
-if [[ -z "${image_id}" ]]; then
-    echo "--region is required (us-east-1 or eu-central-1)"
-    exit 1
-fi
-
-function get_pretty_revision() {
-    local git_rev="$1"
-    local sha1=$(git rev-parse --short "${git_rev}" 2>/dev/null)
-
-    echo "${sha1}"
-}
-
-function wait_for_ssh() {
-    echo "=> Waiting for ssh connection"
-    while true; do
-        echo -n "."
-
-        if $SSH ubuntu@${server_ip} echo "hello"; then
-            echo ""
-            break
-        fi
-
-        sleep 5
-    done
-}
-
-now=$(date "+%Y-%m-%d-%H%M%S")
-pretty_revision=$(get_pretty_revision "${revision}")
-
-if [[ -z "${ami_name}" ]]; then
-    ami_name="box-${deploy_env}-${pretty_revision}-${now}"
-fi
-
-echo "=> Create EC2 instance"
-id=$(aws ec2 run-instances --image-id "${image_id}" --instance-type "${INSTANCE_TYPE}" --security-group-ids "${security_group}" --block-device-mappings "${BLOCK_DEVICE}" --key-name "${SSH_KEY_NAME}" --subnet-id "${subnet_id}" --associate-public-ip-address \
-    | $JSON Instances \
-    | $JSON 0.InstanceId)
-
-[[ -z "$id" ]] && exit 1
-echo "Instance created  ID $id"
-
-echo "=> Waiting for instance to get a public IP"
-while true; do
-    server_ip=$(aws ec2 describe-instances --instance-ids ${id} \
-        | $JSON Reservations.0.Instances \
-        | $JSON 0.PublicIpAddress)
-
-    if [[ ! -z "${server_ip}" ]]; then
-        echo ""
-        break
-    fi
-
-    echo -n "."
-    sleep 1
-done
-
-echo "Got public IP ${server_ip}"
-
-wait_for_ssh
-
-echo "=> Fetching cloudron-setup"
-while true; do
-
-    if $SSH ubuntu@${server_ip} wget "https://cloudron.io/cloudron-setup" -O "cloudron-setup"; then
-        echo ""
-        break
-    fi
-
-    echo -n "."
-    sleep 5
-done
-
-echo "=> Running cloudron-setup"
-$SSH ubuntu@${server_ip} sudo /bin/bash "cloudron-setup" --env "${deploy_env}" --provider "ami" --skip-reboot
-
-wait_for_ssh
-
-echo "=> Removing ssh key"
-$SSH ubuntu@${server_ip} sudo rm /home/ubuntu/.ssh/authorized_keys /root/.ssh/authorized_keys
-
-echo "=> Creating AMI"
-image_id=$(aws ec2 create-image --instance-id "${id}" --name "${ami_name}" | $JSON ImageId)
-[[ -z "$id" ]] && exit 1
-echo "Creating AMI with Id ${image_id}"
-
-echo "=> Waiting for AMI to be created"
-while true; do
-    state=$(aws ec2 describe-images --image-ids ${image_id} \
-        | $JSON Images \
-        | $JSON 0.State)
-
-    if [[ "${state}" == "available" ]]; then
-        echo ""
-        break
-    fi
-
-    echo -n "."
-    sleep 5
-done
-
-if [[ "${destroy_server}" == "yes" ]]; then
-    echo "=> Deleting EC2 instance"
-
-    while true; do
-        state=$(aws ec2 terminate-instances --instance-id "${id}" \
-            | $JSON TerminatingInstances \
-            | $JSON 0.CurrentState.Name)
-
-        if [[ "${state}" == "shutting-down" ]]; then
-            echo ""
-            break
-        fi
-
-        echo -n "."
-        sleep 5
-    done
-fi
-
-echo ""
-echo "Done."
-echo ""
-echo "New AMI is: ${image_id}"
-echo ""

baseimage/digitalocean.sh

@@ -1,261 +0,0 @@
-#!/bin/bash
-
-if [[ -z "${DIGITAL_OCEAN_TOKEN}" ]]; then
-    echo "Script requires DIGITAL_OCEAN_TOKEN env to be set"
-    exit 1
-fi
-
-if [[ -z "${JSON}" ]]; then
-    echo "Script requires JSON env to be set to path of JSON binary"
-    exit 1
-fi
-
-readonly CURL="curl --retry 5 -s -u ${DIGITAL_OCEAN_TOKEN}:"
-
-function debug() {
-    echo "$@" >&2
-}
-
-function get_ssh_key_id() {
-    id=$($CURL "https://api.digitalocean.com/v2/account/keys" \
-        | $JSON ssh_keys \
-        | $JSON -c "this.name === \"$1\"" \
-        | $JSON 0.id)
-    [[ -z "$id" ]] && exit 1
-    echo "$id"
-}
-
-function create_droplet() {
-    local ssh_key_id="$1"
-    local box_name="$2"
-
-    local image_region="sfo2"
-    local ubuntu_image_slug="ubuntu-16-04-x64"
-    local box_size="1gb"
-
-    local data="{\"name\":\"${box_name}\",\"size\":\"${box_size}\",\"region\":\"${image_region}\",\"image\":\"${ubuntu_image_slug}\",\"ssh_keys\":[ \"${ssh_key_id}\" ],\"backups\":false}"
-
-    id=$($CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets" | $JSON droplet.id)
-    [[ -z "$id" ]] && exit 1
-    echo "$id"
-}
-
-function get_droplet_ip() {
-    local droplet_id="$1"
-    ip=$($CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}" | $JSON "droplet.networks.v4[0].ip_address")
-    [[ -z "$ip" ]] && exit 1
-    echo "$ip"
-}
-
-function get_droplet_id() {
-    local droplet_name="$1"
-    id=$($CURL "https://api.digitalocean.com/v2/droplets?per_page=200" | $JSON "droplets" | $JSON -c "this.name === '${droplet_name}'" | $JSON "[0].id")
-    [[ -z "$id" ]] && exit 1
-    echo "$id"
-}
-
-function power_off_droplet() {
-    local droplet_id="$1"
-    local data='{"type":"power_off"}'
-    local response=$($CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions")
-    local event_id=`echo "${response}" | $JSON action.id`
-
-    if [[ -z "${event_id}" ]]; then
-        debug "Got no event id, assuming already powered off."
-        debug "Response: ${response}"
-        return
-    fi
-
-    debug "Powered off droplet. Event id: ${event_id}"
-    debug -n "Waiting for droplet to power off"
-
-    while true; do
-        local event_status=`$CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions/${event_id}" | $JSON action.status`
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug ""
-}
-
-function power_on_droplet() {
-    local droplet_id="$1"
-    local data='{"type":"power_on"}'
-    local event_id=`$CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions" | $JSON action.id`
-
-    debug "Powered on droplet. Event id: ${event_id}"
-
-    if [[ -z "${event_id}" ]]; then
-        debug "Got no event id, assuming already powered on"
-        return
-    fi
-
-    debug -n "Waiting for droplet to power on"
-
-    while true; do
-        local event_status=`$CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions/${event_id}" | $JSON action.status`
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug ""
-}
-
-function get_image_id() {
-    local snapshot_name="$1"
-    local image_id=""
-
-    if ! response=$($CURL "https://api.digitalocean.com/v2/images?per_page=200"); then
-        echo "Failed to get image listing. ${response}"
-        return 1
-    fi
-
-    if ! image_id=$(echo "$response" \
-       | $JSON images \
-       | $JSON -c "this.name === \"${snapshot_name}\"" 0.id); then
-        echo "Failed to parse curl response: ${response}"
-        return 1
-    fi
-
-    if [[ -z "${image_id}" ]]; then
-        echo "Failed to get image id of ${snapshot_name}. reponse: ${response}"
-        return 1
-    fi
-
-    echo "${image_id}"
-}
-
-function snapshot_droplet() {
-    local droplet_id="$1"
-    local snapshot_name="$2"
-    local data="{\"type\":\"snapshot\",\"name\":\"${snapshot_name}\"}"
-    local event_id=`$CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions" | $JSON action.id`
-
-    debug "Droplet snapshotted as ${snapshot_name}. Event id: ${event_id}"
-    debug -n "Waiting for snapshot to complete"
-
-    while true; do
-        if ! response=$($CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions/${event_id}"); then
-            echo "Could not get action status. ${response}"
-            continue
-        fi
-        if ! event_status=$(echo "${response}" | $JSON action.status); then
-            echo "Could not parse action.status from response. ${response}"
-            continue
-        fi
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug "! done"
-
-    if ! image_id=$(get_image_id "${snapshot_name}"); then
-        return 1
-    fi
-    echo "${image_id}"
-}
-
-function destroy_droplet() {
-    local droplet_id="$1"
-    # TODO: check for 204 status
-    $CURL -X DELETE "https://api.digitalocean.com/v2/droplets/${droplet_id}"
-    debug "Droplet destroyed"
-    debug ""
-}
-
-function transfer_image() {
-    local image_id="$1"
-    local region_slug="$2"
-    local data="{\"type\":\"transfer\",\"region\":\"${region_slug}\"}"
-    local event_id=`$CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/images/${image_id}/actions" | $JSON action.id`
-    echo "${event_id}"
-}
-
-function wait_for_image_event() {
-    local image_id="$1"
-    local event_id="$2"
-
-    debug -n "Waiting for ${event_id}"
-
-    while true; do
-        local event_status=`$CURL "https://api.digitalocean.com/v2/images/${image_id}/actions/${event_id}" | $JSON action.status`
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug ""
-}
-
-function transfer_image_to_all_regions() {
-    local image_id="$1"
-
-    xfer_events=()
-    image_regions=(ams2) ## sfo1 is where the image is created
-    for image_region in ${image_regions[@]}; do
-        xfer_event=$(transfer_image ${image_id} ${image_region})
-        echo "Image transfer to ${image_region} initiated. Event id: ${xfer_event}"
-        xfer_events+=("${xfer_event}")
-        sleep 1
-    done
-
-    echo "Image transfer initiated, but they will take some time to get transferred."
-
-    for xfer_event in ${xfer_events[@]}; do
-        $vps wait_for_image_event "${image_id}" "${xfer_event}"
-    done
-}
-
-if [[ $# -lt 1 ]]; then
-    debug "<command> <params...>"
-    exit 1
-fi
-
-case $1 in
-get_ssh_key_id)
-    get_ssh_key_id "${@:2}"
-    ;;
-
-create)
-    create_droplet "${@:2}"
-    ;;
-
-get_id)
-    get_droplet_id "${@:2}"
-    ;;
-
-get_ip)
-    get_droplet_ip "${@:2}"
-    ;;
-
-power_on)
-    power_on_droplet "${@:2}"
-    ;;
-
-power_off)
-    power_off_droplet "${@:2}"
-    ;;
-
-snapshot)
-    snapshot_droplet "${@:2}"
-    ;;
-
-destroy)
-    destroy_droplet "${@:2}"
-    ;;
-
-transfer_image_to_all_regions)
-    transfer_image_to_all_regions "${@:2}"
-    ;;
-
-*)
-    echo "Unknown command $1"
-    exit 1
-esac

baseimage/initializeBaseUbuntuImage.sh

@@ -4,8 +4,7 @@ set -euv -o pipefail
 
 readonly SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
-readonly arg_provider="${1:-generic}"
-readonly arg_infraversionpath="${SOURCE_DIR}/${2:-}"
+readonly arg_infraversionpath="${SOURCE_DIR}/../src"
 
 function die {
     echo $1
@@ -14,6 +13,9 @@ function die {
 
 export DEBIAN_FRONTEND=noninteractive
 
+readonly ubuntu_codename=$(lsb_release -cs)
+readonly ubuntu_version=$(lsb_release -rs)
+
 # hold grub since updating it breaks on some VPS providers. also, dist-upgrade will trigger it
 apt-mark hold grub* >/dev/null
 apt-get -o Dpkg::Options::="--force-confdef" update -y
@@ -27,11 +29,12 @@ debconf-set-selections <<< 'mysql-server mysql-server/root_password_again passwo
 
 # this enables automatic security upgrades (https://help.ubuntu.com/community/AutomaticSecurityUpdates)
 # resolvconf is needed for unbound to work property after disabling systemd-resolved in 18.04
-ubuntu_version=$(lsb_release -rs)
-ubuntu_codename=$(lsb_release -cs)
+
 gpg_package=$([[ "${ubuntu_version}" == "16.04" ]] && echo "gnupg" || echo "gpg")
-apt-get -y install \
+mysql_package=$([[ "${ubuntu_version}" == "20.04" ]] && echo "mysql-server-8.0" || echo "mysql-server-5.7")
+apt-get -y install --no-install-recommends \
     acl \
+    apparmor \
     build-essential \
     cifs-utils \
     cron \
@@ -39,11 +42,12 @@ apt-get -y install \
     debconf-utils \
     dmsetup \
     $gpg_package \
+    ipset \
     iptables \
     libpython2.7 \
     linux-generic \
     logrotate \
-    mysql-server-5.7 \
+    $mysql_package \
     openssh-server \
     pwgen \
     resolvconf \
@@ -51,32 +55,29 @@ apt-get -y install \
     tzdata \
     unattended-upgrades \
     unbound \
+    unzip \
     xfsprogs
 
-if [[ "${ubuntu_version}" == "16.04" ]]; then
-    echo "==> installing nginx for xenial for TLSv3 support"
-
-    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.14.0-1~xenial_amd64.deb -o /tmp/nginx.deb
-    # apt install with install deps (as opposed to dpkg -i)
-    apt install -y /tmp/nginx.deb
-    rm /tmp/nginx.deb
-else
-    apt install -y nginx-full
-fi
+echo "==> installing nginx for xenial for TLSv3 support"
+curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-2~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
+# apt install with install deps (as opposed to dpkg -i)
+apt install -y /tmp/nginx.deb
+rm /tmp/nginx.deb
 
 # on some providers like scaleway the sudo file is changed and we want to keep the old one
-apt-get -o Dpkg::Options::="--force-confold" install -y sudo
+apt-get -o Dpkg::Options::="--force-confold" install -y --no-install-recommends sudo
 
 # this ensures that unattended upgades are enabled, if it was disabled during ubuntu install time (see #346)
 # debconf-set-selection of unattended-upgrades/enable_auto_updates + dpkg-reconfigure does not work
 cp /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades
 
 echo "==> Installing node.js"
-mkdir -p /usr/local/node-10.18.1
-curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-10.18.1
-ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
-ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
-apt-get install -y python   # Install python which is required for npm rebuild
+readonly node_version=14.15.4
+mkdir -p /usr/local/node-${node_version}
+curl -sL https://nodejs.org/dist/v${node_version}/node-v${node_version}-linux-x64.tar.gz | tar zxf - --strip-components=1 -C /usr/local/node-${node_version}
+ln -sf /usr/local/node-${node_version}/bin/node /usr/bin/node
+ln -sf /usr/local/node-${node_version}/bin/npm /usr/bin/npm
+apt-get install -y --no-install-recommends python   # Install python which is required for npm rebuild
 [[ "$(python --version 2>&1)" == "Python 2.7."* ]] || die "Expecting python version to be 2.7.x"
 
 # https://docs.docker.com/engine/installation/linux/ubuntulinux/
@@ -87,9 +88,10 @@ mkdir -p /etc/systemd/system/docker.service.d
 echo -e "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2" > /etc/systemd/system/docker.service.d/cloudron.conf
 
 # there are 3 packages for docker - containerd, CLI and the daemon
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.2-3_amd64.deb" -o /tmp/containerd.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+readonly docker_version=20.10.3
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb" -o /tmp/containerd.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
 # apt install with install deps (as opposed to dpkg -i)
 apt install -y /tmp/containerd.deb  /tmp/docker-ce-cli.deb /tmp/docker.deb
 rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
@@ -102,7 +104,7 @@ fi
 
 # do not upgrade grub because it might prompt user and break this script
 echo "==> Enable memory accounting"
-apt-get -y --no-upgrade install grub2-common
+apt-get -y --no-upgrade --no-install-recommends install grub2-common
 sed -e 's/^GRUB_CMDLINE_LINUX="\(.*\)"$/GRUB_CMDLINE_LINUX="\1 cgroup_enable=memory swapaccount=1 panic_on_oops=1 panic=5"/' -i /etc/default/grub
 update-grub
 
@@ -121,26 +123,37 @@ for image in ${images}; do
 done
 
 echo "==> Install collectd"
-if ! apt-get install -y libcurl3-gnutls collectd collectd-utils; then
+# without this, libnotify4 will install gnome-shell
+apt-get install -y libnotify4 --no-install-recommends
+if ! apt-get install -y --no-install-recommends libcurl3-gnutls collectd collectd-utils; then
     # FQDNLookup is true in default debian config. The box code has a custom collectd.conf that fixes this
     echo "Failed to install collectd. Presumably because of http://mailman.verplant.org/pipermail/collectd/2015-March/006491.html"
     sed -e 's/^FQDNLookup true/FQDNLookup false/' -i /etc/collectd/collectd.conf
 fi
+# https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1872281
+[[ "${ubuntu_version}" == "20.04" ]] && echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
 
+# some hosts like atlantic install ntp which conflicts with timedatectl. https://serverfault.com/questions/1024770/ubuntu-20-04-time-sync-problems-and-possibly-incorrect-status-information
 echo "==> Configuring host"
 sed -e 's/^#NTP=/NTP=0.ubuntu.pool.ntp.org 1.ubuntu.pool.ntp.org 2.ubuntu.pool.ntp.org 3.ubuntu.pool.ntp.org/' -i /etc/systemd/timesyncd.conf
+if systemctl is-active ntp; then
+    systemctl stop ntp
+    apt purge -y ntp
+fi
 timedatectl set-ntp 1
 # mysql follows the system timezone
 timedatectl set-timezone UTC
 
 echo "==> Adding sshd configuration warning"
-sed -e '/Port 22/ i # NOTE: Cloudron only supports moving SSH to port 202. See https://cloudron.io/documentation/security/#securing-ssh-access' -i /etc/ssh/sshd_config
+sed -e '/Port 22/ i # NOTE: Cloudron only supports moving SSH to port 202. See https://docs.cloudron.io/security/#securing-ssh-access' -i /etc/ssh/sshd_config
 
 # https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1701068
 echo "==> Disabling motd news"
-sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
+if [ -f "/etc/default/motd-news" ]; then
+    sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
+fi
 
-# Disable bind for good measure (on online.net, kimsufi servers these are pre-installed and conflicts with unbound)
+# Disable bind for good measure (on online.net, kimsufi servers these are pre-installed)
 systemctl stop bind9 || true
 systemctl disable bind9 || true
 
@@ -152,7 +165,7 @@ systemctl disable dnsmasq || true
 systemctl stop postfix || true
 systemctl disable postfix || true
 
-# on ubuntu 18.04, this is the default. this requires resolvconf for DNS to work further after the disable
+# on ubuntu 18.04 and 20.04, this is the default. this requires resolvconf for DNS to work further after the disable
 systemctl stop systemd-resolved || true
 systemctl disable systemd-resolved || true
 
@@ -161,4 +174,3 @@ systemctl disable systemd-resolved || true
 ip6=$([[ -s /proc/net/if_inet6 ]] && echo "yes" || echo "no")
 echo -e "server:\n\tinterface: 127.0.0.1\n\tdo-ip6: ${ip6}" > /etc/unbound/unbound.conf.d/cloudron-network.conf
 systemctl restart unbound
-

box.js

@@ -2,57 +2,65 @@
 
 'use strict';
 
-// prefix all output with a timestamp
-// debug() already prefixes and uses process.stderr NOT console.*
-['log', 'info', 'warn', 'debug', 'error'].forEach(function (log) {
-    var orig = console[log];
-    console[log] = function () {
-        orig.apply(console, [new Date().toISOString()].concat(Array.prototype.slice.call(arguments)));
-    };
-});
-
-require('supererror')({ splatchError: true });
-
 let async = require('async'),
-    constants = require('./src/constants.js'),
     dockerProxy = require('./src/dockerproxy.js'),
+    fs = require('fs'),
     ldap = require('./src/ldap.js'),
+    paths = require('./src/paths.js'),
+    proxyAuth = require('./src/proxyauth.js'),
     server = require('./src/server.js');
 
-console.log();
-console.log('==========================================');
-console.log(`           Cloudron ${constants.VERSION}  `);
-console.log('==========================================');
-console.log();
+const NOOP_CALLBACK = function () { };
+
+function setupLogging(callback) {
+    if (process.env.BOX_ENV === 'test') return callback();
+
+    var logfileStream = fs.createWriteStream(paths.BOX_LOG_FILE, { flags:'a' });
+    process.stdout.write = process.stderr.write = logfileStream.write.bind(logfileStream);
+
+    callback();
+}
 
 async.series([
-    server.start,
+    setupLogging,
+    server.start, // do this first since it also inits the database
+    proxyAuth.start,
     ldap.start,
     dockerProxy.start
 ], function (error) {
     if (error) {
-        console.error('Error starting server', error);
+        console.log('Error starting server', error);
         process.exit(1);
     }
-    console.log('Cloudron is up and running');
-});
-
-var NOOP_CALLBACK = function () { };
-
-process.on('SIGINT', function () {
-    console.log('Received SIGINT. Shutting down.');
-
-    server.stop(NOOP_CALLBACK);
-    ldap.stop(NOOP_CALLBACK);
-    dockerProxy.stop(NOOP_CALLBACK);
-    setTimeout(process.exit.bind(process), 3000);
-});
-
-process.on('SIGTERM', function () {
-    console.log('Received SIGTERM. Shutting down.');
-
-    server.stop(NOOP_CALLBACK);
-    ldap.stop(NOOP_CALLBACK);
-    dockerProxy.stop(NOOP_CALLBACK);
-    setTimeout(process.exit.bind(process), 3000);
+
+    // require those here so that logging handler is already setup
+    require('supererror');
+    const debug = require('debug')('box:box');
+
+    process.on('SIGINT', function () {
+        debug('Received SIGINT. Shutting down.');
+
+        proxyAuth.stop(NOOP_CALLBACK);
+        server.stop(NOOP_CALLBACK);
+        ldap.stop(NOOP_CALLBACK);
+        dockerProxy.stop(NOOP_CALLBACK);
+        setTimeout(process.exit.bind(process), 3000);
+    });
+
+    process.on('SIGTERM', function () {
+        debug('Received SIGTERM. Shutting down.');
+
+        proxyAuth.stop(NOOP_CALLBACK);
+        server.stop(NOOP_CALLBACK);
+        ldap.stop(NOOP_CALLBACK);
+        dockerProxy.stop(NOOP_CALLBACK);
+        setTimeout(process.exit.bind(process), 3000);
+    });
+
+    process.on('uncaughtException', function (error) {
+        console.error((error && error.stack) ? error.stack : error);
+        setTimeout(process.exit.bind(process, 1), 3000);
+    });
+
+    console.log(`Cloudron is up and running. Logs are at ${paths.BOX_LOG_FILE}`); // this goes to journalctl
 });

migrations/20160208164735-groups-add-admin.js

@@ -12,8 +12,6 @@ exports.up = function(db, callback) {
 			db.all('SELECT * FROM users WHERE admin=1', function (error, results) {
 				if (error) return done(error);
 
-				console.dir(results);
-
 				async.eachSeries(results, function (r, next) {
 					db.runSql('INSERT INTO groupMembers (groupId, userId) VALUES (?, ?)', [ ADMIN_GROUP_ID, r.id ], next);
 				}, done);

migrations/20200528003052-settings-backup-s3-path-style.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        if (backupConfig.provider !== 'minio' && backupConfig.provider !== 's3-v4-compat') return callback();
+
+        backupConfig.s3ForcePathStyle = true; // usually minio is self-hosted. s3 v4 compat, we don't know
+
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200530194049-appPasswords-change-unique-constraint.js

@@ -0,0 +1,17 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+
+    // http://stackoverflow.com/questions/386294/what-is-the-maximum-length-of-a-valid-email-address
+
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE appPasswords DROP INDEX name'),
+        db.runSql.bind(db, 'ALTER TABLE appPasswords ADD CONSTRAINT appPasswords_name_userId_identifier UNIQUE (name, userId, identifier)'),
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200604111448-userGroups-add-source.js

@@ -0,0 +1,17 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE userGroups ADD COLUMN source VARCHAR(128) DEFAULT ""', function (error) {
+        if (error) return callback(error);
+
+        callback();
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE userGroups DROP COLUMN source', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20200614181936-backups-add-identifier.js

@@ -0,0 +1,38 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE backups ADD COLUMN identifier VARCHAR(128)', function (error) {
+        if (error) return callback(error);
+
+
+        db.all('SELECT * FROM backups', function (error, backups) {
+            if (error) return callback(error);
+
+            async.eachSeries(backups, function (backup, next) {
+                let identifier = 'unknown';
+
+                if (backup.type === 'box') {
+                    identifier = 'box';
+                } else {
+                    const match = backup.id.match(/app_(.+?)_.+/);
+                    if (match) identifier = match[1];
+                }
+
+                db.runSql('UPDATE backups SET identifier=? WHERE id=?', [ identifier, backup.id ], next);
+            }, function (error) {
+                if (error) return callback(error);
+
+                db.runSql('ALTER TABLE backups MODIFY COLUMN identifier VARCHAR(128) NOT NULL', callback);
+            });
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE backups DROP COLUMN identifier', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200709224148-users-rename-modifiedAt-to-ts.js

@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE users ADD COLUMN ts TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP', function (error) {
+        if (error) console.error(error);
+
+        db.runSql('ALTER TABLE users DROP COLUMN modifiedAt', callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE users DROP COLUMN ts', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200728232132-settings-set-backups-schedulePattern.js

@@ -0,0 +1,29 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        if (backupConfig.intervalSecs === 6 * 60 * 60) { // every 6 hours
+            backupConfig.schedulePattern = '00 00 5,11,17,23 * * *';
+        } else if (backupConfig.intervalSecs === 12 * 60 * 60) { // every 12 hours
+            backupConfig.schedulePattern = '00 00 5,17 * * *';
+        } else if (backupConfig.intervalSecs === 24 * 60 * 60) { // every day
+            backupConfig.schedulePattern = '00 00 23 * * *';
+        } else if (backupConfig.intervalSecs === 3 * 24 * 60 * 60) { // every 3 days (based on day)
+            backupConfig.schedulePattern = '00 00 23 * * 1,3,5';
+        } else if (backupConfig.intervalSecs === 7 * 24 * 60 * 60) { // every week (saturday)
+            backupConfig.schedulePattern = '00 00 23 * * 6';
+        } else { // default to everyday
+            backupConfig.schedulePattern = '00 00 23 * * *';
+        }
+
+        delete backupConfig.intervalSecs;
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200816002703-settings-add-mail-domain-and-fqdn.js

@@ -0,0 +1,23 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="admin_domain"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        const adminDomain = results[0].value;
+
+        async.series([
+            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_domain', adminDomain ]),
+            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_fqdn', `my.${adminDomain}` ])
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_domain"'),
+        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_fqdn"'),
+    ], callback);
+};

migrations/20200820050259-settings-merge-autoupdate-pattern.js

@@ -0,0 +1,22 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    db.runSql('SELECT * FROM settings WHERE name=?', ['app_autoupdate_pattern'], function (error, results) {
+        if (error || results.length === 0) return callback(error); // will use defaults from box code
+
+        var updatePattern = results[0].value; // use app auto update patter for the box as well
+
+        async.series([
+            db.runSql.bind(db, 'START TRANSACTION;'),
+            db.runSql.bind(db, 'DELETE FROM settings WHERE name=? OR name=?', ['app_autoupdate_pattern', 'box_autoupdate_pattern']),
+            db.runSql.bind(db, 'INSERT settings (name, value) VALUES(?, ?)', ['autoupdate_pattern', updatePattern]),
+            db.runSql.bind(db, 'COMMIT')
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200824004859-mail-add-bannerJson.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mail ADD COLUMN bannerJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mail DROP COLUMN bannerJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200914172345-split-firewall-config.js

@@ -0,0 +1,27 @@
+'use strict';
+
+const OLD_FIREWALL_CONFIG_JSON = '/home/yellowtent/boxdata/firewall-config.json';
+const PORTS_FILE = '/home/yellowtent/boxdata/firewall/ports.json';
+const BLOCKLIST_FILE = '/home/yellowtent/boxdata/firewall/blocklist.txt';
+
+const fs = require('fs');
+
+exports.up = function (db, callback) {
+    if (!fs.existsSync(OLD_FIREWALL_CONFIG_JSON)) return callback();
+
+    try {
+        const dataJson = fs.readFileSync(OLD_FIREWALL_CONFIG_JSON, 'utf8');
+        const data = JSON.parse(dataJson);
+        fs.writeFileSync(BLOCKLIST_FILE, data.blocklist.join('\n') + '\n', 'utf8');
+        fs.writeFileSync(PORTS_FILE, JSON.stringify({ allowed_tcp_ports: data.allowed_tcp_ports }, null, 4), 'utf8');
+        fs.unlinkSync(OLD_FIREWALL_CONFIG_JSON);
+    } catch (error) {
+        console.log('Error migrating old firewall config', error);
+    }
+
+    callback();
+};
+
+exports.down = function (db, callback) {
+    callback();
+};

migrations/20201028052525-volumes-create-table.js

@@ -0,0 +1,40 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    var cmd1 = 'CREATE TABLE volumes(' +
+                'id VARCHAR(128) NOT NULL UNIQUE,' +
+                'name VARCHAR(256) NOT NULL UNIQUE,' +
+                'hostPath VARCHAR(1024) NOT NULL UNIQUE,' +
+                'creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,' +
+                'PRIMARY KEY (id)) CHARACTER SET utf8 COLLATE utf8_bin';
+
+    var cmd2 = 'CREATE TABLE appMounts(' +
+                 'appId VARCHAR(128) NOT NULL,' +
+                 'volumeId VARCHAR(128) NOT NULL,' +
+                 'readOnly BOOLEAN DEFAULT 1,' +
+                 'UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),' +
+                 'FOREIGN KEY(appId) REFERENCES apps(id),' +
+                 'FOREIGN KEY(volumeId) REFERENCES volumes(id)) CHARACTER SET utf8 COLLATE utf8_bin;';
+
+    db.runSql(cmd1, function (error) {
+        if (error) console.error(error);
+
+        db.runSql(cmd2, function (error) {
+            if (error) console.error(error);
+
+            db.runSql('ALTER TABLE apps DROP COLUMN bindsJson', callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('DROP TABLE appMounts', function (error) {
+        if (error) console.error(error);
+
+        db.runSql('DROP TABLE volumes', function (error) {
+            if (error) console.error(error);
+            callback(error);
+        });
+    });
+};
+

migrations/20201110174504-apps-add-proxyAuth.js

@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN proxyAuth BOOLEAN DEFAULT 0', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN proxyAuth', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20201113071728-mailboxes-add-ownerType.js

@@ -0,0 +1,18 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD COLUMN ownerType VARCHAR(16)'),
+        db.runSql.bind(db, 'UPDATE mailboxes SET ownerType=?', [ 'user' ]),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes MODIFY ownerType VARCHAR(16) NOT NULL'),
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes DROP COLUMN ownerType', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20201119072309-apps-remove-httpPort.js

@@ -0,0 +1,13 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE apps DROP COLUMN httpPort')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20201120212726-apps-add-containerIp.js

@@ -0,0 +1,29 @@
+'use strict';
+
+const async = require('async'),
+    iputils = require('../src/iputils.js');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN containerIp VARCHAR(16) UNIQUE', function (error) {
+        if (error) console.error(error);
+
+        let baseIp = iputils.intFromIp('172.18.16.0');
+
+        db.all('SELECT * FROM apps', function (error, apps) {
+            if (error) return callback(error);
+
+            async.eachSeries(apps, function (app, iteratorDone) {
+                const nextIp = iputils.ipFromInt(++baseIp);
+                db.runSql('UPDATE apps SET containerIp=? WHERE id=?', [ nextIp, app.id ], iteratorDone);
+            }, callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN containerIp', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20201204081504-settings-sftp-requireAdmin.js

@@ -0,0 +1,21 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM settings WHERE name=?', ['platform_config'], function (error, results) {
+        let value;
+        if (error || results.length === 0) {
+            value = { sftp: { requireAdmin: true } };
+        } else {
+            value = JSON.parse(results[0].value);
+            if (!value.sftp) value.sftp = {};
+            value.sftp.requireAdmin = true;
+        }
+
+        // existing installations may not even have the key. so use REPLACE instead of UPDATE
+        db.runSql('REPLACE INTO settings (name, value) VALUES (?, ?)', [ 'platform_config', JSON.stringify(value) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20201222182945-groupMembers-unique-members.js

@@ -0,0 +1,18 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'CREATE TABLE groupMembers_copy(groupId VARCHAR(128) NOT NULL, userId VARCHAR(128) NOT NULL, FOREIGN KEY(groupId) REFERENCES userGroups(id), FOREIGN KEY(userId) REFERENCES users(id), UNIQUE (groupId, userId)) CHARACTER SET utf8 COLLATE utf8_bin'),    // In mysql CREATE TABLE.. LIKE does not copy indexes
+        db.runSql.bind(db, 'INSERT INTO groupMembers_copy SELECT * FROM groupMembers GROUP BY groupId, userId'),
+        db.runSql.bind(db, 'DROP TABLE groupMembers'),
+        db.runSql.bind(db, 'ALTER TABLE groupMembers_copy RENAME TO groupMembers')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE groupMembers DROP INDEX groupMembers_member'),
+    ], callback);
+};

migrations/20201223014453-domains-add-wellKnownJson.js

@@ -0,0 +1,51 @@
+'use strict';
+
+const async = require('async'),
+    safe = require('safetydance');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE domains ADD COLUMN wellKnownJson TEXT', function (error) {
+        if (error) return callback(error);
+
+        // keep the paths around, so that we don't need to trigger a re-configure. the old nginx config will use the paths
+        // the new one will proxy calls to the box code
+        const WELLKNOWN_DIR = '/home/yellowtent/boxdata/well-known';
+        const output = safe.child_process.execSync('find . -type f -printf "%P\n"', { cwd: WELLKNOWN_DIR, encoding: 'utf8' });
+        if (!output) return callback();
+        const paths = output.trim().split('\n');
+        if (paths.length === 0) return callback(); // user didn't configure any well-known
+
+        let wellKnown = {};
+        for (let path of paths) {
+            const fqdn = path.split('/', 1)[0];
+            const loc = path.slice(fqdn.length+1);
+            const doc = safe.fs.readFileSync(`${WELLKNOWN_DIR}/${path}`, { encoding: 'utf8' });
+            if (!doc) continue;
+
+            wellKnown[fqdn] = {};
+            wellKnown[fqdn][loc] = doc;
+        }
+
+        console.log('Migrating well-known', JSON.stringify(wellKnown, null, 4));
+
+        async.eachSeries(Object.keys(wellKnown), function (fqdn, iteratorDone) {
+            db.runSql('UPDATE domains SET wellKnownJson=? WHERE domain=?', [ JSON.stringify(wellKnown[fqdn]), fqdn ], function (error, result) {
+                if (error) {
+                    console.error(error); // maybe the domain does not exist anymore
+                } else if (result.affectedRows === 0) {
+                    console.log(`Could not migrate wellknown as domain ${fqdn} is missing`);
+                }
+                iteratorDone();
+            });
+        }, function (error) {
+            callback(error);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE domains DROP COLUMN wellKnownJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20210120025243-settings-platform-config-memoryLimit.js

@@ -0,0 +1,23 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM settings WHERE name=?', ['platform_config'], function (error, results) {
+        if (error || results.length === 0) return callback(null);
+
+        let value = JSON.parse(results[0].value);
+
+        for (const serviceName of Object.keys(value)) {
+            const service = value[serviceName];
+            if (!service.memorySwap) continue;
+            service.memoryLimit = service.memorySwap;
+            delete service.memorySwap;
+            delete service.memory;
+        }
+
+        db.runSql('UPDATE settings SET value=? WHERE name=?', [ JSON.stringify(value), 'platform_config' ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210120025750-apps-servicesConfig-memoryLimit.js

@@ -0,0 +1,28 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM apps', function (error, apps) {
+        if (error) return callback(error);
+
+        async.eachSeries(apps, function (app, iteratorDone) {
+            if (!app.servicesConfigJson) return iteratorDone();
+
+            let servicesConfig = JSON.parse(app.servicesConfigJson);
+            for (const serviceName of Object.keys(servicesConfig)) {
+                const service = servicesConfig[serviceName];
+                if (!service.memorySwap) continue;
+                service.memoryLimit = service.memorySwap;
+                delete service.memorySwap;
+                delete service.memory;
+            }
+
+            db.runSql('UPDATE apps SET servicesConfigJson=? WHERE id=?', [ JSON.stringify(servicesConfig), app.id ], iteratorDone);
+        }, callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210121201822-settings-rename-platform-config-to-services-config.js

@@ -0,0 +1,9 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('UPDATE settings SET name=? WHERE name=?', [ 'services_config', 'platform_config' ], callback);
+};
+
+exports.down = function(db, callback) {
+    db.runSql('UPDATE settings SET name=? WHERE name=?', [ 'platform_config', 'services_config' ], callback);
+};

migrations/20210204181904-settings-change-ovh-url.js

@@ -0,0 +1,10 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    /* this contained an invalid migration of OVH URLs from s3 subdomain to storage subdomain. See https://forum.cloudron.io/topic/4584/issue-with-backups-listings-and-saving-backup-config-in-6-2  */
+    callback();
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210209223121-settings-add-registry-config-provider.js

@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="registry_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var registryConfig = JSON.parse(results[0].value);
+        if (!registryConfig.provider) registryConfig.provider = 'other';
+
+        db.runSql('UPDATE settings SET value=? WHERE name="registry_config"', [ JSON.stringify(registryConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/schema.sql

@@ -21,7 +21,7 @@ CREATE TABLE IF NOT EXISTS users(
     password VARCHAR(1024) NOT NULL,
     salt VARCHAR(512) NOT NULL,
     createdAt VARCHAR(512) NOT NULL,
-    modifiedAt VARCHAR(512) NOT NULL,
+    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
     displayName VARCHAR(512) DEFAULT "",
     fallbackEmail VARCHAR(512) DEFAULT "",
     twoFactorAuthenticationSecret VARCHAR(128) DEFAULT "",
@@ -37,13 +37,15 @@ CREATE TABLE IF NOT EXISTS users(
 CREATE TABLE IF NOT EXISTS userGroups(
     id VARCHAR(128) NOT NULL UNIQUE,
     name VARCHAR(254) NOT NULL UNIQUE,
+    source VARCHAR(128) DEFAULT "",
     PRIMARY KEY(id));
 
 CREATE TABLE IF NOT EXISTS groupMembers(
     groupId VARCHAR(128) NOT NULL,
     userId VARCHAR(128) NOT NULL,
     FOREIGN KEY(groupId) REFERENCES userGroups(id),
-    FOREIGN KEY(userId) REFERENCES users(id));
+    FOREIGN KEY(userId) REFERENCES users(id),
+    UNIQUE (groupId, userId));
 
 CREATE TABLE IF NOT EXISTS tokens(
     id VARCHAR(128) NOT NULL UNIQUE,
@@ -64,9 +66,6 @@ CREATE TABLE IF NOT EXISTS apps(
     healthTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app last responded
     containerId VARCHAR(128),
     manifestJson TEXT,
-    httpPort INTEGER,                        // this is the nginx proxy port and not manifest.httpPort
-    location VARCHAR(128) NOT NULL,
-    domain VARCHAR(128) NOT NULL,
     accessRestrictionJson TEXT, // { users: [ ], groups: [ ] }
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app was installed
     updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,   // when the last app update was done
@@ -86,7 +85,8 @@ CREATE TABLE IF NOT EXISTS apps(
     dataDir VARCHAR(256) UNIQUE,
     taskId INTEGER, // current task
     errorJson TEXT,
-    bindsJson TEXT, // bind mounts
+    servicesConfigJson TEXT, // app services configuration
+    containerIp VARCHAR(16) UNIQUE, // this is not-null because of ip allocation fails, user can 'repair'
 
     FOREIGN KEY(mailboxDomain) REFERENCES domains(domain),
     FOREIGN KEY(taskId) REFERENCES tasks(id),
@@ -124,6 +124,7 @@ CREATE TABLE IF NOT EXISTS backups(
     packageVersion VARCHAR(128) NOT NULL, /* app version or box version */
     encryptionVersion INTEGER, /* when null, unencrypted backup */
     type VARCHAR(16) NOT NULL, /* 'box' or 'app' */
+    identifier VARCHAR(128) NOT NULL, /* 'box' or the app id */
     dependsOn TEXT, /* comma separate list of objects this backup depends on */
     state VARCHAR(16) NOT NULL,
     manifestJson TEXT, /* to validate if the app can be installed in this version of box */
@@ -147,6 +148,7 @@ CREATE TABLE IF NOT EXISTS domains(
     provider VARCHAR(16) NOT NULL,
     configJson TEXT, /* JSON containing the dns backend provider config */
     tlsConfigJson TEXT, /* JSON containing the tls provider config */
+    wellKnownJson TEXT, /* JSON containing well known docs for this domain */
 
     PRIMARY KEY (domain))
 
@@ -160,6 +162,7 @@ CREATE TABLE IF NOT EXISTS mail(
     mailFromValidation BOOLEAN DEFAULT 1,
     catchAllJson TEXT,
     relayJson TEXT,
+    bannerJson TEXT,
 
     dkimSelector VARCHAR(128) NOT NULL DEFAULT "cloudron",
 
@@ -179,6 +182,7 @@ CREATE TABLE IF NOT EXISTS mailboxes(
     name VARCHAR(128) NOT NULL,
     type VARCHAR(16) NOT NULL, /* 'mailbox', 'alias', 'list' */
     ownerId VARCHAR(128) NOT NULL, /* user id */
+    ownerType VARCHAR(16) NOT NULL,
     aliasName VARCHAR(128), /* the target name type is an alias */
     aliasDomain VARCHAR(128), /* the target domain */
     membersJson TEXT, /* members of a group. fully qualified */
@@ -219,7 +223,7 @@ CREATE TABLE IF NOT EXISTS notifications(
     message TEXT,
     acknowledged BOOLEAN DEFAULT false,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-
+    UNIQUE KEY appPasswords_name_appId_identifier (name, userId, identifier),
     PRIMARY KEY (id)
 );
 
@@ -231,8 +235,24 @@ CREATE TABLE IF NOT EXISTS appPasswords(
     hashedPassword VARCHAR(1024) NOT NULL,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     FOREIGN KEY(userId) REFERENCES users(id),
-    UNIQUE (name, userId),
+
     PRIMARY KEY (id)
 );
 
+CREATE TABLE IF NOT EXISTS volumes(
+    id VARCHAR(128) NOT NULL UNIQUE,
+    name VARCHAR(256) NOT NULL UNIQUE,
+    hostPath VARCHAR(1024) NOT NULL UNIQUE,
+    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (id)
+);
+
+CREATE TABLE IF NOT EXISTS appMounts(
+    appId VARCHAR(128) NOT NULL,
+    volumeId VARCHAR(128) NOT NULL,
+    readOnly BOOLEAN DEFAULT 1,
+    UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),
+    FOREIGN KEY(appId) REFERENCES apps(id),
+    FOREIGN KEY(volumeId) REFERENCES volumes(id));
+
 CHARACTER SET utf8 COLLATE utf8_bin;

package.json

@@ -10,76 +10,77 @@
     "type": "git",
     "url": "https://git.cloudron.io/cloudron/box.git"
   },
-  "engines": {
-    "node": ">=4.0.0 <=4.1.1"
-  },
   "dependencies": {
-    "@google-cloud/dns": "^1.1.0",
-    "@google-cloud/storage": "^2.5.0",
+    "@google-cloud/dns": "^2.1.0",
+    "@google-cloud/storage": "^5.8.3",
     "@sindresorhus/df": "git+https://github.com/cloudron-io/df.git#type",
-    "async": "^2.6.3",
-    "aws-sdk": "^2.610.0",
+    "async": "^3.2.0",
+    "aws-sdk": "^2.879.0",
+    "basic-auth": "^2.0.1",
     "body-parser": "^1.19.0",
-    "cloudron-manifestformat": "^5.1.1",
+    "cloudron-manifestformat": "^5.10.1",
     "connect": "^3.7.0",
-    "connect-lastmile": "^1.2.2",
+    "connect-lastmile": "^2.0.0",
     "connect-timeout": "^1.9.0",
+    "cookie-parser": "^1.4.5",
     "cookie-session": "^1.4.0",
     "cron": "^1.8.2",
-    "db-migrate": "^0.11.6",
-    "db-migrate-mysql": "^1.1.10",
-    "debug": "^4.1.1",
-    "dockerode": "^2.5.8",
-    "ejs": "^2.6.1",
-    "ejs-cli": "^2.1.1",
+    "db-migrate": "^0.11.12",
+    "db-migrate-mysql": "^2.1.2",
+    "debug": "^4.3.1",
+    "dockerode": "^3.2.1",
+    "ejs": "^3.1.6",
+    "ejs-cli": "^2.2.1",
     "express": "^4.17.1",
-    "js-yaml": "^3.13.1",
-    "json": "^9.0.6",
-    "ldapjs": "^1.0.2",
-    "lodash": "^4.17.15",
+    "ipaddr.js": "^2.0.0",
+    "js-yaml": "^4.0.0",
+    "json": "^10.0.0",
+    "jsonwebtoken": "^8.5.1",
+    "ldapjs": "^2.2.4",
+    "lodash": "^4.17.21",
     "lodash.chunk": "^4.2.0",
-    "mime": "^2.4.4",
-    "moment": "^2.25.3",
-    "moment-timezone": "^0.5.27",
-    "morgan": "^1.9.1",
-    "multiparty": "^4.2.1",
+    "mime": "^2.5.2",
+    "moment": "^2.29.1",
+    "moment-timezone": "^0.5.33",
+    "morgan": "^1.10.0",
+    "multiparty": "^4.2.2",
+    "mustache-express": "^1.3.0",
     "mysql": "^2.18.1",
-    "nodemailer": "^6.4.2",
+    "nodemailer": "^6.5.0",
     "nodemailer-smtp-transport": "^2.7.4",
     "once": "^1.4.0",
-    "parse-links": "^0.1.0",
-    "pretty-bytes": "^5.3.0",
+    "pretty-bytes": "^5.6.0",
     "progress-stream": "^2.0.0",
     "proxy-middleware": "^0.15.0",
     "qrcode": "^1.4.4",
-    "readdirp": "^3.3.0",
-    "request": "^2.88.0",
-    "rimraf": "^2.6.3",
+    "readdirp": "^3.6.0",
+    "request": "^2.88.2",
+    "rimraf": "^3.0.2",
     "s3-block-read-stream": "^0.5.0",
-    "safetydance": "^1.0.0",
-    "semver": "^6.1.1",
+    "safetydance": "^1.1.1",
+    "semver": "^7.3.5",
     "showdown": "^1.9.1",
     "speakeasy": "^2.0.0",
     "split": "^1.0.1",
-    "superagent": "^5.2.1",
+    "superagent": "^6.1.0",
     "supererror": "^0.7.2",
     "tar-fs": "github:cloudron-io/tar-fs#ignore_stat_error",
-    "tar-stream": "^2.1.0",
+    "tar-stream": "^2.2.0",
     "tldjs": "^2.3.1",
-    "underscore": "^1.9.2",
-    "uuid": "^3.4.0",
-    "validator": "^11.0.0",
-    "ws": "^7.2.1",
+    "underscore": "^1.12.1",
+    "uuid": "^8.3.2",
+    "validator": "^13.5.2",
+    "ws": "^7.4.4",
     "xml2js": "^0.4.23"
   },
   "devDependencies": {
     "expect.js": "*",
-    "hock": "^1.3.3",
-    "js2xmlparser": "^4.0.0",
-    "mocha": "^6.1.4",
+    "hock": "^1.4.1",
+    "js2xmlparser": "^4.0.1",
+    "mocha": "^8.3.2",
     "mock-aws-s3": "git+https://github.com/cloudron-io/mock-aws-s3.git",
-    "nock": "^10.0.6",
-    "node-sass": "^4.12.0",
+    "nock": "^13.0.11",
+    "node-sass": "^5.0.0",
     "recursive-readdir": "^2.2.2"
   },
   "scripts": {

runTests

@@ -2,11 +2,11 @@
 
 set -eu
 
-readonly SOURCE_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly source_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 readonly DATA_DIR="${HOME}/.cloudron_test"
 readonly DEFAULT_TESTS="./src/test/*-test.js ./src/routes/test/*-test.js"
 
-! "${SOURCE_dir}/src/test/checkInstall" && exit 1
+! "${source_dir}/src/test/checkInstall" && exit 1
 
 # cleanup old data dirs some of those docker container data requires sudo to be removed
 echo "=> Provide root password to purge any leftover data in ${DATA_DIR} and load apparmor profile:"
@@ -22,19 +22,27 @@ fi
 mkdir -p ${DATA_DIR}
 cd ${DATA_DIR}
 mkdir -p appsdata
-mkdir -p boxdata/profileicons boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com
-mkdir -p platformdata/addons/mail platformdata/nginx/cert platformdata/nginx/applications platformdata/collectd/collectd.conf.d platformdata/addons platformdata/logrotate.d platformdata/backup platformdata/logs/tasks
+mkdir -p boxdata/profileicons boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com boxdata/sftp/ssh
+mkdir -p platformdata/addons/mail/banner platformdata/nginx/cert platformdata/nginx/applications platformdata/collectd/collectd.conf.d platformdata/addons platformdata/logrotate.d platformdata/backup platformdata/logs/tasks
+sudo mkdir -p /mnt/cloudron-test-music /media/cloudron-test-music # volume test
+
+# translations
+mkdir -p box/dashboard/dist/translation
+cp -r ${source_dir}/../dashboard/dist/translation/* box/dashboard/dist/translation
 
 # put cert
 echo "=> Generating a localhost selfsigned cert"
 openssl req -x509 -newkey rsa:2048 -keyout platformdata/nginx/cert/host.key -out platformdata/nginx/cert/host.cert -days 3650 -subj '/CN=localhost' -nodes -config <(cat /etc/ssl/openssl.cnf <(printf "\n[SAN]\nsubjectAltName=DNS:*.localhost"))
 
+# generate legacy key format for sftp
+ssh-keygen -m PEM -t rsa -f boxdata/sftp/ssh/ssh_host_rsa_key -q -N ""
+
 # clear out any containers
 echo "=> Delete all docker containers first"
 docker ps -qa | xargs --no-run-if-empty docker rm -f
 
 # create docker network (while the infra code does this, most tests skip infra setup)
-docker network create --subnet=172.18.0.0/16 cloudron || true
+docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 cloudron || true
 
 # create the same mysql server version to test with
 OUT=`docker inspect mysql-server` || true
@@ -59,7 +67,7 @@ echo "=> Ensure database"
 mysql -h"${MYSQL_IP}" -uroot -ppassword -e 'CREATE DATABASE IF NOT EXISTS box'
 
 echo "=> Run database migrations"
-cd "${SOURCE_dir}"
+cd "${source_dir}"
 BOX_ENV=test DATABASE_URL=mysql://root:password@${MYSQL_IP}/box node_modules/.bin/db-migrate up
 
 echo "=> Run tests with mocha"

scripts/cloudron-setup

@@ -2,6 +2,12 @@
 
 set -eu -o pipefail
 
+function exitHandler() {
+    rm -f /etc/update-motd.d/91-cloudron-install-in-progress
+}
+
+trap exitHandler EXIT
+
 # change this to a hash when we make a upgrade release
 readonly LOG_FILE="/var/log/cloudron-setup.log"
 readonly MINIMUM_DISK_SIZE_GB="18" # this is the size of "/" and required to fit in docker images 18 is a safe bet for different reporting on 20GB min
@@ -41,35 +47,39 @@ if systemctl -q is-active box; then
 fi
 
 initBaseImage="true"
-# provisioning data
-provider=""
+provider="generic"
 requestedVersion=""
+installServerOrigin="https://api.cloudron.io"
 apiServerOrigin="https://api.cloudron.io"
 webServerOrigin="https://cloudron.io"
 sourceTarballUrl=""
 rebootServer="true"
-license=""
+setupToken=""
 
-args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot,license:" -n "$0" -- "$@")
+args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot,generate-setup-token" -n "$0" -- "$@")
 eval set -- "${args}"
 
 while true; do
     case "$1" in
-    --help) echo "See https://cloudron.io/documentation/installation/ on how to install Cloudron"; exit 0;;
+    --help) echo "See https://docs.cloudron.io/installation/ on how to install Cloudron"; exit 0;;
     --provider) provider="$2"; shift 2;;
     --version) requestedVersion="$2"; shift 2;;
     --env)
         if [[ "$2" == "dev" ]]; then
             apiServerOrigin="https://api.dev.cloudron.io"
             webServerOrigin="https://dev.cloudron.io"
+            installServerOrigin="https://api.dev.cloudron.io"
         elif [[ "$2" == "staging" ]]; then
             apiServerOrigin="https://api.staging.cloudron.io"
             webServerOrigin="https://staging.cloudron.io"
+            installServerOrigin="https://api.staging.cloudron.io"
+        elif [[ "$2" == "unstable" ]]; then
+            installServerOrigin="https://api.dev.cloudron.io"
         fi
         shift 2;;
-    --license) license="$2"; shift 2;;
     --skip-baseimage-init) initBaseImage="false"; shift;;
     --skip-reboot) rebootServer="false"; shift;;
+    --generate-setup-token) setupToken="$(openssl rand -hex 10)"; shift;;
     --) break;;
     *) echo "Unknown option $1"; exit 1;;
     esac
@@ -83,56 +93,34 @@ fi
 
 # Only --help works with mismatched ubuntu
 ubuntu_version=$(lsb_release -rs)
-if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" ]]; then
-    echo "Cloudron requires Ubuntu 16.04 or 18.04" > /dev/stderr
+if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" && "${ubuntu_version}" != "20.04" ]]; then
+    echo "Cloudron requires Ubuntu 16.04, 18.04 or 20.04" > /dev/stderr
     exit 1
 fi
 
+# Install MOTD file for stack script style installations. this is removed by the trap exit handler. Heredoc quotes prevents parameter expansion
+cat > /etc/update-motd.d/91-cloudron-install-in-progress <<'EOF'
+#!/bin/bash
+
+printf "**********************************************************************\n\n"
+
+printf "\t\t\tWELCOME TO CLOUDRON\n"
+printf "\t\t\t-------------------\n"
+
+printf '\n\e[1;32m%-6s\e[m\n\n' "Cloudron is installing. Run 'tail -f /var/log/cloudron-setup.log' to view progress."
+
+printf "Cloudron overview - https://docs.cloudron.io/ \n"
+printf "Cloudron setup - https://docs.cloudron.io/installation/#setup \n"
+
+printf "\nFor help and more information, visit https://forum.cloudron.io\n\n"
+
+printf "**********************************************************************\n"
+EOF
+chmod +x /etc/update-motd.d/91-cloudron-install-in-progress
+
 # Can only write after we have confirmed script has root access
 echo "Running cloudron-setup with args : $@" > "${LOG_FILE}"
 
-# validate arguments in the absence of data
-readonly AVAILABLE_PROVIDERS="azure, caas, cloudscale, contabo, digitalocean, ec2, exoscale, gce, hetzner, interox, lightsail, linode, netcup, ovh, rosehosting, scaleway, skysilk, time4vps, upcloud, vultr or generic"
-if [[ -z "${provider}" ]]; then
-    echo "--provider is required ($AVAILABLE_PROVIDERS)"
-    exit 1
-elif [[  \
-            "${provider}" != "ami" && \
-            "${provider}" != "azure" && \
-            "${provider}" != "azure-image" && \
-            "${provider}" != "caas" && \
-            "${provider}" != "cloudscale" && \
-            "${provider}" != "contabo" && \
-            "${provider}" != "digitalocean" && \
-            "${provider}" != "digitalocean-mp" && \
-            "${provider}" != "ec2" && \
-            "${provider}" != "exoscale" && \
-            "${provider}" != "gce" && \
-            "${provider}" != "hetzner" && \
-            "${provider}" != "interox" && \
-            "${provider}" != "interox-image" && \
-            "${provider}" != "lightsail" && \
-            "${provider}" != "linode" && \
-            "${provider}" != "linode-oneclick" && \
-            "${provider}" != "linode-stackscript" && \
-            "${provider}" != "netcup" && \
-            "${provider}" != "netcup-image" && \
-            "${provider}" != "ovh" && \
-            "${provider}" != "rosehosting" && \
-            "${provider}" != "scaleway" && \
-            "${provider}" != "skysilk" && \
-            "${provider}" != "skysilk-image" && \
-            "${provider}" != "time4vps" && \
-            "${provider}" != "time4vps-image" && \
-            "${provider}" != "upcloud" && \
-            "${provider}" != "upcloud-image" && \
-            "${provider}" != "vultr" && \
-            "${provider}" != "generic" \
-        ]]; then
-    echo "--provider must be one of: $AVAILABLE_PROVIDERS"
-    exit 1
-fi
-
 echo ""
 echo "##############################################"
 echo "         Cloudron Setup (${requestedVersion:-latest})"
@@ -145,32 +133,20 @@ echo " Join us at https://forum.cloudron.io for any questions."
 echo ""
 
 if [[ "${initBaseImage}" == "true" ]]; then
-    echo "=> Installing software-properties-common"
-    if ! apt-get install -y software-properties-common &>> "${LOG_FILE}"; then
-        echo "Could not install software-properties-common (for add-apt-repository below). See ${LOG_FILE}"
-        exit 1
-    fi
-
-    echo "=> Ensure required apt sources"
-    if ! add-apt-repository universe &>> "${LOG_FILE}"; then
-        echo "Could not add required apt sources (for nginx-full). See ${LOG_FILE}"
-        exit 1
-    fi
-
     echo "=> Updating apt and installing script dependencies"
     if ! apt-get update &>> "${LOG_FILE}"; then
         echo "Could not update package repositories. See ${LOG_FILE}"
         exit 1
     fi
 
-    if ! DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install curl python3 ubuntu-standard -y &>> "${LOG_FILE}"; then
+    if ! DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install --no-install-recommends curl python3 ubuntu-standard software-properties-common -y &>> "${LOG_FILE}"; then
         echo "Could not install setup dependencies (curl). See ${LOG_FILE}"
         exit 1
     fi
 fi
 
 echo "=> Checking version"
-if ! releaseJson=$($curl -s "${apiServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
+if ! releaseJson=$($curl -s "${installServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
     echo "Failed to get release information"
     exit 1
 fi
@@ -196,19 +172,19 @@ fi
 
 if [[ "${initBaseImage}" == "true" ]]; then
     echo -n "=> Installing base dependencies and downloading docker images (this takes some time) ..."
-    if ! /bin/bash "${box_src_tmp_dir}/baseimage/initializeBaseUbuntuImage.sh" "${provider}" "../src" &>> "${LOG_FILE}"; then
+    # initializeBaseUbuntuImage.sh args (provider, infraversion path) are only to support installation of pre 5.3 Cloudrons
+    if ! /bin/bash "${box_src_tmp_dir}/baseimage/initializeBaseUbuntuImage.sh" "generic" "../src" &>> "${LOG_FILE}"; then
         echo "Init script failed. See ${LOG_FILE} for details"
         exit 1
     fi
     echo ""
 fi
 
-# NOTE: this install script only supports 4.2 and above
+# The provider flag is still used for marketplace images
 echo "=> Installing version ${version} (this takes some time) ..."
 mkdir -p /etc/cloudron
 echo "${provider}" > /etc/cloudron/PROVIDER
-
-[[ -n "${license}" ]] && echo -n "$license" > /etc/cloudron/LICENSE
+[[ ! -z "${setupToken}" ]] && echo "${setupToken}" > /etc/cloudron/SETUP_TOKEN
 
 if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" &>> "${LOG_FILE}"; then
     echo "Failed to install cloudron. See ${LOG_FILE} for details"
@@ -221,16 +197,21 @@ mysql -uroot -ppassword -e "REPLACE INTO box.settings (name, value) VALUES ('web
 echo -n "=> Waiting for cloudron to be ready (this takes some time) ..."
 while true; do
     echo -n "."
-    if status=$($curl -q -f "http://localhost:3000/api/v1/cloudron/status" 2>/dev/null); then
+    if status=$($curl -s -f "http://localhost:3000/api/v1/cloudron/status" 2>/dev/null); then
         break # we are up and running
     fi
     sleep 10
 done
 
-if ! ip=$(curl --fail --connect-timeout 2 --max-time 2 -q https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
+if ! ip=$(curl -s --fail --connect-timeout 2 --max-time 2 https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
     ip='<IP>'
 fi
-echo -e "\n\n${GREEN}Visit https://${ip} and accept the self-signed certificate to finish setup.${DONE}\n"
+if [[ -z "${setupToken}" ]]; then
+    url="https://${ip}"
+else
+    url="https://${ip}/?setupToken=${setupToken}"
+fi
+echo -e "\n\n${GREEN}After reboot, visit ${url} and accept the self-signed certificate to finish setup.${DONE}\n"
 
 if [[ "${rebootServer}" == "true" ]]; then
     systemctl stop box mysql # sometimes mysql ends up having corrupt privilege tables
@@ -238,7 +219,7 @@ if [[ "${rebootServer}" == "true" ]]; then
     read -p "The server has to be rebooted to apply all the settings. Reboot now ? [Y/n] " yn
     yn=${yn:-y}
     case $yn in
-        [Yy]* ) systemctl reboot;;
+        [Yy]* ) exitHandler; systemctl reboot;;
         * ) exit;;
     esac
 fi

scripts/cloudron-support

@@ -37,12 +37,13 @@ while true; do
         # fall through
         ;&
     --owner-login)
-        admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' LIMIT 1" 2>/dev/null)
+        admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' AND username IS NOT NULL ORDER BY createdAt LIMIT 1" 2>/dev/null)
         admin_password=$(pwgen -1s 12)
+        dashboard_domain=$(mysql -NB -uroot -ppassword -e "SELECT value FROM box.settings WHERE name='admin_fqdn'" 2>/dev/nul)
         ghost_file=/home/yellowtent/platformdata/cloudron_ghost.json
         printf '{"%s":"%s"}\n' "${admin_username}" "${admin_password}" > "${ghost_file}"
         chown yellowtent:yellowtent "${ghost_file}" && chmod o-r,g-r "${ghost_file}"
-        echo "Login as ${admin_username} / ${admin_password} . Remove ${ghost_file} when done."
+        echo "Login at https://${dashboard_domain} as ${admin_username} / ${admin_password} . This password may only be used once. ${ghost_file} will be automatically removed after use."
         exit 0
         ;;
     --) break;;
@@ -57,7 +58,7 @@ if [[ "`df --output="avail" / | sed -n 2p`" -lt "10240" ]]; then
     echo ""
     df -h
     echo ""
-    echo "To recover from a full disk, follow the guide at https://cloudron.io/documentation/troubleshooting/#recovery-after-disk-full"
+    echo "To recover from a full disk, follow the guide at https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full"
     exit 1
 fi
 
@@ -73,6 +74,9 @@ echo -n "Generating Cloudron Support stats..."
 # clear file
 rm -rf $OUT
 
+echo -e $LINE"DASHBOARD DOMAIN"$LINE >> $OUT
+mysql -NB -uroot -ppassword -e "SELECT value FROM box.settings WHERE name='admin_fqdn'" &>> $OUT 2>/dev/null || true
+
 echo -e $LINE"PROVIDER"$LINE >> $OUT
 cat /etc/cloudron/PROVIDER &>> $OUT || true
 
@@ -94,12 +98,12 @@ echo -e $LINE"Backup stats (possibly misleading)"$LINE >> $OUT
 du -hcsL /var/backups/* &>> $OUT || true
 
 echo -e $LINE"System daemon status"$LINE >> $OUT
-systemctl status --lines=100 cloudron.target box mysql unbound cloudron-syslog nginx collectd docker &>> $OUT
+systemctl status --lines=100 box mysql unbound cloudron-syslog nginx collectd docker &>> $OUT
 
 echo -e $LINE"Box logs"$LINE >> $OUT
 tail -n 100 /home/yellowtent/platformdata/logs/box.log &>> $OUT
 
-echo -e $LINE"Firewall chains"$LINE >> $OUT
+echo -e $LINE"Interface Info"$LINE >> $OUT
 ip addr &>> $OUT
 
 echo -e $LINE"Firewall chains"$LINE >> $OUT
@@ -112,7 +116,7 @@ if [[ "${enableSSH}" == "true" ]]; then
     permit_root_login=$(grep -q ^PermitRootLogin.*yes /etc/ssh/sshd_config && echo "yes" || echo "no")
 
     # support.js uses similar logic
-    if $(grep -q "ec2\|lightsail\|ami" /etc/cloudron/PROVIDER); then
+    if [[ -d /home/ubuntu ]]; then
         ssh_user="ubuntu"
         keys_file="/home/ubuntu/.ssh/authorized_keys"
     else

scripts/cloudron-translation-update

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+set -eu -o pipefail
+
+# This script downloads new translation data from weblate at https://translate.cloudron.io
+
+OUT="/home/yellowtent/box/dashboard/dist/translation"
+
+# We require root
+if [[ ${EUID} -ne 0 ]]; then
+    echo "This script should be run as root. Run with sudo"
+    exit 1
+fi
+
+echo "=> Downloading new translation files..."
+curl https://translate.cloudron.io/download/cloudron/dashboard/?format=zip -o /tmp/lang.zip
+
+echo "=> Unpacking..."
+unzip -jo /tmp/lang.zip -d $OUT
+chown -R yellowtent:yellowtent $OUT
+# unzip put very restrictive permissions
+chmod ua+r $OUT/*
+
+echo "=> Cleanup..."
+rm /tmp/lang.zip
+
+echo "=> Done"
+
+echo ""
+echo "Reload the dashboard to see the new translations"
+echo ""

scripts/createReleaseTarball

@@ -41,8 +41,8 @@ if ! $(cd "${SOURCE_DIR}/../dashboard" && git diff --exit-code >/dev/null); then
     exit 1
 fi
 
-if [[ "$(node --version)" != "v10.18.1" ]]; then
-    echo "This script requires node 10.18.1"
+if [[ "$(node --version)" != "v14.15.4" ]]; then
+    echo "This script requires node 14.15.4"
     exit 1
 fi
 

scripts/installer.sh

@@ -11,6 +11,10 @@ if [[ ${EUID} -ne 0 ]]; then
     exit 1
 fi
 
+function log() {
+  echo -e "$(date +'%Y-%m-%dT%H:%M:%S')" "==> installer: $1"
+}
+
 readonly user=yellowtent
 readonly box_src_dir=/home/${user}/box
 
@@ -21,58 +25,60 @@ readonly box_src_tmp_dir="$(realpath ${script_dir}/..)"
 readonly ubuntu_version=$(lsb_release -rs)
 readonly ubuntu_codename=$(lsb_release -cs)
 
-readonly is_update=$(systemctl is-active box && echo "yes" || echo "no")
+readonly is_update=$(systemctl is-active -q box && echo "yes" || echo "no")
 
-echo "==> installer: Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src_tmp_dir/VERSION) <=="
+log "Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src_tmp_dir/VERSION)"
 
-echo "==> installer: updating docker"
+log "updating docker"
 
-if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
+readonly docker_version=20.10.3
+if [[ $(docker version --format {{.Client.Version}}) != "${docker_version}" ]]; then
     # there are 3 packages for docker - containerd, CLI and the daemon
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.2-3_amd64.deb" -o /tmp/containerd.deb
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_18.09.2~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb" -o /tmp/containerd.deb
+    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
+    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
 
-    echo "==> installer: Waiting for all dpkg tasks to finish..."
+    log "Waiting for all dpkg tasks to finish..."
     while fuser /var/lib/dpkg/lock; do
         sleep 1
     done
 
     while ! dpkg --force-confold --configure -a; do
-        echo "==> installer: Failed to fix packages. Retry"
+        log "Failed to fix packages. Retry"
         sleep 1
     done
 
     # the latest docker might need newer packages
     while ! apt update -y; do
-        echo "==> installer: Failed to update packages. Retry"
+        log "Failed to update packages. Retry"
         sleep 1
     done
 
     while ! apt install -y /tmp/containerd.deb  /tmp/docker-ce-cli.deb /tmp/docker.deb; do
-        echo "==> installer: Failed to install docker. Retry"
+        log "Failed to install docker. Retry"
         sleep 1
     done
 
     rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
 fi
 
-readonly nginx_version=$(nginx -v)
-if [[ "${nginx_version}" != *"1.14."* && "${ubuntu_version}" == "16.04" ]]; then
-    echo "==> installer: installing nginx for xenial for TLSv3 support"
-    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.14.0-1~xenial_amd64.deb -o /tmp/nginx.deb
+readonly nginx_version=$(nginx -v 2>&1)
+if [[ "${nginx_version}" != *"1.18."* ]]; then
+    log "installing nginx 1.18"
+    $curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-2~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
     # apt install with install deps (as opposed to dpkg -i)
     apt install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes /tmp/nginx.deb
     rm /tmp/nginx.deb
 fi
 
-echo "==> installer: updating node"
-if [[ "$(node --version)" != "v10.18.1" ]]; then
-    mkdir -p /usr/local/node-10.18.1
-    $curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-10.18.1
-    ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
-    ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
-    rm -rf /usr/local/node-10.15.1
+log "updating node"
+readonly node_version=14.15.4
+if [[ "$(node --version)" != "v${node_version}" ]]; then
+    mkdir -p /usr/local/node-${node_version}
+    $curl -sL https://nodejs.org/dist/v${node_version}/node-v${node_version}-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-${node_version}
+    ln -sf /usr/local/node-${node_version}/bin/node /usr/bin/node
+    ln -sf /usr/local/node-${node_version}/bin/npm /usr/bin/npm
+    rm -rf /usr/local/node-10.18.1
 fi
 
 # this is here (and not in updater.js) because rebuild requires the above node
@@ -83,31 +89,31 @@ for try in `seq 1 10`; do
     # however by default npm drops privileges for npm rebuild
     # https://docs.npmjs.com/misc/config#unsafe-perm
     if cd "${box_src_tmp_dir}" && npm rebuild --unsafe-perm; then break; fi
-    echo "==> installer: Failed to rebuild, trying again"
+    log "Failed to rebuild, trying again"
     sleep 5
 done
 
 if [[ ${try} -eq 10 ]]; then
-    echo "==> installer: npm rebuild failed, giving up"
+    log "npm rebuild failed, giving up"
     exit 4
 fi
 
-echo "==> installer: downloading new addon images"
+log "downloading new addon images"
 images=$(node -e "var i = require('${box_src_tmp_dir}/src/infra_version.js'); console.log(i.baseImages.map(function (x) { return x.tag; }).join(' '), Object.keys(i.images).map(function (x) { return i.images[x].tag; }).join(' '));")
 
-echo -e "\tPulling docker images: ${images}"
+log "\tPulling docker images: ${images}"
 for image in ${images}; do
-    if ! docker pull "${image}"; then           # this pulls the image using the sha256
-        echo "==> installer: Could not pull ${image}"
-        exit 5
-    fi
-    if ! docker pull "${image%@sha256:*}"; then  # this will tag the image for readability
-        echo "==> installer: Could not pull ${image%@sha256:*}"
-        exit 6
-   fi
+    while ! docker pull "${image}"; do           # this pulls the image using the sha256
+        log "Could not pull ${image}"
+        sleep 5
+    done
+    while ! docker pull "${image%@sha256:*}"; do  # this will tag the image for readability
+        log "Could not pull ${image%@sha256:*}"
+        sleep 5
+   done
 done
 
-echo "==> installer: update cloudron-syslog"
+log "update cloudron-syslog"
 CLOUDRON_SYSLOG_DIR=/usr/local/cloudron-syslog
 CLOUDRON_SYSLOG="${CLOUDRON_SYSLOG_DIR}/bin/cloudron-syslog"
 CLOUDRON_SYSLOG_VERSION="1.0.3"
@@ -115,7 +121,7 @@ while [[ ! -f "${CLOUDRON_SYSLOG}" || "$(${CLOUDRON_SYSLOG} --version)" != ${CLO
     rm -rf "${CLOUDRON_SYSLOG_DIR}"
     mkdir -p "${CLOUDRON_SYSLOG_DIR}"
     if npm install --unsafe-perm -g --prefix "${CLOUDRON_SYSLOG_DIR}" cloudron-syslog@${CLOUDRON_SYSLOG_VERSION}; then break; fi
-    echo "===> installer: Failed to install cloudron-syslog, trying again"
+    log "Failed to install cloudron-syslog, trying again"
     sleep 5
 done
 
@@ -124,17 +130,17 @@ if ! id "${user}" 2>/dev/null; then
 fi
 
 if [[ "${is_update}" == "yes" ]]; then
-    echo "==> installer: stop cloudron.target service for update"
+    log "stop box service for update"
     ${box_src_dir}/setup/stop.sh
 fi
 
 # ensure we are not inside the source directory, which we will remove now
 cd /root
 
-echo "==> installer: switching the box code"
+log "switching the box code"
 rm -rf "${box_src_dir}"
 mv "${box_src_tmp_dir}" "${box_src_dir}"
 chown -R "${user}:${user}" "${box_src_dir}"
 
-echo "==> installer: calling box setup script"
+log "calling box setup script"
 "${box_src_dir}/setup/start.sh"

setup/start.sh

@@ -5,7 +5,11 @@ set -eu -o pipefail
 # This script is run after the box code is switched. This means that this script
 # should pretty much always succeed. No network logic/download code here.
 
-echo "==> Cloudron Start"
+function log() {
+  echo -e "$(date +'%Y-%m-%dT%H:%M:%S')" "==> start: $1"
+}
+
+log "Cloudron Start"
 
 readonly USER="yellowtent"
 readonly HOME_DIR="/home/${USER}"
@@ -19,32 +23,34 @@ readonly json="$(realpath ${script_dir}/../node_modules/.bin/json)"
 readonly ubuntu_version=$(lsb_release -rs)
 
 cp -f "${script_dir}/../scripts/cloudron-support" /usr/bin/cloudron-support
+cp -f "${script_dir}/../scripts/cloudron-translation-update" /usr/bin/cloudron-translation-update
 
 # this needs to match the cloudron/base:2.0.0 gid
 if ! getent group media; then
     addgroup --gid 500 --system media
 fi
 
-echo "==> Configuring docker"
+log "Configuring docker"
 cp "${script_dir}/start/docker-cloudron-app.apparmor" /etc/apparmor.d/docker-cloudron-app
 systemctl enable apparmor
 systemctl restart apparmor
 
 usermod ${USER} -a -G docker
-docker network create --subnet=172.18.0.0/16 cloudron || true
+# unbound (which starts after box code) relies on this interface to exist. dockerproxy also relies on this.
+docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 cloudron || true
 
 mkdir -p "${BOX_DATA_DIR}"
 mkdir -p "${APPS_DATA_DIR}"
 
 # keep these in sync with paths.js
-echo "==> Ensuring directories"
+log "Ensuring directories"
 
 mkdir -p "${PLATFORM_DATA_DIR}/graphite"
 mkdir -p "${PLATFORM_DATA_DIR}/mysql"
 mkdir -p "${PLATFORM_DATA_DIR}/postgresql"
 mkdir -p "${PLATFORM_DATA_DIR}/mongodb"
 mkdir -p "${PLATFORM_DATA_DIR}/redis"
-mkdir -p "${PLATFORM_DATA_DIR}/addons/mail"
+mkdir -p "${PLATFORM_DATA_DIR}/addons/mail/banner"
 mkdir -p "${PLATFORM_DATA_DIR}/collectd/collectd.conf.d"
 mkdir -p "${PLATFORM_DATA_DIR}/logrotate.d"
 mkdir -p "${PLATFORM_DATA_DIR}/acme"
@@ -57,17 +63,19 @@ mkdir -p "${PLATFORM_DATA_DIR}/logs/backup" \
 mkdir -p "${PLATFORM_DATA_DIR}/update"
 
 mkdir -p "${BOX_DATA_DIR}/appicons"
+mkdir -p "${BOX_DATA_DIR}/firewall"
 mkdir -p "${BOX_DATA_DIR}/profileicons"
 mkdir -p "${BOX_DATA_DIR}/certs"
 mkdir -p "${BOX_DATA_DIR}/acme" # acme keys
 mkdir -p "${BOX_DATA_DIR}/mail/dkim"
 mkdir -p "${BOX_DATA_DIR}/well-known" # .well-known documents
+mkdir -p "${BOX_DATA_DIR}/sftp/ssh" # sftp keys
 
 # ensure backups folder exists and is writeable
 mkdir -p /var/backups
 chmod 777 /var/backups
 
-echo "==> Configuring journald"
+log "Configuring journald"
 sed -e "s/^#SystemMaxUse=.*$/SystemMaxUse=100M/" \
     -e "s/^#ForwardToSyslog=.*$/ForwardToSyslog=no/" \
     -i /etc/systemd/journald.conf
@@ -85,7 +93,10 @@ systemctl daemon-reload
 systemctl restart systemd-journald
 setfacl -n -m u:${USER}:r /var/log/journal/*/system.journal
 
-echo "==> Setting up unbound"
+# Give user access to nginx logs (uses adm group)
+usermod -a -G adm ${USER}
+
+log "Setting up unbound"
 # DO uses Google nameservers by default. This causes RBL queries to fail (host 2.0.0.127.zen.spamhaus.org)
 # We do not use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!)
 # We listen on 0.0.0.0 because there is no way control ordering of docker (which creates the 172.18.0.0/16) and unbound
@@ -95,14 +106,15 @@ cp -f "${script_dir}/start/unbound.conf" /etc/unbound/unbound.conf.d/cloudron-ne
 # update the root anchor after a out-of-disk-space situation (see #269)
 unbound-anchor -a /var/lib/unbound/root.key
 
-echo "==> Adding systemd services"
+log "Adding systemd services"
 cp -r "${script_dir}/start/systemd/." /etc/systemd/system/
 [[ "${ubuntu_version}" == "16.04" ]] && sed -e 's/MemoryMax/MemoryLimit/g' -i /etc/systemd/system/box.service
 systemctl daemon-reload
+systemctl enable --now cloudron-syslog
 systemctl enable unbound
-systemctl enable cloudron-syslog
-systemctl enable cloudron.target
+systemctl enable box
 systemctl enable cloudron-firewall
+systemctl enable --now cloudron-disable-thp
 
 # update firewall rules
 systemctl restart cloudron-firewall
@@ -116,17 +128,23 @@ systemctl restart unbound
 # ensure cloudron-syslog runs
 systemctl restart cloudron-syslog
 
-echo "==> Configuring sudoers"
+log "Configuring sudoers"
 rm -f /etc/sudoers.d/${USER}
 cp "${script_dir}/start/sudoers" /etc/sudoers.d/${USER}
 
-echo "==> Configuring collectd"
+log "Configuring collectd"
 rm -rf /etc/collectd /var/log/collectd.log
 ln -sfF "${PLATFORM_DATA_DIR}/collectd" /etc/collectd
 cp "${script_dir}/start/collectd/collectd.conf" "${PLATFORM_DATA_DIR}/collectd/collectd.conf"
+if [[ "${ubuntu_version}" == "20.04" ]]; then
+    # https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1872281
+    if ! grep -q LD_PRELOAD /etc/default/collectd; then
+        echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
+    fi
+fi
 systemctl restart collectd
 
-echo "==> Configuring logrotate"
+log "Configuring logrotate"
 if ! grep -q "^include ${PLATFORM_DATA_DIR}/logrotate.d" /etc/logrotate.conf; then
     echo -e "\ninclude ${PLATFORM_DATA_DIR}/logrotate.d\n" >> /etc/logrotate.conf
 fi
@@ -136,10 +154,10 @@ cp "${script_dir}/start/logrotate/"* "${PLATFORM_DATA_DIR}/logrotate.d/"
 # logrotate files have to be owned by root, this is here to fixup existing installations where we were resetting the owner to yellowtent
 chown root:root "${PLATFORM_DATA_DIR}/logrotate.d/"
 
-echo "==> Adding motd message for admins"
+log "Adding motd message for admins"
 cp "${script_dir}/start/cloudron-motd" /etc/update-motd.d/92-cloudron
 
-echo "==> Configuring nginx"
+log "Configuring nginx"
 # link nginx config to system config
 unlink /etc/nginx 2>/dev/null || rm -rf /etc/nginx
 ln -s "${PLATFORM_DATA_DIR}/nginx" /etc/nginx
@@ -150,8 +168,15 @@ cp "${script_dir}/start/nginx/mime.types" "${PLATFORM_DATA_DIR}/nginx/mime.types
 if ! grep -q "^Restart=" /etc/systemd/system/multi-user.target.wants/nginx.service; then
     # default nginx service file does not restart on crash
     echo -e "\n[Service]\nRestart=always\n" >> /etc/systemd/system/multi-user.target.wants/nginx.service
-    systemctl daemon-reload
 fi
+
+# worker_rlimit_nofile in nginx config can be max this number
+mkdir -p /etc/systemd/system/nginx.service.d
+if ! grep -q "^LimitNOFILE=" /etc/systemd/system/nginx.service.d/cloudron.conf; then
+    echo -e "[Service]\nLimitNOFILE=16384\n" > /etc/systemd/system/nginx.service.d/cloudron.conf
+fi
+
+systemctl daemon-reload
 systemctl start nginx
 
 # restart mysql to make sure it has latest config
@@ -160,49 +185,64 @@ if [[ ! -f /etc/mysql/mysql.cnf ]] || ! diff -q "${script_dir}/start/mysql.cnf"
     cp "${script_dir}/start/mysql.cnf" /etc/mysql/mysql.cnf
     while true; do
         if ! systemctl list-jobs | grep mysql; then break; fi
-        echo "Waiting for mysql jobs..."
+        log "Waiting for mysql jobs..."
         sleep 1
     done
-    while true; do
-        if systemctl restart mysql; then break; fi
-        echo "Restarting MySql again after sometime since this fails randomly"
+    log "Stopping mysql"
+    systemctl stop mysql
+    while mysqladmin ping 2>/dev/null; do
+        log "Waiting for mysql to stop..."
         sleep 1
     done
-else
-    systemctl start mysql
 fi
 
+# the start/stop of mysql is separate to make sure it got reloaded with latest config and it's up and running before we start the new box code
+# when using 'system restart mysql', it seems to restart much later and the box code loses connection during platform startup (dangerous!)
+log "Starting mysql"
+systemctl start mysql
+while ! mysqladmin ping 2>/dev/null; do
+    log "Waiting for mysql to start..."
+    sleep 1
+done
+
 readonly mysql_root_password="password"
 mysqladmin -u root -ppassword password password # reset default root password
+if [[ "${ubuntu_version}" == "20.04" ]]; then
+    # mysql 8 added a new caching_sha2_password scheme which mysqljs does not support
+    mysql -u root -p${mysql_root_password} -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '${mysql_root_password}';"
+fi
 mysql -u root -p${mysql_root_password} -e 'CREATE DATABASE IF NOT EXISTS box'
 
 # set HOME explicity, because it's not set when the installer calls it. this is done because
 # paths.js uses this env var and some of the migrate code requires box code
-echo "==> Migrating data"
+log "Migrating data"
 cd "${BOX_SRC_DIR}"
 if ! HOME=${HOME_DIR} BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up; then
-    echo "DB migration failed"
+    log "DB migration failed"
     exit 1
 fi
 
 rm -f /etc/cloudron/cloudron.conf
 
 if [[ ! -f "${BOX_DATA_DIR}/dhparams.pem" ]]; then
-    echo "==> Generating dhparams (takes forever)"
+    log "Generating dhparams (takes forever)"
     openssl dhparam -out "${BOX_DATA_DIR}/dhparams.pem" 2048
     cp "${BOX_DATA_DIR}/dhparams.pem" "${PLATFORM_DATA_DIR}/addons/mail/dhparams.pem"
 else
     cp "${BOX_DATA_DIR}/dhparams.pem" "${PLATFORM_DATA_DIR}/addons/mail/dhparams.pem"
 fi
 
-# old installations used to create appdata/<app>/redis which is now part of old backups and prevents restore
-echo "==> Cleaning up stale redis directories"
-find "${APPS_DATA_DIR}" -maxdepth 2 -type d -name redis -exec rm -rf {} +
+if [[ ! -f "${BOX_DATA_DIR}/sftp/ssh/ssh_host_rsa_key" ]]; then
+    # the key format in Ubuntu 20 changed, so we create keys in legacy format. for older ubuntu, just re-use the host keys
+    # see https://github.com/proftpd/proftpd/issues/793
+    if [[ "${ubuntu_version}" == "20.04" ]]; then
+        ssh-keygen -m PEM -t rsa -f "${BOX_DATA_DIR}/sftp/ssh/ssh_host_rsa_key" -q -N ""
+    else
+        cp /etc/ssh/ssh_host_rsa_key* ${BOX_DATA_DIR}/sftp/ssh
+    fi
+fi
 
-echo "==> Cleaning up old logs"
-rm -f /home/yellowtent/platformdata/logs/*/*.log.* || true
-
-echo "==> Changing ownership"
+log "Changing ownership"
 # be careful of what is chown'ed here. subdirs like mysql,redis etc are owned by the containers and will stop working if perms change
 chown -R "${USER}" /etc/cloudron
 chown "${USER}:${USER}" -R "${PLATFORM_DATA_DIR}/nginx" "${PLATFORM_DATA_DIR}/collectd" "${PLATFORM_DATA_DIR}/addons" "${PLATFORM_DATA_DIR}/acme" "${PLATFORM_DATA_DIR}/backup" "${PLATFORM_DATA_DIR}/logs" "${PLATFORM_DATA_DIR}/update"
@@ -216,9 +256,9 @@ find "${BOX_DATA_DIR}" -mindepth 1 -maxdepth 1 -not -path "${BOX_DATA_DIR}/mail"
 chown "${USER}:${USER}" "${BOX_DATA_DIR}/mail"
 chown "${USER}:${USER}" -R "${BOX_DATA_DIR}/mail/dkim" # this is owned by box currently since it generates the keys
 
-echo "==> Starting Cloudron"
-systemctl start cloudron.target
+log "Starting Cloudron"
+systemctl start box
 
 sleep 2 # give systemd sometime to start the processes
 
-echo "==> Almost done"
+log "Almost done"

setup/start/cloudron-disable-thp.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+set -eu
+
+echo "==> Disabling THP"
+
+# https://docs.couchbase.com/server/current/install/thp-disable.html
+if [[ -d /sys/kernel/mm/transparent_hugepage ]]; then
+    echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
+    echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
+else
+    echo "==> kernel does not have THP"
+fi
+

setup/start/cloudron-firewall.sh

@@ -6,11 +6,35 @@ echo "==> Setting up firewall"
 iptables -t filter -N CLOUDRON || true
 iptables -t filter -F CLOUDRON # empty any existing rules
 
-# NOTE: keep these in sync with src/apps.js validatePortBindings
-# allow ssh, http, https, ping, dns
-iptables -t filter -I CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
-# ssh is allowed alternately on port 202
-iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443,587,993,4190 -j ACCEPT
+# first setup any user IP block lists
+ipset create cloudron_blocklist hash:net || true
+/home/yellowtent/box/src/scripts/setblocklist.sh
+
+iptables -t filter -A CLOUDRON -m set --match-set cloudron_blocklist src -j DROP
+# the DOCKER-USER chain is not cleared on docker restart
+if ! iptables -t filter -C DOCKER-USER -m set --match-set cloudron_blocklist src -j DROP; then
+    iptables -t filter -I DOCKER-USER 1 -m set --match-set cloudron_blocklist src -j DROP
+fi
+
+# allow related and establisted connections
+iptables -t filter -A CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
+iptables -t filter -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443 -j ACCEPT # 202 is the alternate ssh port
+
+# whitelist any user ports. we used to use --dports but it has a 15 port limit (XT_MULTI_PORTS)
+ports_json="/home/yellowtent/boxdata/firewall/ports.json"
+if allowed_tcp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_tcp_ports.join(','))" 2>/dev/null); then
+    IFS=',' arr=(${allowed_tcp_ports})
+    for p in "${arr[@]}"; do
+        iptables -A CLOUDRON -p tcp -m tcp --dport "${p}" -j ACCEPT
+    done
+fi
+
+if allowed_udp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_udp_ports.join(','))" 2>/dev/null); then
+    IFS=',' arr=(${allowed_udp_ports})
+    for p in "${arr[@]}"; do
+        iptables -A CLOUDRON -p udp -m udp --dport "${p}" -j ACCEPT
+    done
+fi
 
 # turn and stun service
 iptables -t filter -A CLOUDRON -p tcp -m multiport --dports 3478,5349 -j ACCEPT
@@ -52,8 +76,6 @@ for port in 22 202; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --dport ${port} -m state --state NEW -m recent --update --name "public-${port}" --seconds 10 --hitcount 5 -j CLOUDRON_RATELIMIT_LOG
 done
 
-# TODO: move docker platform rules to platform.js so it can be specialized to rate limit only when destination is the mail container
-
 # docker translates (dnat) 25, 587, 993, 4190 in the PREROUTING step
 for port in 2525 4190 9993; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --syn ! -s 172.18.0.0/16 -d 172.18.0.0/16 --dport ${port} -m connlimit --connlimit-above 50 -j CLOUDRON_RATELIMIT_LOG
@@ -69,12 +91,12 @@ for port in 3306 5432 6379 27017; do
     iptables -A CLOUDRON_RATELIMIT -p tcp --syn -s 172.18.0.0/16 -d 172.18.0.0/16 --dport ${port} -m connlimit --connlimit-above 5000 -j CLOUDRON_RATELIMIT_LOG
 done
 
-# For ssh, http, https
 if ! iptables -t filter -C INPUT -j CLOUDRON_RATELIMIT 2>/dev/null; then
     iptables -t filter -I INPUT 1 -j CLOUDRON_RATELIMIT
 fi
 
-# For smtp, imap etc routed via docker/nat
-# Workaroud issue where Docker insists on adding itself first in FORWARD table
+# Workaround issue where Docker insists on adding itself first in FORWARD table
 iptables -D FORWARD -j CLOUDRON_RATELIMIT || true
 iptables -I FORWARD 1 -j CLOUDRON_RATELIMIT
+
+echo "==> Setting up firewall done"

setup/start/cloudron-motd

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+[[ -f /etc/update-motd.d/91-cloudron-install-in-progress ]] && exit
+
 printf "**********************************************************************\n\n"
 
 if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
@@ -10,12 +12,19 @@ if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
     fi
     echo "${ip}" > /tmp/.cloudron-motd-cache
 
+    if [[ ! -f /etc/cloudron/SETUP_TOKEN ]]; then
+        url="https://${ip}"
+    else
+        setupToken="$(cat /etc/cloudron/SETUP_TOKEN)"
+        url="https://${ip}/?setupToken=${setupToken}"
+    fi
+
     printf "\t\t\tWELCOME TO CLOUDRON\n"
     printf "\t\t\t-------------------\n"
 
-    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://${ip} on your browser and accept the self-signed certificate to finish setup."
-    printf "Cloudron overview - https://cloudron.io/documentation/ \n"
-    printf "Cloudron setup - https://cloudron.io/documentation/installation/#setup \n"
+    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit ${url} on your browser and accept the self-signed certificate to finish setup."
+    printf "Cloudron overview - https://docs.cloudron.io/ \n"
+    printf "Cloudron setup - https://docs.cloudron.io/installation/#setup \n"
 else
     printf "\t\t\tNOTE TO CLOUDRON ADMINS\n"
     printf "\t\t\t-----------------------\n"
@@ -23,7 +32,7 @@ else
     printf "Cloudron relies on and may break your installation. Ubuntu security updates\n"
     printf "are automatically installed on this server every night.\n"
     printf "\n"
-    printf "Read more at https://cloudron.io/documentation/security/#os-updates\n"
+    printf "Read more at https://docs.cloudron.io/security/#os-updates\n"
 fi
 
 printf "\nFor help and more information, visit https://forum.cloudron.io\n\n"

setup/start/cloudron-resize-fs.sh

@@ -4,6 +4,11 @@ set -eu -o pipefail
 
 readonly APPS_SWAP_FILE="/apps.swap"
 
+if [[ -f "${APPS_SWAP_FILE}" ]]; then
+    echo "Swap file already exists at /apps.swap . Skipping"
+    exit
+fi
+
 # all sizes are in mb
 readonly physical_memory=$(LC_ALL=C free -m | awk '/Mem:/ { print $2 }')
 readonly swap_size=$((${physical_memory} > 4096 ? 4096 : ${physical_memory})) # min(RAM, 4GB) if you change this, fix enoughResourcesAvailable() in client.js

setup/start/collectd/collectd.conf

@@ -121,7 +121,7 @@ LoadPlugin memory
 #LoadPlugin netlink
 #LoadPlugin network
 #LoadPlugin nfs
-LoadPlugin nginx
+#LoadPlugin nginx
 #LoadPlugin notify_desktop
 #LoadPlugin notify_email
 #LoadPlugin ntpd
@@ -149,7 +149,7 @@ LoadPlugin nginx
 #LoadPlugin statsd
 LoadPlugin swap
 #LoadPlugin table
-LoadPlugin tail
+#LoadPlugin tail
 #LoadPlugin tail_csv
 #LoadPlugin tcpconns
 #LoadPlugin teamspeak2
@@ -164,7 +164,9 @@ LoadPlugin tail
 #LoadPlugin vmem
 #LoadPlugin vserver
 #LoadPlugin wireless
-LoadPlugin write_graphite
+<LoadPlugin write_graphite>
+   FlushInterval 20
+</LoadPlugin>
 #LoadPlugin write_http
 #LoadPlugin write_riemann
 
@@ -197,42 +199,11 @@ LoadPlugin write_graphite
    IgnoreSelected false
 </Plugin>
 
-<Plugin nginx>
-   URL "http://127.0.0.1/nginx_status"
-</Plugin>
-
 <Plugin swap>
    ReportByDevice false
    ReportBytes true
 </Plugin>
 
-<Plugin "tail">
-   <File "/var/log/nginx/error.log">
-     Instance "nginx"
-     <Match>
-       Regex ".*"
-       DSType "CounterInc"
-       Type counter
-       Instance "errors"
-     </Match>
-   </File>
-   <File "/var/log/nginx/access.log">
-     Instance "nginx"
-     <Match>
-       Regex ".*"
-       DSType "CounterInc"
-       Type counter
-       Instance "requests"
-     </Match>
-    <Match>
-      Regex " \".*\" [0-9]+ [0-9]+ ([0-9]+)"
-      DSType GaugeAverage
-      Type delay
-      Instance "response"
-    </Match>
-   </File>
-</Plugin>
-
 <Plugin python>
     # https://blog.dbrgn.ch/2017/3/10/write-a-collectd-python-plugin/
     ModulePath "/home/yellowtent/box/setup/start/collectd/"

setup/start/collectd/df.py

@@ -6,7 +6,7 @@ disks = []
 
 def init():
     global disks
-    lines = [s.split() for s in subprocess.check_output(["df", "--type=ext4", "--output=source,target,size,used,avail"]).splitlines()]
+    lines = [s.split() for s in subprocess.check_output(["df", "--type=ext4", "--output=source,target,size,used,avail"]).decode('utf-8').splitlines()]
     disks = lines[1:] # strip header
     collectd.info('custom df plugin initialized with %s' % disks)
 

setup/start/collectd/du.py

@@ -6,19 +6,26 @@ PATHS = [] # { name, dir, exclude }
 # there is a pattern in carbon/storage-schemas.conf which stores values every 12h for a year
 INTERVAL = 60 * 60 * 12 # twice a day. change values in docker-graphite if you change this
 
+# we used to pass the INTERVAL as a parameter to register_read. however, collectd write_graphite
+# takes a bit to load (tcp connection) and drops the du data. this then means that we have to wait
+# for INTERVAL secs for du data. instead, we just cache the value for INTERVAL instead
+CACHE = dict()
+CACHE_TIME = 0
+
 def du(pathinfo):
     # -B1 makes du print block sizes and not apparent sizes (to match df which also uses block sizes)
-    cmd = 'timeout 1800 du -DsB1 "{}"'.format(pathinfo['dir'])
+    dirname = pathinfo['dir']
+    cmd = 'timeout 1800 du -DsB1 "{}"'.format(dirname)
     if pathinfo['exclude'] != '':
         cmd += ' --exclude "{}"'.format(pathinfo['exclude'])
 
     collectd.info('computing size with command: %s' % cmd);
     try:
         size = subprocess.check_output(cmd, shell=True).split()[0].decode('utf-8')
-        collectd.info('\tsize of %s is %s (time: %i)' % (pathinfo['dir'], size, int(time.time())))
+        collectd.info('\tsize of %s is %s (time: %i)' % (dirname, size, int(time.time())))
         return size
     except Exception as e:
-        collectd.info('\terror getting the size of %s: %s' % (pathinfo['dir'], str(e)))
+        collectd.info('\terror getting the size of %s: %s' % (dirname, str(e)))
         return 0
 
 def parseSize(size):
@@ -64,19 +71,35 @@ def init():
     collectd.info('custom du plugin initialized with %s %s' % (PATHS, sys.version))
 
 def read():
+    global CACHE, CACHE_TIME
+
+    # read from cache if < 12 hours
+    read_cache = (time.time() - CACHE_TIME) < INTERVAL
+
+    if not read_cache:
+        CACHE_TIME = time.time()
+
     for pathinfo in PATHS:
-        size = du(pathinfo)
+        dirname = pathinfo['dir']
+        if read_cache and dirname in CACHE:
+            size = CACHE[dirname]
+        else:
+            size = du(pathinfo)
+            CACHE[dirname] = size
 
         # type comes from https://github.com/collectd/collectd/blob/master/src/types.db
         val = collectd.Values(type='capacity', plugin='du', plugin_instance=pathinfo['name'])
         val.dispatch(values=[size], type_instance='usage')
 
-    size = dockerSize()
+    if read_cache and 'docker' in CACHE:
+        size = CACHE['docker']
+    else:
+        size = dockerSize()
+        CACHE['docker'] = size
+
     val = collectd.Values(type='capacity', plugin='du', plugin_instance='docker')
     val.dispatch(values=[size], type_instance='usage')
 
-
-
 collectd.register_init(init)
 collectd.register_config(configure)
-collectd.register_read(read, INTERVAL)
+collectd.register_read(read)

setup/start/mysql.cnf

@@ -4,9 +4,9 @@
 # http://bugs.mysql.com/bug.php?id=68514
 [mysqld]
 performance_schema=OFF
-max_connections=50
+max_connections=200
 # on ec2, without this we get a sporadic connection drop when doing the initial migration
-max_allowed_packet=32M
+max_allowed_packet=64M
 
 # https://mathiasbynens.be/notes/mysql-utf8mb4
 character-set-server = utf8mb4
@@ -15,6 +15,9 @@ collation-server = utf8mb4_unicode_ci
 # set timezone to UTC
 default_time_zone='+00:00'
 
+# disable bin logs. they are only useful in replication mode
+skip-log-bin
+
 [mysqldump]
 quick
 quote-names

setup/start/nginx/nginx.conf

@@ -1,11 +1,18 @@
 user www-data;
 
-worker_processes  1;
+# detect based on available CPU cores
+worker_processes  auto;
+
+# this is 4096 by default. See /proc/<PID>/limits and /etc/security/limits.conf
+# usually twice the worker_connections (one for uptsream, one for downstream)
+# see also LimitNOFILE=16384 in systemd drop-in
+worker_rlimit_nofile 8192;
 
 pid /run/nginx.pid;
 
 events {
-    worker_connections  1024;
+    # a single worker has these many simultaneous connections max
+    worker_connections  4096;
 }
 
 http {
@@ -36,23 +43,5 @@ http {
     # zones for rate limiting
     limit_req_zone $binary_remote_addr zone=admin_login:10m rate=10r/s; # 10 request a second
 
-
-    # default http server that returns 404 for any domain we are not listening on
-    server {
-        listen      80 default_server;
-        listen      [::]:80 default_server;
-        server_name does_not_match_anything;
-
-        # acme challenges (for app installation and re-configure when the vhost config does not exist)
-        location /.well-known/acme-challenge/ {
-            default_type text/plain;
-            alias /home/yellowtent/platformdata/acme/;
-        }
-
-        location / {
-            return 404;
-        }
-    }
-
     include applications/*.conf;
 }

setup/start/sudoers

@@ -13,9 +13,6 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/mkdirvolume.sh
 Defaults!/home/yellowtent/box/src/scripts/rmaddondir.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmaddondir.sh
 
-Defaults!/home/yellowtent/box/src/scripts/reloadnginx.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/reloadnginx.sh
-
 Defaults!/home/yellowtent/box/src/scripts/reboot.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/reboot.sh
 
@@ -25,9 +22,6 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/configurecollec
 Defaults!/home/yellowtent/box/src/scripts/collectlogs.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/collectlogs.sh
 
-Defaults!/home/yellowtent/box/src/scripts/retire.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/retire.sh
-
 Defaults!/home/yellowtent/box/src/scripts/update.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/update.sh
 
@@ -44,9 +38,18 @@ yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/scripts/backupup
 Defaults!/home/yellowtent/box/src/scripts/restart.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restart.sh
 
-Defaults!/home/yellowtent/box/src/scripts/restartdocker.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartdocker.sh
+Defaults!/home/yellowtent/box/src/scripts/restartservice.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartservice.sh
 
-Defaults!/home/yellowtent/box/src/scripts/restartunbound.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartunbound.sh
+Defaults!/home/yellowtent/box/src/scripts/rmmailbox.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmmailbox.sh
+
+Defaults!/home/yellowtent/box/src/scripts/starttask.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/scripts/starttask.sh
+
+Defaults!/home/yellowtent/box/src/scripts/stoptask.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/stoptask.sh
+
+Defaults!/home/yellowtent/box/src/scripts/setblocklist.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/setblocklist.sh
 

setup/start/systemd/box.service

@@ -1,20 +1,20 @@
 [Unit]
 Description=Cloudron Admin
 OnFailure=crashnotifier@%n.service
-StopWhenUnneeded=true
-; journald crashes result in a EPIPE in node. Cannot ignore it as it results in loss of logs.
-BindsTo=systemd-journald.service
 After=mysql.service nginx.service
 ; As cloudron-resize-fs is a one-shot, the Wants= automatically ensures that the service *finishes*
 Wants=cloudron-resize-fs.service
 
+[Install]
+WantedBy=multi-user.target
+
 [Service]
 Type=idle
 WorkingDirectory=/home/yellowtent/box
 Restart=always
-; Systemd does not append logs when logging to files, we spawn a shell first and exec to replace it after setting up the pipes
-ExecStart=/bin/sh -c 'echo "Logging to /home/yellowtent/platformdata/logs/box.log"; exec /usr/bin/node /home/yellowtent/box/box.js >> /home/yellowtent/platformdata/logs/box.log 2>&1'
-Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box*,connect-lastmile" "BOX_ENV=cloudron" "NODE_ENV=production"
+ExecStart=/home/yellowtent/box/box.js
+; we run commands like df which will parse properly only with correct locale
+Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box:*,connect-lastmile,-box:ldap" "BOX_ENV=cloudron" "NODE_ENV=production" "LC_ALL=C"
 ; kill apptask processes as well
 KillMode=control-group
 ; Do not kill this process on OOM. Children inherit this score. Do not set it to -1000 so that MemoryMax can keep working

setup/start/systemd/cloudron-disable-thp.service

@@ -0,0 +1,15 @@
+# https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
+[Unit]
+Description=Disable Transparent Huge Pages (THP)
+DefaultDependencies=no
+After=sysinit.target local-fs.target
+Before=docker.service
+
+[Service]
+Type=oneshot
+ExecStart="/home/yellowtent/box/setup/start/cloudron-disable-thp.sh"
+RemainAfterExit=yes
+
+[Install]
+WantedBy=basic.target
+

setup/start/systemd/cloudron-firewall.service

@@ -5,6 +5,7 @@ PartOf=docker.service
 
 [Service]
 Type=oneshot
+Environment="BOX_ENV=cloudron"
 ExecStart="/home/yellowtent/box/setup/start/cloudron-firewall.sh"
 RemainAfterExit=yes
 

setup/start/systemd/cloudron.target

@@ -1,10 +0,0 @@
-[Unit]
-Description=Cloudron Smartserver
-Documentation=https://cloudron.io/documentation.html
-StopWhenUnneeded=true
-Requires=box.service
-After=box.service
-# AllowIsolate=yes
-
-[Install]
-WantedBy=multi-user.target

setup/start/systemd/unbound.service

@@ -2,7 +2,7 @@
 
 [Unit]
 Description=Unbound DNS Resolver
-After=network.target
+After=network.target docker.service
 
 [Service]
 PIDFile=/run/unbound.pid

setup/start/unbound.conf

@@ -1,5 +1,7 @@
 server:
-        interface: 0.0.0.0
+        port: 53
+        interface: 127.0.0.1
+        interface: 172.18.0.1
         do-ip6: no
         access-control: 127.0.0.1 allow
         access-control: 172.18.0.1/16 allow

setup/stop.sh

@@ -4,4 +4,4 @@ set -eu -o pipefail
 
 echo "Stopping cloudron"
 
-systemctl stop cloudron.target
+systemctl stop box

src/appdb.js

@@ -1,32 +1,32 @@
 'use strict';
 
 exports = module.exports = {
-    get: get,
-    getByHttpPort: getByHttpPort,
-    getByContainerId: getByContainerId,
-    add: add,
-    exists: exists,
-    del: del,
-    update: update,
-    getAll: getAll,
-    getPortBindings: getPortBindings,
-    delPortBinding: delPortBinding,
+    get,
+    add,
+    exists,
+    del,
+    update,
+    getAll,
+    getPortBindings,
+    delPortBinding,
 
-    setAddonConfig: setAddonConfig,
-    getAddonConfig: getAddonConfig,
-    getAddonConfigByAppId: getAddonConfigByAppId,
-    getAddonConfigByName: getAddonConfigByName,
-    unsetAddonConfig: unsetAddonConfig,
-    unsetAddonConfigByAppId: unsetAddonConfigByAppId,
-    getAppIdByAddonConfigValue: getAppIdByAddonConfigValue,
+    setAddonConfig,
+    getAddonConfig,
+    getAddonConfigByAppId,
+    getAddonConfigByName,
+    unsetAddonConfig,
+    unsetAddonConfigByAppId,
+    getAppIdByAddonConfigValue,
+    getByIpAddress,
 
-    setHealth: setHealth,
-    setTask: setTask,
-    getAppStoreIds: getAppStoreIds,
+    setHealth,
+    setTask,
+    getAppStoreIds,
 
     // subdomain table types
     SUBDOMAIN_TYPE_PRIMARY: 'primary',
     SUBDOMAIN_TYPE_REDIRECT: 'redirect',
+    SUBDOMAIN_TYPE_ALIAS: 'alias',
 
     _clear: clear
 };
@@ -38,17 +38,14 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
-    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'subdomains.subdomain AS location', 'subdomains.domain',
-    'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
-    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson', 'apps.bindsJson',
-    'apps.sso', 'apps.debugModeJson', 'apps.enableBackup',
+const APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
+    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
+    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson',
+    'apps.sso', 'apps.debugModeJson', 'apps.enableBackup', 'apps.proxyAuth', 'apps.containerIp',
     'apps.creationTime', 'apps.updateTime', 'apps.mailboxName', 'apps.mailboxDomain', 'apps.enableAutomaticUpdate',
     'apps.dataDir', 'apps.ts', 'apps.healthTime' ].join(',');
 
-var PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');
-
-const SUBDOMAIN_FIELDS = [ 'appId', 'domain', 'subdomain', 'type' ].join(',');
+const PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');
 
 function postProcess(result) {
     assert.strictEqual(typeof result, 'object');
@@ -89,6 +86,7 @@ function postProcess(result) {
     result.sso = !!result.sso; // make it bool
     result.enableBackup = !!result.enableBackup; // make it bool
     result.enableAutomaticUpdate = !!result.enableAutomaticUpdate; // make it bool
+    result.proxyAuth = !!result.proxyAuth;
 
     assert(result.debugModeJson === null || typeof result.debugModeJson === 'string');
     result.debugMode = safe.JSON.parse(result.debugModeJson);
@@ -98,15 +96,23 @@ function postProcess(result) {
     result.servicesConfig = safe.JSON.parse(result.servicesConfigJson) || {};
     delete result.servicesConfigJson;
 
-    assert(result.bindsJson === null || typeof result.bindsJson === 'string');
-    result.binds = safe.JSON.parse(result.bindsJson) || {};
-    delete result.bindsJson;
+    let subdomains = JSON.parse(result.subdomains), domains = JSON.parse(result.domains), subdomainTypes = JSON.parse(result.subdomainTypes);
+    delete result.subdomains;
+    delete result.domains;
+    delete result.subdomainTypes;
 
-    result.alternateDomains = result.alternateDomains || [];
-    result.alternateDomains.forEach(function (d) {
-        delete d.appId;
-        delete d.type;
-    });
+    result.alternateDomains = [];
+    result.aliasDomains = [];
+    for (let i = 0; i < subdomainTypes.length; i++) {
+        if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_PRIMARY) {
+            result.location = subdomains[i];
+            result.domain = domains[i];
+        } else if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_REDIRECT) {
+            result.alternateDomains.push({ domain: domains[i], subdomain: subdomains[i] });
+        } else if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_ALIAS) {
+            result.aliasDomains.push({ domain: domains[i], subdomain: subdomains[i] });
+        }
+    }
 
     let envNames = JSON.parse(result.envNames), envValues = JSON.parse(result.envValues);
     delete result.envNames;
@@ -116,119 +122,67 @@ function postProcess(result) {
         if (envNames[i]) result.env[envNames[i]] = envValues[i];
     }
 
+    let volumeIds = JSON.parse(result.volumeIds);
+    delete result.volumeIds;
+    let volumeReadOnlys = JSON.parse(result.volumeReadOnlys);
+    delete result.volumeReadOnlys;
+
+    result.mounts = volumeIds[0] === null ? [] : volumeIds.map((v, idx) => { return { volumeId: v, readOnly: !!volumeReadOnlys[idx] }; }); // NOTE: volumeIds is [null] when volumes of an app is empty
+
     result.error = safe.JSON.parse(result.errorJson);
     delete result.errorJson;
 
     result.taskId = result.taskId ? String(result.taskId) : null;
 }
 
+// each query simply join apps table with another table by id. we then join the full result together
+const PB_QUERY = 'SELECT id, GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes FROM apps LEFT JOIN appPortBindings ON apps.id = appPortBindings.appId GROUP BY apps.id';
+const ENV_QUERY = 'SELECT id, JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues FROM apps LEFT JOIN appEnvVars ON apps.id = appEnvVars.appId GROUP BY apps.id';
+const SUBDOMAIN_QUERY = 'SELECT id, JSON_ARRAYAGG(subdomains.subdomain) AS subdomains, JSON_ARRAYAGG(subdomains.domain) AS domains, JSON_ARRAYAGG(subdomains.type) AS subdomainTypes FROM apps LEFT JOIN subdomains ON apps.id = subdomains.appId GROUP BY apps.id';
+const MOUNTS_QUERY = 'SELECT id, JSON_ARRAYAGG(appMounts.volumeId) AS volumeIds, JSON_ARRAYAGG(appMounts.readOnly) AS volumeReadOnlys FROM apps LEFT JOIN appMounts ON apps.id = appMounts.appId GROUP BY apps.id';
+const APPS_QUERY = `SELECT ${APPS_FIELDS_PREFIXED}, hostPorts, environmentVariables, portTypes, envNames, envValues, subdomains, domains, subdomainTypes, volumeIds, volumeReadOnlys FROM apps`
+    + ` LEFT JOIN (${PB_QUERY}) AS q1 on q1.id = apps.id`
+    + ` LEFT JOIN (${ENV_QUERY}) AS q2 on q2.id = apps.id`
+    + ` LEFT JOIN (${SUBDOMAIN_QUERY}) AS q3 on q3.id = apps.id`
+    + ` LEFT JOIN (${MOUNTS_QUERY}) AS q4 on q4.id = apps.id`;
+
 function get(id, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes, '
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE apps.id = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, id ], function (error, result) {
+    database.query(`${APPS_QUERY} WHERE apps.id = ?`, [ id ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        postProcess(result[0]);
 
-            result[0].alternateDomains = alternateDomains;
-
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
+        callback(null, result[0]);
     });
 }
 
-function getByHttpPort(httpPort, callback) {
-    assert.strictEqual(typeof httpPort, 'number');
+function getByIpAddress(ip, callback) {
+    assert.strictEqual(typeof ip, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-    + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-    + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE httpPort = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, httpPort ], function (error, result) {
+    database.query(`${APPS_QUERY} WHERE apps.containerIp = ?`, [ ip ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        postProcess(result[0]);
 
-            result[0].alternateDomains = alternateDomains;
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
-    });
-}
-
-function getByContainerId(containerId, callback) {
-    assert.strictEqual(typeof containerId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE containerId = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, containerId ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
-
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-            result[0].alternateDomains = alternateDomains;
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
+        callback(null, result[0]);
     });
 }
 
 function getAll(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' GROUP BY apps.id ORDER BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY ], function (error, results) {
+    database.query(`${APPS_QUERY} ORDER BY apps.id`, [ ], function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE type = ?', [ exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        results.forEach(postProcess);
 
-            alternateDomains.forEach(function (d) {
-                var domain = results.find(function (a) { return d.appId === a.id; });
-                if (!domain) return;
-
-                domain.alternateDomains = domain.alternateDomains || [];
-                domain.alternateDomains.push(d);
-            });
-
-            results.forEach(postProcess);
-
-            callback(null, results);
-        });
+        callback(null, results);
     });
 }
 
@@ -300,6 +254,15 @@ function add(id, appStoreId, manifest, location, domain, portBindings, data, cal
         });
     }
 
+    if (data.aliasDomains) {
+        data.aliasDomains.forEach(function (d) {
+            queries.push({
+                query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)',
+                args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_ALIAS ]
+            });
+        });
+    }
+
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, 'no such domain'));
@@ -358,12 +321,13 @@ function del(id, callback) {
         { query: 'DELETE FROM appPortBindings WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appEnvVars WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appPasswords WHERE identifier = ?', args: [ id ] },
+        { query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM apps WHERE id = ?', args: [ id ] }
     ];
 
     database.transaction(queries, function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results[4].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (results[5].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
         callback(null);
     });
@@ -396,6 +360,7 @@ function updateWithConstraints(id, app, constraints, callback) {
     assert(!('portBindings' in app) || typeof app.portBindings === 'object');
     assert(!('accessRestriction' in app) || typeof app.accessRestriction === 'object' || app.accessRestriction === '');
     assert(!('alternateDomains' in app) || Array.isArray(app.alternateDomains));
+    assert(!('aliasDomains' in app) || Array.isArray(app.aliasDomains));
     assert(!('tags' in app) || Array.isArray(app.tags));
     assert(!('env' in app) || typeof app.env === 'object');
 
@@ -423,22 +388,35 @@ function updateWithConstraints(id, app, constraints, callback) {
     }
 
     if ('location' in app && 'domain' in app) { // must be updated together as they are unique together
-        queries.push({ query: 'UPDATE subdomains SET subdomain = ?, domain = ? WHERE appId = ? AND type = ?', args: [ app.location, app.domain, id, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ?', args: [ id ]}); // all locations of an app must be updated together
+        queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, app.domain, app.location, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+
+        if ('alternateDomains' in app) {
+            app.alternateDomains.forEach(function (d) {
+                queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+            });
+        }
+
+        if ('aliasDomains' in app) {
+            app.aliasDomains.forEach(function (d) {
+                queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_ALIAS ]});
+            });
+        }
     }
 
-    if ('alternateDomains' in app) {
-        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ? AND type = ?', args: [ id, exports.SUBDOMAIN_TYPE_REDIRECT ]});
-        app.alternateDomains.forEach(function (d) {
-            queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+    if ('mounts' in app) {
+        queries.push({ query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ]});
+        app.mounts.forEach(function (m) {
+            queries.push({ query: 'INSERT INTO appMounts (appId, volumeId, readOnly) VALUES (?, ?, ?)', args: [ id, m.volumeId, m.readOnly ]});
         });
     }
 
     var fields = [ ], values = [ ];
     for (var p in app) {
-        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig' || p === 'binds') {
+        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig') {
             fields.push(`${p}Json = ?`);
             values.push(JSON.stringify(app[p]));
-        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env') {
+        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'aliasDomains' && p !== 'env' && p !== 'mounts') {
             fields.push(p + ' = ?');
             values.push(app[p]);
         }

src/apphealthmonitor.js

@@ -10,49 +10,48 @@ var appdb = require('./appdb.js'),
     docker = require('./docker.js'),
     eventlog = require('./eventlog.js'),
     safe = require('safetydance'),
-    superagent = require('superagent'),
-    util = require('util');
+    superagent = require('superagent');
 
 exports = module.exports = {
-    run: run
+    run
 };
 
 const HEALTHCHECK_INTERVAL = 10 * 1000; // every 10 seconds. this needs to be small since the UI makes only healthy apps clickable
-const UNHEALTHY_THRESHOLD = 10 * 60 * 1000; // 10 minutes
+const UNHEALTHY_THRESHOLD = 20 * 60 * 1000; // 20 minutes
 
 const OOM_EVENT_LIMIT = 60 * 60 * 1000; // 60 minutes
+let gStartTime = null; // time when apphealthmonitor was started
 let gLastOomMailTime = Date.now() - (5 * 60 * 1000); // pretend we sent email 5 minutes ago
 
-function debugApp(app) {
-    assert(typeof app === 'object');
-
-    debug(app.fqdn + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)) + ' - ' + app.id);
-}
-
 function setHealth(app, health, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof health, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    let now = new Date(), healthTime = app.healthTime, curHealth = app.health;
+    // app starts out with null health
+    // if it became healthy, we update immediately. this is required for ui to say "running" etc
+    // if it became unhealthy/error/dead, wait for a threshold before updating db
+
+    const now = new Date(), lastHealth = app.health;
+    let healthTime = gStartTime > app.healthTime ? gStartTime : app.healthTime; // on box restart, clamp value to start time
 
     if (health === apps.HEALTH_HEALTHY) {
         healthTime = now;
-        if (curHealth && curHealth !== apps.HEALTH_HEALTHY) { // app starts out with null health
-            debugApp(app, 'app switched from %s to healthy', curHealth);
+        if (lastHealth && lastHealth !== apps.HEALTH_HEALTHY) { // app starts out with null health
+            debug(`setHealth: ${app.id} (${app.fqdn}) switched from ${lastHealth} to healthy`);
 
             // do not send mails for dev apps
             if (!app.debugMode) eventlog.add(eventlog.ACTION_APP_UP, auditSource.HEALTH_MONITOR, { app: app });
         }
     } else if (Math.abs(now - healthTime) > UNHEALTHY_THRESHOLD) {
-        if (curHealth === apps.HEALTH_HEALTHY) {
-            debugApp(app, 'marking as unhealthy since not seen for more than %s minutes', UNHEALTHY_THRESHOLD/(60 * 1000));
+        if (lastHealth === apps.HEALTH_HEALTHY) {
+            debug(`setHealth: marking ${app.id} (${app.fqdn}) as unhealthy since not seen for more than ${UNHEALTHY_THRESHOLD/(60 * 1000)} minutes`);
 
             // do not send mails for dev apps
             if (!app.debugMode) eventlog.add(eventlog.ACTION_APP_DOWN, auditSource.HEALTH_MONITOR, { app: app });
         }
     } else {
-        debugApp(app, 'waiting for %s seconds to update the app health', (UNHEALTHY_THRESHOLD - Math.abs(now - healthTime))/1000);
+        debug(`setHealth: ${app.id} (${app.fqdn}) waiting for ${(UNHEALTHY_THRESHOLD - Math.abs(now - healthTime))/1000} to update health`);
         return callback(null);
     }
 
@@ -61,6 +60,7 @@ function setHealth(app, health, callback) {
         if (error) return callback(error);
 
         app.health = health;
+        app.healthTime = healthTime;
 
         callback(null);
     });
@@ -79,21 +79,13 @@ function checkAppHealth(app, callback) {
     const manifest = app.manifest;
 
     docker.inspect(app.containerId, function (error, data) {
-        if (error || !data || !data.State) {
-            debugApp(app, 'Error inspecting container');
-            return setHealth(app, apps.HEALTH_ERROR, callback);
-        }
-
-        if (data.State.Running !== true) {
-            debugApp(app, 'exited');
-            return setHealth(app, apps.HEALTH_DEAD, callback);
-        }
+        if (error || !data || !data.State) return setHealth(app, apps.HEALTH_ERROR, callback);
+        if (data.State.Running !== true) return setHealth(app, apps.HEALTH_DEAD, callback);
 
         // non-appstore apps may not have healthCheckPath
         if (!manifest.healthCheckPath) return setHealth(app, apps.HEALTH_HEALTHY, callback);
 
-        // poll through docker network instead of nginx to bypass any potential oauth proxy
-        var healthCheckUrl = 'http://127.0.0.1:' + app.httpPort + manifest.healthCheckPath;
+        const healthCheckUrl = `http://${app.containerIp}:${manifest.httpPort}${manifest.healthCheckPath}`;
         superagent
             .get(healthCheckUrl)
             .set('Host', app.fqdn) // required for some apache configs with rewrite rules
@@ -103,7 +95,7 @@ function checkAppHealth(app, callback) {
             .end(function (error, res) {
                 if (error && !error.response) {
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
-                } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
+                } else if (res.statusCode >= 403) { // 2xx and 3xx are ok. even 401 and 403 are ok for now (for WP sites)
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
                 } else {
                     setHealth(app, apps.HEALTH_HEALTHY, callback);
@@ -126,7 +118,7 @@ function getContainerInfo(containerId, callback) {
 
 /*
     OOM can be tested using stress tool like so:
-        docker run -ti -m 100M cloudron/base:0.10.0 /bin/bash
+        docker run -ti -m 100M cloudron/base:2.0.0 /bin/bash
         apt-get update && apt-get install stress
         stress --vm 1 --vm-bytes 200M --vm-hang 0
 */
@@ -181,12 +173,10 @@ function processApp(callback) {
         if (error) return callback(error);
 
         async.each(allApps, checkAppHealth, function (error) {
-            if (error) console.error(error);
-
             const alive = allApps
                 .filter(function (a) { return a.installationState === apps.ISTATE_INSTALLED && a.runState === apps.RSTATE_RUNNING && a.health === apps.HEALTH_HEALTHY; });
 
-            debug(`app health: ${alive.length} alive / ${allApps.length - alive.length} dead`);
+            debug(`app health: ${alive.length} alive / ${allApps.length - alive.length} dead.` + (error ? ` ${error.reason}` : ''));
 
             callback(null);
         });
@@ -197,11 +187,13 @@ function run(intervalSecs, callback) {
     assert.strictEqual(typeof intervalSecs, 'number');
     assert.strictEqual(typeof callback, 'function');
 
+    if (!gStartTime) gStartTime = new Date();
+
     async.series([
         processApp, // this is first because docker.getEvents seems to get 'stuck' sometimes
         processDockerEvents.bind(null, intervalSecs)
     ], function (error) {
-        if (error) debug(error);
+        if (error) debug(`run: could not check app health. ${error.message}`);
 
         callback();
     });

src/apps.js

@@ -1,70 +1,72 @@
 'use strict';
 
 exports = module.exports = {
-    hasAccessTo: hasAccessTo,
-    removeInternalFields: removeInternalFields,
-    removeRestrictedFields: removeRestrictedFields,
+    hasAccessTo,
+    removeInternalFields,
+    removeRestrictedFields,
 
-    get: get,
-    getByContainerId: getByContainerId,
-    getByIpAddress: getByIpAddress,
-    getByFqdn: getByFqdn,
-    getAll: getAll,
-    getAllByUser: getAllByUser,
-    install: install,
-    uninstall: uninstall,
+    get,
+    getByIpAddress,
+    getByFqdn,
+    getAll,
+    getAllByUser,
+    install,
+    uninstall,
 
-    setAccessRestriction: setAccessRestriction,
-    setLabel: setLabel,
-    setIcon: setIcon,
-    setTags: setTags,
-    setMemoryLimit: setMemoryLimit,
-    setCpuShares: setCpuShares,
-    setBinds: setBinds,
-    setAutomaticBackup: setAutomaticBackup,
-    setAutomaticUpdate: setAutomaticUpdate,
-    setReverseProxyConfig: setReverseProxyConfig,
-    setCertificate: setCertificate,
-    setDebugMode: setDebugMode,
-    setEnvironment: setEnvironment,
-    setMailbox: setMailbox,
-    setLocation: setLocation,
-    setDataDir: setDataDir,
-    repair: repair,
+    setAccessRestriction,
+    setLabel,
+    setIcon,
+    setTags,
+    setMemoryLimit,
+    setCpuShares,
+    setMounts,
+    setAutomaticBackup,
+    setAutomaticUpdate,
+    setReverseProxyConfig,
+    setCertificate,
+    setDebugMode,
+    setEnvironment,
+    setMailbox,
+    setLocation,
+    setDataDir,
+    repair,
 
-    restore: restore,
-    importApp: importApp,
-    clone: clone,
+    restore,
+    importApp,
+    exportApp,
+    clone,
 
-    update: update,
+    update,
 
-    backup: backup,
-    listBackups: listBackups,
+    backup,
+    listBackups,
 
-    getLogs: getLogs,
+    getLocalLogfilePaths,
+    getLogs,
 
-    start: start,
-    stop: stop,
-    restart: restart,
+    start,
+    stop,
+    restart,
 
-    exec: exec,
+    exec,
 
-    checkManifestConstraints: checkManifestConstraints,
-    downloadManifest: downloadManifest,
+    checkManifestConstraints,
+    downloadManifest,
 
-    canAutoupdateApp: canAutoupdateApp,
-    autoupdateApps: autoupdateApps,
+    canAutoupdateApp,
+    autoupdateApps,
 
-    restoreInstalledApps: restoreInstalledApps,
-    configureInstalledApps: configureInstalledApps,
-    schedulePendingTasks: schedulePendingTasks,
-    restartAppsUsingAddons: restartAppsUsingAddons,
+    restoreInstalledApps,
+    configureInstalledApps,
+    schedulePendingTasks,
+    restartAppsUsingAddons,
 
-    getDataDir: getDataDir,
-    getIconPath: getIconPath,
+    getDataDir,
+    getIconPath,
+    getMemoryLimit,
 
-    downloadFile: downloadFile,
-    uploadFile: uploadFile,
+    downloadFile,
+    uploadFile,
 
     PORT_TYPE_TCP: 'tcp',
     PORT_TYPE_UDP: 'udp',
@@ -134,7 +136,6 @@ var appdb = require('./appdb.js'),
     superagent = require('superagent'),
     tasks = require('./tasks.js'),
     TransformStream = require('stream').Transform,
-    updateChecker = require('./updatechecker.js'),
     users = require('./users.js'),
     util = require('util'),
     uuid = require('uuid'),
@@ -154,7 +155,6 @@ function validatePortBindings(portBindings, manifest) {
     const RESERVED_PORTS = [
         22, /* ssh */
         25, /* smtp */
-        53, /* dns */
         80, /* http */
         143, /* imap */
         202, /* alternate ssh */
@@ -167,7 +167,7 @@ function validatePortBindings(portBindings, manifest) {
         2004, /* graphite (lo) */
         2514, /* cloudron-syslog (lo) */
         constants.PORT, /* app server (lo) */
-        constants.SYSADMIN_PORT, /* sysadmin app server (lo) */
+        constants.AUTHWALL_PORT, /* protected sites */
         constants.INTERNAL_SMTP_PORT, /* internal smtp port (lo) */
         constants.LDAP_PORT,
         3306, /* mysql (lo) */
@@ -182,6 +182,11 @@ function validatePortBindings(portBindings, manifest) {
         [50000, 51000] /* turn udp ports */
     ];
 
+    const ALLOWED_PORTS = [
+        53, // dns 53 is special and adblocker apps can use them
+        853 // dns over tls
+    ];
+
     if (!portBindings) return null;
 
     for (let portName in portBindings) {
@@ -191,7 +196,7 @@ function validatePortBindings(portBindings, manifest) {
         if (!Number.isInteger(hostPort)) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not an integer`, { field: 'portBindings', portName: portName });
         if (RESERVED_PORTS.indexOf(hostPort) !== -1) return new BoxError(BoxError.BAD_FIELD, `Port ${hostPort} is reserved.`, { field: 'portBindings', portName: portName });
         if (RESERVED_PORT_RANGES.find(range => (hostPort >= range[0] && hostPort <= range[1]))) return new BoxError(BoxError.BAD_FIELD, `Port ${hostPort} is reserved.`, { field: 'portBindings', portName: portName });
-        if (hostPort <= 1023 || hostPort > 65535) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not in permitted range`, { field: 'portBindings', portName: portName });
+        if (ALLOWED_PORTS.indexOf(hostPort) === -1 && (hostPort <= 1023 || hostPort > 65535)) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not in permitted range`, { field: 'portBindings', portName: portName });
     }
 
     // it is OK if there is no 1-1 mapping between values in manifest.tcpPorts and portBindings. missing values implies
@@ -334,20 +339,6 @@ function validateEnv(env) {
     return null;
 }
 
-function validateBinds(binds) {
-    for (let name of Object.keys(binds)) {
-        // just have friendly characters under /media
-        if (!/^[-0-9a-zA-Z_@$=#.%+]+$/.test(name)) return new BoxError(BoxError.BAD_FIELD, `Invalid bind name: ${name}`);
-
-        const bind = binds[name];
-
-        if (!bind.hostPath.startsWith('/mnt') && !bind.hostPath.startsWith('/media')) return new BoxError(BoxError.BAD_FIELD, 'hostPath must be in /mnt or /media');
-        if (path.normalize(bind.hostPath) !== bind.hostPath) return new BoxError(BoxError.BAD_FIELD, 'hostPath is not normalized');
-    }
-
-    return null;
-}
-
 function validateDataDir(dataDir) {
     if (dataDir === null) return null;
 
@@ -398,7 +389,7 @@ function getDuplicateErrorDetails(errorMessage, locations, domainObjectMap, port
 
     // check if any of the port bindings conflict
     for (let portName in portBindings) {
-        if (portBindings[portName] === parseInt(match[1])) return new BoxError(BoxError.ALREADY_EXISTS, `Port ${match[1]} is reserved`, { portName });
+        if (portBindings[portName] === parseInt(match[1])) return new BoxError(BoxError.ALREADY_EXISTS, `Port ${match[1]} is in use`, { portName });
     }
 
     if (match[2] === 'dataDir') {
@@ -420,14 +411,14 @@ function removeInternalFields(app) {
         'location', 'domain', 'fqdn', 'mailboxName', 'mailboxDomain',
         'accessRestriction', 'manifest', 'portBindings', 'iconUrl', 'memoryLimit', 'cpuShares',
         'sso', 'debugMode', 'reverseProxyConfig', 'enableBackup', 'creationTime', 'updateTime', 'ts', 'tags',
-        'label', 'alternateDomains', 'env', 'enableAutomaticUpdate', 'dataDir', 'binds');
+        'label', 'alternateDomains', 'aliasDomains', 'env', 'enableAutomaticUpdate', 'dataDir', 'mounts');
 }
 
 // non-admins can only see these
 function removeRestrictedFields(app) {
     return _.pick(app,
-        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId',
-        'location', 'domain', 'fqdn', 'manifest', 'portBindings', 'iconUrl', 'creationTime', 'ts', 'tags', 'label');
+        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId', 'accessRestriction', 'alternateDomains', 'aliasDomains', 'sso',
+        'location', 'domain', 'fqdn', 'manifest', 'portBindings', 'iconUrl', 'creationTime', 'ts', 'tags', 'label', 'enableBackup');
 }
 
 function getIconUrlSync(app) {
@@ -460,6 +451,20 @@ function getIconPath(app, options, callback) {
     callback(new BoxError(BoxError.NOT_FOUND, 'No icon'));
 }
 
+function getMemoryLimit(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    let memoryLimit = app.memoryLimit || app.manifest.memoryLimit || 0;
+
+    if (memoryLimit === -1) { // unrestricted
+        memoryLimit = 0;
+    } else if (memoryLimit === 0 || memoryLimit < constants.DEFAULT_MEMORY_LIMIT) { // ensure we never go below minimum (in case we change the default)
+        memoryLimit = constants.DEFAULT_MEMORY_LIMIT;
+    }
+
+    return memoryLimit;
+}
+
 function postProcess(app, domainObjectMap) {
     let result = {};
     for (let portName in app.portBindings) {
@@ -470,6 +475,7 @@ function postProcess(app, domainObjectMap) {
     app.iconUrl = getIconUrlSync(app);
     app.fqdn = domains.fqdn(app.location, domainObjectMap[app.domain]);
     app.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+    app.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 }
 
 function hasAccessTo(app, user, callback) {
@@ -512,25 +518,6 @@ function get(appId, callback) {
         if (error) return callback(error);
 
         appdb.get(appId, function (error, app) {
-            if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
-            if (error) return callback(error);
-
-            postProcess(app, domainObjectMap);
-
-            callback(null, app);
-        });
-    });
-}
-
-function getByContainerId(containerId, callback) {
-    assert.strictEqual(typeof containerId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    getDomainObjectMap(function (error, domainObjectMap) {
-        if (error) return callback(error);
-
-        appdb.getByContainerId(containerId, function (error, app) {
-            if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
             if (error) return callback(error);
 
             postProcess(app, domainObjectMap);
@@ -548,16 +535,15 @@ function getByIpAddress(ip, callback) {
     // this is only used by the ldap test. the apps tests still uses proper docker
     if (constants.TEST && exports._MOCK_GET_BY_IP_APP_ID) return get(exports._MOCK_GET_BY_IP_APP_ID, callback);
 
-    docker.getContainerIdByIp(ip, function (error, containerId) {
+    appdb.getByIpAddress(ip, function (error, app) {
         if (error) return callback(error);
 
-        docker.inspect(containerId, function (error, result) {
+        getDomainObjectMap(function (error, domainObjectMap) {
             if (error) return callback(error);
 
-            const appId = safe.query(result, 'Config.Labels.appId', null);
-            if (!appId) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
+            postProcess(app, domainObjectMap);
 
-            get(appId, callback);
+            callback(null, app);
         });
     });
 }
@@ -639,20 +625,31 @@ function scheduleTask(appId, installationState, taskId, callback) {
     assert.strictEqual(typeof taskId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    appTaskManager.scheduleTask(appId, taskId, function (error) {
-        debug(`scheduleTask: task ${taskId} of ${appId} completed`);
-        if (error && (error.code === tasks.ECRASHED || error.code === tasks.ESTOPPED)) { // if task crashed, update the error
-            debug(`Apptask crashed/stopped: ${error.message}`);
-            let boxError = new BoxError(BoxError.TASK_ERROR, error.message);
-            boxError.details.crashed = error.code === tasks.ECRASHED;
-            boxError.details.stopped = error.code === tasks.ESTOPPED;
-            // see also apptask makeTaskError
-            boxError.details.taskId = taskId;
-            boxError.details.installationState = installationState;
-            appdb.update(appId, { installationState: exports.ISTATE_ERROR, error: boxError.toPlainObject(), taskId: null }, callback);
-        } else if (!(installationState === exports.ISTATE_PENDING_UNINSTALL && !error)) { // clear out taskId except for successful uninstall
-            appdb.update(appId, { taskId: null }, callback);
+    settings.getBackupConfig(function (error, backupConfig) {
+        if (error) return callback(error);
+
+        let memoryLimit = 400;
+        if (installationState === exports.ISTATE_PENDING_BACKUP || installationState === exports.ISTATE_PENDING_CLONE || installationState === exports.ISTATE_PENDING_RESTORE || installationState === exports.ISTATE_PENDING_UPDATE) {
+            memoryLimit = 'memoryLimit' in backupConfig ? Math.max(backupConfig.memoryLimit/1024/1024, 400) : 400;
         }
+
+        const options = { timeout: 20 * 60 * 60 * 1000 /* 20 hours */, nice: 15, memoryLimit };
+
+        appTaskManager.scheduleTask(appId, taskId, options, function (error) {
+            debug(`scheduleTask: task ${taskId} of ${appId} completed`);
+            if (error && (error.code === tasks.ECRASHED || error.code === tasks.ESTOPPED)) { // if task crashed, update the error
+                debug(`Apptask crashed/stopped: ${error.message}`);
+                let boxError = new BoxError(BoxError.TASK_ERROR, error.message);
+                boxError.details.crashed = error.code === tasks.ECRASHED;
+                boxError.details.stopped = error.code === tasks.ESTOPPED;
+                // see also apptask makeTaskError
+                boxError.details.taskId = taskId;
+                boxError.details.installationState = installationState;
+                appdb.update(appId, { installationState: exports.ISTATE_ERROR, error: boxError.toPlainObject(), taskId: null }, callback);
+            } else if (!(installationState === exports.ISTATE_PENDING_UNINSTALL && !error)) { // clear out taskId except for successful uninstall
+                appdb.update(appId, { taskId: null }, callback);
+            }
+        });
     });
 }
 
@@ -696,6 +693,11 @@ function checkAppState(app, state) {
         if (state !== exports.ISTATE_PENDING_UNINSTALL && state !== exports.ISTATE_PENDING_RESTORE) return new BoxError(BoxError.BAD_STATE, 'Not allowed in error state');
     }
 
+    if (app.runState === exports.RSTATE_STOPPED) {
+        // can't backup or restore since app addons are down. can't update because migration scripts won't run
+        if (state === exports.ISTATE_PENDING_UPDATE || state === exports.ISTATE_PENDING_BACKUP || state === exports.ISTATE_PENDING_RESTORE)  return new BoxError(BoxError.BAD_STATE, 'Not allowed in stopped state');
+    }
+
     return null;
 }
 
@@ -709,7 +711,13 @@ function validateLocations(locations, callback) {
         for (let location of locations) {
             if (!(location.domain in domainObjectMap)) return callback(new BoxError(BoxError.BAD_FIELD, 'No such domain', { field: 'location', domain: location.domain, subdomain: location.subdomain }));
 
-            error = domains.validateHostname(location.subdomain, domainObjectMap[location.domain]);
+            let subdomain = location.subdomain;
+            if (location.type === 'alias' && subdomain.startsWith('*')) {
+                if (subdomain === '*') continue;
+                subdomain = subdomain.replace(/^\*\./, ''); // remove *.
+            }
+
+            error = domains.validateHostname(subdomain, domainObjectMap[location.domain]);
             if (error) return callback(new BoxError(BoxError.BAD_FIELD, 'Bad location: ' + error.message, { field: 'location', domain: location.domain, subdomain: location.subdomain }));
         }
 
@@ -741,10 +749,12 @@ function install(data, auditSource, callback) {
         enableBackup = 'enableBackup' in data ? data.enableBackup : true,
         enableAutomaticUpdate = 'enableAutomaticUpdate' in data ? data.enableAutomaticUpdate : true,
         alternateDomains = data.alternateDomains || [],
+        aliasDomains = data.aliasDomains || [],
         env = data.env || {},
         label = data.label || null,
         tags = data.tags || [],
         overwriteDns = 'overwriteDns' in data ? data.overwriteDns : false,
+        skipDnsSetup = 'skipDnsSetup' in data ? data.skipDnsSetup : false,
         appStoreId = data.appStoreId,
         manifest = data.manifest;
 
@@ -774,7 +784,7 @@ function install(data, auditSource, callback) {
 
     if ('sso' in data && !('optionalSso' in manifest)) return callback(new BoxError(BoxError.BAD_FIELD, 'sso can only be specified for apps with optionalSso'));
     // if sso was unspecified, enable it by default if possible
-    if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['oauth'];
+    if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['proxyAuth'];
 
     error = validateEnv(env);
     if (error) return callback(error);
@@ -793,7 +803,10 @@ function install(data, auditSource, callback) {
         }
     }
 
-    const locations = [{subdomain: location, domain}].concat(alternateDomains);
+    const locations = [{ subdomain: location, domain, type: 'primary' }]
+        .concat(alternateDomains.map(ad => _.extend(ad, { type: 'redirect' })))
+        .concat(aliasDomains.map(ad => _.extend(ad, { type: 'alias' })));
+
     validateLocations(locations, function (error, domainObjectMap) {
         if (error) return callback(error);
 
@@ -805,18 +818,19 @@ function install(data, auditSource, callback) {
         debug('Will install app with id : ' + appId);
 
         var data = {
-            accessRestriction: accessRestriction,
-            memoryLimit: memoryLimit,
-            sso: sso,
-            debugMode: debugMode,
-            mailboxName: mailboxName,
-            mailboxDomain: mailboxDomain,
-            enableBackup: enableBackup,
-            enableAutomaticUpdate: enableAutomaticUpdate,
-            alternateDomains: alternateDomains,
-            env: env,
-            label: label,
-            tags: tags,
+            accessRestriction,
+            memoryLimit,
+            sso,
+            debugMode,
+            mailboxName,
+            mailboxDomain,
+            enableBackup,
+            enableAutomaticUpdate,
+            alternateDomains,
+            aliasDomains,
+            env,
+            label,
+            tags,
             runState: exports.RSTATE_RUNNING,
             installationState: exports.ISTATE_PENDING_INSTALL
         };
@@ -835,7 +849,7 @@ function install(data, auditSource, callback) {
                 }
 
                 const task = {
-                    args: { restoreConfig: null, overwriteDns },
+                    args: { restoreConfig: null, skipDnsSetup, overwriteDns },
                     values: { },
                     requiredState: data.installationState
                 };
@@ -846,6 +860,7 @@ function install(data, auditSource, callback) {
                     const newApp = _.extend({}, data, { appStoreId, manifest, location, domain, portBindings });
                     newApp.fqdn = domains.fqdn(newApp.location, domainObjectMap[newApp.domain]);
                     newApp.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+                    newApp.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 
                     eventlog.add(eventlog.ACTION_APP_INSTALL, auditSource, { appId, app: newApp, taskId: result.taskId });
 
@@ -988,9 +1003,9 @@ function setCpuShares(app, cpuShares, auditSource, callback) {
     });
 }
 
-function setBinds(app, binds, auditSource, callback) {
+function setMounts(app, mounts, auditSource, callback) {
     assert.strictEqual(typeof app, 'object');
-    assert(binds && typeof binds === 'object');
+    assert(Array.isArray(mounts));
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -998,17 +1013,15 @@ function setBinds(app, binds, auditSource, callback) {
     let error = checkAppState(app, exports.ISTATE_PENDING_RECREATE_CONTAINER);
     if (error) return callback(error);
 
-    error = validateBinds(binds);
-    if (error) return callback(error);
-
     const task = {
         args: {},
-        values: { binds }
+        values: { mounts }
     };
     addTask(appId, exports.ISTATE_PENDING_RECREATE_CONTAINER, task, function (error, result) {
+        if (error && error.reason === BoxError.ALREADY_EXISTS) return callback(new BoxError(BoxError.CONFLICT, 'Duplicate mount points'));
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId, app, binds, taskId: result.taskId });
+        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId, app, mounts, taskId: result.taskId });
 
         callback(null, { taskId: result.taskId });
     });
@@ -1203,7 +1216,8 @@ function setLocation(app, data, auditSource, callback) {
         domain: data.domain.toLowerCase(),
         // these are intentionally reset, if not set
         portBindings: null,
-        alternateDomains: []
+        alternateDomains: [],
+        aliasDomains: []
     };
 
     if ('portBindings' in data) {
@@ -1223,14 +1237,21 @@ function setLocation(app, data, auditSource, callback) {
         values.alternateDomains = data.alternateDomains;
     }
 
-    const locations = [{subdomain: values.location, domain: values.domain}].concat(values.alternateDomains);
+    if ('aliasDomains' in data) {
+        values.aliasDomains = data.aliasDomains;
+    }
+
+    const locations = [{ subdomain: values.location, domain: values.domain, type: 'primary' }]
+        .concat(values.alternateDomains.map(ad => _.extend(ad, { type: 'redirect' })))
+        .concat(values.aliasDomains.map(ad => _.extend(ad, { type: 'alias' })));
 
     validateLocations(locations, function (error, domainObjectMap) {
         if (error) return callback(error);
 
         const task = {
             args: {
-                oldConfig: _.pick(app, 'location', 'domain', 'fqdn', 'alternateDomains', 'portBindings'),
+                oldConfig: _.pick(app, 'location', 'domain', 'fqdn', 'alternateDomains', 'aliasDomains', 'portBindings'),
+                skipDnsSetup: !!data.skipDnsSetup,
                 overwriteDns: !!data.overwriteDns
             },
             values
@@ -1241,6 +1262,7 @@ function setLocation(app, data, auditSource, callback) {
 
             values.fqdn = domains.fqdn(values.location, domainObjectMap[values.domain]);
             values.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+            values.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 
             eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, _.extend({ appId, app, taskId: result.taskId }, values));
 
@@ -1284,7 +1306,8 @@ function update(app, data, auditSource, callback) {
 
     const skipBackup = !!data.skipBackup,
         appId = app.id,
-        manifest = data.manifest;
+        manifest = data.manifest,
+        appStoreId = data.appStoreId;
 
     let values = {};
 
@@ -1299,14 +1322,12 @@ function update(app, data, auditSource, callback) {
     error = checkManifestConstraints(manifest);
     if (error) return callback(error);
 
-    var updateConfig = { skipBackup, manifest };
+    var updateConfig = { skipBackup, manifest, appStoreId }; // this will clear appStoreId when updating from a repo and set it if passed in for update route
 
     // prevent user from installing a app with different manifest id over an existing app
     // this allows cloudron install -f --app <appid> for an app installed from the appStore
     if (app.manifest.id !== updateConfig.manifest.id) {
         if (!data.force) return callback(new BoxError(BoxError.BAD_FIELD, 'manifest id does not match. force to override'));
-        // clear appStoreId so that this app does not get updates anymore
-        updateConfig.appStoreId = '';
     }
 
     // suffix '0' if prerelease is missing for semver.lte to work as expected
@@ -1353,13 +1374,23 @@ function update(app, data, auditSource, callback) {
 
         eventlog.add(eventlog.ACTION_APP_UPDATE, auditSource, { appId, app, skipBackup, toManifest: manifest, fromManifest: app.manifest, force: data.force, taskId: result.taskId });
 
-        // clear update indicator, if update fails, it will come back through the update checker
-        updateChecker.resetAppUpdateInfo(appId);
-
         callback(null, { taskId: result.taskId });
     });
 }
 
+function getLocalLogfilePaths(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    const appId = app.id;
+
+    var filePaths = [];
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'app.log'));
+    if (app.manifest.addons && app.manifest.addons.redis) filePaths.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
+
+    return filePaths;
+}
+
 function getLogs(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
     assert(options && typeof options === 'object');
@@ -1379,11 +1410,8 @@ function getLogs(app, options, callback) {
 
     var args = [ '--lines=' + lines ];
     if (follow) args.push('--follow', '--retry', '--quiet'); // same as -F. to make it work if file doesn't exist, --quiet to not output file headers, which are no logs
-    args.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
-    args.push(path.join(paths.LOG_DIR, appId, 'app.log'));
-    if (app.manifest.addons && app.manifest.addons.redis) args.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
 
-    var cp = spawn('/usr/bin/tail', args);
+    var cp = spawn('/usr/bin/tail', args.concat(getLocalLogfilePaths(app)));
 
     var transformStream = split(function mapper(line) {
         if (format !== 'json') return line + '\n';
@@ -1429,7 +1457,7 @@ function repair(app, data, auditSource, callback) {
 
     // maybe split this into a separate route like reinstall?
     if (errorState === exports.ISTATE_PENDING_INSTALL || errorState === exports.ISTATE_PENDING_CLONE) {
-        task.args = { overwriteDns: true };
+        task.args = { skipDnsSetup: false, overwriteDns: true };
         if (data.manifest) {
             let error = manifestFormat.parse(data.manifest);
             if (error) return callback(new BoxError(BoxError.BAD_FIELD, `manifest error: ${error.message}`));
@@ -1502,6 +1530,7 @@ function restore(app, backupId, auditSource, callback) {
             args: {
                 restoreConfig,
                 oldManifest: app.manifest,
+                skipDnsSetup: !!backupId, // if this is a restore, just skip dns setup. only re-installs should setup dns
                 overwriteDns: true
             },
             values
@@ -1557,6 +1586,7 @@ function importApp(app, data, auditSource, callback) {
             args: {
                 restoreConfig,
                 oldManifest: app.manifest,
+                skipDnsSetup: false,
                 overwriteDns: true
             },
             values: {}
@@ -1571,6 +1601,26 @@ function importApp(app, data, auditSource, callback) {
     });
 }
 
+function exportApp(app, data, auditSource, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const appId = app.id;
+
+    let error = checkAppState(app, exports.ISTATE_PENDING_BACKUP);
+    if (error) return callback(error);
+
+    const task = {
+        args: { snapshotOnly: true },
+        values: {}
+    };
+    addTask(appId, exports.ISTATE_PENDING_BACKUP, task, (error, result) => {
+        if (error) return callback(error);
+
+        callback(null, { taskId: result.taskId });
+    });
+}
+
 function purchaseApp(data, callback) {
     assert.strictEqual(typeof data, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -1599,6 +1649,7 @@ function clone(app, data, user, auditSource, callback) {
         portBindings = data.portBindings || null,
         backupId = data.backupId,
         overwriteDns = 'overwriteDns' in data ? data.overwriteDns : false,
+        skipDnsSetup = 'skipDnsSetup' in data ? data.skipDnsSetup : false,
         appId = app.id;
 
     assert.strictEqual(typeof backupId, 'string');
@@ -1625,7 +1676,7 @@ function clone(app, data, user, auditSource, callback) {
         let mailboxName = hasMailAddon(manifest) ? mailboxNameForLocation(location, manifest) : null;
         let mailboxDomain = hasMailAddon(manifest) ? domain : null;
 
-        const locations = [{subdomain: location, domain}];
+        const locations = [{ subdomain: location, domain, type: 'primary' }];
         validateLocations(locations, function (error, domainObjectMap) {
             if (error) return callback(error);
 
@@ -1642,7 +1693,8 @@ function clone(app, data, user, auditSource, callback) {
                 enableBackup: app.enableBackup,
                 reverseProxyConfig: app.reverseProxyConfig,
                 env: app.env,
-                alternateDomains: []
+                alternateDomains: [],
+                aliasDomains: []
             };
 
             appdb.add(newAppId, appStoreId, manifest, location, domain, translatePortBindings(portBindings, manifest), data, function (error) {
@@ -1654,7 +1706,7 @@ function clone(app, data, user, auditSource, callback) {
 
                     const restoreConfig = { backupId: backupId, backupFormat: backupInfo.format };
                     const task = {
-                        args: { restoreConfig, overwriteDns, oldManifest: null },
+                        args: { restoreConfig, overwriteDns, skipDnsSetup, oldManifest: null },
                         values: {},
                         requiredState: exports.ISTATE_PENDING_CLONE
                     };
@@ -1664,6 +1716,8 @@ function clone(app, data, user, auditSource, callback) {
                         const newApp = _.extend({}, data, { appStoreId, manifest, location, domain, portBindings });
                         newApp.fqdn = domains.fqdn(newApp.location, domainObjectMap[newApp.domain]);
                         newApp.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+                        newApp.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+
                         eventlog.add(eventlog.ACTION_APP_CLONE, auditSource, { appId: newAppId, oldAppId: appId, backupId: backupId, oldApp: app, newApp: newApp, taskId: result.taskId });
 
                         callback(null, { id: newAppId, taskId: result.taskId });
@@ -1832,18 +1886,25 @@ function exec(app, options, callback) {
     });
 }
 
-function canAutoupdateApp(app, newManifest) {
+function canAutoupdateApp(app, updateInfo) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof updateInfo, 'object');
+
+    const manifest = updateInfo.manifest;
+
     if (!app.enableAutomaticUpdate) return false;
 
     // for invalid subscriptions the appstore does not return a dockerImage
-    if (!newManifest.dockerImage) return false;
+    if (!manifest.dockerImage) return false;
 
-    if ((semver.major(app.manifest.version) !== 0) && (semver.major(app.manifest.version) !== semver.major(newManifest.version))) return false; // major changes are blocking
+    if (updateInfo.unstable) return false; // only manual update allowed for unstable updates
+
+    if ((semver.major(app.manifest.version) !== 0) && (semver.major(app.manifest.version) !== semver.major(manifest.version))) return false; // major changes are blocking
 
     if (app.runState === exports.RSTATE_STOPPED) return false; // stopped apps won't run migration scripts and shouldn't be updated
 
-    const newTcpPorts = newManifest.tcpPorts || { };
-    const newUdpPorts = newManifest.udpPorts || { };
+    const newTcpPorts = manifest.tcpPorts || { };
+    const newUdpPorts = manifest.udpPorts || { };
     const portBindings = app.portBindings; // this is never null
 
     for (let portName in portBindings) {
@@ -1859,8 +1920,6 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!updateInfo) return callback(null);
-
     async.eachSeries(Object.keys(updateInfo), function iterator(appId, iteratorDone) {
         get(appId, function (error, app) {
             if (error) {
@@ -1868,7 +1927,7 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
                 return iteratorDone();
             }
 
-            if (!canAutoupdateApp(app, updateInfo[appId].manifest)) {
+            if (!canAutoupdateApp(app, updateInfo[appId])) {
                 debug(`app ${app.fqdn} requires manual update`);
                 return iteratorDone();
             }
@@ -1913,14 +1972,15 @@ function listBackups(app, page, perPage, callback) {
     assert(typeof perPage === 'number' && perPage > 0);
     assert.strictEqual(typeof callback, 'function');
 
-    backups.getByAppIdPaged(page, perPage, app.id, function (error, results) {
+    backups.getByIdentifierAndStatePaged(app.id, backups.BACKUP_STATE_NORMAL, page, perPage, function (error, results) {
         if (error) return callback(error);
 
         callback(null, results);
     });
 }
 
-function restoreInstalledApps(callback) {
+function restoreInstalledApps(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     getAll(function (error, apps) {
@@ -1930,7 +1990,7 @@ function restoreInstalledApps(callback) {
         apps = apps.filter(app => app.installationState !== exports.ISTATE_PENDING_RESTORE); // safeguard against tasks being created non-stop if we crash on startup
 
         async.eachSeries(apps, function (app, iteratorDone) {
-            backups.getByAppIdPaged(1, 1, app.id, function (error, results) {
+            backups.getByIdentifierAndStatePaged(app.id, backups.BACKUP_STATE_NORMAL, 1, 1, function (error, results) {
                 let installationState, restoreConfig, oldManifest;
                 if (!error && results.length) {
                     installationState = exports.ISTATE_PENDING_RESTORE;
@@ -1943,7 +2003,7 @@ function restoreInstalledApps(callback) {
                 }
 
                 const task = {
-                    args: { restoreConfig, overwriteDns: true, oldManifest },
+                    args: { restoreConfig, skipDnsSetup: options.skipDnsSetup, overwriteDns: true, oldManifest },
                     values: {},
                     scheduleNow: false, // task will be scheduled by autoRestartTasks when platform is ready
                     requireNullTaskId: false // ignore existing stale taskId
@@ -2001,6 +2061,7 @@ function restartAppsUsingAddons(changedAddons, callback) {
         apps = apps.filter(app => app.manifest.addons && _.intersection(Object.keys(app.manifest.addons), changedAddons).length !== 0);
         apps = apps.filter(app => app.installationState !== exports.ISTATE_ERROR); // remove errored apps. let them be 'repaired' by hand
         apps = apps.filter(app => app.installationState !== exports.ISTATE_PENDING_RESTART); // safeguard against tasks being created non-stop restart if we crash on startup
+        apps = apps.filter(app => app.runState !== exports.RSTATE_STOPPED); // don't start stopped apps
 
         async.eachSeries(apps, function (app, iteratorDone) {
             debug(`restartAppsUsingAddons: marking ${app.fqdn} for restart`);
@@ -2009,11 +2070,17 @@ function restartAppsUsingAddons(changedAddons, callback) {
                 args: {},
                 values: { runState: exports.RSTATE_RUNNING }
             };
-            addTask(app.id, exports.ISTATE_PENDING_RESTART, task, function (error, result) {
-                if (error) debug(`restartAppsUsingAddons: error marking ${app.fqdn} for restart: ${JSON.stringify(error)}`);
-                else debug(`restartAppsUsingAddons: marked ${app.id} for restart with taskId ${result.taskId}`);
 
-                iteratorDone(); // ignore error
+            // stop apps before updating the databases because postgres will "lock" them preventing import
+            docker.stopContainers(app.id, function (error) {
+                if (error) debug(`restartAppsUsingAddons: error stopping ${app.fqdn}`, error);
+
+                addTask(app.id, exports.ISTATE_PENDING_RESTART, task, function (error, result) {
+                    if (error) debug(`restartAppsUsingAddons: error marking ${app.fqdn} for restart: ${JSON.stringify(error)}`);
+                    else debug(`restartAppsUsingAddons: marked ${app.id} for restart with taskId ${result.taskId}`);
+
+                    iteratorDone(); // ignore error
+                });
             });
         }, callback);
     });

src/appstore.js

@@ -1,31 +1,28 @@
 'use strict';
 
 exports = module.exports = {
-    getFeatures: getFeatures,
+    getFeatures,
 
-    getApps: getApps,
-    getApp: getApp,
-    getAppVersion: getAppVersion,
+    getApps,
+    getApp,
+    getAppVersion,
 
-    trackBeginSetup: trackBeginSetup,
-    trackFinishedSetup: trackFinishedSetup,
+    trackBeginSetup,
+    trackFinishedSetup,
 
-    registerWithLoginCredentials: registerWithLoginCredentials,
-    registerWithLicense: registerWithLicense,
+    registerWithLoginCredentials,
 
-    purchaseApp: purchaseApp,
-    unpurchaseApp: unpurchaseApp,
+    purchaseApp,
+    unpurchaseApp,
 
-    getUserToken: getUserToken,
-    getSubscription: getSubscription,
-    isFreePlan: isFreePlan,
+    getUserToken,
+    getSubscription,
+    isFreePlan,
 
-    sendAliveStatus: sendAliveStatus,
+    getAppUpdate,
+    getBoxUpdate,
 
-    getAppUpdate: getAppUpdate,
-    getBoxUpdate: getBoxUpdate,
-
-    createTicket: createTicket
+    createTicket
 };
 
 var apps = require('./apps.js'),
@@ -34,32 +31,30 @@ var apps = require('./apps.js'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:appstore'),
-    domains = require('./domains.js'),
     eventlog = require('./eventlog.js'),
-    groups = require('./groups.js'),
-    mail = require('./mail.js'),
-    os = require('os'),
+    path = require('path'),
     paths = require('./paths.js'),
     safe = require('safetydance'),
     semver = require('semver'),
     settings = require('./settings.js'),
     superagent = require('superagent'),
-    users = require('./users.js'),
+    support = require('./support.js'),
     util = require('util');
 
-const NOOP_CALLBACK = function (error) { if (error) debug(error); };
-
 // These are the default options and will be adjusted once a subscription state is obtained
 // Keep in sync with appstore/routes/cloudrons.js
 let gFeatures = {
-    userMaxCount: null,
-    externalLdap: true,
-    eventLog: true,
-    privateDockerRegistry: true,
-    branding: true,
-    userManager: true,
-    multiAdmin: true,
-    support: true
+    userMaxCount: 5,
+    userGroups: false,
+    userRoles: false,
+    domainMaxCount: 1,
+    externalLdap: false,
+    privateDockerRegistry: false,
+    branding: false,
+    support: false,
+    directoryConfig: false,
+    mailboxMaxCount: 5,
+    emailPremium: false
 };
 
 // attempt to load feature cache in case appstore would be down
@@ -235,111 +230,6 @@ function unpurchaseApp(appId, data, callback) {
     });
 }
 
-function sendAliveStatus(callback) {
-    callback = callback || NOOP_CALLBACK;
-
-    let allSettings, allDomains, mailDomains, loginEvents, userCount, groupCount;
-
-    async.series([
-        function (callback) {
-            settings.getAll(function (error, result) {
-                if (error) return callback(error);
-                allSettings = result;
-                callback();
-            });
-        },
-        function (callback) {
-            domains.getAll(function (error, result) {
-                if (error) return callback(error);
-                allDomains = result;
-                callback();
-            });
-        },
-        function (callback) {
-            mail.getDomains(function (error, result) {
-                if (error) return callback(error);
-                mailDomains = result;
-                callback();
-            });
-        },
-        function (callback) {
-            eventlog.getAllPaged([ eventlog.ACTION_USER_LOGIN ], null, 1, 1, function (error, result) {
-                if (error) return callback(error);
-                loginEvents = result;
-                callback();
-            });
-        },
-        function (callback) {
-            users.count(function (error, result) {
-                if (error) return callback(error);
-                userCount = result;
-                callback();
-            });
-        },
-        function (callback) {
-            groups.count(function (error, result) {
-                if (error) return callback(error);
-                groupCount = result;
-                callback();
-            });
-        }
-    ], function (error) {
-        if (error) return callback(error);
-
-        var backendSettings = {
-            backupConfig: {
-                provider: allSettings[settings.BACKUP_CONFIG_KEY].provider,
-                hardlinks: !allSettings[settings.BACKUP_CONFIG_KEY].noHardlinks
-            },
-            domainConfig: {
-                count: allDomains.length,
-                domains: Array.from(new Set(allDomains.map(function (d) { return { domain: d.domain, provider: d.provider }; })))
-            },
-            mailConfig: {
-                outboundCount: mailDomains.length,
-                inboundCount: mailDomains.filter(function (d) { return d.enabled; }).length,
-                catchAllCount: mailDomains.filter(function (d) { return d.catchAll.length !== 0; }).length,
-                relayProviders: Array.from(new Set(mailDomains.map(function (d) { return d.relay.provider; })))
-            },
-            userCount: userCount,
-            groupCount: groupCount,
-            appAutoupdatePattern: allSettings[settings.APP_AUTOUPDATE_PATTERN_KEY],
-            boxAutoupdatePattern: allSettings[settings.BOX_AUTOUPDATE_PATTERN_KEY],
-            timeZone: allSettings[settings.TIME_ZONE_KEY],
-            sysinfoProvider: allSettings[settings.SYSINFO_CONFIG_KEY].provider
-        };
-
-        var data = {
-            version: constants.VERSION,
-            adminFqdn: settings.adminFqdn(),
-            provider: settings.provider(),
-            backendSettings: backendSettings,
-            machine: {
-                cpus: os.cpus(),
-                totalmem: os.totalmem()
-            },
-            events: {
-                lastLogin: loginEvents[0] ? (new Date(loginEvents[0].creationTime).getTime()) : 0
-            }
-        };
-
-        getCloudronToken(function (error, token) {
-            if (error) return callback(error);
-
-            const url = `${settings.apiServerOrigin()}/api/v1/alive`;
-            superagent.post(url).send(data).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
-                if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Sending alive status failed. %s %j', result.status, result.body)));
-
-                callback(null);
-            });
-        });
-    });
-}
-
 function getBoxUpdate(options, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -355,17 +245,17 @@ function getBoxUpdate(options, callback) {
             automatic: options.automatic
         };
 
-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-            if (result.statusCode === 204) return callback(null); // no update
+            if (result.statusCode === 204) return callback(null, null); // no update
             if (result.statusCode !== 200 || !result.body) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
 
             var updateInfo = result.body;
 
             if (!semver.valid(updateInfo.version) || semver.gt(constants.VERSION, updateInfo.version)) {
-                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Invalid update version: %s %s', result.statusCode, result.text)));
+                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Update version invalid or is a downgrade: %s %s', result.statusCode, result.text)));
             }
 
             // updateInfo: { version, changelog, sourceTarballUrl, sourceTarballSigUrl, boxVersionsUrl, boxVersionsSigUrl }
@@ -398,7 +288,7 @@ function getAppUpdate(app, options, callback) {
             automatic: options.automatic
         };
 
-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -416,7 +306,9 @@ function getAppUpdate(app, options, callback) {
                 return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Malformed update: %s %s', result.statusCode, result.text)));
             }
 
-            // { id, creationDate, manifest }
+            updateInfo.unstable = !!updateInfo.unstable;
+
+            // { id, creationDate, manifest, unstable }
             callback(null, updateInfo);
         });
     });
@@ -453,18 +345,16 @@ function registerCloudron(data, callback) {
 
 // This works without a Cloudron token as this Cloudron was not yet registered
 let gBeginSetupAlreadyTracked = false;
-function trackBeginSetup(provider) {
-    assert.strictEqual(typeof provider, 'string');
-
+function trackBeginSetup() {
     // avoid browser reload double tracking, not perfect since box might restart, but covers most cases and is simple
     if (gBeginSetupAlreadyTracked) return;
     gBeginSetupAlreadyTracked = true;
 
     const url = `${settings.apiServerOrigin()}/api/v1/helper/setup_begin`;
 
-    superagent.post(url).send({ provider }).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return console.error(error.message);
-        if (result.statusCode !== 200) return console.error(error.message);
+    superagent.post(url).send({}).timeout(30 * 1000).end(function (error, result) {
+        if (error && !error.response) return debug(`trackBeginSetup: ${error.message}`);
+        if (result.statusCode !== 200) return debug(`trackBeginSetup: ${result.statusCode} ${error.message}`);
     });
 }
 
@@ -475,23 +365,8 @@ function trackFinishedSetup(domain) {
     const url = `${settings.apiServerOrigin()}/api/v1/helper/setup_finished`;
 
     superagent.post(url).send({ domain }).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return console.error(error.message);
-        if (result.statusCode !== 200) return console.error(error.message);
-    });
-}
-
-function registerWithLicense(license, domain, callback) {
-    assert.strictEqual(typeof license, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    getCloudronToken(function (error, token) {
-        if (token) return callback(new BoxError(BoxError.CONFLICT, 'Cloudron is already registered'));
-
-        const provider = settings.provider();
-        const version = constants.VERSION;
-
-        registerCloudron({ license, domain, provider, version }, callback);
+        if (error && !error.response) return debug(`trackFinishedSetup: ${error.message}`);
+        if (result.statusCode !== 200) return debug(`trackFinishedSetup: ${result.statusCode} ${error.message}`);
     });
 }
 
@@ -514,7 +389,7 @@ function registerWithLoginCredentials(options, callback) {
             login(options.email, options.password, options.totpToken || '', function (error, result) {
                 if (error) return callback(error);
 
-                registerCloudron({ domain: settings.adminDomain(), accessToken: result.accessToken, provider: settings.provider(), version: constants.VERSION, purpose: options.purpose || '' }, callback);
+                registerCloudron({ domain: settings.adminDomain(), accessToken: result.accessToken, version: constants.VERSION, purpose: options.purpose || '' }, callback);
             });
         });
     });
@@ -535,26 +410,55 @@ function createTicket(info, auditSource, callback) {
         apps.get(info.appId, callback);
     }
 
+    function enableSshIfNeeded(callback) {
+        if (!info.enableSshSupport) return callback();
+
+        support.enableRemoteSupport(true, auditSource, function (error) {
+            // ensure we can at least get the ticket through
+            if (error) debug('Unable to enable SSH support.', error);
+
+            callback();
+        });
+    }
+
     getCloudronToken(function (error, token) {
         if (error) return callback(error);
 
-        collectAppInfoIfNeeded(function (error, result) {
-            if (error) console.error('Unable to get app info', error);
-            if (result) info.app = result;
+        enableSshIfNeeded(function (error) {
+            if (error) return callback(error);
 
-            let url = settings.apiServerOrigin() + '/api/v1/ticket';
+            collectAppInfoIfNeeded(function (error, app) {
+                if (error) return callback(error);
+                if (app) info.app = app;
 
-            info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
+                info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
 
-            superagent.post(url).query({ accessToken: token }).send(info).timeout(10 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+                var req = superagent.post(`${settings.apiServerOrigin()}/api/v1/ticket`)
+                    .query({ accessToken: token })
+                    .timeout(30 * 1000);
 
-                eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+                // either send as JSON through body or as multipart, depending on attachments
+                if (info.app) {
+                    req.field('infoJSON', JSON.stringify(info));
 
-                callback(null, { message: `An email for sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                    apps.getLocalLogfilePaths(info.app).forEach(function (filePath) {
+                        var logs = safe.child_process.execSync(`tail --lines=1000 ${filePath}`);
+                        if (logs) req.attach(path.basename(filePath), logs, path.basename(filePath));
+                    });
+                } else {
+                    req.send(info);
+                }
+
+                req.end(function (error, result) {
+                    if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                    if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+                    if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
+                    if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+
+                    eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+
+                    callback(null, { message: `An email was sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                });
             });
         });
     });
@@ -570,7 +474,7 @@ function getApps(callback) {
             if (error) return callback(error);
 
             const url = `${settings.apiServerOrigin()}/api/v1/apps`;
-            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -605,7 +509,7 @@ function getAppVersion(appId, version, callback) {
             let url = `${settings.apiServerOrigin()}/api/v1/apps/${appId}`;
             if (version !== 'latest') url += `/versions/${version}`;
 
-            superagent.get(url).query({ accessToken: token }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));

src/apptask.js

@@ -3,24 +3,18 @@
 'use strict';
 
 exports = module.exports = {
-    run: run,
+    run,
 
     // exported for testing
-    _reserveHttpPort: reserveHttpPort,
     _configureReverseProxy: configureReverseProxy,
     _unconfigureReverseProxy: unconfigureReverseProxy,
     _createAppDir: createAppDir,
     _deleteAppDir: deleteAppDir,
     _verifyManifest: verifyManifest,
-    _registerSubdomains: registerSubdomains,
-    _unregisterSubdomains: unregisterSubdomains,
     _waitForDnsPropagation: waitForDnsPropagation
 };
 
-require('supererror')({ splatchError: true });
-
-var addons = require('./addons.js'),
-    appdb = require('./appdb.js'),
+const appdb = require('./appdb.js'),
     apps = require('./apps.js'),
     assert = require('assert'),
     async = require('async'),
@@ -36,15 +30,15 @@ var addons = require('./addons.js'),
     ejs = require('ejs'),
     eventlog = require('./eventlog.js'),
     fs = require('fs'),
+    iputils = require('./iputils.js'),
     manifestFormat = require('cloudron-manifestformat'),
-    mkdirp = require('mkdirp'),
-    net = require('net'),
     os = require('os'),
     path = require('path'),
     paths = require('./paths.js'),
     reverseProxy = require('./reverseproxy.js'),
     rimraf = require('rimraf'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     superagent = require('superagent'),
@@ -92,22 +86,16 @@ function updateApp(app, values, callback) {
     });
 }
 
-function reserveHttpPort(app, callback) {
+function allocateContainerIp(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let server = net.createServer();
-    server.listen(0, function () {
-        let port = server.address().port;
-
-        updateApp(app, { httpPort: port }, function (error) {
-            server.close(function (/* closeError */) {
-                if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Failed to allocate http port ${port}: ${error.message}`));
-
-                callback(null);
-            });
-        });
-    });
+    async.retry({ times: 10 }, function (retryCallback) {
+        const iprange = iputils.intFromIp('172.18.20.255') - iputils.intFromIp('172.18.16.1');
+        let rnd = Math.floor(Math.random() * iprange);
+        const containerIp = iputils.ipFromInt(iputils.intFromIp('172.18.16.1') + rnd);
+        updateApp(app, { containerIp }, retryCallback);
+    }, callback);
 }
 
 function configureReverseProxy(app, callback) {
@@ -176,7 +164,7 @@ function createAppDir(app, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     const appDir = path.join(paths.APPS_DATA_DIR, app.id);
-    mkdirp(appDir, function (error) {
+    fs.mkdir(appDir, { recursive: true }, function (error) {
         if (error) return callback(new BoxError(BoxError.FS_ERROR, `Error creating directory: ${error.message}`, { appDir }));
 
         callback(null);
@@ -343,82 +331,6 @@ function removeIcon(app, callback) {
     callback(null);
 }
 
-function registerSubdomains(app, overwrite, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof overwrite, 'boolean');
-    assert.strictEqual(typeof callback, 'function');
-
-    sysinfo.getServerIp(function (error, ip) {
-        if (error) return callback(error);
-
-        const allDomains = [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains);
-
-        debug(`registerSubdomain: Will register ${JSON.stringify(allDomains)}`);
-
-        async.eachSeries(allDomains, function (domain, iteratorDone) {
-            async.retry({ times: 200, interval: 5000 }, function (retryCallback) {
-                debugApp(app, 'Registering subdomain: %s%s', domain.subdomain ? (domain.subdomain + '.') : '', domain.domain);
-
-                // get the current record before updating it
-                domains.getDnsRecords(domain.subdomain, domain.domain, 'A', function (error, values) {
-                    if (error && error.reason === BoxError.EXTERNAL_ERROR) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
-                    if (error && error.reason === BoxError.ACCESS_DENIED) return retryCallback(null, new BoxError(BoxError.ACCESS_DENIED, error.message, { domain }));
-                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, new BoxError(BoxError.NOT_FOUND, error.message, { domain }));
-                    if (error) return retryCallback(null, new BoxError(BoxError.EXTERNAL_ERROR, error.message, domain)); // give up for other errors
-
-                    if (values.length !== 0 && values[0] === ip) return retryCallback(null); // up-to-date
-
-                    // refuse to update any existing DNS record for custom domains that we did not create
-                    if (values.length !== 0 && !overwrite) return retryCallback(null, new BoxError(BoxError.ALREADY_EXISTS, 'DNS Record already exists', { domain }));
-
-                    domains.upsertDnsRecords(domain.subdomain, domain.domain, 'A', [ ip ], function (error) {
-                        if (error && (error.reason === BoxError.BUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
-                            debug('registerSubdomains: Upsert error. Will retry.', error.message);
-                            return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
-                        }
-
-                        retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, domain) : null);
-                    });
-                });
-            }, function (error, result) {
-                if (error || result) return iteratorDone(error || result);
-
-                iteratorDone(null);
-            });
-        }, callback);
-    });
-}
-
-function unregisterSubdomains(app, allDomains, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert(Array.isArray(allDomains));
-    assert.strictEqual(typeof callback, 'function');
-
-    sysinfo.getServerIp(function (error, ip) {
-        if (error) return callback(error);
-
-        async.eachSeries(allDomains, function (domain, iteratorDone) {
-            async.retry({ times: 30, interval: 5000 }, function (retryCallback) {
-                debugApp(app, 'Unregistering subdomain: %s%s', domain.subdomain ? (domain.subdomain + '.') : '', domain.domain);
-
-                domains.removeDnsRecords(domain.subdomain, domain.domain, 'A', [ ip ], function (error) {
-                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, null);
-                    if (error && (error.reason === BoxError.SBUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
-                        debug('registerSubdomains: Remove error. Will retry.', error.message);
-                        return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
-                    }
-
-                    retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain }) : null);
-                });
-            }, function (error, result) {
-                if (error || result) return iteratorDone(error || result);
-
-                iteratorDone();
-            });
-        }, callback);
-    });
-}
-
 function waitForDnsPropagation(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -434,8 +346,8 @@ function waitForDnsPropagation(app, callback) {
         domains.waitForDnsRecord(app.location, app.domain, 'A', ip, { times: 240 }, function (error) {
             if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: app.location, domain: app.domain }));
 
-            // now wait for alternateDomains, if any
-            async.eachSeries(app.alternateDomains, function (domain, iteratorCallback) {
+            // now wait for alternateDomains and aliasDomains, if any
+            async.eachSeries(app.alternateDomains.concat(app.aliasDomains), function (domain, iteratorCallback) {
                 domains.waitForDnsRecord(domain.subdomain, domain.domain, 'A', ip, { times: 240 }, function (error) {
                     if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: domain.subdomain, domain: domain.domain }));
 
@@ -454,7 +366,7 @@ function moveDataDir(app, targetDir, callback) {
     let resolvedSourceDir = apps.getDataDir(app, app.dataDir);
     let resolvedTargetDir = apps.getDataDir(app, targetDir);
 
-    debug(`moveDataDir: migrating data from ${resolvedSourceDir} to ${resolvedTargetDir}`);
+    debugApp(app, `moveDataDir: migrating data from ${resolvedSourceDir} to ${resolvedTargetDir}`);
 
     if (resolvedSourceDir === resolvedTargetDir) return callback();
 
@@ -487,6 +399,8 @@ function downloadImage(manifest, callback) {
 }
 
 function startApp(app, callback){
+    debugApp(app, 'startApp: starting container');
+
     if (app.runState === apps.RSTATE_STOPPED) return callback();
 
     docker.startContainer(app.id, callback);
@@ -500,6 +414,7 @@ function install(app, args, progressCallback, callback) {
 
     const restoreConfig = args.restoreConfig; // has to be set when restoring
     const overwriteDns = args.overwriteDns;
+    const skipDnsSetup = args.skipDnsSetup;
     const oldManifest = args.oldManifest;
 
     async.series([
@@ -520,7 +435,7 @@ function install(app, args, progressCallback, callback) {
                 addonsToRemove = app.manifest.addons;
             }
 
-            addons.teardownAddons(app, addonsToRemove, next);
+            services.teardownAddons(app, addonsToRemove, next);
         },
 
         function deleteAppDirIfNeeded(done) {
@@ -535,13 +450,21 @@ function install(app, args, progressCallback, callback) {
             docker.deleteImage(oldManifest, done);
         },
 
-        reserveHttpPort.bind(null, app),
+        // allocating container ip here, lets the users "repair" an app if allocation fails at appdb.add time
+        allocateContainerIp.bind(null, app),
 
         progressCallback.bind(null, { percent: 20, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
 
-        progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
-        registerSubdomains.bind(null, app, overwriteDns),
+        function setupDnsIfNeeded(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
+
+                domains.registerLocations.bind(null, [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains).concat(app.aliasDomains), { overwriteDns }, progressCallback)
+            ], done);
+        },
 
         progressCallback.bind(null, { percent: 40, message: 'Downloading image' }),
         downloadImage.bind(null, app.manifest),
@@ -553,24 +476,24 @@ function install(app, args, progressCallback, callback) {
             if (!restoreConfig) {
                 async.series([
                     progressCallback.bind(null, { percent: 60, message: 'Setting up addons' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
                 ], next);
             } else if (!restoreConfig.backupId) { // in-place import
                 async.series([
                     progressCallback.bind(null, { percent: 60, message: 'Importing addons in-place' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
-                    addons.clearAddons.bind(null, app, _.omit(app.manifest.addons, 'localstorage')),
-                    addons.restoreAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
+                    services.clearAddons.bind(null, app, _.omit(app.manifest.addons, 'localstorage')),
+                    services.restoreAddons.bind(null, app, app.manifest.addons),
                 ], next);
             } else {
                 async.series([
                     progressCallback.bind(null, { percent: 65, message: 'Download backup and restoring addons' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
-                    addons.clearAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
+                    services.clearAddons.bind(null, app, app.manifest.addons),
                     backups.downloadApp.bind(null, app, restoreConfig, (progress) => {
                         progressCallback({ percent: 65, message: progress.message });
                     }),
-                    addons.restoreAddons.bind(null, app, app.manifest.addons)
+                    services.restoreAddons.bind(null, app, app.manifest.addons)
                 ], next);
             }
         },
@@ -580,8 +503,14 @@ function install(app, args, progressCallback, callback) {
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 85, message: 'Waiting for DNS propagation' }),
-        exports._waitForDnsPropagation.bind(null, app),
+        function waitForDns(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 85, message: 'Waiting for DNS propagation' }),
+                exports._waitForDnsPropagation.bind(null, app),
+            ], done);
+        },
 
         progressCallback.bind(null, { percent: 95, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
@@ -605,7 +534,7 @@ function backup(app, args, progressCallback, callback) {
 
     async.series([
         progressCallback.bind(null, { percent: 10, message: 'Backing up' }),
-        backups.backupApp.bind(null, app, { /* options */ }, (progress) => {
+        backups.backupApp.bind(null, app, { snapshotOnly: !!args.snapshotOnly }, (progress) => {
             progressCallback({ percent: 30, message: progress.message });
         }),
 
@@ -633,7 +562,7 @@ function create(app, args, progressCallback, callback) {
 
         // FIXME: re-setup addons only because sendmail addon to re-inject env vars on mailboxName change
         progressCallback.bind(null, { percent: 30, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
@@ -659,6 +588,7 @@ function changeLocation(app, args, progressCallback, callback) {
 
     const oldConfig = args.oldConfig;
     const locationChanged = oldConfig.fqdn !== app.fqdn;
+    const skipDnsSetup = args.skipDnsSetup;
     const overwriteDns = args.overwriteDns;
 
     async.series([
@@ -670,27 +600,45 @@ function changeLocation(app, args, progressCallback, callback) {
                 return !app.alternateDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; });
             });
 
+            if (oldConfig.aliasDomains) {
+                obsoleteDomains = obsoleteDomains.concat(oldConfig.aliasDomains.filter(function (o) {
+                    return !app.aliasDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; });
+                }));
+            }
+
             if (locationChanged) obsoleteDomains.push({ subdomain: oldConfig.location, domain: oldConfig.domain });
 
             if (obsoleteDomains.length === 0) return next();
 
-            unregisterSubdomains(app, obsoleteDomains, next);
+            domains.unregisterLocations(obsoleteDomains, progressCallback, next);
         },
 
-        progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
-        registerSubdomains.bind(null, app, overwriteDns),
+        function setupDnsIfNeeded(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
+                domains.registerLocations.bind(null, [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains).concat(app.aliasDomains), { overwriteDns }, progressCallback)
+            ], done);
+        },
 
         // re-setup addons since they rely on the app's fqdn (e.g oauth)
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 80, message: 'Waiting for DNS propagation' }),
-        exports._waitForDnsPropagation.bind(null, app),
+        function waitForDns(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 80, message: 'Waiting for DNS propagation' }),
+                exports._waitForDnsPropagation.bind(null, app),
+            ], done);
+        },
 
         progressCallback.bind(null, { percent: 90, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
@@ -724,7 +672,7 @@ function migrateDataDir(app, args, progressCallback, callback) {
 
         // re-setup addons since this creates the localStorage volume
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, _.extend({}, app, { dataDir: newDataDir }), app.manifest.addons),
+        services.setupAddons.bind(null, _.extend({}, app, { dataDir: newDataDir }), app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Moving data dir' }),
         moveDataDir.bind(null, app, newDataDir),
@@ -741,7 +689,8 @@ function migrateDataDir(app, args, progressCallback, callback) {
             debugApp(app, 'error migrating data dir : %s', error);
             return updateApp(app, { installationState: apps.ISTATE_ERROR, error: makeTaskError(error, app) }, callback.bind(null, error));
         }
-        callback(null);
+
+        callback();
     });
 }
 
@@ -756,7 +705,6 @@ function configure(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 10, message: 'Cleaning up old install' }),
         unconfigureReverseProxy.bind(null, app),
         deleteContainers.bind(null, app, { managedOnly: true }),
-        reserveHttpPort.bind(null, app),
 
         progressCallback.bind(null, { percent: 20, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
@@ -769,7 +717,7 @@ function configure(app, args, progressCallback, callback) {
 
         // re-setup addons since they rely on the app's fqdn (e.g oauth)
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
@@ -786,11 +734,11 @@ function configure(app, args, progressCallback, callback) {
             debugApp(app, 'error reconfiguring : %s', error);
             return updateApp(app, { installationState: apps.ISTATE_ERROR, error: makeTaskError(error, app) }, callback.bind(null, error));
         }
-        callback(null);
+
+        callback();
     });
 }
 
-// nginx configuration is skipped because app.httpPort is expected to be available
 function update(app, args, progressCallback, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof args, 'object');
@@ -802,7 +750,10 @@ function update(app, args, progressCallback, callback) {
 
     // app does not want these addons anymore
     // FIXME: this does not handle option changes (like multipleDatabases)
-    var unusedAddons = _.omit(app.manifest.addons, Object.keys(updateConfig.manifest.addons));
+    const unusedAddons = _.omit(app.manifest.addons, Object.keys(updateConfig.manifest.addons));
+    const httpPathsChanged = app.manifest.httpPaths !== updateConfig.manifest.httpPaths;
+    const httpPortChanged = app.manifest.httpPort !== updateConfig.manifest.httpPort;
+    const proxyAuthChanged = !_.isEqual(safe.query(app.manifest, 'addons.proxyAuth'), safe.query(updateConfig.manifest, 'addons.proxyAuth'));
 
     async.series([
         // this protects against the theoretical possibility of an app being marked for update from
@@ -841,7 +792,7 @@ function update(app, args, progressCallback, callback) {
         },
 
         // only delete unused addons after backup
-        addons.teardownAddons.bind(null, app, unusedAddons),
+        services.teardownAddons.bind(null, app, unusedAddons),
 
         // free unused ports
         function (next) {
@@ -853,7 +804,7 @@ function update(app, args, progressCallback, callback) {
                 if (newTcpPorts[portName] || newUdpPorts[portName]) return callback(null); // port still in use
 
                 appdb.delPortBinding(currentPorts[portName], apps.PORT_TYPE_TCP, function (error) {
-                    if (error && error.reason === BoxError.NOT_FOUND) console.error('Portbinding does not exist in database.');
+                    if (error && error.reason === BoxError.NOT_FOUND) debugApp(app, 'update: portbinding does not exist in database', error);
                     else if (error) return next(error);
 
                     // also delete from app object for further processing (the db is updated in the next step)
@@ -869,14 +820,21 @@ function update(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 45, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
 
-        progressCallback.bind(null, { percent: 70, message: 'Updating addons' }),
-        addons.setupAddons.bind(null, app, updateConfig.manifest.addons),
+        progressCallback.bind(null, { percent: 60, message: 'Updating addons' }),
+        services.setupAddons.bind(null, app, updateConfig.manifest.addons),
 
-        progressCallback.bind(null, { percent: 80, message: 'Creating container' }),
+        progressCallback.bind(null, { percent: 70, message: 'Creating container' }),
         createContainer.bind(null, app),
 
         startApp.bind(null, app),
 
+        progressCallback.bind(null, { percent: 90, message: 'Configuring reverse proxy' }),
+        function (next) {
+            if (!httpPathsChanged && !proxyAuthChanged && !httpPortChanged) return next();
+
+            configureReverseProxy(app, next);
+        },
+
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null, updateTime: new Date() })
     ], function seriesDone(error) {
@@ -900,13 +858,16 @@ function start(app, args, progressCallback, callback) {
 
     async.series([
         progressCallback.bind(null, { percent: 10, message: 'Starting app services' }),
-        addons.startAppServices.bind(null, app),
+        services.startAppServices.bind(null, app),
 
         progressCallback.bind(null, { percent: 35, message: 'Starting container' }),
         docker.startContainer.bind(null, app.id),
 
+        progressCallback.bind(null, { percent: 60, message: 'Adding collectd profile' }),
+        addCollectdProfile.bind(null, app),
+
         // stopped apps do not renew certs. currently, we don't do DNS to not overwrite existing user settings
-        progressCallback.bind(null, { percent: 60, message: 'Configuring reverse proxy' }),
+        progressCallback.bind(null, { percent: 80, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
 
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
@@ -931,7 +892,10 @@ function stop(app, args, progressCallback, callback) {
         docker.stopContainers.bind(null, app.id),
 
         progressCallback.bind(null, { percent: 50, message: 'Stopping app services' }),
-        addons.stopAppServices.bind(null, app),
+        services.stopAppServices.bind(null, app),
+
+        progressCallback.bind(null, { percent: 80, message: 'Removing collectd profile' }),
+        removeCollectdProfile.bind(null, app),
 
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null })
@@ -977,18 +941,20 @@ function uninstall(app, args, progressCallback, callback) {
         deleteContainers.bind(null, app, {}),
 
         progressCallback.bind(null, { percent: 30, message: 'Teardown addons' }),
-        addons.teardownAddons.bind(null, app, app.manifest.addons),
+        services.teardownAddons.bind(null, app, app.manifest.addons),
 
-        progressCallback.bind(null, { percent: 40, message: 'Deleting app data directory' }),
+        progressCallback.bind(null, { percent: 40, message: 'Cleanup file manager' }),
+
+        progressCallback.bind(null, { percent: 50, message: 'Deleting app data directory' }),
         deleteAppDir.bind(null, app, { removeDirectory: true }),
 
-        progressCallback.bind(null, { percent: 50, message: 'Deleting image' }),
+        progressCallback.bind(null, { percent: 60, message: 'Deleting image' }),
         docker.deleteImage.bind(null, app.manifest),
 
-        progressCallback.bind(null, { percent: 60, message: 'Unregistering domains' }),
-        unregisterSubdomains.bind(null, app, [ { subdomain: app.location, domain: app.domain } ].concat(app.alternateDomains)),
+        progressCallback.bind(null, { percent: 70, message: 'Unregistering domains' }),
+        domains.unregisterLocations.bind(null, [ { subdomain: app.location, domain: app.domain } ].concat(app.alternateDomains).concat(app.aliasDomains), progressCallback),
 
-        progressCallback.bind(null, { percent: 70, message: 'Cleanup icon' }),
+        progressCallback.bind(null, { percent: 80, message: 'Cleanup icon' }),
         removeIcon.bind(null, app),
 
         progressCallback.bind(null, { percent: 90, message: 'Cleanup logs' }),
@@ -1044,6 +1010,8 @@ function run(appId, args, progressCallback, callback) {
             return stop(app, args, progressCallback, callback);
         case apps.ISTATE_PENDING_RESTART:
             return restart(app, args, progressCallback, callback);
+        case apps.ISTATE_INSTALLED: // can only happen when we have a bug in our code while testing/development
+            return updateApp(app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null }, callback);
         default:
             debugApp(app, 'apptask launched with invalid command');
             return callback(new BoxError(BoxError.INTERNAL_ERROR, 'Unknown install command in apptask:' + app.installationState));

src/apptaskmanager.js

@@ -1,7 +1,7 @@
 'use strict';
 
 exports = module.exports = {
-    scheduleTask: scheduleTask
+    scheduleTask
 };
 
 let assert = require('assert'),
@@ -12,6 +12,8 @@ let assert = require('assert'),
     safe = require('safetydance'),
     path = require('path'),
     paths = require('./paths.js'),
+    scheduler = require('./scheduler.js'),
+    services = require('./services.js'),
     tasks = require('./tasks.js');
 
 let gActiveTasks = { }; // indexed by app id
@@ -35,9 +37,10 @@ function initializeSync() {
 }
 
 // callback is called when task is finished
-function scheduleTask(appId, taskId, callback) {
+function scheduleTask(appId, taskId, options, callback) {
     assert.strictEqual(typeof appId, 'string');
     assert.strictEqual(typeof taskId, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     if (!gInitialized) initializeSync();
@@ -49,7 +52,7 @@ function scheduleTask(appId, taskId, callback) {
     if (Object.keys(gActiveTasks).length >= TASK_CONCURRENCY) {
         debug(`Reached concurrency limit, queueing task id ${taskId}`);
         tasks.update(taskId, { percent: 1, message: 'Waiting for other app tasks to complete' }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
+        gPendingTasks.push({ appId, taskId, options, callback });
         return;
     }
 
@@ -58,7 +61,7 @@ function scheduleTask(appId, taskId, callback) {
     if (lockError) {
         debug(`Could not get lock. ${lockError.message}, queueing task id ${taskId}`);
         tasks.update(taskId, { percent: 1, message: waitText(lockError.operation) }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
+        gPendingTasks.push({ appId, taskId, options, callback });
         return;
     }
 
@@ -68,20 +71,28 @@ function scheduleTask(appId, taskId, callback) {
 
     if (!fs.existsSync(path.dirname(logFile))) safe.fs.mkdirSync(path.dirname(logFile)); // ensure directory
 
-    tasks.startTask(taskId, { logFile, timeout: 20 * 60 * 60 * 1000 /* 20 hours */ }, function (error, result) {
+    scheduler.suspendJobs(appId);
+
+    tasks.startTask(taskId, Object.assign(options, { logFile }), function (error, result) {
         callback(error, result);
 
         delete gActiveTasks[appId];
         locker.unlock(locker.OP_APPTASK); // unlock event will trigger next task
+
+        scheduler.resumeJobs(appId);
     });
 }
 
 function startNextTask() {
-    if (gPendingTasks.length === 0) return;
+    if (gPendingTasks.length === 0) {
+        // rebuild sftp when task queue is empty. this minimizes risk of sftp rebuild overlapping with other app tasks
+        services.rebuildService('sftp', error => { if (error) debug('Unable to rebuild sftp:', error); });
+        return;
+    }
 
     assert(Object.keys(gActiveTasks).length < TASK_CONCURRENCY);
 
     const t = gPendingTasks.shift();
-    scheduleTask(t.appId, t.taskId, t.callback);
+    scheduleTask(t.appId, t.taskId, t.options, t.callback);
 }
 

src/autoconfig.xml.ejs

@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<clientConfig version="1.1">
+  <emailProvider id="<%= domain %>">
+    <domain><%= domain %></domain>
+    <displayName>Cloudron Mail</displayName>
+    <displayShortName>Cloudron</displayShortName>
+    <incomingServer type="imap">
+      <hostname><%= mailFqdn %></hostname>
+      <port>993</port>
+      <socketType>SSL</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </incomingServer>
+    <outgoingServer type="smtp">
+      <hostname><%= mailFqdn %></hostname>
+      <port>587</port>
+      <socketType>STARTTLS</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+      <addThisServer>true</addThisServer>
+    </outgoingServer>
+
+    <documentation url="http://cloudron.io/email/#autodiscover">
+       <descr lang="en">Cloudron Email</descr>
+    </documentation>
+
+  </emailProvider>
+</clientConfig>
+

src/backupdb.js

@@ -6,27 +6,21 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var BACKUPS_FIELDS = [ 'id', 'creationTime', 'packageVersion', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs', 'encryptionVersion' ];
+var BACKUPS_FIELDS = [ 'id', 'identifier', 'creationTime', 'packageVersion', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs', 'encryptionVersion' ];
 
 exports = module.exports = {
-    add: add,
+    add,
 
-    getByTypeAndStatePaged: getByTypeAndStatePaged,
-    getByTypePaged: getByTypePaged,
+    getByTypePaged,
+    getByIdentifierPaged,
+    getByIdentifierAndStatePaged,
 
-    get: get,
-    del: del,
-    update: update,
-    getByAppIdPaged: getByAppIdPaged,
+    get,
+    del,
+    update,
+    list,
 
-    _clear: clear,
-
-    BACKUP_TYPE_APP: 'app',
-    BACKUP_TYPE_BOX: 'box',
-
-    BACKUP_STATE_NORMAL: 'normal', // should rename to created to avoid listing in UI?
-    BACKUP_STATE_CREATING: 'creating',
-    BACKUP_STATE_ERROR: 'error'
+    _clear: clear
 };
 
 function postProcess(result) {
@@ -38,15 +32,15 @@ function postProcess(result) {
     delete result.manifestJson;
 }
 
-function getByTypeAndStatePaged(type, state, page, perPage, callback) {
-    assert(type === exports.BACKUP_TYPE_APP || type === exports.BACKUP_TYPE_BOX);
+function getByIdentifierAndStatePaged(identifier, state, page, perPage, callback) {
+    assert.strictEqual(typeof identifier, 'string');
     assert.strictEqual(typeof state, 'string');
     assert(typeof page === 'number' && page > 0);
     assert(typeof perPage === 'number' && perPage > 0);
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? ORDER BY creationTime DESC LIMIT ?,?',
-        [ type, state, (page-1)*perPage, perPage ], function (error, results) {
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE identifier = ? AND state = ? ORDER BY creationTime DESC LIMIT ?,?',
+        [ identifier, state, (page-1)*perPage, perPage ], function (error, results) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
             results.forEach(function (result) { postProcess(result); });
@@ -56,7 +50,7 @@ function getByTypeAndStatePaged(type, state, page, perPage, callback) {
 }
 
 function getByTypePaged(type, page, perPage, callback) {
-    assert(type === exports.BACKUP_TYPE_APP || type === exports.BACKUP_TYPE_BOX);
+    assert.strictEqual(typeof type, 'string');
     assert(typeof page === 'number' && page > 0);
     assert(typeof perPage === 'number' && perPage > 0);
     assert.strictEqual(typeof callback, 'function');
@@ -71,15 +65,29 @@ function getByTypePaged(type, page, perPage, callback) {
         });
 }
 
-function getByAppIdPaged(page, perPage, appId, callback) {
+function getByIdentifierPaged(identifier, page, perPage, callback) {
+    assert.strictEqual(typeof identifier, 'string');
     assert(typeof page === 'number' && page > 0);
     assert(typeof perPage === 'number' && perPage > 0);
-    assert.strictEqual(typeof appId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    // box versions (0.93.x and below) used to use appbackup_ prefix
-    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? AND id LIKE ? ORDER BY creationTime DESC LIMIT ?,?',
-        [ exports.BACKUP_TYPE_APP, exports.BACKUP_STATE_NORMAL, '%app%\\_' + appId + '\\_%', (page-1)*perPage, perPage ], function (error, results) {
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE identifier = ? ORDER BY creationTime DESC LIMIT ?,?',
+        [ identifier, (page-1)*perPage, perPage ], function (error, results) {
+            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+
+            results.forEach(function (result) { postProcess(result); });
+
+            callback(null, results);
+        });
+}
+
+function list(page, perPage, callback) {
+    assert(typeof page === 'number' && page > 0);
+    assert(typeof perPage === 'number' && perPage > 0);
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups ORDER BY creationTime DESC LIMIT ?,?',
+        [ (page-1)*perPage, perPage ], function (error, results) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
             results.forEach(function (result) { postProcess(result); });
@@ -108,7 +116,9 @@ function add(id, data, callback) {
     assert.strictEqual(typeof id, 'string');
     assert(data.encryptionVersion === null || typeof data.encryptionVersion === 'number');
     assert.strictEqual(typeof data.packageVersion, 'string');
-    assert(data.type === exports.BACKUP_TYPE_APP || data.type === exports.BACKUP_TYPE_BOX);
+    assert.strictEqual(typeof data.type, 'string');
+    assert.strictEqual(typeof data.identifier, 'string');
+    assert.strictEqual(typeof data.state, 'string');
     assert(util.isArray(data.dependsOn));
     assert.strictEqual(typeof data.manifest, 'object');
     assert.strictEqual(typeof data.format, 'string');
@@ -117,8 +127,8 @@ function add(id, data, callback) {
     var creationTime = data.creationTime || new Date(); // allow tests to set the time
     var manifestJson = JSON.stringify(data.manifest);
 
-    database.query('INSERT INTO backups (id, encryptionVersion, packageVersion, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)',
-        [ id, data.encryptionVersion, data.packageVersion, data.type, creationTime, exports.BACKUP_STATE_NORMAL, data.dependsOn.join(','), manifestJson, data.format ],
+    database.query('INSERT INTO backups (id, identifier, encryptionVersion, packageVersion, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
+        [ id, data.identifier, data.encryptionVersion, data.packageVersion, data.type, creationTime, data.state, data.dependsOn.join(','), manifestJson, data.format ],
         function (error) {
             if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS));
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));

src/backups.js

@@ -1,38 +1,45 @@
 'use strict';
 
 exports = module.exports = {
-    testConfig: testConfig,
-    testProviderConfig: testProviderConfig,
+    testConfig,
+    testProviderConfig,
 
-    getByStatePaged: getByStatePaged,
-    getByAppIdPaged: getByAppIdPaged,
+    getByIdentifierAndStatePaged,
 
-    get: get,
+    get,
 
-    startBackupTask: startBackupTask,
-    ensureBackup: ensureBackup,
+    startBackupTask,
 
-    restore: restore,
+    restore,
 
-    backupApp: backupApp,
-    downloadApp: downloadApp,
+    backupApp,
+    downloadApp,
 
-    backupBoxAndApps: backupBoxAndApps,
+    backupBoxAndApps,
 
-    upload: upload,
+    upload,
 
-    startCleanupTask: startCleanupTask,
-    cleanup: cleanup,
-    cleanupCacheFilesSync: cleanupCacheFilesSync,
+    startCleanupTask,
+    cleanup,
+    cleanupCacheFilesSync,
 
-    injectPrivateFields: injectPrivateFields,
-    removePrivateFields: removePrivateFields,
+    injectPrivateFields,
+    removePrivateFields,
 
-    checkConfiguration: checkConfiguration,
+    checkConfiguration,
 
-    configureCollectd: configureCollectd,
+    configureCollectd,
 
-    generateEncryptionKeysSync: generateEncryptionKeysSync,
+    generateEncryptionKeysSync,
+
+    BACKUP_IDENTIFIER_BOX: 'box',
+
+    BACKUP_TYPE_APP: 'app',
+    BACKUP_TYPE_BOX: 'box',
+
+    BACKUP_STATE_NORMAL: 'normal', // should rename to created to avoid listing in UI?
+    BACKUP_STATE_CREATING: 'creating',
+    BACKUP_STATE_ERROR: 'error',
 
     // for testing
     _getBackupFilePath: getBackupFilePath,
@@ -41,39 +48,38 @@ exports = module.exports = {
     _applyBackupRetentionPolicy: applyBackupRetentionPolicy
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     async = require('async'),
     assert = require('assert'),
     backupdb = require('./backupdb.js'),
     BoxError = require('./boxerror.js'),
     collectd = require('./collectd.js'),
     constants = require('./constants.js'),
+    CronJob = require('cron').CronJob,
     crypto = require('crypto'),
     database = require('./database.js'),
     DataLayout = require('./datalayout.js'),
     debug = require('debug')('box:backups'),
-    df = require('@sindresorhus/df'),
     ejs = require('ejs'),
     eventlog = require('./eventlog.js'),
     fs = require('fs'),
     locker = require('./locker.js'),
     moment = require('moment'),
-    mkdirp = require('mkdirp'),
     once = require('once'),
     path = require('path'),
     paths = require('./paths.js'),
     progressStream = require('progress-stream'),
-    prettyBytes = require('pretty-bytes'),
     safe = require('safetydance'),
     shell = require('./shell.js'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     syncer = require('./syncer.js'),
     tar = require('tar-fs'),
     tasks = require('./tasks.js'),
     TransformStream = require('stream').Transform,
     util = require('util'),
-    zlib = require('zlib');
+    zlib = require('zlib'),
+    _ = require('underscore');
 
 const BACKUP_UPLOAD_CMD = path.join(__dirname, 'scripts/backupupload.js');
 const COLLECTD_CONFIG_EJS = fs.readFileSync(__dirname + '/collectd/cloudron-backup.ejs', { encoding: 'utf8' });
@@ -87,6 +93,9 @@ function debugApp(app) {
 // choose which storage backend we use for test purpose we use s3
 function api(provider) {
     switch (provider) {
+    case 'nfs': return require('./storage/filesystem.js');
+    case 'cifs': return require('./storage/filesystem.js');
+    case 'sshfs': return require('./storage/filesystem.js');
     case 's3': return require('./storage/s3.js');
     case 'gcs': return require('./storage/gcs.js');
     case 'filesystem': return require('./storage/filesystem.js');
@@ -96,8 +105,10 @@ function api(provider) {
     case 'exoscale-sos': return require('./storage/s3.js');
     case 'wasabi': return require('./storage/s3.js');
     case 'scaleway-objectstorage': return require('./storage/s3.js');
+    case 'backblaze-b2': return require('./storage/s3.js');
     case 'linode-objectstorage': return require('./storage/s3.js');
     case 'ovh-objectstorage': return require('./storage/s3.js');
+    case 'ionos-objectstorage': return require('./storage/s3.js');
     case 'noop': return require('./storage/noop.js');
     default: return null;
     }
@@ -133,8 +144,8 @@ function testConfig(backupConfig, callback) {
 
     if (backupConfig.format !== 'tgz' && backupConfig.format !== 'rsync') return callback(new BoxError(BoxError.BAD_FIELD, 'unknown format', { field: 'format' }));
 
-    // remember to adjust the cron ensureBackup task interval accordingly
-    if (backupConfig.intervalSecs < 6 * 60 * 60) return callback(new BoxError(BoxError.BAD_FIELD, 'Interval must be atleast 6 hours', { field: 'intervalSecs' }));
+    const job = safe.safeCall(function () { return new CronJob(backupConfig.schedulePattern); });
+    if (!job) return callback(new BoxError(BoxError.BAD_FIELD, 'Invalid schedule pattern', { field: 'schedulePattern' }));
 
     if ('password' in backupConfig) {
         if (typeof backupConfig.password !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'password must be a string', { field: 'password' }));
@@ -143,6 +154,7 @@ function testConfig(backupConfig, callback) {
 
     const policy = backupConfig.retentionPolicy;
     if (!policy) return callback(new BoxError(BoxError.BAD_FIELD, 'retentionPolicy is required', { field: 'retentionPolicy' }));
+    if (!['keepWithinSecs','keepDaily','keepWeekly','keepMonthly','keepYearly'].find(k => !!policy[k])) return callback(new BoxError(BoxError.BAD_FIELD, 'properties missing', { field: 'retentionPolicy' }));
     if ('keepWithinSecs' in policy && typeof policy.keepWithinSecs !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepWithinSecs must be a number', { field: 'retentionPolicy' }));
     if ('keepDaily' in policy && typeof policy.keepDaily !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepDaily must be a number', { field: 'retentionPolicy' }));
     if ('keepWeekly' in policy && typeof policy.keepWeekly !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepWeekly must be a number', { field: 'retentionPolicy' }));
@@ -175,26 +187,14 @@ function generateEncryptionKeysSync(password) {
     };
 }
 
-function getByStatePaged(state, page, perPage, callback) {
+function getByIdentifierAndStatePaged(identifier, state, page, perPage, callback) {
+    assert.strictEqual(typeof identifier, 'string');
     assert.strictEqual(typeof state, 'string');
     assert(typeof page === 'number' && page > 0);
     assert(typeof perPage === 'number' && perPage > 0);
     assert.strictEqual(typeof callback, 'function');
 
-    backupdb.getByTypeAndStatePaged(backupdb.BACKUP_TYPE_BOX, state, page, perPage, function (error, results) {
-        if (error) return callback(error);
-
-        callback(null, results);
-    });
-}
-
-function getByAppIdPaged(page, perPage, appId, callback) {
-    assert(typeof page === 'number' && page > 0);
-    assert(typeof perPage === 'number' && perPage > 0);
-    assert.strictEqual(typeof appId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    backupdb.getByAppIdPaged(page, perPage, appId, function (error, results) {
+    backupdb.getByIdentifierAndStatePaged(identifier, state, page, perPage, function (error, results) {
         if (error) return callback(error);
 
         callback(null, results);
@@ -212,16 +212,19 @@ function get(backupId, callback) {
     });
 }
 
+// This is not part of the storage api, since we don't want to pull the "format" logistics into that
 function getBackupFilePath(backupConfig, backupId, format) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof backupId, 'string');
     assert.strictEqual(typeof format, 'string');
 
+    const backupPath = api(backupConfig.provider).getBackupPath(backupConfig);
+
     if (format === 'tgz') {
         const fileType = backupConfig.encryption ? '.tar.gz.enc' : '.tar.gz';
-        return path.join(backupConfig.prefix || backupConfig.backupFolder || '', backupId+fileType);
+        return path.join(backupPath, backupId+fileType);
     } else {
-        return path.join(backupConfig.prefix || backupConfig.backupFolder || '', backupId);
+        return path.join(backupPath, backupId);
     }
 }
 
@@ -378,9 +381,11 @@ function createReadStream(sourceFile, encryption) {
 
     stream.on('error', function (error) {
         debug(`createReadStream: read stream error at ${sourceFile}`, error);
-        ps.emit('error', new BoxError(BoxError.FS_ERROR, `Error reading ${sourceFile}: ${error.message}`));
+        ps.emit('error', new BoxError(BoxError.FS_ERROR, `Error reading ${sourceFile}: ${error.message} ${error.code}`));
     });
 
+    stream.on('open', () => ps.emit('open'));
+
     if (encryption) {
         let encryptStream = new EncryptStream(encryption);
 
@@ -513,18 +518,19 @@ function sync(backupConfig, backupId, dataLayout, progressCallback, callback) {
                 progressCallback({ message: `Adding ${task.path}` + (retryCount > 1 ?  ` (Try ${retryCount})` : '') });
                 debug(`Adding ${task.path} position ${task.position} try ${retryCount}`);
                 var stream = createReadStream(dataLayout.toLocalPath('./' + task.path), backupConfig.encryption);
-                stream.on('error', function (error) {
-                    debug(`read stream error for ${task.path}: ${error.message}`);
-                    retryCallback();
-                }); // ignore error if file disappears
+                stream.on('error', (error) => retryCallback(error.message.includes('ENOENT') ? null : error)); // ignore error if file disappears
                 stream.on('progress', function (progress) {
                     const transferred = Math.round(progress.transferred/1024/1024), speed = Math.round(progress.speed/1024/1024);
                     if (!transferred && !speed) return progressCallback({ message: `Uploading ${task.path}` }); // 0M@0MBps looks wrong
                     progressCallback({ message: `Uploading ${task.path}: ${transferred}M@${speed}MBps` }); // 0M@0MBps looks wrong
                 });
-                api(backupConfig.provider).upload(backupConfig, backupFilePath, stream, function (error) {
-                    debug(error ? `Error uploading ${task.path} try ${retryCount}: ${error.message}` : `Uploaded ${task.path}`);
-                    retryCallback(error);
+                // only create the destination path when we have confirmation that the source is available. otherwise, we end up with
+                // files owned as 'root' and the cp later will fail
+                stream.on('open', function () {
+                    api(backupConfig.provider).upload(backupConfig, backupFilePath, stream, function (error) {
+                        debug(error ? `Error uploading ${task.path} try ${retryCount}: ${error.message}` : `Uploaded ${task.path}`);
+                        retryCallback(error);
+                    });
                 });
             }
         }, iteratorCallback);
@@ -544,53 +550,33 @@ function saveFsMetadata(dataLayout, metadataFile, callback) {
     // contains paths prefixed with './'
     let metadata = {
         emptyDirs: [],
-        execFiles: []
+        execFiles: [],
+        symlinks: []
     };
 
+    // we assume small number of files. spawnSync will raise a ENOBUFS error after maxBuffer
     for (let lp of dataLayout.localPaths()) {
-        var emptyDirs = safe.child_process.execSync(`find ${lp} -type d -empty\n`, { encoding: 'utf8' });
-        if (emptyDirs === null) return callback(safe.error);
+        const emptyDirs = safe.child_process.execSync(`find ${lp} -type d -empty`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 30 });
+        if (emptyDirs === null) return callback(new BoxError(BoxError.FS_ERROR, `Error finding empty dirs: ${safe.error.message}`));
         if (emptyDirs.length) metadata.emptyDirs = metadata.emptyDirs.concat(emptyDirs.trim().split('\n').map((ed) => dataLayout.toRemotePath(ed)));
 
-        var execFiles = safe.child_process.execSync(`find ${lp} -type f -executable\n`, { encoding: 'utf8' });
-        if (execFiles === null) return callback(safe.error);
-
+        const execFiles = safe.child_process.execSync(`find ${lp} -type f -executable`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 30 });
+        if (execFiles === null) return callback(new BoxError(BoxError.FS_ERROR, `Error finding executables: ${safe.error.message}`));
         if (execFiles.length) metadata.execFiles = metadata.execFiles.concat(execFiles.trim().split('\n').map((ef) => dataLayout.toRemotePath(ef)));
+
+        const symlinks = safe.child_process.execSync(`find ${lp} -type l`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 30 });
+        if (symlinks === null) return callback(new BoxError(BoxError.FS_ERROR, `Error finding symlinks: ${safe.error.message}`));
+        if (symlinks.length) metadata.symlinks = metadata.symlinks.concat(symlinks.trim().split('\n').map((sl) => {
+            const target = safe.fs.readlinkSync(sl);
+            return { path: dataLayout.toRemotePath(sl), target };
+        }));
     }
 
-    if (!safe.fs.writeFileSync(metadataFile, JSON.stringify(metadata, null, 4))) return callback(safe.error);
+    if (!safe.fs.writeFileSync(metadataFile, JSON.stringify(metadata, null, 4))) return callback(new BoxError(BoxError.FS_ERROR, `Error writing fs metadata: ${safe.error.message}`));
 
     callback();
 }
 
-// the du call in the function below requires root
-function checkFreeDiskSpace(backupConfig, dataLayout, callback) {
-    assert.strictEqual(typeof backupConfig, 'object');
-    assert(dataLayout instanceof DataLayout, 'dataLayout must be a DataLayout');
-    assert.strictEqual(typeof callback, 'function');
-
-    if (backupConfig.provider !== 'filesystem') return callback();
-
-    let used = 0;
-    for (let localPath of dataLayout.localPaths()) {
-        debug(`checkFreeDiskSpace: getting disk usage of ${localPath}`);
-        let result = safe.child_process.execSync(`du -Dsb ${localPath}`, { encoding: 'utf8' });
-        if (!result) return callback(new BoxError(BoxError.FS_ERROR, safe.error));
-        used += parseInt(result, 10);
-    }
-
-    debug(`checkFreeDiskSpace: ${used} bytes`);
-
-    df.file(backupConfig.backupFolder).then(function (diskUsage) {
-        const needed = used + (1024 * 1024 * 1024); // check if there is atleast 1GB left afterwards
-        if (diskUsage.available <= needed) return callback(new BoxError(BoxError.FS_ERROR, `Not enough disk space for backup. Needed: ${prettyBytes(needed)} Available: ${prettyBytes(diskUsage.available)}`));
-
-        callback(null);
-    }).catch(function (error) {
-        callback(new BoxError(BoxError.FS_ERROR, error));
-    });
-}
-
 // this function is called via backupupload (since it needs root to traverse app's directory)
 function upload(backupId, format, dataLayoutString, progressCallback, callback) {
     assert.strictEqual(typeof backupId, 'string');
@@ -606,7 +592,7 @@ function upload(backupId, format, dataLayoutString, progressCallback, callback)
     settings.getBackupConfig(function (error, backupConfig) {
         if (error) return callback(error);
 
-        checkFreeDiskSpace(backupConfig, dataLayout, function (error) {
+        api(backupConfig.provider).checkPreconditions(backupConfig, dataLayout, function (error) {
             if (error) return callback(error);
 
             if (format === 'tgz') {
@@ -705,7 +691,7 @@ function restoreFsMetadata(dataLayout, metadataFile, callback) {
     if (metadata === null) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Error parsing fsmetadata.json:' + safe.error.message));
 
     async.eachSeries(metadata.emptyDirs, function createPath(emptyDir, iteratorDone) {
-        mkdirp(dataLayout.toLocalPath(emptyDir), iteratorDone);
+        fs.mkdir(dataLayout.toLocalPath(emptyDir), { recursive: true }, iteratorDone);
     }, function (error) {
         if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `unable to create path: ${error.message}`));
 
@@ -714,7 +700,19 @@ function restoreFsMetadata(dataLayout, metadataFile, callback) {
         }, function (error) {
             if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `unable to chmod: ${error.message}`));
 
-            callback();
+            async.eachSeries(metadata.symlinks || [], function createSymlink(symlink, iteratorDone) {
+                if (!symlink.target) return iteratorDone();
+                // the path may not exist if we had a directory full of symlinks
+                fs.mkdir(path.dirname(dataLayout.toLocalPath(symlink.path)), { recursive: true }, function (error) {
+                    if (error) return iteratorDone(error);
+
+                    fs.symlink(symlink.target, dataLayout.toLocalPath(symlink.path), 'file', iteratorDone);
+                });
+            }, function (error) {
+                if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `unable to symlink: ${error.message}`));
+
+                callback();
+            });
         });
     });
 }
@@ -737,7 +735,7 @@ function downloadDir(backupConfig, backupFilePath, dataLayout, progressCallback,
         }
         const destFilePath = dataLayout.toLocalPath('./' + relativePath);
 
-        mkdirp(path.dirname(destFilePath), function (error) {
+        fs.mkdir(path.dirname(destFilePath), { recursive: true }, function (error) {
             if (error) return done(new BoxError(BoxError.FS_ERROR, error.message));
 
             async.retry({ times: 5, interval: 20000 }, function (retryCallback) {
@@ -828,7 +826,9 @@ function restore(backupConfig, backupId, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    const dataLayout = new DataLayout(paths.BOX_DATA_DIR, []);
+    const boxDataDir = safe.fs.realpathSync(paths.BOX_DATA_DIR);
+    if (!boxDataDir) return callback(new BoxError(BoxError.FS_ERROR, `Error resolving boxdata: ${safe.error.message}`));
+    const dataLayout = new DataLayout(boxDataDir, []);
 
     download(backupConfig, backupId, backupConfig.format, dataLayout, progressCallback, function (error) {
         if (error) return callback(error);
@@ -852,7 +852,7 @@ function downloadApp(app, restoreConfig, progressCallback, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     const appDataDir = safe.fs.realpathSync(path.join(paths.APPS_DATA_DIR, app.id));
-    if (!appDataDir) return callback(safe.error);
+    if (!appDataDir) return callback(new BoxError(BoxError.FS_ERROR, safe.error.message));
     const dataLayout = new DataLayout(appDataDir, app.dataDir ? [{ localDir: app.dataDir, remoteDir: 'data' }] : []);
 
     const startTime = new Date();
@@ -874,15 +874,23 @@ function runBackupUpload(uploadConfig, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    const { backupId, format, dataLayout, progressTag } = uploadConfig;
+    const { backupId, backupConfig, dataLayout, progressTag } = uploadConfig;
     assert.strictEqual(typeof backupId, 'string');
-    assert.strictEqual(typeof format, 'string');
+    assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof progressTag, 'string');
     assert(dataLayout instanceof DataLayout, 'dataLayout must be a DataLayout');
 
     let result = ''; // the script communicates error result as a string
 
-    shell.sudo(`backup-${backupId}`, [ BACKUP_UPLOAD_CMD, backupId, format, dataLayout.toString() ], { preserveEnv: true, ipc: true }, function (error) {
+    // https://stackoverflow.com/questions/48387040/node-js-recommended-max-old-space-size
+    const envCopy = Object.assign({}, process.env);
+    if (backupConfig.memoryLimit && backupConfig.memoryLimit >= 2*1024*1024*1024) {
+        const heapSize = Math.min((backupConfig.memoryLimit/1024/1024) - 256, 8192);
+        debug(`runBackupUpload: adjusting heap size to ${heapSize}M`);
+        envCopy.NODE_OPTIONS = `--max-old-space-size=${heapSize}`;
+    }
+
+    shell.sudo(`backup-${backupId}`, [ BACKUP_UPLOAD_CMD, backupId, backupConfig.format, dataLayout.toString() ], { env: envCopy, preserveEnv: true, ipc: true }, function (error) {
         if (error && (error.code === null /* signal */ || (error.code !== 0 && error.code !== 50))) { // backuptask crashed
             return callback(new BoxError(BoxError.INTERNAL_ERROR, 'Backuptask crashed'));
         } else if (error && error.code === 50) { // exited with error
@@ -927,9 +935,13 @@ function snapshotBox(progressCallback, callback) {
 
     progressCallback({ message: 'Snapshotting box' });
 
+    const startTime = new Date();
+
     database.exportToFile(`${paths.BOX_DATA_DIR}/box.mysqldump`, function (error) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
+        debug(`snapshotBox: took ${(new Date() - startTime)/1000} seconds`);
+
         return callback();
     });
 }
@@ -939,34 +951,37 @@ function uploadBoxSnapshot(backupConfig, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    var startTime = new Date();
-
     snapshotBox(progressCallback, function (error) {
         if (error) return callback(error);
 
         const boxDataDir = safe.fs.realpathSync(paths.BOX_DATA_DIR);
-        if (!boxDataDir) return callback(safe.error);
+        if (!boxDataDir) return callback(new BoxError(BoxError.FS_ERROR, `Error resolving boxdata: ${safe.error.message}`));
 
         const uploadConfig = {
             backupId: 'snapshot/box',
-            format: backupConfig.format,
+            backupConfig,
             dataLayout: new DataLayout(boxDataDir, []),
             progressTag: 'box'
         };
 
+        progressCallback({ message: 'Uploading box snapshot' });
+
+        const startTime = new Date();
+
         runBackupUpload(uploadConfig, progressCallback, function (error) {
             if (error) return callback(error);
 
-            debug('uploadBoxSnapshot: time: %s secs', (new Date() - startTime)/1000);
+            debug(`uploadBoxSnapshot: took ${(new Date() - startTime)/1000} seconds`);
 
             setSnapshotInfo('box', { timestamp: new Date().toISOString(), format: backupConfig.format }, callback);
         });
     });
 }
 
-function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, callback) {
+function rotateBoxBackup(backupConfig, tag, options, appBackupIds, progressCallback, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof tag, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert(Array.isArray(appBackupIds));
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
@@ -982,7 +997,9 @@ function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, call
     const data = {
         encryptionVersion: backupConfig.encryption ? 2 : null,
         packageVersion: constants.VERSION,
-        type: backupdb.BACKUP_TYPE_BOX,
+        type: exports.BACKUP_TYPE_BOX,
+        state: exports.BACKUP_STATE_CREATING,
+        identifier: 'box',
         dependsOn: appBackupIds,
         manifest: null,
         format: format
@@ -993,9 +1010,9 @@ function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, call
         var copy = api(backupConfig.provider).copy(backupConfig, getBackupFilePath(backupConfig, 'snapshot/box', format), getBackupFilePath(backupConfig, backupId, format));
         copy.on('progress', (message) => progressCallback({ message: `box: ${message}` }));
         copy.on('done', function (copyBackupError) {
-            const state = copyBackupError ? backupdb.BACKUP_STATE_ERROR : backupdb.BACKUP_STATE_NORMAL;
+            const state = copyBackupError ? exports.BACKUP_STATE_ERROR : exports.BACKUP_STATE_NORMAL;
 
-            backupdb.update(backupId, { state: state }, function (error) {
+            backupdb.update(backupId, { preserveSecs: options.preserveSecs || 0, state }, function (error) {
                 if (copyBackupError) return callback(copyBackupError);
                 if (error) return callback(error);
 
@@ -1007,9 +1024,10 @@ function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, call
     });
 }
 
-function backupBoxWithAppBackupIds(appBackupIds, tag, progressCallback, callback) {
+function backupBoxWithAppBackupIds(appBackupIds, tag, options, progressCallback, callback) {
     assert(Array.isArray(appBackupIds));
     assert.strictEqual(typeof tag, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1019,13 +1037,17 @@ function backupBoxWithAppBackupIds(appBackupIds, tag, progressCallback, callback
         uploadBoxSnapshot(backupConfig, progressCallback, function (error) {
             if (error) return callback(error);
 
-            rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, callback);
+            rotateBoxBackup(backupConfig, tag, options, appBackupIds, progressCallback, callback);
         });
     });
 }
 
 function canBackupApp(app) {
     // only backup apps that are installed or specific pending states
+
+    // stopped apps cannot be backed up because addons might be down (redis)
+    if (app.runState === apps.RSTATE_STOPPED) return false;
+
     // we used to check the health here but that doesn't work for stopped apps. it's better to just fail
     // and inform the user if the backup fails and the app addons have not been setup yet.
     return  app.installationState === apps.ISTATE_INSTALLED ||
@@ -1039,15 +1061,18 @@ function snapshotApp(app, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
+    const startTime = new Date();
     progressCallback({ message: `Snapshotting app ${app.fqdn}` });
 
     if (!safe.fs.writeFileSync(path.join(paths.APPS_DATA_DIR, app.id + '/config.json'), JSON.stringify(app))) {
         return callback(new BoxError(BoxError.FS_ERROR, 'Error creating config.json: ' + safe.error.message));
     }
 
-    addons.backupAddons(app, app.manifest.addons, function (error) {
+    services.backupAddons(app, app.manifest.addons, function (error) {
         if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error.message));
 
+        debugApp(app, `snapshotApp: took ${(new Date() - startTime)/1000} seconds`);
+
         return callback(null);
     });
 }
@@ -1060,6 +1085,8 @@ function rotateAppBackup(backupConfig, app, tag, options, progressCallback, call
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
+    const startTime = new Date();
+
     var snapshotInfo = getSnapshotInfo(app.id);
 
     var manifest = snapshotInfo.restoreConfig ? snapshotInfo.restoreConfig.manifest : snapshotInfo.manifest; // compat
@@ -1072,7 +1099,9 @@ function rotateAppBackup(backupConfig, app, tag, options, progressCallback, call
     const data = {
         encryptionVersion: backupConfig.encryption ? 2 : null,
         packageVersion: manifest.version,
-        type: backupdb.BACKUP_TYPE_APP,
+        type: exports.BACKUP_TYPE_APP,
+        state: exports.BACKUP_STATE_CREATING,
+        identifier: app.id,
         dependsOn: [ ],
         manifest,
         format: format
@@ -1084,13 +1113,13 @@ function rotateAppBackup(backupConfig, app, tag, options, progressCallback, call
         var copy = api(backupConfig.provider).copy(backupConfig, getBackupFilePath(backupConfig, `snapshot/app_${app.id}`, format), getBackupFilePath(backupConfig, backupId, format));
         copy.on('progress', (message) => progressCallback({ message: `${message} (${app.fqdn})` }));
         copy.on('done', function (copyBackupError) {
-            const state = copyBackupError ? backupdb.BACKUP_STATE_ERROR : backupdb.BACKUP_STATE_NORMAL;
+            const state = copyBackupError ? exports.BACKUP_STATE_ERROR : exports.BACKUP_STATE_NORMAL;
 
-            backupdb.update(backupId, { preserveSecs: options.preserveSecs || 0, state: state }, function (error) {
+            backupdb.update(backupId, { preserveSecs: options.preserveSecs || 0, state }, function (error) {
                 if (copyBackupError) return callback(copyBackupError);
                 if (error) return callback(error);
 
-                debug(`Rotated app backup of ${app.id} successfully to id ${backupId}`);
+                debug(`Rotated app backup of ${app.id} successfully to id ${backupId}. Took ${(new Date() - startTime)/1000} seconds`);
 
                 callback(null, backupId);
             });
@@ -1104,30 +1133,30 @@ function uploadAppSnapshot(backupConfig, app, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!canBackupApp(app)) return callback(); // nothing to do
-
-    var startTime = new Date();
-
     snapshotApp(app, progressCallback, function (error) {
         if (error) return callback(error);
 
         const backupId = util.format('snapshot/app_%s', app.id);
         const appDataDir = safe.fs.realpathSync(path.join(paths.APPS_DATA_DIR, app.id));
-        if (!appDataDir) return callback(safe.error);
+        if (!appDataDir) return callback(new BoxError(BoxError.FS_ERROR, `Error resolving appsdata: ${safe.error.message}`));
 
         const dataLayout = new DataLayout(appDataDir, app.dataDir ? [{ localDir: app.dataDir, remoteDir: 'data' }] : []);
 
+        progressCallback({ message: `Uploading app snapshot ${app.fqdn}`});
+
         const uploadConfig = {
             backupId,
-            format: backupConfig.format,
+            backupConfig,
             dataLayout,
             progressTag: app.fqdn
         };
 
+        const startTime = new Date();
+
         runBackupUpload(uploadConfig, progressCallback, function (error) {
             if (error) return callback(error);
 
-            debugApp(app, 'uploadAppSnapshot: %s done time: %s secs', backupId, (new Date() - startTime)/1000);
+            debugApp(app, `uploadAppSnapshot: ${backupId} done. ${(new Date() - startTime)/1000} seconds`);
 
             setSnapshotInfo(app.id, { timestamp: new Date().toISOString(), manifest: app.manifest, format: backupConfig.format }, callback);
         });
@@ -1141,7 +1170,16 @@ function backupAppWithTag(app, tag, options, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!canBackupApp(app)) return callback(); // nothing to do
+    if (!canBackupApp(app)) { // if we cannot backup, reuse it's most recent backup
+        getByIdentifierAndStatePaged(app.id, exports.BACKUP_STATE_NORMAL, 1, 1, function (error, results) {
+            if (error) return callback(error);
+            if (results.length === 0) return callback(null, null); // no backup to re-use
+
+            callback(null, results[0].id);
+        });
+
+        return;
+    }
 
     settings.getBackupConfig(function (error, backupConfig) {
         if (error) return callback(error);
@@ -1160,6 +1198,8 @@ function backupApp(app, options, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
+    if (options.snapshotOnly) return snapshotApp(app, progressCallback, callback);
+
     const tag = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
 
     debug(`backupApp - Backing up ${app.fqdn} with tag ${tag}`);
@@ -1168,7 +1208,8 @@ function backupApp(app, options, progressCallback, callback) {
 }
 
 // this function expects you to have a lock. Unlike other progressCallback this also has a progress field
-function backupBoxAndApps(progressCallback, callback) {
+function backupBoxAndApps(options, progressCallback, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1189,13 +1230,14 @@ function backupBoxAndApps(progressCallback, callback) {
                 return iteratorCallback(null, null); // nothing to backup
             }
 
-            backupAppWithTag(app, tag, { /* options */ }, (progress) => progressCallback({ percent: percent, message: progress.message }), function (error, backupId) {
-                if (error && error.reason !== BoxError.BAD_STATE) {
+            const startTime = new Date();
+            backupAppWithTag(app, tag, options, (progress) => progressCallback({ percent: percent, message: progress.message }), function (error, backupId) {
+                if (error) {
                     debugApp(app, 'Unable to backup', error);
                     return iteratorCallback(error);
                 }
 
-                debugApp(app, 'Backed up');
+                debugApp(app, `Backed up. Took ${(new Date() - startTime)/1000} seconds`);
 
                 iteratorCallback(null, backupId || null); // clear backupId if is in BAD_STATE and never backed up
             });
@@ -1207,7 +1249,7 @@ function backupBoxAndApps(progressCallback, callback) {
             progressCallback({ percent: percent, message: 'Backing up system data' });
             percent += step;
 
-            backupBoxWithAppBackupIds(backupIds, tag, (progress) => progressCallback({ percent: percent, message: progress.message }), callback);
+            backupBoxWithAppBackupIds(backupIds, tag, options, (progress) => progressCallback({ percent: percent, message: progress.message }), callback);
         });
     });
 }
@@ -1216,59 +1258,46 @@ function startBackupTask(auditSource, callback) {
     let error = locker.lock(locker.OP_FULL_BACKUP);
     if (error) return callback(new BoxError(BoxError.BAD_STATE, `Cannot backup now: ${error.message}`));
 
-    tasks.add(tasks.TASK_BACKUP, [ ], function (error, taskId) {
+    settings.getBackupConfig(function (error, backupConfig) {
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_BACKUP_START, auditSource, { taskId });
+        const memoryLimit = 'memoryLimit' in backupConfig ? Math.max(backupConfig.memoryLimit/1024/1024, 400) : 400;
 
-        tasks.startTask(taskId, { timeout: 12 * 60 * 60 * 1000 /* 12 hours */ }, function (error, backupId) {
-            locker.unlock(locker.OP_FULL_BACKUP);
-
-            const errorMessage = error ? error.message : '';
-            const timedOut = error ? error.code === tasks.ETIMEOUT : false;
-
-            eventlog.add(eventlog.ACTION_BACKUP_FINISH, auditSource, { taskId, errorMessage, timedOut, backupId });
-        });
-
-        callback(null, taskId);
-    });
-}
-
-function ensureBackup(auditSource, callback) {
-    assert.strictEqual(typeof auditSource, 'object');
-
-    debug('ensureBackup: %j', auditSource);
-
-    getByStatePaged(backupdb.BACKUP_STATE_NORMAL, 1, 1, function (error, backups) {
-        if (error) {
-            debug('Unable to list backups', error);
-            return callback(error);
-        }
-
-        settings.getBackupConfig(function (error, backupConfig) {
+        tasks.add(tasks.TASK_BACKUP, [ { /* options */ } ], function (error, taskId) {
             if (error) return callback(error);
 
-            if (backups.length !== 0 && (new Date() - new Date(backups[0].creationTime) < (backupConfig.intervalSecs - 3600) * 1000)) { // adjust 1 hour
-                debug('Previous backup was %j, no need to backup now', backups[0]);
-                return callback(null);
-            }
+            eventlog.add(eventlog.ACTION_BACKUP_START, auditSource, { taskId });
 
-            startBackupTask(auditSource, callback);
+            tasks.startTask(taskId, { timeout: 24 * 60 * 60 * 1000 /* 24 hours */, nice: 15, memoryLimit }, function (error, backupId) {
+                locker.unlock(locker.OP_FULL_BACKUP);
+
+                const errorMessage = error ? error.message : '';
+                const timedOut = error ? error.code === tasks.ETIMEOUT : false;
+
+                eventlog.add(eventlog.ACTION_BACKUP_FINISH, auditSource, { taskId, errorMessage, timedOut, backupId });
+            });
+
+            callback(null, taskId);
         });
     });
 }
 
-// backups must be descending in creationTime
-function applyBackupRetentionPolicy(backups, policy) {
+function applyBackupRetentionPolicy(backups, policy, referencedBackupIds) {
     assert(Array.isArray(backups));
     assert.strictEqual(typeof policy, 'object');
+    assert(Array.isArray(referencedBackupIds));
 
     const now = new Date();
 
     for (const backup of backups) {
-        if (backup.keepReason) continue; // already kept for some other reason
-
-        if ((now - backup.creationTime) < (backup.preserveSecs * 1000)) {
+        if (backup.state === exports.BACKUP_STATE_ERROR) {
+            backup.discardReason = 'error';
+        } else if (backup.state === exports.BACKUP_STATE_CREATING) {
+            if ((now - backup.creationTime) < 48*60*60*1000) backup.keepReason = 'creating';
+            else backup.discardReason = 'creating-too-long';
+        } else if (referencedBackupIds.includes(backup.id)) {
+            backup.keepReason = 'reference';
+        } else if ((now - backup.creationTime) < (backup.preserveSecs * 1000)) {
             backup.keepReason = 'preserveSecs';
         } else if ((now - backup.creationTime < policy.keepWithinSecs * 1000) || policy.keepWithinSecs < 0) {
             backup.keepReason = 'keepWithinSecs';
@@ -1290,18 +1319,24 @@ function applyBackupRetentionPolicy(backups, policy) {
 
         let lastPeriod = null, keptSoFar = 0;
         for (const backup of backups) {
-            if (backup.keepReason) continue; // already kept for some other reason
+            if (backup.discardReason) continue; // already discarded for some reason
+            if (backup.keepReason && backup.keepReason !== 'reference') continue; // kept for some other reason
             const period = moment(backup.creationTime).format(KEEP_FORMATS[format]);
             if (period === lastPeriod) continue; // already kept for this period
 
             lastPeriod = period;
-            backup.keepReason = format;
+            backup.keepReason = backup.keepReason ? `${backup.keepReason}+${format}` : format;
             if (++keptSoFar === n) break;
         }
     }
 
+    if (policy.keepLatest) {
+        let latestNormalBackup = backups.find(b => b.state === exports.BACKUP_STATE_NORMAL);
+        if (latestNormalBackup && !latestNormalBackup.keepReason) latestNormalBackup.keepReason = 'latest';
+    }
+
     for (const backup of backups) {
-        if (backup.keepReason) debug(`applyBackupRetentionPolicy: ${backup.id} ${backup.type} ${backup.keepReason}`);
+        debug(`applyBackupRetentionPolicy: ${backup.id} ${backup.type} ${backup.keepReason || backup.discardReason || 'unprocessed'}`);
     }
 }
 
@@ -1311,7 +1346,7 @@ function cleanupBackup(backupConfig, backup, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    var backupFilePath = getBackupFilePath(backupConfig, backup.id, backup.format);
+    const backupFilePath = getBackupFilePath(backupConfig, backup.id, backup.format);
 
     function done(error) {
         if (error) {
@@ -1350,60 +1385,52 @@ function cleanupAppBackups(backupConfig, referencedAppBackupIds, progressCallbac
 
     let removedAppBackupIds = [];
 
-    // we clean app backups of any state because the ones to keep are determined by the box cleanup code
-    backupdb.getByTypePaged(backupdb.BACKUP_TYPE_APP, 1, 1000, function (error, appBackups) {
+    apps.getAll(function (error, allApps) {
         if (error) return callback(error);
 
-        for (const appBackup of appBackups) { // set the reason so that policy filter can skip it
-            if (referencedAppBackupIds.includes(appBackup.id)) appBackup.keepReason = 'reference';
-        }
+        const allAppIds = allApps.map(a => a.id);
 
-        applyBackupRetentionPolicy(appBackups, backupConfig.retentionPolicy);
+        backupdb.getByTypePaged(exports.BACKUP_TYPE_APP, 1, 1000, function (error, appBackups) {
+            if (error) return callback(error);
 
-        async.eachSeries(appBackups, function iterator(appBackup, iteratorDone) {
-            if (appBackup.keepReason) return iteratorDone();
+            // collate the backups by app id. note that the app could already have been uninstalled
+            let appBackupsById = {};
+            for (const appBackup of appBackups) {
+                if (!appBackupsById[appBackup.identifier]) appBackupsById[appBackup.identifier] = [];
+                appBackupsById[appBackup.identifier].push(appBackup);
+            }
 
-            progressCallback({ message: `Removing app backup ${appBackup.id}`});
+            // apply backup policy per app. keep latest backup only for existing apps
+            let appBackupsToRemove = [];
+            for (const appId of Object.keys(appBackupsById)) {
+                applyBackupRetentionPolicy(appBackupsById[appId], _.extend({ keepLatest: allAppIds.includes(appId) }, backupConfig.retentionPolicy), referencedAppBackupIds);
+                appBackupsToRemove = appBackupsToRemove.concat(appBackupsById[appId].filter(b => !b.keepReason));
+            }
 
-            removedAppBackupIds.push(appBackup.id);
-            cleanupBackup(backupConfig, appBackup, progressCallback, iteratorDone);
-        }, function () {
-            debug('cleanupAppBackups: done');
+            async.eachSeries(appBackupsToRemove, function iterator(appBackup, iteratorDone) {
+                progressCallback({ message: `Removing app backup (${appBackup.identifier}): ${appBackup.id}`});
+                removedAppBackupIds.push(appBackup.id);
+                cleanupBackup(backupConfig, appBackup, progressCallback, iteratorDone);
+            }, function () {
+                debug('cleanupAppBackups: done');
 
-            callback(null, removedAppBackupIds);
+                callback(null, removedAppBackupIds);
+            });
         });
     });
 }
 
-function cleanupBoxBackups(backupConfig, progressCallback, auditSource, callback) {
+function cleanupBoxBackups(backupConfig, progressCallback, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     let referencedAppBackupIds = [], removedBoxBackupIds = [];
 
-    backupdb.getByTypePaged(backupdb.BACKUP_TYPE_BOX, 1, 1000, function (error, boxBackups) {
+    backupdb.getByTypePaged(exports.BACKUP_TYPE_BOX, 1, 1000, function (error, boxBackups) {
         if (error) return callback(error);
 
-        if (boxBackups.length === 0) return callback(null, { removedBoxBackupIds, referencedAppBackupIds });
-
-        // search for the first valid backup
-        var i;
-        for (i = 0; i < boxBackups.length; i++) {
-            if (boxBackups[i].state === backupdb.BACKUP_STATE_NORMAL) break;
-        }
-
-        // keep the first valid backup
-        if (i !== boxBackups.length) {
-            debug('cleanupBoxBackups: preserving box backup %s (%j)', boxBackups[i].id, boxBackups[i].dependsOn);
-            referencedAppBackupIds = boxBackups[i].dependsOn;
-            boxBackups.splice(i, 1);
-        } else {
-            debug('cleanupBoxBackups: no box backup to preserve');
-        }
-
-        applyBackupRetentionPolicy(boxBackups, backupConfig.retentionPolicy);
+        applyBackupRetentionPolicy(boxBackups, _.extend({ keepLatest: true }, backupConfig.retentionPolicy), [] /* references */);
 
         async.eachSeries(boxBackups, function iterator(boxBackup, iteratorNext) {
             if (boxBackup.keepReason) {
@@ -1423,6 +1450,46 @@ function cleanupBoxBackups(backupConfig, progressCallback, auditSource, callback
     });
 }
 
+function cleanupMissingBackups(backupConfig, progressCallback, callback) {
+    assert.strictEqual(typeof backupConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    let page = 1, perPage = 1000, more = false, missingBackupIds = [];
+
+    async.doWhilst(function (whilstCallback) {
+        backupdb.list(page, perPage, function (error, result) {
+            if (error) return whilstCallback(error);
+
+            async.eachSeries(result, function (backup, next) {
+                let backupFilePath = getBackupFilePath(backupConfig, backup.id, backup.format);
+                if (backup.format === 'rsync') backupFilePath = backupFilePath + '/'; // add trailing slash to indicate directory
+
+                api(backupConfig.provider).exists(backupConfig, backupFilePath, function (error, exists) {
+                    if (error || exists) return next();
+
+                    progressCallback({ message: `Removing missing backup ${backup.id}`});
+
+                    backupdb.del(backup.id, function (error) {
+                        if (error) debug(`cleanupBackup: error removing ${backup.id} from database`, error);
+
+                        missingBackupIds.push(backup.id);
+
+                        next();
+                    });
+                });
+            }, function () {
+                more = result.length === perPage;
+                whilstCallback();
+            });
+        });
+    }, function (testDone) { return testDone(null, more); }, function (error) {
+        if (error) return callback(error);
+
+        return callback(null, missingBackupIds);
+    });
+}
+
 function cleanupCacheFilesSync() {
     var files = safe.fs.readdirSync(path.join(paths.BACKUP_INFO_DIR));
     if (!files) return;
@@ -1472,8 +1539,7 @@ function cleanupSnapshots(backupConfig, callback) {
     });
 }
 
-function cleanup(auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof auditSource, 'object');
+function cleanup(progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1487,7 +1553,7 @@ function cleanup(auditSource, progressCallback, callback) {
 
         progressCallback({ percent: 10, message: 'Cleaning box backups' });
 
-        cleanupBoxBackups(backupConfig, progressCallback, auditSource, function (error, { removedBoxBackupIds, referencedAppBackupIds }) {
+        cleanupBoxBackups(backupConfig, progressCallback, function (error, { removedBoxBackupIds, referencedAppBackupIds }) {
             if (error) return callback(error);
 
             progressCallback({ percent: 40, message: 'Cleaning app backups' });
@@ -1495,12 +1561,18 @@ function cleanup(auditSource, progressCallback, callback) {
             cleanupAppBackups(backupConfig, referencedAppBackupIds, progressCallback, function (error, removedAppBackupIds) {
                 if (error) return callback(error);
 
-                progressCallback({ percent: 90, message: 'Cleaning snapshots' });
+                progressCallback({ percent: 70, message: 'Cleaning missing backups' });
 
-                cleanupSnapshots(backupConfig, function (error) {
+                cleanupMissingBackups(backupConfig, progressCallback, function (error, missingBackupIds) {
                     if (error) return callback(error);
 
-                    callback(null, { removedBoxBackupIds, removedAppBackupIds });
+                    progressCallback({ percent: 90, message: 'Cleaning snapshots' });
+
+                    cleanupSnapshots(backupConfig, function (error) {
+                        if (error) return callback(error);
+
+                        callback(null, { removedBoxBackupIds, removedAppBackupIds, missingBackupIds });
+                    });
                 });
             });
         });
@@ -1509,15 +1581,16 @@ function cleanup(auditSource, progressCallback, callback) {
 
 function startCleanupTask(auditSource, callback) {
 
-    tasks.add(tasks.TASK_CLEAN_BACKUPS, [ auditSource ], function (error, taskId) {
+    tasks.add(tasks.TASK_CLEAN_BACKUPS, [], function (error, taskId) {
         if (error) return callback(error);
 
-        tasks.startTask(taskId, {}, (error, result) => { // result is { removedBoxBackups, removedAppBackups }
+        tasks.startTask(taskId, {}, (error, result) => { // result is { removedBoxBackupIds, removedAppBackupIds, missingBackupIds }
             eventlog.add(eventlog.ACTION_BACKUP_CLEANUP_FINISH, auditSource, {
                 taskId,
                 errorMessage: error ? error.message : null,
-                removedBoxBackups: result ? result.removedBoxBackups : [],
-                removedAppBackups: result ? result.removedAppBackups : []
+                removedBoxBackupIds: result ? result.removedBoxBackupIds : [],
+                removedAppBackupIds: result ? result.removedAppBackupIds : [],
+                missingBackupIds: result ? result.missingBackupIds : []
             });
         });
 
@@ -1533,9 +1606,9 @@ function checkConfiguration(callback) {
 
         let message = '';
         if (backupConfig.provider === 'noop') {
-            message = 'Cloudron backups are disabled. Please ensure this server is backed up using alternate means. See https://cloudron.io/documentation/backups/#storage-providers for more information.';
+            message = 'Cloudron backups are disabled. Please ensure this server is backed up using alternate means. See https://docs.cloudron.io/backups/#storage-providers for more information.';
         } else if (backupConfig.provider === 'filesystem' && !backupConfig.externalDisk) {
-            message = 'Cloudron backups are currently on the same disk as the Cloudron server instance. This is dangerous and can lead to complete data loss if the disk fails. See https://cloudron.io/documentation/backups/#storage-providers for storing backups in an external location.';
+            message = 'Cloudron backups are currently on the same disk as the Cloudron server instance. This is dangerous and can lead to complete data loss if the disk fails. See https://docs.cloudron.io/backups/#storage-providers for storing backups in an external location.';
         }
 
         callback(null, message);

src/boxerror.js

@@ -50,6 +50,7 @@ BoxError.FS_ERROR = 'FileSystem Error';
 BoxError.INACTIVE = 'Inactive';
 BoxError.INTERNAL_ERROR = 'Internal Error';
 BoxError.INVALID_CREDENTIALS = 'Invalid Credentials';
+BoxError.IPTABLES_ERROR = 'IPTables Error';
 BoxError.LICENSE_ERROR = 'License Error';
 BoxError.LOGROTATE_ERROR = 'Logrotate Error';
 BoxError.MAIL_ERROR = 'Mail Error';
@@ -92,6 +93,7 @@ BoxError.toHttpError = function (error) {
     case BoxError.MAIL_ERROR:
     case BoxError.DOCKER_ERROR:
     case BoxError.ADDONS_ERROR:
+    case BoxError.IPTABLES_ERROR:
         return new HttpError(424, error);
     case BoxError.DATABASE_ERROR:
     case BoxError.INTERNAL_ERROR:

src/branding.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports = module.exports = {
+    renderFooter
+};
+
+const assert = require('assert'),
+    constants = require('./constants.js');
+
+function renderFooter(footer) {
+    assert.strictEqual(typeof footer, 'string');
+
+    const year = new Date().getFullYear();
+
+    return footer.replace(/%YEAR%/g, year)
+        .replace(/%VERSION%/g, constants.VERSION);
+}
+

src/cert/acme2.js

@@ -139,7 +139,7 @@ Acme2.prototype.updateContact = function (registrationUri, callback) {
     const that = this;
     this.sendSignedRequest(registrationUri, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to update contact. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to update contact. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug(`updateContact: contact of user updated to ${that.email}`);
 
@@ -160,7 +160,7 @@ Acme2.prototype.registerUser = function (callback) {
     this.sendSignedRequest(this.directory.newAccount, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
         // 200 if already exists. 201 for new accounts
-        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register new account. Expecting 200 or 201, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to register new account. Expecting 200 or 201, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug(`registerUser: user registered keyid: ${result.headers.location}`);
 
@@ -185,8 +185,8 @@ Acme2.prototype.newOrder = function (domain, callback) {
 
     this.sendSignedRequest(this.directory.newOrder, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode === 403) return callback(new BoxError(BoxError.ACCESS_DENIED, `Forbidden sending signed request: ${result.body.detail}`));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register user. Expecting 201, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode === 403) return callback(new BoxError(BoxError.ACCESS_DENIED, `Forbidden sending new order: ${result.body.detail}`));
+        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to send new order. Expecting 201, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug('newOrder: created order %s %j', domain, result.body);
 
@@ -259,7 +259,7 @@ Acme2.prototype.notifyChallengeReady = function (challenge, callback) {
 
     this.sendSignedRequest(challenge.url, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to notify challenge. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to notify challenge. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         callback();
     });
@@ -313,7 +313,7 @@ Acme2.prototype.signCertificate = function (domain, finalizationUrl, csrDer, cal
     this.sendSignedRequest(finalizationUrl, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
         // 429 means we reached the cert limit for this domain
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to sign certificate. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to sign certificate. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         return callback(null);
     });
@@ -362,7 +362,7 @@ Acme2.prototype.downloadCertificate = function (hostname, certUrl, callback) {
         that.postAsGet(certUrl, function (error, result) {
             if (error) return retryCallback(new BoxError(BoxError.NETWORK_ERROR, `Network error when downloading certificate: ${error.message}`));
             if (result.statusCode === 202) return retryCallback(new BoxError(BoxError.TRY_AGAIN, 'Retry downloading certificate'));
-            if (result.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to get cert. Expecting 200, got %s %s', result.statusCode, result.text)));
+            if (result.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to get cert. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
             const fullChainPem = result.body; // buffer
 
@@ -572,48 +572,50 @@ Acme2.prototype.acmeFlow = function (hostname, domain, callback) {
 Acme2.prototype.getDirectory = function (callback) {
     const that = this;
 
-    request.get(this.caDirectory, { json: true, timeout: 30000 }, function (error, response) {
-        if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Network error getting directory: ${error.message}`));
-        if (response.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response code when fetching directory : ' + response.statusCode));
+    async.retry({ times: 3, interval: 20000 }, function (retryCallback) {
+        request.get(that.caDirectory, { json: true, timeout: 30000 }, function (error, response) {
+            if (error) return retryCallback(new BoxError(BoxError.NETWORK_ERROR, `Network error getting directory: ${error.message}`));
+            if (response.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response code when fetching directory : ' + response.statusCode));
 
-        if (typeof response.body.newNonce !== 'string' ||
-            typeof response.body.newOrder !== 'string' ||
-            typeof response.body.newAccount !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Invalid response body : ${response.body}`));
+            if (typeof response.body.newNonce !== 'string' ||
+                typeof response.body.newOrder !== 'string' ||
+                typeof response.body.newAccount !== 'string') return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, `Invalid response body : ${response.body}`));
 
-        that.directory = response.body;
+            that.directory = response.body;
 
-        callback(null);
-    });
+            retryCallback(null);
+        });
+    }, callback);
 };
 
-Acme2.prototype.getCertificate = function (hostname, domain, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+Acme2.prototype.getCertificate = function (vhost, domain, callback) {
+    assert.strictEqual(typeof vhost, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`getCertificate: start acme flow for ${hostname} from ${this.caDirectory}`);
+    debug(`getCertificate: start acme flow for ${vhost} from ${this.caDirectory}`);
 
-    if (hostname !== domain && this.wildcard) { // bare domain is not part of wildcard SAN
-        hostname = domains.makeWildcard(hostname);
-        debug(`getCertificate: will get wildcard cert for ${hostname}`);
+    if (vhost !== domain && this.wildcard) { // bare domain is not part of wildcard SAN
+        vhost = domains.makeWildcard(vhost);
+        debug(`getCertificate: will get wildcard cert for ${vhost}`);
     }
 
     const that = this;
     this.getDirectory(function (error) {
         if (error) return callback(error);
 
-        that.acmeFlow(hostname, domain, function (error) {
+        that.acmeFlow(vhost, domain, function (error) {
             if (error) return callback(error);
 
             var outdir = paths.APP_CERTS_DIR;
-            const certName = hostname.replace('*.', '_.');
+            const certName = vhost.replace('*.', '_.');
             callback(null, path.join(outdir, `${certName}.cert`), path.join(outdir, `${certName}.key`));
         });
     });
 };
 
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+function getCertificate(vhost, domain, options, callback) {
+    assert.strictEqual(typeof vhost, 'string'); // this can also be a wildcard domain (for alias domains)
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -623,6 +625,6 @@ function getCertificate(hostname, domain, options, callback) {
         debug(`getCertificate: attempt ${attempt++}`);
 
         let acme = new Acme2(options || { });
-        acme.getCertificate(hostname, domain, retryCallback);
+        acme.getCertificate(vhost, domain, retryCallback);
     }, callback);
 }

src/cert/caas.js

@@ -1,22 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    getCertificate: getCertificate,
-
-    // testing
-    _name: 'caas'
-};
-
-var assert = require('assert'),
-    debug = require('debug')('box:cert/caas.js');
-
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('getCertificate: using fallback certificate', hostname);
-
-    return callback(null, '', '');
-}

src/cert/fallback.js

@@ -1,22 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    getCertificate: getCertificate,
-
-    // testing
-    _name: 'fallback'
-};
-
-var assert = require('assert'),
-    debug = require('debug')('box:cert/fallback.js');
-
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('getCertificate: using fallback certificate', hostname);
-
-    return callback(null, '', '');
-}

src/cert/interface.js

@@ -1,24 +0,0 @@
-'use strict';
-
-// -------------------------------------------
-//  This file just describes the interface
-//
-//  New backends can start from here
-// -------------------------------------------
-
-exports = module.exports = {
-    getCertificate: getCertificate
-};
-
-var assert = require('assert'),
-    BoxError = require('../boxerror.js');
-
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    return callback(new BoxError(BoxError.NOT_IMPLEMENTED, 'getCertificate is not implemented'));
-}
-

src/cloudron.js

@@ -1,34 +1,35 @@
 'use strict';
 
 exports = module.exports = {
-    initialize: initialize,
-    uninitialize: uninitialize,
-    getConfig: getConfig,
-    getLogs: getLogs,
+    initialize,
+    uninitialize,
+    getConfig,
+    getLogs,
 
-    reboot: reboot,
-    isRebootRequired: isRebootRequired,
+    reboot,
+    isRebootRequired,
 
-    onActivated: onActivated,
+    onActivated,
 
-    prepareDashboardDomain: prepareDashboardDomain,
-    setDashboardDomain: setDashboardDomain,
-    setDashboardAndMailDomain: setDashboardAndMailDomain,
-    renewCerts: renewCerts,
+    setupDnsAndCert,
 
-    setupDashboard: setupDashboard,
+    prepareDashboardDomain,
+    setDashboardDomain,
+    updateDashboardDomain,
+    renewCerts,
+    syncDnsRecords,
 
-    runSystemChecks: runSystemChecks,
+    runSystemChecks
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     appstore = require('./appstore.js'),
     assert = require('assert'),
     async = require('async'),
     auditSource = require('./auditsource.js'),
     backups = require('./backups.js'),
     BoxError = require('./boxerror.js'),
+    branding = require('./branding.js'),
     constants = require('./constants.js'),
     cron = require('./cron.js'),
     debug = require('debug')('box:cloudron'),
@@ -42,10 +43,12 @@ var addons = require('./addons.js'),
     platform = require('./platform.js'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     spawn = require('child_process').spawn,
     split = require('split'),
+    sysinfo = require('./sysinfo.js'),
     tasks = require('./tasks.js'),
     users = require('./users.js');
 
@@ -66,19 +69,29 @@ function uninitialize(callback) {
 
     async.series([
         cron.stopJobs,
-        platform.stop
+        platform.stopAllTasks
     ], callback);
 }
 
-function onActivated(callback) {
+function onActivated(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     // Starting the platform after a user is available means:
     // 1. mail bounces can now be sent to the cloudron owner
     // 2. the restore code path can run without sudo (since mail/ is non-root)
     async.series([
-        platform.start,
-        cron.startJobs
+        platform.start.bind(null, options),
+        cron.startJobs,
+        function checkBackupConfiguration(done) {
+            backups.checkConfiguration(function (error, message) {
+                if (error) return done(error);
+                notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, done);
+            });
+        },
+        // disable responding to api calls via IP to not leak domain info. this is carefully placed as the last item, so it buys
+        // the UI some time to query the dashboard domain in the restore code path
+        (done) => setTimeout(() => reverseProxy.writeDefaultConfig({ activated :true }, done), 30000)
     ], callback);
 }
 
@@ -103,24 +116,49 @@ function notifyUpdate(callback) {
 
 // each of these tasks can fail. we will add some routes to fix/re-run them
 function runStartupTasks() {
-    // configure nginx to be reachable by IP
-    reverseProxy.writeDefaultConfig(NOOP_CALLBACK);
+    const tasks = [
+        // stop all the systemd tasks
+        platform.stopAllTasks,
 
-    // this configures collectd to collect backup storage metrics if filesystem is used. This is also triggerd when the settings change with the rest api
-    settings.getBackupConfig(function (error, backupConfig) {
-        if (error) return console.error('Failed to read backup config.', error);
-        backups.configureCollectd(backupConfig, NOOP_CALLBACK);
-    });
+        // this configures collectd to collect backup storage metrics if filesystem is used. This is also triggerd when the settings change with the rest api
+        function (callback) {
+            settings.getBackupConfig(function (error, backupConfig) {
+                if (error) return callback(error);
 
-    // always generate webadmin config since we have no versioning mechanism for the ejs
-    if (settings.adminDomain()) reverseProxy.writeAdminConfig(settings.adminDomain(), NOOP_CALLBACK);
+                backups.configureCollectd(backupConfig, callback);
+            });
+        },
 
-    // check activation state and start the platform
-    users.isActivated(function (error, activated) {
-        if (error) return debug(error);
-        if (!activated) return debug('initialize: not activated yet'); // not activated
+        // always generate webadmin config since we have no versioning mechanism for the ejs
+        function (callback) {
+            if (!settings.adminDomain()) return callback();
 
-        onActivated(NOOP_CALLBACK);
+            reverseProxy.writeDashboardConfig(settings.adminDomain(), callback);
+        },
+
+        // check activation state and start the platform
+        function (callback) {
+            users.isActivated(function (error, activated) {
+                if (error) return callback(error);
+
+                // configure nginx to be reachable by IP when not activated. for the moment, the IP based redirect exists even after domain is setup
+                // just in case user forgot or some network error happenned in the middle (then browser refresh takes you to activation page)
+                // we remove the config as a simple security measure to not expose IP <-> domain
+                if (!activated) {
+                    debug('runStartupTasks: not activated. generating IP based redirection config');
+                    return reverseProxy.writeDefaultConfig({ activated: false }, callback);
+                }
+
+                onActivated({}, callback);
+            });
+        }
+    ];
+
+    // we used to run tasks in parallel but simultaneous nginx reloads was causing issues
+    async.series(async.reflectAll(tasks), function (error, results) {
+        results.forEach((result, idx) => {
+            if (result.error) debug(`Startup task at index ${idx} failed: ${result.error.message}`);
+        });
     });
 }
 
@@ -139,17 +177,18 @@ function getConfig(callback) {
             mailFqdn: settings.mailFqdn(),
             version: constants.VERSION,
             isDemo: settings.isDemo(),
-            provider: settings.provider(),
             cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-            footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
-            features: appstore.getFeatures()
+            footer: branding.renderFooter(allSettings[settings.FOOTER_KEY] || constants.FOOTER),
+            features: appstore.getFeatures(),
+            profileLocked: allSettings[settings.DIRECTORY_CONFIG_KEY].lockUserProfiles,
+            mandatory2FA: allSettings[settings.DIRECTORY_CONFIG_KEY].mandatory2FA
         });
     });
 }
 
 function reboot(callback) {
     notifications.alert(notifications.ALERT_REBOOT, 'Reboot Required', '', function (error) {
-        if (error) console.error('Failed to clear reboot notification.', error);
+        if (error) debug('reboot: failed to clear reboot notification.', error);
 
         shell.sudo('reboot', [ REBOOT_CMD ], {}, callback);
     });
@@ -167,24 +206,11 @@ function runSystemChecks(callback) {
     assert.strictEqual(typeof callback, 'function');
 
     async.parallel([
-        checkBackupConfiguration,
         checkMailStatus,
         checkRebootRequired
     ], callback);
 }
 
-function checkBackupConfiguration(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('checking backup configuration');
-
-    backups.checkConfiguration(function (error, message) {
-        if (error) return callback(error);
-
-        notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, callback);
-    });
-}
-
 function checkMailStatus(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -275,7 +301,7 @@ function prepareDashboardDomain(domain, auditSource, callback) {
             const conflict = result.filter(app => app.fqdn === fqdn);
             if (conflict.length) return callback(new BoxError(BoxError.BAD_STATE, 'Dashboard location conflicts with an existing app'));
 
-            tasks.add(tasks.TASK_PREPARE_DASHBOARD_DOMAIN, [ domain, auditSource ], function (error, taskId) {
+            tasks.add(tasks.TASK_SETUP_DNS_AND_CERT, [ constants.ADMIN_LOCATION, domain, auditSource ], function (error, taskId) {
                 if (error) return callback(error);
 
                 tasks.startTask(taskId, {}, NOOP_CALLBACK);
@@ -297,12 +323,12 @@ function setDashboardDomain(domain, auditSource, callback) {
     domains.get(domain, function (error, domainObject) {
         if (error) return callback(error);
 
-        reverseProxy.writeAdminConfig(domain, function (error) {
+        reverseProxy.writeDashboardConfig(domain, function (error) {
             if (error) return callback(error);
 
             const fqdn = domains.fqdn(constants.ADMIN_LOCATION, domainObject);
 
-            settings.setAdmin(domain, fqdn, function (error) {
+            settings.setAdminLocation(domain, fqdn, function (error) {
                 if (error) return callback(error);
 
                 eventlog.add(eventlog.ACTION_DASHBOARD_DOMAIN_UPDATE, auditSource, { domain: domain, fqdn: fqdn });
@@ -314,36 +340,24 @@ function setDashboardDomain(domain, auditSource, callback) {
 }
 
 // call this only post activation because it will restart mail server
-function setDashboardAndMailDomain(domain, auditSource, callback) {
+function updateDashboardDomain(domain, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`setDashboardAndMailDomain: ${domain}`);
+    debug(`updateDashboardDomain: ${domain}`);
 
     if (settings.isDemo()) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
 
     setDashboardDomain(domain, auditSource, function (error) {
         if (error) return callback(error);
 
-        mail.onMailFqdnChanged(NOOP_CALLBACK); // this will update dns and re-configure mail server
-        addons.restartService('turn', NOOP_CALLBACK); // to update the realm variable
+        services.rebuildService('turn', NOOP_CALLBACK); // to update the realm variable
 
         callback(null);
     });
 }
 
-function setupDashboard(auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof callback, 'function');
-
-    async.series([
-        domains.prepareDashboardDomain.bind(null, settings.adminDomain(), auditSource, progressCallback),
-        setDashboardDomain.bind(null, settings.adminDomain(), auditSource)
-    ], callback);
-}
-
 function renewCerts(options, auditSource, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof auditSource, 'object');
@@ -357,3 +371,47 @@ function renewCerts(options, auditSource, callback) {
         callback(null, taskId);
     });
 }
+
+function setupDnsAndCert(subdomain, domain, auditSource, progressCallback, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    domains.get(domain, function (error, domainObject) {
+        if (error) return callback(error);
+
+        const adminFqdn = domains.fqdn(subdomain, domainObject);
+
+        sysinfo.getServerIp(function (error, ip) {
+            if (error) return callback(error);
+
+            async.series([
+                (done) => { progressCallback({ message: `Updating DNS of ${adminFqdn}` }); done(); },
+                domains.upsertDnsRecords.bind(null, subdomain, domain, 'A', [ ip ]),
+                (done) => { progressCallback({ message: `Waiting for DNS of ${adminFqdn}` }); done(); },
+                domains.waitForDnsRecord.bind(null, subdomain, domain, 'A', ip, { interval: 30000, times: 50000 }),
+                (done) => { progressCallback({ message: `Getting certificate of ${adminFqdn}` }); done(); },
+                reverseProxy.ensureCertificate.bind(null, domains.fqdn(subdomain, domainObject), domain, auditSource)
+            ], function (error) {
+                if (error) return callback(error);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function syncDnsRecords(options, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    tasks.add(tasks.TASK_SYNC_DNS_RECORDS, [ options ], function (error, taskId) {
+        if (error) return callback(error);
+
+        tasks.startTask(taskId, {}, NOOP_CALLBACK);
+
+        callback(null, taskId);
+    });
+}

src/collectd/volume.ejs

@@ -0,0 +1,9 @@
+<Plugin python>
+  <Module du>
+    <Path>
+        Instance "<%= volumeId %>"
+        Dir "<%= hostPath %>"
+    </Path>
+  </Module>
+</Plugin>
+

src/constants.js

@@ -26,7 +26,7 @@ exports = module.exports = {
 
     PORT: CLOUDRON ? 3000 : 5454,
     INTERNAL_SMTP_PORT: 2525, // this value comes from the mail container
-    SYSADMIN_PORT: 3001, // unused
+    AUTHWALL_PORT: 3001,
     LDAP_PORT: 3002,
     DOCKER_PROXY_PORT: 3003,
 
@@ -37,7 +37,14 @@ exports = module.exports = {
     DEFAULT_MEMORY_LIMIT: (256 * 1024 * 1024), // see also client.js
 
     DEMO_USERNAME: 'cloudron',
-    DEMO_BLACKLISTED_APPS: [ 'com.github.cloudtorrent' ],
+    DEMO_BLACKLISTED_APPS: [
+        'com.github.cloudtorrent',
+        'net.alltubedownload.cloudronapp',
+        'com.adguard.home.cloudronapp',
+        'com.transmissionbt.cloudronapp',
+        'io.github.sickchill.cloudronapp',
+        'to.couchpota.cloudronapp'
+    ],
 
     AUTOUPDATE_PATTERN_NEVER: 'never',
 
@@ -48,8 +55,8 @@ exports = module.exports = {
 
     SUPPORT_EMAIL: 'support@cloudron.io',
 
-    FOOTER: '&copy; 2020  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
+    FOOTER: '&copy; %YEAR%  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
 
-    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '5.1.1-test'
+    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '6.0.1-test'
 };
 

src/cron.js

@@ -1,16 +1,23 @@
 'use strict';
 
+// IMPORTANT: These patterns are together because they spin tasks which acquire a lock
+// If the patterns overlap all the time, then the task may not ever get a chance to run!
+// If you change this change dashboard patterns in settings.html
+const DEFAULT_CLEANUP_BACKUPS_PATTERN = '00 30 1,3,5,23 * * *',
+    DEFAULT_AUTOUPDATE_PATTERN = '00 00 1,3,5,23 * * *';
+
 exports = module.exports = {
-    startJobs: startJobs,
+    startJobs,
 
-    stopJobs: stopJobs,
+    stopJobs,
 
-    handleSettingsChanged: handleSettingsChanged
+    handleSettingsChanged,
+
+    DEFAULT_AUTOUPDATE_PATTERN,
 };
 
 var appHealthMonitor = require('./apphealthmonitor.js'),
     apps = require('./apps.js'),
-    appstore = require('./appstore.js'),
     assert = require('assert'),
     async = require('async'),
     auditSource = require('./auditsource.js'),
@@ -26,15 +33,13 @@ var appHealthMonitor = require('./apphealthmonitor.js'),
     settings = require('./settings.js'),
     system = require('./system.js'),
     updater = require('./updater.js'),
-    updateChecker = require('./updatechecker.js');
+    updateChecker = require('./updatechecker.js'),
+    _ = require('underscore');
 
-var gJobs = {
-    alive: null, // send periodic stats
-    appAutoUpdater: null,
-    boxAutoUpdater: null,
-    appUpdateChecker: null,
+const gJobs = {
+    autoUpdater: null,
     backup: null,
-    boxUpdateChecker: null,
+    updateChecker: null,
     systemChecks: null,
     diskSpaceChecker: null,
     certificateRenew: null,
@@ -47,7 +52,7 @@ var gJobs = {
     appHealthMonitor: null
 };
 
-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+const NOOP_CALLBACK = function (error) { if (error) debug(error); };
 
 // cron format
 // Seconds: 0-59
@@ -60,15 +65,11 @@ var NOOP_CALLBACK = function (error) { if (error) debug(error); };
 function startJobs(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    const randomMinute = Math.floor(60*Math.random());
-    gJobs.alive = new CronJob({
-        cronTime: '00 ' + randomMinute + ' * * * *', // every hour on a random minute
-        onTick: appstore.sendAliveStatus,
-        start: true
-    });
+    debug('startJobs: starting cron jobs');
 
+    const randomTick = Math.floor(60*Math.random());
     gJobs.systemChecks = new CronJob({
-        cronTime: '00 30 * * * *', // every 30 minutes. if you change this interval, change the notification messages with correct duration
+        cronTime: '00 30 2 * * *', // once a day. if you change this interval, change the notification messages with correct duration
         onTick: () => cloudron.runSystemChecks(NOOP_CALLBACK),
         start: true
     });
@@ -79,15 +80,10 @@ function startJobs(callback) {
         start: true
     });
 
-    gJobs.boxUpdateCheckerJob = new CronJob({
-        cronTime: '00 ' + randomMinute + ' 23 * * *', // once an day
-        onTick: () => updateChecker.checkBoxUpdates({ automatic: true }, NOOP_CALLBACK),
-        start: true
-    });
-
-    gJobs.appUpdateChecker = new CronJob({
-        cronTime: '00 ' + randomMinute + ' 22 * * *', // once an day
-        onTick: () => updateChecker.checkAppUpdates({ automatic: true }, NOOP_CALLBACK),
+    // this is run separately from the update itself so that the user can disable automatic updates but can still get a notification
+    gJobs.updateCheckerJob = new CronJob({
+        cronTime: `${randomTick} ${randomTick} 1,5,9,13,17,21,23 * * *`,
+        onTick: () => updateChecker.checkForUpdates({ automatic: true }, NOOP_CALLBACK),
         start: true
     });
 
@@ -98,7 +94,7 @@ function startJobs(callback) {
     });
 
     gJobs.cleanupBackups = new CronJob({
-        cronTime: '00 45 1,3,5,23 * * *', // every 6 hours. try not to overlap with ensureBackup job
+        cronTime: DEFAULT_CLEANUP_BACKUPS_PATTERN,
         onTick: backups.startCleanupTask.bind(null, auditSource.CRON, NOOP_CALLBACK),
         start: true
     });
@@ -138,8 +134,7 @@ function startJobs(callback) {
 
         const tz = allSettings[settings.TIME_ZONE_KEY];
         backupConfigChanged(allSettings[settings.BACKUP_CONFIG_KEY], tz);
-        appAutoupdatePatternChanged(allSettings[settings.APP_AUTOUPDATE_PATTERN_KEY], tz);
-        boxAutoupdatePatternChanged(allSettings[settings.BOX_AUTOUPDATE_PATTERN_KEY], tz);
+        autoupdatePatternChanged(allSettings[settings.AUTOUPDATE_PATTERN_KEY], tz);
         dynamicDnsChanged(allSettings[settings.DYNAMIC_DNS_KEY]);
 
         callback();
@@ -154,8 +149,7 @@ function handleSettingsChanged(key, value) {
     switch (key) {
     case settings.TIME_ZONE_KEY:
     case settings.BACKUP_CONFIG_KEY:
-    case settings.APP_AUTOUPDATE_PATTERN_KEY:
-    case settings.BOX_AUTOUPDATE_PATTERN_KEY:
+    case settings.AUTOUPDATE_PATTERN_KEY:
     case settings.DYNAMIC_DNS_KEY:
         debug('handleSettingsChanged: recreating all jobs');
         async.series([
@@ -172,71 +166,48 @@ function backupConfigChanged(value, tz) {
     assert.strictEqual(typeof value, 'object');
     assert.strictEqual(typeof tz, 'string');
 
-    debug(`backupConfigChanged: interval ${value.intervalSecs} (${tz})`);
+    debug(`backupConfigChanged: schedule ${value.schedulePattern} (${tz})`);
 
     if (gJobs.backup) gJobs.backup.stop();
-    let pattern;
-    if (value.intervalSecs <= 6 * 60 * 60) {
-        pattern = '00 00 1,7,13,19 * * *'; // no option but to backup in the middle of the day
-    } else {
-        pattern = '00 00 1,3,5,23 * * *'; // avoid middle of the day backups
-    }
 
     gJobs.backup = new CronJob({
-        cronTime: pattern,
-        onTick: backups.ensureBackup.bind(null, auditSource.CRON, NOOP_CALLBACK),
+        cronTime: value.schedulePattern,
+        onTick: backups.startBackupTask.bind(null, auditSource.CRON, NOOP_CALLBACK),
         start: true,
         timeZone: tz
     });
 }
 
-function boxAutoupdatePatternChanged(pattern, tz) {
+function autoupdatePatternChanged(pattern, tz) {
     assert.strictEqual(typeof pattern, 'string');
     assert.strictEqual(typeof tz, 'string');
 
-    debug(`boxAutoupdatePatternChanged: pattern - ${pattern} (${tz})`);
+    debug(`autoupdatePatternChanged: pattern - ${pattern} (${tz})`);
 
-    if (gJobs.boxAutoUpdater) gJobs.boxAutoUpdater.stop();
+    if (gJobs.autoUpdater) gJobs.autoUpdater.stop();
 
     if (pattern === constants.AUTOUPDATE_PATTERN_NEVER) return;
 
-    gJobs.boxAutoUpdater = new CronJob({
+    gJobs.autoUpdater = new CronJob({
         cronTime: pattern,
         onTick: function() {
-            var updateInfo = updateChecker.getUpdateInfo();
-            if (updateInfo.box) {
-                debug('Starting autoupdate to %j', updateInfo.box);
+            const updateInfo = updateChecker.getUpdateInfo();
+            // do box before app updates. for the off chance that the box logic fixes some app update logic issue
+            if (updateInfo.box && !updateInfo.box.unstable) {
+                debug('Starting box autoupdate to %j', updateInfo.box);
                 updater.updateToLatest({ skipBackup: false }, auditSource.CRON, NOOP_CALLBACK);
-            } else {
-                debug('No box auto updates available');
+                return;
             }
-        },
-        start: true,
-        timeZone: tz
-    });
-}
 
-function appAutoupdatePatternChanged(pattern, tz) {
-    assert.strictEqual(typeof pattern, 'string');
-    assert.strictEqual(typeof tz, 'string');
-
-    debug(`appAutoupdatePatternChanged: pattern ${pattern} (${tz})`);
-
-    if (gJobs.appAutoUpdater) gJobs.appAutoUpdater.stop();
-
-    if (pattern === constants.AUTOUPDATE_PATTERN_NEVER) return;
-
-    gJobs.appAutoUpdater = new CronJob({
-        cronTime: pattern,
-        onTick: function() {
-            var updateInfo = updateChecker.getUpdateInfo();
-            if (updateInfo.apps) {
-                debug('Starting app update to %j', updateInfo.apps);
-                apps.autoupdateApps(updateInfo.apps, auditSource.CRON, NOOP_CALLBACK);
+            const appUpdateInfo = _.omit(updateInfo, 'box');
+            if (Object.keys(appUpdateInfo).length > 0) {
+                debug('Starting app update to %j', appUpdateInfo);
+                apps.autoupdateApps(appUpdateInfo, auditSource.CRON, NOOP_CALLBACK);
             } else {
                 debug('No app auto updates available');
             }
         },
+
         start: true,
         timeZone: tz
     });

src/database.js

@@ -17,12 +17,12 @@ var assert = require('assert'),
     BoxError = require('./boxerror.js'),
     child_process = require('child_process'),
     constants = require('./constants.js'),
+    debug = require('debug')('box:database'),
     mysql = require('mysql'),
     once = require('once'),
     util = require('util');
 
-var gConnectionPool = null,
-    gDefaultConnection = null;
+var gConnectionPool = null;
 
 const gDatabase = {
     hostname: '127.0.0.1',
@@ -42,59 +42,37 @@ function initialize(callback) {
         gDatabase.hostname = require('child_process').execSync('docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" mysql-server').toString().trim();
     }
 
+    // https://github.com/mysqljs/mysql#pool-options
     gConnectionPool  = mysql.createPool({
-        connectionLimit: 5, // this has to be > 1 since we store one connection as 'default'. the rest for transactions
+        connectionLimit: 5,
         host: gDatabase.hostname,
         user: gDatabase.username,
         password: gDatabase.password,
         port: gDatabase.port,
         database: gDatabase.name,
         multipleStatements: false,
+        waitForConnections: true, // getConnection() will wait until a connection is avaiable
         ssl: false,
         timezone: 'Z' // mysql follows the SYSTEM timezone. on Cloudron, this is UTC
     });
 
     gConnectionPool.on('connection', function (connection) {
+        // connection objects are re-used. so we have to attach to the event here (once) to prevent crash
+        // note the pool also has an 'acquire' event but that is called whenever we do a getConnection()
+        connection.on('error', (error) => debug(`Connection ${connection.threadId} error: ${error.message} ${error.code}`));
+
         connection.query('USE ' + gDatabase.name);
         connection.query('SET SESSION sql_mode = \'strict_all_tables\'');
     });
 
-    reconnect(callback);
+    callback(null);
 }
 
 function uninitialize(callback) {
-    if (gConnectionPool) {
-        gConnectionPool.end(callback);
-        gConnectionPool = null;
-    } else {
-        callback(null);
-    }
-}
+    if (!gConnectionPool) return callback(null);
 
-function reconnect(callback) {
-    callback = callback ? once(callback) : function () {};
-
-    gConnectionPool.getConnection(function (error, connection) {
-        if (error) {
-            console.error('Unable to reestablish connection to database. Try again in a bit.', error.message);
-            return setTimeout(reconnect.bind(null, callback), 1000);
-        }
-
-        connection.on('error', function (error) {
-            // by design, we catch all normal errors by providing callbacks.
-            // this function should be invoked only when we have no callbacks pending and we have a fatal error
-            assert(error.fatal, 'Non-fatal error on connection object');
-
-            console.error('Unhandled mysql connection error.', error);
-
-            // This is most likely an issue an can cause double callbacks from reconnect()
-            setTimeout(reconnect.bind(null, callback), 1000);
-        });
-
-        gDefaultConnection = connection;
-
-        callback(null);
-    });
+    gConnectionPool.end(callback);
+    gConnectionPool = null;
 }
 
 function clear(callback) {
@@ -107,80 +85,43 @@ function clear(callback) {
     child_process.exec(cmd, callback);
 }
 
-function beginTransaction(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    if (gConnectionPool === null) return callback(new BoxError(BoxError.DATABASE_ERROR, 'No database connection pool.'));
-
-    gConnectionPool.getConnection(function (error, connection) {
-        if (error) {
-            console.error('Unable to get connection to database. Try again in a bit.', error.message);
-            return setTimeout(beginTransaction.bind(null, callback), 1000);
-        }
-
-        connection.beginTransaction(function (error) {
-            if (error) return callback(error);
-
-            return callback(null, connection);
-        });
-    });
-}
-
-function rollback(connection, callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    connection.rollback(function (error) {
-        if (error) console.error(error); // can this happen?
-
-        connection.release();
-        callback(null);
-    });
-}
-
-// FIXME: if commit fails, is it supposed to return an error ?
-function commit(connection, callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    connection.commit(function (error) {
-        if (error) return rollback(connection, callback);
-
-        connection.release();
-        return callback(null);
-    });
-}
-
 function query() {
-    var args = Array.prototype.slice.call(arguments);
-    var callback = args[args.length - 1];
+    const args = Array.prototype.slice.call(arguments);
+    const callback = args[args.length - 1];
     assert.strictEqual(typeof callback, 'function');
 
-    if (gDefaultConnection === null) return callback(new BoxError(BoxError.DATABASE_ERROR, 'No connection to database'));
+    if (constants.TEST && !gConnectionPool) return callback(new BoxError(BoxError.DATABASE_ERROR, 'database.js not initialized'));
 
-    args[args.length -1 ] = function (error, result) {
-        if (error && error.fatal) {
-            gDefaultConnection = null;
-            setTimeout(reconnect, 1000);
-        }
-
-        callback(error, result);
-    };
-
-    gDefaultConnection.query.apply(gDefaultConnection, args);
+    gConnectionPool.query.apply(gConnectionPool, args); // this is same as getConnection/query/release
 }
 
 function transaction(queries, callback) {
     assert(util.isArray(queries));
     assert.strictEqual(typeof callback, 'function');
 
-    beginTransaction(function (error, conn) {
+    callback = once(callback);
+
+    gConnectionPool.getConnection(function (error, connection) {
         if (error) return callback(error);
 
-        async.mapSeries(queries, function iterator(query, done) {
-            conn.query(query.query, query.args, done);
-        }, function seriesDone(error, results) {
-            if (error) return rollback(conn, callback.bind(null, error));
+        const releaseConnection = (error) => { connection.release(); callback(error); };
 
-            commit(conn, callback.bind(null, null, results));
+        connection.beginTransaction(function (error) {
+            if (error) return releaseConnection(error);
+
+            async.mapSeries(queries, function iterator(query, done) {
+                connection.query(query.query, query.args, done);
+            }, function seriesDone(error, results) {
+                if (error) return connection.rollback(() => releaseConnection(error));
+
+                connection.commit(function (error) {
+                    if (error) return connection.rollback(() => releaseConnection(error));
+
+                    connection.release();
+
+                    callback(null, results);
+                });
+            });
         });
     });
 }
@@ -203,7 +144,7 @@ function exportToFile(file, callback) {
 
     // latest mysqldump enables column stats by default which is not present in MySQL 5.7 server
     // this option must not be set in production cloudrons which still use the old mysqldump
-    const disableColStats = (constants.TEST && process.env.DESKTOP_SESSION !== 'ubuntu') ? '--column-statistics=0' : '';
+    const disableColStats = (constants.TEST && require('fs').readFileSync('/etc/lsb-release', 'utf-8').includes('20.04')) ? '--column-statistics=0' : '';
 
     var cmd = `/usr/bin/mysqldump -h "${gDatabase.hostname}" -u root -p${gDatabase.password} ${disableColStats} --single-transaction --routines --triggers ${gDatabase.name} > "${file}"`;
 

src/dns/caas.js

@@ -1,177 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    removePrivateFields: removePrivateFields,
-    injectPrivateFields: injectPrivateFields,
-    upsert: upsert,
-    get: get,
-    del: del,
-    wait: wait,
-    verifyDnsConfig: verifyDnsConfig
-};
-
-var assert = require('assert'),
-    BoxError = require('../boxerror.js'),
-    constants = require('../constants.js'),
-    debug = require('debug')('box:dns/caas'),
-    domains = require('../domains.js'),
-    settings = require('../settings.js'),
-    superagent = require('superagent'),
-    util = require('util'),
-    waitForDns = require('./waitfordns.js');
-
-function formatError(response) {
-    return util.format('Caas DNS error [%s] %j', response.statusCode, response.body);
-}
-
-function getFqdn(location, domain) {
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof domain, 'string');
-
-    return (location === '') ? domain : location + '-' + domain;
-}
-
-function removePrivateFields(domainObject) {
-    domainObject.config.token = constants.SECRET_PLACEHOLDER;
-
-    // do not return the 'key'. in caas, this is private
-    delete domainObject.fallbackCertificate.key;
-
-    return domainObject;
-}
-
-function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
-}
-
-function upsert(domainObject, location, type, values, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert(util.isArray(values));
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-
-    let fqdn = location !== '' && type === 'TXT' ? location + '.' + domainObject.domain : getFqdn(location, domainObject.domain);
-
-    debug('add: %s for zone %s of type %s with values %j', location, domainObject.domain, type, values);
-
-    var data = {
-        type: type,
-        values: values
-    };
-
-    superagent
-        .post(settings.apiServerOrigin() + '/api/v1/caas/domains/' + fqdn)
-        .query({ token: dnsConfig.token })
-        .send(data)
-        .timeout(30 * 1000)
-        .end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode === 400) return callback(new BoxError(BoxError.BAD_FIELD, result.body.message));
-            if (result.statusCode === 420) return callback(new BoxError(BoxError.BUSY));
-            if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
-
-            return callback(null);
-        });
-}
-
-function get(domainObject, location, type, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-    const fqdn = location !== '' && type === 'TXT' ? location + '.' + domainObject.domain : getFqdn(location, domainObject.domain);
-
-    debug('get: zoneName: %s subdomain: %s type: %s fqdn: %s', domainObject.domain, location, type, fqdn);
-
-    superagent
-        .get(settings.apiServerOrigin() + '/api/v1/caas/domains/' + fqdn)
-        .query({ token: dnsConfig.token, type: type })
-        .timeout(30 * 1000)
-        .end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
-
-            return callback(null, result.body.values);
-        });
-}
-
-function del(domainObject, location, type, values, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert(util.isArray(values));
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-    debug('del: %s for zone %s of type %s with values %j', location, domainObject.domain, type, values);
-
-    var data = {
-        type: type,
-        values: values
-    };
-
-    superagent
-        .del(settings.apiServerOrigin() + '/api/v1/caas/domains/' + getFqdn(location, domainObject.domain))
-        .query({ token: dnsConfig.token })
-        .send(data)
-        .timeout(30 * 1000)
-        .end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode === 400) return callback(new BoxError(BoxError.BAD_FIELD, result.body.message));
-            if (result.statusCode === 420) return callback(new BoxError(BoxError.BUSY));
-            if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
-            if (result.statusCode !== 204) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
-
-            return callback(null);
-        });
-}
-
-function wait(domainObject, location, type, value, options, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof type, 'string');
-    assert.strictEqual(typeof value, 'string');
-    assert(options && typeof options === 'object'); // { interval: 5000, times: 50000 }
-    assert.strictEqual(typeof callback, 'function');
-
-    const fqdn = domains.fqdn(location, domainObject);
-
-    waitForDns(fqdn, domainObject.zoneName, type, value, options, callback);
-}
-
-function verifyDnsConfig(domainObject, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    const dnsConfig = domainObject.config;
-
-    if (!dnsConfig.token || typeof dnsConfig.token !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'token must be a non-empty string', { field: 'token' }));
-
-    const ip = '127.0.0.1';
-
-    var credentials = {
-        token: dnsConfig.token,
-        hyphenatedSubdomains: true  // this will ensure we always use them, regardless of passed-in configs
-    };
-
-    const location = 'cloudrontestdns';
-
-    upsert(domainObject, location, 'A', [ ip ], function (error) {
-        if (error) return callback(error);
-
-        debug('verifyDnsConfig: Test A record added');
-
-        del(domainObject, location, 'A', [ ip ], function (error) {
-            if (error) return callback(error);
-
-            debug('verifyDnsConfig: Test A record removed again');
-
-            callback(null, credentials);
-        });
-    });
-}

src/dns/digitalocean.js

@@ -1,13 +1,13 @@
 'use strict';
 
 exports = module.exports = {
-    removePrivateFields: removePrivateFields,
-    injectPrivateFields: injectPrivateFields,
-    upsert: upsert,
-    get: get,
-    del: del,
-    wait: wait,
-    verifyDnsConfig: verifyDnsConfig
+    removePrivateFields,
+    injectPrivateFields,
+    upsert,
+    get,
+    del,
+    wait,
+    verifyDnsConfig
 };
 
 var assert = require('assert'),
@@ -69,7 +69,7 @@ function getInternal(dnsConfig, zoneName, name, type, callback) {
 
                 iteratorDone();
             });
-    }, function () { return !!nextPage; }, function (error) {
+    }, function (testDone) { return testDone(null, !!nextPage); }, function (error) {
         debug('getInternal:', error, JSON.stringify(matchingRecords));
 
         if (error) return callback(error);

src/dns/linode.js

@@ -1,13 +1,13 @@
 'use strict';
 
 exports = module.exports = {
-    removePrivateFields: removePrivateFields,
-    injectPrivateFields: injectPrivateFields,
-    upsert: upsert,
-    get: get,
-    del: del,
-    wait: wait,
-    verifyDnsConfig: verifyDnsConfig
+    removePrivateFields,
+    injectPrivateFields,
+    upsert,
+    get,
+    del,
+    wait,
+    verifyDnsConfig
 };
 
 let async = require('async'),
@@ -99,7 +99,7 @@ function getZoneRecords(dnsConfig, zoneName, name, type, callback) {
 
                     iteratorDone();
                 });
-        }, function () { return more; }, function (error) {
+        }, function (testDone) { return testDone(null, more); }, function (error) {
             debug('getZoneRecords:', error, JSON.stringify(records));
 
             if (error) return callback(error);
@@ -119,9 +119,10 @@ function get(domainObject, location, type, callback) {
         zoneName = domainObject.zoneName,
         name = domains.getName(domainObject, location, type) || '';
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { records } = result;
         var tmp = records.map(function (record) { return record.target; });
 
         debug('get: %j', tmp);
@@ -143,9 +144,10 @@ function upsert(domainObject, location, type, values, callback) {
 
     debug('upsert: %s for zone %s of type %s with values %j', name, zoneName, type, values);
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { zoneId, records } = result;
         let i = 0, recordIds = []; // used to track available records to update instead of create
 
         async.eachSeries(values, function (value, iteratorCallback) {
@@ -222,9 +224,10 @@ function del(domainObject, location, type, values, callback) {
         zoneName = domainObject.zoneName,
         name = domains.getName(domainObject, location, type) || '';
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { zoneId, records } = result;
         if (records.length === 0) return callback(null);
 
         var tmp = records.filter(function (record) { return values.some(function (value) { return value === record.target; }); });
@@ -287,7 +290,7 @@ function verifyDnsConfig(domainObject, callback) {
         if (error || !nameservers) return callback(new BoxError(BoxError.BAD_FIELD, error ? error.message : 'Unable to get nameservers', { field: 'nameservers' }));
 
         if (nameservers.map(function (n) { return n.toLowerCase(); }).indexOf('ns1.linode.com') === -1) {
-            debug('verifyDnsConfig: %j does not contains DO NS', nameservers);
+            debug('verifyDnsConfig: %j does not contains linode NS', nameservers);
             return callback(new BoxError(BoxError.BAD_FIELD, 'Domain nameservers are not set to Linode', { field: 'nameservers' }));
         }
 

src/dns/namecheap.js

@@ -16,10 +16,10 @@ var assert = require('assert'),
     debug = require('debug')('box:dns/namecheap'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
+    querystring = require('querystring'),
     safe = require('safetydance'),
     superagent = require('superagent'),
     sysinfo = require('../sysinfo.js'),
-    util = require('util'),
     waitForDns = require('./waitfordns.js'),
     xml2js = require('xml2js');
 
@@ -50,11 +50,9 @@ function getQuery(dnsConfig, callback) {
     });
 }
 
-function getInternal(dnsConfig, zoneName, subdomain, type, callback) {
+function getZone(dnsConfig, zoneName, callback) {
     assert.strictEqual(typeof dnsConfig, 'object');
     assert.strictEqual(typeof zoneName, 'string');
-    assert.strictEqual(typeof subdomain, 'string');
-    assert.strictEqual(typeof type, 'string');
     assert.strictEqual(typeof callback, 'function');
 
     getQuery(dnsConfig, function (error, query) {
@@ -78,20 +76,18 @@ function getInternal(dnsConfig, zoneName, subdomain, type, callback) {
 
                     return callback(new BoxError(BoxError.EXTERNAL_ERROR, errorMessage));
                 }
-                if (!tmp.CommandResponse[0]) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response'));
-                if (!tmp.CommandResponse[0].DomainDNSGetHostsResult[0]) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response'));
-
-                var hosts = result.ApiResponse.CommandResponse[0].DomainDNSGetHostsResult[0].host.map(function (h) {
-                    return h['$'];
-                });
+                const host = safe.query(tmp, 'CommandResponse[0].DomainDNSGetHostsResult[0].host');
+                if (!host) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Invalid response: ${JSON.stringify(tmp)}`));
+                if (!Array.isArray(host)) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `host is not an array: ${JSON.stringify(tmp)}`));
 
+                const hosts = host.map(h => h['$']);
                 callback(null, hosts);
             });
         });
     });
 }
 
-function setInternal(dnsConfig, zoneName, hosts, callback) {
+function setZone(dnsConfig, zoneName, hosts, callback) {
     assert.strictEqual(typeof dnsConfig, 'object');
     assert.strictEqual(typeof zoneName, 'string');
     assert(Array.isArray(hosts));
@@ -118,7 +114,10 @@ function setInternal(dnsConfig, zoneName, hosts, callback) {
             }
         });
 
-        superagent.post(ENDPOINT).query(query).end(function (error, result) {
+        // namecheap recommends sending as POSTDATA with > 10 records
+        const qs = querystring.stringify(query);
+
+        superagent.post(ENDPOINT).set('Content-Type', 'application/x-www-form-urlencoded').send(qs).end(function (error, result) {
             if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
 
             var parser = new xml2js.Parser();
@@ -146,7 +145,7 @@ function upsert(domainObject, subdomain, type, values, callback) {
     assert.strictEqual(typeof domainObject, 'object');
     assert.strictEqual(typeof subdomain, 'string');
     assert.strictEqual(typeof type, 'string');
-    assert(util.isArray(values));
+    assert(Array.isArray(values));
     assert.strictEqual(typeof callback, 'function');
 
     const dnsConfig = domainObject.config;
@@ -156,17 +155,17 @@ function upsert(domainObject, subdomain, type, values, callback) {
 
     debug('upsert: %s for zone %s of type %s with values %j', subdomain, zoneName, type, values);
 
-    getInternal(dnsConfig, zoneName, subdomain, type, function (error, result) {
+    getZone(dnsConfig, zoneName, function (error, result) {
         if (error) return callback(error);
 
         // Array to keep track of records that need to be inserted
         let toInsert = [];
 
-        for (var i = 0; i < values.length; i++) {
+        for (let i = 0; i < values.length; i++) {
             let curValue = values[i];
             let wasUpdate = false;
 
-            for (var j = 0; j < result.length; j++) {
+            for (let j = 0; j < result.length; j++) {
                 let curHost = result[j];
 
                 if (curHost.Type === type && curHost.Name === subdomain) {
@@ -200,9 +199,9 @@ function upsert(domainObject, subdomain, type, values, callback) {
             }
         }
 
-        let toUpsert = result.concat(toInsert);
+        const hosts = result.concat(toInsert);
 
-        setInternal(dnsConfig, zoneName, toUpsert, callback);
+        setZone(dnsConfig, zoneName, hosts, callback);
     });
 }
 
@@ -217,16 +216,16 @@ function get(domainObject, subdomain, type, callback) {
 
     subdomain = domains.getName(domainObject, subdomain, type) || '@';
 
-    getInternal(dnsConfig, zoneName, subdomain, type, function (error, result) {
+    getZone(dnsConfig, zoneName, function (error, result) {
         if (error) return callback(error);
 
         // We need to filter hosts to ones with this subdomain and type
-        let actualHosts = result.filter((host) => host.Type === type && host.Name === subdomain);
+        const actualHosts = result.filter((host) => host.Type === type && host.Name === subdomain);
 
         // We only return the value string
-        var tmp = actualHosts.map(function (record) { return record.Address; });
+        const tmp = actualHosts.map(function (record) { return record.Address; });
 
-        debug('get: %j', tmp);
+        debug(`get: subdomain: ${subdomain} type:${type} value:${JSON.stringify(tmp)}`);
 
         return callback(null, tmp);
     });
@@ -236,7 +235,7 @@ function del(domainObject, subdomain, type, values, callback) {
     assert.strictEqual(typeof domainObject, 'object');
     assert.strictEqual(typeof subdomain, 'string');
     assert.strictEqual(typeof type, 'string');
-    assert(util.isArray(values));
+    assert(Array.isArray(values));
     assert.strictEqual(typeof callback, 'function');
 
     const dnsConfig = domainObject.config;
@@ -246,29 +245,19 @@ function del(domainObject, subdomain, type, values, callback) {
 
     debug('del: %s for zone %s of type %s with values %j', subdomain, zoneName, type, values);
 
-    getInternal(dnsConfig, zoneName, subdomain, type, function (error, result) {
+    getZone(dnsConfig, zoneName, function (error, result) {
         if (error) return callback(error);
 
         if (result.length === 0) return callback();
+        const originalLength = result.length;
 
-        let removed = false;
-
-        for (var i = 0; i < values.length; i++) {
+        for (let i = 0; i < values.length; i++) {
             let curValue = values[i];
 
-            for (var j = 0; j < result.length; j++) {
-                let curHost = result[i];
-
-                if (curHost.Type === type && curHost.Name === subdomain && curHost.Address === curValue) {
-                    removed = true;
-
-                    result.splice(i, 1); // Remove element from result array
-                }
-            }
+            result = result.filter(curHost => curHost.Type !== type || curHost.Name !== subdomain || curHost.Address !== curValue);
         }
 
-        // Only set hosts if we actually removed a host
-        if (removed) return setInternal(dnsConfig, zoneName, result, callback);
+        if (result.length !== originalLength) return setZone(dnsConfig, zoneName, result, callback);
 
         callback();
     });

src/dns/netcup.js

@@ -0,0 +1,303 @@
+'use strict';
+
+exports = module.exports = {
+    removePrivateFields: removePrivateFields,
+    injectPrivateFields: injectPrivateFields,
+    upsert: upsert,
+    get: get,
+    del: del,
+    wait: wait,
+    verifyDnsConfig: verifyDnsConfig
+};
+
+var assert = require('assert'),
+    BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
+    debug = require('debug')('box:dns/netcup'),
+    dns = require('../native-dns.js'),
+    domains = require('../domains.js'),
+    superagent = require('superagent'),
+    util = require('util'),
+    waitForDns = require('./waitfordns.js');
+
+var API_ENDPOINT = 'https://ccp.netcup.net/run/webservice/servers/endpoint.php?JSON';
+
+function formatError(response) {
+    if (response.body) return util.format('Netcup DNS error [%s] %s', response.body.statuscode, response.body.longmessage);
+    else return util.format('Netcup DNS error [%s] %s', response.statusCode, response.text);
+}
+
+function removePrivateFields(domainObject) {
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
+    return domainObject;
+}
+
+function injectPrivateFields(newConfig, currentConfig) {
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+}
+
+// returns a api session id
+function login(dnsConfig, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const data = {
+        action: 'login',
+        param:{
+            apikey: dnsConfig.apiKey,
+            apipassword: dnsConfig.apiPassword,
+            customernumber: dnsConfig.customerNumber
+        }
+    };
+
+    superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+        callback(null, result.body.responsedata.apisessionid);
+    });
+}
+
+function getAllRecords(dnsConfig, apiSessionId, zoneName, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof apiSessionId, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug(`getAllRecords: getting dns records of ${zoneName}`);
+
+    const data = {
+        action: 'infoDnsRecords',
+        param:{
+            apikey: dnsConfig.apiKey,
+            apisessionid: apiSessionId,
+            customernumber: dnsConfig.customerNumber,
+            domainname: zoneName,
+        }
+    };
+
+    superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+        debug('getAllRecords:', JSON.stringify(result.body.responsedata.dnsrecords || []));
+
+        callback(null, result.body.responsedata.dnsrecords || []);
+    });
+}
+
+function upsert(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('upsert: %s for zone %s of type %s with values %j', name, zoneName, type, values);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            let records = [];
+
+            values.forEach(function (value) {
+                // remove possible quotation
+                if (value.charAt(0) === '"') value = value.slice(1);
+                if (value.charAt(value.length -1) === '"') value = value.slice(0, -1);
+
+                let priority = null;
+                if (type === 'MX') {
+                    priority = parseInt(value.split(' ')[0], 10);
+                    value = value.split(' ')[1];
+                }
+
+                let record = result.find(function (r) { return r.hostname === name && r.type === type; });
+                if (!record) record = { hostname: name, type: type, destination: value, deleterecord: false };
+                else record.destination = value;
+
+                if (priority !== null) record.priority = priority;
+
+                records.push(record);
+            });
+
+            const data = {
+                action: 'updateDnsRecords',
+                param:{
+                    apikey: dnsConfig.apiKey,
+                    apisessionid: apiSessionId,
+                    customernumber: dnsConfig.customerNumber,
+                    domainname: zoneName,
+                    dnsrecordset: {
+                        dnsrecords: records
+                    }
+                }
+            };
+
+            debug('upserting', JSON.stringify(data));
+
+            superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+                if (result.body.statuscode !== 2000) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                debug('upsert:', result.body);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function get(domainObject, location, type, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('get: %s for zone %s of type %s', name, zoneName, type);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            // We only return the value string
+            callback(null, result.filter(function (r) { return r.hostname === name && r.type === type; }).map(function (r) { return r.destination; }));
+        });
+    });
+}
+
+function del(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('del: %s for zone %s of type %s with values %j', name, zoneName, type, values);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            let records = [];
+
+            values.forEach(function (value) {
+                // remove possible quotation
+                if (value.charAt(0) === '"') value = value.slice(1);
+                if (value.charAt(value.length -1) === '"') value = value.slice(0, -1);
+
+                let record = result.find(function (r) { return r.hostname === name && r.type === type && r.destination === value; });
+                if (!record) return;
+
+                record.deleterecord = true;
+
+                records.push(record);
+            });
+
+            if (records.length === 0) return callback(null);
+
+            const data = {
+                action: 'updateDnsRecords',
+                param:{
+                    apikey: dnsConfig.apiKey,
+                    apisessionid: apiSessionId,
+                    customernumber: dnsConfig.customerNumber,
+                    domainname: zoneName,
+                    dnsrecordset: {
+                        dnsrecords: records
+                    }
+                }
+            };
+
+            superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+                if (result.body.statuscode !== 2000) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                debug('del:', result.body.responsedata);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function wait(domainObject, location, type, value, options, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof value, 'string');
+    assert(options && typeof options === 'object'); // { interval: 5000, times: 50000 }
+    assert.strictEqual(typeof callback, 'function');
+
+    const fqdn = domains.fqdn(location, domainObject);
+
+    waitForDns(fqdn, domainObject.zoneName, type, value, options, callback);
+}
+
+function verifyDnsConfig(domainObject, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName;
+
+    if (!dnsConfig.customerNumber || typeof dnsConfig.customerNumber !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'customerNumber must be a non-empty string', { field: 'customerNumber' }));
+    if (!dnsConfig.apiKey || typeof dnsConfig.apiKey !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'apiKey must be a non-empty string', { field: 'apiKey' }));
+    if (!dnsConfig.apiPassword || typeof dnsConfig.apiPassword !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'apiPassword must be a non-empty string', { field: 'apiPassword' }));
+
+    const ip = '127.0.0.1';
+
+    var credentials = {
+        customerNumber: dnsConfig.customerNumber,
+        apiKey: dnsConfig.apiKey,
+        apiPassword: dnsConfig.apiPassword,
+    };
+
+    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here
+
+    dns.resolve(zoneName, 'NS', { timeout: 5000 }, function (error, nameservers) {
+        if (error && error.code === 'ENOTFOUND') return callback(new BoxError(BoxError.BAD_FIELD, 'Unable to resolve nameservers for this domain', { field: 'nameservers' }));
+        if (error || !nameservers) return callback(new BoxError(BoxError.BAD_FIELD, error ? error.message : 'Unable to get nameservers', { field: 'nameservers' }));
+
+        if (!nameservers.every(function (n) { return n.toLowerCase().indexOf('dns.netcup.net') !== -1; })) {
+            debug('verifyDnsConfig: %j does not contains Netcup NS', nameservers);
+            return callback(new BoxError(BoxError.BAD_FIELD, 'Domain nameservers are not set to Netcup', { field: 'nameservers' }));
+        }
+
+        const location = 'cloudrontestdns';
+
+        upsert(domainObject, location, 'A', [ ip ], function (error) {
+            if (error) return callback(error);
+
+            debug('verifyDnsConfig: Test A record added');
+
+            del(domainObject, location, 'A', [ ip ], function (error) {
+                if (error) return callback(error);
+
+                debug('verifyDnsConfig: Test A record removed again');
+
+                callback(null, credentials);
+            });
+        });
+    });
+}

src/dns/route53.js

@@ -281,13 +281,14 @@ function verifyDnsConfig(domainObject, callback) {
             }
 
             const location = 'cloudrontestdns';
+            const newDomainObject = Object.assign({ }, domainObject, { config: credentials });
 
-            upsert(domainObject, location, 'A', [ ip ], function (error) {
+            upsert(newDomainObject, location, 'A', [ ip ], function (error) {
                 if (error) return callback(error);
 
                 debug('verifyDnsConfig: Test A record added');
 
-                del(domainObject, location, 'A', [ ip ], function (error) {
+                del(newDomainObject, location, 'A', [ ip ], function (error) {
                     if (error) return callback(error);
 
                     debug('verifyDnsConfig: Test A record removed again');

src/docker.js

@@ -1,38 +1,39 @@
 'use strict';
 
 exports = module.exports = {
-    testRegistryConfig: testRegistryConfig,
-    setRegistryConfig: setRegistryConfig,
-    injectPrivateFields: injectPrivateFields,
-    removePrivateFields: removePrivateFields,
+    testRegistryConfig,
+    setRegistryConfig,
+    injectPrivateFields,
+    removePrivateFields,
 
-    ping: ping,
+    ping,
 
-    info: info,
-    downloadImage: downloadImage,
-    createContainer: createContainer,
-    startContainer: startContainer,
-    restartContainer: restartContainer,
-    stopContainer: stopContainer,
+    info,
+    downloadImage,
+    createContainer,
+    startContainer,
+    restartContainer,
+    stopContainer,
     stopContainerByName: stopContainer,
-    stopContainers: stopContainers,
-    deleteContainer: deleteContainer,
-    deleteContainerByName: deleteContainer,
-    deleteImage: deleteImage,
-    deleteContainers: deleteContainers,
-    createSubcontainer: createSubcontainer,
-    getContainerIdByIp: getContainerIdByIp,
-    inspect: inspect,
+    stopContainers,
+    deleteContainer,
+    deleteImage,
+    deleteContainers,
+    createSubcontainer,
+    getContainerIdByIp,
+    inspect,
+    getContainerIp,
     inspectByName: inspect,
-    execContainer: execContainer,
-    getEvents: getEvents,
-    memoryUsage: memoryUsage,
-    createVolume: createVolume,
-    removeVolume: removeVolume,
-    clearVolume: clearVolume
+    execContainer,
+    getEvents,
+    memoryUsage,
+    createVolume,
+    removeVolume,
+    clearVolume,
+    update
 };
 
-var addons = require('./addons.js'),
+const apps = require('./apps.js'),
     async = require('async'),
     assert = require('assert'),
     BoxError = require('./boxerror.js'),
@@ -40,11 +41,16 @@ var addons = require('./addons.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:docker'),
     Docker = require('dockerode'),
+    os = require('os'),
     path = require('path'),
+    reverseProxy = require('./reverseproxy.js'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     safe = require('safetydance'),
+    system = require('./system.js'),
     util = require('util'),
+    volumes = require('./volumes.js'),
     _ = require('underscore');
 
 const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
@@ -53,11 +59,13 @@ const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
 const DOCKER_SOCKET_PATH = '/var/run/docker.sock';
 const gConnection = new Docker({ socketPath: DOCKER_SOCKET_PATH });
 
-function testRegistryConfig(auth, callback) {
-    assert.strictEqual(typeof auth, 'object');
+function testRegistryConfig(config, callback) {
+    assert.strictEqual(typeof config, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    gConnection.checkAuth(auth, function (error /*, data */) { // this returns a 500 even for auth errors
+    if (config.provider === 'noop') return callback();
+
+    gConnection.checkAuth(config, function (error /*, data */) { // this returns a 500 even for auth errors
         if (error) return callback(new BoxError(BoxError.BAD_FIELD, error, { field: 'serverAddress' }));
 
         callback();
@@ -76,14 +84,14 @@ function removePrivateFields(registryConfig) {
     return registryConfig;
 }
 
-function setRegistryConfig(auth, callback) {
-    assert.strictEqual(typeof auth, 'object');
+function setRegistryConfig(config, callback) {
+    assert.strictEqual(typeof config, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    const isLogin = !!auth.password;
+    const isLogin = !!config.password;
 
     // currently, auth info is not stashed in the db but maybe it should for restore to work?
-    const cmd = isLogin ? `docker login ${auth.serverAddress} --username ${auth.username} --password ${auth.password}` : `docker logout ${auth.serverAddress}`;
+    const cmd = isLogin ? `docker login ${config.serverAddress} --username ${config.username} --password ${config.password}` : `docker logout ${config.serverAddress}`;
 
     child_process.exec(cmd, { }, function (error /*, stdout, stderr */) {
         if (error) return callback(new BoxError(BoxError.ACCESS_DENIED, error.message));
@@ -129,12 +137,12 @@ function getRegistryConfig(image, callback) {
 }
 
 function pullImage(manifest, callback) {
-    getRegistryConfig(manifest.dockerImage, function (error, authConfig) {
+    getRegistryConfig(manifest.dockerImage, function (error, config) {
         if (error) return callback(error);
 
-        debug(`pullImage: will pull ${manifest.dockerImage}. auth: ${authConfig ? 'yes' : 'no'}`);
+        debug(`pullImage: will pull ${manifest.dockerImage}. auth: ${config ? 'yes' : 'no'}`);
 
-        gConnection.pull(manifest.dockerImage, { authconfig: authConfig }, function (error, stream) {
+        gConnection.pull(manifest.dockerImage, { authconfig: config }, function (error, stream) {
             if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND, `Unable to pull image ${manifest.dockerImage}. message: ${error.message} statusCode: ${error.statusCode}`));
             if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, `Unable to pull image ${manifest.dockerImage}. Please check the network or if the image needs authentication. statusCode: ${error.statusCode}`));
 
@@ -171,26 +179,125 @@ function downloadImage(manifest, callback) {
 
     debug('downloadImage %s', manifest.dockerImage);
 
-    var attempt = 1;
+    const image = gConnection.getImage(manifest.dockerImage);
 
-    async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
-        debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+    image.inspect(function (error, result) {
+        if (!error && result) return callback(null); // image is already present locally
 
-        pullImage(manifest, retryCallback);
-    }, callback);
+        let attempt = 1;
+
+        async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
+            debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+
+            pullImage(manifest, retryCallback);
+        }, callback);
+    });
 }
 
-function getBindsSync(app) {
+function getVolumeMounts(app, callback) {
     assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
 
-    let binds = [];
+    let mounts = [];
 
-    for (let name of Object.keys(app.binds)) {
-        const bind = app.binds[name];
-        binds.push(`${bind.hostPath}:/media/${name}:${bind.readOnly ? 'ro' : 'rw'}`);
+    if (app.mounts.length === 0) return callback(null, []);
+
+    volumes.list(function (error, result) {
+        if (error) return callback(error);
+
+        let volumesById = {};
+        result.forEach(r => volumesById[r.id] = r);
+
+        for (const mount of app.mounts) {
+            const volume = volumesById[mount.volumeId];
+
+            mounts.push({
+                Source: volume.hostPath,
+                Target: `/media/${volume.name}`,
+                Type: 'bind',
+                ReadOnly: mount.readOnly
+            });
+        }
+
+        callback(null, mounts);
+    });
+}
+
+function getAddonMounts(app, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    let mounts = [];
+
+    const addons = app.manifest.addons;
+    if (!addons) return callback(null, mounts);
+
+    async.eachSeries(Object.keys(addons), function (addon, iteratorDone) {
+        switch (addon) {
+        case 'localstorage':
+            mounts.push({
+                Target: '/app/data',
+                Source: `${app.id}-localstorage`,
+                Type: 'volume',
+                ReadOnly: false
+            });
+
+            return iteratorDone();
+        case 'tls':
+            reverseProxy.getCertificate(app.fqdn, app.domain, function (error, bundle) {
+                if (error) return iteratorDone(error);
+
+                mounts.push({
+                    Target: '/etc/certs/tls_cert.pem',
+                    Source: bundle.certFilePath,
+                    Type: 'bind',
+                    ReadOnly: true
+                });
+
+                mounts.push({
+                    Target: '/etc/certs/tls_key.pem',
+                    Source: bundle.keyFilePath,
+                    Type: 'bind',
+                    ReadOnly: true
+                });
+
+                iteratorDone();
+            });
+
+            return;
+        default:
+            iteratorDone();
+        }
+    }, function (error) {
+        callback(error, mounts);
+    });
+}
+
+function getMounts(app, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    getVolumeMounts(app, function (error, volumeMounts) {
+        if (error) return callback(error);
+
+        getAddonMounts(app, function (error, addonMounts) {
+            if (error) return callback(error);
+
+            callback(null, volumeMounts.concat(addonMounts));
+        });
+    });
+}
+
+function getLowerUpIp() { // see getifaddrs and IFF_LOWER_UP and netdevice
+    const ni = os.networkInterfaces(); // { lo: [], eth0: [] }
+    for (const iname of Object.keys(ni)) {
+        if (iname === 'lo') continue;
+        for (const address of ni[iname]) {
+            if (!address.internal && address.family === 'IPv4') return address.address;
+        }
     }
 
-    return binds;
+    return null;
 }
 
 function createSubcontainer(app, name, cmd, options, callback) {
@@ -200,12 +307,11 @@ function createSubcontainer(app, name, cmd, options, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let isAppContainer = !cmd; // non app-containers are like scheduler and exec (terminal) containers
+    let isAppContainer = !cmd; // non app-containers are like scheduler
 
     var manifest = app.manifest;
     var exposedPorts = {}, dockerPortBindings = { };
     var domain = app.fqdn;
-    const hostname = isAppContainer ? app.id : name;
 
     const envPrefix = manifest.manifestVersion <= 1 ? '' : 'CLOUDRON_';
 
@@ -219,11 +325,6 @@ function createSubcontainer(app, name, cmd, options, callback) {
         `${envPrefix}APP_DOMAIN=${domain}`
     ];
 
-    // docker portBindings requires ports to be exposed
-    exposedPorts[manifest.httpPort + '/tcp'] = {};
-
-    dockerPortBindings[manifest.httpPort + '/tcp'] = [ { HostIp: '127.0.0.1', HostPort: app.httpPort + '' } ];
-
     var portEnv = [];
     for (let portName in app.portBindings) {
         const hostPort = app.portBindings[portName];
@@ -232,104 +333,120 @@ function createSubcontainer(app, name, cmd, options, callback) {
 
         var containerPort = ports[portName].containerPort || hostPort;
 
+        // docker portBindings requires ports to be exposed
         exposedPorts[`${containerPort}/${portType}`] = {};
         portEnv.push(`${portName}=${hostPort}`);
 
-        dockerPortBindings[`${containerPort}/${portType}`] = [ { HostIp: '0.0.0.0', HostPort: hostPort + '' } ];
+        const hostIp = hostPort === 53 ? getLowerUpIp() : '0.0.0.0'; // port 53 is special because it is possibly taken by systemd-resolved
+        dockerPortBindings[`${containerPort}/${portType}`] = [ { HostIp: hostIp, HostPort: hostPort + '' } ];
     }
 
     let appEnv = [];
     Object.keys(app.env).forEach(function (name) { appEnv.push(`${name}=${app.env[name]}`); });
 
-    // first check db record, then manifest
-    var memoryLimit = app.memoryLimit || manifest.memoryLimit || 0;
-
-    if (memoryLimit === -1) { // unrestricted
-        memoryLimit = 0;
-    } else if (memoryLimit === 0 || memoryLimit < constants.DEFAULT_MEMORY_LIMIT) { // ensure we never go below minimum (in case we change the default)
-        memoryLimit = constants.DEFAULT_MEMORY_LIMIT;
-    }
+    let memoryLimit = apps.getMemoryLimit(app);
 
     // give scheduler tasks twice the memory limit since background jobs take more memory
     // if required, we can make this a manifest and runtime argument later
     if (!isAppContainer) memoryLimit *= 2;
 
-    addons.getEnvironment(app, function (error, addonEnv) {
+    services.getEnvironment(app, function (error, addonEnv) {
         if (error) return callback(error);
 
-        // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
-        // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
-        // name to look up the internal docker ip. this makes curl from within container fail
-        // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
-        // Hostname cannot be set with container NetworkMode
-        var containerOptions = {
-            name: name, // for referencing containers
-            Tty: isAppContainer,
-            Hostname: hostname,
-            Image: app.manifest.dockerImage,
-            Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
-            Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
-            ExposedPorts: isAppContainer ? exposedPorts : { },
-            Volumes: { // see also ReadonlyRootfs
-                '/tmp': {},
-                '/run': {}
-            },
-            Labels: {
-                'fqdn': app.fqdn,
-                'appId': app.id,
-                'isSubcontainer': String(!isAppContainer),
-                'isCloudronManaged': String(true)
-            },
-            HostConfig: {
-                Mounts: addons.getMountsSync(app, app.manifest.addons),
-                Binds: getBindsSync(app), // ideally, we have to use 'Mounts' but we have to create volumes then
-                LogConfig: {
-                    Type: 'syslog',
-                    Config: {
-                        'tag': app.id,
-                        'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
-                        'syslog-format': 'rfc5424'
-                    }
+        getMounts(app, function (error, mounts) {
+            if (error) return callback(error);
+
+            let containerOptions = {
+                name: name, // for referencing containers
+                Tty: isAppContainer,
+                Image: app.manifest.dockerImage,
+                Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
+                Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
+                ExposedPorts: isAppContainer ? exposedPorts : { },
+                Volumes: { // see also ReadonlyRootfs
+                    '/tmp': {},
+                    '/run': {}
                 },
-                Memory: memoryLimit / 2,
-                MemorySwap: memoryLimit, // Memory + Swap
-                PortBindings: isAppContainer ? dockerPortBindings : { },
-                PublishAllPorts: false,
-                ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
-                RestartPolicy: {
-                    'Name': isAppContainer ? 'unless-stopped' : 'no',
-                    'MaximumRetryCount': 0
+                Labels: {
+                    'fqdn': app.fqdn,
+                    'appId': app.id,
+                    'isSubcontainer': String(!isAppContainer),
+                    'isCloudronManaged': String(true)
                 },
-                CpuShares: app.cpuShares,
-                VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
-                NetworkMode: 'cloudron', // user defined bridge network
-                Dns: ['172.18.0.1'], // use internal dns
-                DnsSearch: ['.'], // use internal dns
-                SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
-                CapDrop: [ 'NET_RAW' ] // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
-            },
-            NetworkingConfig: {
-                EndpointsConfig: {
-                    cloudron: {
-                        Aliases: [ name ] // this allows sub-containers reach app containers by name
-                    }
+                HostConfig: {
+                    Mounts: mounts,
+                    LogConfig: {
+                        Type: 'syslog',
+                        Config: {
+                            'tag': app.id,
+                            'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
+                            'syslog-format': 'rfc5424'
+                        }
+                    },
+                    Memory: system.getMemoryAllocation(memoryLimit),
+                    MemorySwap: memoryLimit, // Memory + Swap
+                    PortBindings: isAppContainer ? dockerPortBindings : { },
+                    PublishAllPorts: false,
+                    ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
+                    RestartPolicy: {
+                        'Name': isAppContainer ? 'unless-stopped' : 'no',
+                        'MaximumRetryCount': 0
+                    },
+                    CpuShares: app.cpuShares,
+                    VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
+                    SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
+                    CapAdd: [],
+                    CapDrop: []
                 }
+            };
+
+            // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
+            // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
+            // name to look up the internal docker ip. this makes curl from within container fail
+            // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
+            // Hostname cannot be set with container NetworkMode. Subcontainers run is the network space of the app container
+            // This is done to prevent lots of up/down events and iptables locking
+            if (isAppContainer) {
+                containerOptions.Hostname = app.id;
+                containerOptions.HostConfig.NetworkMode = 'cloudron'; // user defined bridge network
+                containerOptions.HostConfig.Dns = ['172.18.0.1']; // use internal dns
+                containerOptions.HostConfig.DnsSearch = ['.']; // use internal dns
+
+                containerOptions.NetworkingConfig = {
+                    EndpointsConfig: {
+                        cloudron: {
+                            IPAMConfig: {
+                                IPv4Address: app.containerIp
+                            },
+                            Aliases: [ name ] // adds hostname entry with container name
+                        }
+                    }
+                };
+            } else {
+                containerOptions.HostConfig.NetworkMode = `container:${app.containerId}`; // scheduler containers must have same IP as app for various addon auth
             }
-        };
 
-        var capabilities = manifest.capabilities || [];
-        if (capabilities.includes('net_admin')) {
-            containerOptions.HostConfig.CapAdd = [
-                'NET_ADMIN', 'NET_RAW'
-            ];
-        }
+            var capabilities = manifest.capabilities || [];
 
-        containerOptions = _.extend(containerOptions, options);
+            // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
+            if (capabilities.includes('net_admin')) containerOptions.HostConfig.CapAdd.push('NET_ADMIN', 'NET_RAW');
+            if (capabilities.includes('mlock')) containerOptions.HostConfig.CapAdd.push('IPC_LOCK'); // mlock prevents swapping
+            if (!capabilities.includes('ping')) containerOptions.HostConfig.CapDrop.push('NET_RAW'); // NET_RAW is included by default by Docker
 
-        gConnection.createContainer(containerOptions, function (error, container) {
-            if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+            if (capabilities.includes('vaapi') && safe.fs.existsSync('/dev/dri')) {
+                containerOptions.HostConfig.Devices = [
+                    { PathOnHost: '/dev/dri', PathInContainer: '/dev/dri', CgroupPermissions: 'rwm' }
+                ];
+            }
 
-            callback(null, container);
+            containerOptions = _.extend(containerOptions, options);
+
+            gConnection.createContainer(containerOptions, function (error, container) {
+                if (error && error.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
+                if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+
+                callback(null, container);
+            });
         });
     });
 }
@@ -394,7 +511,7 @@ function stopContainer(containerId, callback) {
     });
 }
 
-function deleteContainer(containerId, callback) {
+function deleteContainer(containerId, callback) { // id can also be name
     assert(!containerId || typeof containerId === 'string');
     assert.strictEqual(typeof callback, 'function');
 
@@ -506,13 +623,29 @@ function inspect(containerId, callback) {
     var container = gConnection.getContainer(containerId);
 
     container.inspect(function (error, result) {
-        if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
+        if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND, `Unable to find container ${containerId}`));
         if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
 
         callback(null, result);
     });
 }
 
+function getContainerIp(containerId, callback) {
+    assert.strictEqual(typeof containerId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (constants.TEST) return callback(null, '127.0.5.5');
+
+    inspect(containerId, function (error, result) {
+        if (error) return callback(error);
+
+        const ip = safe.query(result, 'NetworkSettings.Networks.cloudron.IPAddress', null);
+        if (!ip) return callback(new BoxError(BoxError.DOCKER_ERROR, 'Error getting container IP'));
+
+        callback(null, ip);
+    });
+}
+
 function execContainer(containerId, options, callback) {
     assert.strictEqual(typeof containerId, 'string');
     assert.strictEqual(typeof options, 'object');
@@ -635,3 +768,17 @@ function info(callback) {
         callback(null, result);
     });
 }
+
+function update(name, memory, memorySwap, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof memory, 'number');
+    assert.strictEqual(typeof memorySwap, 'number');
+    assert.strictEqual(typeof callback, 'function');
+
+    const args = `update --memory ${memory} --memory-swap ${memorySwap} ${name}`.split(' ');
+    // scale back db containers, if possible. this is retried because updating memory constraints can fail
+    // with failed to write to memory.memsw.limit_in_bytes: write /sys/fs/cgroup/memory/docker/xx/memory.memsw.limit_in_bytes: device or resource busy
+    async.retry({ times: 10, interval: 60 * 1000 }, function (retryCallback) {
+        shell.spawn(`update(${name})`, '/usr/bin/docker', args, { }, retryCallback);
+    }, callback);
+}

src/dockerproxy.js

@@ -55,7 +55,7 @@ function attachDockerRequest(req, res, next) {
         // Force node to send out the headers, this is required for the /container/wait api to make the docker cli proceed
         res.write(' ');
 
-        dockerResponse.on('error', function (error) { console.error('dockerResponse error:', error); });
+        dockerResponse.on('error', function (error) { debug('dockerResponse error:', error); });
         dockerResponse.pipe(res, { end: true });
     });
 

src/domaindb.js

@@ -16,14 +16,18 @@ var assert = require('assert'),
     database = require('./database.js'),
     safe = require('safetydance');
 
-var DOMAINS_FIELDS = [ 'domain', 'zoneName', 'provider', 'configJson', 'tlsConfigJson' ].join(',');
+var DOMAINS_FIELDS = [ 'domain', 'zoneName', 'provider', 'configJson', 'tlsConfigJson', 'wellKnownJson' ].join(',');
 
 function postProcess(data) {
     data.config = safe.JSON.parse(data.configJson);
-    data.tlsConfig = safe.JSON.parse(data.tlsConfigJson);
     delete data.configJson;
+
+    data.tlsConfig = safe.JSON.parse(data.tlsConfigJson);
     delete data.tlsConfigJson;
 
+    data.wellKnown = safe.JSON.parse(data.wellKnownJson);
+    delete data.wellKnownJson;
+
     return data;
 }
 
@@ -66,7 +70,7 @@ function add(name, data, callback) {
     ];
 
     database.transaction(queries, function (error) {
-        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'Domain already exists'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(null);
@@ -86,6 +90,9 @@ function update(name, domain, callback) {
         } else if (k === 'tlsConfig') {
             fields.push('tlsConfigJson = ?');
             args.push(JSON.stringify(domain[k]));
+        } else if (k === 'wellKnown') {
+            fields.push('wellKnownJson = ?');
+            args.push(JSON.stringify(domain[k]));
         } else {
             fields.push(k + ' = ?');
             args.push(domain[k]);

src/domains.js

@@ -1,40 +1,44 @@
 'use strict';
 
 module.exports = exports = {
-    add: add,
-    get: get,
-    getAll: getAll,
-    update: update,
-    del: del,
-    clear: clear,
+    add,
+    get,
+    getAll,
+    update,
+    del,
+    clear,
 
-    fqdn: fqdn,
-    getName: getName,
+    fqdn,
+    getName,
 
-    getDnsRecords: getDnsRecords,
-    upsertDnsRecords: upsertDnsRecords,
-    removeDnsRecords: removeDnsRecords,
+    getDnsRecords,
+    upsertDnsRecords,
+    removeDnsRecords,
 
-    waitForDnsRecord: waitForDnsRecord,
+    waitForDnsRecord,
 
-    removePrivateFields: removePrivateFields,
-    removeRestrictedFields: removeRestrictedFields,
+    removePrivateFields,
+    removeRestrictedFields,
 
-    validateHostname: validateHostname,
+    validateHostname,
 
-    makeWildcard: makeWildcard,
+    makeWildcard,
 
-    parentDomain: parentDomain,
+    parentDomain,
 
-    checkDnsRecords: checkDnsRecords,
+    registerLocations,
+    unregisterLocations,
 
-    prepareDashboardDomain: prepareDashboardDomain
+    checkDnsRecords,
+    syncDnsRecords
 };
 
-var assert = require('assert'),
+const apps = require('./apps.js'),
+    assert = require('assert'),
     async = require('async'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
+    crypto = require('crypto'),
     debug = require('debug')('box:domains'),
     domaindb = require('./domaindb.js'),
     eventlog = require('./eventlog.js'),
@@ -54,7 +58,6 @@ function api(provider) {
     assert.strictEqual(typeof provider, 'string');
 
     switch (provider) {
-    case 'caas': return require('./dns/caas.js');
     case 'cloudflare': return require('./dns/cloudflare.js');
     case 'route53': return require('./dns/route53.js');
     case 'gcdns': return require('./dns/gcdns.js');
@@ -64,6 +67,7 @@ function api(provider) {
     case 'linode': return require('./dns/linode.js');
     case 'namecom': return require('./dns/namecom.js');
     case 'namecheap': return require('./dns/namecheap.js');
+    case 'netcup': return require('./dns/netcup.js');
     case 'noop': return require('./dns/noop.js');
     case 'manual': return require('./dns/manual.js');
     case 'wildcard': return require('./dns/wildcard.js');
@@ -93,14 +97,12 @@ function verifyDnsConfig(dnsConfig, domain, zoneName, provider, callback) {
         if (error && error.reason === BoxError.EXTERNAL_ERROR) return callback(new BoxError(BoxError.BAD_FIELD, `Configuration error: ${error.message}`));
         if (error) return callback(error);
 
-        result.hyphenatedSubdomains = !!dnsConfig.hyphenatedSubdomains;
-
         callback(null, result);
     });
 }
 
 function fqdn(location, domainObject) {
-    return location + (location ? (domainObject.config.hyphenatedSubdomains ? '-' : '.') : '') + domainObject.domain;
+    return location + (location ? '.' : '') + domainObject.domain;
 }
 
 // Hostname validation comes from RFC 1123 (section 2.1)
@@ -134,10 +136,6 @@ function validateHostname(location, domainObject) {
         if (/^[-.]/.test(location)) return new BoxError(BoxError.BAD_FIELD, 'Subdomain cannot start or end with hyphen or dot', { field: 'location' });
     }
 
-    if (domainObject.config.hyphenatedSubdomains) {
-        if (location.indexOf('.') !== -1) return new BoxError(BoxError.BAD_FIELD, 'Subdomain cannot contain a dot', { field: 'location' });
-    }
-
     return null;
 }
 
@@ -149,10 +147,9 @@ function validateTlsConfig(tlsConfig, dnsProvider) {
     case 'letsencrypt-prod':
     case 'letsencrypt-staging':
     case 'fallback':
-    case 'caas':
         break;
     default:
-        return new BoxError(BoxError.BAD_FIELD, 'tlsConfig.provider must be caas, fallback, letsencrypt-prod/staging', { field: 'tlsProvider' });
+        return new BoxError(BoxError.BAD_FIELD, 'tlsConfig.provider must be fallback, letsencrypt-prod/staging', { field: 'tlsProvider' });
     }
 
     if (tlsConfig.wildcard) {
@@ -163,6 +160,12 @@ function validateTlsConfig(tlsConfig, dnsProvider) {
     return null;
 }
 
+function validateWellKnown(wellKnown) {
+    assert.strictEqual(typeof wellKnown, 'object');
+
+    return null;
+}
+
 function add(domain, data, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof data.zoneName, 'string');
@@ -189,13 +192,17 @@ function add(domain, data, auditSource, callback) {
         if (error) return callback(error);
     } else {
         fallbackCertificate = reverseProxy.generateFallbackCertificateSync({ domain, config });
-        if (fallbackCertificate.error) return callback(error);
+        if (fallbackCertificate.error) return callback(fallbackCertificate.error);
     }
 
     let error = validateTlsConfig(tlsConfig, provider);
     if (error) return callback(error);
 
-    if (!dkimSelector) dkimSelector = 'cloudron-' + settings.adminDomain().replace(/\./g, '');
+    if (!dkimSelector) {
+        // create a unique suffix. this lets one add this domain can be added in another cloudron instance and not have their dkim selector conflict
+        const suffix = crypto.createHash('sha256').update(settings.adminDomain()).digest('hex').substr(0, 6);
+        dkimSelector = `cloudron-${suffix}`;
+    }
 
     verifyDnsConfig(config, domain, zoneName, provider, function (error, sanitizedConfig) {
         if (error) return callback(error);
@@ -257,7 +264,7 @@ function update(domain, data, auditSource, callback) {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let { zoneName, provider, config, fallbackCertificate, tlsConfig } = data;
+    let { zoneName, provider, config, fallbackCertificate, tlsConfig, wellKnown } = data;
 
     if (settings.isDemo() && (domain === settings.adminDomain())) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
 
@@ -278,6 +285,9 @@ function update(domain, data, auditSource, callback) {
         error = validateTlsConfig(tlsConfig, provider);
         if (error) return callback(error);
 
+        error = validateWellKnown(wellKnown, provider);
+        if (error) return callback(error);
+
         if (provider === domainObject.provider) api(provider).injectPrivateFields(config, domainObject.config);
 
         verifyDnsConfig(config, domain, zoneName, provider, function (error, sanitizedConfig) {
@@ -285,9 +295,10 @@ function update(domain, data, auditSource, callback) {
 
             let newData = {
                 config: sanitizedConfig,
-                zoneName: zoneName,
-                provider: provider,
-                tlsConfig: tlsConfig
+                zoneName,
+                provider,
+                tlsConfig,
+                wellKnown
             };
 
             domaindb.update(domain, newData, function (error) {
@@ -313,6 +324,7 @@ function del(domain, auditSource, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     if (domain === settings.adminDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove admin domain'));
+    if (domain === settings.mailDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove mail domain. Change the mail server location first'));
 
     domaindb.del(domain, function (error) {
         if (error) return callback(error);
@@ -336,24 +348,13 @@ function clear(callback) {
 }
 
 // returns the 'name' that needs to be inserted into zone
+// eslint-disable-next-line no-unused-vars
 function getName(domain, location, type) {
     const part = domain.domain.slice(0, -domain.zoneName.length - 1);
 
     if (location === '') return part;
 
-    if (!domain.config.hyphenatedSubdomains) return part ? `${location}.${part}` : location;
-
-    // hyphenatedSubdomains
-    if (type !== 'TXT') return `${location}-${part}`;
-
-    if (location.startsWith('_acme-challenge.')) {
-        return `${location}-${part}`;
-    } else if (location === '_acme-challenge') {
-        const up = part.replace(/^[^.]*\.?/, ''); // this gets the domain one level up
-        return up ? `${location}.${up}` : location;
-    } else {
-        return `${location}.${part}`;
-    }
+    return part ? `${location}.${part}` : location;
 }
 
 function getDnsRecords(location, domain, type, callback) {
@@ -453,7 +454,7 @@ function waitForDnsRecord(location, domain, type, value, options, callback) {
 
 // removes all fields that are strictly private and should never be returned by API calls
 function removePrivateFields(domain) {
-    var result = _.pick(domain, 'domain', 'zoneName', 'provider', 'config', 'tlsConfig', 'fallbackCertificate');
+    var result = _.pick(domain, 'domain', 'zoneName', 'provider', 'config', 'tlsConfig', 'fallbackCertificate', 'wellKnown');
     return api(result.provider).removePrivateFields(result);
 }
 
@@ -461,46 +462,139 @@ function removePrivateFields(domain) {
 function removeRestrictedFields(domain) {
     var result = _.pick(domain, 'domain', 'zoneName', 'provider');
 
-    // always ensure config object
-    result.config = { hyphenatedSubdomains: !!domain.config.hyphenatedSubdomains };
+    result.config = {}; // always ensure config object
 
     return result;
 }
 
-function makeWildcard(hostname) {
-    assert.strictEqual(typeof hostname, 'string');
+function makeWildcard(vhost) {
+    assert.strictEqual(typeof vhost, 'string');
 
-    let parts = hostname.split('.');
+    // if the vhost is like *.example.com, this function will do nothing
+    let parts = vhost.split('.');
     parts[0] = '*';
     return parts.join('.');
 }
 
-function prepareDashboardDomain(domain, auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
+function registerLocations(locations, options, progressCallback, callback) {
+    assert(Array.isArray(locations));
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    get(domain, function (error, domainObject) {
+    debug(`registerLocations: Will register ${JSON.stringify(locations)} with options ${JSON.stringify(options)}`);
+
+    const overwriteDns = options.overwriteDns || false;
+
+    sysinfo.getServerIp(function (error, ip) {
         if (error) return callback(error);
 
-        const adminFqdn = fqdn(constants.ADMIN_LOCATION, domainObject);
+        async.eachSeries(locations, function (location, iteratorDone) {
+            async.retry({ times: 200, interval: 5000 }, function (retryCallback) {
+                progressCallback({ message: `Registering location: ${location.subdomain ? (location.subdomain + '.') : ''}${location.domain}` });
 
-        sysinfo.getServerIp(function (error, ip) {
+                // get the current record before updating it
+                getDnsRecords(location.subdomain, location.domain, 'A', function (error, values) {
+                    if (error && error.reason === BoxError.EXTERNAL_ERROR) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location })); // try again
+                    if (error && error.reason === BoxError.ACCESS_DENIED) return retryCallback(null, new BoxError(BoxError.ACCESS_DENIED, error.message, { domain: location }));
+                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, new BoxError(BoxError.NOT_FOUND, error.message, { domain: location }));
+                    if (error) return retryCallback(null, new BoxError(BoxError.EXTERNAL_ERROR, error.message, location)); // give up for other errors
+
+                    if (values.length !== 0 && values[0] === ip) return retryCallback(null); // up-to-date
+
+                    // refuse to update any existing DNS record for custom domains that we did not create
+                    if (values.length !== 0 && !overwriteDns) return retryCallback(null, new BoxError(BoxError.ALREADY_EXISTS, 'DNS Record already exists', { domain: location }));
+
+                    upsertDnsRecords(location.subdomain, location.domain, 'A', [ ip ], function (error) {
+                        if (error && (error.reason === BoxError.BUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
+                            progressCallback({ message: `registerSubdomains: Upsert error. Will retry. ${error.message}` });
+                            return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location })); // try again
+                        }
+
+                        retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, location) : null);
+                    });
+                });
+            }, function (error, result) {
+                if (error || result) return iteratorDone(error || result);
+
+                iteratorDone(null);
+            });
+        }, callback);
+    });
+}
+
+function unregisterLocations(locations, progressCallback, callback) {
+    assert(Array.isArray(locations));
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    sysinfo.getServerIp(function (error, ip) {
+        if (error) return callback(error);
+
+        async.eachSeries(locations, function (location, iteratorDone) {
+            async.retry({ times: 30, interval: 5000 }, function (retryCallback) {
+                progressCallback({ message: `Unregistering location: ${location.subdomain ? (location.subdomain + '.') : ''}${location.domain}` });
+
+                removeDnsRecords(location.subdomain, location.domain, 'A', [ ip ], function (error) {
+                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, null);
+                    if (error && (error.reason === BoxError.SBUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
+                        progressCallback({ message: `Error unregistering location. Will retry. ${error.message}`});
+                        return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location })); // try again
+                    }
+
+                    retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location }) : null);
+                });
+            }, function (error, result) {
+                if (error || result) return iteratorDone(error || result);
+
+                iteratorDone();
+            });
+        }, callback);
+    });
+}
+
+function syncDnsRecords(options, progressCallback, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (options.domain && options.type === 'mail') return mail.setDnsRecords(options.domain, callback);
+
+    getAll(function (error, domains) {
+        if (error) return callback(error);
+
+        if (options.domain) domains = domains.filter(d => d.domain === options.domain);
+
+        const mailSubdomain = settings.mailFqdn().substr(0, settings.mailFqdn().length - settings.mailDomain().length - 1);
+
+        apps.getAll(function (error, allApps) {
             if (error) return callback(error);
 
-            async.series([
-                (done) => { progressCallback({ percent: 10, message: `Updating DNS of ${adminFqdn}` }); done(); },
-                upsertDnsRecords.bind(null, constants.ADMIN_LOCATION, domain, 'A', [ ip ]),
-                (done) => { progressCallback({ percent: 40, message: `Waiting for DNS of ${adminFqdn}` }); done(); },
-                waitForDnsRecord.bind(null, constants.ADMIN_LOCATION, domain, 'A', ip, { interval: 30000, times: 50000 }),
-                (done) => { progressCallback({ percent: 70, message: `Getting certificate of ${adminFqdn}` }); done(); },
-                reverseProxy.ensureCertificate.bind(null, fqdn(constants.ADMIN_LOCATION, domainObject), domain, auditSource)
-            ], function (error) {
-                if (error) return callback(error);
+            let progress = 1, errors = [];
 
-                callback(null);
-            });
+            // we sync by domain only to get some nice progress
+            async.eachSeries(domains, function (domain, iteratorDone) {
+                progressCallback({ percent: progress, message: `Updating DNS of ${domain.domain}`});
+                progress += Math.round(100/(1+domains.length));
+
+                let locations = [];
+                if (domain.domain === settings.adminDomain()) locations.push({ subdomain: constants.ADMIN_LOCATION, domain: settings.adminDomain() });
+                if (domain.domain === settings.mailDomain() && settings.mailFqdn() !== settings.adminFqdn()) locations.push({ subdomain: mailSubdomain, domain: settings.mailDomain() });
+
+                allApps.forEach(function (app) {
+                    const appLocations = [{ subdomain: app.location, domain: app.domain }].concat(app.alternateDomains).concat(app.aliasDomains);
+                    locations = locations.concat(appLocations.filter(al => al.domain === domain.domain));
+                });
+
+                async.series([
+                    registerLocations.bind(null, locations, { overwriteDns: true }, progressCallback),
+                    progressCallback.bind(null, { message: `Updating mail DNS of ${domain.domain}`}),
+                    mail.setDnsRecords.bind(null, domain.domain)
+                ], function (error) {
+                    if (error) errors.push({ domain: domain.domain, message: error.message });
+                    iteratorDone();
+                });
+            }, () => callback(null, { errors }));
         });
     });
 }

src/eventlog.js

@@ -1,11 +1,12 @@
 'use strict';
 
 exports = module.exports = {
-    add: add,
-    get: get,
-    getAllPaged: getAllPaged,
-    getByCreationTime: getByCreationTime,
-    cleanup: cleanup,
+    add,
+    upsert,
+    get,
+    getAllPaged,
+    getByCreationTime,
+    cleanup,
 
     // keep in sync with webadmin index.js filter
     ACTION_ACTIVATE: 'cloudron.activate',
@@ -39,6 +40,7 @@ exports = module.exports = {
     ACTION_DOMAIN_UPDATE: 'domain.update',
     ACTION_DOMAIN_REMOVE: 'domain.remove',
 
+    ACTION_MAIL_LOCATION: 'mail.location',
     ACTION_MAIL_ENABLED: 'mail.enabled',
     ACTION_MAIL_DISABLED: 'mail.disabled',
     ACTION_MAIL_MAILBOX_ADD: 'mail.box.add',
@@ -56,10 +58,15 @@ exports = module.exports = {
 
     ACTION_USER_ADD: 'user.add',
     ACTION_USER_LOGIN: 'user.login',
+    ACTION_USER_LOGOUT: 'user.logout',
     ACTION_USER_REMOVE: 'user.remove',
     ACTION_USER_UPDATE: 'user.update',
     ACTION_USER_TRANSFER: 'user.transfer',
 
+    ACTION_VOLUME_ADD: 'volume.add',
+    ACTION_VOLUME_UPDATE: 'volume.update',
+    ACTION_VOLUME_REMOVE: 'volume.remove',
+
     ACTION_DYNDNS_UPDATE: 'dyndns.update',
 
     ACTION_SUPPORT_TICKET: 'support.ticket',
@@ -85,9 +92,24 @@ function add(action, source, data, callback) {
 
     callback = callback || NOOP_CALLBACK;
 
-    // we do only daily upserts for login actions, so they don't spam the db
-    var api = action === exports.ACTION_USER_LOGIN ? eventlogdb.upsert : eventlogdb.add;
-    api(uuid.v4(), action, source, data, function (error, id) {
+    eventlogdb.add(uuid.v4(), action, source, data, function (error, id) {
+        if (error) return callback(error);
+
+        callback(null, { id: id });
+
+        notifications.onEvent(id, action, source, data, NOOP_CALLBACK);
+    });
+}
+
+function upsert(action, source, data, callback) {
+    assert.strictEqual(typeof action, 'string');
+    assert.strictEqual(typeof source, 'object');
+    assert.strictEqual(typeof data, 'object');
+    assert(!callback || typeof callback === 'function');
+
+    callback = callback || NOOP_CALLBACK;
+
+    eventlogdb.upsert(uuid.v4(), action, source, data, function (error, id) {
         if (error) return callback(error);
 
         callback(null, { id: id });

src/externalldap.js

@@ -20,7 +20,9 @@ var assert = require('assert'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:externalldap'),
+    groups = require('./groups.js'),
     ldap = require('ldapjs'),
+    once = require('once'),
     settings = require('./settings.js'),
     tasks = require('./tasks.js'),
     users = require('./users.js');
@@ -40,14 +42,14 @@ function translateUser(ldapConfig, ldapUser) {
 
     return {
         username: ldapUser[ldapConfig.usernameField],
-        email: ldapUser.mail,
+        email: ldapUser.mail || ldapUser.mailPrimaryAddress,
         displayName: ldapUser.cn // user.giveName + ' ' + user.sn
     };
 }
 
 function validUserRequirements(user) {
     if (!user.username || !user.email || !user.displayName) {
-        debug(`[LDAP user empty username/email/displayName] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+        debug(`[Invalid LDAP user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
         return false;
     } else {
         return true;
@@ -55,40 +57,95 @@ function validUserRequirements(user) {
 }
 
 // performs service bind if required
-function getClient(externalLdapConfig, callback) {
+function getClient(externalLdapConfig, doBindAuth, callback) {
     assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof doBindAuth, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
+    // ensure we only callback once since we also have to listen to client.error events
+    callback = once(callback);
+
     // basic validation to not crash
     try { ldap.parseDN(externalLdapConfig.baseDn); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid baseDn')); }
     try { ldap.parseFilter(externalLdapConfig.filter); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid filter')); }
 
+    var config = {
+        url: externalLdapConfig.url,
+        tlsOptions: {
+            rejectUnauthorized: externalLdapConfig.acceptSelfSignedCerts ? false : true
+        }
+    };
+
     var client;
     try {
-        client = ldap.createClient({ url: externalLdapConfig.url });
+        client = ldap.createClient(config);
     } catch (e) {
         if (e instanceof ldap.ProtocolError) return callback(new BoxError(BoxError.BAD_FIELD, 'url protocol is invalid'));
         return callback(new BoxError(BoxError.INTERNAL_ERROR, e));
     }
 
-    if (!externalLdapConfig.bindDn) return callback(null, client);
+    // ensure we don't just crash
+    client.on('error', function (error) {
+        callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+    });
+
+    // skip bind auth if none exist or if not wanted
+    if (!externalLdapConfig.bindDn || !doBindAuth) return callback(null, client);
 
     client.bind(externalLdapConfig.bindDn, externalLdapConfig.bindPassword, function (error) {
         if (error instanceof ldap.InvalidCredentialsError) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
         if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
 
-        callback(null, client, externalLdapConfig);
+        callback(null, client);
     });
 }
 
+function ldapGetByDN(externalLdapConfig, dn, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof dn, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    getClient(externalLdapConfig, true, function (error, client) {
+        if (error) return callback(error);
+
+        let searchOptions = {
+            paged: true,
+            scope: 'sub'    // We may have to make this configurable
+        };
+
+        debug(`Get object at ${dn}`);
+
+        // basic validation to not crash
+        try { ldap.parseDN(dn); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid DN')); }
+
+        client.search(dn, searchOptions, function (error, result) {
+            if (error instanceof ldap.NoSuchObjectError) return callback(new BoxError(BoxError.NOT_FOUND));
+            if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+
+            let ldapObjects = [];
+
+            result.on('searchEntry', entry => ldapObjects.push(entry.object));
+            result.on('error', error => callback(new BoxError(BoxError.EXTERNAL_ERROR, error)));
+
+            result.on('end', function (result) {
+                client.unbind();
+
+                if (result.status !== 0) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Server returned status ' + result.status));
+                if (ldapObjects.length === 0) return callback(new BoxError(BoxError.NOT_FOUND));
+
+                callback(null, ldapObjects[0]);
+            });
+        });
+    });
+}
 
 // TODO support search by email
-function ldapSearch(externalLdapConfig, options, callback) {
+function ldapUserSearch(externalLdapConfig, options, callback) {
     assert.strictEqual(typeof externalLdapConfig, 'object');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    getClient(externalLdapConfig, function (error, client) {
+    getClient(externalLdapConfig, true, function (error, client) {
         if (error) return callback(error);
 
         let searchOptions = {
@@ -124,6 +181,48 @@ function ldapSearch(externalLdapConfig, options, callback) {
     });
 }
 
+function ldapGroupSearch(externalLdapConfig, options, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    getClient(externalLdapConfig, true, function (error, client) {
+        if (error) return callback(error);
+
+        let searchOptions = {
+            paged: true,
+            scope: 'sub'    // We may have to make this configurable
+        };
+
+        if (externalLdapConfig.groupFilter) searchOptions.filter = ldap.parseFilter(externalLdapConfig.groupFilter);
+
+        if (options.filter) { // https://github.com/ldapjs/node-ldapjs/blob/master/docs/filters.md
+            let extraFilter = ldap.parseFilter(options.filter);
+            searchOptions.filter = new ldap.AndFilter({ filters: [ extraFilter, searchOptions.filter ] });
+        }
+
+        debug(`Listing groups at ${externalLdapConfig.groupBaseDn} with filter ${searchOptions.filter.toString()}`);
+
+        client.search(externalLdapConfig.groupBaseDn, searchOptions, function (error, result) {
+            if (error instanceof ldap.NoSuchObjectError) return callback(new BoxError(BoxError.NOT_FOUND));
+            if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+
+            let ldapGroups = [];
+
+            result.on('searchEntry', entry => ldapGroups.push(entry.object));
+            result.on('error', error => callback(new BoxError(BoxError.EXTERNAL_ERROR, error)));
+
+            result.on('end', function (result) {
+                client.unbind();
+
+                if (result.status !== 0) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Server returned status ' + result.status));
+
+                callback(null, ldapGroups);
+            });
+        });
+    });
+}
+
 function testConfig(config, callback) {
     assert.strictEqual(typeof config, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -141,7 +240,20 @@ function testConfig(config, callback) {
     if (!config.filter) return callback(new BoxError(BoxError.BAD_FIELD, 'filter must not be empty'));
     try { ldap.parseFilter(config.filter); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid filter')); }
 
-    getClient(config, function (error, client) {
+    if ('syncGroups' in config && typeof config.syncGroups !== 'boolean') return callback(new BoxError(BoxError.BAD_FIELD, 'syncGroups must be a boolean'));
+    if ('acceptSelfSignedCerts' in config && typeof config.acceptSelfSignedCerts !== 'boolean') return callback(new BoxError(BoxError.BAD_FIELD, 'acceptSelfSignedCerts must be a boolean'));
+
+    if (config.syncGroups) {
+        if (!config.groupBaseDn) return callback(new BoxError(BoxError.BAD_FIELD, 'groupBaseDn must not be empty'));
+        try { ldap.parseDN(config.groupBaseDn); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid groupBaseDn')); }
+
+        if (!config.groupFilter) return callback(new BoxError(BoxError.BAD_FIELD, 'groupFilter must not be empty'));
+        try { ldap.parseFilter(config.groupFilter); } catch (e) { return callback(new BoxError(BoxError.BAD_FIELD, 'invalid groupFilter')); }
+
+        if (!config.groupnameField || typeof config.groupnameField !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'groupFilter must not be empty'));
+    }
+
+    getClient(config, true, function (error, client) {
         if (error) return callback(error);
 
         var opts = {
@@ -167,7 +279,7 @@ function search(identifier, callback) {
         if (error) return callback(error);
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
 
-        ldapSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
+        ldapUserSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
             if (error) return callback(error);
 
             // translate ldap properties to ours
@@ -188,7 +300,7 @@ function createAndVerifyUserIfNotExist(identifier, password, callback) {
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
         if (!externalLdapConfig.autoCreate) return callback(new BoxError(BoxError.BAD_STATE, 'auto create not enabled'));
 
-        ldapSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
+        ldapUserSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${identifier}` }, function (error, ldapUsers) {
             if (error) return callback(error);
             if (ldapUsers.length === 0) return callback(new BoxError(BoxError.NOT_FOUND));
             if (ldapUsers.length > 1) return callback(new BoxError(BoxError.CONFLICT));
@@ -198,7 +310,7 @@ function createAndVerifyUserIfNotExist(identifier, password, callback) {
 
             users.create(user.username, null /* password */, user.email, user.displayName, { source: 'ldap' }, auditSource.EXTERNAL_LDAP_AUTO_CREATE, function (error, user) {
                 if (error) {
-                    console.error('Failed to auto create user', user.username, error);
+                    debug(`createAndVerifyUserIfNotExist: Failed to auto create user ${user.username}`, error);
                     return callback(new BoxError(BoxError.INTERNAL_ERROR));
                 }
 
@@ -220,17 +332,20 @@ function verifyPassword(user, password, callback) {
         if (error) return callback(error);
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
 
-        ldapSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${user.username}` }, function (error, ldapUsers) {
+        ldapUserSearch(externalLdapConfig, { filter: `${externalLdapConfig.usernameField}=${user.username}` }, function (error, ldapUsers) {
             if (error) return callback(error);
             if (ldapUsers.length === 0) return callback(new BoxError(BoxError.NOT_FOUND));
             if (ldapUsers.length > 1) return callback(new BoxError(BoxError.CONFLICT));
 
-            let client = ldap.createClient({ url: externalLdapConfig.url });
-            client.bind(ldapUsers[0].dn, password, function (error) {
-                if (error instanceof ldap.InvalidCredentialsError) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+            getClient(externalLdapConfig, false, function (error, client) {
+                if (error) return callback(error);
 
-                callback(null, translateUser(externalLdapConfig, ldapUsers[0]));
+                client.bind(ldapUsers[0].dn, password, function (error) {
+                    if (error instanceof ldap.InvalidCredentialsError) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+                    if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error));
+
+                    callback(null, translateUser(externalLdapConfig, ldapUsers[0]));
+                });
             });
         });
     });
@@ -255,6 +370,197 @@ function startSyncer(callback) {
     });
 }
 
+function syncUsers(externalLdapConfig, progressCallback, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    ldapUserSearch(externalLdapConfig, {}, function (error, ldapUsers) {
+        if (error) return callback(error);
+
+        debug(`Found ${ldapUsers.length} users`);
+
+        let percent = 10;
+        let step = 30/(ldapUsers.length+1); // ensure no divide by 0
+
+        // we ignore all errors here and just log them for now
+        async.eachSeries(ldapUsers, function (user, iteratorCallback) {
+            user = translateUser(externalLdapConfig, user);
+
+            if (!validUserRequirements(user)) return iteratorCallback();
+
+            percent += step;
+            progressCallback({ percent, message: `Syncing... ${user.username}` });
+
+            users.getByUsername(user.username, function (error, result) {
+                if (error && error.reason !== BoxError.NOT_FOUND) return iteratorCallback(error);
+
+                if (!result) {
+                    debug(`[adding user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    users.create(user.username, null /* password */, user.email, user.displayName, { source: 'ldap' }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
+                        if (error) debug('syncUsers: Failed to create user', user, error.message);
+                        iteratorCallback();
+                    });
+                } else if (result.source !== 'ldap') {
+                    debug(`[conflicting user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    iteratorCallback();
+                } else if (result.email !== user.email || result.displayName !== user.displayName) {
+                    debug(`[updating user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    users.update(result, { email: user.email, fallbackEmail: user.email, displayName: user.displayName }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
+                        if (error) debug('Failed to update user', user, error);
+
+                        iteratorCallback();
+                    });
+                } else {
+                    // user known and up-to-date
+                    debug(`[up-to-date user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
+
+                    iteratorCallback();
+                }
+            });
+        }, callback);
+    });
+}
+
+function syncGroups(externalLdapConfig, progressCallback, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!externalLdapConfig.syncGroups) {
+        debug('Group sync is disabled');
+        progressCallback({ percent: 70, message: 'Skipping group sync...' });
+        return callback(null, []);
+    }
+
+    ldapGroupSearch(externalLdapConfig, {}, function (error, ldapGroups) {
+        if (error) return callback(error);
+
+        debug(`Found ${ldapGroups.length} groups`);
+
+        let percent = 40;
+        let step = 30/(ldapGroups.length+1); // ensure no divide by 0
+
+        // we ignore all non internal errors here and just log them for now
+        async.eachSeries(ldapGroups, function (ldapGroup, iteratorCallback) {
+            var groupName = ldapGroup[externalLdapConfig.groupnameField];
+            if (!groupName) return iteratorCallback();
+            // some servers return empty array for unknown properties :-/
+            if (typeof groupName !== 'string') return iteratorCallback();
+
+            // groups are lowercase
+            groupName = groupName.toLowerCase();
+
+            percent += step;
+            progressCallback({ percent, message: `Syncing... ${groupName}` });
+
+            groups.getByName(groupName, function (error, result) {
+                if (error && error.reason !== BoxError.NOT_FOUND) return iteratorCallback(error);
+
+                if (!result) {
+                    debug(`[adding group] groupname=${groupName}`);
+
+                    groups.create(groupName, 'ldap', function (error) {
+                        if (error) debug('syncGroups: Failed to create group', groupName, error);
+                        iteratorCallback();
+                    });
+                } else {
+                    debug(`[up-to-date group] groupname=${groupName}`);
+
+                    iteratorCallback();
+                }
+            });
+        }, function (error) {
+            if (error) return callback(error);
+
+            debug('sync: ldap sync is done', error);
+
+            callback(error);
+        });
+    });
+}
+
+function syncGroupUsers(externalLdapConfig, progressCallback, callback) {
+    assert.strictEqual(typeof externalLdapConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!externalLdapConfig.syncGroups) {
+        debug('Group users sync is disabled');
+        progressCallback({ percent: 99, message: 'Skipping group users sync...' });
+        return callback(null, []);
+    }
+
+    groups.getAll(function (error, result) {
+        if (error) return callback(error);
+
+        var ldapGroups = result.filter(function (g) { return g.source === 'ldap'; });
+        debug(`Found ${ldapGroups.length} groups to sync users`);
+
+        async.eachSeries(ldapGroups, function (group, iteratorCallback) {
+            debug(`Sync users for group ${group.name}`);
+
+            ldapGroupSearch(externalLdapConfig, {}, function (error, result) {
+                if (error) return callback(error);
+                if (!result || result.length === 0) {
+                    debug(`syncGroupUsers: Unable to find group ${group.name} ignoring for now.`);
+                    return callback();
+                }
+
+                // since our group names are lowercase we cannot use potentially case matching ldap filters
+                let found = result.find(function (r) {
+                    if (!r[externalLdapConfig.groupnameField]) return false;
+                    return r[externalLdapConfig.groupnameField].toLowerCase() === group.name;
+                });
+
+                if (!found) {
+                    debug(`syncGroupUsers: Unable to find group ${group.name} ignoring for now.`);
+                    return callback();
+                }
+
+                var ldapGroupMembers = found.member || found.uniqueMember || [];
+
+                // if only one entry is in the group ldap returns a string, not an array!
+                if (typeof ldapGroupMembers === 'string') ldapGroupMembers = [ ldapGroupMembers ];
+
+                debug(`Group ${group.name} has ${ldapGroupMembers.length} members.`);
+
+                async.eachSeries(ldapGroupMembers, function (memberDn, iteratorCallback) {
+                    ldapGetByDN(externalLdapConfig, memberDn, function (error, result) {
+                        if (error) {
+                            debug(`Failed to get ${memberDn}:`, error);
+                            return iteratorCallback();
+                        }
+
+                        debug(`Found member object at ${memberDn} adding to group ${group.name}`);
+
+                        const username = result[externalLdapConfig.usernameField];
+                        if (!username) return iteratorCallback();
+
+                        users.getByUsername(username, function (error, result) {
+                            if (error) {
+                                debug(`syncGroupUsers: Failed to get user by username ${username}`, error);
+                                return iteratorCallback();
+                            }
+
+                            groups.addMember(group.id, result.id, function (error) {
+                                if (error && error.reason !== BoxError.ALREADY_EXISTS) debug('syncGroupUsers: Failed to add member', error);
+                                iteratorCallback();
+                            });
+                        });
+                    });
+                }, function (error) {
+                    if (error) debug('syncGroupUsers: ', error);
+                    iteratorCallback();
+                });
+            });
+        }, callback);
+    });
+}
+
 function sync(progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
@@ -265,58 +571,18 @@ function sync(progressCallback, callback) {
         if (error) return callback(error);
         if (externalLdapConfig.provider === 'noop') return callback(new BoxError(BoxError.BAD_STATE, 'not enabled'));
 
-        ldapSearch(externalLdapConfig, {}, function (error, ldapUsers) {
+        async.series([
+            syncUsers.bind(null, externalLdapConfig, progressCallback),
+            syncGroups.bind(null, externalLdapConfig, progressCallback),
+            syncGroupUsers.bind(null, externalLdapConfig, progressCallback)
+        ], function (error) {
             if (error) return callback(error);
 
-            debug(`Found ${ldapUsers.length} users`);
-            let percent = 10;
-            let step = 90/(ldapUsers.length+1); // ensure no divide by 0
+            progressCallback({ percent: 100, message: 'Done' });
 
-            // we ignore all errors here and just log them for now
-            async.eachSeries(ldapUsers, function (user, iteratorCallback) {
-                user = translateUser(externalLdapConfig, user);
+            debug('sync: ldap sync is done', error);
 
-                if (!validUserRequirements(user)) return iteratorCallback();
-
-                percent += step;
-                progressCallback({ percent, message: `Syncing... ${user.username}` });
-
-                users.getByUsername(user.username, function (error, result) {
-                    if (error && error.reason !== BoxError.NOT_FOUND) {
-                        debug(`Could not find user with username ${user.username}: ${error.message}`);
-                        return iteratorCallback();
-                    }
-
-                    if (error) {
-                        debug(`[adding user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        users.create(user.username, null /* password */, user.email, user.displayName, { source: 'ldap' }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
-                            if (error) console.error('Failed to create user', user, error);
-                            iteratorCallback();
-                        });
-                    } else if (result.source !== 'ldap') {
-                        debug(`[conflicting user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        iteratorCallback();
-                    } else if (result.email !== user.email || result.displayName !== user.displayName) {
-                        debug(`[updating user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        users.update(result, { email: user.email, fallbackEmail: user.email, displayName: user.displayName }, auditSource.EXTERNAL_LDAP_TASK, function (error) {
-                            if (error) debug('Failed to update user', user, error);
-
-                            iteratorCallback();
-                        });
-                    } else {
-                        // user known and up-to-date
-                        debug(`[up-to-date user] username=${user.username} email=${user.email} displayName=${user.displayName}`);
-
-                        iteratorCallback();
-                    }
-                });
-            }, function (error) {
-                debug('sync: ldap sync is done', error);
-                callback(error);
-            });
+            callback(error);
         });
     });
 }

src/graphs.js

@@ -1,41 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    startGraphite: startGraphite
-};
-
-var assert = require('assert'),
-    infra = require('./infra_version.js'),
-    paths = require('./paths.js'),
-    shell = require('./shell.js');
-
-function startGraphite(existingInfra, callback) {
-    assert.strictEqual(typeof existingInfra, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    const tag = infra.images.graphite.tag;
-    const dataDir = paths.PLATFORM_DATA_DIR;
-
-    if (existingInfra.version === infra.version && infra.images.graphite.tag === existingInfra.images.graphite.tag) return callback();
-
-    const cmd = `docker run --restart=always -d --name="graphite" \
-                --hostname graphite \
-                --net cloudron \
-                --net-alias graphite \
-                --log-driver syslog \
-                --log-opt syslog-address=udp://127.0.0.1:2514 \
-                --log-opt syslog-format=rfc5424 \
-                --log-opt tag=graphite \
-                -m 75m \
-                --memory-swap 150m \
-                --dns 172.18.0.1 \
-                --dns-search=. \
-                -p 127.0.0.1:2003:2003 \
-                -p 127.0.0.1:2004:2004 \
-                -p 127.0.0.1:8417:8000 \
-                -v "${dataDir}/graphite:/var/lib/graphite" \
-                --label isCloudronManaged=true \
-                --read-only -v /tmp -v /run "${tag}"`;
-
-    shell.exec('startGraphite', cmd, callback);
-}

src/groupdb.js

@@ -2,6 +2,7 @@
 
 exports = module.exports = {
     get: get,
+    getByName: getByName,
     getWithMembers: getWithMembers,
     getAll: getAll,
     getAllWithMembers: getAllWithMembers,
@@ -19,8 +20,6 @@ exports = module.exports = {
     getMembership: getMembership,
     setMembership: setMembership,
 
-    getGroups: getGroups,
-
     _clear: clear
 };
 
@@ -28,7 +27,7 @@ var assert = require('assert'),
     BoxError = require('./boxerror.js'),
     database = require('./database.js');
 
-var GROUPS_FIELDS = [ 'id', 'name' ].join(',');
+var GROUPS_FIELDS = [ 'id', 'name', 'source' ].join(',');
 
 function get(groupId, callback) {
     assert.strictEqual(typeof groupId, 'string');
@@ -42,6 +41,18 @@ function get(groupId, callback) {
     });
 }
 
+function getByName(name, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups WHERE name = ?', [ name ], function (error, result) {
+        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Group not found'));
+
+        callback(null, result[0]);
+    });
+}
+
 function getWithMembers(groupId, callback) {
     assert.strictEqual(typeof groupId, 'string');
     assert.strictEqual(typeof callback, 'function');
@@ -63,7 +74,7 @@ function getWithMembers(groupId, callback) {
 function getAll(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups', function (error, results) {
+    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups ORDER BY name', function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(null, results);
@@ -73,9 +84,8 @@ function getAll(callback) {
 function getAllWithMembers(callback) {
     database.query('SELECT ' + GROUPS_FIELDS + ',GROUP_CONCAT(groupMembers.userId) AS userIds ' +
                     ' FROM userGroups LEFT OUTER JOIN groupMembers ON userGroups.id = groupMembers.groupId ' +
-                    ' GROUP BY userGroups.id', function (error, results) {
+                    ' GROUP BY userGroups.id ORDER BY name', function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Group not found'));
 
         results.forEach(function (result) { result.userIds = result.userIds ? result.userIds.split(',') : [ ]; });
 
@@ -83,12 +93,13 @@ function getAllWithMembers(callback) {
     });
 }
 
-function add(id, name, callback) {
+function add(id, name, source, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof source, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO userGroups (id, name) VALUES (?, ?)', [ id, name ], function (error, result) {
+    database.query('INSERT INTO userGroups (id, name, source) VALUES (?, ?, ?)', [ id, name, source ], function (error, result) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
         if (error || result.affectedRows !== 1) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -185,6 +196,7 @@ function setMembers(groupId, userIds, callback) {
 
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, 'Group not found'));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.CONFLICT, 'Duplicate member in list'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(error);
@@ -216,6 +228,7 @@ function setMembership(userId, groupIds, callback) {
 
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, error.message));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.CONFLICT, 'Already member'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(null);
@@ -260,15 +273,3 @@ function isMember(groupId, userId, callback) {
         callback(null, result.length !== 0);
     });
 }
-
-function getGroups(userId, callback) {
-    assert.strictEqual(typeof userId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query('SELECT ' + GROUPS_FIELDS + ' ' +
-        ' FROM userGroups INNER JOIN groupMembers ON userGroups.id = groupMembers.groupId AND groupMembers.userId = ?', [ userId ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-        callback(null, results);
-    });
-}

src/groups.js

@@ -1,26 +1,25 @@
 'use strict';
 
 exports = module.exports = {
-    create: create,
-    remove: remove,
-    get: get,
-    update: update,
-    getWithMembers: getWithMembers,
-    getAll: getAll,
-    getAllWithMembers: getAllWithMembers,
+    create,
+    remove,
+    get,
+    getByName,
+    update,
+    getWithMembers,
+    getAll,
+    getAllWithMembers,
 
-    getMembers: getMembers,
-    addMember: addMember,
-    setMembers: setMembers,
-    removeMember: removeMember,
-    isMember: isMember,
+    getMembers,
+    addMember,
+    setMembers,
+    removeMember,
+    isMember,
 
-    getGroups: getGroups,
+    setMembership,
+    getMembership,
 
-    setMembership: setMembership,
-    getMembership: getMembership,
-
-    count: count
+    count
 };
 
 var assert = require('assert'),
@@ -45,8 +44,17 @@ function validateGroupname(name) {
     return null;
 }
 
-function create(name, callback) {
+function validateGroupSource(source) {
+    assert.strictEqual(typeof source, 'string');
+
+    if (source !== '' && source !== 'ldap') return new BoxError(BoxError.BAD_FIELD, 'source must be "" or "ldap"', { field: source });
+
+    return null;
+}
+
+function create(name, source, callback) {
     assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof source, 'string');
     assert.strictEqual(typeof callback, 'function');
 
     // we store names in lowercase
@@ -55,8 +63,11 @@ function create(name, callback) {
     var error = validateGroupname(name);
     if (error) return callback(error);
 
+    error = validateGroupSource(source);
+    if (error) return callback(error);
+
     var id = 'gid-' + uuid.v4();
-    groupdb.add(id, name, function (error) {
+    groupdb.add(id, name, source, function (error) {
         if (error) return callback(error);
 
         callback(null, { id: id, name: name });
@@ -85,6 +96,17 @@ function get(id, callback) {
     });
 }
 
+function getByName(name, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    groupdb.getByName(name, function (error, result) {
+        if (error) return callback(error);
+
+        return callback(null, result);
+    });
+}
+
 function getWithMembers(id, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');
@@ -217,17 +239,6 @@ function update(groupId, data, callback) {
     });
 }
 
-function getGroups(userId, callback) {
-    assert.strictEqual(typeof userId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    groupdb.getGroups(userId, function (error, results) {
-        if (error) return callback(error);
-
-        callback(null, results);
-    });
-}
-
 function count(callback) {
     assert.strictEqual(typeof callback, 'function');
 

src/infra_version.js

@@ -6,22 +6,22 @@
 
 exports = module.exports = {
     // a version change recreates all containers with latest docker config
-    'version': '48.17.0',
+    'version': '48.18.0',
 
     'baseImages': [
-        { repo: 'cloudron/base', tag: 'cloudron/base:2.0.0@sha256:f9fea80513aa7c92fe2e7bf3978b54c8ac5222f47a9a32a7f8833edf0eb5a4f4' }
+        { repo: 'cloudron/base', tag: 'cloudron/base:3.0.0@sha256:455c70428723e3a823198c57472785437eb6eab082e79b3ff04ea584faf46e92' }
     ],
 
     // a major version bump in the db containers will trigger the restore logic that uses the db dumps
     // docker inspect --format='{{index .RepoDigests 0}}' $IMAGE to get the sha256
     'images': {
-        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.1.0@sha256:e1dd22aa6eef5beb7339834b200a8bb787ffc2264ce11139857a054108fefb4f' },
-        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.3.0@sha256:0cc39004b80eb5ee570b9fd4f38859410a49692e93e0f0d8b104b0b524d7030a' },
-        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:2.1.1@sha256:6ea1bcf9b655d628230acda01d46cf21883b112111d89583c94138646895c14c' },
-        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:2.2.0@sha256:205486ff0f6bf6854610572df401cf3651bc62baf28fd26e9c5632497f10c2cb' },
-        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.2.0@sha256:cfdcc1a54cf29818cac99eacedc2ecf04e44995be3d06beea11dcaa09d90ed8d' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.9.3@sha256:2c062e8883cadae4b4c22553f9db141fc441d6c627aa85e2a5ad71e3fcbf2b42' },
-        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.3.0@sha256:b7bc1ca4f4d0603a01369a689129aa273a938ce195fe43d00d42f4f2d5212f50' },
-        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:1.1.0@sha256:0c1fe4dd6121900624dcb383251ecb0084c3810e095064933de671409d8d6d7b' }
+        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.3.1@sha256:759cafab7625ff538418a1f2ed5558b1d5bff08c576bba577d865d6d02b49091' },
+        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:3.0.4@sha256:4d688c746f27b195d98f35a7d24ec01f3f754e0ca61e9de0b0bc9793553880f1' },
+        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:4.0.2@sha256:424081fd38ebd35f3606c64f8f99138570e5f4d5066f12cfb4142447d249d3e7' },
+        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:4.0.1@sha256:ad20a9a5dcb2ab132374a7c8d44b89af0ec37651cf889e570f7625b02ee85fdf' },
+        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:3.0.2@sha256:caaa1f7f4055ae8990d8ec65bd100567496df7e4ed5eb427867f3717a8dcbf92' },
+        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:3.2.3@sha256:fdc4aa6d2c85aeafe65eaa4243aada0cc2e57b94f6eaee02c9b1a8fb89b01dd7' },
+        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:3.0.1@sha256:bed9f6b5d06fe2c5289e895e806cfa5b74ad62993d705be55d4554a67d128029' },
+        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:3.2.0@sha256:61e8247ded1e07cf882ca478dab180960357c614472e80b938f1f690a46788c2' }
     }
 };

src/iputils.js

@@ -0,0 +1,35 @@
+'use strict';
+
+exports = module.exports = {
+    ipFromInt,
+    intFromIp
+};
+
+const assert = require('assert');
+
+function intFromIp(address) {
+    assert.strictEqual(typeof address, 'string');
+
+    const parts = address.split('.');
+
+    if (parts.length !== 4) return null;
+
+    return (parseInt(parts[0], 10) << (8*3)) & 0xFF000000 |
+           (parseInt(parts[1], 10) << (8*2)) & 0x00FF0000 |
+           (parseInt(parts[2], 10) << (8*1)) & 0x0000FF00 |
+           (parseInt(parts[3], 10) << (8*0)) & 0x000000FF;
+}
+
+function ipFromInt(input) {
+    assert.strictEqual(typeof input, 'number');
+
+    let output = [];
+
+    for (let i = 3; i >= 0; --i) {
+        const octet = (input >> (i*8)) & 0x000000FF;
+        output.push(octet);
+    }
+
+    return output.join('.');
+}
+

src/janitor.js

@@ -16,21 +16,15 @@ const NOOP_CALLBACK = function () { };
 
 const gConnection = new Docker({ socketPath: '/var/run/docker.sock' });
 
-function ignoreError(func) {
-    return function (callback) {
-        func(function (error) {
-            if (error) console.error('Ignored error:', error);
+function cleanupTokens(callback) {
+    assert(!callback || typeof callback === 'function'); // callback is null when called from cronjob
 
-            callback();
-        });
-    };
-}
+    callback = callback || NOOP_CALLBACK;
 
-function cleanupExpiredTokens(callback) {
-    assert.strictEqual(typeof callback, 'function');
+    debug('Cleaning up expired tokens');
 
     tokendb.delExpired(function (error, result) {
-        if (error) return callback(error);
+        if (error) return debug('cleanupTokens: error removing expired tokens', error);
 
         debug('Cleaned up %s expired tokens.', result);
 
@@ -38,16 +32,6 @@ function cleanupExpiredTokens(callback) {
     });
 }
 
-function cleanupTokens(callback) {
-    assert(!callback || typeof callback === 'function'); // callback is null when called from cronjob
-
-    debug('Cleaning up expired tokens');
-
-    async.series([
-        ignoreError(cleanupExpiredTokens)
-    ], callback);
-}
-
 function cleanupTmpVolume(containerInfo, callback) {
     assert.strictEqual(typeof containerInfo, 'object');
     assert.strictEqual(typeof callback, 'function');

src/ldap.js

@@ -1,11 +1,11 @@
 'use strict';
 
 exports = module.exports = {
-    start: start,
-    stop: stop
+    start,
+    stop
 };
 
-var assert = require('assert'),
+const assert = require('assert'),
     appdb = require('./appdb.js'),
     apps = require('./apps.js'),
     async = require('async'),
@@ -13,11 +13,13 @@ var assert = require('assert'),
     constants = require('./constants.js'),
     debug = require('debug')('box:ldap'),
     eventlog = require('./eventlog.js'),
+    groups = require('./groups.js'),
     ldap = require('ldapjs'),
     mail = require('./mail.js'),
     mailboxdb = require('./mailboxdb.js'),
     path = require('path'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     users = require('./users.js');
 
 var gServer = null;
@@ -132,8 +134,8 @@ function userSearch(req, res, next) {
 
             var dn = ldap.parseDN('cn=' + user.id + ',ou=users,dc=cloudron');
 
-            var groups = [ GROUP_USERS_DN ];
-            if (users.compareRoles(user.role, users.ROLE_ADMIN) >= 0) groups.push(GROUP_ADMINS_DN);
+            var memberof = [ GROUP_USERS_DN ];
+            if (users.compareRoles(user.role, users.ROLE_ADMIN) >= 0) memberof.push(GROUP_ADMINS_DN);
 
             var displayName = user.displayName || user.username || ''; // displayName can be empty and username can be null
             var nameParts = displayName.split(' ');
@@ -154,7 +156,7 @@ function userSearch(req, res, next) {
                     givenName: firstName,
                     username: user.username,
                     samaccountname: user.username,      // to support ActiveDirectory clients
-                    memberof: groups
+                    memberof: memberof
                 }
             };
 
@@ -286,14 +288,14 @@ function mailboxSearch(req, res, next) {
     } else if (req.dn.rdns[0].attrs.domain) { // legacy ldap mailbox search for old sogo
         var domain = req.dn.rdns[0].attrs.domain.value.toLowerCase();
 
-        mailboxdb.listMailboxes(domain, 1, 1000, function (error, result) {
+        mailboxdb.listMailboxes(domain, 1, 1000, function (error, mailboxes) {
 
             if (error) return next(new ldap.OperationsError(error.toString()));
 
             var results = [];
 
             // send mailbox objects
-            result.forEach(function (mailbox) {
+            mailboxes.forEach(function (mailbox) {
                 var dn = ldap.parseDN(`cn=${mailbox.name}@${domain},domain=${domain},ou=mailboxes,dc=cloudron`);
 
                 var obj = {
@@ -327,7 +329,9 @@ function mailboxSearch(req, res, next) {
             async.eachSeries(mailboxes, function (mailbox, callback) {
                 var dn = ldap.parseDN(`cn=${mailbox.name}@${mailbox.domain},ou=mailboxes,dc=cloudron`);
 
-                users.get(mailbox.ownerId, function (error, userObject) {
+                let getFunc = mailbox.ownerType === mail.OWNERTYPE_USER ? users.get : groups.get;
+
+                getFunc(mailbox.ownerId, function (error, ownerObject) {
                     if (error) return callback(); // skip mailboxes with unknown owner
 
                     var obj = {
@@ -335,30 +339,26 @@ function mailboxSearch(req, res, next) {
                         attributes: {
                             objectclass: ['mailbox'],
                             objectcategory: 'mailbox',
-                            displayname: userObject.displayName,
+                            displayname: mailbox.ownerType === mail.OWNERTYPE_USER ? ownerObject.displayName : ownerObject.name,
                             cn: `${mailbox.name}@${mailbox.domain}`,
                             uid: `${mailbox.name}@${mailbox.domain}`,
                             mail: `${mailbox.name}@${mailbox.domain}`
                         }
                     };
 
-                    mailboxdb.getAliasesForName(mailbox.name, mailbox.domain, function (error, aliases) {
-                        if (error) return callback(error);
-
-                        aliases.forEach(function (a, idx) {
-                            obj.attributes['mail' + idx] = `${a}@${mailbox.domain}`;
-                        });
-
-                        // ensure all filter values are also lowercase
-                        var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
-                        if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));
-
-                        if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
-                            results.push(obj);
-                        }
-
-                        callback();
+                    mailbox.aliases.forEach(function (a, idx) {
+                        obj.attributes['mail' + idx] = `${a.name}@${a.domain}`;
                     });
+
+                    // ensure all filter values are also lowercase
+                    var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
+                    if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));
+
+                    if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
+                        results.push(obj);
+                    }
+
+                    callback();
                 });
             }, function (error) {
                 if (error) return next(new ldap.OperationsError(error.toString()));
@@ -482,18 +482,42 @@ function authorizeUserForApp(req, res, next) {
     assert.strictEqual(typeof req.user, 'object');
     assert.strictEqual(typeof req.app, 'object');
 
-    apps.hasAccessTo(req.app, req.user, function (error, result) {
+    apps.hasAccessTo(req.app, req.user, function (error, hasAccess) {
         if (error) return next(new ldap.OperationsError(error.toString()));
 
         // we return no such object, to avoid leakage of a users existence
-        if (!result) return next(new ldap.NoSuchObjectError(req.dn.toString()));
+        if (!hasAccess) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-        eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', appId: req.app.id }, { userId: req.user.id, user: users.removePrivateFields(req.user) });
+        eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', appId: req.app.id }, { userId: req.user.id, user: users.removePrivateFields(req.user) });
 
         res.end();
     });
 }
 
+function verifyMailboxPassword(mailbox, password, callback) {
+    assert.strictEqual(typeof mailbox, 'object');
+    assert.strictEqual(typeof password, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (mailbox.ownerType === mail.OWNERTYPE_USER) return users.verify(mailbox.ownerId, password, users.AP_MAIL /* identifier */, callback);
+
+    groups.getMembers(mailbox.ownerId, function (error, userIds) {
+        if (error) return callback(error);
+
+        let verifiedUser = null;
+        async.someSeries(userIds, function iterator(userId, iteratorDone) {
+            users.verify(userId, password, users.AP_MAIL /* identifier */, function (error, result) {
+                if (error) return iteratorDone(null, false);
+                verifiedUser = result;
+                iteratorDone(null, true);
+            });
+        }, function (error, result) {
+            if (!result) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+            callback(null, verifiedUser);
+        });
+    });
+}
+
 function authenticateUserMailbox(req, res, next) {
     debug('user mailbox auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
 
@@ -513,12 +537,12 @@ function authenticateUserMailbox(req, res, next) {
             if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
             if (error) return next(new ldap.OperationsError(error.message));
 
-            users.verify(mailbox.ownerId, req.credentials || '', users.AP_MAIL, function (error, result) {
+            verifyMailboxPassword(mailbox, req.credentials || '', function (error, result) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                 if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
                 if (error) return next(new ldap.OperationsError(error.message));
 
-                eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
+                eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
                 res.end();
             });
         });
@@ -547,6 +571,16 @@ function authenticateSftp(req, res, next) {
     });
 }
 
+function loadSftpConfig(req, res, next) {
+    services.getServiceConfig('sftp', function (error, serviceConfig) {
+        if (error) return next(new ldap.OperationsError(error.toString()));
+
+        req.requireAdmin = serviceConfig.requireAdmin;
+
+        next();
+    });
+}
+
 function userSearchSftp(req, res, next) {
     debug('sftp user search: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);
 
@@ -570,6 +604,8 @@ function userSearchSftp(req, res, next) {
         users.getByUsername(username, function (error, user) {
             if (error) return next(new ldap.OperationsError(error.toString()));
 
+            if (req.requireAdmin && users.compareRoles(user.role, users.ROLE_ADMIN) < 0) return next(new ldap.InsufficientAccessRightsError('Insufficient previleges'));
+
             apps.hasAccessTo(app, user, function (error, hasAccess) {
                 if (error) return next(new ldap.OperationsError(error.toString()));
                 if (!hasAccess) return next(new ldap.InsufficientAccessRightsError('Not authorized'));
@@ -577,7 +613,7 @@ function userSearchSftp(req, res, next) {
                 var obj = {
                     dn: ldap.parseDN(`cn=${username}@${appFqdn},ou=sftp,dc=cloudron`).toString(),
                     attributes: {
-                        homeDirectory: path.join('/app/data', app.id, 'data'),
+                        homeDirectory: path.join('/app/data', app.id),
                         objectclass: ['user'],
                         objectcategory: 'person',
                         cn: user.id,
@@ -593,16 +629,37 @@ function userSearchSftp(req, res, next) {
     });
 }
 
+function verifyAppMailboxPassword(addonId, username, password, callback) {
+    assert.strictEqual(typeof addonId, 'string');
+    assert.strictEqual(typeof username, 'string');
+    assert.strictEqual(typeof password, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const pattern = addonId === 'sendmail' ? 'MAIL_SMTP' : 'MAIL_IMAP';
+    appdb.getAppIdByAddonConfigValue(addonId, `%${pattern}_PASSWORD`, password, function (error, appId) { // search by password because this is unique for each app
+        if (error) return callback(error);
+
+        appdb.getAddonConfig(appId, addonId, function (error, result) {
+            if (error) return callback(error);
+
+            if (!result.some(r => r.name.endsWith(`${pattern}_USERNAME`) && r.value === username)) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+
+            callback(null);
+        });
+    });
+}
+
 function authenticateMailAddon(req, res, next) {
     debug('mail addon auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
 
     if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-    var email = req.dn.rdns[0].attrs.cn.value.toLowerCase();
-    var parts = email.split('@');
+    const email = req.dn.rdns[0].attrs.cn.value.toLowerCase();
+    const parts = email.split('@');
     if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
     const addonId = req.dn.rdns[1].attrs.ou.value.toLowerCase(); // 'sendmail' or 'recvmail'
+    if (addonId !== 'sendmail' && addonId !== 'recvmail') return next(new ldap.OperationsError('Invalid DN'));
 
     mail.getDomain(parts[1], function (error, domain) {
         if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
@@ -610,26 +667,22 @@ function authenticateMailAddon(req, res, next) {
 
         if (addonId === 'recvmail' && !domain.enabled) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-        let namePattern; // manifest v2 has a CLOUDRON_ prefix for names
-        if (addonId === 'sendmail') namePattern = '%MAIL_SMTP_PASSWORD';
-        else if (addonId === 'recvmail') namePattern = '%MAIL_IMAP_PASSWORD';
-        else return next(new ldap.OperationsError('Invalid DN'));
+        verifyAppMailboxPassword(addonId, email, req.credentials || '', function (error) {
+            if (!error) return res.end(); // validated as app
 
-        // note: with sendmail addon, apps can send mail without a mailbox (unlike users)
-        appdb.getAppIdByAddonConfigValue(addonId, namePattern, req.credentials || '', function (error, appId) {
+            if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
             if (error && error.reason !== BoxError.NOT_FOUND) return next(new ldap.OperationsError(error.message));
-            if (appId) return res.end();
 
             mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                 if (error) return next(new ldap.OperationsError(error.message));
 
-                users.verify(mailbox.ownerId, req.credentials || '', users.AP_MAIL, function (error, result) {
+                verifyMailboxPassword(mailbox, req.credentials || '', function (error, result) {
                     if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                     if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
                     if (error) return next(new ldap.OperationsError(error.message));
 
-                    eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
+                    eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
                     res.end();
                 });
             });
@@ -645,14 +698,14 @@ function start(callback) {
         debug: NOOP,
         info: debug,
         warn: debug,
-        error: console.error,
-        fatal: console.error
+        error: debug,
+        fatal: debug
     };
 
     gServer = ldap.createServer({ log: logger });
 
     gServer.on('error', function (error) {
-        console.error('LDAP:', error);
+        debug('start: server error ', error);
     });
 
     gServer.search('ou=users,dc=cloudron', authenticateApp, userSearch);
@@ -660,16 +713,16 @@ function start(callback) {
     gServer.bind('ou=users,dc=cloudron', authenticateApp, authenticateUser, authorizeUserForApp);
 
     // http://www.ietf.org/proceedings/43/I-D/draft-srivastava-ldap-mail-00.txt
-    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch); // haraka, dovecot
+    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch); // haraka (address translation), dovecot (LMTP), sogo (mailbox search)
     gServer.bind('ou=mailboxes,dc=cloudron', authenticateUserMailbox); // apps like sogo can use domain=${domain} to authenticate a mailbox
     gServer.search('ou=mailaliases,dc=cloudron', mailAliasSearch); // haraka
     gServer.search('ou=mailinglists,dc=cloudron', mailingListSearch); // haraka
 
-    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailAddon); // dovecot
-    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka
+    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailAddon); // dovecot (IMAP auth)
+    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka (MSA auth)
 
     gServer.bind('ou=sftp,dc=cloudron', authenticateSftp);    // sftp
-    gServer.search('ou=sftp,dc=cloudron', userSearchSftp);
+    gServer.search('ou=sftp,dc=cloudron', loadSftpConfig, userSearchSftp);
 
     gServer.compare('cn=users,ou=groups,dc=cloudron', authenticateApp, groupUsersCompare);
     gServer.compare('cn=admins,ou=groups,dc=cloudron', authenticateApp, groupAdminsCompare);
@@ -687,6 +740,12 @@ function start(callback) {
         res.end();
     });
 
+    // just log that an attempt was made to unknown route, this helps a lot during app packaging
+    gServer.use(function(req, res, next) {
+        debug('not handled: dn %s, scope %s, filter %s (from %s)', req.dn ? req.dn.toString() : '-', req.scope, req.filter ? req.filter.toString() : '-', req.connection.ldap.id);
+        return next();
+    });
+
     gServer.listen(constants.LDAP_PORT, '0.0.0.0', callback);
 }
 

src/mail.js

@@ -1,59 +1,70 @@
 'use strict';
 
 exports = module.exports = {
-    getStatus: getStatus,
-    checkConfiguration: checkConfiguration,
+    getStatus,
+    checkConfiguration,
 
-    getDomains: getDomains,
+    getLocation,
+    setLocation, // triggers the change task
+    changeLocation, // does the actual changing
 
-    getDomain: getDomain,
-    clearDomains: clearDomains,
+    getDomains,
 
-    onDomainAdded: onDomainAdded,
-    onDomainRemoved: onDomainRemoved,
+    getDomain,
+    clearDomains,
 
-    removePrivateFields: removePrivateFields,
+    onDomainAdded,
+    onDomainRemoved,
 
-    setDnsRecords: setDnsRecords,
-    onMailFqdnChanged: onMailFqdnChanged,
+    removePrivateFields,
 
-    validateName: validateName,
+    setDnsRecords,
 
-    setMailFromValidation: setMailFromValidation,
-    setCatchAllAddress: setCatchAllAddress,
-    setMailRelay: setMailRelay,
-    setMailEnabled: setMailEnabled,
+    validateName,
+
+    setMailFromValidation,
+    setCatchAllAddress,
+    setMailRelay,
+    setMailEnabled,
+    setBanner,
 
     startMail: restartMail,
-    restartMail: restartMail,
-    handleCertChanged: handleCertChanged,
-    getMailAuth: getMailAuth,
+    restartMail,
+    handleCertChanged,
+    getMailAuth,
 
-    sendTestMail: sendTestMail,
+    sendTestMail,
 
-    listMailboxes: listMailboxes,
-    removeMailboxes: removeMailboxes,
-    getMailbox: getMailbox,
-    addMailbox: addMailbox,
-    updateMailboxOwner: updateMailboxOwner,
-    removeMailbox: removeMailbox,
+    getMailboxCount,
+    listMailboxes,
+    getMailbox,
+    addMailbox,
+    updateMailboxOwner,
+    removeMailbox,
 
-    getAliases: getAliases,
-    setAliases: setAliases,
+    getAliases,
+    setAliases,
 
-    getLists: getLists,
-    getList: getList,
-    addList: addList,
-    updateList: updateList,
-    removeList: removeList,
-    resolveList: resolveList,
+    getLists,
+    getList,
+    addList,
+    updateList,
+    removeList,
+    resolveList,
 
+    OWNERTYPE_USER: 'user',
+    OWNERTYPE_GROUP: 'group',
+
+    DEFAULT_MEMORY_LIMIT: 512 * 1024 * 1024,
+
+    _removeMailboxes: removeMailboxes,
     _readDkimPublicKeySync: readDkimPublicKeySync
 };
 
-var assert = require('assert'),
+const assert = require('assert'),
     async = require('async'),
     BoxError = require('./boxerror.js'),
+    cloudron = require('./cloudron.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:mail'),
     dns = require('./native-dns.js'),
@@ -69,18 +80,23 @@ var assert = require('assert'),
     nodemailer = require('nodemailer'),
     path = require('path'),
     paths = require('./paths.js'),
+    request = require('request'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     smtpTransport = require('nodemailer-smtp-transport'),
     sysinfo = require('./sysinfo.js'),
+    system = require('./system.js'),
+    tasks = require('./tasks.js'),
     users = require('./users.js'),
     validator = require('validator'),
     _ = require('underscore');
 
 const DNS_OPTIONS = { timeout: 5000 };
 var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+const REMOVE_MAILBOX = path.join(__dirname, 'scripts/rmmailbox.sh');
 
 function validateName(name) {
     assert.strictEqual(typeof name, 'string');
@@ -100,7 +116,6 @@ function checkOutboundPort25(callback) {
     var smtpServer = _.sample([
         'smtp.gmail.com',
         'smtp.live.com',
-        'smtp.mail.yahoo.com',
         'smtp.1und1.de',
     ]);
 
@@ -237,14 +252,14 @@ function checkSpf(domain, mailFqdn, callback) {
             let txtRecord = txtRecords[i].join(''); // https://agari.zendesk.com/hc/en-us/articles/202952749-How-long-can-my-SPF-record-be-
             if (txtRecord.indexOf('v=spf1 ') !== 0) continue; // not SPF
             spf.value = txtRecord;
-            spf.status = spf.value.indexOf(' a:' + settings.adminFqdn()) !== -1;
+            spf.status = spf.value.indexOf(' a:' + settings.mailFqdn()) !== -1;
             break;
         }
 
         if (spf.status) {
             spf.expected = spf.value;
         } else if (i !== txtRecords.length) {
-            spf.expected = 'v=spf1 a:' + settings.adminFqdn() + ' ' + spf.value.slice('v=spf1 '.length);
+            spf.expected = 'v=spf1 a:' + settings.mailFqdn() + ' ' + spf.value.slice('v=spf1 '.length);
         }
 
         callback(null, spf);
@@ -543,7 +558,7 @@ function checkConfiguration(callback) {
                 markdownMessage += '\n\n';
             });
 
-            if (markdownMessage) markdownMessage += 'Email Status is checked every 30 minutes.\n See the [troubleshooting docs](https://cloudron.io/documentation/troubleshooting/#mail-dns) for more information.\n';
+            if (markdownMessage) markdownMessage += 'Email Status is checked every 30 minutes.\n See the [troubleshooting docs](https://docs.cloudron.io/troubleshooting/#mail-dns) for more information.\n';
 
             callback(null, markdownMessage); // empty message means all status checks succeeded
         });
@@ -575,15 +590,18 @@ function createMailConfig(mailFqdn, mailDomain, callback) {
         }
 
         // create sections for per-domain configuration
-        mailDomains.forEach(function (domain) {
+        async.eachSeries(mailDomains, function (domain, iteratorDone) {
             const catchAll = domain.catchAll.map(function (c) { return `${c}@${domain.domain}`; }).join(',');
             const mailFromValidation = domain.mailFromValidation;
 
             if (!safe.fs.appendFileSync(path.join(paths.ADDON_CONFIG_DIR, 'mail/mail.ini'),
                 `[${domain.domain}]\ncatch_all=${catchAll}\nmail_from_validation=${mailFromValidation}\n\n`, 'utf8')) {
-                return callback(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
+                return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
             }
 
+            if (!safe.fs.writeFileSync(`${paths.ADDON_CONFIG_DIR}/mail/banner/${domain.domain}.text`, domain.banner.text || '')) return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create text banner file:' + safe.error.message));
+            if (!safe.fs.writeFileSync(`${paths.ADDON_CONFIG_DIR}/mail/banner/${domain.domain}.html`, domain.banner.html || '')) return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create html banner file:' + safe.error.message));
+
             const relay = domain.relay;
 
             const enableRelay = relay.provider !== 'cloudron-smtp' && relay.provider !== 'noop',
@@ -593,21 +611,26 @@ function createMailConfig(mailFqdn, mailDomain, callback) {
                 username = relay.username || '',
                 password = relay.password || '';
 
-            if (!enableRelay) return;
+            if (!enableRelay) return iteratorDone();
 
             if (!safe.fs.appendFileSync(paths.ADDON_CONFIG_DIR + '/mail/smtp_forward.ini',
                 `[${domain.domain}]\nenable_outbound=true\nhost=${host}\nport=${port}\nenable_tls=true\nauth_type=${authType}\nauth_user=${username}\nauth_pass=${password}\n\n`, 'utf8')) {
-                return callback(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
+                return iteratorDone(new BoxError(BoxError.FS_ERROR, 'Could not create mail var file:' + safe.error.message));
             }
-        });
 
-        callback(null, mailInDomains.length !== 0 /* allowInbound */);
+            iteratorDone();
+        }, function (error) {
+            if (error) return callback(error);
+
+            callback(null, mailInDomains.length !== 0 /* allowInbound */);
+        });
     });
 }
 
-function configureMail(mailFqdn, mailDomain, callback) {
+function configureMail(mailFqdn, mailDomain, serviceConfig, callback) {
     assert.strictEqual(typeof mailFqdn, 'string');
     assert.strictEqual(typeof mailDomain, 'string');
+    assert.strictEqual(typeof serviceConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     // mail (note: 2525 is hardcoded in mail container and app use this port)
@@ -616,7 +639,8 @@ function configureMail(mailFqdn, mailDomain, callback) {
     // mail container uses /app/data for backed up data and /run for restart-able data
 
     const tag = infra.images.mail.tag;
-    const memoryLimit = 4 * 256;
+    const memoryLimit = serviceConfig.memoryLimit || exports.DEFAULT_MEMORY_LIMIT;
+    const memory = system.getMemoryAllocation(memoryLimit);
     const cloudronToken = hat(8 * 128), relayToken = hat(8 * 128);
 
     reverseProxy.getCertificate(mailFqdn, mailDomain, function (error, bundle) {
@@ -629,7 +653,10 @@ function configureMail(mailFqdn, mailDomain, callback) {
         if (!safe.child_process.execSync(`cp ${bundle.certFilePath} ${mailCertFilePath}`)) return callback(new BoxError(BoxError.FS_ERROR, 'Could not create cert file:' + safe.error.message));
         if (!safe.child_process.execSync(`cp ${bundle.keyFilePath} ${mailKeyFilePath}`)) return callback(new BoxError(BoxError.FS_ERROR, 'Could not create key file:' + safe.error.message));
 
-        shell.exec('startMail', 'docker rm -f mail || true', function (error) {
+        async.series([
+            shell.exec.bind(null, 'stopMail', 'docker stop mail || true'),
+            shell.exec.bind(null, 'removeMail', 'docker rm -f mail || true'),
+        ], function (error) {
             if (error) return callback(error);
 
             createMailConfig(mailFqdn, mailDomain, function (error, allowInbound) {
@@ -644,8 +671,8 @@ function configureMail(mailFqdn, mailDomain, callback) {
                             --log-opt syslog-address=udp://127.0.0.1:2514 \
                             --log-opt syslog-format=rfc5424 \
                             --log-opt tag=mail \
-                            -m ${memoryLimit}m \
-                            --memory-swap ${memoryLimit * 2}m \
+                            -m ${memory} \
+                            --memory-swap ${memoryLimit} \
                             --dns 172.18.0.1 \
                             --dns-search=. \
                             -e CLOUDRON_MAIL_TOKEN="${cloudronToken}" \
@@ -692,8 +719,12 @@ function restartMail(callback) {
 
     if (process.env.BOX_ENV === 'test' && !process.env.TEST_CREATE_INFRA) return callback();
 
-    debug(`restartMail: restarting mail container with ${settings.mailFqdn()} ${settings.adminDomain()}`);
-    configureMail(settings.mailFqdn(), settings.adminDomain(), callback);
+    services.getServiceConfig('mail', function (error, serviceConfig) {
+        if (error) return callback(error);
+
+        debug(`restartMail: restarting mail container with ${settings.mailFqdn()} ${settings.adminDomain()}`);
+        configureMail(settings.mailFqdn(), settings.adminDomain(), serviceConfig, callback);
+    });
 }
 
 function restartMailIfActivated(callback) {
@@ -746,7 +777,7 @@ function txtRecordsWithSpf(domain, mailFqdn, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     domains.getDnsRecords('', domain, 'TXT', function (error, txtRecords) {
-        if (error) return error;
+        if (error) return callback(error);
 
         debug('txtRecordsWithSpf: current txt records - %j', txtRecords);
 
@@ -844,37 +875,40 @@ function upsertDnsRecords(domain, mailFqdn, callback) {
 
         if (process.env.BOX_ENV === 'test') return callback();
 
-        var dkimKey = readDkimPublicKeySync(domain);
+        const dkimKey = readDkimPublicKeySync(domain);
         if (!dkimKey) return callback(new BoxError(BoxError.FS_ERROR, 'Failed to read dkim public key'));
 
         // t=s limits the domainkey to this domain and not it's subdomains
-        var dkimRecord = { subdomain: `${mailDomain.dkimSelector}._domainkey`, domain: domain, type: 'TXT', values: [ '"v=DKIM1; t=s; p=' + dkimKey + '"' ] };
+        const dkimRecord = { subdomain: `${mailDomain.dkimSelector}._domainkey`, domain: domain, type: 'TXT', values: [ `"v=DKIM1; t=s; p=${dkimKey}"` ] };
 
-        var records = [ ];
+        let records = [];
         records.push(dkimRecord);
-        if (mailDomain.enabled) {
-            records.push({ subdomain: '_dmarc', domain: domain, type: 'TXT', values: [ '"v=DMARC1; p=reject; pct=100"' ] });
-            records.push({ subdomain: '', domain: domain, type: 'MX', values: [ '10 ' + mailFqdn + '.' ] });
-        }
+        if (mailDomain.enabled) records.push({ subdomain: '', domain: domain, type: 'MX', values: [ '10 ' + mailFqdn + '.' ] });
 
         txtRecordsWithSpf(domain, mailFqdn, function (error, txtRecords) {
             if (error) return callback(error);
 
             if (txtRecords) records.push({ subdomain: '', domain: domain, type: 'TXT', values: txtRecords });
 
-            debug('upsertDnsRecords: will update %j', records);
+            domains.getDnsRecords('_dmarc', domain, 'TXT', function (error, dmarcRecords) { // only update dmarc if absent. this allows user to set email for reporting
+                if (error) return callback(error);
 
-            async.mapSeries(records, function (record, iteratorCallback) {
-                domains.upsertDnsRecords(record.subdomain, record.domain, record.type, record.values, iteratorCallback);
-            }, function (error, changeIds) {
-                if (error) {
-                    debug(`upsertDnsRecords: failed to update: ${error}`);
-                    return callback(error);
-                }
+                if (dmarcRecords.length === 0) records.push({ subdomain: '_dmarc', domain: domain, type: 'TXT', values: [ '"v=DMARC1; p=reject; pct=100"' ] });
 
-                debug('upsertDnsRecords: records %j added with changeIds %j', records, changeIds);
+                debug('upsertDnsRecords: will update %j', records);
 
-                callback(null);
+                async.mapSeries(records, function (record, iteratorCallback) {
+                    domains.upsertDnsRecords(record.subdomain, record.domain, record.type, record.values, iteratorCallback);
+                }, function (error, changeIds) {
+                    if (error) {
+                        debug(`upsertDnsRecords: failed to update: ${error}`);
+                        return callback(error);
+                    }
+
+                    debug('upsertDnsRecords: records %j added with changeIds %j', records, changeIds);
+
+                    callback(null);
+                });
             });
         });
     });
@@ -887,21 +921,73 @@ function setDnsRecords(domain, callback) {
     upsertDnsRecords(domain, settings.mailFqdn(), callback);
 }
 
-function onMailFqdnChanged(callback) {
+function getLocation(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    const mailFqdn = settings.mailFqdn(),
-        mailDomain = settings.adminDomain();
+    const domain = settings.mailDomain(), fqdn = settings.mailFqdn();
+    const subdomain = fqdn.substr(0, fqdn.length - domain.length - 1);
 
-    domains.getAll(function (error, allDomains) {
+    callback(null, { domain, subdomain });
+}
+
+function changeLocation(auditSource, progressCallback, callback) {
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    const fqdn = settings.mailFqdn(), domain = settings.mailDomain();
+    const subdomain = fqdn.substr(0, fqdn.length - domain.length - 1);
+
+    let progress = 20;
+    progressCallback({ percent: progress, message: `Setting up DNS of certs of mail server ${fqdn}` });
+
+    cloudron.setupDnsAndCert(subdomain, domain, auditSource, progressCallback, function (error) {
         if (error) return callback(error);
 
-        async.eachOfSeries(allDomains, function (domainObject, idx, iteratorDone) {
-            upsertDnsRecords(domainObject.domain, mailFqdn, iteratorDone);
-        }, function (error) {
+        domains.getAll(function (error, allDomains) {
             if (error) return callback(error);
 
-            configureMail(mailFqdn, mailDomain, callback);
+            async.eachOfSeries(allDomains, function (domainObject, idx, iteratorDone) {
+                progressCallback({ percent: progress, message: `Updating DNS of ${domainObject.domain}` });
+                progress += Math.round(70/allDomains.length);
+
+                upsertDnsRecords(domainObject.domain, fqdn, function (error) { // ignore any errors. we anyway report dns errors in status tab
+                    progressCallback({ percent: progress, message: `Updated DNS of ${domainObject.domain}: ${error ? error.message : 'success'}` });
+                    iteratorDone();
+                });
+            }, function (error) {
+                if (error) return callback(error);
+
+                progressCallback({ percent: 90, message: 'Restarting mail server' });
+
+                restartMailIfActivated(callback);
+            });
+        });
+    });
+}
+
+function setLocation(subdomain, domain, auditSource, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    domains.get(domain, function (error, domainObject) {
+        if (error) return callback(error);
+
+        const fqdn = domains.fqdn(subdomain, domainObject);
+
+        settings.setMailLocation(domain, fqdn, function (error) {
+            if (error) return callback(error);
+
+            tasks.add(tasks.TASK_CHANGE_MAIL_LOCATION, [ auditSource ], function (error, taskId) {
+                if (error) return callback(error);
+
+                tasks.startTask(taskId, {}, NOOP_CALLBACK);
+                eventlog.add(eventlog.ACTION_MAIL_LOCATION, auditSource, { subdomain, domain, taskId });
+
+                callback(null, taskId);
+            });
         });
     });
 }
@@ -910,6 +996,8 @@ function onDomainAdded(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
+    if (!settings.mailFqdn()) return callback(); // mail domain is not set yet (when provisioning)
+
     async.series([
         upsertDnsRecords.bind(null, domain, settings.mailFqdn()), // do this first to ensure DKIM keys
         restartMailIfActivated
@@ -935,7 +1023,7 @@ function clearDomains(callback) {
 
 // remove all fields that should never be sent out via REST API
 function removePrivateFields(domain) {
-    let result = _.pick(domain, 'domain', 'enabled', 'mailFromValidation', 'catchAll', 'relay');
+    let result = _.pick(domain, 'domain', 'enabled', 'mailFromValidation', 'catchAll', 'relay', 'banner');
     if (result.relay.provider !== 'cloudron-smtp') {
         if (result.relay.username === result.relay.password) result.relay.username = constants.SECRET_PLACEHOLDER;
         result.relay.password = constants.SECRET_PLACEHOLDER;
@@ -957,6 +1045,20 @@ function setMailFromValidation(domain, enabled, callback) {
     });
 }
 
+function setBanner(domain, banner, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof banner, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    maildb.update(domain, { banner }, function (error) {
+        if (error) return callback(error);
+
+        restartMail(NOOP_CALLBACK);
+
+        callback(null);
+    });
+}
+
 function setCatchAllAddress(domain, addresses, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(addresses));
@@ -1032,13 +1134,25 @@ function sendTestMail(domain, to, callback) {
     });
 }
 
-function listMailboxes(domain, page, perPage, callback) {
+function listMailboxes(domain, search, page, perPage, callback) {
     assert.strictEqual(typeof domain, 'string');
+    assert(typeof search === 'string' || search === null);
     assert.strictEqual(typeof page, 'number');
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    mailboxdb.listMailboxes(domain, page, perPage, function (error, result) {
+    mailboxdb.listMailboxes(domain, search, page, perPage, function (error, result) {
+        if (error) return callback(error);
+
+        callback(null, result);
+    });
+}
+
+function getMailboxCount(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    mailboxdb.getMailboxCount(domain, function (error, result) {
         if (error) return callback(error);
 
         callback(null, result);
@@ -1068,10 +1182,11 @@ function getMailbox(name, domain, callback) {
     });
 }
 
-function addMailbox(name, domain, userId, auditSource, callback) {
+function addMailbox(name, domain, ownerId, ownerType, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1080,49 +1195,80 @@ function addMailbox(name, domain, userId, auditSource, callback) {
     var error = validateName(name);
     if (error) return callback(error);
 
-    mailboxdb.addMailbox(name, domain, userId, function (error) {
+    if (ownerType !== exports.OWNERTYPE_USER && ownerType !== exports.OWNERTYPE_GROUP) return callback(new BoxError(BoxError.BAD_FIELD, 'bad owner type'));
+
+    mailboxdb.addMailbox(name, domain, ownerId, ownerType, function (error) {
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_ADD, auditSource, { name, domain, userId });
+        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_ADD, auditSource, { name, domain, ownerId, ownerType });
 
         callback(null);
     });
 }
 
-function updateMailboxOwner(name, domain, userId, auditSource, callback) {
+function updateMailboxOwner(name, domain, ownerId, ownerType, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     name = name.toLowerCase();
 
+    if (ownerType !== exports.OWNERTYPE_USER && ownerType !== exports.OWNERTYPE_GROUP) return callback(new BoxError(BoxError.BAD_FIELD, 'bad owner type'));
+
     getMailbox(name, domain, function (error, result) {
         if (error) return callback(error);
 
-        mailboxdb.updateMailboxOwner(name, domain, userId, function (error) {
+        mailboxdb.updateMailboxOwner(name, domain, ownerId, ownerType, function (error) {
             if (error) return callback(error);
 
-            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldUserId: result.userId, userId });
+            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldUserId: result.userId, ownerId, ownerType });
 
             callback(null);
         });
     });
 }
 
-function removeMailbox(name, domain, auditSource, callback) {
+function removeSolrIndex(mailbox, callback) {
+    assert.strictEqual(typeof mailbox, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    services.getContainerDetails('mail', 'CLOUDRON_MAIL_TOKEN', function (error, addonDetails) {
+        if (error) return callback(error);
+
+        request.post(`https://${addonDetails.ip}:3000/solr_delete_index?access_token=${addonDetails.token}`, { timeout: 2000, rejectUnauthorized: false, json: { mailbox } }, function (error, response) {
+            if (error) return callback(error);
+
+            if (response.statusCode !== 200) return callback(new Error(`Error removing solr index - ${response.statusCode} ${JSON.stringify(response.body)}`));
+
+            callback(null);
+        });
+    });
+}
+
+function removeMailbox(name, domain, options, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    mailboxdb.del(name, domain, function (error) {
-        if (error) return callback(error);
+    const mailbox =`${name}@${domain}`;
+    const deleteMailFunc = options.deleteMails ? shell.sudo.bind(null, 'removeMailbox', [ REMOVE_MAILBOX, mailbox ], {}) : (next) => next();
 
-        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_REMOVE, auditSource, { name, domain });
+    deleteMailFunc(function (error) {
+        if (error) return callback(new BoxError(BoxError.FS_ERROR, `Error removing mailbox: ${error.message}`));
 
-        callback(null);
+        mailboxdb.del(name, domain, function (error) {
+            if (error) return callback(error);
+
+            removeSolrIndex(mailbox, NOOP_CALLBACK);
+            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_REMOVE, auditSource, { name, domain });
+
+            callback();
+        });
     });
 }
 
@@ -1165,11 +1311,14 @@ function setAliases(name, domain, aliases, callback) {
     });
 }
 
-function getLists(domain, callback) {
+function getLists(domain, search, page, perPage, callback) {
     assert.strictEqual(typeof domain, 'string');
+    assert(typeof search === 'string' || search === null);
+    assert.strictEqual(typeof page, 'number');
+    assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    mailboxdb.getLists(domain, function (error, result) {
+    mailboxdb.getLists(domain, search, page, perPage, function (error, result) {
         if (error) return callback(error);
 
         callback(null, result);
@@ -1275,7 +1424,7 @@ function resolveList(listName, listDomain, callback) {
 
             let result = [], toResolve = list.members.slice(), visited = []; // slice creates a copy of array
 
-            async.whilst(() => toResolve.length != 0, function (iteratorCallback) {
+            async.whilst((testDone) => testDone(null, toResolve.length != 0), function (iteratorCallback) {
                 const toProcess = toResolve.shift();
                 const parts = toProcess.split('@');
                 const memberName = parts[0].split('+')[0], memberDomain = parts[1];
@@ -1296,7 +1445,7 @@ function resolveList(listName, listDomain, callback) {
                     if (entry.type === mailboxdb.TYPE_MAILBOX) { // concrete mailbox
                         result.push(member);
                     } else if (entry.type === mailboxdb.TYPE_ALIAS) { // resolve aliases
-                        toResolve = toResolve.concat(`${entry.aliasName}@${entry.aliasTarget}`);
+                        toResolve = toResolve.concat(`${entry.aliasName}@${entry.aliasDomain}`);
                     } else { // resolve list members
                         toResolve = toResolve.concat(entry.members);
                     }

src/mail_templates/app_down.ejs

@@ -7,8 +7,8 @@ The application '<%= title %>' installed at <%= appFqdn %> is not responding.
 This is most likely a problem in the application.
 
 To resolve this, you can try the following:
-* Restart the app by opening the app's web terminal - https://cloudron.io/documentation/apps/#web-terminal
-* Restore the app to the latest backup - https://cloudron.io/documentation/backups/#restoring-an-app
+* Restart the app by opening the app's web terminal - https://docs.cloudron.io/apps/#web-terminal
+* Restore the app to the latest backup - https://docs.cloudron.io/backups/#restoring-an-app
 * Contact us via <%= supportEmail %> or https://forum.cloudron.io
 
 

src/mail_templates/app_updates_available.ejs

@@ -3,18 +3,14 @@
 Dear Cloudron Admin,
 
 <% for (var i = 0; i < apps.length; i++) { -%>
-A new version <%= apps[i].updateInfo.manifest.version %> of the app '<%= apps[i].app.manifest.title %>' installed at <%= apps[i].app.fqdn %> is available.
+The app '<%= apps[i].app.manifest.title %>' installed at <%= apps[i].app.fqdn %> has an update available.
 
-Changes:
+<%= apps[i].app.manifest.title %> v<%= apps[i].updateInfo.manifest.version %> changes:
 <%= apps[i].updateInfo.manifest.changelog %>
 
 <% } -%>
 
-<% if (!hasSubscription) { -%>
-*Keep your Cloudron automatically up-to-date and secure by upgrading to a paid plan at* <%= webadminUrl %>/#/settings
-<% } else { -%>
 Update now at <%= webadminUrl %>
-<% } -%>
 
 Powered by https://cloudron.io
 
@@ -33,24 +29,20 @@ Sent at: <%= new Date().toUTCString() %>
     <div style="width: 650px; text-align: left;">
 <% for (var i = 0; i < apps.length; i++) { -%>
         <p>
-            A new version <%= apps[i].updateInfo.manifest.version %> of the app '<%= apps[i].app.manifest.title %>' installed at <a href="https://<%= apps[i].app.fqdn %>"><%= apps[i].app.fqdn %></a> is available.
+            The app '<%= apps[i].app.manifest.title %>' installed at <a href="https://<%= apps[i].app.fqdn %>"><%= apps[i].app.fqdn %></a> has an update available.
         </p>
 
-        <h5>Changelog:</h5>
+        <h5><%= apps[i].app.manifest.title %> v<%= apps[i].updateInfo.manifest.version %> changes:</h5>
         <%- apps[i].changelogHTML %>
 
         <br/>
 <% } -%>
 
-<% if (!hasSubscription) { -%>
-        <p>Keep your Cloudron automatically up-to-date and secure by upgrading to a <a href="<%= webadminUrl %>/#/settings">paid plan</a>.</p>
-<% } else { -%>
         <p>
             <br/>
             <center><a href="<%= webadminUrl %>">Update now</a></center>
             <br/>
         </p>
-<% } -%>
     </div>
 
     <div style="font-size: 10px; color: #333333; background: #ffffff;">

src/mail_templates/backup_failed.ejs

@@ -2,7 +2,7 @@
 
 Dear <%= cloudronName %> Admin,
 
-Cloudron failed to create a complete backup. Please see https://cloudron.io/documentation/troubleshooting/#backups
+Cloudron failed to create a complete backup. Please see https://docs.cloudron.io/troubleshooting/#backups
 for troubleshooting.
 
 Logs for this failure are available at <%= logUrl %>