typo

this happens because we have some bug in sftp container causing uninstall(s) to
fail. the database of those apps are gone but the export logic then tries to export
them and it all fails.

CHANGES

@@ -2112,6 +2112,119 @@
 * postgresql: set collation order explicity when creating database to C.UTF-8 (for confluence)
 * rsync: fix error while goes missing when syncing
 * Pre-select app domain by default in the redirection drop down
+* robots: preseve leading and trailing whitespaces/newlines
+
+[5.6.3]
+* Fix postgres locale issue
 
 [6.0.0]
 * Focal support
+* Reduce duration of self-signed certs to 800 days
+* Better backup config filename when downloading
+* branding: footer can have template variables like %YEAR% and %VERSION%
+* sftp: secure the API with a token
+* filemanager: Add extract context menu item
+* Do not download docker images if present locally
+* sftp: disable access to non-admins by default
+* postgresql: whitelist pgcrypto extension for loomio
+* filemanager: Add new file creation action and collapse new and upload actions
+* rsync: add warning to remove lifecycle rules
+* Add volume management
+* backups: adjust node's heap size based on memory limit
+* s3: diasble per-chunk timeout
+* logs: more descriptive log file names on download
+* collectd: remove collectd config when app stopped (and add it back when started)
+* Apps can optionally request an authwall to be installed in front of them
+* mailbox can now owned by a group
+* linode: enable dns provider in setup view
+* dns: apps can now use the dns port
+* httpPaths: allow apps to specify forwarding from custom paths to container ports (for OLS)
+* add elasticemail smtp relay option
+* mail: add option to fts using solr
+* mail: change the namespace separator of new installations to /
+* mail: enable acl
+* Disable THP
+* filemanager: allow download dirs as zip files
+* aws: add china region
+* security: fix issue where apps could send with any username (but valid password)
+* i18n support
+
+[6.0.1]
+* app: add export route
+* mail: on location change, fix lock up when one or more domains have invalid credentials
+* mail: fix crash because of write after timeout closure
+* scaleway: fix installation issue where THP is not enabled in kernel
+
+[6.1.0]
+* mail: update haraka to 2.8.27. this fixes zero-length queue file crash
+* update: set/unset appStoreId from the update route
+* proxyauth: Do not follow redirects
+* proxyauth: add 2FA
+* appstore: add category translations
+* appstore: add media category
+* prepend the version to assets when sourcing to avoid cache hits on update
+* filemanger: list volumes of the app
+* Display upload size and size progress
+* nfs: chown the backups for hardlinks to work
+* remove user add/remove/role change email notifications
+* persist update indicator across restarts
+* cloudron-setup: add --generate-setup-token
+* dashboard: pass accessToken query param to automatically login
+* wellknown: add a way to set well known docs
+* oom: notification mails have links to dashboard
+* collectd: do not install xorg* packages
+* apptask: backup/restore tasks now use the backup memory limit configuration
+* eventlog: add logout event
+* mailbox: include alias in mailbox search
+* proxyAuth: add path exclusion
+* turn: fix for CVE-2020-26262
+* app password: fix regression where apps are not listed anymore in the UI
+* Support for multiDomain apps (domain aliases)
+* netcup: add dns provider
+* Container swap size is now dynamically determined based on system RAM/swap ratio
+
+[6.1.1]
+* Fix bug where platform does not start if memory limits could not be applied
+
+[6.1.2]
+* App disk usage was not shown in graphs
+* Email autoconfig
+* Fix SOGo login
+
+[6.2.0]
+* ovh: object storage URL has changed from s3 to storage subdomain
+* ionos: add profit bricks object storage
+* update node to 14.15.4
+* update docker to 20.10.3
+* new base image 3.0.0
+* postgresql updated to 12.5
+* redis updated to 5.0.7
+* dovecot updated to 2.3.7
+* proxyAuth: fix docker UA detection
+* registry config: add UI to disable it
+* update solr to 8.8.1
+* firewall: fix issue where script errored when having more than 15 wl/bl ports
+* If groups are used, do not allow app installation without choosing the access settings
+* tls addon
+* Do not overwrite existing DMARC record
+* Sync dns records
+* Dry run restore
+* linode: show cloudron is installing when user SSHs
+* mysql: disable bin logs
+* Show cancel task button if task is still running after 2 minutes
+* filemanager: fix various bugs involving file names with spaces
+* Change Referrer-policy default to 'same-origin'
+* rsync: preserve and restore symlinks
+* Clean up backups function now removes missing backups
+
+[6.2.1]
+* Avoid updown notifications on full restore
+* Add retries to downloader logic in installer
+
+[6.2.2]
+* Fix ENOBUFS issue with backups when collecting fs metadata
+
+[6.2.3]
+* Fix addon crashes with missing databases
+* Update mail container for LMTP cert fix
+* Fix services view showing yellow icon

baseimage/createAMI

@@ -1,193 +0,0 @@
-#!/bin/bash
-
-set -eu -o pipefail
-
-assertNotEmpty() {
-    : "${!1:? "$1 is not set."}"
-}
-
-readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-readonly SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")"/.. && pwd)"
-export JSON="${SOURCE_DIR}/node_modules/.bin/json"
-
-INSTANCE_TYPE="t2.micro"
-BLOCK_DEVICE="DeviceName=/dev/sda1,Ebs={VolumeSize=20,DeleteOnTermination=true,VolumeType=gp2}"
-SSH_KEY_NAME="id_rsa_yellowtent"
-
-revision=$(git rev-parse HEAD)
-ami_name=""
-server_id=""
-server_ip=""
-destroy_server="yes"
-deploy_env="prod"
-image_id=""
-
-args=$(getopt -o "" -l "revision:,name:,no-destroy,env:,region:" -n "$0" -- "$@")
-eval set -- "${args}"
-
-while true; do
-    case "$1" in
-    --env) deploy_env="$2"; shift 2;;
-    --revision) revision="$2"; shift 2;;
-    --name) ami_name="$2"; shift 2;;
-    --no-destroy) destroy_server="no"; shift 2;;
-    --region)
-        case "$2" in
-        "us-east-1")
-            image_id="ami-6edd3078"
-            security_group="sg-a5e17fd9"
-            subnet_id="subnet-b8fbc0f1"
-            ;;
-        "eu-central-1")
-            image_id="ami-5aee2235"
-            security_group="sg-19f5a770" # everything open on eu-central-1
-            subnet_id=""
-            ;;
-        *)
-            echo "Unknown aws region $2"
-            exit 1
-            ;;
-        esac
-        export AWS_DEFAULT_REGION="$2"    # used by the aws cli tool
-        shift 2
-        ;;
-    --) break;;
-    *) echo "Unknown option $1"; exit 1;;
-    esac
-done
-
-# TODO fix this
-export AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY}"
-export AWS_SECRET_ACCESS_KEY="${AWS_ACCESS_SECRET}"
-
-readonly ssh_keys="${HOME}/.ssh/id_rsa_yellowtent"
-readonly SSH="ssh -o IdentitiesOnly=yes -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${ssh_keys}"
-
-if [[ ! -f "${ssh_keys}" ]]; then
-    echo "caas ssh key is missing at ${ssh_keys} (pick it up from secrets repo)"
-    exit 1
-fi
-
-if [[ -z "${image_id}" ]]; then
-    echo "--region is required (us-east-1 or eu-central-1)"
-    exit 1
-fi
-
-function get_pretty_revision() {
-    local git_rev="$1"
-    local sha1=$(git rev-parse --short "${git_rev}" 2>/dev/null)
-
-    echo "${sha1}"
-}
-
-function wait_for_ssh() {
-    echo "=> Waiting for ssh connection"
-    while true; do
-        echo -n "."
-
-        if $SSH ubuntu@${server_ip} echo "hello"; then
-            echo ""
-            break
-        fi
-
-        sleep 5
-    done
-}
-
-now=$(date "+%Y-%m-%d-%H%M%S")
-pretty_revision=$(get_pretty_revision "${revision}")
-
-if [[ -z "${ami_name}" ]]; then
-    ami_name="box-${deploy_env}-${pretty_revision}-${now}"
-fi
-
-echo "=> Create EC2 instance"
-id=$(aws ec2 run-instances --image-id "${image_id}" --instance-type "${INSTANCE_TYPE}" --security-group-ids "${security_group}" --block-device-mappings "${BLOCK_DEVICE}" --key-name "${SSH_KEY_NAME}" --subnet-id "${subnet_id}" --associate-public-ip-address \
-    | $JSON Instances \
-    | $JSON 0.InstanceId)
-
-[[ -z "$id" ]] && exit 1
-echo "Instance created  ID $id"
-
-echo "=> Waiting for instance to get a public IP"
-while true; do
-    server_ip=$(aws ec2 describe-instances --instance-ids ${id} \
-        | $JSON Reservations.0.Instances \
-        | $JSON 0.PublicIpAddress)
-
-    if [[ ! -z "${server_ip}" ]]; then
-        echo ""
-        break
-    fi
-
-    echo -n "."
-    sleep 1
-done
-
-echo "Got public IP ${server_ip}"
-
-wait_for_ssh
-
-echo "=> Fetching cloudron-setup"
-while true; do
-
-    if $SSH ubuntu@${server_ip} wget "https://cloudron.io/cloudron-setup" -O "cloudron-setup"; then
-        echo ""
-        break
-    fi
-
-    echo -n "."
-    sleep 5
-done
-
-echo "=> Running cloudron-setup"
-$SSH ubuntu@${server_ip} sudo /bin/bash "cloudron-setup" --env "${deploy_env}" --provider "ami" --skip-reboot
-
-wait_for_ssh
-
-echo "=> Removing ssh key"
-$SSH ubuntu@${server_ip} sudo rm /home/ubuntu/.ssh/authorized_keys /root/.ssh/authorized_keys
-
-echo "=> Creating AMI"
-image_id=$(aws ec2 create-image --instance-id "${id}" --name "${ami_name}" | $JSON ImageId)
-[[ -z "$id" ]] && exit 1
-echo "Creating AMI with Id ${image_id}"
-
-echo "=> Waiting for AMI to be created"
-while true; do
-    state=$(aws ec2 describe-images --image-ids ${image_id} \
-        | $JSON Images \
-        | $JSON 0.State)
-
-    if [[ "${state}" == "available" ]]; then
-        echo ""
-        break
-    fi
-
-    echo -n "."
-    sleep 5
-done
-
-if [[ "${destroy_server}" == "yes" ]]; then
-    echo "=> Deleting EC2 instance"
-
-    while true; do
-        state=$(aws ec2 terminate-instances --instance-id "${id}" \
-            | $JSON TerminatingInstances \
-            | $JSON 0.CurrentState.Name)
-
-        if [[ "${state}" == "shutting-down" ]]; then
-            echo ""
-            break
-        fi
-
-        echo -n "."
-        sleep 5
-    done
-fi
-
-echo ""
-echo "Done."
-echo ""
-echo "New AMI is: ${image_id}"
-echo ""

baseimage/digitalocean.sh

@@ -1,261 +0,0 @@
-#!/bin/bash
-
-if [[ -z "${DIGITAL_OCEAN_TOKEN}" ]]; then
-    echo "Script requires DIGITAL_OCEAN_TOKEN env to be set"
-    exit 1
-fi
-
-if [[ -z "${JSON}" ]]; then
-    echo "Script requires JSON env to be set to path of JSON binary"
-    exit 1
-fi
-
-readonly CURL="curl --retry 5 -s -u ${DIGITAL_OCEAN_TOKEN}:"
-
-function debug() {
-    echo "$@" >&2
-}
-
-function get_ssh_key_id() {
-    id=$($CURL "https://api.digitalocean.com/v2/account/keys" \
-        | $JSON ssh_keys \
-        | $JSON -c "this.name === \"$1\"" \
-        | $JSON 0.id)
-    [[ -z "$id" ]] && exit 1
-    echo "$id"
-}
-
-function create_droplet() {
-    local ssh_key_id="$1"
-    local box_name="$2"
-
-    local image_region="sfo2"
-    local ubuntu_image_slug="ubuntu-16-04-x64"
-    local box_size="1gb"
-
-    local data="{\"name\":\"${box_name}\",\"size\":\"${box_size}\",\"region\":\"${image_region}\",\"image\":\"${ubuntu_image_slug}\",\"ssh_keys\":[ \"${ssh_key_id}\" ],\"backups\":false}"
-
-    id=$($CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets" | $JSON droplet.id)
-    [[ -z "$id" ]] && exit 1
-    echo "$id"
-}
-
-function get_droplet_ip() {
-    local droplet_id="$1"
-    ip=$($CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}" | $JSON "droplet.networks.v4[0].ip_address")
-    [[ -z "$ip" ]] && exit 1
-    echo "$ip"
-}
-
-function get_droplet_id() {
-    local droplet_name="$1"
-    id=$($CURL "https://api.digitalocean.com/v2/droplets?per_page=200" | $JSON "droplets" | $JSON -c "this.name === '${droplet_name}'" | $JSON "[0].id")
-    [[ -z "$id" ]] && exit 1
-    echo "$id"
-}
-
-function power_off_droplet() {
-    local droplet_id="$1"
-    local data='{"type":"power_off"}'
-    local response=$($CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions")
-    local event_id=`echo "${response}" | $JSON action.id`
-
-    if [[ -z "${event_id}" ]]; then
-        debug "Got no event id, assuming already powered off."
-        debug "Response: ${response}"
-        return
-    fi
-
-    debug "Powered off droplet. Event id: ${event_id}"
-    debug -n "Waiting for droplet to power off"
-
-    while true; do
-        local event_status=`$CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions/${event_id}" | $JSON action.status`
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug ""
-}
-
-function power_on_droplet() {
-    local droplet_id="$1"
-    local data='{"type":"power_on"}'
-    local event_id=`$CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions" | $JSON action.id`
-
-    debug "Powered on droplet. Event id: ${event_id}"
-
-    if [[ -z "${event_id}" ]]; then
-        debug "Got no event id, assuming already powered on"
-        return
-    fi
-
-    debug -n "Waiting for droplet to power on"
-
-    while true; do
-        local event_status=`$CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions/${event_id}" | $JSON action.status`
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug ""
-}
-
-function get_image_id() {
-    local snapshot_name="$1"
-    local image_id=""
-
-    if ! response=$($CURL "https://api.digitalocean.com/v2/images?per_page=200"); then
-        echo "Failed to get image listing. ${response}"
-        return 1
-    fi
-
-    if ! image_id=$(echo "$response" \
-       | $JSON images \
-       | $JSON -c "this.name === \"${snapshot_name}\"" 0.id); then
-        echo "Failed to parse curl response: ${response}"
-        return 1
-    fi
-
-    if [[ -z "${image_id}" ]]; then
-        echo "Failed to get image id of ${snapshot_name}. reponse: ${response}"
-        return 1
-    fi
-
-    echo "${image_id}"
-}
-
-function snapshot_droplet() {
-    local droplet_id="$1"
-    local snapshot_name="$2"
-    local data="{\"type\":\"snapshot\",\"name\":\"${snapshot_name}\"}"
-    local event_id=`$CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions" | $JSON action.id`
-
-    debug "Droplet snapshotted as ${snapshot_name}. Event id: ${event_id}"
-    debug -n "Waiting for snapshot to complete"
-
-    while true; do
-        if ! response=$($CURL "https://api.digitalocean.com/v2/droplets/${droplet_id}/actions/${event_id}"); then
-            echo "Could not get action status. ${response}"
-            continue
-        fi
-        if ! event_status=$(echo "${response}" | $JSON action.status); then
-            echo "Could not parse action.status from response. ${response}"
-            continue
-        fi
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug "! done"
-
-    if ! image_id=$(get_image_id "${snapshot_name}"); then
-        return 1
-    fi
-    echo "${image_id}"
-}
-
-function destroy_droplet() {
-    local droplet_id="$1"
-    # TODO: check for 204 status
-    $CURL -X DELETE "https://api.digitalocean.com/v2/droplets/${droplet_id}"
-    debug "Droplet destroyed"
-    debug ""
-}
-
-function transfer_image() {
-    local image_id="$1"
-    local region_slug="$2"
-    local data="{\"type\":\"transfer\",\"region\":\"${region_slug}\"}"
-    local event_id=`$CURL -X POST -H 'Content-Type: application/json' -d "${data}" "https://api.digitalocean.com/v2/images/${image_id}/actions" | $JSON action.id`
-    echo "${event_id}"
-}
-
-function wait_for_image_event() {
-    local image_id="$1"
-    local event_id="$2"
-
-    debug -n "Waiting for ${event_id}"
-
-    while true; do
-        local event_status=`$CURL "https://api.digitalocean.com/v2/images/${image_id}/actions/${event_id}" | $JSON action.status`
-        if [[ "${event_status}" == "completed" ]]; then
-            break
-        fi
-        debug -n "."
-        sleep 10
-    done
-    debug ""
-}
-
-function transfer_image_to_all_regions() {
-    local image_id="$1"
-
-    xfer_events=()
-    image_regions=(ams2) ## sfo1 is where the image is created
-    for image_region in ${image_regions[@]}; do
-        xfer_event=$(transfer_image ${image_id} ${image_region})
-        echo "Image transfer to ${image_region} initiated. Event id: ${xfer_event}"
-        xfer_events+=("${xfer_event}")
-        sleep 1
-    done
-
-    echo "Image transfer initiated, but they will take some time to get transferred."
-
-    for xfer_event in ${xfer_events[@]}; do
-        $vps wait_for_image_event "${image_id}" "${xfer_event}"
-    done
-}
-
-if [[ $# -lt 1 ]]; then
-    debug "<command> <params...>"
-    exit 1
-fi
-
-case $1 in
-get_ssh_key_id)
-    get_ssh_key_id "${@:2}"
-    ;;
-
-create)
-    create_droplet "${@:2}"
-    ;;
-
-get_id)
-    get_droplet_id "${@:2}"
-    ;;
-
-get_ip)
-    get_droplet_ip "${@:2}"
-    ;;
-
-power_on)
-    power_on_droplet "${@:2}"
-    ;;
-
-power_off)
-    power_off_droplet "${@:2}"
-    ;;
-
-snapshot)
-    snapshot_droplet "${@:2}"
-    ;;
-
-destroy)
-    destroy_droplet "${@:2}"
-    ;;
-
-transfer_image_to_all_regions)
-    transfer_image_to_all_regions "${@:2}"
-    ;;
-
-*)
-    echo "Unknown command $1"
-    exit 1
-esac

baseimage/initializeBaseUbuntuImage.sh

@@ -29,10 +29,12 @@ debconf-set-selections <<< 'mysql-server mysql-server/root_password_again passwo
 
 # this enables automatic security upgrades (https://help.ubuntu.com/community/AutomaticSecurityUpdates)
 # resolvconf is needed for unbound to work property after disabling systemd-resolved in 18.04
+
 gpg_package=$([[ "${ubuntu_version}" == "16.04" ]] && echo "gnupg" || echo "gpg")
 mysql_package=$([[ "${ubuntu_version}" == "20.04" ]] && echo "mysql-server-8.0" || echo "mysql-server-5.7")
-apt-get -y install \
+apt-get -y install --no-install-recommends \
     acl \
+    apparmor \
     build-essential \
     cifs-utils \
     cron \
@@ -53,27 +55,29 @@ apt-get -y install \
     tzdata \
     unattended-upgrades \
     unbound \
+    unzip \
     xfsprogs
 
 echo "==> installing nginx for xenial for TLSv3 support"
-curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
+curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-2~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
 # apt install with install deps (as opposed to dpkg -i)
 apt install -y /tmp/nginx.deb
 rm /tmp/nginx.deb
 
 # on some providers like scaleway the sudo file is changed and we want to keep the old one
-apt-get -o Dpkg::Options::="--force-confold" install -y sudo
+apt-get -o Dpkg::Options::="--force-confold" install -y --no-install-recommends sudo
 
 # this ensures that unattended upgades are enabled, if it was disabled during ubuntu install time (see #346)
 # debconf-set-selection of unattended-upgrades/enable_auto_updates + dpkg-reconfigure does not work
 cp /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades
 
 echo "==> Installing node.js"
-mkdir -p /usr/local/node-10.18.1
-curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxf - --strip-components=1 -C /usr/local/node-10.18.1
-ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
-ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
-apt-get install -y python   # Install python which is required for npm rebuild
+readonly node_version=14.15.4
+mkdir -p /usr/local/node-${node_version}
+curl -sL https://nodejs.org/dist/v${node_version}/node-v${node_version}-linux-x64.tar.gz | tar zxf - --strip-components=1 -C /usr/local/node-${node_version}
+ln -sf /usr/local/node-${node_version}/bin/node /usr/bin/node
+ln -sf /usr/local/node-${node_version}/bin/npm /usr/bin/npm
+apt-get install -y --no-install-recommends python   # Install python which is required for npm rebuild
 [[ "$(python --version 2>&1)" == "Python 2.7."* ]] || die "Expecting python version to be 2.7.x"
 
 # https://docs.docker.com/engine/installation/linux/ubuntulinux/
@@ -84,9 +88,10 @@ mkdir -p /etc/systemd/system/docker.service.d
 echo -e "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2" > /etc/systemd/system/docker.service.d/cloudron.conf
 
 # there are 3 packages for docker - containerd, CLI and the daemon
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.13-2_amd64.deb" -o /tmp/containerd.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+readonly docker_version=20.10.3
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb" -o /tmp/containerd.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
+curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
 # apt install with install deps (as opposed to dpkg -i)
 apt install -y /tmp/containerd.deb  /tmp/docker-ce-cli.deb /tmp/docker.deb
 rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
@@ -99,7 +104,7 @@ fi
 
 # do not upgrade grub because it might prompt user and break this script
 echo "==> Enable memory accounting"
-apt-get -y --no-upgrade install grub2-common
+apt-get -y --no-upgrade --no-install-recommends install grub2-common
 sed -e 's/^GRUB_CMDLINE_LINUX="\(.*\)"$/GRUB_CMDLINE_LINUX="\1 cgroup_enable=memory swapaccount=1 panic_on_oops=1 panic=5"/' -i /etc/default/grub
 update-grub
 
@@ -118,7 +123,9 @@ for image in ${images}; do
 done
 
 echo "==> Install collectd"
-if ! apt-get install -y libcurl3-gnutls collectd collectd-utils; then
+# without this, libnotify4 will install gnome-shell
+apt-get install -y libnotify4 --no-install-recommends
+if ! apt-get install -y --no-install-recommends libcurl3-gnutls collectd collectd-utils; then
     # FQDNLookup is true in default debian config. The box code has a custom collectd.conf that fixes this
     echo "Failed to install collectd. Presumably because of http://mailman.verplant.org/pipermail/collectd/2015-March/006491.html"
     sed -e 's/^FQDNLookup true/FQDNLookup false/' -i /etc/collectd/collectd.conf
@@ -126,8 +133,13 @@ fi
 # https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1872281
 [[ "${ubuntu_version}" == "20.04" ]] && echo -e "\nLD_PRELOAD=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" >> /etc/default/collectd
 
+# some hosts like atlantic install ntp which conflicts with timedatectl. https://serverfault.com/questions/1024770/ubuntu-20-04-time-sync-problems-and-possibly-incorrect-status-information
 echo "==> Configuring host"
 sed -e 's/^#NTP=/NTP=0.ubuntu.pool.ntp.org 1.ubuntu.pool.ntp.org 2.ubuntu.pool.ntp.org 3.ubuntu.pool.ntp.org/' -i /etc/systemd/timesyncd.conf
+if systemctl is-active ntp; then
+    systemctl stop ntp
+    apt purge -y ntp
+fi
 timedatectl set-ntp 1
 # mysql follows the system timezone
 timedatectl set-timezone UTC
@@ -141,7 +153,7 @@ if [ -f "/etc/default/motd-news" ]; then
     sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
 fi
 
-# Disable bind for good measure (on online.net, kimsufi servers these are pre-installed and conflicts with unbound)
+# Disable bind for good measure (on online.net, kimsufi servers these are pre-installed)
 systemctl stop bind9 || true
 systemctl disable bind9 || true
 
@@ -153,7 +165,7 @@ systemctl disable dnsmasq || true
 systemctl stop postfix || true
 systemctl disable postfix || true
 
-# on ubuntu 18.04, this is the default. this requires resolvconf for DNS to work further after the disable
+# on ubuntu 18.04 and 20.04, this is the default. this requires resolvconf for DNS to work further after the disable
 systemctl stop systemd-resolved || true
 systemctl disable systemd-resolved || true
 
@@ -162,4 +174,3 @@ systemctl disable systemd-resolved || true
 ip6=$([[ -s /proc/net/if_inet6 ]] && echo "yes" || echo "no")
 echo -e "server:\n\tinterface: 127.0.0.1\n\tdo-ip6: ${ip6}" > /etc/unbound/unbound.conf.d/cloudron-network.conf
 systemctl restart unbound
-

box.js

@@ -7,6 +7,7 @@ let async = require('async'),
     fs = require('fs'),
     ldap = require('./src/ldap.js'),
     paths = require('./src/paths.js'),
+    proxyAuth = require('./src/proxyauth.js'),
     server = require('./src/server.js');
 
 const NOOP_CALLBACK = function () { };
@@ -22,7 +23,8 @@ function setupLogging(callback) {
 
 async.series([
     setupLogging,
-    server.start,
+    server.start, // do this first since it also inits the database
+    proxyAuth.start,
     ldap.start,
     dockerProxy.start
 ], function (error) {
@@ -38,6 +40,7 @@ async.series([
     process.on('SIGINT', function () {
         debug('Received SIGINT. Shutting down.');
 
+        proxyAuth.stop(NOOP_CALLBACK);
         server.stop(NOOP_CALLBACK);
         ldap.stop(NOOP_CALLBACK);
         dockerProxy.stop(NOOP_CALLBACK);
@@ -47,6 +50,7 @@ async.series([
     process.on('SIGTERM', function () {
         debug('Received SIGTERM. Shutting down.');
 
+        proxyAuth.stop(NOOP_CALLBACK);
         server.stop(NOOP_CALLBACK);
         ldap.stop(NOOP_CALLBACK);
         dockerProxy.stop(NOOP_CALLBACK);

migrations/20201028052525-volumes-create-table.js

@@ -0,0 +1,40 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    var cmd1 = 'CREATE TABLE volumes(' +
+                'id VARCHAR(128) NOT NULL UNIQUE,' +
+                'name VARCHAR(256) NOT NULL UNIQUE,' +
+                'hostPath VARCHAR(1024) NOT NULL UNIQUE,' +
+                'creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,' +
+                'PRIMARY KEY (id)) CHARACTER SET utf8 COLLATE utf8_bin';
+
+    var cmd2 = 'CREATE TABLE appMounts(' +
+                 'appId VARCHAR(128) NOT NULL,' +
+                 'volumeId VARCHAR(128) NOT NULL,' +
+                 'readOnly BOOLEAN DEFAULT 1,' +
+                 'UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),' +
+                 'FOREIGN KEY(appId) REFERENCES apps(id),' +
+                 'FOREIGN KEY(volumeId) REFERENCES volumes(id)) CHARACTER SET utf8 COLLATE utf8_bin;';
+
+    db.runSql(cmd1, function (error) {
+        if (error) console.error(error);
+
+        db.runSql(cmd2, function (error) {
+            if (error) console.error(error);
+
+            db.runSql('ALTER TABLE apps DROP COLUMN bindsJson', callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('DROP TABLE appMounts', function (error) {
+        if (error) console.error(error);
+
+        db.runSql('DROP TABLE volumes', function (error) {
+            if (error) console.error(error);
+            callback(error);
+        });
+    });
+};
+

migrations/20201110174504-apps-add-proxyAuth.js

@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN proxyAuth BOOLEAN DEFAULT 0', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN proxyAuth', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20201113071728-mailboxes-add-ownerType.js

@@ -0,0 +1,18 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD COLUMN ownerType VARCHAR(16)'),
+        db.runSql.bind(db, 'UPDATE mailboxes SET ownerType=?', [ 'user' ]),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes MODIFY ownerType VARCHAR(16) NOT NULL'),
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes DROP COLUMN ownerType', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20201119072309-apps-remove-httpPort.js

@@ -0,0 +1,13 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE apps DROP COLUMN httpPort')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20201120212726-apps-add-containerIp.js

@@ -0,0 +1,29 @@
+'use strict';
+
+const async = require('async'),
+    iputils = require('../src/iputils.js');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN containerIp VARCHAR(16) UNIQUE', function (error) {
+        if (error) console.error(error);
+
+        let baseIp = iputils.intFromIp('172.18.16.0');
+
+        db.all('SELECT * FROM apps', function (error, apps) {
+            if (error) return callback(error);
+
+            async.eachSeries(apps, function (app, iteratorDone) {
+                const nextIp = iputils.ipFromInt(++baseIp);
+                db.runSql('UPDATE apps SET containerIp=? WHERE id=?', [ nextIp, app.id ], iteratorDone);
+            }, callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN containerIp', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20201204081504-settings-sftp-requireAdmin.js

@@ -0,0 +1,21 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM settings WHERE name=?', ['platform_config'], function (error, results) {
+        let value;
+        if (error || results.length === 0) {
+            value = { sftp: { requireAdmin: true } };
+        } else {
+            value = JSON.parse(results[0].value);
+            if (!value.sftp) value.sftp = {};
+            value.sftp.requireAdmin = true;
+        }
+
+        // existing installations may not even have the key. so use REPLACE instead of UPDATE
+        db.runSql('REPLACE INTO settings (name, value) VALUES (?, ?)', [ 'platform_config', JSON.stringify(value) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20201222182945-groupMembers-unique-members.js

@@ -0,0 +1,18 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'CREATE TABLE groupMembers_copy(groupId VARCHAR(128) NOT NULL, userId VARCHAR(128) NOT NULL, FOREIGN KEY(groupId) REFERENCES userGroups(id), FOREIGN KEY(userId) REFERENCES users(id), UNIQUE (groupId, userId)) CHARACTER SET utf8 COLLATE utf8_bin'),    // In mysql CREATE TABLE.. LIKE does not copy indexes
+        db.runSql.bind(db, 'INSERT INTO groupMembers_copy SELECT * FROM groupMembers GROUP BY groupId, userId'),
+        db.runSql.bind(db, 'DROP TABLE groupMembers'),
+        db.runSql.bind(db, 'ALTER TABLE groupMembers_copy RENAME TO groupMembers')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE groupMembers DROP INDEX groupMembers_member'),
+    ], callback);
+};

migrations/20201223014453-domains-add-wellKnownJson.js

@@ -0,0 +1,51 @@
+'use strict';
+
+const async = require('async'),
+    safe = require('safetydance');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE domains ADD COLUMN wellKnownJson TEXT', function (error) {
+        if (error) return callback(error);
+
+        // keep the paths around, so that we don't need to trigger a re-configure. the old nginx config will use the paths
+        // the new one will proxy calls to the box code
+        const WELLKNOWN_DIR = '/home/yellowtent/boxdata/well-known';
+        const output = safe.child_process.execSync('find . -type f -printf "%P\n"', { cwd: WELLKNOWN_DIR, encoding: 'utf8' });
+        if (!output) return callback();
+        const paths = output.trim().split('\n');
+        if (paths.length === 0) return callback(); // user didn't configure any well-known
+
+        let wellKnown = {};
+        for (let path of paths) {
+            const fqdn = path.split('/', 1)[0];
+            const loc = path.slice(fqdn.length+1);
+            const doc = safe.fs.readFileSync(`${WELLKNOWN_DIR}/${path}`, { encoding: 'utf8' });
+            if (!doc) continue;
+
+            wellKnown[fqdn] = {};
+            wellKnown[fqdn][loc] = doc;
+        }
+
+        console.log('Migrating well-known', JSON.stringify(wellKnown, null, 4));
+
+        async.eachSeries(Object.keys(wellKnown), function (fqdn, iteratorDone) {
+            db.runSql('UPDATE domains SET wellKnownJson=? WHERE domain=?', [ JSON.stringify(wellKnown[fqdn]), fqdn ], function (error, result) {
+                if (error) {
+                    console.error(error); // maybe the domain does not exist anymore
+                } else if (result.affectedRows === 0) {
+                    console.log(`Could not migrate wellknown as domain ${fqdn} is missing`);
+                }
+                iteratorDone();
+            });
+        }, function (error) {
+            callback(error);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE domains DROP COLUMN wellKnownJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20210120025243-settings-platform-config-memoryLimit.js

@@ -0,0 +1,23 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM settings WHERE name=?', ['platform_config'], function (error, results) {
+        if (error || results.length === 0) return callback(null);
+
+        let value = JSON.parse(results[0].value);
+
+        for (const serviceName of Object.keys(value)) {
+            const service = value[serviceName];
+            if (!service.memorySwap) continue;
+            service.memoryLimit = service.memorySwap;
+            delete service.memorySwap;
+            delete service.memory;
+        }
+
+        db.runSql('UPDATE settings SET value=? WHERE name=?', [ JSON.stringify(value), 'platform_config' ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210120025750-apps-servicesConfig-memoryLimit.js

@@ -0,0 +1,28 @@
+'use strict';
+
+const async = require('async');
+
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM apps', function (error, apps) {
+        if (error) return callback(error);
+
+        async.eachSeries(apps, function (app, iteratorDone) {
+            if (!app.servicesConfigJson) return iteratorDone();
+
+            let servicesConfig = JSON.parse(app.servicesConfigJson);
+            for (const serviceName of Object.keys(servicesConfig)) {
+                const service = servicesConfig[serviceName];
+                if (!service.memorySwap) continue;
+                service.memoryLimit = service.memorySwap;
+                delete service.memorySwap;
+                delete service.memory;
+            }
+
+            db.runSql('UPDATE apps SET servicesConfigJson=? WHERE id=?', [ JSON.stringify(servicesConfig), app.id ], iteratorDone);
+        }, callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210121201822-settings-rename-platform-config-to-services-config.js

@@ -0,0 +1,9 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('UPDATE settings SET name=? WHERE name=?', [ 'services_config', 'platform_config' ], callback);
+};
+
+exports.down = function(db, callback) {
+    db.runSql('UPDATE settings SET name=? WHERE name=?', [ 'platform_config', 'services_config' ], callback);
+};

migrations/20210204181904-settings-change-ovh-url.js

@@ -0,0 +1,10 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    /* this contained an invalid migration of OVH URLs from s3 subdomain to storage subdomain. See https://forum.cloudron.io/topic/4584/issue-with-backups-listings-and-saving-backup-config-in-6-2  */
+    callback();
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20210209223121-settings-add-registry-config-provider.js

@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="registry_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var registryConfig = JSON.parse(results[0].value);
+        if (!registryConfig.provider) registryConfig.provider = 'other';
+
+        db.runSql('UPDATE settings SET value=? WHERE name="registry_config"', [ JSON.stringify(registryConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/schema.sql

@@ -44,7 +44,8 @@ CREATE TABLE IF NOT EXISTS groupMembers(
     groupId VARCHAR(128) NOT NULL,
     userId VARCHAR(128) NOT NULL,
     FOREIGN KEY(groupId) REFERENCES userGroups(id),
-    FOREIGN KEY(userId) REFERENCES users(id));
+    FOREIGN KEY(userId) REFERENCES users(id),
+    UNIQUE (groupId, userId));
 
 CREATE TABLE IF NOT EXISTS tokens(
     id VARCHAR(128) NOT NULL UNIQUE,
@@ -65,7 +66,6 @@ CREATE TABLE IF NOT EXISTS apps(
     healthTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app last responded
     containerId VARCHAR(128),
     manifestJson TEXT,
-    httpPort INTEGER,                        // this is the nginx proxy port and not manifest.httpPort
     accessRestrictionJson TEXT, // { users: [ ], groups: [ ] }
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app was installed
     updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,   // when the last app update was done
@@ -85,8 +85,8 @@ CREATE TABLE IF NOT EXISTS apps(
     dataDir VARCHAR(256) UNIQUE,
     taskId INTEGER, // current task
     errorJson TEXT,
-    bindsJson TEXT, // bind mounts
     servicesConfigJson TEXT, // app services configuration
+    containerIp VARCHAR(16) UNIQUE, // this is not-null because of ip allocation fails, user can 'repair'
 
     FOREIGN KEY(mailboxDomain) REFERENCES domains(domain),
     FOREIGN KEY(taskId) REFERENCES tasks(id),
@@ -148,6 +148,7 @@ CREATE TABLE IF NOT EXISTS domains(
     provider VARCHAR(16) NOT NULL,
     configJson TEXT, /* JSON containing the dns backend provider config */
     tlsConfigJson TEXT, /* JSON containing the tls provider config */
+    wellKnownJson TEXT, /* JSON containing well known docs for this domain */
 
     PRIMARY KEY (domain))
 
@@ -181,6 +182,7 @@ CREATE TABLE IF NOT EXISTS mailboxes(
     name VARCHAR(128) NOT NULL,
     type VARCHAR(16) NOT NULL, /* 'mailbox', 'alias', 'list' */
     ownerId VARCHAR(128) NOT NULL, /* user id */
+    ownerType VARCHAR(16) NOT NULL,
     aliasName VARCHAR(128), /* the target name type is an alias */
     aliasDomain VARCHAR(128), /* the target domain */
     membersJson TEXT, /* members of a group. fully qualified */
@@ -237,4 +239,20 @@ CREATE TABLE IF NOT EXISTS appPasswords(
     PRIMARY KEY (id)
 );
 
+CREATE TABLE IF NOT EXISTS volumes(
+    id VARCHAR(128) NOT NULL UNIQUE,
+    name VARCHAR(256) NOT NULL UNIQUE,
+    hostPath VARCHAR(1024) NOT NULL UNIQUE,
+    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (id)
+);
+
+CREATE TABLE IF NOT EXISTS appMounts(
+    appId VARCHAR(128) NOT NULL,
+    volumeId VARCHAR(128) NOT NULL,
+    readOnly BOOLEAN DEFAULT 1,
+    UNIQUE KEY appMounts_appId_volumeId (appId, volumeId),
+    FOREIGN KEY(appId) REFERENCES apps(id),
+    FOREIGN KEY(volumeId) REFERENCES volumes(id));
+
 CHARACTER SET utf8 COLLATE utf8_bin;

package.json

@@ -10,76 +10,77 @@
     "type": "git",
     "url": "https://git.cloudron.io/cloudron/box.git"
   },
-  "engines": {
-    "node": ">=4.0.0 <=4.1.1"
-  },
   "dependencies": {
-    "@google-cloud/dns": "^1.2.9",
-    "@google-cloud/storage": "^2.5.0",
+    "@google-cloud/dns": "^2.1.0",
+    "@google-cloud/storage": "^5.8.0",
     "@sindresorhus/df": "git+https://github.com/cloudron-io/df.git#type",
-    "async": "^2.6.3",
-    "aws-sdk": "^2.759.0",
+    "async": "^3.2.0",
+    "aws-sdk": "^2.850.0",
+    "basic-auth": "^2.0.1",
     "body-parser": "^1.19.0",
-    "cloudron-manifestformat": "^5.6.0",
+    "cloudron-manifestformat": "^5.10.1",
     "connect": "^3.7.0",
     "connect-lastmile": "^2.0.0",
     "connect-timeout": "^1.9.0",
+    "cookie-parser": "^1.4.5",
     "cookie-session": "^1.4.0",
     "cron": "^1.8.2",
-    "db-migrate": "^0.11.11",
-    "db-migrate-mysql": "^2.1.1",
-    "debug": "^4.2.0",
-    "dockerode": "^2.5.8",
-    "ejs": "^2.6.1",
+    "db-migrate": "^0.11.12",
+    "db-migrate-mysql": "^2.1.2",
+    "debug": "^4.3.1",
+    "dockerode": "^3.2.1",
+    "ejs": "^3.1.6",
     "ejs-cli": "^2.2.1",
     "express": "^4.17.1",
     "ipaddr.js": "^2.0.0",
-    "js-yaml": "^3.14.0",
-    "json": "^9.0.6",
-    "ldapjs": "^2.2.0",
-    "lodash": "^4.17.20",
+    "js-yaml": "^4.0.0",
+    "json": "^10.0.0",
+    "jsonwebtoken": "^8.5.1",
+    "ldapjs": "^2.2.4",
+    "lodash": "^4.17.21",
     "lodash.chunk": "^4.2.0",
-    "mime": "^2.4.6",
-    "moment": "^2.29.0",
-    "moment-timezone": "^0.5.31",
+    "mime": "^2.5.2",
+    "moment": "^2.29.1",
+    "moment-timezone": "^0.5.33",
     "morgan": "^1.10.0",
     "multiparty": "^4.2.2",
+    "mustache-express": "^1.3.0",
     "mysql": "^2.18.1",
-    "nodemailer": "^6.4.11",
+    "nodemailer": "^6.4.18",
     "nodemailer-smtp-transport": "^2.7.4",
     "once": "^1.4.0",
-    "pretty-bytes": "^5.4.1",
+    "pretty-bytes": "^5.6.0",
     "progress-stream": "^2.0.0",
     "proxy-middleware": "^0.15.0",
     "qrcode": "^1.4.4",
-    "readdirp": "^3.4.0",
+    "readdirp": "^3.5.0",
     "request": "^2.88.2",
-    "rimraf": "^2.6.3",
+    "rimraf": "^3.0.2",
     "s3-block-read-stream": "^0.5.0",
     "safetydance": "^1.1.1",
-    "semver": "^6.1.1",
+    "semver": "^7.3.4",
     "showdown": "^1.9.1",
     "speakeasy": "^2.0.0",
     "split": "^1.0.1",
-    "superagent": "^5.3.1",
+    "superagent": "^6.1.0",
     "supererror": "^0.7.2",
     "tar-fs": "github:cloudron-io/tar-fs#ignore_stat_error",
-    "tar-stream": "^2.1.4",
+    "tar-stream": "^2.2.0",
     "tldjs": "^2.3.1",
-    "underscore": "^1.11.0",
-    "uuid": "^3.4.0",
-    "validator": "^11.0.0",
-    "ws": "^7.3.1",
+    "underscore": "^1.12.0",
+    "uuid": "^8.3.2",
+    "validator": "^13.5.2",
+    "ws": "^7.4.3",
     "xml2js": "^0.4.23"
   },
   "devDependencies": {
     "expect.js": "*",
     "hock": "^1.4.1",
     "js2xmlparser": "^4.0.1",
-    "mocha": "^6.2.3",
+    "mocha": "^8.3.0",
     "mock-aws-s3": "git+https://github.com/cloudron-io/mock-aws-s3.git",
-    "nock": "^10.0.6",
-    "node-sass": "^4.14.1",
+    "nock": "^13.0.7",
+    "node-sass": "^5.0.0",
     "recursive-readdir": "^2.2.2"
   },
   "scripts": {

runTests

@@ -2,11 +2,11 @@
 
 set -eu
 
-readonly SOURCE_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly source_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 readonly DATA_DIR="${HOME}/.cloudron_test"
 readonly DEFAULT_TESTS="./src/test/*-test.js ./src/routes/test/*-test.js"
 
-! "${SOURCE_dir}/src/test/checkInstall" && exit 1
+! "${source_dir}/src/test/checkInstall" && exit 1
 
 # cleanup old data dirs some of those docker container data requires sudo to be removed
 echo "=> Provide root password to purge any leftover data in ${DATA_DIR} and load apparmor profile:"
@@ -22,19 +22,27 @@ fi
 mkdir -p ${DATA_DIR}
 cd ${DATA_DIR}
 mkdir -p appsdata
-mkdir -p boxdata/profileicons boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com
-mkdir -p platformdata/addons/mail platformdata/nginx/cert platformdata/nginx/applications platformdata/collectd/collectd.conf.d platformdata/addons platformdata/logrotate.d platformdata/backup platformdata/logs/tasks
+mkdir -p boxdata/profileicons boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com boxdata/sftp/ssh
+mkdir -p platformdata/addons/mail/banner platformdata/nginx/cert platformdata/nginx/applications platformdata/collectd/collectd.conf.d platformdata/addons platformdata/logrotate.d platformdata/backup platformdata/logs/tasks
+sudo mkdir -p /mnt/cloudron-test-music /media/cloudron-test-music # volume test
+
+# translations
+mkdir -p box/dashboard/dist/translation
+cp -r ${source_dir}/../dashboard/dist/translation/* box/dashboard/dist/translation
 
 # put cert
 echo "=> Generating a localhost selfsigned cert"
 openssl req -x509 -newkey rsa:2048 -keyout platformdata/nginx/cert/host.key -out platformdata/nginx/cert/host.cert -days 3650 -subj '/CN=localhost' -nodes -config <(cat /etc/ssl/openssl.cnf <(printf "\n[SAN]\nsubjectAltName=DNS:*.localhost"))
 
+# generate legacy key format for sftp
+ssh-keygen -m PEM -t rsa -f boxdata/sftp/ssh/ssh_host_rsa_key -q -N ""
+
 # clear out any containers
 echo "=> Delete all docker containers first"
 docker ps -qa | xargs --no-run-if-empty docker rm -f
 
 # create docker network (while the infra code does this, most tests skip infra setup)
-docker network create --subnet=172.18.0.0/16 cloudron || true
+docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 cloudron || true
 
 # create the same mysql server version to test with
 OUT=`docker inspect mysql-server` || true
@@ -59,7 +67,7 @@ echo "=> Ensure database"
 mysql -h"${MYSQL_IP}" -uroot -ppassword -e 'CREATE DATABASE IF NOT EXISTS box'
 
 echo "=> Run database migrations"
-cd "${SOURCE_dir}"
+cd "${source_dir}"
 BOX_ENV=test DATABASE_URL=mysql://root:password@${MYSQL_IP}/box node_modules/.bin/db-migrate up
 
 echo "=> Run tests with mocha"

scripts/cloudron-setup

@@ -2,6 +2,12 @@
 
 set -eu -o pipefail
 
+function exitHandler() {
+    rm -f /etc/update-motd.d/91-cloudron-install-in-progress
+}
+
+trap exitHandler EXIT
+
 # change this to a hash when we make a upgrade release
 readonly LOG_FILE="/var/log/cloudron-setup.log"
 readonly MINIMUM_DISK_SIZE_GB="18" # this is the size of "/" and required to fit in docker images 18 is a safe bet for different reporting on 20GB min
@@ -43,12 +49,14 @@ fi
 initBaseImage="true"
 provider="generic"
 requestedVersion=""
+installServerOrigin="https://api.cloudron.io"
 apiServerOrigin="https://api.cloudron.io"
 webServerOrigin="https://cloudron.io"
 sourceTarballUrl=""
 rebootServer="true"
+setupToken=""
 
-args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot" -n "$0" -- "$@")
+args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot,generate-setup-token" -n "$0" -- "$@")
 eval set -- "${args}"
 
 while true; do
@@ -60,13 +68,18 @@ while true; do
         if [[ "$2" == "dev" ]]; then
             apiServerOrigin="https://api.dev.cloudron.io"
             webServerOrigin="https://dev.cloudron.io"
+            installServerOrigin="https://api.dev.cloudron.io"
         elif [[ "$2" == "staging" ]]; then
             apiServerOrigin="https://api.staging.cloudron.io"
             webServerOrigin="https://staging.cloudron.io"
+            installServerOrigin="https://api.staging.cloudron.io"
+        elif [[ "$2" == "unstable" ]]; then
+            installServerOrigin="https://api.dev.cloudron.io"
         fi
         shift 2;;
     --skip-baseimage-init) initBaseImage="false"; shift;;
     --skip-reboot) rebootServer="false"; shift;;
+    --generate-setup-token) setupToken="$(openssl rand -hex 10)"; shift;;
     --) break;;
     *) echo "Unknown option $1"; exit 1;;
     esac
@@ -85,6 +98,26 @@ if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" && "${ubu
     exit 1
 fi
 
+# Install MOTD file for stack script style installations. this is removed by the trap exit handler. Heredoc quotes prevents parameter expansion
+cat > /etc/update-motd.d/91-cloudron-install-in-progress <<'EOF'
+#!/bin/bash
+
+printf "**********************************************************************\n\n"
+
+printf "\t\t\tWELCOME TO CLOUDRON\n"
+printf "\t\t\t-------------------\n"
+
+printf '\n\e[1;32m%-6s\e[m\n\n' "Cloudron is installing. Run 'tail -f /var/log/cloudron-setup.log' to view progress."
+
+printf "Cloudron overview - https://docs.cloudron.io/ \n"
+printf "Cloudron setup - https://docs.cloudron.io/installation/#setup \n"
+
+printf "\nFor help and more information, visit https://forum.cloudron.io\n\n"
+
+printf "**********************************************************************\n"
+EOF
+chmod +x /etc/update-motd.d/91-cloudron-install-in-progress
+
 # Can only write after we have confirmed script has root access
 echo "Running cloudron-setup with args : $@" > "${LOG_FILE}"
 
@@ -100,26 +133,20 @@ echo " Join us at https://forum.cloudron.io for any questions."
 echo ""
 
 if [[ "${initBaseImage}" == "true" ]]; then
-    echo "=> Installing software-properties-common"
-    if ! apt-get install -y software-properties-common &>> "${LOG_FILE}"; then
-        echo "Could not install software-properties-common (for add-apt-repository below). See ${LOG_FILE}"
-        exit 1
-    fi
-
     echo "=> Updating apt and installing script dependencies"
     if ! apt-get update &>> "${LOG_FILE}"; then
         echo "Could not update package repositories. See ${LOG_FILE}"
         exit 1
     fi
 
-    if ! DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install curl python3 ubuntu-standard -y &>> "${LOG_FILE}"; then
+    if ! DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install --no-install-recommends curl python3 ubuntu-standard software-properties-common -y &>> "${LOG_FILE}"; then
         echo "Could not install setup dependencies (curl). See ${LOG_FILE}"
         exit 1
     fi
 fi
 
 echo "=> Checking version"
-if ! releaseJson=$($curl -s "${apiServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
+if ! releaseJson=$($curl -s "${installServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
     echo "Failed to get release information"
     exit 1
 fi
@@ -157,6 +184,7 @@ fi
 echo "=> Installing version ${version} (this takes some time) ..."
 mkdir -p /etc/cloudron
 echo "${provider}" > /etc/cloudron/PROVIDER
+[[ ! -z "${setupToken}" ]] && echo "${setupToken}" > /etc/cloudron/SETUP_TOKEN
 
 if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" &>> "${LOG_FILE}"; then
     echo "Failed to install cloudron. See ${LOG_FILE} for details"
@@ -178,7 +206,12 @@ done
 if ! ip=$(curl -s --fail --connect-timeout 2 --max-time 2 https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
     ip='<IP>'
 fi
-echo -e "\n\n${GREEN}Visit https://${ip} and accept the self-signed certificate to finish setup.${DONE}\n"
+if [[ -z "${setupToken}" ]]; then
+    url="https://${ip}"
+else
+    url="https://${ip}/?setupToken=${setupToken}"
+fi
+echo -e "\n\n${GREEN}After reboot, visit ${url} and accept the self-signed certificate to finish setup.${DONE}\n"
 
 if [[ "${rebootServer}" == "true" ]]; then
     systemctl stop box mysql # sometimes mysql ends up having corrupt privilege tables
@@ -186,7 +219,7 @@ if [[ "${rebootServer}" == "true" ]]; then
     read -p "The server has to be rebooted to apply all the settings. Reboot now ? [Y/n] " yn
     yn=${yn:-y}
     case $yn in
-        [Yy]* ) systemctl reboot;;
+        [Yy]* ) exitHandler; systemctl reboot;;
         * ) exit;;
     esac
 fi

scripts/cloudron-support

@@ -42,7 +42,7 @@ while true; do
         ghost_file=/home/yellowtent/platformdata/cloudron_ghost.json
         printf '{"%s":"%s"}\n' "${admin_username}" "${admin_password}" > "${ghost_file}"
         chown yellowtent:yellowtent "${ghost_file}" && chmod o-r,g-r "${ghost_file}"
-        echo "Login as ${admin_username} / ${admin_password} . Remove ${ghost_file} when done."
+        echo "Login as ${admin_username} / ${admin_password} . This password may only be used once. ${ghost_file} will be automatically removed after use."
         exit 0
         ;;
     --) break;;
@@ -73,6 +73,9 @@ echo -n "Generating Cloudron Support stats..."
 # clear file
 rm -rf $OUT
 
+echo -e $LINE"DASHBOARD DOMAIN"$LINE >> $OUT
+mysql -NB -uroot -ppassword -e "SELECT value FROM box.settings WHERE name='admin_fqdn'" &>> $OUT 2>/dev/null || true
+
 echo -e $LINE"PROVIDER"$LINE >> $OUT
 cat /etc/cloudron/PROVIDER &>> $OUT || true
 
@@ -99,7 +102,7 @@ systemctl status --lines=100 box mysql unbound cloudron-syslog nginx collectd do
 echo -e $LINE"Box logs"$LINE >> $OUT
 tail -n 100 /home/yellowtent/platformdata/logs/box.log &>> $OUT
 
-echo -e $LINE"Firewall chains"$LINE >> $OUT
+echo -e $LINE"Interface Info"$LINE >> $OUT
 ip addr &>> $OUT
 
 echo -e $LINE"Firewall chains"$LINE >> $OUT

scripts/cloudron-translation-update

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+set -eu -o pipefail
+
+# This script downloads new translation data from weblate at https://translate.cloudron.io
+
+OUT="/home/yellowtent/box/dashboard/dist/translation"
+
+# We require root
+if [[ ${EUID} -ne 0 ]]; then
+    echo "This script should be run as root. Run with sudo"
+    exit 1
+fi
+
+echo "=> Downloading new translation files..."
+curl https://translate.cloudron.io/download/cloudron/dashboard/?format=zip -o /tmp/lang.zip
+
+echo "=> Unpacking..."
+unzip -jo /tmp/lang.zip -d $OUT
+chown -R yellowtent:yellowtent $OUT
+# unzip put very restrictive permissions
+chmod ua+r $OUT/*
+
+echo "=> Cleanup..."
+rm /tmp/lang.zip
+
+echo "=> Done"
+
+echo ""
+echo "Reload the dashboard to see the new translations"
+echo ""

scripts/createReleaseTarball

@@ -41,8 +41,8 @@ if ! $(cd "${SOURCE_DIR}/../dashboard" && git diff --exit-code >/dev/null); then
     exit 1
 fi
 
-if [[ "$(node --version)" != "v10.18.1" ]]; then
-    echo "This script requires node 10.18.1"
+if [[ "$(node --version)" != "v14.15.4" ]]; then
+    echo "This script requires node 14.15.4"
     exit 1
 fi
 

scripts/installer.sh

@@ -11,6 +11,10 @@ if [[ ${EUID} -ne 0 ]]; then
     exit 1
 fi
 
+function log() {
+  echo -e "$(date +'%Y-%m-%dT%H:%M:%S')" "==> installer: $1"
+}
+
 readonly user=yellowtent
 readonly box_src_dir=/home/${user}/box
 
@@ -21,36 +25,37 @@ readonly box_src_tmp_dir="$(realpath ${script_dir}/..)"
 readonly ubuntu_version=$(lsb_release -rs)
 readonly ubuntu_codename=$(lsb_release -cs)
 
-readonly is_update=$(systemctl is-active box && echo "yes" || echo "no")
+readonly is_update=$(systemctl is-active -q box && echo "yes" || echo "no")
 
-echo "==> installer: Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src_tmp_dir/VERSION) <=="
+log "Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src_tmp_dir/VERSION)"
 
-echo "==> installer: updating docker"
+log "updating docker"
 
-if [[ $(docker version --format {{.Client.Version}}) != "19.03.12" ]]; then
+readonly docker_version=20.10.3
+if [[ $(docker version --format {{.Client.Version}}) != "${docker_version}" ]]; then
     # there are 3 packages for docker - containerd, CLI and the daemon
-    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.2.13-2_amd64.deb" -o /tmp/containerd.deb
-    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
-    curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_19.03.12~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
+    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb" -o /tmp/containerd.deb
+    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker-ce-cli.deb
+    $curl -sL "https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb" -o /tmp/docker.deb
 
-    echo "==> installer: Waiting for all dpkg tasks to finish..."
+    log "Waiting for all dpkg tasks to finish..."
     while fuser /var/lib/dpkg/lock; do
         sleep 1
     done
 
     while ! dpkg --force-confold --configure -a; do
-        echo "==> installer: Failed to fix packages. Retry"
+        log "Failed to fix packages. Retry"
         sleep 1
     done
 
     # the latest docker might need newer packages
     while ! apt update -y; do
-        echo "==> installer: Failed to update packages. Retry"
+        log "Failed to update packages. Retry"
         sleep 1
     done
 
     while ! apt install -y /tmp/containerd.deb  /tmp/docker-ce-cli.deb /tmp/docker.deb; do
-        echo "==> installer: Failed to install docker. Retry"
+        log "Failed to install docker. Retry"
         sleep 1
     done
 
@@ -59,25 +64,21 @@ fi
 
 readonly nginx_version=$(nginx -v 2>&1)
 if [[ "${nginx_version}" != *"1.18."* ]]; then
-    echo "==> installer: installing nginx 1.18"
-    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
+    log "installing nginx 1.18"
+    $curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-2~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
     # apt install with install deps (as opposed to dpkg -i)
     apt install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes /tmp/nginx.deb
     rm /tmp/nginx.deb
 fi
 
-if ! which ipset; then
-    echo "==> installer: installing ipset"
-    apt install -y ipset
-fi
-
-echo "==> installer: updating node"
-if [[ "$(node --version)" != "v10.18.1" ]]; then
-    mkdir -p /usr/local/node-10.18.1
-    $curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-10.18.1
-    ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
-    ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
-    rm -rf /usr/local/node-10.15.1
+log "updating node"
+readonly node_version=14.15.4
+if [[ "$(node --version)" != "v${node_version}" ]]; then
+    mkdir -p /usr/local/node-${node_version}
+    $curl -sL https://nodejs.org/dist/v${node_version}/node-v${node_version}-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-${node_version}
+    ln -sf /usr/local/node-${node_version}/bin/node /usr/bin/node
+    ln -sf /usr/local/node-${node_version}/bin/npm /usr/bin/npm
+    rm -rf /usr/local/node-10.18.1
 fi
 
 # this is here (and not in updater.js) because rebuild requires the above node
@@ -88,31 +89,31 @@ for try in `seq 1 10`; do
     # however by default npm drops privileges for npm rebuild
     # https://docs.npmjs.com/misc/config#unsafe-perm
     if cd "${box_src_tmp_dir}" && npm rebuild --unsafe-perm; then break; fi
-    echo "==> installer: Failed to rebuild, trying again"
+    log "Failed to rebuild, trying again"
     sleep 5
 done
 
 if [[ ${try} -eq 10 ]]; then
-    echo "==> installer: npm rebuild failed, giving up"
+    log "npm rebuild failed, giving up"
     exit 4
 fi
 
-echo "==> installer: downloading new addon images"
+log "downloading new addon images"
 images=$(node -e "var i = require('${box_src_tmp_dir}/src/infra_version.js'); console.log(i.baseImages.map(function (x) { return x.tag; }).join(' '), Object.keys(i.images).map(function (x) { return i.images[x].tag; }).join(' '));")
 
-echo -e "\tPulling docker images: ${images}"
+log "\tPulling docker images: ${images}"
 for image in ${images}; do
-    if ! docker pull "${image}"; then           # this pulls the image using the sha256
-        echo "==> installer: Could not pull ${image}"
-        exit 5
-    fi
-    if ! docker pull "${image%@sha256:*}"; then  # this will tag the image for readability
-        echo "==> installer: Could not pull ${image%@sha256:*}"
-        exit 6
-   fi
+    while ! docker pull "${image}"; do           # this pulls the image using the sha256
+        log "Could not pull ${image}"
+        sleep 5
+    done
+    while ! docker pull "${image%@sha256:*}"; do  # this will tag the image for readability
+        log "Could not pull ${image%@sha256:*}"
+        sleep 5
+   done
 done
 
-echo "==> installer: update cloudron-syslog"
+log "update cloudron-syslog"
 CLOUDRON_SYSLOG_DIR=/usr/local/cloudron-syslog
 CLOUDRON_SYSLOG="${CLOUDRON_SYSLOG_DIR}/bin/cloudron-syslog"
 CLOUDRON_SYSLOG_VERSION="1.0.3"
@@ -120,7 +121,7 @@ while [[ ! -f "${CLOUDRON_SYSLOG}" || "$(${CLOUDRON_SYSLOG} --version)" != ${CLO
     rm -rf "${CLOUDRON_SYSLOG_DIR}"
     mkdir -p "${CLOUDRON_SYSLOG_DIR}"
     if npm install --unsafe-perm -g --prefix "${CLOUDRON_SYSLOG_DIR}" cloudron-syslog@${CLOUDRON_SYSLOG_VERSION}; then break; fi
-    echo "===> installer: Failed to install cloudron-syslog, trying again"
+    log "Failed to install cloudron-syslog, trying again"
     sleep 5
 done
 
@@ -129,17 +130,17 @@ if ! id "${user}" 2>/dev/null; then
 fi
 
 if [[ "${is_update}" == "yes" ]]; then
-    echo "==> installer: stop box service for update"
+    log "stop box service for update"
     ${box_src_dir}/setup/stop.sh
 fi
 
 # ensure we are not inside the source directory, which we will remove now
 cd /root
 
-echo "==> installer: switching the box code"
+log "switching the box code"
 rm -rf "${box_src_dir}"
 mv "${box_src_tmp_dir}" "${box_src_dir}"
 chown -R "${user}:${user}" "${box_src_dir}"
 
-echo "==> installer: calling box setup script"
+log "calling box setup script"
 "${box_src_dir}/setup/start.sh"

setup/start.sh

@@ -5,7 +5,11 @@ set -eu -o pipefail
 # This script is run after the box code is switched. This means that this script
 # should pretty much always succeed. No network logic/download code here.
 
-echo "==> Cloudron Start"
+function log() {
+  echo -e "$(date +'%Y-%m-%dT%H:%M:%S')" "==> start: $1"
+}
+
+log "Cloudron Start"
 
 readonly USER="yellowtent"
 readonly HOME_DIR="/home/${USER}"
@@ -19,25 +23,27 @@ readonly json="$(realpath ${script_dir}/../node_modules/.bin/json)"
 readonly ubuntu_version=$(lsb_release -rs)
 
 cp -f "${script_dir}/../scripts/cloudron-support" /usr/bin/cloudron-support
+cp -f "${script_dir}/../scripts/cloudron-translation-update" /usr/bin/cloudron-translation-update
 
 # this needs to match the cloudron/base:2.0.0 gid
 if ! getent group media; then
     addgroup --gid 500 --system media
 fi
 
-echo "==> Configuring docker"
+log "Configuring docker"
 cp "${script_dir}/start/docker-cloudron-app.apparmor" /etc/apparmor.d/docker-cloudron-app
 systemctl enable apparmor
 systemctl restart apparmor
 
 usermod ${USER} -a -G docker
-docker network create --subnet=172.18.0.0/16 cloudron || true
+# unbound (which starts after box code) relies on this interface to exist. dockerproxy also relies on this.
+docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 cloudron || true
 
 mkdir -p "${BOX_DATA_DIR}"
 mkdir -p "${APPS_DATA_DIR}"
 
 # keep these in sync with paths.js
-echo "==> Ensuring directories"
+log "Ensuring directories"
 
 mkdir -p "${PLATFORM_DATA_DIR}/graphite"
 mkdir -p "${PLATFORM_DATA_DIR}/mysql"
@@ -63,12 +69,13 @@ mkdir -p "${BOX_DATA_DIR}/certs"
 mkdir -p "${BOX_DATA_DIR}/acme" # acme keys
 mkdir -p "${BOX_DATA_DIR}/mail/dkim"
 mkdir -p "${BOX_DATA_DIR}/well-known" # .well-known documents
+mkdir -p "${BOX_DATA_DIR}/sftp/ssh" # sftp keys
 
 # ensure backups folder exists and is writeable
 mkdir -p /var/backups
 chmod 777 /var/backups
 
-echo "==> Configuring journald"
+log "Configuring journald"
 sed -e "s/^#SystemMaxUse=.*$/SystemMaxUse=100M/" \
     -e "s/^#ForwardToSyslog=.*$/ForwardToSyslog=no/" \
     -i /etc/systemd/journald.conf
@@ -89,7 +96,7 @@ setfacl -n -m u:${USER}:r /var/log/journal/*/system.journal
 # Give user access to nginx logs (uses adm group)
 usermod -a -G adm ${USER}
 
-echo "==> Setting up unbound"
+log "Setting up unbound"
 # DO uses Google nameservers by default. This causes RBL queries to fail (host 2.0.0.127.zen.spamhaus.org)
 # We do not use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!)
 # We listen on 0.0.0.0 because there is no way control ordering of docker (which creates the 172.18.0.0/16) and unbound
@@ -99,16 +106,15 @@ cp -f "${script_dir}/start/unbound.conf" /etc/unbound/unbound.conf.d/cloudron-ne
 # update the root anchor after a out-of-disk-space situation (see #269)
 unbound-anchor -a /var/lib/unbound/root.key
 
-echo "==> Adding systemd services"
+log "Adding systemd services"
 cp -r "${script_dir}/start/systemd/." /etc/systemd/system/
-systemctl disable cloudron.target || true
-rm -f /etc/systemd/system/cloudron.target
 [[ "${ubuntu_version}" == "16.04" ]] && sed -e 's/MemoryMax/MemoryLimit/g' -i /etc/systemd/system/box.service
 systemctl daemon-reload
+systemctl enable --now cloudron-syslog
 systemctl enable unbound
-systemctl enable cloudron-syslog
 systemctl enable box
 systemctl enable cloudron-firewall
+systemctl enable --now cloudron-disable-thp
 
 # update firewall rules
 systemctl restart cloudron-firewall
@@ -122,11 +128,11 @@ systemctl restart unbound
 # ensure cloudron-syslog runs
 systemctl restart cloudron-syslog
 
-echo "==> Configuring sudoers"
+log "Configuring sudoers"
 rm -f /etc/sudoers.d/${USER}
 cp "${script_dir}/start/sudoers" /etc/sudoers.d/${USER}
 
-echo "==> Configuring collectd"
+log "Configuring collectd"
 rm -rf /etc/collectd /var/log/collectd.log
 ln -sfF "${PLATFORM_DATA_DIR}/collectd" /etc/collectd
 cp "${script_dir}/start/collectd/collectd.conf" "${PLATFORM_DATA_DIR}/collectd/collectd.conf"
@@ -138,7 +144,7 @@ if [[ "${ubuntu_version}" == "20.04" ]]; then
 fi
 systemctl restart collectd
 
-echo "==> Configuring logrotate"
+log "Configuring logrotate"
 if ! grep -q "^include ${PLATFORM_DATA_DIR}/logrotate.d" /etc/logrotate.conf; then
     echo -e "\ninclude ${PLATFORM_DATA_DIR}/logrotate.d\n" >> /etc/logrotate.conf
 fi
@@ -148,10 +154,10 @@ cp "${script_dir}/start/logrotate/"* "${PLATFORM_DATA_DIR}/logrotate.d/"
 # logrotate files have to be owned by root, this is here to fixup existing installations where we were resetting the owner to yellowtent
 chown root:root "${PLATFORM_DATA_DIR}/logrotate.d/"
 
-echo "==> Adding motd message for admins"
+log "Adding motd message for admins"
 cp "${script_dir}/start/cloudron-motd" /etc/update-motd.d/92-cloudron
 
-echo "==> Configuring nginx"
+log "Configuring nginx"
 # link nginx config to system config
 unlink /etc/nginx 2>/dev/null || rm -rf /etc/nginx
 ln -s "${PLATFORM_DATA_DIR}/nginx" /etc/nginx
@@ -179,18 +185,26 @@ if [[ ! -f /etc/mysql/mysql.cnf ]] || ! diff -q "${script_dir}/start/mysql.cnf"
     cp "${script_dir}/start/mysql.cnf" /etc/mysql/mysql.cnf
     while true; do
         if ! systemctl list-jobs | grep mysql; then break; fi
-        echo "Waiting for mysql jobs..."
+        log "Waiting for mysql jobs..."
         sleep 1
     done
-    while true; do
-        if systemctl restart mysql; then break; fi
-        echo "Restarting MySql again after sometime since this fails randomly"
+    log "Stopping mysql"
+    systemctl stop mysql
+    while mysqladmin ping 2>/dev/null; do
+        log "Waiting for mysql to stop..."
         sleep 1
     done
-else
-    systemctl start mysql
 fi
 
+# the start/stop of mysql is separate to make sure it got reloaded with latest config and it's up and running before we start the new box code
+# when using 'system restart mysql', it seems to restart much later and the box code loses connection during platform startup (dangerous!)
+log "Starting mysql"
+systemctl start mysql
+while ! mysqladmin ping 2>/dev/null; do
+    log "Waiting for mysql to start..."
+    sleep 1
+done
+
 readonly mysql_root_password="password"
 mysqladmin -u root -ppassword password password # reset default root password
 if [[ "${ubuntu_version}" == "20.04" ]]; then
@@ -201,31 +215,34 @@ mysql -u root -p${mysql_root_password} -e 'CREATE DATABASE IF NOT EXISTS box'
 
 # set HOME explicity, because it's not set when the installer calls it. this is done because
 # paths.js uses this env var and some of the migrate code requires box code
-echo "==> Migrating data"
+log "Migrating data"
 cd "${BOX_SRC_DIR}"
 if ! HOME=${HOME_DIR} BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up; then
-    echo "DB migration failed"
+    log "DB migration failed"
     exit 1
 fi
 
 rm -f /etc/cloudron/cloudron.conf
 
 if [[ ! -f "${BOX_DATA_DIR}/dhparams.pem" ]]; then
-    echo "==> Generating dhparams (takes forever)"
+    log "Generating dhparams (takes forever)"
     openssl dhparam -out "${BOX_DATA_DIR}/dhparams.pem" 2048
     cp "${BOX_DATA_DIR}/dhparams.pem" "${PLATFORM_DATA_DIR}/addons/mail/dhparams.pem"
 else
     cp "${BOX_DATA_DIR}/dhparams.pem" "${PLATFORM_DATA_DIR}/addons/mail/dhparams.pem"
 fi
 
-# old installations used to create appdata/<app>/redis which is now part of old backups and prevents restore
-echo "==> Cleaning up stale redis directories"
-find "${APPS_DATA_DIR}" -maxdepth 2 -type d -name redis -exec rm -rf {} +
+if [[ ! -f "${BOX_DATA_DIR}/sftp/ssh/ssh_host_rsa_key" ]]; then
+    # the key format in Ubuntu 20 changed, so we create keys in legacy format. for older ubuntu, just re-use the host keys
+    # see https://github.com/proftpd/proftpd/issues/793
+    if [[ "${ubuntu_version}" == "20.04" ]]; then
+        ssh-keygen -m PEM -t rsa -f "${BOX_DATA_DIR}/sftp/ssh/ssh_host_rsa_key" -q -N ""
+    else
+        cp /etc/ssh/ssh_host_rsa_key* ${BOX_DATA_DIR}/sftp/ssh
+    fi
+fi
 
-echo "==> Cleaning up old logs"
-rm -f /home/yellowtent/platformdata/logs/*/*.log.* || true
-
-echo "==> Changing ownership"
+log "Changing ownership"
 # be careful of what is chown'ed here. subdirs like mysql,redis etc are owned by the containers and will stop working if perms change
 chown -R "${USER}" /etc/cloudron
 chown "${USER}:${USER}" -R "${PLATFORM_DATA_DIR}/nginx" "${PLATFORM_DATA_DIR}/collectd" "${PLATFORM_DATA_DIR}/addons" "${PLATFORM_DATA_DIR}/acme" "${PLATFORM_DATA_DIR}/backup" "${PLATFORM_DATA_DIR}/logs" "${PLATFORM_DATA_DIR}/update"
@@ -239,9 +256,9 @@ find "${BOX_DATA_DIR}" -mindepth 1 -maxdepth 1 -not -path "${BOX_DATA_DIR}/mail"
 chown "${USER}:${USER}" "${BOX_DATA_DIR}/mail"
 chown "${USER}:${USER}" -R "${BOX_DATA_DIR}/mail/dkim" # this is owned by box currently since it generates the keys
 
-echo "==> Starting Cloudron"
+log "Starting Cloudron"
 systemctl start box
 
 sleep 2 # give systemd sometime to start the processes
 
-echo "==> Almost done"
+log "Almost done"

setup/start/cloudron-disable-thp.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+set -eu
+
+echo "==> Disabling THP"
+
+# https://docs.couchbase.com/server/current/install/thp-disable.html
+if [[ -d /sys/kernel/mm/transparent_hugepage ]]; then
+    echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
+    echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
+else
+    echo "==> kernel does not have THP"
+fi
+

setup/start/cloudron-firewall.sh

@@ -20,10 +20,20 @@ fi
 iptables -t filter -A CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
 iptables -t filter -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443 -j ACCEPT # 202 is the alternate ssh port
 
-# whitelist any user ports
+# whitelist any user ports. we used to use --dports but it has a 15 port limit (XT_MULTI_PORTS)
 ports_json="/home/yellowtent/boxdata/firewall/ports.json"
 if allowed_tcp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_tcp_ports.join(','))" 2>/dev/null); then
-    [[ -n "${allowed_tcp_ports}" ]] && iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports "${allowed_tcp_ports}" -j ACCEPT
+    IFS=',' arr=(${allowed_tcp_ports})
+    for p in "${arr[@]}"; do
+        iptables -A CLOUDRON -p tcp -m tcp --dport "${p}" -j ACCEPT
+    done
+fi
+
+if allowed_udp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_udp_ports.join(','))" 2>/dev/null); then
+    IFS=',' arr=(${allowed_udp_ports})
+    for p in "${arr[@]}"; do
+        iptables -A CLOUDRON -p udp -m udp --dport "${p}" -j ACCEPT
+    done
 fi
 
 # turn and stun service
@@ -88,3 +98,5 @@ fi
 # Workaround issue where Docker insists on adding itself first in FORWARD table
 iptables -D FORWARD -j CLOUDRON_RATELIMIT || true
 iptables -I FORWARD 1 -j CLOUDRON_RATELIMIT
+
+echo "==> Setting up firewall done"

setup/start/cloudron-motd

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+[[ -f /etc/update-motd.d/91-cloudron-install-in-progress ]] && exit
+
 printf "**********************************************************************\n\n"
 
 if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
@@ -10,10 +12,17 @@ if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
     fi
     echo "${ip}" > /tmp/.cloudron-motd-cache
 
+    if [[ ! -f /etc/cloudron/SETUP_TOKEN ]]; then
+        url="https://${ip}"
+    else
+        setupToken="$(cat /etc/cloudron/SETUP_TOKEN)"
+        url="https://${ip}/?setupToken=${setupToken}"
+    fi
+
     printf "\t\t\tWELCOME TO CLOUDRON\n"
     printf "\t\t\t-------------------\n"
 
-    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://${ip} on your browser and accept the self-signed certificate to finish setup."
+    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit ${url} on your browser and accept the self-signed certificate to finish setup."
     printf "Cloudron overview - https://docs.cloudron.io/ \n"
     printf "Cloudron setup - https://docs.cloudron.io/installation/#setup \n"
 else

setup/start/collectd/df.py

@@ -6,7 +6,7 @@ disks = []
 
 def init():
     global disks
-    lines = [s.split() for s in subprocess.check_output(["df", "--type=ext4", "--output=source,target,size,used,avail"]).splitlines()]
+    lines = [s.split() for s in subprocess.check_output(["df", "--type=ext4", "--output=source,target,size,used,avail"]).decode('utf-8').splitlines()]
     disks = lines[1:] # strip header
     collectd.info('custom df plugin initialized with %s' % disks)
 

setup/start/mysql.cnf

@@ -6,7 +6,7 @@
 performance_schema=OFF
 max_connections=50
 # on ec2, without this we get a sporadic connection drop when doing the initial migration
-max_allowed_packet=32M
+max_allowed_packet=64M
 
 # https://mathiasbynens.be/notes/mysql-utf8mb4
 character-set-server = utf8mb4
@@ -15,6 +15,9 @@ collation-server = utf8mb4_unicode_ci
 # set timezone to UTC
 default_time_zone='+00:00'
 
+# disable bin logs. they are only useful in replication mode
+skip-log-bin
+
 [mysqldump]
 quick
 quote-names

setup/start/sudoers

@@ -25,9 +25,6 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/configurecollec
 Defaults!/home/yellowtent/box/src/scripts/collectlogs.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/collectlogs.sh
 
-Defaults!/home/yellowtent/box/src/scripts/retire.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/retire.sh
-
 Defaults!/home/yellowtent/box/src/scripts/update.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/update.sh
 

setup/start/systemd/box.service

@@ -1,8 +1,6 @@
 [Unit]
 Description=Cloudron Admin
 OnFailure=crashnotifier@%n.service
-; journald crashes result in a EPIPE in node. Cannot ignore it as it results in loss of logs.
-BindsTo=systemd-journald.service
 After=mysql.service nginx.service
 ; As cloudron-resize-fs is a one-shot, the Wants= automatically ensures that the service *finishes*
 Wants=cloudron-resize-fs.service

setup/start/systemd/cloudron-disable-thp.service

@@ -0,0 +1,15 @@
+# https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
+[Unit]
+Description=Disable Transparent Huge Pages (THP)
+DefaultDependencies=no
+After=sysinit.target local-fs.target
+Before=docker.service
+
+[Service]
+Type=oneshot
+ExecStart="/home/yellowtent/box/setup/start/cloudron-disable-thp.sh"
+RemainAfterExit=yes
+
+[Install]
+WantedBy=basic.target
+

setup/start/systemd/unbound.service

@@ -2,7 +2,7 @@
 
 [Unit]
 Description=Unbound DNS Resolver
-After=network.target
+After=network.target docker.service
 
 [Service]
 PIDFile=/run/unbound.pid

setup/start/unbound.conf

@@ -1,5 +1,7 @@
 server:
-        interface: 0.0.0.0
+        port: 53
+        interface: 127.0.0.1
+        interface: 172.18.0.1
         do-ip6: no
         access-control: 127.0.0.1 allow
         access-control: 172.18.0.1/16 allow

src/appdb.js

@@ -1,32 +1,32 @@
 'use strict';
 
 exports = module.exports = {
-    get: get,
-    getByHttpPort: getByHttpPort,
-    getByContainerId: getByContainerId,
-    add: add,
-    exists: exists,
-    del: del,
-    update: update,
-    getAll: getAll,
-    getPortBindings: getPortBindings,
-    delPortBinding: delPortBinding,
+    get,
+    add,
+    exists,
+    del,
+    update,
+    getAll,
+    getPortBindings,
+    delPortBinding,
 
-    setAddonConfig: setAddonConfig,
-    getAddonConfig: getAddonConfig,
-    getAddonConfigByAppId: getAddonConfigByAppId,
-    getAddonConfigByName: getAddonConfigByName,
-    unsetAddonConfig: unsetAddonConfig,
-    unsetAddonConfigByAppId: unsetAddonConfigByAppId,
-    getAppIdByAddonConfigValue: getAppIdByAddonConfigValue,
+    setAddonConfig,
+    getAddonConfig,
+    getAddonConfigByAppId,
+    getAddonConfigByName,
+    unsetAddonConfig,
+    unsetAddonConfigByAppId,
+    getAppIdByAddonConfigValue,
+    getByIpAddress,
 
-    setHealth: setHealth,
-    setTask: setTask,
-    getAppStoreIds: getAppStoreIds,
+    setHealth,
+    setTask,
+    getAppStoreIds,
 
     // subdomain table types
     SUBDOMAIN_TYPE_PRIMARY: 'primary',
     SUBDOMAIN_TYPE_REDIRECT: 'redirect',
+    SUBDOMAIN_TYPE_ALIAS: 'alias',
 
     _clear: clear
 };
@@ -38,17 +38,14 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
-    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'subdomains.subdomain AS location', 'subdomains.domain',
-    'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
-    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson', 'apps.bindsJson',
-    'apps.sso', 'apps.debugModeJson', 'apps.enableBackup',
+const APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
+    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
+    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson',
+    'apps.sso', 'apps.debugModeJson', 'apps.enableBackup', 'apps.proxyAuth', 'apps.containerIp',
     'apps.creationTime', 'apps.updateTime', 'apps.mailboxName', 'apps.mailboxDomain', 'apps.enableAutomaticUpdate',
     'apps.dataDir', 'apps.ts', 'apps.healthTime' ].join(',');
 
-var PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');
-
-const SUBDOMAIN_FIELDS = [ 'appId', 'domain', 'subdomain', 'type' ].join(',');
+const PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');
 
 function postProcess(result) {
     assert.strictEqual(typeof result, 'object');
@@ -89,6 +86,7 @@ function postProcess(result) {
     result.sso = !!result.sso; // make it bool
     result.enableBackup = !!result.enableBackup; // make it bool
     result.enableAutomaticUpdate = !!result.enableAutomaticUpdate; // make it bool
+    result.proxyAuth = !!result.proxyAuth;
 
     assert(result.debugModeJson === null || typeof result.debugModeJson === 'string');
     result.debugMode = safe.JSON.parse(result.debugModeJson);
@@ -98,15 +96,23 @@ function postProcess(result) {
     result.servicesConfig = safe.JSON.parse(result.servicesConfigJson) || {};
     delete result.servicesConfigJson;
 
-    assert(result.bindsJson === null || typeof result.bindsJson === 'string');
-    result.binds = safe.JSON.parse(result.bindsJson) || {};
-    delete result.bindsJson;
+    let subdomains = JSON.parse(result.subdomains), domains = JSON.parse(result.domains), subdomainTypes = JSON.parse(result.subdomainTypes);
+    delete result.subdomains;
+    delete result.domains;
+    delete result.subdomainTypes;
 
-    result.alternateDomains = result.alternateDomains || [];
-    result.alternateDomains.forEach(function (d) {
-        delete d.appId;
-        delete d.type;
-    });
+    result.alternateDomains = [];
+    result.aliasDomains = [];
+    for (let i = 0; i < subdomainTypes.length; i++) {
+        if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_PRIMARY) {
+            result.location = subdomains[i];
+            result.domain = domains[i];
+        } else if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_REDIRECT) {
+            result.alternateDomains.push({ domain: domains[i], subdomain: subdomains[i] });
+        } else if (subdomainTypes[i] === exports.SUBDOMAIN_TYPE_ALIAS) {
+            result.aliasDomains.push({ domain: domains[i], subdomain: subdomains[i] });
+        }
+    }
 
     let envNames = JSON.parse(result.envNames), envValues = JSON.parse(result.envValues);
     delete result.envNames;
@@ -116,119 +122,67 @@ function postProcess(result) {
         if (envNames[i]) result.env[envNames[i]] = envValues[i];
     }
 
+    let volumeIds = JSON.parse(result.volumeIds);
+    delete result.volumeIds;
+    let volumeReadOnlys = JSON.parse(result.volumeReadOnlys);
+    delete result.volumeReadOnlys;
+
+    result.mounts = volumeIds[0] === null ? [] : volumeIds.map((v, idx) => { return { volumeId: v, readOnly: !!volumeReadOnlys[idx] }; }); // NOTE: volumeIds is [null] when volumes of an app is empty
+
     result.error = safe.JSON.parse(result.errorJson);
     delete result.errorJson;
 
     result.taskId = result.taskId ? String(result.taskId) : null;
 }
 
+// each query simply join apps table with another table by id. we then join the full result together
+const PB_QUERY = 'SELECT id, GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes FROM apps LEFT JOIN appPortBindings ON apps.id = appPortBindings.appId GROUP BY apps.id';
+const ENV_QUERY = 'SELECT id, JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues FROM apps LEFT JOIN appEnvVars ON apps.id = appEnvVars.appId GROUP BY apps.id';
+const SUBDOMAIN_QUERY = 'SELECT id, JSON_ARRAYAGG(subdomains.subdomain) AS subdomains, JSON_ARRAYAGG(subdomains.domain) AS domains, JSON_ARRAYAGG(subdomains.type) AS subdomainTypes FROM apps LEFT JOIN subdomains ON apps.id = subdomains.appId GROUP BY apps.id';
+const MOUNTS_QUERY = 'SELECT id, JSON_ARRAYAGG(appMounts.volumeId) AS volumeIds, JSON_ARRAYAGG(appMounts.readOnly) AS volumeReadOnlys FROM apps LEFT JOIN appMounts ON apps.id = appMounts.appId GROUP BY apps.id';
+const APPS_QUERY = `SELECT ${APPS_FIELDS_PREFIXED}, hostPorts, environmentVariables, portTypes, envNames, envValues, subdomains, domains, subdomainTypes, volumeIds, volumeReadOnlys FROM apps`
+    + ` LEFT JOIN (${PB_QUERY}) AS q1 on q1.id = apps.id`
+    + ` LEFT JOIN (${ENV_QUERY}) AS q2 on q2.id = apps.id`
+    + ` LEFT JOIN (${SUBDOMAIN_QUERY}) AS q3 on q3.id = apps.id`
+    + ` LEFT JOIN (${MOUNTS_QUERY}) AS q4 on q4.id = apps.id`;
+
 function get(id, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes, '
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE apps.id = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, id ], function (error, result) {
+    database.query(`${APPS_QUERY} WHERE apps.id = ?`, [ id ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        postProcess(result[0]);
 
-            result[0].alternateDomains = alternateDomains;
-
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
+        callback(null, result[0]);
     });
 }
 
-function getByHttpPort(httpPort, callback) {
-    assert.strictEqual(typeof httpPort, 'number');
+function getByIpAddress(ip, callback) {
+    assert.strictEqual(typeof ip, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-    + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-    + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE httpPort = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, httpPort ], function (error, result) {
+    database.query(`${APPS_QUERY} WHERE apps.containerIp = ?`, [ ip ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        postProcess(result[0]);
 
-            result[0].alternateDomains = alternateDomains;
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
-    });
-}
-
-function getByContainerId(containerId, callback) {
-    assert.strictEqual(typeof containerId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' WHERE containerId = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, containerId ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
-
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-            result[0].alternateDomains = alternateDomains;
-            postProcess(result[0]);
-
-            callback(null, result[0]);
-        });
+        callback(null, result[0]);
     });
 }
 
 function getAll(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
-        + ' FROM apps'
-        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
-        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
-        + ' GROUP BY apps.id ORDER BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY ], function (error, results) {
+    database.query(`${APPS_QUERY} ORDER BY apps.id`, [ ], function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE type = ?', [ exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        results.forEach(postProcess);
 
-            alternateDomains.forEach(function (d) {
-                var domain = results.find(function (a) { return d.appId === a.id; });
-                if (!domain) return;
-
-                domain.alternateDomains = domain.alternateDomains || [];
-                domain.alternateDomains.push(d);
-            });
-
-            results.forEach(postProcess);
-
-            callback(null, results);
-        });
+        callback(null, results);
     });
 }
 
@@ -300,6 +254,15 @@ function add(id, appStoreId, manifest, location, domain, portBindings, data, cal
         });
     }
 
+    if (data.aliasDomains) {
+        data.aliasDomains.forEach(function (d) {
+            queries.push({
+                query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)',
+                args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_ALIAS ]
+            });
+        });
+    }
+
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, 'no such domain'));
@@ -358,12 +321,13 @@ function del(id, callback) {
         { query: 'DELETE FROM appPortBindings WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appEnvVars WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM appPasswords WHERE identifier = ?', args: [ id ] },
+        { query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ] },
         { query: 'DELETE FROM apps WHERE id = ?', args: [ id ] }
     ];
 
     database.transaction(queries, function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results[4].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (results[5].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
 
         callback(null);
     });
@@ -396,6 +360,7 @@ function updateWithConstraints(id, app, constraints, callback) {
     assert(!('portBindings' in app) || typeof app.portBindings === 'object');
     assert(!('accessRestriction' in app) || typeof app.accessRestriction === 'object' || app.accessRestriction === '');
     assert(!('alternateDomains' in app) || Array.isArray(app.alternateDomains));
+    assert(!('aliasDomains' in app) || Array.isArray(app.aliasDomains));
     assert(!('tags' in app) || Array.isArray(app.tags));
     assert(!('env' in app) || typeof app.env === 'object');
 
@@ -423,22 +388,35 @@ function updateWithConstraints(id, app, constraints, callback) {
     }
 
     if ('location' in app && 'domain' in app) { // must be updated together as they are unique together
-        queries.push({ query: 'UPDATE subdomains SET subdomain = ?, domain = ? WHERE appId = ? AND type = ?', args: [ app.location, app.domain, id, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ?', args: [ id ]}); // all locations of an app must be updated together
+        queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, app.domain, app.location, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+
+        if ('alternateDomains' in app) {
+            app.alternateDomains.forEach(function (d) {
+                queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+            });
+        }
+
+        if ('aliasDomains' in app) {
+            app.aliasDomains.forEach(function (d) {
+                queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_ALIAS ]});
+            });
+        }
     }
 
-    if ('alternateDomains' in app) {
-        queries.push({ query: 'DELETE FROM subdomains WHERE appId = ? AND type = ?', args: [ id, exports.SUBDOMAIN_TYPE_REDIRECT ]});
-        app.alternateDomains.forEach(function (d) {
-            queries.push({ query: 'INSERT INTO subdomains (appId, domain, subdomain, type) VALUES (?, ?, ?, ?)', args: [ id, d.domain, d.subdomain, exports.SUBDOMAIN_TYPE_REDIRECT ]});
+    if ('mounts' in app) {
+        queries.push({ query: 'DELETE FROM appMounts WHERE appId = ?', args: [ id ]});
+        app.mounts.forEach(function (m) {
+            queries.push({ query: 'INSERT INTO appMounts (appId, volumeId, readOnly) VALUES (?, ?, ?)', args: [ id, m.volumeId, m.readOnly ]});
         });
     }
 
     var fields = [ ], values = [ ];
     for (var p in app) {
-        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig' || p === 'binds') {
+        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig') {
             fields.push(`${p}Json = ?`);
             values.push(JSON.stringify(app[p]));
-        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env') {
+        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'aliasDomains' && p !== 'env' && p !== 'mounts') {
             fields.push(p + ' = ?');
             values.push(app[p]);
         }

src/apphealthmonitor.js

@@ -10,49 +10,48 @@ var appdb = require('./appdb.js'),
     docker = require('./docker.js'),
     eventlog = require('./eventlog.js'),
     safe = require('safetydance'),
-    superagent = require('superagent'),
-    util = require('util');
+    superagent = require('superagent');
 
 exports = module.exports = {
-    run: run
+    run
 };
 
 const HEALTHCHECK_INTERVAL = 10 * 1000; // every 10 seconds. this needs to be small since the UI makes only healthy apps clickable
-const UNHEALTHY_THRESHOLD = 10 * 60 * 1000; // 10 minutes
+const UNHEALTHY_THRESHOLD = 20 * 60 * 1000; // 20 minutes
 
 const OOM_EVENT_LIMIT = 60 * 60 * 1000; // 60 minutes
+let gStartTime = null; // time when apphealthmonitor was started
 let gLastOomMailTime = Date.now() - (5 * 60 * 1000); // pretend we sent email 5 minutes ago
 
-function debugApp(app) {
-    assert(typeof app === 'object');
-
-    debug(app.fqdn + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)) + ' - ' + app.id);
-}
-
 function setHealth(app, health, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof health, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    let now = new Date(), healthTime = app.healthTime, curHealth = app.health;
+    // app starts out with null health
+    // if it became healthy, we update immediately. this is required for ui to say "running" etc
+    // if it became unhealthy/error/dead, wait for a threshold before updating db
+
+    const now = new Date(), lastHealth = app.health;
+    let healthTime = gStartTime > app.healthTime ? gStartTime : app.healthTime; // on box restart, clamp value to start time
 
     if (health === apps.HEALTH_HEALTHY) {
         healthTime = now;
-        if (curHealth && curHealth !== apps.HEALTH_HEALTHY) { // app starts out with null health
-            debugApp(app, 'app switched from %s to healthy', curHealth);
+        if (lastHealth && lastHealth !== apps.HEALTH_HEALTHY) { // app starts out with null health
+            debug(`setHealth: ${app.id} (${app.fqdn}) switched from ${lastHealth} to healthy`);
 
             // do not send mails for dev apps
             if (!app.debugMode) eventlog.add(eventlog.ACTION_APP_UP, auditSource.HEALTH_MONITOR, { app: app });
         }
     } else if (Math.abs(now - healthTime) > UNHEALTHY_THRESHOLD) {
-        if (curHealth === apps.HEALTH_HEALTHY) {
-            debugApp(app, 'marking as unhealthy since not seen for more than %s minutes', UNHEALTHY_THRESHOLD/(60 * 1000));
+        if (lastHealth === apps.HEALTH_HEALTHY) {
+            debug(`setHealth: marking ${app.id} (${app.fqdn}) as unhealthy since not seen for more than ${UNHEALTHY_THRESHOLD/(60 * 1000)} minutes`);
 
             // do not send mails for dev apps
             if (!app.debugMode) eventlog.add(eventlog.ACTION_APP_DOWN, auditSource.HEALTH_MONITOR, { app: app });
         }
     } else {
-        debugApp(app, 'waiting for %s seconds to update the app health', (UNHEALTHY_THRESHOLD - Math.abs(now - healthTime))/1000);
+        debug(`setHealth: ${app.id} (${app.fqdn}) waiting for ${(UNHEALTHY_THRESHOLD - Math.abs(now - healthTime))/1000} to update health`);
         return callback(null);
     }
 
@@ -61,6 +60,7 @@ function setHealth(app, health, callback) {
         if (error) return callback(error);
 
         app.health = health;
+        app.healthTime = healthTime;
 
         callback(null);
     });
@@ -85,8 +85,7 @@ function checkAppHealth(app, callback) {
         // non-appstore apps may not have healthCheckPath
         if (!manifest.healthCheckPath) return setHealth(app, apps.HEALTH_HEALTHY, callback);
 
-        // poll through docker network instead of nginx to bypass any potential oauth proxy
-        var healthCheckUrl = 'http://127.0.0.1:' + app.httpPort + manifest.healthCheckPath;
+        const healthCheckUrl = `http://${app.containerIp}:${manifest.httpPort}${manifest.healthCheckPath}`;
         superagent
             .get(healthCheckUrl)
             .set('Host', app.fqdn) // required for some apache configs with rewrite rules
@@ -96,7 +95,7 @@ function checkAppHealth(app, callback) {
             .end(function (error, res) {
                 if (error && !error.response) {
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
-                } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
+                } else if (res.statusCode >= 403) { // 2xx and 3xx are ok. even 401 and 403 are ok for now (for WP sites)
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
                 } else {
                     setHealth(app, apps.HEALTH_HEALTHY, callback);
@@ -119,7 +118,7 @@ function getContainerInfo(containerId, callback) {
 
 /*
     OOM can be tested using stress tool like so:
-        docker run -ti -m 100M cloudron/base:0.10.0 /bin/bash
+        docker run -ti -m 100M cloudron/base:2.0.0 /bin/bash
         apt-get update && apt-get install stress
         stress --vm 1 --vm-bytes 200M --vm-hang 0
 */
@@ -188,6 +187,8 @@ function run(intervalSecs, callback) {
     assert.strictEqual(typeof intervalSecs, 'number');
     assert.strictEqual(typeof callback, 'function');
 
+    if (!gStartTime) gStartTime = new Date();
+
     async.series([
         processApp, // this is first because docker.getEvents seems to get 'stuck' sometimes
         processDockerEvents.bind(null, intervalSecs)

src/apps.js

@@ -1,70 +1,72 @@
 'use strict';
 
 exports = module.exports = {
-    hasAccessTo: hasAccessTo,
-    removeInternalFields: removeInternalFields,
-    removeRestrictedFields: removeRestrictedFields,
+    hasAccessTo,
+    removeInternalFields,
+    removeRestrictedFields,
 
-    get: get,
-    getByContainerId: getByContainerId,
-    getByIpAddress: getByIpAddress,
-    getByFqdn: getByFqdn,
-    getAll: getAll,
-    getAllByUser: getAllByUser,
-    install: install,
-    uninstall: uninstall,
+    get,
+    getByIpAddress,
+    getByFqdn,
+    getAll,
+    getAllByUser,
+    install,
+    uninstall,
 
-    setAccessRestriction: setAccessRestriction,
-    setLabel: setLabel,
-    setIcon: setIcon,
-    setTags: setTags,
-    setMemoryLimit: setMemoryLimit,
-    setCpuShares: setCpuShares,
-    setBinds: setBinds,
-    setAutomaticBackup: setAutomaticBackup,
-    setAutomaticUpdate: setAutomaticUpdate,
-    setReverseProxyConfig: setReverseProxyConfig,
-    setCertificate: setCertificate,
-    setDebugMode: setDebugMode,
-    setEnvironment: setEnvironment,
-    setMailbox: setMailbox,
-    setLocation: setLocation,
-    setDataDir: setDataDir,
-    repair: repair,
+    setAccessRestriction,
+    setLabel,
+    setIcon,
+    setTags,
+    setMemoryLimit,
+    setCpuShares,
+    setMounts,
+    setAutomaticBackup,
+    setAutomaticUpdate,
+    setReverseProxyConfig,
+    setCertificate,
+    setDebugMode,
+    setEnvironment,
+    setMailbox,
+    setLocation,
+    setDataDir,
+    repair,
 
-    restore: restore,
-    importApp: importApp,
-    clone: clone,
+    restore,
+    importApp,
+    exportApp,
+    clone,
 
-    update: update,
+    update,
 
-    backup: backup,
-    listBackups: listBackups,
+    backup,
+    listBackups,
 
-    getLogs: getLogs,
+    getLocalLogfilePaths,
+    getLogs,
 
-    start: start,
-    stop: stop,
-    restart: restart,
+    start,
+    stop,
+    restart,
 
-    exec: exec,
+    exec,
 
-    checkManifestConstraints: checkManifestConstraints,
-    downloadManifest: downloadManifest,
+    checkManifestConstraints,
+    downloadManifest,
 
-    canAutoupdateApp: canAutoupdateApp,
-    autoupdateApps: autoupdateApps,
+    canAutoupdateApp,
+    autoupdateApps,
 
-    restoreInstalledApps: restoreInstalledApps,
-    configureInstalledApps: configureInstalledApps,
-    schedulePendingTasks: schedulePendingTasks,
-    restartAppsUsingAddons: restartAppsUsingAddons,
+    restoreInstalledApps,
+    configureInstalledApps,
+    schedulePendingTasks,
+    restartAppsUsingAddons,
 
-    getDataDir: getDataDir,
-    getIconPath: getIconPath,
+    getDataDir,
+    getIconPath,
+    getMemoryLimit,
 
-    downloadFile: downloadFile,
-    uploadFile: uploadFile,
+    downloadFile,
+    uploadFile,
 
     PORT_TYPE_TCP: 'tcp',
     PORT_TYPE_UDP: 'udp',
@@ -134,7 +136,6 @@ var appdb = require('./appdb.js'),
     superagent = require('superagent'),
     tasks = require('./tasks.js'),
     TransformStream = require('stream').Transform,
-    updateChecker = require('./updatechecker.js'),
     users = require('./users.js'),
     util = require('util'),
     uuid = require('uuid'),
@@ -154,7 +155,6 @@ function validatePortBindings(portBindings, manifest) {
     const RESERVED_PORTS = [
         22, /* ssh */
         25, /* smtp */
-        53, /* dns */
         80, /* http */
         143, /* imap */
         202, /* alternate ssh */
@@ -167,7 +167,7 @@ function validatePortBindings(portBindings, manifest) {
         2004, /* graphite (lo) */
         2514, /* cloudron-syslog (lo) */
         constants.PORT, /* app server (lo) */
-        constants.SYSADMIN_PORT, /* sysadmin app server (lo) */
+        constants.AUTHWALL_PORT, /* protected sites */
         constants.INTERNAL_SMTP_PORT, /* internal smtp port (lo) */
         constants.LDAP_PORT,
         3306, /* mysql (lo) */
@@ -182,6 +182,11 @@ function validatePortBindings(portBindings, manifest) {
         [50000, 51000] /* turn udp ports */
     ];
 
+    const ALLOWED_PORTS = [
+        53, // dns 53 is special and adblocker apps can use them
+        853 // dns over tls
+    ];
+
     if (!portBindings) return null;
 
     for (let portName in portBindings) {
@@ -191,7 +196,7 @@ function validatePortBindings(portBindings, manifest) {
         if (!Number.isInteger(hostPort)) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not an integer`, { field: 'portBindings', portName: portName });
         if (RESERVED_PORTS.indexOf(hostPort) !== -1) return new BoxError(BoxError.BAD_FIELD, `Port ${hostPort} is reserved.`, { field: 'portBindings', portName: portName });
         if (RESERVED_PORT_RANGES.find(range => (hostPort >= range[0] && hostPort <= range[1]))) return new BoxError(BoxError.BAD_FIELD, `Port ${hostPort} is reserved.`, { field: 'portBindings', portName: portName });
-        if (hostPort <= 1023 || hostPort > 65535) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not in permitted range`, { field: 'portBindings', portName: portName });
+        if (ALLOWED_PORTS.indexOf(hostPort) === -1 && (hostPort <= 1023 || hostPort > 65535)) return new BoxError(BoxError.BAD_FIELD, `${hostPort} is not in permitted range`, { field: 'portBindings', portName: portName });
     }
 
     // it is OK if there is no 1-1 mapping between values in manifest.tcpPorts and portBindings. missing values implies
@@ -334,20 +339,6 @@ function validateEnv(env) {
     return null;
 }
 
-function validateBinds(binds) {
-    for (let name of Object.keys(binds)) {
-        // just have friendly characters under /media
-        if (!/^[-0-9a-zA-Z_@$=#.%+]+$/.test(name)) return new BoxError(BoxError.BAD_FIELD, `Invalid bind name: ${name}`);
-
-        const bind = binds[name];
-
-        if (!bind.hostPath.startsWith('/mnt') && !bind.hostPath.startsWith('/media')) return new BoxError(BoxError.BAD_FIELD, 'hostPath must be in /mnt or /media');
-        if (path.normalize(bind.hostPath) !== bind.hostPath) return new BoxError(BoxError.BAD_FIELD, 'hostPath is not normalized');
-    }
-
-    return null;
-}
-
 function validateDataDir(dataDir) {
     if (dataDir === null) return null;
 
@@ -398,7 +389,7 @@ function getDuplicateErrorDetails(errorMessage, locations, domainObjectMap, port
 
     // check if any of the port bindings conflict
     for (let portName in portBindings) {
-        if (portBindings[portName] === parseInt(match[1])) return new BoxError(BoxError.ALREADY_EXISTS, `Port ${match[1]} is reserved`, { portName });
+        if (portBindings[portName] === parseInt(match[1])) return new BoxError(BoxError.ALREADY_EXISTS, `Port ${match[1]} is in use`, { portName });
     }
 
     if (match[2] === 'dataDir') {
@@ -420,13 +411,13 @@ function removeInternalFields(app) {
         'location', 'domain', 'fqdn', 'mailboxName', 'mailboxDomain',
         'accessRestriction', 'manifest', 'portBindings', 'iconUrl', 'memoryLimit', 'cpuShares',
         'sso', 'debugMode', 'reverseProxyConfig', 'enableBackup', 'creationTime', 'updateTime', 'ts', 'tags',
-        'label', 'alternateDomains', 'env', 'enableAutomaticUpdate', 'dataDir', 'binds');
+        'label', 'alternateDomains', 'aliasDomains', 'env', 'enableAutomaticUpdate', 'dataDir', 'mounts');
 }
 
 // non-admins can only see these
 function removeRestrictedFields(app) {
     return _.pick(app,
-        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId', 'alternateDomains', 'sso',
+        'id', 'appStoreId', 'installationState', 'error', 'runState', 'health', 'taskId', 'accessRestriction', 'alternateDomains', 'aliasDomains', 'sso',
         'location', 'domain', 'fqdn', 'manifest', 'portBindings', 'iconUrl', 'creationTime', 'ts', 'tags', 'label', 'enableBackup');
 }
 
@@ -460,6 +451,20 @@ function getIconPath(app, options, callback) {
     callback(new BoxError(BoxError.NOT_FOUND, 'No icon'));
 }
 
+function getMemoryLimit(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    let memoryLimit = app.memoryLimit || app.manifest.memoryLimit || 0;
+
+    if (memoryLimit === -1) { // unrestricted
+        memoryLimit = 0;
+    } else if (memoryLimit === 0 || memoryLimit < constants.DEFAULT_MEMORY_LIMIT) { // ensure we never go below minimum (in case we change the default)
+        memoryLimit = constants.DEFAULT_MEMORY_LIMIT;
+    }
+
+    return memoryLimit;
+}
+
 function postProcess(app, domainObjectMap) {
     let result = {};
     for (let portName in app.portBindings) {
@@ -470,6 +475,7 @@ function postProcess(app, domainObjectMap) {
     app.iconUrl = getIconUrlSync(app);
     app.fqdn = domains.fqdn(app.location, domainObjectMap[app.domain]);
     app.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+    app.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 }
 
 function hasAccessTo(app, user, callback) {
@@ -512,25 +518,6 @@ function get(appId, callback) {
         if (error) return callback(error);
 
         appdb.get(appId, function (error, app) {
-            if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
-            if (error) return callback(error);
-
-            postProcess(app, domainObjectMap);
-
-            callback(null, app);
-        });
-    });
-}
-
-function getByContainerId(containerId, callback) {
-    assert.strictEqual(typeof containerId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    getDomainObjectMap(function (error, domainObjectMap) {
-        if (error) return callback(error);
-
-        appdb.getByContainerId(containerId, function (error, app) {
-            if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
             if (error) return callback(error);
 
             postProcess(app, domainObjectMap);
@@ -548,16 +535,15 @@ function getByIpAddress(ip, callback) {
     // this is only used by the ldap test. the apps tests still uses proper docker
     if (constants.TEST && exports._MOCK_GET_BY_IP_APP_ID) return get(exports._MOCK_GET_BY_IP_APP_ID, callback);
 
-    docker.getContainerIdByIp(ip, function (error, containerId) {
+    appdb.getByIpAddress(ip, function (error, app) {
         if (error) return callback(error);
 
-        docker.inspect(containerId, function (error, result) {
+        getDomainObjectMap(function (error, domainObjectMap) {
             if (error) return callback(error);
 
-            const appId = safe.query(result, 'Config.Labels.appId', null);
-            if (!appId) return callback(new BoxError(BoxError.NOT_FOUND, 'No such app'));
+            postProcess(app, domainObjectMap);
 
-            get(appId, callback);
+            callback(null, app);
         });
     });
 }
@@ -639,20 +625,31 @@ function scheduleTask(appId, installationState, taskId, callback) {
     assert.strictEqual(typeof taskId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    appTaskManager.scheduleTask(appId, taskId, function (error) {
-        debug(`scheduleTask: task ${taskId} of ${appId} completed`);
-        if (error && (error.code === tasks.ECRASHED || error.code === tasks.ESTOPPED)) { // if task crashed, update the error
-            debug(`Apptask crashed/stopped: ${error.message}`);
-            let boxError = new BoxError(BoxError.TASK_ERROR, error.message);
-            boxError.details.crashed = error.code === tasks.ECRASHED;
-            boxError.details.stopped = error.code === tasks.ESTOPPED;
-            // see also apptask makeTaskError
-            boxError.details.taskId = taskId;
-            boxError.details.installationState = installationState;
-            appdb.update(appId, { installationState: exports.ISTATE_ERROR, error: boxError.toPlainObject(), taskId: null }, callback);
-        } else if (!(installationState === exports.ISTATE_PENDING_UNINSTALL && !error)) { // clear out taskId except for successful uninstall
-            appdb.update(appId, { taskId: null }, callback);
+    settings.getBackupConfig(function (error, backupConfig) {
+        if (error) return callback(error);
+
+        let memoryLimit = 400;
+        if (installationState === exports.ISTATE_PENDING_BACKUP || installationState === exports.ISTATE_PENDING_CLONE || installationState === exports.ISTATE_PENDING_RESTORE) {
+            memoryLimit = 'memoryLimit' in backupConfig ? Math.max(backupConfig.memoryLimit/1024/1024, 400) : 400;
         }
+
+        const options = { timeout: 20 * 60 * 60 * 1000 /* 20 hours */, nice: 15, memoryLimit };
+
+        appTaskManager.scheduleTask(appId, taskId, options, function (error) {
+            debug(`scheduleTask: task ${taskId} of ${appId} completed`);
+            if (error && (error.code === tasks.ECRASHED || error.code === tasks.ESTOPPED)) { // if task crashed, update the error
+                debug(`Apptask crashed/stopped: ${error.message}`);
+                let boxError = new BoxError(BoxError.TASK_ERROR, error.message);
+                boxError.details.crashed = error.code === tasks.ECRASHED;
+                boxError.details.stopped = error.code === tasks.ESTOPPED;
+                // see also apptask makeTaskError
+                boxError.details.taskId = taskId;
+                boxError.details.installationState = installationState;
+                appdb.update(appId, { installationState: exports.ISTATE_ERROR, error: boxError.toPlainObject(), taskId: null }, callback);
+            } else if (!(installationState === exports.ISTATE_PENDING_UNINSTALL && !error)) { // clear out taskId except for successful uninstall
+                appdb.update(appId, { taskId: null }, callback);
+            }
+        });
     });
 }
 
@@ -714,7 +711,13 @@ function validateLocations(locations, callback) {
         for (let location of locations) {
             if (!(location.domain in domainObjectMap)) return callback(new BoxError(BoxError.BAD_FIELD, 'No such domain', { field: 'location', domain: location.domain, subdomain: location.subdomain }));
 
-            error = domains.validateHostname(location.subdomain, domainObjectMap[location.domain]);
+            let subdomain = location.subdomain;
+            if (location.type === 'alias' && subdomain.startsWith('*')) {
+                if (subdomain === '*') continue;
+                subdomain = subdomain.replace(/^\*\./, ''); // remove *.
+            }
+
+            error = domains.validateHostname(subdomain, domainObjectMap[location.domain]);
             if (error) return callback(new BoxError(BoxError.BAD_FIELD, 'Bad location: ' + error.message, { field: 'location', domain: location.domain, subdomain: location.subdomain }));
         }
 
@@ -746,10 +749,12 @@ function install(data, auditSource, callback) {
         enableBackup = 'enableBackup' in data ? data.enableBackup : true,
         enableAutomaticUpdate = 'enableAutomaticUpdate' in data ? data.enableAutomaticUpdate : true,
         alternateDomains = data.alternateDomains || [],
+        aliasDomains = data.aliasDomains || [],
         env = data.env || {},
         label = data.label || null,
         tags = data.tags || [],
         overwriteDns = 'overwriteDns' in data ? data.overwriteDns : false,
+        skipDnsSetup = 'skipDnsSetup' in data ? data.skipDnsSetup : false,
         appStoreId = data.appStoreId,
         manifest = data.manifest;
 
@@ -779,7 +784,7 @@ function install(data, auditSource, callback) {
 
     if ('sso' in data && !('optionalSso' in manifest)) return callback(new BoxError(BoxError.BAD_FIELD, 'sso can only be specified for apps with optionalSso'));
     // if sso was unspecified, enable it by default if possible
-    if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['oauth'];
+    if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['proxyAuth'];
 
     error = validateEnv(env);
     if (error) return callback(error);
@@ -798,7 +803,10 @@ function install(data, auditSource, callback) {
         }
     }
 
-    const locations = [{subdomain: location, domain}].concat(alternateDomains);
+    const locations = [{ subdomain: location, domain, type: 'primary' }]
+        .concat(alternateDomains.map(ad => _.extend(ad, { type: 'redirect' })))
+        .concat(aliasDomains.map(ad => _.extend(ad, { type: 'alias' })));
+
     validateLocations(locations, function (error, domainObjectMap) {
         if (error) return callback(error);
 
@@ -810,18 +818,19 @@ function install(data, auditSource, callback) {
         debug('Will install app with id : ' + appId);
 
         var data = {
-            accessRestriction: accessRestriction,
-            memoryLimit: memoryLimit,
-            sso: sso,
-            debugMode: debugMode,
-            mailboxName: mailboxName,
-            mailboxDomain: mailboxDomain,
-            enableBackup: enableBackup,
-            enableAutomaticUpdate: enableAutomaticUpdate,
-            alternateDomains: alternateDomains,
-            env: env,
-            label: label,
-            tags: tags,
+            accessRestriction,
+            memoryLimit,
+            sso,
+            debugMode,
+            mailboxName,
+            mailboxDomain,
+            enableBackup,
+            enableAutomaticUpdate,
+            alternateDomains,
+            aliasDomains,
+            env,
+            label,
+            tags,
             runState: exports.RSTATE_RUNNING,
             installationState: exports.ISTATE_PENDING_INSTALL
         };
@@ -840,7 +849,7 @@ function install(data, auditSource, callback) {
                 }
 
                 const task = {
-                    args: { restoreConfig: null, overwriteDns },
+                    args: { restoreConfig: null, skipDnsSetup, overwriteDns },
                     values: { },
                     requiredState: data.installationState
                 };
@@ -851,6 +860,7 @@ function install(data, auditSource, callback) {
                     const newApp = _.extend({}, data, { appStoreId, manifest, location, domain, portBindings });
                     newApp.fqdn = domains.fqdn(newApp.location, domainObjectMap[newApp.domain]);
                     newApp.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+                    newApp.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 
                     eventlog.add(eventlog.ACTION_APP_INSTALL, auditSource, { appId, app: newApp, taskId: result.taskId });
 
@@ -993,9 +1003,9 @@ function setCpuShares(app, cpuShares, auditSource, callback) {
     });
 }
 
-function setBinds(app, binds, auditSource, callback) {
+function setMounts(app, mounts, auditSource, callback) {
     assert.strictEqual(typeof app, 'object');
-    assert(binds && typeof binds === 'object');
+    assert(Array.isArray(mounts));
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1003,17 +1013,15 @@ function setBinds(app, binds, auditSource, callback) {
     let error = checkAppState(app, exports.ISTATE_PENDING_RECREATE_CONTAINER);
     if (error) return callback(error);
 
-    error = validateBinds(binds);
-    if (error) return callback(error);
-
     const task = {
         args: {},
-        values: { binds }
+        values: { mounts }
     };
     addTask(appId, exports.ISTATE_PENDING_RECREATE_CONTAINER, task, function (error, result) {
+        if (error && error.reason === BoxError.ALREADY_EXISTS) return callback(new BoxError(BoxError.CONFLICT, 'Duplicate mount points'));
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId, app, binds, taskId: result.taskId });
+        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId, app, mounts, taskId: result.taskId });
 
         callback(null, { taskId: result.taskId });
     });
@@ -1208,7 +1216,8 @@ function setLocation(app, data, auditSource, callback) {
         domain: data.domain.toLowerCase(),
         // these are intentionally reset, if not set
         portBindings: null,
-        alternateDomains: []
+        alternateDomains: [],
+        aliasDomains: []
     };
 
     if ('portBindings' in data) {
@@ -1228,14 +1237,21 @@ function setLocation(app, data, auditSource, callback) {
         values.alternateDomains = data.alternateDomains;
     }
 
-    const locations = [{subdomain: values.location, domain: values.domain}].concat(values.alternateDomains);
+    if ('aliasDomains' in data) {
+        values.aliasDomains = data.aliasDomains;
+    }
+
+    const locations = [{ subdomain: values.location, domain: values.domain, type: 'primary' }]
+        .concat(values.alternateDomains.map(ad => _.extend(ad, { type: 'redirect' })))
+        .concat(values.aliasDomains.map(ad => _.extend(ad, { type: 'alias' })));
 
     validateLocations(locations, function (error, domainObjectMap) {
         if (error) return callback(error);
 
         const task = {
             args: {
-                oldConfig: _.pick(app, 'location', 'domain', 'fqdn', 'alternateDomains', 'portBindings'),
+                oldConfig: _.pick(app, 'location', 'domain', 'fqdn', 'alternateDomains', 'aliasDomains', 'portBindings'),
+                skipDnsSetup: !!data.skipDnsSetup,
                 overwriteDns: !!data.overwriteDns
             },
             values
@@ -1246,6 +1262,7 @@ function setLocation(app, data, auditSource, callback) {
 
             values.fqdn = domains.fqdn(values.location, domainObjectMap[values.domain]);
             values.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+            values.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
 
             eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, _.extend({ appId, app, taskId: result.taskId }, values));
 
@@ -1289,7 +1306,8 @@ function update(app, data, auditSource, callback) {
 
     const skipBackup = !!data.skipBackup,
         appId = app.id,
-        manifest = data.manifest;
+        manifest = data.manifest,
+        appStoreId = data.appStoreId;
 
     let values = {};
 
@@ -1304,14 +1322,12 @@ function update(app, data, auditSource, callback) {
     error = checkManifestConstraints(manifest);
     if (error) return callback(error);
 
-    var updateConfig = { skipBackup, manifest };
+    var updateConfig = { skipBackup, manifest, appStoreId }; // this will clear appStoreId when updating from a repo and set it if passed in for update route
 
     // prevent user from installing a app with different manifest id over an existing app
     // this allows cloudron install -f --app <appid> for an app installed from the appStore
     if (app.manifest.id !== updateConfig.manifest.id) {
         if (!data.force) return callback(new BoxError(BoxError.BAD_FIELD, 'manifest id does not match. force to override'));
-        // clear appStoreId so that this app does not get updates anymore
-        updateConfig.appStoreId = '';
     }
 
     // suffix '0' if prerelease is missing for semver.lte to work as expected
@@ -1358,13 +1374,23 @@ function update(app, data, auditSource, callback) {
 
         eventlog.add(eventlog.ACTION_APP_UPDATE, auditSource, { appId, app, skipBackup, toManifest: manifest, fromManifest: app.manifest, force: data.force, taskId: result.taskId });
 
-        // clear update indicator, if update fails, it will come back through the update checker
-        updateChecker.resetAppUpdateInfo(appId);
-
         callback(null, { taskId: result.taskId });
     });
 }
 
+function getLocalLogfilePaths(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    const appId = app.id;
+
+    var filePaths = [];
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
+    filePaths.push(path.join(paths.LOG_DIR, appId, 'app.log'));
+    if (app.manifest.addons && app.manifest.addons.redis) filePaths.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
+
+    return filePaths;
+}
+
 function getLogs(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
     assert(options && typeof options === 'object');
@@ -1384,11 +1410,8 @@ function getLogs(app, options, callback) {
 
     var args = [ '--lines=' + lines ];
     if (follow) args.push('--follow', '--retry', '--quiet'); // same as -F. to make it work if file doesn't exist, --quiet to not output file headers, which are no logs
-    args.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
-    args.push(path.join(paths.LOG_DIR, appId, 'app.log'));
-    if (app.manifest.addons && app.manifest.addons.redis) args.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));
 
-    var cp = spawn('/usr/bin/tail', args);
+    var cp = spawn('/usr/bin/tail', args.concat(getLocalLogfilePaths(app)));
 
     var transformStream = split(function mapper(line) {
         if (format !== 'json') return line + '\n';
@@ -1434,7 +1457,7 @@ function repair(app, data, auditSource, callback) {
 
     // maybe split this into a separate route like reinstall?
     if (errorState === exports.ISTATE_PENDING_INSTALL || errorState === exports.ISTATE_PENDING_CLONE) {
-        task.args = { overwriteDns: true };
+        task.args = { skipDnsSetup: false, overwriteDns: true };
         if (data.manifest) {
             let error = manifestFormat.parse(data.manifest);
             if (error) return callback(new BoxError(BoxError.BAD_FIELD, `manifest error: ${error.message}`));
@@ -1507,6 +1530,7 @@ function restore(app, backupId, auditSource, callback) {
             args: {
                 restoreConfig,
                 oldManifest: app.manifest,
+                skipDnsSetup: !!backupId, // if this is a restore, just skip dns setup. only re-installs should setup dns
                 overwriteDns: true
             },
             values
@@ -1562,6 +1586,7 @@ function importApp(app, data, auditSource, callback) {
             args: {
                 restoreConfig,
                 oldManifest: app.manifest,
+                skipDnsSetup: false,
                 overwriteDns: true
             },
             values: {}
@@ -1576,6 +1601,26 @@ function importApp(app, data, auditSource, callback) {
     });
 }
 
+function exportApp(app, data, auditSource, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const appId = app.id;
+
+    let error = checkAppState(app, exports.ISTATE_PENDING_BACKUP);
+    if (error) return callback(error);
+
+    const task = {
+        args: { snapshotOnly: true },
+        values: {}
+    };
+    addTask(appId, exports.ISTATE_PENDING_BACKUP, task, (error, result) => {
+        if (error) return callback(error);
+
+        callback(null, { taskId: result.taskId });
+    });
+}
+
 function purchaseApp(data, callback) {
     assert.strictEqual(typeof data, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -1604,6 +1649,7 @@ function clone(app, data, user, auditSource, callback) {
         portBindings = data.portBindings || null,
         backupId = data.backupId,
         overwriteDns = 'overwriteDns' in data ? data.overwriteDns : false,
+        skipDnsSetup = 'skipDnsSetup' in data ? data.skipDnsSetup : false,
         appId = app.id;
 
     assert.strictEqual(typeof backupId, 'string');
@@ -1630,7 +1676,7 @@ function clone(app, data, user, auditSource, callback) {
         let mailboxName = hasMailAddon(manifest) ? mailboxNameForLocation(location, manifest) : null;
         let mailboxDomain = hasMailAddon(manifest) ? domain : null;
 
-        const locations = [{subdomain: location, domain}];
+        const locations = [{ subdomain: location, domain, type: 'primary' }];
         validateLocations(locations, function (error, domainObjectMap) {
             if (error) return callback(error);
 
@@ -1647,7 +1693,8 @@ function clone(app, data, user, auditSource, callback) {
                 enableBackup: app.enableBackup,
                 reverseProxyConfig: app.reverseProxyConfig,
                 env: app.env,
-                alternateDomains: []
+                alternateDomains: [],
+                aliasDomains: []
             };
 
             appdb.add(newAppId, appStoreId, manifest, location, domain, translatePortBindings(portBindings, manifest), data, function (error) {
@@ -1659,7 +1706,7 @@ function clone(app, data, user, auditSource, callback) {
 
                     const restoreConfig = { backupId: backupId, backupFormat: backupInfo.format };
                     const task = {
-                        args: { restoreConfig, overwriteDns, oldManifest: null },
+                        args: { restoreConfig, overwriteDns, skipDnsSetup, oldManifest: null },
                         values: {},
                         requiredState: exports.ISTATE_PENDING_CLONE
                     };
@@ -1669,6 +1716,8 @@ function clone(app, data, user, auditSource, callback) {
                         const newApp = _.extend({}, data, { appStoreId, manifest, location, domain, portBindings });
                         newApp.fqdn = domains.fqdn(newApp.location, domainObjectMap[newApp.domain]);
                         newApp.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+                        newApp.aliasDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });
+
                         eventlog.add(eventlog.ACTION_APP_CLONE, auditSource, { appId: newAppId, oldAppId: appId, backupId: backupId, oldApp: app, newApp: newApp, taskId: result.taskId });
 
                         callback(null, { id: newAppId, taskId: result.taskId });
@@ -1871,8 +1920,6 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!updateInfo) return callback(null);
-
     async.eachSeries(Object.keys(updateInfo), function iterator(appId, iteratorDone) {
         get(appId, function (error, app) {
             if (error) {
@@ -1932,7 +1979,8 @@ function listBackups(app, page, perPage, callback) {
     });
 }
 
-function restoreInstalledApps(callback) {
+function restoreInstalledApps(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     getAll(function (error, apps) {
@@ -1955,7 +2003,7 @@ function restoreInstalledApps(callback) {
                 }
 
                 const task = {
-                    args: { restoreConfig, overwriteDns: true, oldManifest },
+                    args: { restoreConfig, skipDnsSetup: options.skipDnsSetup, overwriteDns: true, oldManifest },
                     values: {},
                     scheduleNow: false, // task will be scheduled by autoRestartTasks when platform is ready
                     requireNullTaskId: false // ignore existing stale taskId

src/appstore.js

@@ -1,28 +1,28 @@
 'use strict';
 
 exports = module.exports = {
-    getFeatures: getFeatures,
+    getFeatures,
 
-    getApps: getApps,
-    getApp: getApp,
-    getAppVersion: getAppVersion,
+    getApps,
+    getApp,
+    getAppVersion,
 
-    trackBeginSetup: trackBeginSetup,
-    trackFinishedSetup: trackFinishedSetup,
+    trackBeginSetup,
+    trackFinishedSetup,
 
-    registerWithLoginCredentials: registerWithLoginCredentials,
+    registerWithLoginCredentials,
 
-    purchaseApp: purchaseApp,
-    unpurchaseApp: unpurchaseApp,
+    purchaseApp,
+    unpurchaseApp,
 
-    getUserToken: getUserToken,
-    getSubscription: getSubscription,
-    isFreePlan: isFreePlan,
+    getUserToken,
+    getSubscription,
+    isFreePlan,
 
-    getAppUpdate: getAppUpdate,
-    getBoxUpdate: getBoxUpdate,
+    getAppUpdate,
+    getBoxUpdate,
 
-    createTicket: createTicket
+    createTicket
 };
 
 var apps = require('./apps.js'),
@@ -32,17 +32,21 @@ var apps = require('./apps.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:appstore'),
     eventlog = require('./eventlog.js'),
+    path = require('path'),
     paths = require('./paths.js'),
     safe = require('safetydance'),
     semver = require('semver'),
     settings = require('./settings.js'),
     superagent = require('superagent'),
+    support = require('./support.js'),
     util = require('util');
 
 // These are the default options and will be adjusted once a subscription state is obtained
 // Keep in sync with appstore/routes/cloudrons.js
 let gFeatures = {
     userMaxCount: 5,
+    userGroups: false,
+    userRoles: false,
     domainMaxCount: 1,
     externalLdap: false,
     privateDockerRegistry: false,
@@ -241,17 +245,17 @@ function getBoxUpdate(options, callback) {
             automatic: options.automatic
         };
 
-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-            if (result.statusCode === 204) return callback(null); // no update
+            if (result.statusCode === 204) return callback(null, null); // no update
             if (result.statusCode !== 200 || !result.body) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
 
             var updateInfo = result.body;
 
             if (!semver.valid(updateInfo.version) || semver.gt(constants.VERSION, updateInfo.version)) {
-                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Invalid update version: %s %s', result.statusCode, result.text)));
+                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Update version invalid or is a downgrade: %s %s', result.statusCode, result.text)));
             }
 
             // updateInfo: { version, changelog, sourceTarballUrl, sourceTarballSigUrl, boxVersionsUrl, boxVersionsSigUrl }
@@ -284,7 +288,7 @@ function getAppUpdate(app, options, callback) {
             automatic: options.automatic
         };
 
-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(30 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -406,26 +410,55 @@ function createTicket(info, auditSource, callback) {
         apps.get(info.appId, callback);
     }
 
+    function enableSshIfNeeded(callback) {
+        if (!info.enableSshSupport) return callback();
+
+        support.enableRemoteSupport(true, auditSource, function (error) {
+            // ensure we can at least get the ticket through
+            if (error) debug('Unable to enable SSH support.', error);
+
+            callback();
+        });
+    }
+
     getCloudronToken(function (error, token) {
         if (error) return callback(error);
 
-        collectAppInfoIfNeeded(function (error, result) {
+        enableSshIfNeeded(function (error) {
             if (error) return callback(error);
-            if (result) info.app = result;
 
-            let url = settings.apiServerOrigin() + '/api/v1/ticket';
+            collectAppInfoIfNeeded(function (error, app) {
+                if (error) return callback(error);
+                if (app) info.app = app;
 
-            info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
+                info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
 
-            superagent.post(url).query({ accessToken: token }).send(info).timeout(10 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+                var req = superagent.post(`${settings.apiServerOrigin()}/api/v1/ticket`)
+                    .query({ accessToken: token })
+                    .timeout(30 * 1000);
 
-                eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+                // either send as JSON through body or as multipart, depending on attachments
+                if (info.app) {
+                    req.field('infoJSON', JSON.stringify(info));
 
-                callback(null, { message: `An email for sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                    apps.getLocalLogfilePaths(info.app).forEach(function (filePath) {
+                        var logs = safe.child_process.execSync(`tail --lines=1000 ${filePath}`);
+                        if (logs) req.attach(path.basename(filePath), logs, path.basename(filePath));
+                    });
+                } else {
+                    req.send(info);
+                }
+
+                req.end(function (error, result) {
+                    if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                    if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+                    if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
+                    if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+
+                    eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
+
+                    callback(null, { message: `An email was sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                });
             });
         });
     });
@@ -441,7 +474,7 @@ function getApps(callback) {
             if (error) return callback(error);
 
             const url = `${settings.apiServerOrigin()}/api/v1/apps`;
-            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -476,7 +509,7 @@ function getAppVersion(appId, version, callback) {
             let url = `${settings.apiServerOrigin()}/api/v1/apps/${appId}`;
             if (version !== 'latest') url += `/versions/${version}`;
 
-            superagent.get(url).query({ accessToken: token }).timeout(10 * 1000).end(function (error, result) {
+            superagent.get(url).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
                 if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
                 if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
                 if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));

src/apptask.js

@@ -3,22 +3,18 @@
 'use strict';
 
 exports = module.exports = {
-    run: run,
+    run,
 
     // exported for testing
-    _reserveHttpPort: reserveHttpPort,
     _configureReverseProxy: configureReverseProxy,
     _unconfigureReverseProxy: unconfigureReverseProxy,
     _createAppDir: createAppDir,
     _deleteAppDir: deleteAppDir,
     _verifyManifest: verifyManifest,
-    _registerSubdomains: registerSubdomains,
-    _unregisterSubdomains: unregisterSubdomains,
     _waitForDnsPropagation: waitForDnsPropagation
 };
 
-var addons = require('./addons.js'),
-    appdb = require('./appdb.js'),
+const appdb = require('./appdb.js'),
     apps = require('./apps.js'),
     assert = require('assert'),
     async = require('async'),
@@ -34,14 +30,15 @@ var addons = require('./addons.js'),
     ejs = require('ejs'),
     eventlog = require('./eventlog.js'),
     fs = require('fs'),
+    iputils = require('./iputils.js'),
     manifestFormat = require('cloudron-manifestformat'),
-    net = require('net'),
     os = require('os'),
     path = require('path'),
     paths = require('./paths.js'),
     reverseProxy = require('./reverseproxy.js'),
     rimraf = require('rimraf'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     superagent = require('superagent'),
@@ -89,22 +86,16 @@ function updateApp(app, values, callback) {
     });
 }
 
-function reserveHttpPort(app, callback) {
+function allocateContainerIp(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let server = net.createServer();
-    server.listen(0, function () {
-        let port = server.address().port;
-
-        updateApp(app, { httpPort: port }, function (error) {
-            server.close(function (/* closeError */) {
-                if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Failed to allocate http port ${port}: ${error.message}`));
-
-                callback(null);
-            });
-        });
-    });
+    async.retry({ times: 10 }, function (retryCallback) {
+        const iprange = iputils.intFromIp('172.18.20.255') - iputils.intFromIp('172.18.16.1');
+        let rnd = Math.floor(Math.random() * iprange);
+        const containerIp = iputils.ipFromInt(iputils.intFromIp('172.18.16.1') + rnd);
+        updateApp(app, { containerIp }, retryCallback);
+    }, callback);
 }
 
 function configureReverseProxy(app, callback) {
@@ -340,82 +331,6 @@ function removeIcon(app, callback) {
     callback(null);
 }
 
-function registerSubdomains(app, overwrite, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof overwrite, 'boolean');
-    assert.strictEqual(typeof callback, 'function');
-
-    sysinfo.getServerIp(function (error, ip) {
-        if (error) return callback(error);
-
-        const allDomains = [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains);
-
-        debug(`registerSubdomain: Will register ${JSON.stringify(allDomains)}`);
-
-        async.eachSeries(allDomains, function (domain, iteratorDone) {
-            async.retry({ times: 200, interval: 5000 }, function (retryCallback) {
-                debugApp(app, 'Registering subdomain: %s%s', domain.subdomain ? (domain.subdomain + '.') : '', domain.domain);
-
-                // get the current record before updating it
-                domains.getDnsRecords(domain.subdomain, domain.domain, 'A', function (error, values) {
-                    if (error && error.reason === BoxError.EXTERNAL_ERROR) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
-                    if (error && error.reason === BoxError.ACCESS_DENIED) return retryCallback(null, new BoxError(BoxError.ACCESS_DENIED, error.message, { domain }));
-                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, new BoxError(BoxError.NOT_FOUND, error.message, { domain }));
-                    if (error) return retryCallback(null, new BoxError(BoxError.EXTERNAL_ERROR, error.message, domain)); // give up for other errors
-
-                    if (values.length !== 0 && values[0] === ip) return retryCallback(null); // up-to-date
-
-                    // refuse to update any existing DNS record for custom domains that we did not create
-                    if (values.length !== 0 && !overwrite) return retryCallback(null, new BoxError(BoxError.ALREADY_EXISTS, 'DNS Record already exists', { domain }));
-
-                    domains.upsertDnsRecords(domain.subdomain, domain.domain, 'A', [ ip ], function (error) {
-                        if (error && (error.reason === BoxError.BUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
-                            debug('registerSubdomains: Upsert error. Will retry.', error.message);
-                            return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
-                        }
-
-                        retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, domain) : null);
-                    });
-                });
-            }, function (error, result) {
-                if (error || result) return iteratorDone(error || result);
-
-                iteratorDone(null);
-            });
-        }, callback);
-    });
-}
-
-function unregisterSubdomains(app, allDomains, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert(Array.isArray(allDomains));
-    assert.strictEqual(typeof callback, 'function');
-
-    sysinfo.getServerIp(function (error, ip) {
-        if (error) return callback(error);
-
-        async.eachSeries(allDomains, function (domain, iteratorDone) {
-            async.retry({ times: 30, interval: 5000 }, function (retryCallback) {
-                debugApp(app, 'Unregistering subdomain: %s%s', domain.subdomain ? (domain.subdomain + '.') : '', domain.domain);
-
-                domains.removeDnsRecords(domain.subdomain, domain.domain, 'A', [ ip ], function (error) {
-                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, null);
-                    if (error && (error.reason === BoxError.SBUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
-                        debug('registerSubdomains: Remove error. Will retry.', error.message);
-                        return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain })); // try again
-                    }
-
-                    retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain }) : null);
-                });
-            }, function (error, result) {
-                if (error || result) return iteratorDone(error || result);
-
-                iteratorDone();
-            });
-        }, callback);
-    });
-}
-
 function waitForDnsPropagation(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -431,8 +346,8 @@ function waitForDnsPropagation(app, callback) {
         domains.waitForDnsRecord(app.location, app.domain, 'A', ip, { times: 240 }, function (error) {
             if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: app.location, domain: app.domain }));
 
-            // now wait for alternateDomains, if any
-            async.eachSeries(app.alternateDomains, function (domain, iteratorCallback) {
+            // now wait for alternateDomains and aliasDomains, if any
+            async.eachSeries(app.alternateDomains.concat(app.aliasDomains), function (domain, iteratorCallback) {
                 domains.waitForDnsRecord(domain.subdomain, domain.domain, 'A', ip, { times: 240 }, function (error) {
                     if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: domain.subdomain, domain: domain.domain }));
 
@@ -451,7 +366,7 @@ function moveDataDir(app, targetDir, callback) {
     let resolvedSourceDir = apps.getDataDir(app, app.dataDir);
     let resolvedTargetDir = apps.getDataDir(app, targetDir);
 
-    debug(`moveDataDir: migrating data from ${resolvedSourceDir} to ${resolvedTargetDir}`);
+    debugApp(app, `moveDataDir: migrating data from ${resolvedSourceDir} to ${resolvedTargetDir}`);
 
     if (resolvedSourceDir === resolvedTargetDir) return callback();
 
@@ -484,6 +399,8 @@ function downloadImage(manifest, callback) {
 }
 
 function startApp(app, callback){
+    debugApp(app, 'startApp: starting container');
+
     if (app.runState === apps.RSTATE_STOPPED) return callback();
 
     docker.startContainer(app.id, callback);
@@ -497,6 +414,7 @@ function install(app, args, progressCallback, callback) {
 
     const restoreConfig = args.restoreConfig; // has to be set when restoring
     const overwriteDns = args.overwriteDns;
+    const skipDnsSetup = args.skipDnsSetup;
     const oldManifest = args.oldManifest;
 
     async.series([
@@ -517,7 +435,7 @@ function install(app, args, progressCallback, callback) {
                 addonsToRemove = app.manifest.addons;
             }
 
-            addons.teardownAddons(app, addonsToRemove, next);
+            services.teardownAddons(app, addonsToRemove, next);
         },
 
         function deleteAppDirIfNeeded(done) {
@@ -532,13 +450,21 @@ function install(app, args, progressCallback, callback) {
             docker.deleteImage(oldManifest, done);
         },
 
-        reserveHttpPort.bind(null, app),
+        // allocating container ip here, lets the users "repair" an app if allocation fails at appdb.add time
+        allocateContainerIp.bind(null, app),
 
         progressCallback.bind(null, { percent: 20, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
 
-        progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
-        registerSubdomains.bind(null, app, overwriteDns),
+        function setupDnsIfNeeded(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
+
+                domains.registerLocations.bind(null, [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains).concat(app.aliasDomains), { overwriteDns }, progressCallback)
+            ], done);
+        },
 
         progressCallback.bind(null, { percent: 40, message: 'Downloading image' }),
         downloadImage.bind(null, app.manifest),
@@ -550,24 +476,24 @@ function install(app, args, progressCallback, callback) {
             if (!restoreConfig) {
                 async.series([
                     progressCallback.bind(null, { percent: 60, message: 'Setting up addons' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
                 ], next);
             } else if (!restoreConfig.backupId) { // in-place import
                 async.series([
                     progressCallback.bind(null, { percent: 60, message: 'Importing addons in-place' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
-                    addons.clearAddons.bind(null, app, _.omit(app.manifest.addons, 'localstorage')),
-                    addons.restoreAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
+                    services.clearAddons.bind(null, app, _.omit(app.manifest.addons, 'localstorage')),
+                    services.restoreAddons.bind(null, app, app.manifest.addons),
                 ], next);
             } else {
                 async.series([
                     progressCallback.bind(null, { percent: 65, message: 'Download backup and restoring addons' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
-                    addons.clearAddons.bind(null, app, app.manifest.addons),
+                    services.setupAddons.bind(null, app, app.manifest.addons),
+                    services.clearAddons.bind(null, app, app.manifest.addons),
                     backups.downloadApp.bind(null, app, restoreConfig, (progress) => {
                         progressCallback({ percent: 65, message: progress.message });
                     }),
-                    addons.restoreAddons.bind(null, app, app.manifest.addons)
+                    services.restoreAddons.bind(null, app, app.manifest.addons)
                 ], next);
             }
         },
@@ -577,8 +503,14 @@ function install(app, args, progressCallback, callback) {
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 85, message: 'Waiting for DNS propagation' }),
-        exports._waitForDnsPropagation.bind(null, app),
+        function waitForDns(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 85, message: 'Waiting for DNS propagation' }),
+                exports._waitForDnsPropagation.bind(null, app),
+            ], done);
+        },
 
         progressCallback.bind(null, { percent: 95, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
@@ -602,7 +534,7 @@ function backup(app, args, progressCallback, callback) {
 
     async.series([
         progressCallback.bind(null, { percent: 10, message: 'Backing up' }),
-        backups.backupApp.bind(null, app, { /* options */ }, (progress) => {
+        backups.backupApp.bind(null, app, { snapshotOnly: !!args.snapshotOnly }, (progress) => {
             progressCallback({ percent: 30, message: progress.message });
         }),
 
@@ -630,7 +562,7 @@ function create(app, args, progressCallback, callback) {
 
         // FIXME: re-setup addons only because sendmail addon to re-inject env vars on mailboxName change
         progressCallback.bind(null, { percent: 30, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
@@ -656,6 +588,7 @@ function changeLocation(app, args, progressCallback, callback) {
 
     const oldConfig = args.oldConfig;
     const locationChanged = oldConfig.fqdn !== app.fqdn;
+    const skipDnsSetup = args.skipDnsSetup;
     const overwriteDns = args.overwriteDns;
 
     async.series([
@@ -667,27 +600,45 @@ function changeLocation(app, args, progressCallback, callback) {
                 return !app.alternateDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; });
             });
 
+            if (oldConfig.aliasDomains) {
+                obsoleteDomains = obsoleteDomains.concat(oldConfig.aliasDomains.filter(function (o) {
+                    return !app.aliasDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; });
+                }));
+            }
+
             if (locationChanged) obsoleteDomains.push({ subdomain: oldConfig.location, domain: oldConfig.domain });
 
             if (obsoleteDomains.length === 0) return next();
 
-            unregisterSubdomains(app, obsoleteDomains, next);
+            domains.unregisterLocations(obsoleteDomains, progressCallback, next);
         },
 
-        progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
-        registerSubdomains.bind(null, app, overwriteDns),
+        function setupDnsIfNeeded(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 30, message: 'Registering subdomains' }),
+                domains.registerLocations.bind(null, [ { subdomain: app.location, domain: app.domain }].concat(app.alternateDomains).concat(app.aliasDomains), { overwriteDns }, progressCallback)
+            ], done);
+        },
 
         // re-setup addons since they rely on the app's fqdn (e.g oauth)
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 80, message: 'Waiting for DNS propagation' }),
-        exports._waitForDnsPropagation.bind(null, app),
+        function waitForDns(done) {
+            if (skipDnsSetup) return done();
+
+            async.series([
+                progressCallback.bind(null, { percent: 80, message: 'Waiting for DNS propagation' }),
+                exports._waitForDnsPropagation.bind(null, app),
+            ], done);
+        },
 
         progressCallback.bind(null, { percent: 90, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
@@ -721,7 +672,7 @@ function migrateDataDir(app, args, progressCallback, callback) {
 
         // re-setup addons since this creates the localStorage volume
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, _.extend({}, app, { dataDir: newDataDir }), app.manifest.addons),
+        services.setupAddons.bind(null, _.extend({}, app, { dataDir: newDataDir }), app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Moving data dir' }),
         moveDataDir.bind(null, app, newDataDir),
@@ -754,7 +705,6 @@ function configure(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 10, message: 'Cleaning up old install' }),
         unconfigureReverseProxy.bind(null, app),
         deleteContainers.bind(null, app, { managedOnly: true }),
-        reserveHttpPort.bind(null, app),
 
         progressCallback.bind(null, { percent: 20, message: 'Downloading icon' }),
         downloadIcon.bind(null, app),
@@ -767,7 +717,7 @@ function configure(app, args, progressCallback, callback) {
 
         // re-setup addons since they rely on the app's fqdn (e.g oauth)
         progressCallback.bind(null, { percent: 50, message: 'Setting up addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        services.setupAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 60, message: 'Creating container' }),
         createContainer.bind(null, app),
@@ -789,7 +739,6 @@ function configure(app, args, progressCallback, callback) {
     });
 }
 
-// nginx configuration is skipped because app.httpPort is expected to be available
 function update(app, args, progressCallback, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof args, 'object');
@@ -801,7 +750,10 @@ function update(app, args, progressCallback, callback) {
 
     // app does not want these addons anymore
     // FIXME: this does not handle option changes (like multipleDatabases)
-    var unusedAddons = _.omit(app.manifest.addons, Object.keys(updateConfig.manifest.addons));
+    const unusedAddons = _.omit(app.manifest.addons, Object.keys(updateConfig.manifest.addons));
+    const httpPathsChanged = app.manifest.httpPaths !== updateConfig.manifest.httpPaths;
+    const httpPortChanged = app.manifest.httpPort !== updateConfig.manifest.httpPort;
+    const proxyAuthChanged = !_.isEqual(safe.query(app.manifest, 'addons.proxyAuth'), safe.query(updateConfig.manifest, 'addons.proxyAuth'));
 
     async.series([
         // this protects against the theoretical possibility of an app being marked for update from
@@ -840,7 +792,7 @@ function update(app, args, progressCallback, callback) {
         },
 
         // only delete unused addons after backup
-        addons.teardownAddons.bind(null, app, unusedAddons),
+        services.teardownAddons.bind(null, app, unusedAddons),
 
         // free unused ports
         function (next) {
@@ -852,7 +804,7 @@ function update(app, args, progressCallback, callback) {
                 if (newTcpPorts[portName] || newUdpPorts[portName]) return callback(null); // port still in use
 
                 appdb.delPortBinding(currentPorts[portName], apps.PORT_TYPE_TCP, function (error) {
-                    if (error && error.reason === BoxError.NOT_FOUND) debug('update: portbinding does not exist in database', error);
+                    if (error && error.reason === BoxError.NOT_FOUND) debugApp(app, 'update: portbinding does not exist in database', error);
                     else if (error) return next(error);
 
                     // also delete from app object for further processing (the db is updated in the next step)
@@ -869,13 +821,20 @@ function update(app, args, progressCallback, callback) {
         downloadIcon.bind(null, app),
 
         progressCallback.bind(null, { percent: 60, message: 'Updating addons' }),
-        addons.setupAddons.bind(null, app, updateConfig.manifest.addons),
+        services.setupAddons.bind(null, app, updateConfig.manifest.addons),
 
         progressCallback.bind(null, { percent: 70, message: 'Creating container' }),
         createContainer.bind(null, app),
 
         startApp.bind(null, app),
 
+        progressCallback.bind(null, { percent: 90, message: 'Configuring reverse proxy' }),
+        function (next) {
+            if (!httpPathsChanged && !proxyAuthChanged && !httpPortChanged) return next();
+
+            configureReverseProxy(app, next);
+        },
+
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null, updateTime: new Date() })
     ], function seriesDone(error) {
@@ -899,13 +858,16 @@ function start(app, args, progressCallback, callback) {
 
     async.series([
         progressCallback.bind(null, { percent: 10, message: 'Starting app services' }),
-        addons.startAppServices.bind(null, app),
+        services.startAppServices.bind(null, app),
 
         progressCallback.bind(null, { percent: 35, message: 'Starting container' }),
         docker.startContainer.bind(null, app.id),
 
+        progressCallback.bind(null, { percent: 60, message: 'Adding collectd profile' }),
+        addCollectdProfile.bind(null, app),
+
         // stopped apps do not renew certs. currently, we don't do DNS to not overwrite existing user settings
-        progressCallback.bind(null, { percent: 60, message: 'Configuring reverse proxy' }),
+        progressCallback.bind(null, { percent: 80, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
 
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
@@ -930,7 +892,10 @@ function stop(app, args, progressCallback, callback) {
         docker.stopContainers.bind(null, app.id),
 
         progressCallback.bind(null, { percent: 50, message: 'Stopping app services' }),
-        addons.stopAppServices.bind(null, app),
+        services.stopAppServices.bind(null, app),
+
+        progressCallback.bind(null, { percent: 80, message: 'Removing collectd profile' }),
+        removeCollectdProfile.bind(null, app),
 
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null })
@@ -976,7 +941,7 @@ function uninstall(app, args, progressCallback, callback) {
         deleteContainers.bind(null, app, {}),
 
         progressCallback.bind(null, { percent: 30, message: 'Teardown addons' }),
-        addons.teardownAddons.bind(null, app, app.manifest.addons),
+        services.teardownAddons.bind(null, app, app.manifest.addons),
 
         progressCallback.bind(null, { percent: 40, message: 'Cleanup file manager' }),
 
@@ -987,7 +952,7 @@ function uninstall(app, args, progressCallback, callback) {
         docker.deleteImage.bind(null, app.manifest),
 
         progressCallback.bind(null, { percent: 70, message: 'Unregistering domains' }),
-        unregisterSubdomains.bind(null, app, [ { subdomain: app.location, domain: app.domain } ].concat(app.alternateDomains)),
+        domains.unregisterLocations.bind(null, [ { subdomain: app.location, domain: app.domain } ].concat(app.alternateDomains).concat(app.aliasDomains), progressCallback),
 
         progressCallback.bind(null, { percent: 80, message: 'Cleanup icon' }),
         removeIcon.bind(null, app),

src/apptaskmanager.js

@@ -1,7 +1,7 @@
 'use strict';
 
 exports = module.exports = {
-    scheduleTask: scheduleTask
+    scheduleTask
 };
 
 let assert = require('assert'),
@@ -12,7 +12,8 @@ let assert = require('assert'),
     safe = require('safetydance'),
     path = require('path'),
     paths = require('./paths.js'),
-    sftp = require('./sftp.js'),
+    scheduler = require('./scheduler.js'),
+    services = require('./services.js'),
     tasks = require('./tasks.js');
 
 let gActiveTasks = { }; // indexed by app id
@@ -36,9 +37,10 @@ function initializeSync() {
 }
 
 // callback is called when task is finished
-function scheduleTask(appId, taskId, callback) {
+function scheduleTask(appId, taskId, options, callback) {
     assert.strictEqual(typeof appId, 'string');
     assert.strictEqual(typeof taskId, 'string');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     if (!gInitialized) initializeSync();
@@ -50,7 +52,7 @@ function scheduleTask(appId, taskId, callback) {
     if (Object.keys(gActiveTasks).length >= TASK_CONCURRENCY) {
         debug(`Reached concurrency limit, queueing task id ${taskId}`);
         tasks.update(taskId, { percent: 1, message: 'Waiting for other app tasks to complete' }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
+        gPendingTasks.push({ appId, taskId, options, callback });
         return;
     }
 
@@ -59,7 +61,7 @@ function scheduleTask(appId, taskId, callback) {
     if (lockError) {
         debug(`Could not get lock. ${lockError.message}, queueing task id ${taskId}`);
         tasks.update(taskId, { percent: 1, message: waitText(lockError.operation) }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
+        gPendingTasks.push({ appId, taskId, options, callback });
         return;
     }
 
@@ -69,17 +71,17 @@ function scheduleTask(appId, taskId, callback) {
 
     if (!fs.existsSync(path.dirname(logFile))) safe.fs.mkdirSync(path.dirname(logFile)); // ensure directory
 
-    // TODO: set memory limit for app backup task
-    tasks.startTask(taskId, { logFile, timeout: 20 * 60 * 60 * 1000 /* 20 hours */, nice: 15 }, function (error, result) {
+    scheduler.suspendJobs(appId);
+
+    tasks.startTask(taskId, Object.assign(options, { logFile }), function (error, result) {
         callback(error, result);
 
         delete gActiveTasks[appId];
         locker.unlock(locker.OP_APPTASK); // unlock event will trigger next task
 
         // post app task hooks
-        sftp.rebuild(function (error) {
-            if (error) console.error('Unable to rebuild sftp:', error);
-        });
+        services.rebuildService('sftp', error => { if (error) debug('Unable to rebuild sftp:', error); });
+        scheduler.resumeJobs(appId);
     });
 }
 
@@ -89,6 +91,6 @@ function startNextTask() {
     assert(Object.keys(gActiveTasks).length < TASK_CONCURRENCY);
 
     const t = gPendingTasks.shift();
-    scheduleTask(t.appId, t.taskId, t.callback);
+    scheduleTask(t.appId, t.taskId, t.options, t.callback);
 }
 

src/autoconfig.xml.ejs

@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<clientConfig version="1.1">
+  <emailProvider id="<%= domain %>">
+    <domain><%= domain %></domain>
+    <displayName>Cloudron Mail</displayName>
+    <displayShortName>Cloudron</displayShortName>
+    <incomingServer type="imap">
+      <hostname><%= mailFqdn %></hostname>
+      <port>993</port>
+      <socketType>SSL</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </incomingServer>
+    <outgoingServer type="smtp">
+      <hostname><%= mailFqdn %></hostname>
+      <port>587</port>
+      <socketType>STARTTLS</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+      <addThisServer>true</addThisServer>
+    </outgoingServer>
+
+    <documentation url="http://cloudron.io/email/#autodiscover">
+       <descr lang="en">Cloudron Email</descr>
+    </documentation>
+
+  </emailProvider>
+</clientConfig>
+

src/backupdb.js

@@ -18,6 +18,7 @@ exports = module.exports = {
     get,
     del,
     update,
+    list,
 
     _clear: clear
 };
@@ -80,6 +81,21 @@ function getByIdentifierPaged(identifier, page, perPage, callback) {
         });
 }
 
+function list(page, perPage, callback) {
+    assert(typeof page === 'number' && page > 0);
+    assert(typeof perPage === 'number' && perPage > 0);
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups ORDER BY creationTime DESC LIMIT ?,?',
+        [ (page-1)*perPage, perPage ], function (error, results) {
+            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+
+            results.forEach(function (result) { postProcess(result); });
+
+            callback(null, results);
+        });
+}
+
 function get(id, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');

src/backups.js

@@ -1,36 +1,36 @@
 'use strict';
 
 exports = module.exports = {
-    testConfig: testConfig,
-    testProviderConfig: testProviderConfig,
+    testConfig,
+    testProviderConfig,
 
     getByIdentifierAndStatePaged,
 
-    get: get,
+    get,
 
-    startBackupTask: startBackupTask,
+    startBackupTask,
 
-    restore: restore,
+    restore,
 
-    backupApp: backupApp,
-    downloadApp: downloadApp,
+    backupApp,
+    downloadApp,
 
-    backupBoxAndApps: backupBoxAndApps,
+    backupBoxAndApps,
 
-    upload: upload,
+    upload,
 
-    startCleanupTask: startCleanupTask,
-    cleanup: cleanup,
-    cleanupCacheFilesSync: cleanupCacheFilesSync,
+    startCleanupTask,
+    cleanup,
+    cleanupCacheFilesSync,
 
-    injectPrivateFields: injectPrivateFields,
-    removePrivateFields: removePrivateFields,
+    injectPrivateFields,
+    removePrivateFields,
 
-    checkConfiguration: checkConfiguration,
+    checkConfiguration,
 
-    configureCollectd: configureCollectd,
+    configureCollectd,
 
-    generateEncryptionKeysSync: generateEncryptionKeysSync,
+    generateEncryptionKeysSync,
 
     BACKUP_IDENTIFIER_BOX: 'box',
 
@@ -48,8 +48,7 @@ exports = module.exports = {
     _applyBackupRetentionPolicy: applyBackupRetentionPolicy
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     async = require('async'),
     assert = require('assert'),
     backupdb = require('./backupdb.js'),
@@ -72,6 +71,7 @@ var addons = require('./addons.js'),
     progressStream = require('progress-stream'),
     safe = require('safetydance'),
     shell = require('./shell.js'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     syncer = require('./syncer.js'),
     tar = require('tar-fs'),
@@ -108,6 +108,7 @@ function api(provider) {
     case 'backblaze-b2': return require('./storage/s3.js');
     case 'linode-objectstorage': return require('./storage/s3.js');
     case 'ovh-objectstorage': return require('./storage/s3.js');
+    case 'ionos-objectstorage': return require('./storage/s3.js');
     case 'noop': return require('./storage/noop.js');
     default: return null;
     }
@@ -549,21 +550,29 @@ function saveFsMetadata(dataLayout, metadataFile, callback) {
     // contains paths prefixed with './'
     let metadata = {
         emptyDirs: [],
-        execFiles: []
+        execFiles: [],
+        symlinks: []
     };
 
+    // we assume small number of files. spawnSync will raise a ENOBUFS error after maxBuffer
     for (let lp of dataLayout.localPaths()) {
-        var emptyDirs = safe.child_process.execSync(`find ${lp} -type d -empty\n`, { encoding: 'utf8' });
-        if (emptyDirs === null) return callback(safe.error);
+        const emptyDirs = safe.child_process.execSync(`find ${lp} -type d -empty`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 30 });
+        if (emptyDirs === null) return callback(new BoxError(BoxError.FS_ERROR, `Error finding empty dirs: ${safe.error.message}`));
         if (emptyDirs.length) metadata.emptyDirs = metadata.emptyDirs.concat(emptyDirs.trim().split('\n').map((ed) => dataLayout.toRemotePath(ed)));
 
-        var execFiles = safe.child_process.execSync(`find ${lp} -type f -executable\n`, { encoding: 'utf8' });
-        if (execFiles === null) return callback(safe.error);
-
+        const execFiles = safe.child_process.execSync(`find ${lp} -type f -executable`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 30 });
+        if (execFiles === null) return callback(new BoxError(BoxError.FS_ERROR, `Error finding executables: ${safe.error.message}`));
         if (execFiles.length) metadata.execFiles = metadata.execFiles.concat(execFiles.trim().split('\n').map((ef) => dataLayout.toRemotePath(ef)));
+
+        const symlinks = safe.child_process.execSync(`find ${lp} -type l`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 30 });
+        if (symlinks === null) return callback(new BoxError(BoxError.FS_ERROR, `Error finding symlinks: ${safe.error.message}`));
+        if (symlinks.length) metadata.symlinks = metadata.symlinks.concat(symlinks.trim().split('\n').map((sl) => {
+            const target = safe.fs.readlinkSync(sl);
+            return { path: dataLayout.toRemotePath(sl), target };
+        }));
     }
 
-    if (!safe.fs.writeFileSync(metadataFile, JSON.stringify(metadata, null, 4))) return callback(safe.error);
+    if (!safe.fs.writeFileSync(metadataFile, JSON.stringify(metadata, null, 4))) return callback(new BoxError(BoxError.FS_ERROR, `Error writing fs metadata: ${safe.error.message}`));
 
     callback();
 }
@@ -691,7 +700,19 @@ function restoreFsMetadata(dataLayout, metadataFile, callback) {
         }, function (error) {
             if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `unable to chmod: ${error.message}`));
 
-            callback();
+            async.eachSeries(metadata.symlinks || [], function createSymlink(symlink, iteratorDone) {
+                if (!symlink.target) return iteratorDone();
+                // the path may not exist if we had a directory full of symlinks
+                fs.mkdir(path.dirname(dataLayout.toLocalPath(symlink.path)), { recursive: true }, function (error) {
+                    if (error) return iteratorDone(error);
+
+                    fs.symlink(symlink.target, dataLayout.toLocalPath(symlink.path), 'file', iteratorDone);
+                });
+            }, function (error) {
+                if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `unable to symlink: ${error.message}`));
+
+                callback();
+            });
         });
     });
 }
@@ -805,7 +826,9 @@ function restore(backupConfig, backupId, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    const dataLayout = new DataLayout(paths.BOX_DATA_DIR, []);
+    const boxDataDir = safe.fs.realpathSync(paths.BOX_DATA_DIR);
+    if (!boxDataDir) return callback(new BoxError(BoxError.FS_ERROR, `Error resolving boxdata: ${safe.error.message}`));
+    const dataLayout = new DataLayout(boxDataDir, []);
 
     download(backupConfig, backupId, backupConfig.format, dataLayout, progressCallback, function (error) {
         if (error) return callback(error);
@@ -829,7 +852,7 @@ function downloadApp(app, restoreConfig, progressCallback, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     const appDataDir = safe.fs.realpathSync(path.join(paths.APPS_DATA_DIR, app.id));
-    if (!appDataDir) return callback(safe.error);
+    if (!appDataDir) return callback(new BoxError(BoxError.FS_ERROR, safe.error.message));
     const dataLayout = new DataLayout(appDataDir, app.dataDir ? [{ localDir: app.dataDir, remoteDir: 'data' }] : []);
 
     const startTime = new Date();
@@ -851,15 +874,23 @@ function runBackupUpload(uploadConfig, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    const { backupId, format, dataLayout, progressTag } = uploadConfig;
+    const { backupId, backupConfig, dataLayout, progressTag } = uploadConfig;
     assert.strictEqual(typeof backupId, 'string');
-    assert.strictEqual(typeof format, 'string');
+    assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof progressTag, 'string');
     assert(dataLayout instanceof DataLayout, 'dataLayout must be a DataLayout');
 
     let result = ''; // the script communicates error result as a string
 
-    shell.sudo(`backup-${backupId}`, [ BACKUP_UPLOAD_CMD, backupId, format, dataLayout.toString() ], { preserveEnv: true, ipc: true }, function (error) {
+    // https://stackoverflow.com/questions/48387040/node-js-recommended-max-old-space-size
+    const envCopy = Object.assign({}, process.env);
+    if (backupConfig.memoryLimit && backupConfig.memoryLimit >= 2*1024*1024*1024) {
+        const heapSize = Math.min((backupConfig.memoryLimit/1024/1024) - 256, 8192);
+        debug(`runBackupUpload: adjusting heap size to ${heapSize}M`);
+        envCopy.NODE_OPTIONS = `--max-old-space-size=${heapSize}`;
+    }
+
+    shell.sudo(`backup-${backupId}`, [ BACKUP_UPLOAD_CMD, backupId, backupConfig.format, dataLayout.toString() ], { env: envCopy, preserveEnv: true, ipc: true }, function (error) {
         if (error && (error.code === null /* signal */ || (error.code !== 0 && error.code !== 50))) { // backuptask crashed
             return callback(new BoxError(BoxError.INTERNAL_ERROR, 'Backuptask crashed'));
         } else if (error && error.code === 50) { // exited with error
@@ -924,11 +955,11 @@ function uploadBoxSnapshot(backupConfig, progressCallback, callback) {
         if (error) return callback(error);
 
         const boxDataDir = safe.fs.realpathSync(paths.BOX_DATA_DIR);
-        if (!boxDataDir) return callback(safe.error);
+        if (!boxDataDir) return callback(new BoxError(BoxError.FS_ERROR, `Error resolving boxdata: ${safe.error.message}`));
 
         const uploadConfig = {
             backupId: 'snapshot/box',
-            format: backupConfig.format,
+            backupConfig,
             dataLayout: new DataLayout(boxDataDir, []),
             progressTag: 'box'
         };
@@ -1037,7 +1068,7 @@ function snapshotApp(app, progressCallback, callback) {
         return callback(new BoxError(BoxError.FS_ERROR, 'Error creating config.json: ' + safe.error.message));
     }
 
-    addons.backupAddons(app, app.manifest.addons, function (error) {
+    services.backupAddons(app, app.manifest.addons, function (error) {
         if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error.message));
 
         debugApp(app, `snapshotApp: took ${(new Date() - startTime)/1000} seconds`);
@@ -1107,7 +1138,7 @@ function uploadAppSnapshot(backupConfig, app, progressCallback, callback) {
 
         const backupId = util.format('snapshot/app_%s', app.id);
         const appDataDir = safe.fs.realpathSync(path.join(paths.APPS_DATA_DIR, app.id));
-        if (!appDataDir) return callback(safe.error);
+        if (!appDataDir) return callback(new BoxError(BoxError.FS_ERROR, `Error resolving appsdata: ${safe.error.message}`));
 
         const dataLayout = new DataLayout(appDataDir, app.dataDir ? [{ localDir: app.dataDir, remoteDir: 'data' }] : []);
 
@@ -1115,7 +1146,7 @@ function uploadAppSnapshot(backupConfig, app, progressCallback, callback) {
 
         const uploadConfig = {
             backupId,
-            format: backupConfig.format,
+            backupConfig,
             dataLayout,
             progressTag: app.fqdn
         };
@@ -1167,6 +1198,8 @@ function backupApp(app, options, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
+    if (options.snapshotOnly) return snapshotApp(app, progressCallback, callback);
+
     const tag = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
 
     debug(`backupApp - Backing up ${app.fqdn} with tag ${tag}`);
@@ -1313,7 +1346,7 @@ function cleanupBackup(backupConfig, backup, progressCallback, callback) {
     assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    var backupFilePath = getBackupFilePath(backupConfig, backup.id, backup.format);
+    const backupFilePath = getBackupFilePath(backupConfig, backup.id, backup.format);
 
     function done(error) {
         if (error) {
@@ -1417,6 +1450,46 @@ function cleanupBoxBackups(backupConfig, progressCallback, callback) {
     });
 }
 
+function cleanupMissingBackups(backupConfig, progressCallback, callback) {
+    assert.strictEqual(typeof backupConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    let page = 1, perPage = 1000, more = false, missingBackupIds = [];
+
+    async.doWhilst(function (whilstCallback) {
+        backupdb.list(page, perPage, function (error, result) {
+            if (error) return whilstCallback(error);
+
+            async.eachSeries(result, function (backup, next) {
+                let backupFilePath = getBackupFilePath(backupConfig, backup.id, backup.format);
+                if (backup.format === 'rsync') backupFilePath = backupFilePath + '/'; // add trailing slash to indicate directory
+
+                api(backupConfig.provider).exists(backupConfig, backupFilePath, function (error, exists) {
+                    if (error || exists) return next();
+
+                    progressCallback({ message: `Removing missing backup ${backup.id}`});
+
+                    backupdb.del(backup.id, function (error) {
+                        if (error) debug(`cleanupBackup: error removing ${backup.id} from database`, error);
+
+                        missingBackupIds.push(backup.id);
+
+                        next();
+                    });
+                });
+            }, function () {
+                more = result.length === perPage;
+                whilstCallback();
+            });
+        });
+    }, function (testDone) { return testDone(null, more); }, function (error) {
+        if (error) return callback(error);
+
+        return callback(null, missingBackupIds);
+    });
+}
+
 function cleanupCacheFilesSync() {
     var files = safe.fs.readdirSync(path.join(paths.BACKUP_INFO_DIR));
     if (!files) return;
@@ -1488,12 +1561,18 @@ function cleanup(progressCallback, callback) {
             cleanupAppBackups(backupConfig, referencedAppBackupIds, progressCallback, function (error, removedAppBackupIds) {
                 if (error) return callback(error);
 
-                progressCallback({ percent: 90, message: 'Cleaning snapshots' });
+                progressCallback({ percent: 70, message: 'Cleaning missing backups' });
 
-                cleanupSnapshots(backupConfig, function (error) {
+                cleanupMissingBackups(backupConfig, progressCallback, function (error, missingBackupIds) {
                     if (error) return callback(error);
 
-                    callback(null, { removedBoxBackupIds, removedAppBackupIds });
+                    progressCallback({ percent: 90, message: 'Cleaning snapshots' });
+
+                    cleanupSnapshots(backupConfig, function (error) {
+                        if (error) return callback(error);
+
+                        callback(null, { removedBoxBackupIds, removedAppBackupIds, missingBackupIds });
+                    });
                 });
             });
         });
@@ -1505,12 +1584,13 @@ function startCleanupTask(auditSource, callback) {
     tasks.add(tasks.TASK_CLEAN_BACKUPS, [], function (error, taskId) {
         if (error) return callback(error);
 
-        tasks.startTask(taskId, {}, (error, result) => { // result is { removedBoxBackups, removedAppBackups }
+        tasks.startTask(taskId, {}, (error, result) => { // result is { removedBoxBackupIds, removedAppBackupIds, missingBackupIds }
             eventlog.add(eventlog.ACTION_BACKUP_CLEANUP_FINISH, auditSource, {
                 taskId,
                 errorMessage: error ? error.message : null,
-                removedBoxBackups: result ? result.removedBoxBackups : [],
-                removedAppBackups: result ? result.removedAppBackups : []
+                removedBoxBackupIds: result ? result.removedBoxBackupIds : [],
+                removedAppBackupIds: result ? result.removedAppBackupIds : [],
+                missingBackupIds: result ? result.missingBackupIds : []
             });
         });
 

src/branding.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports = module.exports = {
+    renderFooter
+};
+
+const assert = require('assert'),
+    constants = require('./constants.js');
+
+function renderFooter(footer) {
+    assert.strictEqual(typeof footer, 'string');
+
+    const year = new Date().getFullYear();
+
+    return footer.replace(/%YEAR%/g, year)
+        .replace(/%VERSION%/g, constants.VERSION);
+}
+

src/cert/acme2.js

@@ -139,7 +139,7 @@ Acme2.prototype.updateContact = function (registrationUri, callback) {
     const that = this;
     this.sendSignedRequest(registrationUri, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to update contact. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to update contact. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug(`updateContact: contact of user updated to ${that.email}`);
 
@@ -160,7 +160,7 @@ Acme2.prototype.registerUser = function (callback) {
     this.sendSignedRequest(this.directory.newAccount, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
         // 200 if already exists. 201 for new accounts
-        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register new account. Expecting 200 or 201, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to register new account. Expecting 200 or 201, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug(`registerUser: user registered keyid: ${result.headers.location}`);
 
@@ -185,8 +185,8 @@ Acme2.prototype.newOrder = function (domain, callback) {
 
     this.sendSignedRequest(this.directory.newOrder, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode === 403) return callback(new BoxError(BoxError.ACCESS_DENIED, `Forbidden sending signed request: ${result.body.detail}`));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register user. Expecting 201, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode === 403) return callback(new BoxError(BoxError.ACCESS_DENIED, `Forbidden sending new order: ${result.body.detail}`));
+        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to send new order. Expecting 201, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         debug('newOrder: created order %s %j', domain, result.body);
 
@@ -259,7 +259,7 @@ Acme2.prototype.notifyChallengeReady = function (challenge, callback) {
 
     this.sendSignedRequest(challenge.url, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to notify challenge. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to notify challenge. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         callback();
     });
@@ -313,7 +313,7 @@ Acme2.prototype.signCertificate = function (domain, finalizationUrl, csrDer, cal
     this.sendSignedRequest(finalizationUrl, JSON.stringify(payload), function (error, result) {
         if (error) return callback(error);
         // 429 means we reached the cert limit for this domain
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to sign certificate. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to sign certificate. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
         return callback(null);
     });
@@ -362,7 +362,7 @@ Acme2.prototype.downloadCertificate = function (hostname, certUrl, callback) {
         that.postAsGet(certUrl, function (error, result) {
             if (error) return retryCallback(new BoxError(BoxError.NETWORK_ERROR, `Network error when downloading certificate: ${error.message}`));
             if (result.statusCode === 202) return retryCallback(new BoxError(BoxError.TRY_AGAIN, 'Retry downloading certificate'));
-            if (result.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to get cert. Expecting 200, got %s %s', result.statusCode, result.text)));
+            if (result.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to get cert. Expecting 200, got ${result.statusCode} ${JSON.stringify(result.body)}`));
 
             const fullChainPem = result.body; // buffer
 
@@ -586,34 +586,34 @@ Acme2.prototype.getDirectory = function (callback) {
     });
 };
 
-Acme2.prototype.getCertificate = function (hostname, domain, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+Acme2.prototype.getCertificate = function (vhost, domain, callback) {
+    assert.strictEqual(typeof vhost, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`getCertificate: start acme flow for ${hostname} from ${this.caDirectory}`);
+    debug(`getCertificate: start acme flow for ${vhost} from ${this.caDirectory}`);
 
-    if (hostname !== domain && this.wildcard) { // bare domain is not part of wildcard SAN
-        hostname = domains.makeWildcard(hostname);
-        debug(`getCertificate: will get wildcard cert for ${hostname}`);
+    if (vhost !== domain && this.wildcard) { // bare domain is not part of wildcard SAN
+        vhost = domains.makeWildcard(vhost);
+        debug(`getCertificate: will get wildcard cert for ${vhost}`);
     }
 
     const that = this;
     this.getDirectory(function (error) {
         if (error) return callback(error);
 
-        that.acmeFlow(hostname, domain, function (error) {
+        that.acmeFlow(vhost, domain, function (error) {
             if (error) return callback(error);
 
             var outdir = paths.APP_CERTS_DIR;
-            const certName = hostname.replace('*.', '_.');
+            const certName = vhost.replace('*.', '_.');
             callback(null, path.join(outdir, `${certName}.cert`), path.join(outdir, `${certName}.key`));
         });
     });
 };
 
-function getCertificate(hostname, domain, options, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+function getCertificate(vhost, domain, options, callback) {
+    assert.strictEqual(typeof vhost, 'string'); // this can also be a wildcard domain (for alias domains)
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -623,6 +623,6 @@ function getCertificate(hostname, domain, options, callback) {
         debug(`getCertificate: attempt ${attempt++}`);
 
         let acme = new Acme2(options || { });
-        acme.getCertificate(hostname, domain, retryCallback);
+        acme.getCertificate(vhost, domain, retryCallback);
     }, callback);
 }

src/cloudron.js

@@ -17,18 +17,19 @@ exports = module.exports = {
     setDashboardDomain,
     updateDashboardDomain,
     renewCerts,
+    syncDnsRecords,
 
     runSystemChecks
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     appstore = require('./appstore.js'),
     assert = require('assert'),
     async = require('async'),
     auditSource = require('./auditsource.js'),
     backups = require('./backups.js'),
     BoxError = require('./boxerror.js'),
+    branding = require('./branding.js'),
     constants = require('./constants.js'),
     cron = require('./cron.js'),
     debug = require('debug')('box:cloudron'),
@@ -42,6 +43,7 @@ var addons = require('./addons.js'),
     platform = require('./platform.js'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     spawn = require('child_process').spawn,
@@ -71,22 +73,25 @@ function uninitialize(callback) {
     ], callback);
 }
 
-function onActivated(callback) {
+function onActivated(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     // Starting the platform after a user is available means:
     // 1. mail bounces can now be sent to the cloudron owner
     // 2. the restore code path can run without sudo (since mail/ is non-root)
     async.series([
-        (done) => reverseProxy.writeDefaultConfig({ activated :true }, done), // update IP based nginx config once user is created
-        platform.start,
+        platform.start.bind(null, options),
         cron.startJobs,
-        function checkBackupConfiguration(callback) {
+        function checkBackupConfiguration(done) {
             backups.checkConfiguration(function (error, message) {
-                if (error) return callback(error);
-                notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, callback);
+                if (error) return done(error);
+                notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, done);
             });
-        }
+        },
+        // disable responding to api calls via IP to not leak domain info. this is carefully placed as the last item, so it buys
+        // the UI some time to query the dashboard domain in the restore code path
+        (done) => setTimeout(() => reverseProxy.writeDefaultConfig({ activated :true }, done), 30000)
     ], callback);
 }
 
@@ -144,7 +149,7 @@ function runStartupTasks() {
                     return reverseProxy.writeDefaultConfig({ activated: false }, callback);
                 }
 
-                onActivated(callback);
+                onActivated({}, callback);
             });
         }
     ];
@@ -173,7 +178,7 @@ function getConfig(callback) {
             version: constants.VERSION,
             isDemo: settings.isDemo(),
             cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-            footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
+            footer: branding.renderFooter(allSettings[settings.FOOTER_KEY] || constants.FOOTER),
             features: appstore.getFeatures(),
             profileLocked: allSettings[settings.DIRECTORY_CONFIG_KEY].lockUserProfiles,
             mandatory2FA: allSettings[settings.DIRECTORY_CONFIG_KEY].mandatory2FA
@@ -347,7 +352,7 @@ function updateDashboardDomain(domain, auditSource, callback) {
     setDashboardDomain(domain, auditSource, function (error) {
         if (error) return callback(error);
 
-        addons.rebuildService('turn', NOOP_CALLBACK); // to update the realm variable
+        services.rebuildService('turn', NOOP_CALLBACK); // to update the realm variable
 
         callback(null);
     });
@@ -397,3 +402,16 @@ function setupDnsAndCert(subdomain, domain, auditSource, progressCallback, callb
         });
     });
 }
+
+function syncDnsRecords(options, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    tasks.add(tasks.TASK_SYNC_DNS_RECORDS, [ options ], function (error, taskId) {
+        if (error) return callback(error);
+
+        tasks.startTask(taskId, {}, NOOP_CALLBACK);
+
+        callback(null, taskId);
+    });
+}

src/collectd/volume.ejs

@@ -0,0 +1,9 @@
+<Plugin python>
+  <Module du>
+    <Path>
+        Instance "<%= volumeId %>"
+        Dir "<%= hostPath %>"
+    </Path>
+  </Module>
+</Plugin>
+

src/constants.js

@@ -26,7 +26,7 @@ exports = module.exports = {
 
     PORT: CLOUDRON ? 3000 : 5454,
     INTERNAL_SMTP_PORT: 2525, // this value comes from the mail container
-    SYSADMIN_PORT: 3001, // unused
+    AUTHWALL_PORT: 3001,
     LDAP_PORT: 3002,
     DOCKER_PROXY_PORT: 3003,
 
@@ -37,7 +37,14 @@ exports = module.exports = {
     DEFAULT_MEMORY_LIMIT: (256 * 1024 * 1024), // see also client.js
 
     DEMO_USERNAME: 'cloudron',
-    DEMO_BLACKLISTED_APPS: [ 'com.github.cloudtorrent' ],
+    DEMO_BLACKLISTED_APPS: [
+        'com.github.cloudtorrent',
+        'net.alltubedownload.cloudronapp',
+        'com.adguard.home.cloudronapp',
+        'com.transmissionbt.cloudronapp',
+        'io.github.sickchill.cloudronapp',
+        'to.couchpota.cloudronapp'
+    ],
 
     AUTOUPDATE_PATTERN_NEVER: 'never',
 
@@ -48,8 +55,8 @@ exports = module.exports = {
 
     SUPPORT_EMAIL: 'support@cloudron.io',
 
-    FOOTER: '&copy; 2020  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
+    FOOTER: '&copy; %YEAR%  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
 
-    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '5.1.1-test'
+    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '6.0.1-test'
 };
 

src/cron.js

@@ -33,9 +33,10 @@ var appHealthMonitor = require('./apphealthmonitor.js'),
     settings = require('./settings.js'),
     system = require('./system.js'),
     updater = require('./updater.js'),
-    updateChecker = require('./updatechecker.js');
+    updateChecker = require('./updatechecker.js'),
+    _ = require('underscore');
 
-var gJobs = {
+const gJobs = {
     autoUpdater: null,
     backup: null,
     updateChecker: null,
@@ -51,7 +52,7 @@ var gJobs = {
     appHealthMonitor: null
 };
 
-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+const NOOP_CALLBACK = function (error) { if (error) debug(error); };
 
 // cron format
 // Seconds: 0-59
@@ -64,6 +65,8 @@ var NOOP_CALLBACK = function (error) { if (error) debug(error); };
 function startJobs(callback) {
     assert.strictEqual(typeof callback, 'function');
 
+    debug('startJobs: starting cron jobs');
+
     const randomTick = Math.floor(60*Math.random());
     gJobs.systemChecks = new CronJob({
         cronTime: '00 30 2 * * *', // once a day. if you change this interval, change the notification messages with correct duration
@@ -196,9 +199,10 @@ function autoupdatePatternChanged(pattern, tz) {
                 return;
             }
 
-            if (updateInfo.apps && Object.keys(updateInfo.apps).length > 0) {
-                debug('Starting app update to %j', updateInfo.apps);
-                apps.autoupdateApps(updateInfo.apps, auditSource.CRON, NOOP_CALLBACK);
+            const appUpdateInfo = _.omit(updateInfo, 'box');
+            if (Object.keys(appUpdateInfo).length > 0) {
+                debug('Starting app update to %j', appUpdateInfo);
+                apps.autoupdateApps(appUpdateInfo, auditSource.CRON, NOOP_CALLBACK);
             } else {
                 debug('No app auto updates available');
             }

src/dns/digitalocean.js

@@ -1,13 +1,13 @@
 'use strict';
 
 exports = module.exports = {
-    removePrivateFields: removePrivateFields,
-    injectPrivateFields: injectPrivateFields,
-    upsert: upsert,
-    get: get,
-    del: del,
-    wait: wait,
-    verifyDnsConfig: verifyDnsConfig
+    removePrivateFields,
+    injectPrivateFields,
+    upsert,
+    get,
+    del,
+    wait,
+    verifyDnsConfig
 };
 
 var assert = require('assert'),
@@ -69,7 +69,7 @@ function getInternal(dnsConfig, zoneName, name, type, callback) {
 
                 iteratorDone();
             });
-    }, function () { return !!nextPage; }, function (error) {
+    }, function (testDone) { return testDone(null, !!nextPage); }, function (error) {
         debug('getInternal:', error, JSON.stringify(matchingRecords));
 
         if (error) return callback(error);

src/dns/linode.js

@@ -1,13 +1,13 @@
 'use strict';
 
 exports = module.exports = {
-    removePrivateFields: removePrivateFields,
-    injectPrivateFields: injectPrivateFields,
-    upsert: upsert,
-    get: get,
-    del: del,
-    wait: wait,
-    verifyDnsConfig: verifyDnsConfig
+    removePrivateFields,
+    injectPrivateFields,
+    upsert,
+    get,
+    del,
+    wait,
+    verifyDnsConfig
 };
 
 let async = require('async'),
@@ -99,7 +99,7 @@ function getZoneRecords(dnsConfig, zoneName, name, type, callback) {
 
                     iteratorDone();
                 });
-        }, function () { return more; }, function (error) {
+        }, function (testDone) { return testDone(null, more); }, function (error) {
             debug('getZoneRecords:', error, JSON.stringify(records));
 
             if (error) return callback(error);
@@ -119,9 +119,10 @@ function get(domainObject, location, type, callback) {
         zoneName = domainObject.zoneName,
         name = domains.getName(domainObject, location, type) || '';
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { records } = result;
         var tmp = records.map(function (record) { return record.target; });
 
         debug('get: %j', tmp);
@@ -143,9 +144,10 @@ function upsert(domainObject, location, type, values, callback) {
 
     debug('upsert: %s for zone %s of type %s with values %j', name, zoneName, type, values);
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { zoneId, records } = result;
         let i = 0, recordIds = []; // used to track available records to update instead of create
 
         async.eachSeries(values, function (value, iteratorCallback) {
@@ -222,9 +224,10 @@ function del(domainObject, location, type, values, callback) {
         zoneName = domainObject.zoneName,
         name = domains.getName(domainObject, location, type) || '';
 
-    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, result) {
         if (error) return callback(error);
 
+        const { zoneId, records } = result;
         if (records.length === 0) return callback(null);
 
         var tmp = records.filter(function (record) { return values.some(function (value) { return value === record.target; }); });
@@ -287,7 +290,7 @@ function verifyDnsConfig(domainObject, callback) {
         if (error || !nameservers) return callback(new BoxError(BoxError.BAD_FIELD, error ? error.message : 'Unable to get nameservers', { field: 'nameservers' }));
 
         if (nameservers.map(function (n) { return n.toLowerCase(); }).indexOf('ns1.linode.com') === -1) {
-            debug('verifyDnsConfig: %j does not contains DO NS', nameservers);
+            debug('verifyDnsConfig: %j does not contains linode NS', nameservers);
             return callback(new BoxError(BoxError.BAD_FIELD, 'Domain nameservers are not set to Linode', { field: 'nameservers' }));
         }
 

src/dns/netcup.js

@@ -0,0 +1,303 @@
+'use strict';
+
+exports = module.exports = {
+    removePrivateFields: removePrivateFields,
+    injectPrivateFields: injectPrivateFields,
+    upsert: upsert,
+    get: get,
+    del: del,
+    wait: wait,
+    verifyDnsConfig: verifyDnsConfig
+};
+
+var assert = require('assert'),
+    BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
+    debug = require('debug')('box:dns/netcup'),
+    dns = require('../native-dns.js'),
+    domains = require('../domains.js'),
+    superagent = require('superagent'),
+    util = require('util'),
+    waitForDns = require('./waitfordns.js');
+
+var API_ENDPOINT = 'https://ccp.netcup.net/run/webservice/servers/endpoint.php?JSON';
+
+function formatError(response) {
+    if (response.body) return util.format('Netcup DNS error [%s] %s', response.body.statuscode, response.body.longmessage);
+    else return util.format('Netcup DNS error [%s] %s', response.statusCode, response.text);
+}
+
+function removePrivateFields(domainObject) {
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
+    return domainObject;
+}
+
+function injectPrivateFields(newConfig, currentConfig) {
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+}
+
+// returns a api session id
+function login(dnsConfig, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const data = {
+        action: 'login',
+        param:{
+            apikey: dnsConfig.apiKey,
+            apipassword: dnsConfig.apiPassword,
+            customernumber: dnsConfig.customerNumber
+        }
+    };
+
+    superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+        callback(null, result.body.responsedata.apisessionid);
+    });
+}
+
+function getAllRecords(dnsConfig, apiSessionId, zoneName, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof apiSessionId, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug(`getAllRecords: getting dns records of ${zoneName}`);
+
+    const data = {
+        action: 'infoDnsRecords',
+        param:{
+            apikey: dnsConfig.apiKey,
+            apisessionid: apiSessionId,
+            customernumber: dnsConfig.customerNumber,
+            domainname: zoneName,
+        }
+    };
+
+    superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+        debug('getAllRecords:', JSON.stringify(result.body.responsedata.dnsrecords || []));
+
+        callback(null, result.body.responsedata.dnsrecords || []);
+    });
+}
+
+function upsert(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('upsert: %s for zone %s of type %s with values %j', name, zoneName, type, values);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            let records = [];
+
+            values.forEach(function (value) {
+                // remove possible quotation
+                if (value.charAt(0) === '"') value = value.slice(1);
+                if (value.charAt(value.length -1) === '"') value = value.slice(0, -1);
+
+                let priority = null;
+                if (type === 'MX') {
+                    priority = parseInt(value.split(' ')[0], 10);
+                    value = value.split(' ')[1];
+                }
+
+                let record = result.find(function (r) { return r.hostname === name && r.type === type; });
+                if (!record) record = { hostname: name, type: type, destination: value, deleterecord: false };
+                else record.destination = value;
+
+                if (priority !== null) record.priority = priority;
+
+                records.push(record);
+            });
+
+            const data = {
+                action: 'updateDnsRecords',
+                param:{
+                    apikey: dnsConfig.apiKey,
+                    apisessionid: apiSessionId,
+                    customernumber: dnsConfig.customerNumber,
+                    domainname: zoneName,
+                    dnsrecordset: {
+                        dnsrecords: records
+                    }
+                }
+            };
+
+            debug('upserting', JSON.stringify(data));
+
+            superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+                if (result.body.statuscode !== 2000) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                debug('upsert:', result.body);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function get(domainObject, location, type, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('get: %s for zone %s of type %s', name, zoneName, type);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            // We only return the value string
+            callback(null, result.filter(function (r) { return r.hostname === name && r.type === type; }).map(function (r) { return r.destination; }));
+        });
+    });
+}
+
+function del(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '@';
+
+    debug('del: %s for zone %s of type %s with values %j', name, zoneName, type, values);
+
+    login(dnsConfig, function (error, apiSessionId) {
+        if (error) return callback(error);
+
+        getAllRecords(dnsConfig, apiSessionId, zoneName, function (error, result) {
+            if (error) return callback(error);
+
+            let records = [];
+
+            values.forEach(function (value) {
+                // remove possible quotation
+                if (value.charAt(0) === '"') value = value.slice(1);
+                if (value.charAt(value.length -1) === '"') value = value.slice(0, -1);
+
+                let record = result.find(function (r) { return r.hostname === name && r.type === type && r.destination === value; });
+                if (!record) return;
+
+                record.deleterecord = true;
+
+                records.push(record);
+            });
+
+            if (records.length === 0) return callback(null);
+
+            const data = {
+                action: 'updateDnsRecords',
+                param:{
+                    apikey: dnsConfig.apiKey,
+                    apisessionid: apiSessionId,
+                    customernumber: dnsConfig.customerNumber,
+                    domainname: zoneName,
+                    dnsrecordset: {
+                        dnsrecords: records
+                    }
+                }
+            };
+
+            superagent.post(API_ENDPOINT).send(data).end(function (error, result) {
+                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+                if (result.body.statuscode !== 2000) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                debug('del:', result.body.responsedata);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function wait(domainObject, location, type, value, options, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof value, 'string');
+    assert(options && typeof options === 'object'); // { interval: 5000, times: 50000 }
+    assert.strictEqual(typeof callback, 'function');
+
+    const fqdn = domains.fqdn(location, domainObject);
+
+    waitForDns(fqdn, domainObject.zoneName, type, value, options, callback);
+}
+
+function verifyDnsConfig(domainObject, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName;
+
+    if (!dnsConfig.customerNumber || typeof dnsConfig.customerNumber !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'customerNumber must be a non-empty string', { field: 'customerNumber' }));
+    if (!dnsConfig.apiKey || typeof dnsConfig.apiKey !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'apiKey must be a non-empty string', { field: 'apiKey' }));
+    if (!dnsConfig.apiPassword || typeof dnsConfig.apiPassword !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'apiPassword must be a non-empty string', { field: 'apiPassword' }));
+
+    const ip = '127.0.0.1';
+
+    var credentials = {
+        customerNumber: dnsConfig.customerNumber,
+        apiKey: dnsConfig.apiKey,
+        apiPassword: dnsConfig.apiPassword,
+    };
+
+    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here
+
+    dns.resolve(zoneName, 'NS', { timeout: 5000 }, function (error, nameservers) {
+        if (error && error.code === 'ENOTFOUND') return callback(new BoxError(BoxError.BAD_FIELD, 'Unable to resolve nameservers for this domain', { field: 'nameservers' }));
+        if (error || !nameservers) return callback(new BoxError(BoxError.BAD_FIELD, error ? error.message : 'Unable to get nameservers', { field: 'nameservers' }));
+
+        if (!nameservers.every(function (n) { return n.toLowerCase().indexOf('dns.netcup.net') !== -1; })) {
+            debug('verifyDnsConfig: %j does not contains Netcup NS', nameservers);
+            return callback(new BoxError(BoxError.BAD_FIELD, 'Domain nameservers are not set to Netcup', { field: 'nameservers' }));
+        }
+
+        const location = 'cloudrontestdns';
+
+        upsert(domainObject, location, 'A', [ ip ], function (error) {
+            if (error) return callback(error);
+
+            debug('verifyDnsConfig: Test A record added');
+
+            del(domainObject, location, 'A', [ ip ], function (error) {
+                if (error) return callback(error);
+
+                debug('verifyDnsConfig: Test A record removed again');
+
+                callback(null, credentials);
+            });
+        });
+    });
+}

src/docker.js

@@ -1,37 +1,39 @@
 'use strict';
 
 exports = module.exports = {
-    testRegistryConfig: testRegistryConfig,
-    setRegistryConfig: setRegistryConfig,
-    injectPrivateFields: injectPrivateFields,
-    removePrivateFields: removePrivateFields,
+    testRegistryConfig,
+    setRegistryConfig,
+    injectPrivateFields,
+    removePrivateFields,
 
-    ping: ping,
+    ping,
 
-    info: info,
-    downloadImage: downloadImage,
-    createContainer: createContainer,
-    startContainer: startContainer,
-    restartContainer: restartContainer,
-    stopContainer: stopContainer,
+    info,
+    downloadImage,
+    createContainer,
+    startContainer,
+    restartContainer,
+    stopContainer,
     stopContainerByName: stopContainer,
-    stopContainers: stopContainers,
-    deleteContainer: deleteContainer,
-    deleteImage: deleteImage,
-    deleteContainers: deleteContainers,
-    createSubcontainer: createSubcontainer,
-    getContainerIdByIp: getContainerIdByIp,
-    inspect: inspect,
+    stopContainers,
+    deleteContainer,
+    deleteImage,
+    deleteContainers,
+    createSubcontainer,
+    getContainerIdByIp,
+    inspect,
+    getContainerIp,
     inspectByName: inspect,
-    execContainer: execContainer,
-    getEvents: getEvents,
-    memoryUsage: memoryUsage,
-    createVolume: createVolume,
-    removeVolume: removeVolume,
-    clearVolume: clearVolume
+    execContainer,
+    getEvents,
+    memoryUsage,
+    createVolume,
+    removeVolume,
+    clearVolume,
+    update
 };
 
-var addons = require('./addons.js'),
+const apps = require('./apps.js'),
     async = require('async'),
     assert = require('assert'),
     BoxError = require('./boxerror.js'),
@@ -39,11 +41,16 @@ var addons = require('./addons.js'),
     constants = require('./constants.js'),
     debug = require('debug')('box:docker'),
     Docker = require('dockerode'),
+    os = require('os'),
     path = require('path'),
+    reverseProxy = require('./reverseproxy.js'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     safe = require('safetydance'),
+    system = require('./system.js'),
     util = require('util'),
+    volumes = require('./volumes.js'),
     _ = require('underscore');
 
 const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
@@ -52,11 +59,13 @@ const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
 const DOCKER_SOCKET_PATH = '/var/run/docker.sock';
 const gConnection = new Docker({ socketPath: DOCKER_SOCKET_PATH });
 
-function testRegistryConfig(auth, callback) {
-    assert.strictEqual(typeof auth, 'object');
+function testRegistryConfig(config, callback) {
+    assert.strictEqual(typeof config, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    gConnection.checkAuth(auth, function (error /*, data */) { // this returns a 500 even for auth errors
+    if (config.provider === 'noop') return callback();
+
+    gConnection.checkAuth(config, function (error /*, data */) { // this returns a 500 even for auth errors
         if (error) return callback(new BoxError(BoxError.BAD_FIELD, error, { field: 'serverAddress' }));
 
         callback();
@@ -75,14 +84,14 @@ function removePrivateFields(registryConfig) {
     return registryConfig;
 }
 
-function setRegistryConfig(auth, callback) {
-    assert.strictEqual(typeof auth, 'object');
+function setRegistryConfig(config, callback) {
+    assert.strictEqual(typeof config, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    const isLogin = !!auth.password;
+    const isLogin = !!config.password;
 
     // currently, auth info is not stashed in the db but maybe it should for restore to work?
-    const cmd = isLogin ? `docker login ${auth.serverAddress} --username ${auth.username} --password ${auth.password}` : `docker logout ${auth.serverAddress}`;
+    const cmd = isLogin ? `docker login ${config.serverAddress} --username ${config.username} --password ${config.password}` : `docker logout ${config.serverAddress}`;
 
     child_process.exec(cmd, { }, function (error /*, stdout, stderr */) {
         if (error) return callback(new BoxError(BoxError.ACCESS_DENIED, error.message));
@@ -128,12 +137,12 @@ function getRegistryConfig(image, callback) {
 }
 
 function pullImage(manifest, callback) {
-    getRegistryConfig(manifest.dockerImage, function (error, authConfig) {
+    getRegistryConfig(manifest.dockerImage, function (error, config) {
         if (error) return callback(error);
 
-        debug(`pullImage: will pull ${manifest.dockerImage}. auth: ${authConfig ? 'yes' : 'no'}`);
+        debug(`pullImage: will pull ${manifest.dockerImage}. auth: ${config ? 'yes' : 'no'}`);
 
-        gConnection.pull(manifest.dockerImage, { authconfig: authConfig }, function (error, stream) {
+        gConnection.pull(manifest.dockerImage, { authconfig: config }, function (error, stream) {
             if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND, `Unable to pull image ${manifest.dockerImage}. message: ${error.message} statusCode: ${error.statusCode}`));
             if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, `Unable to pull image ${manifest.dockerImage}. Please check the network or if the image needs authentication. statusCode: ${error.statusCode}`));
 
@@ -170,26 +179,125 @@ function downloadImage(manifest, callback) {
 
     debug('downloadImage %s', manifest.dockerImage);
 
-    var attempt = 1;
+    const image = gConnection.getImage(manifest.dockerImage);
 
-    async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
-        debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+    image.inspect(function (error, result) {
+        if (!error && result) return callback(null); // image is already present locally
 
-        pullImage(manifest, retryCallback);
-    }, callback);
+        let attempt = 1;
+
+        async.retry({ times: 10, interval: 5000, errorFilter: e => e.reason !== BoxError.NOT_FOUND }, function (retryCallback) {
+            debug('Downloading image %s. attempt: %s', manifest.dockerImage, attempt++);
+
+            pullImage(manifest, retryCallback);
+        }, callback);
+    });
 }
 
-function getBindsSync(app) {
+function getVolumeMounts(app, callback) {
     assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
 
-    let binds = [];
+    let mounts = [];
 
-    for (let name of Object.keys(app.binds)) {
-        const bind = app.binds[name];
-        binds.push(`${bind.hostPath}:/media/${name}:${bind.readOnly ? 'ro' : 'rw'}`);
+    if (app.mounts.length === 0) return callback(null, []);
+
+    volumes.list(function (error, result) {
+        if (error) return callback(error);
+
+        let volumesById = {};
+        result.forEach(r => volumesById[r.id] = r);
+
+        for (const mount of app.mounts) {
+            const volume = volumesById[mount.volumeId];
+
+            mounts.push({
+                Source: volume.hostPath,
+                Target: `/media/${volume.name}`,
+                Type: 'bind',
+                ReadOnly: mount.readOnly
+            });
+        }
+
+        callback(null, mounts);
+    });
+}
+
+function getAddonMounts(app, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    let mounts = [];
+
+    const addons = app.manifest.addons;
+    if (!addons) return callback(null, mounts);
+
+    async.eachSeries(Object.keys(addons), function (addon, iteratorDone) {
+        switch (addon) {
+        case 'localstorage':
+            mounts.push({
+                Target: '/app/data',
+                Source: `${app.id}-localstorage`,
+                Type: 'volume',
+                ReadOnly: false
+            });
+
+            return iteratorDone();
+        case 'tls':
+            reverseProxy.getCertificate(app.fqdn, app.domain, function (error, bundle) {
+                if (error) return iteratorDone(error);
+
+                mounts.push({
+                    Target: '/etc/certs/tls_cert.pem',
+                    Source: bundle.certFilePath,
+                    Type: 'bind',
+                    ReadOnly: true
+                });
+
+                mounts.push({
+                    Target: '/etc/certs/tls_key.pem',
+                    Source: bundle.keyFilePath,
+                    Type: 'bind',
+                    ReadOnly: true
+                });
+
+                iteratorDone();
+            });
+
+            return;
+        default:
+            iteratorDone();
+        }
+    }, function (error) {
+        callback(error, mounts);
+    });
+}
+
+function getMounts(app, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    getVolumeMounts(app, function (error, volumeMounts) {
+        if (error) return callback(error);
+
+        getAddonMounts(app, function (error, addonMounts) {
+            if (error) return callback(error);
+
+            callback(null, volumeMounts.concat(addonMounts));
+        });
+    });
+}
+
+function getLowerUpIp() { // see getifaddrs and IFF_LOWER_UP and netdevice
+    const ni = os.networkInterfaces(); // { lo: [], eth0: [] }
+    for (const iname of Object.keys(ni)) {
+        if (iname === 'lo') continue;
+        for (const address of ni[iname]) {
+            if (!address.internal && address.family === 'IPv4') return address.address;
+        }
     }
 
-    return binds;
+    return null;
 }
 
 function createSubcontainer(app, name, cmd, options, callback) {
@@ -217,11 +325,6 @@ function createSubcontainer(app, name, cmd, options, callback) {
         `${envPrefix}APP_DOMAIN=${domain}`
     ];
 
-    // docker portBindings requires ports to be exposed
-    exposedPorts[manifest.httpPort + '/tcp'] = {};
-
-    dockerPortBindings[manifest.httpPort + '/tcp'] = [ { HostIp: '127.0.0.1', HostPort: app.httpPort + '' } ];
-
     var portEnv = [];
     for (let portName in app.portBindings) {
         const hostPort = app.portBindings[portName];
@@ -230,119 +333,120 @@ function createSubcontainer(app, name, cmd, options, callback) {
 
         var containerPort = ports[portName].containerPort || hostPort;
 
+        // docker portBindings requires ports to be exposed
         exposedPorts[`${containerPort}/${portType}`] = {};
         portEnv.push(`${portName}=${hostPort}`);
 
-        dockerPortBindings[`${containerPort}/${portType}`] = [ { HostIp: '0.0.0.0', HostPort: hostPort + '' } ];
+        const hostIp = hostPort === 53 ? getLowerUpIp() : '0.0.0.0'; // port 53 is special because it is possibly taken by systemd-resolved
+        dockerPortBindings[`${containerPort}/${portType}`] = [ { HostIp: hostIp, HostPort: hostPort + '' } ];
     }
 
     let appEnv = [];
     Object.keys(app.env).forEach(function (name) { appEnv.push(`${name}=${app.env[name]}`); });
 
-    // first check db record, then manifest
-    var memoryLimit = app.memoryLimit || manifest.memoryLimit || 0;
-
-    if (memoryLimit === -1) { // unrestricted
-        memoryLimit = 0;
-    } else if (memoryLimit === 0 || memoryLimit < constants.DEFAULT_MEMORY_LIMIT) { // ensure we never go below minimum (in case we change the default)
-        memoryLimit = constants.DEFAULT_MEMORY_LIMIT;
-    }
+    let memoryLimit = apps.getMemoryLimit(app);
 
     // give scheduler tasks twice the memory limit since background jobs take more memory
     // if required, we can make this a manifest and runtime argument later
     if (!isAppContainer) memoryLimit *= 2;
 
-    addons.getEnvironment(app, function (error, addonEnv) {
+    services.getEnvironment(app, function (error, addonEnv) {
         if (error) return callback(error);
 
-        let containerOptions = {
-            name: name, // for referencing containers
-            Tty: isAppContainer,
-            Image: app.manifest.dockerImage,
-            Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
-            Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
-            ExposedPorts: isAppContainer ? exposedPorts : { },
-            Volumes: { // see also ReadonlyRootfs
-                '/tmp': {},
-                '/run': {}
-            },
-            Labels: {
-                'fqdn': app.fqdn,
-                'appId': app.id,
-                'isSubcontainer': String(!isAppContainer),
-                'isCloudronManaged': String(true)
-            },
-            HostConfig: {
-                Mounts: addons.getMountsSync(app, app.manifest.addons),
-                Binds: getBindsSync(app), // ideally, we have to use 'Mounts' but we have to create volumes then
-                LogConfig: {
-                    Type: 'syslog',
-                    Config: {
-                        'tag': app.id,
-                        'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
-                        'syslog-format': 'rfc5424'
-                    }
-                },
-                Memory: memoryLimit / 2,
-                MemorySwap: memoryLimit, // Memory + Swap
-                PortBindings: isAppContainer ? dockerPortBindings : { },
-                PublishAllPorts: false,
-                ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
-                RestartPolicy: {
-                    'Name': isAppContainer ? 'unless-stopped' : 'no',
-                    'MaximumRetryCount': 0
-                },
-                CpuShares: app.cpuShares,
-                VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
-                SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
-                CapAdd: [],
-                CapDrop: []
-            }
-        };
+        getMounts(app, function (error, mounts) {
+            if (error) return callback(error);
 
-        // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
-        // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
-        // name to look up the internal docker ip. this makes curl from within container fail
-        // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
-        // Hostname cannot be set with container NetworkMode. Subcontainers run is the network space of the app container
-        // This is done to prevent lots of up/down events and iptables locking
-        if (isAppContainer) {
-            containerOptions.Hostname = app.id;
-            containerOptions.HostConfig.NetworkMode = 'cloudron'; // user defined bridge network
-            containerOptions.HostConfig.Dns = ['172.18.0.1']; // use internal dns
-            containerOptions.HostConfig.DnsSearch = ['.']; // use internal dns
-
-            containerOptions.NetworkingConfig = {
-                EndpointsConfig: {
-                    cloudron: {
-                        Aliases: [ name ] // adds hostname entry with container name
-                    }
+            let containerOptions = {
+                name: name, // for referencing containers
+                Tty: isAppContainer,
+                Image: app.manifest.dockerImage,
+                Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
+                Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
+                ExposedPorts: isAppContainer ? exposedPorts : { },
+                Volumes: { // see also ReadonlyRootfs
+                    '/tmp': {},
+                    '/run': {}
+                },
+                Labels: {
+                    'fqdn': app.fqdn,
+                    'appId': app.id,
+                    'isSubcontainer': String(!isAppContainer),
+                    'isCloudronManaged': String(true)
+                },
+                HostConfig: {
+                    Mounts: mounts,
+                    LogConfig: {
+                        Type: 'syslog',
+                        Config: {
+                            'tag': app.id,
+                            'syslog-address': 'udp://127.0.0.1:2514', // see apps.js:validatePortBindings()
+                            'syslog-format': 'rfc5424'
+                        }
+                    },
+                    Memory: system.getMemoryAllocation(memoryLimit),
+                    MemorySwap: memoryLimit, // Memory + Swap
+                    PortBindings: isAppContainer ? dockerPortBindings : { },
+                    PublishAllPorts: false,
+                    ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
+                    RestartPolicy: {
+                        'Name': isAppContainer ? 'unless-stopped' : 'no',
+                        'MaximumRetryCount': 0
+                    },
+                    CpuShares: app.cpuShares,
+                    VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
+                    SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
+                    CapAdd: [],
+                    CapDrop: []
                 }
             };
-        } else {
-            containerOptions.HostConfig.NetworkMode = `container:${app.containerId}`;
-        }
 
-        var capabilities = manifest.capabilities || [];
+            // do no set hostname of containers to location as it might conflict with addons names. for example, an app installed in mail
+            // location may not reach mail container anymore by DNS. We cannot set hostname to fqdn either as that sets up the dns
+            // name to look up the internal docker ip. this makes curl from within container fail
+            // Note that Hostname has no effect on DNS. We have to use the --net-alias for dns.
+            // Hostname cannot be set with container NetworkMode. Subcontainers run is the network space of the app container
+            // This is done to prevent lots of up/down events and iptables locking
+            if (isAppContainer) {
+                containerOptions.Hostname = app.id;
+                containerOptions.HostConfig.NetworkMode = 'cloudron'; // user defined bridge network
+                containerOptions.HostConfig.Dns = ['172.18.0.1']; // use internal dns
+                containerOptions.HostConfig.DnsSearch = ['.']; // use internal dns
 
-        // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
-        if (capabilities.includes('net_admin')) containerOptions.HostConfig.CapAdd.push('NET_ADMIN', 'NET_RAW');
-        if (capabilities.includes('mlock')) containerOptions.HostConfig.CapAdd.push('IPC_LOCK'); // mlock prevents swapping
-        if (!capabilities.includes('ping')) containerOptions.HostConfig.CapDrop.push('NET_RAW'); // NET_RAW is included by default by Docker
+                containerOptions.NetworkingConfig = {
+                    EndpointsConfig: {
+                        cloudron: {
+                            IPAMConfig: {
+                                IPv4Address: app.containerIp
+                            },
+                            Aliases: [ name ] // adds hostname entry with container name
+                        }
+                    }
+                };
+            } else {
+                containerOptions.HostConfig.NetworkMode = `container:${app.containerId}`; // scheduler containers must have same IP as app for various addon auth
+            }
 
-        if (capabilities.includes('vaapi') && safe.fs.existsSync('/dev/dri')) {
-            containerOptions.HostConfig.Devices = [
-                { PathOnHost: '/dev/dri', PathInContainer: '/dev/dri', CgroupPermissions: 'rwm' }
-            ];
-        }
+            var capabilities = manifest.capabilities || [];
 
-        containerOptions = _.extend(containerOptions, options);
+            // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
+            if (capabilities.includes('net_admin')) containerOptions.HostConfig.CapAdd.push('NET_ADMIN', 'NET_RAW');
+            if (capabilities.includes('mlock')) containerOptions.HostConfig.CapAdd.push('IPC_LOCK'); // mlock prevents swapping
+            if (!capabilities.includes('ping')) containerOptions.HostConfig.CapDrop.push('NET_RAW'); // NET_RAW is included by default by Docker
 
-        gConnection.createContainer(containerOptions, function (error, container) {
-            if (error && error.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
-            if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+            if (capabilities.includes('vaapi') && safe.fs.existsSync('/dev/dri')) {
+                containerOptions.HostConfig.Devices = [
+                    { PathOnHost: '/dev/dri', PathInContainer: '/dev/dri', CgroupPermissions: 'rwm' }
+                ];
+            }
 
-            callback(null, container);
+            containerOptions = _.extend(containerOptions, options);
+
+            gConnection.createContainer(containerOptions, function (error, container) {
+                if (error && error.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
+                if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
+
+                callback(null, container);
+            });
         });
     });
 }
@@ -526,6 +630,22 @@ function inspect(containerId, callback) {
     });
 }
 
+function getContainerIp(containerId, callback) {
+    assert.strictEqual(typeof containerId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (constants.TEST) return callback(null, '127.0.5.5');
+
+    inspect(containerId, function (error, result) {
+        if (error) return callback(error);
+
+        const ip = safe.query(result, 'NetworkSettings.Networks.cloudron.IPAddress', null);
+        if (!ip) return callback(new BoxError(BoxError.DOCKER_ERROR, 'Error getting container IP'));
+
+        callback(null, ip);
+    });
+}
+
 function execContainer(containerId, options, callback) {
     assert.strictEqual(typeof containerId, 'string');
     assert.strictEqual(typeof options, 'object');
@@ -648,3 +768,17 @@ function info(callback) {
         callback(null, result);
     });
 }
+
+function update(name, memory, memorySwap, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof memory, 'number');
+    assert.strictEqual(typeof memorySwap, 'number');
+    assert.strictEqual(typeof callback, 'function');
+
+    const args = `update --memory ${memory} --memory-swap ${memorySwap} ${name}`.split(' ');
+    // scale back db containers, if possible. this is retried because updating memory constraints can fail
+    // with failed to write to memory.memsw.limit_in_bytes: write /sys/fs/cgroup/memory/docker/xx/memory.memsw.limit_in_bytes: device or resource busy
+    async.retry({ times: 10, interval: 60 * 1000 }, function (retryCallback) {
+        shell.spawn(`update(${name})`, '/usr/bin/docker', args, { }, retryCallback);
+    }, callback);
+}

src/domaindb.js

@@ -16,14 +16,18 @@ var assert = require('assert'),
     database = require('./database.js'),
     safe = require('safetydance');
 
-var DOMAINS_FIELDS = [ 'domain', 'zoneName', 'provider', 'configJson', 'tlsConfigJson' ].join(',');
+var DOMAINS_FIELDS = [ 'domain', 'zoneName', 'provider', 'configJson', 'tlsConfigJson', 'wellKnownJson' ].join(',');
 
 function postProcess(data) {
     data.config = safe.JSON.parse(data.configJson);
-    data.tlsConfig = safe.JSON.parse(data.tlsConfigJson);
     delete data.configJson;
+
+    data.tlsConfig = safe.JSON.parse(data.tlsConfigJson);
     delete data.tlsConfigJson;
 
+    data.wellKnown = safe.JSON.parse(data.wellKnownJson);
+    delete data.wellKnownJson;
+
     return data;
 }
 
@@ -86,6 +90,9 @@ function update(name, domain, callback) {
         } else if (k === 'tlsConfig') {
             fields.push('tlsConfigJson = ?');
             args.push(JSON.stringify(domain[k]));
+        } else if (k === 'wellKnown') {
+            fields.push('wellKnownJson = ?');
+            args.push(JSON.stringify(domain[k]));
         } else {
             fields.push(k + ' = ?');
             args.push(domain[k]);

src/domains.js

@@ -1,37 +1,44 @@
 'use strict';
 
 module.exports = exports = {
-    add: add,
-    get: get,
-    getAll: getAll,
-    update: update,
-    del: del,
-    clear: clear,
+    add,
+    get,
+    getAll,
+    update,
+    del,
+    clear,
 
-    fqdn: fqdn,
-    getName: getName,
+    fqdn,
+    getName,
 
-    getDnsRecords: getDnsRecords,
-    upsertDnsRecords: upsertDnsRecords,
-    removeDnsRecords: removeDnsRecords,
+    getDnsRecords,
+    upsertDnsRecords,
+    removeDnsRecords,
 
-    waitForDnsRecord: waitForDnsRecord,
+    waitForDnsRecord,
 
-    removePrivateFields: removePrivateFields,
-    removeRestrictedFields: removeRestrictedFields,
+    removePrivateFields,
+    removeRestrictedFields,
 
-    validateHostname: validateHostname,
+    validateHostname,
 
-    makeWildcard: makeWildcard,
+    makeWildcard,
 
-    parentDomain: parentDomain,
+    parentDomain,
 
-    checkDnsRecords: checkDnsRecords
+    registerLocations,
+    unregisterLocations,
+
+    checkDnsRecords,
+    syncDnsRecords
 };
 
-var assert = require('assert'),
+const apps = require('./apps.js'),
+    assert = require('assert'),
+    async = require('async'),
     BoxError = require('./boxerror.js'),
     constants = require('./constants.js'),
+    crypto = require('crypto'),
     debug = require('debug')('box:domains'),
     domaindb = require('./domaindb.js'),
     eventlog = require('./eventlog.js'),
@@ -60,6 +67,7 @@ function api(provider) {
     case 'linode': return require('./dns/linode.js');
     case 'namecom': return require('./dns/namecom.js');
     case 'namecheap': return require('./dns/namecheap.js');
+    case 'netcup': return require('./dns/netcup.js');
     case 'noop': return require('./dns/noop.js');
     case 'manual': return require('./dns/manual.js');
     case 'wildcard': return require('./dns/wildcard.js');
@@ -152,6 +160,12 @@ function validateTlsConfig(tlsConfig, dnsProvider) {
     return null;
 }
 
+function validateWellKnown(wellKnown) {
+    assert.strictEqual(typeof wellKnown, 'object');
+
+    return null;
+}
+
 function add(domain, data, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof data.zoneName, 'string');
@@ -178,13 +192,17 @@ function add(domain, data, auditSource, callback) {
         if (error) return callback(error);
     } else {
         fallbackCertificate = reverseProxy.generateFallbackCertificateSync({ domain, config });
-        if (fallbackCertificate.error) return callback(error);
+        if (fallbackCertificate.error) return callback(fallbackCertificate.error);
     }
 
     let error = validateTlsConfig(tlsConfig, provider);
     if (error) return callback(error);
 
-    if (!dkimSelector) dkimSelector = 'cloudron-' + settings.adminDomain().replace(/\./g, '');
+    if (!dkimSelector) {
+        // create a unique suffix. this lets one add this domain can be added in another cloudron instance and not have their dkim selector conflict
+        const suffix = crypto.createHash('sha256').update(settings.adminDomain()).digest('hex').substr(0, 6);
+        dkimSelector = `cloudron-${suffix}`;
+    }
 
     verifyDnsConfig(config, domain, zoneName, provider, function (error, sanitizedConfig) {
         if (error) return callback(error);
@@ -246,7 +264,7 @@ function update(domain, data, auditSource, callback) {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let { zoneName, provider, config, fallbackCertificate, tlsConfig } = data;
+    let { zoneName, provider, config, fallbackCertificate, tlsConfig, wellKnown } = data;
 
     if (settings.isDemo() && (domain === settings.adminDomain())) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
 
@@ -267,6 +285,9 @@ function update(domain, data, auditSource, callback) {
         error = validateTlsConfig(tlsConfig, provider);
         if (error) return callback(error);
 
+        error = validateWellKnown(wellKnown, provider);
+        if (error) return callback(error);
+
         if (provider === domainObject.provider) api(provider).injectPrivateFields(config, domainObject.config);
 
         verifyDnsConfig(config, domain, zoneName, provider, function (error, sanitizedConfig) {
@@ -274,9 +295,10 @@ function update(domain, data, auditSource, callback) {
 
             let newData = {
                 config: sanitizedConfig,
-                zoneName: zoneName,
-                provider: provider,
-                tlsConfig: tlsConfig
+                zoneName,
+                provider,
+                tlsConfig,
+                wellKnown
             };
 
             domaindb.update(domain, newData, function (error) {
@@ -302,7 +324,7 @@ function del(domain, auditSource, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     if (domain === settings.adminDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove admin domain'));
-    if (domain === settings.mailDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove mail domain'));
+    if (domain === settings.mailDomain()) return callback(new BoxError(BoxError.CONFLICT, 'Cannot remove mail domain. Change the mail server location first'));
 
     domaindb.del(domain, function (error) {
         if (error) return callback(error);
@@ -326,6 +348,7 @@ function clear(callback) {
 }
 
 // returns the 'name' that needs to be inserted into zone
+// eslint-disable-next-line no-unused-vars
 function getName(domain, location, type) {
     const part = domain.domain.slice(0, -domain.zoneName.length - 1);
 
@@ -431,7 +454,7 @@ function waitForDnsRecord(location, domain, type, value, options, callback) {
 
 // removes all fields that are strictly private and should never be returned by API calls
 function removePrivateFields(domain) {
-    var result = _.pick(domain, 'domain', 'zoneName', 'provider', 'config', 'tlsConfig', 'fallbackCertificate');
+    var result = _.pick(domain, 'domain', 'zoneName', 'provider', 'config', 'tlsConfig', 'fallbackCertificate', 'wellKnown');
     return api(result.provider).removePrivateFields(result);
 }
 
@@ -444,10 +467,134 @@ function removeRestrictedFields(domain) {
     return result;
 }
 
-function makeWildcard(hostname) {
-    assert.strictEqual(typeof hostname, 'string');
+function makeWildcard(vhost) {
+    assert.strictEqual(typeof vhost, 'string');
 
-    let parts = hostname.split('.');
+    // if the vhost is like *.example.com, this function will do nothing
+    let parts = vhost.split('.');
     parts[0] = '*';
     return parts.join('.');
 }
+
+function registerLocations(locations, options, progressCallback, callback) {
+    assert(Array.isArray(locations));
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug(`registerLocations: Will register ${JSON.stringify(locations)} with options ${JSON.stringify(options)}`);
+
+    const overwriteDns = options.overwriteDns || false;
+
+    sysinfo.getServerIp(function (error, ip) {
+        if (error) return callback(error);
+
+        async.eachSeries(locations, function (location, iteratorDone) {
+            async.retry({ times: 200, interval: 5000 }, function (retryCallback) {
+                progressCallback({ message: `Registering location: ${location.subdomain ? (location.subdomain + '.') : ''}${location.domain}` });
+
+                // get the current record before updating it
+                getDnsRecords(location.subdomain, location.domain, 'A', function (error, values) {
+                    if (error && error.reason === BoxError.EXTERNAL_ERROR) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location })); // try again
+                    if (error && error.reason === BoxError.ACCESS_DENIED) return retryCallback(null, new BoxError(BoxError.ACCESS_DENIED, error.message, { domain: location }));
+                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, new BoxError(BoxError.NOT_FOUND, error.message, { domain: location }));
+                    if (error) return retryCallback(null, new BoxError(BoxError.EXTERNAL_ERROR, error.message, location)); // give up for other errors
+
+                    if (values.length !== 0 && values[0] === ip) return retryCallback(null); // up-to-date
+
+                    // refuse to update any existing DNS record for custom domains that we did not create
+                    if (values.length !== 0 && !overwriteDns) return retryCallback(null, new BoxError(BoxError.ALREADY_EXISTS, 'DNS Record already exists', { domain: location }));
+
+                    upsertDnsRecords(location.subdomain, location.domain, 'A', [ ip ], function (error) {
+                        if (error && (error.reason === BoxError.BUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
+                            progressCallback({ message: `registerSubdomains: Upsert error. Will retry. ${error.message}` });
+                            return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location })); // try again
+                        }
+
+                        retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, location) : null);
+                    });
+                });
+            }, function (error, result) {
+                if (error || result) return iteratorDone(error || result);
+
+                iteratorDone(null);
+            });
+        }, callback);
+    });
+}
+
+function unregisterLocations(locations, progressCallback, callback) {
+    assert(Array.isArray(locations));
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    sysinfo.getServerIp(function (error, ip) {
+        if (error) return callback(error);
+
+        async.eachSeries(locations, function (location, iteratorDone) {
+            async.retry({ times: 30, interval: 5000 }, function (retryCallback) {
+                progressCallback({ message: `Unregistering location: ${location.subdomain ? (location.subdomain + '.') : ''}${location.domain}` });
+
+                removeDnsRecords(location.subdomain, location.domain, 'A', [ ip ], function (error) {
+                    if (error && error.reason === BoxError.NOT_FOUND) return retryCallback(null, null);
+                    if (error && (error.reason === BoxError.SBUSY || error.reason === BoxError.EXTERNAL_ERROR)) {
+                        progressCallback({ message: `Error unregistering location. Will retry. ${error.message}`});
+                        return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location })); // try again
+                    }
+
+                    retryCallback(null, error ? new BoxError(BoxError.EXTERNAL_ERROR, error.message, { domain: location }) : null);
+                });
+            }, function (error, result) {
+                if (error || result) return iteratorDone(error || result);
+
+                iteratorDone();
+            });
+        }, callback);
+    });
+}
+
+function syncDnsRecords(options, progressCallback, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (options.domain && options.type === 'mail') return mail.setDnsRecords(options.domain, callback);
+
+    getAll(function (error, domains) {
+        if (error) return callback(error);
+
+        if (options.domain) domains = domains.filter(d => d.domain === options.domain);
+
+        const mailSubdomain = settings.mailFqdn().substr(0, settings.mailFqdn().length - settings.mailDomain().length - 1);
+
+        apps.getAll(function (error, allApps) {
+            if (error) return callback(error);
+
+            let progress = 1, errors = [];
+
+            // we sync by domain only to get some nice progress
+            async.eachSeries(domains, function (domain, iteratorDone) {
+                progressCallback({ percent: progress, message: `Updating DNS of ${domain.domain}`});
+                progress += Math.round(100/(1+domains.length));
+
+                let locations = [];
+                if (domain.domain === settings.adminDomain()) locations.push({ subdomain: constants.ADMIN_LOCATION, domain: settings.adminDomain() });
+                if (domain.domain === settings.mailDomain() && settings.mailFqdn() !== settings.adminFqdn()) locations.push({ subdomain: mailSubdomain, domain: settings.mailDomain() });
+
+                allApps.forEach(function (app) {
+                    const appLocations = [{ subdomain: app.location, domain: app.domain }].concat(app.alternateDomains).concat(app.aliasDomains);
+                    locations = locations.concat(appLocations.filter(al => al.domain === domain.domain));
+                });
+
+                async.series([
+                    registerLocations.bind(null, locations, { overwriteDns: true }, progressCallback),
+                    progressCallback.bind(null, { message: `Updating mail DNS of ${domain.domain}`}),
+                    mail.setDnsRecords.bind(null, domain.domain)
+                ], function (error) {
+                    if (error) errors.push({ domain: domain.domain, message: error.message });
+                    iteratorDone();
+                });
+            }, () => callback(null, { errors }));
+        });
+    });
+}

src/eventlog.js

@@ -1,11 +1,12 @@
 'use strict';
 
 exports = module.exports = {
-    add: add,
-    get: get,
-    getAllPaged: getAllPaged,
-    getByCreationTime: getByCreationTime,
-    cleanup: cleanup,
+    add,
+    upsert,
+    get,
+    getAllPaged,
+    getByCreationTime,
+    cleanup,
 
     // keep in sync with webadmin index.js filter
     ACTION_ACTIVATE: 'cloudron.activate',
@@ -57,10 +58,15 @@ exports = module.exports = {
 
     ACTION_USER_ADD: 'user.add',
     ACTION_USER_LOGIN: 'user.login',
+    ACTION_USER_LOGOUT: 'user.logout',
     ACTION_USER_REMOVE: 'user.remove',
     ACTION_USER_UPDATE: 'user.update',
     ACTION_USER_TRANSFER: 'user.transfer',
 
+    ACTION_VOLUME_ADD: 'volume.add',
+    ACTION_VOLUME_UPDATE: 'volume.update',
+    ACTION_VOLUME_REMOVE: 'volume.remove',
+
     ACTION_DYNDNS_UPDATE: 'dyndns.update',
 
     ACTION_SUPPORT_TICKET: 'support.ticket',
@@ -86,9 +92,24 @@ function add(action, source, data, callback) {
 
     callback = callback || NOOP_CALLBACK;
 
-    // we do only daily upserts for login actions, so they don't spam the db
-    var api = action === exports.ACTION_USER_LOGIN ? eventlogdb.upsert : eventlogdb.add;
-    api(uuid.v4(), action, source, data, function (error, id) {
+    eventlogdb.add(uuid.v4(), action, source, data, function (error, id) {
+        if (error) return callback(error);
+
+        callback(null, { id: id });
+
+        notifications.onEvent(id, action, source, data, NOOP_CALLBACK);
+    });
+}
+
+function upsert(action, source, data, callback) {
+    assert.strictEqual(typeof action, 'string');
+    assert.strictEqual(typeof source, 'object');
+    assert.strictEqual(typeof data, 'object');
+    assert(!callback || typeof callback === 'function');
+
+    callback = callback || NOOP_CALLBACK;
+
+    eventlogdb.upsert(uuid.v4(), action, source, data, function (error, id) {
         if (error) return callback(error);
 
         callback(null, { id: id });

src/graphite.js

@@ -1,23 +1,27 @@
 'use strict';
 
 exports = module.exports = {
-    startGraphite: startGraphite
+    start,
+
+    DEFAULT_MEMORY_LIMIT: 256 * 1024 * 1024
 };
 
 var assert = require('assert'),
     async = require('async'),
     infra = require('./infra_version.js'),
     paths = require('./paths.js'),
-    shell = require('./shell.js');
+    shell = require('./shell.js'),
+    system = require('./system.js');
 
-function startGraphite(existingInfra, callback) {
+function start(existingInfra, serviceConfig, callback) {
     assert.strictEqual(typeof existingInfra, 'object');
+    assert.strictEqual(typeof serviceConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     const tag = infra.images.graphite.tag;
     const dataDir = paths.PLATFORM_DATA_DIR;
-
-    if (existingInfra.version === infra.version && infra.images.graphite.tag === existingInfra.images.graphite.tag) return callback();
+    const memoryLimit = serviceConfig.memoryLimit || exports.DEFAULT_MEMORY_LIMIT;
+    const memory = system.getMemoryAllocation(memoryLimit);
 
     const cmd = `docker run --restart=always -d --name="graphite" \
                 --hostname graphite \
@@ -27,8 +31,8 @@ function startGraphite(existingInfra, callback) {
                 --log-opt syslog-address=udp://127.0.0.1:2514 \
                 --log-opt syslog-format=rfc5424 \
                 --log-opt tag=graphite \
-                -m 150m \
-                --memory-swap 150m \
+                -m ${memory} \
+                --memory-swap ${memoryLimit} \
                 --dns 172.18.0.1 \
                 --dns-search=. \
                 -p 127.0.0.1:2003:2003 \

src/groupdb.js

@@ -196,6 +196,7 @@ function setMembers(groupId, userIds, callback) {
 
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, 'Group not found'));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.CONFLICT, 'Duplicate member in list'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(error);
@@ -227,6 +228,7 @@ function setMembership(userId, groupIds, callback) {
 
     database.transaction(queries, function (error) {
         if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, error.message));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.CONFLICT, 'Already member'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
         callback(null);

src/groups.js

@@ -1,25 +1,25 @@
 'use strict';
 
 exports = module.exports = {
-    create: create,
-    remove: remove,
-    get: get,
-    getByName: getByName,
-    update: update,
-    getWithMembers: getWithMembers,
-    getAll: getAll,
-    getAllWithMembers: getAllWithMembers,
+    create,
+    remove,
+    get,
+    getByName,
+    update,
+    getWithMembers,
+    getAll,
+    getAllWithMembers,
 
-    getMembers: getMembers,
-    addMember: addMember,
-    setMembers: setMembers,
-    removeMember: removeMember,
-    isMember: isMember,
+    getMembers,
+    addMember,
+    setMembers,
+    removeMember,
+    isMember,
 
-    setMembership: setMembership,
-    getMembership: getMembership,
+    setMembership,
+    getMembership,
 
-    count: count
+    count
 };
 
 var assert = require('assert'),

src/infra_version.js

@@ -6,22 +6,22 @@
 
 exports = module.exports = {
     // a version change recreates all containers with latest docker config
-    'version': '48.17.1',
+    'version': '48.18.0',
 
     'baseImages': [
-        { repo: 'cloudron/base', tag: 'cloudron/base:2.0.0@sha256:f9fea80513aa7c92fe2e7bf3978b54c8ac5222f47a9a32a7f8833edf0eb5a4f4' }
+        { repo: 'cloudron/base', tag: 'cloudron/base:3.0.0@sha256:455c70428723e3a823198c57472785437eb6eab082e79b3ff04ea584faf46e92' }
     ],
 
     // a major version bump in the db containers will trigger the restore logic that uses the db dumps
     // docker inspect --format='{{index .RepoDigests 0}}' $IMAGE to get the sha256
     'images': {
-        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.1.0@sha256:e1dd22aa6eef5beb7339834b200a8bb787ffc2264ce11139857a054108fefb4f' },
-        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.3.2@sha256:dd624870c7f8ba9b2759f93ce740d1e092a1ac4b2d6af5007a01b30ad6b316d0' },
-        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:3.2.1@sha256:ca45ba2c8356fd1ec5ec996a4e8ce1e9df6711b36c358ca19f6ab4bdc476695e' },
-        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:3.0.0@sha256:59e50b1f55e433ffdf6d678f8c658812b4119f631db8325572a52ee40d3bc562' },
-        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.3.0@sha256:0e31ec817e235b1814c04af97b1e7cf0053384aca2569570ce92bef0d95e94d2' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.10.0@sha256:3aff92bfc85d6ca3cc6fc381c8a89625d2af95cc55ed2db692ef4e483e600372' },
-        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.3.0@sha256:b7bc1ca4f4d0603a01369a689129aa273a938ce195fe43d00d42f4f2d5212f50' },
-        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:2.0.2@sha256:cbd604eaa970c99ba5c4c2e7984929668e05de824172f880e8c576b2fb7c976d' }
+        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.3.0@sha256:386fb755fc41edd7086f7bcb230f7f28078936f9ae4ead6d97c741df1cc194ae' },
+        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:3.0.4@sha256:4d688c746f27b195d98f35a7d24ec01f3f754e0ca61e9de0b0bc9793553880f1' },
+        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:4.0.2@sha256:424081fd38ebd35f3606c64f8f99138570e5f4d5066f12cfb4142447d249d3e7' },
+        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:4.0.1@sha256:ad20a9a5dcb2ab132374a7c8d44b89af0ec37651cf889e570f7625b02ee85fdf' },
+        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:3.0.2@sha256:caaa1f7f4055ae8990d8ec65bd100567496df7e4ed5eb427867f3717a8dcbf92' },
+        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:3.2.3@sha256:fdc4aa6d2c85aeafe65eaa4243aada0cc2e57b94f6eaee02c9b1a8fb89b01dd7' },
+        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.4.0@sha256:953bbd8b72a9108a8526d2c0bdbba67e1e1563ff59d0a117f0884dba1576f3dd' },
+        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:3.2.0@sha256:61e8247ded1e07cf882ca478dab180960357c614472e80b938f1f690a46788c2' }
     }
 };

src/iputils.js

@@ -0,0 +1,35 @@
+'use strict';
+
+exports = module.exports = {
+    ipFromInt,
+    intFromIp
+};
+
+const assert = require('assert');
+
+function intFromIp(address) {
+    assert.strictEqual(typeof address, 'string');
+
+    const parts = address.split('.');
+
+    if (parts.length !== 4) return null;
+
+    return (parseInt(parts[0], 10) << (8*3)) & 0xFF000000 |
+           (parseInt(parts[1], 10) << (8*2)) & 0x00FF0000 |
+           (parseInt(parts[2], 10) << (8*1)) & 0x0000FF00 |
+           (parseInt(parts[3], 10) << (8*0)) & 0x000000FF;
+}
+
+function ipFromInt(input) {
+    assert.strictEqual(typeof input, 'number');
+
+    let output = [];
+
+    for (let i = 3; i >= 0; --i) {
+        const octet = (input >> (i*8)) & 0x000000FF;
+        output.push(octet);
+    }
+
+    return output.join('.');
+}
+

src/ldap.js

@@ -1,11 +1,11 @@
 'use strict';
 
 exports = module.exports = {
-    start: start,
-    stop: stop
+    start,
+    stop
 };
 
-var assert = require('assert'),
+const assert = require('assert'),
     appdb = require('./appdb.js'),
     apps = require('./apps.js'),
     async = require('async'),
@@ -13,11 +13,13 @@ var assert = require('assert'),
     constants = require('./constants.js'),
     debug = require('debug')('box:ldap'),
     eventlog = require('./eventlog.js'),
+    groups = require('./groups.js'),
     ldap = require('ldapjs'),
     mail = require('./mail.js'),
     mailboxdb = require('./mailboxdb.js'),
     path = require('path'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     users = require('./users.js');
 
 var gServer = null;
@@ -132,8 +134,8 @@ function userSearch(req, res, next) {
 
             var dn = ldap.parseDN('cn=' + user.id + ',ou=users,dc=cloudron');
 
-            var groups = [ GROUP_USERS_DN ];
-            if (users.compareRoles(user.role, users.ROLE_ADMIN) >= 0) groups.push(GROUP_ADMINS_DN);
+            var memberof = [ GROUP_USERS_DN ];
+            if (users.compareRoles(user.role, users.ROLE_ADMIN) >= 0) memberof.push(GROUP_ADMINS_DN);
 
             var displayName = user.displayName || user.username || ''; // displayName can be empty and username can be null
             var nameParts = displayName.split(' ');
@@ -154,7 +156,7 @@ function userSearch(req, res, next) {
                     givenName: firstName,
                     username: user.username,
                     samaccountname: user.username,      // to support ActiveDirectory clients
-                    memberof: groups
+                    memberof: memberof
                 }
             };
 
@@ -286,14 +288,14 @@ function mailboxSearch(req, res, next) {
     } else if (req.dn.rdns[0].attrs.domain) { // legacy ldap mailbox search for old sogo
         var domain = req.dn.rdns[0].attrs.domain.value.toLowerCase();
 
-        mailboxdb.listMailboxes(domain, 1, 1000, function (error, result) {
+        mailboxdb.listMailboxes(domain, 1, 1000, function (error, mailboxes) {
 
             if (error) return next(new ldap.OperationsError(error.toString()));
 
             var results = [];
 
             // send mailbox objects
-            result.forEach(function (mailbox) {
+            mailboxes.forEach(function (mailbox) {
                 var dn = ldap.parseDN(`cn=${mailbox.name}@${domain},domain=${domain},ou=mailboxes,dc=cloudron`);
 
                 var obj = {
@@ -327,7 +329,9 @@ function mailboxSearch(req, res, next) {
             async.eachSeries(mailboxes, function (mailbox, callback) {
                 var dn = ldap.parseDN(`cn=${mailbox.name}@${mailbox.domain},ou=mailboxes,dc=cloudron`);
 
-                users.get(mailbox.ownerId, function (error, userObject) {
+                let getFunc = mailbox.ownerType === mail.OWNERTYPE_USER ? users.get : groups.get;
+
+                getFunc(mailbox.ownerId, function (error, ownerObject) {
                     if (error) return callback(); // skip mailboxes with unknown owner
 
                     var obj = {
@@ -335,30 +339,26 @@ function mailboxSearch(req, res, next) {
                         attributes: {
                             objectclass: ['mailbox'],
                             objectcategory: 'mailbox',
-                            displayname: userObject.displayName,
+                            displayname: mailbox.ownerType === mail.OWNERTYPE_USER ? ownerObject.displayName : ownerObject.name,
                             cn: `${mailbox.name}@${mailbox.domain}`,
                             uid: `${mailbox.name}@${mailbox.domain}`,
                             mail: `${mailbox.name}@${mailbox.domain}`
                         }
                     };
 
-                    mailboxdb.getAliasesForName(mailbox.name, mailbox.domain, function (error, aliases) {
-                        if (error) return callback(error);
-
-                        aliases.forEach(function (a, idx) {
-                            obj.attributes['mail' + idx] = `${a.name}@${a.domain}`;
-                        });
-
-                        // ensure all filter values are also lowercase
-                        var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
-                        if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));
-
-                        if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
-                            results.push(obj);
-                        }
-
-                        callback();
+                    mailbox.aliases.forEach(function (a, idx) {
+                        obj.attributes['mail' + idx] = `${a.name}@${a.domain}`;
                     });
+
+                    // ensure all filter values are also lowercase
+                    var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
+                    if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));
+
+                    if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
+                        results.push(obj);
+                    }
+
+                    callback();
                 });
             }, function (error) {
                 if (error) return next(new ldap.OperationsError(error.toString()));
@@ -482,18 +482,42 @@ function authorizeUserForApp(req, res, next) {
     assert.strictEqual(typeof req.user, 'object');
     assert.strictEqual(typeof req.app, 'object');
 
-    apps.hasAccessTo(req.app, req.user, function (error, result) {
+    apps.hasAccessTo(req.app, req.user, function (error, hasAccess) {
         if (error) return next(new ldap.OperationsError(error.toString()));
 
         // we return no such object, to avoid leakage of a users existence
-        if (!result) return next(new ldap.NoSuchObjectError(req.dn.toString()));
+        if (!hasAccess) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-        eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', appId: req.app.id }, { userId: req.user.id, user: users.removePrivateFields(req.user) });
+        eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', appId: req.app.id }, { userId: req.user.id, user: users.removePrivateFields(req.user) });
 
         res.end();
     });
 }
 
+function verifyMailboxPassword(mailbox, password, callback) {
+    assert.strictEqual(typeof mailbox, 'object');
+    assert.strictEqual(typeof password, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (mailbox.ownerType === mail.OWNERTYPE_USER) return users.verify(mailbox.ownerId, password, users.AP_MAIL /* identifier */, callback);
+
+    groups.getMembers(mailbox.ownerId, function (error, userIds) {
+        if (error) return callback(error);
+
+        let verifiedUser = null;
+        async.someSeries(userIds, function iterator(userId, iteratorDone) {
+            users.verify(userId, password, users.AP_MAIL /* identifier */, function (error, result) {
+                if (error) return iteratorDone(null, false);
+                verifiedUser = result;
+                iteratorDone(null, true);
+            });
+        }, function (error, result) {
+            if (!result) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+            callback(null, verifiedUser);
+        });
+    });
+}
+
 function authenticateUserMailbox(req, res, next) {
     debug('user mailbox auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
 
@@ -513,12 +537,12 @@ function authenticateUserMailbox(req, res, next) {
             if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
             if (error) return next(new ldap.OperationsError(error.message));
 
-            users.verify(mailbox.ownerId, req.credentials || '', users.AP_MAIL, function (error, result) {
+            verifyMailboxPassword(mailbox, req.credentials || '', function (error, result) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                 if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
                 if (error) return next(new ldap.OperationsError(error.message));
 
-                eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
+                eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
                 res.end();
             });
         });
@@ -547,6 +571,16 @@ function authenticateSftp(req, res, next) {
     });
 }
 
+function loadSftpConfig(req, res, next) {
+    services.getServiceConfig('sftp', function (error, serviceConfig) {
+        if (error) return next(new ldap.OperationsError(error.toString()));
+
+        req.requireAdmin = serviceConfig.requireAdmin;
+
+        next();
+    });
+}
+
 function userSearchSftp(req, res, next) {
     debug('sftp user search: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);
 
@@ -570,6 +604,8 @@ function userSearchSftp(req, res, next) {
         users.getByUsername(username, function (error, user) {
             if (error) return next(new ldap.OperationsError(error.toString()));
 
+            if (req.requireAdmin && users.compareRoles(user.role, users.ROLE_ADMIN) < 0) return next(new ldap.InsufficientAccessRightsError('Insufficient previleges'));
+
             apps.hasAccessTo(app, user, function (error, hasAccess) {
                 if (error) return next(new ldap.OperationsError(error.toString()));
                 if (!hasAccess) return next(new ldap.InsufficientAccessRightsError('Not authorized'));
@@ -593,16 +629,37 @@ function userSearchSftp(req, res, next) {
     });
 }
 
+function verifyAppMailboxPassword(addonId, username, password, callback) {
+    assert.strictEqual(typeof addonId, 'string');
+    assert.strictEqual(typeof username, 'string');
+    assert.strictEqual(typeof password, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const pattern = addonId === 'sendmail' ? 'MAIL_SMTP' : 'MAIL_IMAP';
+    appdb.getAppIdByAddonConfigValue(addonId, `%${pattern}_PASSWORD`, password, function (error, appId) { // search by password because this is unique for each app
+        if (error) return callback(error);
+
+        appdb.getAddonConfig(appId, addonId, function (error, result) {
+            if (error) return callback(error);
+
+            if (!result.some(r => r.name.endsWith(`${pattern}_USERNAME`) && r.value === username)) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+
+            callback(null);
+        });
+    });
+}
+
 function authenticateMailAddon(req, res, next) {
     debug('mail addon auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
 
     if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-    var email = req.dn.rdns[0].attrs.cn.value.toLowerCase();
-    var parts = email.split('@');
+    const email = req.dn.rdns[0].attrs.cn.value.toLowerCase();
+    const parts = email.split('@');
     if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
     const addonId = req.dn.rdns[1].attrs.ou.value.toLowerCase(); // 'sendmail' or 'recvmail'
+    if (addonId !== 'sendmail' && addonId !== 'recvmail') return next(new ldap.OperationsError('Invalid DN'));
 
     mail.getDomain(parts[1], function (error, domain) {
         if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
@@ -610,26 +667,22 @@ function authenticateMailAddon(req, res, next) {
 
         if (addonId === 'recvmail' && !domain.enabled) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-        let namePattern; // manifest v2 has a CLOUDRON_ prefix for names
-        if (addonId === 'sendmail') namePattern = '%MAIL_SMTP_PASSWORD';
-        else if (addonId === 'recvmail') namePattern = '%MAIL_IMAP_PASSWORD';
-        else return next(new ldap.OperationsError('Invalid DN'));
+        verifyAppMailboxPassword(addonId, email, req.credentials || '', function (error) {
+            if (!error) return res.end(); // validated as app
 
-        // note: with sendmail addon, apps can send mail without a mailbox (unlike users)
-        appdb.getAppIdByAddonConfigValue(addonId, namePattern, req.credentials || '', function (error, appId) {
+            if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
             if (error && error.reason !== BoxError.NOT_FOUND) return next(new ldap.OperationsError(error.message));
-            if (appId) return res.end();
 
             mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                 if (error) return next(new ldap.OperationsError(error.message));
 
-                users.verify(mailbox.ownerId, req.credentials || '', users.AP_MAIL, function (error, result) {
+                verifyMailboxPassword(mailbox, req.credentials || '', function (error, result) {
                     if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
                     if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
                     if (error) return next(new ldap.OperationsError(error.message));
 
-                    eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
+                    eventlog.upsert(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
                     res.end();
                 });
             });
@@ -660,16 +713,16 @@ function start(callback) {
     gServer.bind('ou=users,dc=cloudron', authenticateApp, authenticateUser, authorizeUserForApp);
 
     // http://www.ietf.org/proceedings/43/I-D/draft-srivastava-ldap-mail-00.txt
-    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch); // haraka, dovecot
+    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch); // haraka (address translation), dovecot (LMTP), sogo (mailbox search)
     gServer.bind('ou=mailboxes,dc=cloudron', authenticateUserMailbox); // apps like sogo can use domain=${domain} to authenticate a mailbox
     gServer.search('ou=mailaliases,dc=cloudron', mailAliasSearch); // haraka
     gServer.search('ou=mailinglists,dc=cloudron', mailingListSearch); // haraka
 
-    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailAddon); // dovecot
-    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka
+    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailAddon); // dovecot (IMAP auth)
+    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka (MSA auth)
 
     gServer.bind('ou=sftp,dc=cloudron', authenticateSftp);    // sftp
-    gServer.search('ou=sftp,dc=cloudron', userSearchSftp);
+    gServer.search('ou=sftp,dc=cloudron', loadSftpConfig, userSearchSftp);
 
     gServer.compare('cn=users,ou=groups,dc=cloudron', authenticateApp, groupUsersCompare);
     gServer.compare('cn=admins,ou=groups,dc=cloudron', authenticateApp, groupAdminsCompare);
@@ -687,6 +740,12 @@ function start(callback) {
         res.end();
     });
 
+    // just log that an attempt was made to unknown route, this helps a lot during app packaging
+    gServer.use(function(req, res, next) {
+        debug('not handled: dn %s, scope %s, filter %s (from %s)', req.dn ? req.dn.toString() : '-', req.scope, req.filter ? req.filter.toString() : '-', req.connection.ldap.id);
+        return next();
+    });
+
     gServer.listen(constants.LDAP_PORT, '0.0.0.0', callback);
 }
 

src/mail.js

@@ -52,11 +52,16 @@ exports = module.exports = {
     removeList,
     resolveList,
 
+    OWNERTYPE_USER: 'user',
+    OWNERTYPE_GROUP: 'group',
+
+    DEFAULT_MEMORY_LIMIT: 512 * 1024 * 1024,
+
     _removeMailboxes: removeMailboxes,
     _readDkimPublicKeySync: readDkimPublicKeySync
 };
 
-var assert = require('assert'),
+const assert = require('assert'),
     async = require('async'),
     BoxError = require('./boxerror.js'),
     cloudron = require('./cloudron.js'),
@@ -75,12 +80,15 @@ var assert = require('assert'),
     nodemailer = require('nodemailer'),
     path = require('path'),
     paths = require('./paths.js'),
+    request = require('request'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
+    services = require('./services.js'),
     settings = require('./settings.js'),
     shell = require('./shell.js'),
     smtpTransport = require('nodemailer-smtp-transport'),
     sysinfo = require('./sysinfo.js'),
+    system = require('./system.js'),
     tasks = require('./tasks.js'),
     users = require('./users.js'),
     validator = require('validator'),
@@ -619,9 +627,10 @@ function createMailConfig(mailFqdn, mailDomain, callback) {
     });
 }
 
-function configureMail(mailFqdn, mailDomain, callback) {
+function configureMail(mailFqdn, mailDomain, serviceConfig, callback) {
     assert.strictEqual(typeof mailFqdn, 'string');
     assert.strictEqual(typeof mailDomain, 'string');
+    assert.strictEqual(typeof serviceConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     // mail (note: 2525 is hardcoded in mail container and app use this port)
@@ -630,7 +639,8 @@ function configureMail(mailFqdn, mailDomain, callback) {
     // mail container uses /app/data for backed up data and /run for restart-able data
 
     const tag = infra.images.mail.tag;
-    const memoryLimit = 4 * 256;
+    const memoryLimit = serviceConfig.memoryLimit || exports.DEFAULT_MEMORY_LIMIT;
+    const memory = system.getMemoryAllocation(memoryLimit);
     const cloudronToken = hat(8 * 128), relayToken = hat(8 * 128);
 
     reverseProxy.getCertificate(mailFqdn, mailDomain, function (error, bundle) {
@@ -661,8 +671,8 @@ function configureMail(mailFqdn, mailDomain, callback) {
                             --log-opt syslog-address=udp://127.0.0.1:2514 \
                             --log-opt syslog-format=rfc5424 \
                             --log-opt tag=mail \
-                            -m ${memoryLimit}m \
-                            --memory-swap ${memoryLimit * 2}m \
+                            -m ${memory} \
+                            --memory-swap ${memoryLimit} \
                             --dns 172.18.0.1 \
                             --dns-search=. \
                             -e CLOUDRON_MAIL_TOKEN="${cloudronToken}" \
@@ -709,8 +719,12 @@ function restartMail(callback) {
 
     if (process.env.BOX_ENV === 'test' && !process.env.TEST_CREATE_INFRA) return callback();
 
-    debug(`restartMail: restarting mail container with ${settings.mailFqdn()} ${settings.adminDomain()}`);
-    configureMail(settings.mailFqdn(), settings.adminDomain(), callback);
+    services.getServiceConfig('mail', function (error, serviceConfig) {
+        if (error) return callback(error);
+
+        debug(`restartMail: restarting mail container with ${settings.mailFqdn()} ${settings.adminDomain()}`);
+        configureMail(settings.mailFqdn(), settings.adminDomain(), serviceConfig, callback);
+    });
 }
 
 function restartMailIfActivated(callback) {
@@ -763,7 +777,7 @@ function txtRecordsWithSpf(domain, mailFqdn, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     domains.getDnsRecords('', domain, 'TXT', function (error, txtRecords) {
-        if (error) return error;
+        if (error) return callback(error);
 
         debug('txtRecordsWithSpf: current txt records - %j', txtRecords);
 
@@ -861,37 +875,40 @@ function upsertDnsRecords(domain, mailFqdn, callback) {
 
         if (process.env.BOX_ENV === 'test') return callback();
 
-        var dkimKey = readDkimPublicKeySync(domain);
+        const dkimKey = readDkimPublicKeySync(domain);
         if (!dkimKey) return callback(new BoxError(BoxError.FS_ERROR, 'Failed to read dkim public key'));
 
         // t=s limits the domainkey to this domain and not it's subdomains
-        var dkimRecord = { subdomain: `${mailDomain.dkimSelector}._domainkey`, domain: domain, type: 'TXT', values: [ '"v=DKIM1; t=s; p=' + dkimKey + '"' ] };
+        const dkimRecord = { subdomain: `${mailDomain.dkimSelector}._domainkey`, domain: domain, type: 'TXT', values: [ `"v=DKIM1; t=s; p=${dkimKey}"` ] };
 
-        var records = [ ];
+        let records = [];
         records.push(dkimRecord);
-        if (mailDomain.enabled) {
-            records.push({ subdomain: '_dmarc', domain: domain, type: 'TXT', values: [ '"v=DMARC1; p=reject; pct=100"' ] });
-            records.push({ subdomain: '', domain: domain, type: 'MX', values: [ '10 ' + mailFqdn + '.' ] });
-        }
+        if (mailDomain.enabled) records.push({ subdomain: '', domain: domain, type: 'MX', values: [ '10 ' + mailFqdn + '.' ] });
 
         txtRecordsWithSpf(domain, mailFqdn, function (error, txtRecords) {
             if (error) return callback(error);
 
             if (txtRecords) records.push({ subdomain: '', domain: domain, type: 'TXT', values: txtRecords });
 
-            debug('upsertDnsRecords: will update %j', records);
+            domains.getDnsRecords('_dmarc', domain, 'TXT', function (error, dmarcRecords) { // only update dmarc if absent. this allows user to set email for reporting
+                if (error) return callback(error);
 
-            async.mapSeries(records, function (record, iteratorCallback) {
-                domains.upsertDnsRecords(record.subdomain, record.domain, record.type, record.values, iteratorCallback);
-            }, function (error, changeIds) {
-                if (error) {
-                    debug(`upsertDnsRecords: failed to update: ${error}`);
-                    return callback(error);
-                }
+                if (dmarcRecords.length === 0) records.push({ subdomain: '_dmarc', domain: domain, type: 'TXT', values: [ '"v=DMARC1; p=reject; pct=100"' ] });
 
-                debug('upsertDnsRecords: records %j added with changeIds %j', records, changeIds);
+                debug('upsertDnsRecords: will update %j', records);
 
-                callback(null);
+                async.mapSeries(records, function (record, iteratorCallback) {
+                    domains.upsertDnsRecords(record.subdomain, record.domain, record.type, record.values, iteratorCallback);
+                }, function (error, changeIds) {
+                    if (error) {
+                        debug(`upsertDnsRecords: failed to update: ${error}`);
+                        return callback(error);
+                    }
+
+                    debug('upsertDnsRecords: records %j added with changeIds %j', records, changeIds);
+
+                    callback(null);
+                });
             });
         });
     });
@@ -934,7 +951,10 @@ function changeLocation(auditSource, progressCallback, callback) {
                 progressCallback({ percent: progress, message: `Updating DNS of ${domainObject.domain}` });
                 progress += Math.round(70/allDomains.length);
 
-                upsertDnsRecords(domainObject.domain, fqdn, iteratorDone);
+                upsertDnsRecords(domainObject.domain, fqdn, function (error) { // ignore any errors. we anyway report dns errors in status tab
+                    progressCallback({ percent: progress, message: `Updated DNS of ${domainObject.domain}: ${error ? error.message : 'success'}` });
+                    iteratorDone();
+                });
             }, function (error) {
                 if (error) return callback(error);
 
@@ -1162,10 +1182,11 @@ function getMailbox(name, domain, callback) {
     });
 }
 
-function addMailbox(name, domain, userId, auditSource, callback) {
+function addMailbox(name, domain, ownerId, ownerType, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1174,31 +1195,53 @@ function addMailbox(name, domain, userId, auditSource, callback) {
     var error = validateName(name);
     if (error) return callback(error);
 
-    mailboxdb.addMailbox(name, domain, userId, function (error) {
+    if (ownerType !== exports.OWNERTYPE_USER && ownerType !== exports.OWNERTYPE_GROUP) return callback(new BoxError(BoxError.BAD_FIELD, 'bad owner type'));
+
+    mailboxdb.addMailbox(name, domain, ownerId, ownerType, function (error) {
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_ADD, auditSource, { name, domain, userId });
+        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_ADD, auditSource, { name, domain, ownerId, ownerType });
 
         callback(null);
     });
 }
 
-function updateMailboxOwner(name, domain, userId, auditSource, callback) {
+function updateMailboxOwner(name, domain, ownerId, ownerType, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     name = name.toLowerCase();
 
+    if (ownerType !== exports.OWNERTYPE_USER && ownerType !== exports.OWNERTYPE_GROUP) return callback(new BoxError(BoxError.BAD_FIELD, 'bad owner type'));
+
     getMailbox(name, domain, function (error, result) {
         if (error) return callback(error);
 
-        mailboxdb.updateMailboxOwner(name, domain, userId, function (error) {
+        mailboxdb.updateMailboxOwner(name, domain, ownerId, ownerType, function (error) {
             if (error) return callback(error);
 
-            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldUserId: result.userId, userId });
+            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldUserId: result.userId, ownerId, ownerType });
+
+            callback(null);
+        });
+    });
+}
+
+function removeSolrIndex(mailbox, callback) {
+    assert.strictEqual(typeof mailbox, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    services.getContainerDetails('mail', 'CLOUDRON_MAIL_TOKEN', function (error, addonDetails) {
+        if (error) return callback(error);
+
+        request.post(`https://${addonDetails.ip}:3000/solr_delete_index?access_token=${addonDetails.token}`, { timeout: 2000, rejectUnauthorized: false, json: { mailbox } }, function (error, response) {
+            if (error) return callback(error);
+
+            if (response.statusCode !== 200) return callback(new Error(`Error removing solr index - ${response.statusCode} ${JSON.stringify(response.body)}`));
 
             callback(null);
         });
@@ -1212,7 +1255,8 @@ function removeMailbox(name, domain, options, auditSource, callback) {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    const deleteMailFunc = options.deleteMails ? shell.sudo.bind(null, 'removeMailbox', [ REMOVE_MAILBOX, `${name}@${domain}` ], {}) : (next) => next();
+    const mailbox =`${name}@${domain}`;
+    const deleteMailFunc = options.deleteMails ? shell.sudo.bind(null, 'removeMailbox', [ REMOVE_MAILBOX, mailbox ], {}) : (next) => next();
 
     deleteMailFunc(function (error) {
         if (error) return callback(new BoxError(BoxError.FS_ERROR, `Error removing mailbox: ${error.message}`));
@@ -1220,6 +1264,7 @@ function removeMailbox(name, domain, options, auditSource, callback) {
         mailboxdb.del(name, domain, function (error) {
             if (error) return callback(error);
 
+            removeSolrIndex(mailbox, NOOP_CALLBACK);
             eventlog.add(eventlog.ACTION_MAIL_MAILBOX_REMOVE, auditSource, { name, domain });
 
             callback();
@@ -1379,7 +1424,7 @@ function resolveList(listName, listDomain, callback) {
 
             let result = [], toResolve = list.members.slice(), visited = []; // slice creates a copy of array
 
-            async.whilst(() => toResolve.length != 0, function (iteratorCallback) {
+            async.whilst((testDone) => testDone(null, toResolve.length != 0), function (iteratorCallback) {
                 const toProcess = toResolve.shift();
                 const parts = toProcess.split('@');
                 const memberName = parts[0].split('+')[0], memberDomain = parts[1];

src/mail_templates/app_updates_available.ejs

@@ -3,18 +3,14 @@
 Dear Cloudron Admin,
 
 <% for (var i = 0; i < apps.length; i++) { -%>
-A new version <%= apps[i].updateInfo.manifest.version %> of the app '<%= apps[i].app.manifest.title %>' installed at <%= apps[i].app.fqdn %> is available.
+The app '<%= apps[i].app.manifest.title %>' installed at <%= apps[i].app.fqdn %> has an update available.
 
-Changes:
+<%= apps[i].app.manifest.title %> v<%= apps[i].updateInfo.manifest.version %> changes:
 <%= apps[i].updateInfo.manifest.changelog %>
 
 <% } -%>
 
-<% if (!hasSubscription) { -%>
-*Keep your Cloudron automatically up-to-date and secure by upgrading to a paid plan at* <%= webadminUrl %>/#/settings
-<% } else { -%>
 Update now at <%= webadminUrl %>
-<% } -%>
 
 Powered by https://cloudron.io
 
@@ -33,24 +29,20 @@ Sent at: <%= new Date().toUTCString() %>
     <div style="width: 650px; text-align: left;">
 <% for (var i = 0; i < apps.length; i++) { -%>
         <p>
-            A new version <%= apps[i].updateInfo.manifest.version %> of the app '<%= apps[i].app.manifest.title %>' installed at <a href="https://<%= apps[i].app.fqdn %>"><%= apps[i].app.fqdn %></a> is available.
+            The app '<%= apps[i].app.manifest.title %>' installed at <a href="https://<%= apps[i].app.fqdn %>"><%= apps[i].app.fqdn %></a> has an update available.
         </p>
 
-        <h5>Changelog:</h5>
+        <h5><%= apps[i].app.manifest.title %> v<%= apps[i].updateInfo.manifest.version %> changes:</h5>
         <%- apps[i].changelogHTML %>
 
         <br/>
 <% } -%>
 
-<% if (!hasSubscription) { -%>
-        <p>Keep your Cloudron automatically up-to-date and secure by upgrading to a <a href="<%= webadminUrl %>/#/settings">paid plan</a>.</p>
-<% } else { -%>
         <p>
             <br/>
             <center><a href="<%= webadminUrl %>">Update now</a></center>
             <br/>
         </p>
-<% } -%>
     </div>
 
     <div style="font-size: 10px; color: #333333; background: #ffffff;">

src/mail_templates/box_update_available.ejs

@@ -0,0 +1,45 @@
+<%if (format === 'text') { %>
+
+Dear <%= cloudronName %> Admin,
+
+Cloudron v<%= newBoxVersion %> is now available!
+
+Changes:
+<% for (var i = 0; i < changelog.length; i++) { %>
+  * <%- changelog[i] %>
+<% } %>
+
+Powered by https://cloudron.io
+
+Sent at: <%= new Date().toUTCString() %>
+
+<% } else { %>
+
+<center>
+
+<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
+
+<h3>Dear <%= cloudronName %> Admin,</h3>
+
+<div style="width: 650px; text-align: left;">
+    <p>
+        Cloudron v<%= newBoxVersion %> is now available!
+    </p>
+
+    <h5>Changes:</h5>
+    <ul>
+    <% for (var i = 0; i < changelogHTML.length; i++) { %>
+        <li><%- changelogHTML[i] %></li>
+    <% } %>
+    </ul>
+
+    <br/>
+</div>
+
+<div style="font-size: 10px; color: #333333; background: #ffffff;">
+    Powered by <a href="https://cloudron.io">Cloudron</a>.
+</div>
+
+</center>
+
+<% } %>

src/mail_templates/oom_event.ejs

@@ -2,12 +2,13 @@
 
 Dear <%= cloudronName %> Admin,
 
-<%= program %> was restarted now as it ran out of memory.
-
-If this message appears repeatedly, give the app more memory.
-
-* To increase an app's memory limit - https://docs.cloudron.io/apps/#memory-limit
-* To increase a service's memory limit - https://docs.cloudron.io/troubleshooting/#services
+<%if (app) { %>
+The application at <%= app.fqdn %> ran out of memory. The application has been restarted automatically. If you see this notification often,
+consider increasing the memory limit - <%= webadminUrl %>/#/app/<%= app.id %>/resources .
+<% } else { %>
+The addon <%= addon.name %> service ran out of memory. The service has been restarted automatically. If you see this notification often,
+consider increasing the memory limit - <%= webadminUrl %>/#/services .
+<% } %>
 
 Out of memory event:
 

src/mail_templates/password_reset-html.ejs

@@ -0,0 +1,24 @@
+<center>
+
+<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
+
+<h3>{{ passwordResetEmail.salutation }}</h3>
+
+<p>{{ passwordResetEmail.description }}</p>
+
+<p>
+  <a href="<%= resetLink %>">{{ passwordResetEmail.resetAction }}</a>
+</p>
+
+<br/>
+
+{{ passwordResetEmail.expireNote }}
+
+<br/>
+<br/>
+
+<div style="font-size: 10px; color: #333333; background: #ffffff;">
+  Powered by <a href="https://cloudron.io">Cloudron</a>
+</div>
+
+</center>

src/mail_templates/password_reset-text.ejs

@@ -0,0 +1,9 @@
+{{ passwordResetEmail.salutation }}
+
+{{ passwordResetEmail.description }}
+
+{{ passwordResetEmail.resetActionText }}
+
+{{ passwordResetEmail.expireNote }}
+
+Powered by https://cloudron.io

src/mail_templates/password_reset.ejs

@@ -1,45 +0,0 @@
-<%if (format === 'text') { %>
-
-Hi <%= user.displayName || user.username || user.email %>,
-
-Someone, hopefully you, has requested your account's password
-be reset. If you did not request this reset, please ignore this message.
-
-To reset your password, please visit the following page:
-<%- resetLink %>
-
-Please note that the password reset link will expire in 24 hours.
-
-Powered by https://cloudron.io
-
-<% } else { %>
-
-<center>
-
-<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
-
-<h3>Hi <%= user.displayName || user.username || user.email %>,</h3>
-
-<p>
-    Someone, hopefully you, has requested your account's password be reset.<br/>
-    If you did not request this reset, please ignore this message.
-</p>
-
-<p>
-    <a href="<%= resetLink %>">Click to reset your password</a>
-</p>
-
-<br/>
-
-Please note that the password reset link will expire in 24 hours.
-
-<br/>
-<br/>
-
-<div style="font-size: 10px; color: #333333; background: #ffffff;">
-    Powered by <a href="https://cloudron.io">Cloudron</a>
-</div>
-
-</center>
-
-<% } %>

src/mail_templates/user_added.ejs

@@ -1,30 +0,0 @@
-<%if (format === 'text') { %>
-
-Dear <%= cloudronName %> Admin,
-
-A new user with email <%= user.email %> was added to <%= cloudronName %>.
-
-Powered by https://cloudron.io
-
-<% } else { %>
-
-<center>
-
-<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
-
-<h3>Dear <%= cloudronName %> Admin,</h3>
-
-<p>
-    A new user with email <%= user.email %> was added to <%= cloudronName %>.
-</p>
-
-<br/>
-<br/>
-
-<div style="font-size: 10px; color: #333333; background: #ffffff;">
-    Powered by <a href="https://cloudron.io">Cloudron</a>.
-</div>
-
-</center>
-
-<% } %>

src/mail_templates/user_event.ejs

@@ -1,14 +0,0 @@
-<%if (format === 'text') { %>
-
-Dear Cloudron Admin,
-
-User <%= user.username || user.email %> <%= event %>.
-
-
-Powered by https://cloudron.io
-
-Sent at: <%= new Date().toUTCString() %>
-
-<% } else { %>
-
-<% } %>

src/mail_templates/welcome_user-html.ejs

@@ -0,0 +1,28 @@
+<center>
+
+<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
+
+<h3>{{ welcomeEmail.salutation }}</h3>
+<h2>{{ welcomeEmail.welcomeTo }}</h2>
+
+<p>
+  <a href="<%= inviteLink %>">{{ welcomeEmail.inviteLinkAction }}</a>
+</p>
+
+<br/>
+<br/>
+
+<div style="font-size: 10px; color: #333333; background: #ffffff;">
+<% if (invitor) { -%>
+  {{ welcomeEmail.invitor }}
+<% } -%>
+
+  <br/>
+
+  {{ welcomeEmail.expireNote }}
+  <br/>
+
+  Powered by <a href="https://cloudron.io">Cloudron</a>
+</div>
+
+</center>

src/mail_templates/welcome_user-text.ejs

@@ -0,0 +1,13 @@
+{{ welcomeEmail.salutation }}
+
+{{ welcomeEmail.welcomeTo }}
+
+{{ welcomeEmail.inviteLinkActionText }}
+
+<% if (invitor) { %>
+{{ welcomeEmail.invitor }}
+<% } %>
+
+{{ welcomeEmail.expireNote }}
+
+Powered by https://cloudron.io

src/mail_templates/welcome_user.ejs

@@ -1,50 +0,0 @@
-<%if (format === 'text') { %>
-
-Dear <%= user.displayName || user.username || user.email %>,
-
-Welcome to <%= cloudronName %>!
-
-Follow the link to get started.
-<%- inviteLink %>
-
-<% if (invitor && invitor.email) { %>
-You are receiving this email because you were invited by <%= invitor.email %>.
-<% } %>
-
-Please note that the invite link will expire in 7 days.
-
-Powered by https://cloudron.io
-
-<% } else { %>
-
-<center>
-
-<img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
-
-<h3>Hi <%= user.displayName || user.username || user.email %>,</h3>
-
-<h2>Welcome to <%= cloudronName %>!</h2>
-
-<p>
-    <a href="<%= inviteLink %>">Get started</a>.
-</p>
-
-<br/>
-<br/>
-
-<div style="font-size: 10px; color: #333333; background: #ffffff;">
-    <% if (invitor && invitor.email) { %>
-        You are receiving this email because you were invited by <%= invitor.email %>.
-    <% } %>
-
-    <br/>
-
-        Please note that the invite link will expire in 7 days.
-    <br/>
-
-    Powered by <a href="https://cloudron.io">Cloudron</a>
-</div>
-
-</center>
-
-<% } %>

src/mailboxdb.js

@@ -42,7 +42,7 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'aliasName', 'aliasDomain', 'creationTime', 'membersJson', 'membersOnly', 'domain' ].join(',');
+var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'ownerType', 'aliasName', 'aliasDomain', 'creationTime', 'membersJson', 'membersOnly', 'domain' ].join(',');
 
 function postProcess(data) {
     data.members = safe.JSON.parse(data.membersJson) || [ ];
@@ -53,13 +53,24 @@ function postProcess(data) {
     return data;
 }
 
-function addMailbox(name, domain, ownerId, callback) {
+function postProcessAliases(data) {
+    const aliasNames = JSON.parse(data.aliasNames), aliasDomains = JSON.parse(data.aliasDomains);
+    delete data.aliasNames;
+    delete data.aliasDomains;
+    data.aliases = [];
+    for (let i = 0; i < aliasNames.length; i++) { // NOTE: aliasNames is [ null ] when no aliases
+        if (aliasNames[i]) data.aliases[i] = { name: aliasNames[i], domain: aliasDomains[i] };
+    }
+}
+
+function addMailbox(name, domain, ownerId, ownerType, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId) VALUES (?, ?, ?, ?)', [ name, exports.TYPE_MAILBOX, domain, ownerId ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, ownerType) VALUES (?, ?, ?, ?, ?)', [ name, exports.TYPE_MAILBOX, domain, ownerId, ownerType ], function (error) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'mailbox already exists'));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -67,13 +78,14 @@ function addMailbox(name, domain, ownerId, callback) {
     });
 }
 
-function updateMailboxOwner(name, domain, ownerId, callback) {
+function updateMailboxOwner(name, domain, ownerId, ownerType, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('UPDATE mailboxes SET ownerId = ? WHERE name = ? AND domain = ?', [ ownerId, name, domain ], function (error, result) {
+    database.query('UPDATE mailboxes SET ownerId = ?, ownerType = ? WHERE name = ? AND domain = ?', [ ownerId, ownerType, name, domain ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Mailbox not found'));
 
@@ -88,8 +100,8 @@ function addList(name, domain, members, membersOnly, callback) {
     assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, membersJson, membersOnly) VALUES (?, ?, ?, ?, ?, ?)',
-        [ name, exports.TYPE_LIST, domain, 'admin', JSON.stringify(members), membersOnly ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, ownerType, membersJson, membersOnly) VALUES (?, ?, ?, ?, ?, ?, ?)',
+        [ name, exports.TYPE_LIST, domain, 'admin', 'user', JSON.stringify(members), membersOnly ], function (error) {
             if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'mailbox already exists'));
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -223,14 +235,22 @@ function listMailboxes(domain, search, page, perPage, callback) {
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    let query = `SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? AND domain = ?`;
-    if (search) query += ' AND (name LIKE ' + mysql.escape('%' + search + '%') + ')';
-    query += 'ORDER BY name LIMIT ?,?';
+    const escapedSearch = mysql.escape('%' + search + '%'); // this also quotes the string
+    const searchQuery = search ? ` HAVING (name LIKE ${escapedSearch} OR aliasNames LIKE ${escapedSearch} OR aliasDomains LIKE ${escapedSearch})` : ''; // having instead of where because of aggregated columns use
 
-    database.query(query, [ exports.TYPE_MAILBOX, domain, (page-1)*perPage, perPage ], function (error, results) {
+    const query = 'SELECT m1.name AS name, m1.domain AS domain, m1.ownerId AS ownerId, m1.ownerType as ownerType, JSON_ARRAYAGG(m2.name) AS aliasNames, JSON_ARRAYAGG(m2.domain) AS aliasDomains '
+        + ` FROM (SELECT * FROM mailboxes WHERE type='${exports.TYPE_MAILBOX}') AS m1`
+        + ` LEFT JOIN (SELECT * FROM mailboxes WHERE type='${exports.TYPE_ALIAS}') AS m2`
+        + '     ON m1.name=m2.aliasName AND m1.domain=m2.aliasDomain AND m1.ownerId=m2.ownerId'
+        + ' WHERE m1.domain = ?'
+        + ' GROUP BY m1.name, m1.domain, m1.ownerId'
+        + searchQuery
+        + ' ORDER BY name LIMIT ?,?';
+
+    database.query(query, [ domain, (page-1)*perPage, perPage ], function (error, results) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-        results.forEach(function (result) { postProcess(result); });
+        results.forEach(postProcessAliases);
 
         callback(null, results);
     });
@@ -241,14 +261,20 @@ function listAllMailboxes(page, perPage, callback) {
     assert.strictEqual(typeof perPage, 'number');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query(`SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE type = ? ORDER BY name LIMIT ?,?`,
-        [ exports.TYPE_MAILBOX, (page-1)*perPage, perPage ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+    const query = 'SELECT m1.name AS name, m1.domain AS domain, m1.ownerId AS ownerId, m1.ownerType as ownerType, JSON_ARRAYAGG(m2.name) AS aliasNames, JSON_ARRAYAGG(m2.domain) AS aliasDomains '
+        + ` FROM (SELECT * FROM mailboxes WHERE type='${exports.TYPE_MAILBOX}') AS m1`
+        + ` LEFT JOIN (SELECT * FROM mailboxes WHERE type='${exports.TYPE_ALIAS}') AS m2`
+        + '     ON m1.name=m2.aliasName AND m1.domain=m2.aliasDomain AND m1.ownerId=m2.ownerId'
+        + ' GROUP BY m1.name, m1.domain, m1.ownerId'
+        + ' ORDER BY name LIMIT ?,?';
 
-            results.forEach(function (result) { postProcess(result); });
+    database.query(query, [ (page-1)*perPage, perPage ], function (error, results) {
+        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-            callback(null, results);
-        });
+        results.forEach(postProcessAliases);
+
+        callback(null, results);
+    });
 }
 
 function getLists(domain, search, page, perPage, callback) {
@@ -314,8 +340,8 @@ function setAliasesForName(name, domain, aliases, callback) {
         // clear existing aliases
         queries.push({ query: 'DELETE FROM mailboxes WHERE aliasName = ? AND aliasDomain = ? AND type = ?', args: [ name, domain, exports.TYPE_ALIAS ] });
         aliases.forEach(function (alias) {
-            queries.push({ query: 'INSERT INTO mailboxes (name, domain, type, aliasName, aliasDomain, ownerId) VALUES (?, ?, ?, ?, ?, ?)',
-                args: [ alias.name, alias.domain, exports.TYPE_ALIAS, name, domain, results[0].ownerId ] });
+            queries.push({ query: 'INSERT INTO mailboxes (name, domain, type, aliasName, aliasDomain, ownerId, ownerType) VALUES (?, ?, ?, ?, ?, ?, ?)',
+                args: [ alias.name, alias.domain, exports.TYPE_ALIAS, name, domain, results[0].ownerId, results[0].ownerType ] });
         });
 
         database.transaction(queries, function (error) {

src/mailer.js

@@ -1,25 +1,23 @@
 'use strict';
 
 exports = module.exports = {
-    userAdded: userAdded,
-    userRemoved: userRemoved,
-    roleChanged: roleChanged,
-    passwordReset: passwordReset,
-    appUpdatesAvailable: appUpdatesAvailable,
+    passwordReset,
+    boxUpdateAvailable,
+    appUpdatesAvailable,
 
-    sendInvite: sendInvite,
+    sendInvite,
 
-    appUp: appUp,
-    appDied: appDied,
-    appUpdated: appUpdated,
-    oomEvent: oomEvent,
+    appUp,
+    appDied,
+    appUpdated,
+    oomEvent,
 
-    backupFailed: backupFailed,
+    backupFailed,
 
-    certificateRenewalError: certificateRenewalError,
-    boxUpdateError: boxUpdateError,
+    certificateRenewalError,
+    boxUpdateError,
 
-    sendTestMail: sendTestMail,
+    sendTestMail,
 
     _mailQueue: [] // accumulate mails in test mode
 };
@@ -34,8 +32,8 @@ var assert = require('assert'),
     safe = require('safetydance'),
     settings = require('./settings.js'),
     showdown = require('showdown'),
-    smtpTransport = require('nodemailer-smtp-transport'),
-    util = require('util');
+    translation = require('./translation.js'),
+    smtpTransport = require('nodemailer-smtp-transport');
 
 var NOOP_CALLBACK = function (error) { if (error) debug(error); };
 
@@ -91,14 +89,21 @@ function sendMail(mailOptions, callback) {
     });
 }
 
-function render(templateFile, params) {
+function render(templateFile, params, translationAssets) {
     assert.strictEqual(typeof templateFile, 'string');
     assert.strictEqual(typeof params, 'object');
 
     var content = null;
+    var raw = safe.fs.readFileSync(path.join(MAIL_TEMPLATES_DIR, templateFile), 'utf8');
+    if (raw === null) {
+        debug(`Error loading ${templateFile}`);
+        return '';
+    }
+
+    if (typeof translationAssets === 'object') raw = translation.translate(raw, translationAssets.translations || {}, translationAssets.fallback || {});
 
     try {
-        content = ejs.render(safe.fs.readFileSync(path.join(MAIL_TEMPLATES_DIR, templateFile), 'utf8'), params);
+        content = ejs.render(raw, params);
     } catch (e) {
         debug(`Error rendering ${templateFile}`, e);
     }
@@ -106,25 +111,6 @@ function render(templateFile, params) {
     return content;
 }
 
-function mailUserEvent(mailTo, user, event) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-    assert.strictEqual(typeof event, 'string');
-
-    getMailConfig(function (error, mailConfig) {
-        if (error) return debug('Error getting mail details:', error);
-
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: mailTo,
-            subject: util.format('[%s] %s %s', mailConfig.cloudronName, user.username || user.fallbackEmail || user.email, event),
-            text: render('user_event.ejs', { user: user, event: event, format: 'text' }),
-        };
-
-        sendMail(mailOptions);
-    });
-}
-
 function sendInvite(user, invitor, inviteLink) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof invitor, 'object');
@@ -135,84 +121,31 @@ function sendInvite(user, invitor, inviteLink) {
     getMailConfig(function (error, mailConfig) {
         if (error) return debug('Error getting mail details:', error);
 
-        var templateData = {
-            user: user,
-            webadminUrl: settings.adminOrigin(),
-            inviteLink: inviteLink,
-            invitor: invitor,
-            cloudronName: mailConfig.cloudronName,
-            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
-        };
+        translation.getTranslations(function (error, translationAssets) {
+            if (error) return debug('Error getting translations:', error);
 
-        var templateDataText = JSON.parse(JSON.stringify(templateData));
-        templateDataText.format = 'text';
+            var templateData = {
+                user: user.displayName || user.username || user.email,
+                webadminUrl: settings.adminOrigin(),
+                inviteLink: inviteLink,
+                invitor: invitor ? invitor.email : null,
+                cloudronName: mailConfig.cloudronName,
+                cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
+            };
 
-        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
-        templateDataHTML.format = 'html';
+            var mailOptions = {
+                from: mailConfig.notificationFrom,
+                to: user.fallbackEmail,
+                subject: ejs.render(translation.translate('{{ welcomeEmail.subject }}', translationAssets.translations || {}, translationAssets.fallback || {}), { cloudron: mailConfig.cloudronName }),
+                text: render('welcome_user-text.ejs', templateData, translationAssets),
+                html: render('welcome_user-html.ejs', templateData, translationAssets)
+            };
 
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: user.fallbackEmail,
-            subject: util.format('Welcome to %s', mailConfig.cloudronName),
-            text: render('welcome_user.ejs', templateDataText),
-            html: render('welcome_user.ejs', templateDataHTML)
-        };
-
-        sendMail(mailOptions);
+            sendMail(mailOptions);
+        });
     });
 }
 
-function userAdded(mailTo, user) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-
-    debug(`userAdded: Sending mail for added users ${user.fallbackEmail} to ${mailTo}`);
-
-    getMailConfig(function (error, mailConfig) {
-        if (error) return debug('Error getting mail details:', error);
-
-        var templateData = {
-            user: user,
-            cloudronName: mailConfig.cloudronName,
-            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
-        };
-
-        var templateDataText = JSON.parse(JSON.stringify(templateData));
-        templateDataText.format = 'text';
-
-        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
-        templateDataHTML.format = 'html';
-
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: mailTo,
-            subject: util.format('[%s] User %s added', mailConfig.cloudronName, user.fallbackEmail),
-            text: render('user_added.ejs', templateDataText),
-            html: render('user_added.ejs', templateDataHTML)
-        };
-
-        sendMail(mailOptions);
-    });
-}
-
-function userRemoved(mailTo, user) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-
-    debug('Sending mail for userRemoved.', user.id, user.username, user.email);
-
-    mailUserEvent(mailTo, user, 'was removed');
-}
-
-function roleChanged(mailTo, user) {
-    assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof user, 'object');
-
-    debug('Sending mail for roleChanged');
-
-    mailUserEvent(mailTo, user, `now has the role '${user.role}'`);
-}
-
 function passwordReset(user) {
     assert.strictEqual(typeof user, 'object');
 
@@ -221,28 +154,26 @@ function passwordReset(user) {
     getMailConfig(function (error, mailConfig) {
         if (error) return debug('Error getting mail details:', error);
 
-        var templateData = {
-            user: user,
-            resetLink: `${settings.adminOrigin()}/login.html?resetToken=${user.resetToken}`,
-            cloudronName: mailConfig.cloudronName,
-            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
-        };
+        translation.getTranslations(function (error, translationAssets) {
+            if (error) return debug('Error getting translations:', error);
 
-        var templateDataText = JSON.parse(JSON.stringify(templateData));
-        templateDataText.format = 'text';
+            var templateData = {
+                user: user.displayName || user.username || user.email,
+                resetLink: `${settings.adminOrigin()}/login.html?resetToken=${user.resetToken}`,
+                cloudronName: mailConfig.cloudronName,
+                cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
+            };
 
-        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
-        templateDataHTML.format = 'html';
+            var mailOptions = {
+                from: mailConfig.notificationFrom,
+                to: user.fallbackEmail,
+                subject: ejs.render(translation.translate('{{ passwordResetEmail.subject }}', translationAssets.translations || {}, translationAssets.fallback || {}), { cloudron: mailConfig.cloudronName }),
+                text: render('password_reset-text.ejs', templateData, translationAssets),
+                html: render('password_reset-html.ejs', templateData, translationAssets)
+            };
 
-        var mailOptions = {
-            from: mailConfig.notificationFrom,
-            to: user.fallbackEmail,
-            subject: util.format('[%s] Password Reset', mailConfig.cloudronName),
-            text: render('password_reset.ejs', templateDataText),
-            html: render('password_reset.ejs', templateDataHTML)
-        };
-
-        sendMail(mailOptions);
+            sendMail(mailOptions);
+        });
     });
 }
 
@@ -258,7 +189,7 @@ function appUp(mailTo, app) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] App %s is back online', mailConfig.cloudronName, app.fqdn),
+            subject: `[${mailConfig.cloudronName}] App ${app.fqdn} is back online`,
             text: render('app_up.ejs', { title: app.manifest.title, appFqdn: app.fqdn, format: 'text' })
         };
 
@@ -278,7 +209,7 @@ function appDied(mailTo, app) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] App %s is down', mailConfig.cloudronName, app.fqdn),
+            subject: `[${mailConfig.cloudronName}] App ${app.fqdn} is down`,
             text: render('app_down.ejs', { title: app.manifest.title, appFqdn: app.fqdn, supportEmail: mailConfig.supportEmail, format: 'text' })
         };
 
@@ -325,10 +256,46 @@ function appUpdated(mailTo, app, callback) {
     });
 }
 
-function appUpdatesAvailable(mailTo, apps, hasSubscription, callback) {
+function boxUpdateAvailable(mailTo, updateInfo, callback) {
+    assert.strictEqual(typeof mailTo, 'string');
+    assert.strictEqual(typeof updateInfo, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    getMailConfig(function (error, mailConfig) {
+        if (error) return debug('Error getting mail details:', error);
+
+        var converter = new showdown.Converter();
+
+        var templateData = {
+            webadminUrl: settings.adminOrigin(),
+            newBoxVersion: updateInfo.version,
+            changelog: updateInfo.changelog,
+            changelogHTML: updateInfo.changelog.map(function (e) { return converter.makeHtml(e); }),
+            cloudronName: mailConfig.cloudronName,
+            cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
+        };
+
+        var templateDataText = JSON.parse(JSON.stringify(templateData));
+        templateDataText.format = 'text';
+
+        var templateDataHTML = JSON.parse(JSON.stringify(templateData));
+        templateDataHTML.format = 'html';
+
+        var mailOptions = {
+            from: mailConfig.notificationFrom,
+            to: mailTo,
+            subject: `[${mailConfig.cloudronName}] Cloudron update available`,
+            text: render('box_update_available.ejs', templateDataText),
+            html: render('box_update_available.ejs', templateDataHTML)
+        };
+
+        sendMail(mailOptions, callback);
+    });
+}
+
+function appUpdatesAvailable(mailTo, apps, callback) {
     assert.strictEqual(typeof mailTo, 'string');
     assert.strictEqual(typeof apps, 'object');
-    assert.strictEqual(typeof hasSubscription, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
     getMailConfig(function (error, mailConfig) {
@@ -341,7 +308,6 @@ function appUpdatesAvailable(mailTo, apps, hasSubscription, callback) {
 
         var templateData = {
             webadminUrl: settings.adminOrigin(),
-            hasSubscription: hasSubscription,
             apps: apps,
             cloudronName: mailConfig.cloudronName,
             cloudronAvatarUrl: settings.adminOrigin() + '/api/v1/cloudron/avatar'
@@ -356,7 +322,7 @@ function appUpdatesAvailable(mailTo, apps, hasSubscription, callback) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: `New app updates available for ${mailConfig.cloudronName}`,
+            subject: `[${mailConfig.cloudronName}] App update available`,
             text: render('app_updates_available.ejs', templateDataText),
             html: render('app_updates_available.ejs', templateDataHTML)
         };
@@ -374,7 +340,7 @@ function backupFailed(mailTo, errorMessage, logUrl) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] Failed to backup', mailConfig.cloudronName),
+            subject: `[${mailConfig.cloudronName}] Failed to backup`,
             text: render('backup_failed.ejs', { cloudronName: mailConfig.cloudronName, message: errorMessage, logUrl: logUrl, format: 'text' })
         };
 
@@ -393,7 +359,7 @@ function certificateRenewalError(mailTo, domain, message) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] Certificate renewal error', domain),
+            subject: `[${mailConfig.cloudronName}] Certificate renewal error`,
             text: render('certificate_renewal_error.ejs', { domain: domain, message: message, format: 'text' })
         };
 
@@ -411,7 +377,7 @@ function boxUpdateError(mailTo, message) {
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] Cloudron update error', mailConfig.cloudronName),
+            subject: `[${mailConfig.cloudronName}] Cloudron update error`,
             text: render('box_update_error.ejs', { message: message, format: 'text' })
         };
 
@@ -419,19 +385,30 @@ function boxUpdateError(mailTo, message) {
     });
 }
 
-function oomEvent(mailTo, program, event) {
+function oomEvent(mailTo, app, addon, containerId, event) {
     assert.strictEqual(typeof mailTo, 'string');
-    assert.strictEqual(typeof program, 'string');
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof addon, 'object');
+    assert.strictEqual(typeof containerId, 'string');
     assert.strictEqual(typeof event, 'object');
 
     getMailConfig(function (error, mailConfig) {
         if (error) return debug('Error getting mail details:', error);
 
+        const templateData = {
+            webadminUrl: settings.adminOrigin(),
+            cloudronName: mailConfig.cloudronName,
+            app,
+            addon,
+            event: JSON.stringify(event),
+            format: 'text'
+        };
+
         var mailOptions = {
             from: mailConfig.notificationFrom,
             to: mailTo,
-            subject: util.format('[%s] %s was restarted (OOM)', mailConfig.cloudronName, program),
-            text: render('oom_event.ejs', { cloudronName: mailConfig.cloudronName, program: program, event: JSON.stringify(event), format: 'text' })
+            subject: `[${mailConfig.cloudronName}] ${app ? app.fqdn : addon.name} was restarted (OOM)`,
+            text: render('oom_event.ejs', templateData)
         };
 
         sendMail(mailOptions);
@@ -450,7 +427,7 @@ function sendTestMail(domain, email, callback) {
             authUser: `no-reply@${domain}`,
             from: `"${mailConfig.cloudronName}" <no-reply@${domain}>`,
             to: email,
-            subject: util.format('Test Email from %s', mailConfig.cloudronName),
+            subject: `[${mailConfig.cloudronName}] Test Email`,
             text: render('test.ejs', { cloudronName: mailConfig.cloudronName, format: 'text'})
         };
 

src/middleware/index.js

@@ -1,10 +1,11 @@
 'use strict';
 
 exports = module.exports = {
+    cookieParser: require('cookie-parser'),
     cors: require('./cors'),
     json: require('body-parser').json,
     morgan: require('morgan'),
-    proxy: require('proxy-middleware'),
+    proxy: require('./proxy-middleware.js'),
     lastMile: require('connect-lastmile'),
     multipart: require('./multipart.js'),
     timeout: require('connect-timeout'),

src/middleware/proxy-middleware.js

@@ -0,0 +1,149 @@
+// https://github.com/cloudron-io/node-proxy-middleware
+// MIT license
+// contains https://github.com/gonzalocasas/node-proxy-middleware/pull/59
+
+var os = require('os');
+var http = require('http');
+var https = require('https');
+var owns = {}.hasOwnProperty;
+
+module.exports = function proxyMiddleware(options) {
+  //enable ability to quickly pass a url for shorthand setup
+  if(typeof options === 'string'){
+      options = require('url').parse(options);
+  }
+
+  var httpLib = options.protocol === 'https:' ? https : http;
+  var request = httpLib.request;
+
+  options = options || {};
+  options.hostname = options.hostname;
+  options.port = options.port;
+  options.pathname = options.pathname || '/';
+
+  return function (req, resp, next) {
+    var url = req.url;
+    // You can pass the route within the options, as well
+    if (typeof options.route === 'string') {
+      if (url === options.route) {
+        url = '';
+      } else if (url.slice(0, options.route.length) === options.route) {
+        url = url.slice(options.route.length);
+      } else {
+        return next();
+      }
+    }
+
+    //options for this request
+    var opts = extend({}, options);
+    if (url && url.charAt(0) === '?') { // prevent /api/resource/?offset=0
+      if (options.pathname.length > 1 && options.pathname.charAt(options.pathname.length - 1) === '/') {
+        opts.path = options.pathname.substring(0, options.pathname.length - 1) + url;
+      } else {
+        opts.path = options.pathname + url;
+      }
+    } else if (url) {
+      opts.path = slashJoin(options.pathname, url);
+    } else {
+      opts.path = options.pathname;
+    }
+    opts.method = req.method;
+    opts.headers = options.headers ? merge(req.headers, options.headers) : req.headers;
+
+    applyViaHeader(req.headers, opts, opts.headers);
+
+    if (!options.preserveHost) {
+      // Forwarding the host breaks dotcloud
+      delete opts.headers.host;
+    }
+
+    var myReq = request(opts, function (myRes) {
+      var statusCode = myRes.statusCode
+        , headers = myRes.headers
+        , location = headers.location;
+      // Fix the location
+      if (((statusCode > 300 && statusCode < 304) || statusCode === 201) && location && location.indexOf(options.href) > -1) {
+        // absoulte path
+        headers.location = location.replace(options.href, slashJoin('/', slashJoin((options.route || ''), '')));
+      }
+      applyViaHeader(myRes.headers, opts, myRes.headers);
+      rewriteCookieHosts(myRes.headers, opts, myRes.headers, req);
+      resp.writeHead(myRes.statusCode, myRes.headers);
+      myRes.on('error', function (err) {
+        next(err);
+      });
+      myRes.on('end', function (err) {
+        next();
+      });
+      myRes.pipe(resp);
+    });
+    myReq.on('error', function (err) {
+      next(err);
+    });
+    if (!req.readable) {
+      myReq.end();
+    } else {
+      req.pipe(myReq);
+    }
+  };
+};
+
+function applyViaHeader(existingHeaders, opts, applyTo) {
+  if (!opts.via) return;
+
+  var viaName = (true === opts.via) ?  os.hostname() : opts.via;
+  var viaHeader = '1.1 ' + viaName;
+  if(existingHeaders.via) {
+    viaHeader = existingHeaders.via + ', ' + viaHeader;
+  }
+
+  applyTo.via = viaHeader;
+}
+
+function rewriteCookieHosts(existingHeaders, opts, applyTo, req) {
+  if (!opts.cookieRewrite || !owns.call(existingHeaders, 'set-cookie')) {
+    return;
+  }
+
+  var existingCookies = existingHeaders['set-cookie'],
+      rewrittenCookies = [],
+      rewriteHostname = (true === opts.cookieRewrite) ? os.hostname() : opts.cookieRewrite;
+
+  if (!Array.isArray(existingCookies)) {
+    existingCookies = [ existingCookies ];
+  }
+
+  for (var i = 0; i < existingCookies.length; i++) {
+    var rewrittenCookie = existingCookies[i].replace(/(Domain)=[a-z\.-_]*?(;|$)/gi, '$1=' + rewriteHostname + '$2');
+
+    if (!req.connection.encrypted) {
+      rewrittenCookie = rewrittenCookie.replace(/;\s*?(Secure)/i, '');
+    }
+    rewrittenCookies.push(rewrittenCookie);
+  }
+
+  applyTo['set-cookie'] = rewrittenCookies;
+}
+
+function slashJoin(p1, p2) {
+  var trailing_slash = false;
+
+  if (p1.length && p1[p1.length - 1] === '/') { trailing_slash = true; }
+  if (trailing_slash && p2.length && p2[0] === '/') {p2 = p2.substring(1); }
+
+  return p1 + p2;
+}
+
+function extend(obj, src) {
+  for (var key in src) if (owns.call(src, key)) obj[key] = src[key];
+  return obj;
+}
+
+//merges data without changing state in either argument
+function merge(src1, src2) {
+    var merged = {};
+    extend(merged, src1);
+    extend(merged, src2);
+    return merged;
+}
+

src/nginxconfig.ejs

@@ -28,7 +28,13 @@ server {
         alias /home/yellowtent/platformdata/acme/;
     }
 
-    # for default server, serve the splash page. for other endpoints, redirect to HTTPS
+    location /notfound.html {
+        root <%= sourceDir %>/dashboard/dist;
+        try_files /notfound.html =404;
+        internal;
+    }
+
+    # for default server, serve the notfound page. for other endpoints, redirect to HTTPS
     location / {
 <% if ( endpoint === 'admin' || endpoint === 'setup' ) { %>
         return 301 https://$host$request_uri;
@@ -37,8 +43,8 @@ server {
 <% } else if ( endpoint === 'redirect' ) { %>
         return 301 https://<%= redirectTo %>$request_uri;
 <% } else if ( endpoint === 'ip' ) { %>
-        root <%= sourceDir %>/dashboard/dist;
-        try_files /splash.html =404;
+        root /home/yellowtent/boxdata;
+        try_files /custom_pages/notfound.html /notfound.html;
 <% } %>
     }
 }
@@ -89,12 +95,20 @@ server {
     proxy_hide_header X-Content-Type-Options;
     add_header X-Permitted-Cross-Domain-Policies "none";
     proxy_hide_header X-Permitted-Cross-Domain-Policies;
-    add_header Referrer-Policy "no-referrer-when-downgrade";
+    add_header Referrer-Policy "same-origin";
     proxy_hide_header Referrer-Policy;
 
+    # workaround caching issue after /logout. if max-age is set, browser uses cache and user thinks they have not logged out
+    # have to keep all the add_header here to avoid repeating all add_header in location block
+    <% if (proxyAuth.enabled) { %>
+    proxy_hide_header Cache-Control;
+    add_header Cache-Control no-cache;
+    add_header Set-Cookie $auth_cookie;
+    <% } %>
+
     # gzip responses that are > 50k and not images
     gzip on;
-    gzip_min_length 50k;
+    gzip_min_length 18k;
     gzip_types text/css text/javascript text/xml text/plain application/javascript application/x-javascript application/json;
 
     # enable for proxied requests as well
@@ -147,86 +161,152 @@ server {
 <% if ( endpoint === 'admin' ) { %>
         proxy_pass http://127.0.0.1:3000;
 <% } else if ( endpoint === 'app' ) { %>
-        proxy_pass http://127.0.0.1:<%= port %>;
+        proxy_pass http://<%= ip %>:<%= port %>;
 <% } else if ( endpoint === 'redirect' ) { %>
         return 302 https://<%= redirectTo %>$request_uri;
 <% } %>
     }
 
     # user defined .well-known resources
-    location ~ ^/.well-known/(.*)$ {
-        root /home/yellowtent/boxdata/well-known/$host;
-        try_files /$1 @wellknown-upstream;
+    location /.well-known/ {
+        error_page 404 = @wellknown-upstream;
+        proxy_pass http://127.0.0.1:3000/well-known-handler/;
     }
 
-    location / {
-        # increase the proxy buffer sizes to not run into buffer issues (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers)
-        proxy_buffer_size       128k;
-        proxy_buffers           4 256k;
-        proxy_busy_buffers_size 256k;
+<% if (proxyAuth.enabled) { %>
+    proxy_set_header X-App-ID "<%= proxyAuth.id %>";
+<% } %>
 
-        # No buffering to temp files, it fails for large downloads
-        proxy_max_temp_file_size 0;
+    # increase the proxy buffer sizes to not run into buffer issues (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers)
+    proxy_buffer_size       128k;
+    proxy_buffers           4 256k;
+    proxy_busy_buffers_size 256k;
 
-        # Disable check to allow unlimited body sizes. this allows apps to accept whatever size they want
-        client_max_body_size 0;
+    # No buffering to temp files, it fails for large downloads
+    proxy_max_temp_file_size 0;
+
+    # Disable check to allow unlimited body sizes. this allows apps to accept whatever size they want
+    client_max_body_size 0;
 
 <% if (robotsTxtQuoted) { %>
-        location = /robots.txt {
-            return 200 <%- robotsTxtQuoted %>;
-        }
+    location = /robots.txt {
+        return 200 <%- robotsTxtQuoted %>;
+    }
 <% } %>
 
 <% if ( endpoint === 'admin' || endpoint === 'setup' ) { %>
-        location /api/ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 1m;
-        }
+    location /api/ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 1m;
+    }
 
-        location ~ ^/api/v1/(developer|session)/login$ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 1m;
-            limit_req zone=admin_login burst=5;
-        }
+    location ~ ^/api/v1/(developer|session)/login$ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 1m;
+        limit_req zone=admin_login burst=5;
+    }
 
-        # the read timeout is between successive reads and not the whole connection
-        location ~ ^/api/v1/apps/.*/exec$ {
-            proxy_pass   http://127.0.0.1:3000;
-            proxy_read_timeout 30m;
-        }
+    # the read timeout is between successive reads and not the whole connection
+    location ~ ^/api/v1/apps/.*/exec$ {
+        proxy_pass   http://127.0.0.1:3000;
+        proxy_read_timeout 30m;
+    }
 
-        location ~ ^/api/v1/apps/.*/upload$ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 0;
-        }
+    location ~ ^/api/v1/apps/.*/upload$ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 0;
+    }
 
-        location ~ ^/api/v1/apps/.*/files/ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 0;
-        }
+    location ~ ^/api/v1/apps/.*/files/ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 0;
+    }
 
-        # graphite paths (uncomment block below and visit /graphite-web/dashboard)
-        # remember to comment out the CSP policy as well to access the graphite dashboard
-        # location ~ ^/graphite-web/ {
-        #     proxy_pass   http://127.0.0.1:8417;
-        #     client_max_body_size 1m;
-        # }
+    location ~ ^/api/v1/volumes/.*/files/ {
+        proxy_pass   http://127.0.0.1:3000;
+        client_max_body_size 0;
+    }
 
-        location / {
-            root   <%= sourceDir %>/dashboard/dist;
-            index  index.html index.htm;
-        }
+    # graphite paths (uncomment block below and visit /graphite-web/dashboard)
+    # remember to comment out the CSP policy as well to access the graphite dashboard
+    # location ~ ^/graphite-web/ {
+    #     proxy_pass   http://127.0.0.1:8417;
+    #     client_max_body_size 1m;
+    # }
+
+    location / {
+        root   <%= sourceDir %>/dashboard/dist;
+        index  index.html index.htm;
+    }
 <% } else if ( endpoint === 'app' ) { %>
-        proxy_pass http://127.0.0.1:<%= port %>;
+    location = /appstatus.html {
+        root /home/yellowtent/box/dashboard/dist;
+    }
+
+    <% if (proxyAuth.enabled) { %>
+    location = /proxy-auth {
+        internal;
+        proxy_pass http://127.0.0.1:3001/auth;
+        proxy_pass_request_body off;
+        # repeat proxy headers since we addded proxy_set_header at this location level
+        proxy_set_header X-App-ID "<%= proxyAuth.id %>";
+        proxy_set_header Content-Length "";
+    }
+
+    location ~ ^/(login|logout)$ {
+        proxy_pass http://127.0.0.1:3001;
+    }
+
+    location @proxy-auth-login {
+        if ($http_user_agent ~* "docker") {
+            return 401;
+        }
+        return 302 /login?redirect=$request_uri;
+    }
+
+    location <%= proxyAuth.location %> {
+        auth_request /proxy-auth;
+        auth_request_set $auth_cookie $upstream_http_set_cookie;
+        error_page 401 = @proxy-auth-login;
+
+        proxy_pass http://<%= ip %>:<%= port %>;
+    }
+
+    <% if (proxyAuth.location !== '/') { %>
+    location / {
+        proxy_pass http://<%= ip %>:<%= port %>;
+    }
+    <% } %>
+
+    <% } else { %>
+    location / {
+        proxy_pass http://<%= ip %>:<%= port %>;
+    }
+    <% } %>
+
+    <% Object.keys(httpPaths).forEach(function (path) { -%>
+    location "<%= path %>" {
+        # the trailing / will replace part of the original URI matched by the location.
+        proxy_pass http://<%= ip %>:<%= httpPaths[path] %>/;
+    }
+    <% }); %>
+
 <% } else if ( endpoint === 'redirect' ) { %>
+    location / {
         # redirect everything to the app. this is temporary because there is no way
         # to clear a permanent redirect on the browser
         return 302 https://<%= redirectTo %>$request_uri;
-<% } else if ( endpoint === 'ip' ) { %>
-        location / {
-            root   <%= sourceDir %>/dashboard/dist;
-            try_files /splash.html =404;
-        }
-<% } %>
     }
+<% } else if ( endpoint === 'ip' ) { %>
+    location /notfound.html {
+        root <%= sourceDir %>/dashboard/dist;
+        try_files /notfound.html =404;
+        internal;
+    }
+
+    location / {
+        root /home/yellowtent/boxdata;
+        try_files /custom_pages/notfound.html /notfound.html;
+    }
+<% } %>
 }

src/notifications.js

@@ -1,11 +1,14 @@
 'use strict';
 
 exports = module.exports = {
-    get: get,
-    ack: ack,
-    getAllPaged: getAllPaged,
+    get,
+    ack,
+    getAllPaged,
 
-    onEvent: onEvent,
+    onEvent,
+
+    appUpdatesAvailable,
+    boxUpdateAvailable,
 
     // NOTE: if you add an alert, be sure to add title below
     ALERT_BACKUP_CONFIG: 'backupConfig',
@@ -20,11 +23,13 @@ exports = module.exports = {
     _add: add
 };
 
-let assert = require('assert'),
+let apps = require('./apps.js'),
+    assert = require('assert'),
     async = require('async'),
     auditSource = require('./auditsource.js'),
     BoxError = require('./boxerror.js'),
     changelog = require('./changelog.js'),
+    constants = require('./constants.js'),
     debug = require('debug')('box:notifications'),
     eventlog = require('./eventlog.js'),
     mailer = require('./mailer.js'),
@@ -94,8 +99,8 @@ function getAllPaged(userId, acknowledged, page, perPage, callback) {
 }
 
 // Calls iterator with (admin, callback)
-function actionForAllAdmins(skippingUserIds, iterator, callback) {
-    assert(Array.isArray(skippingUserIds));
+function forEachAdmin(options, iterator, callback) {
+    assert(Array.isArray(options.skip));
     assert.strictEqual(typeof iterator, 'function');
     assert.strictEqual(typeof callback, 'function');
 
@@ -104,7 +109,7 @@ function actionForAllAdmins(skippingUserIds, iterator, callback) {
         if (error) return callback(error);
 
         // filter out users we want to skip (like the user who did the action or the user the action was performed on)
-        result = result.filter(function (r) { return skippingUserIds.indexOf(r.id) === -1; });
+        result = result.filter(function (r) { return options.skip.indexOf(r.id) === -1; });
 
         async.each(result, iterator, callback);
     });
@@ -116,8 +121,7 @@ function userAdded(performedBy, eventId, user, callback) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([ performedBy, user.id ], function (admin, done) {
-        mailer.userAdded(admin.email, user);
+    forEachAdmin({ skip: [ performedBy, user.id ] }, function (admin, done) {
         add(admin.id, eventId, `User '${user.displayName}' added`, `User '${user.username || user.email || user.fallbackEmail}' was added.`, done);
     }, callback);
 }
@@ -128,8 +132,7 @@ function userRemoved(performedBy, eventId, user, callback) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([ performedBy, user.id ], function (admin, done) {
-        mailer.userRemoved(admin.email, user);
+    forEachAdmin({ skip: [ performedBy, user.id ] }, function (admin, done) {
         add(admin.id, eventId, `User '${user.displayName}' removed`, `User '${user.username || user.email || user.fallbackEmail}' was removed.`, done);
     }, callback);
 }
@@ -139,8 +142,7 @@ function roleChanged(performedBy, eventId, user, callback) {
     assert.strictEqual(typeof user, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([ performedBy, user.id ], function (admin, done) {
-        mailer.roleChanged(admin.email, user);
+    forEachAdmin({ skip: [ performedBy, user.id ] }, function (admin, done) {
         add(admin.id, eventId, `User '${user.displayName}'s role changed`, `User '${user.username || user.email || user.fallbackEmail}' now has the role ${user.role}.`, done);
     }, callback);
 }
@@ -152,23 +154,19 @@ function oomEvent(eventId, app, addon, containerId, event, callback) {
     assert.strictEqual(typeof containerId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    let title, message, program;
+    assert(app || addon);
+
+    let title, message;
     if (app) {
-        program = `App ${app.fqdn}`;
         title = `The application at ${app.fqdn} ran out of memory.`;
-        message = 'The application has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://docs.cloudron.io/apps/#memory-limit)';
+        message = `The application has been restarted automatically. If you see this notification often, consider increasing the [memory limit](${settings.adminOrigin()}/#/app/${app.id}/resources)`;
     } else if (addon) {
-        program = `${addon.name} service`;
         title = `The ${addon.name} service ran out of memory`;
-        message = 'The service has been restarted automatically. If you see this notification often, consider increasing the [memory limit](https://docs.cloudron.io/troubleshooting/#services)';
-    } else { // this never happens currently
-        program = `Container ${containerId}`;
-        title = `The container ${containerId} ran out of memory`;
-        message = 'The container has been restarted automatically. Consider increasing the [memory limit](https://docs.docker.com/v17.09/edge/engine/reference/commandline/update/#update-a-containers-kernel-memory-constraints)';
+        message = `The service has been restarted automatically. If you see this notification often, consider increasing the [memory limit](${settings.adminOrigin()}/#/services)`;
     }
 
-    actionForAllAdmins([], function (admin, done) {
-        mailer.oomEvent(admin.email, program, event);
+    forEachAdmin({ skip: [] }, function (admin, done) {
+        mailer.oomEvent(admin.email, app, addon, containerId, event);
 
         add(admin.id, eventId, title, message, done);
     }, callback);
@@ -179,7 +177,7 @@ function appUp(eventId, app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, done) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
         mailer.appUp(admin.email, app);
         add(admin.id, eventId, `App ${app.fqdn} is back online`, `The application installed at ${app.fqdn} is back online.`, done);
     }, callback);
@@ -190,7 +188,7 @@ function appDied(eventId, app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         mailer.appDied(admin.email, app);
         add(admin.id, eventId, `App ${app.fqdn} is down`, `The application installed at ${app.fqdn} is not responding.`, callback);
     }, callback);
@@ -208,7 +206,7 @@ function appUpdated(eventId, app, callback) {
     const title = upstreamVersion ? `${app.manifest.title} at ${app.fqdn} updated to ${upstreamVersion} (package version ${app.manifest.version})`
         : `${app.manifest.title} at ${app.fqdn} updated to package version ${app.manifest.version}`;
 
-    actionForAllAdmins([], function (admin, done) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
         add(admin.id, eventId, title, `The application installed at https://${app.fqdn} was updated.\n\nChangelog:\n${app.manifest.changelog}\n`, function (error) {
             if (error) return callback(error);
 
@@ -220,6 +218,39 @@ function appUpdated(eventId, app, callback) {
     }, callback);
 }
 
+function boxUpdateAvailable(updateInfo, callback) {
+    assert.strictEqual(typeof updateInfo, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    settings.getAutoupdatePattern(function (error, result) {
+        if (error) return callback(error);
+
+        if (result !== constants.AUTOUPDATE_PATTERN_NEVER) return callback();
+
+        forEachAdmin({ skip: [] }, function (admin, done) {
+            mailer.boxUpdateAvailable(admin.email, updateInfo, done);
+        }, callback);
+    });
+}
+
+function appUpdatesAvailable(appUpdates, callback) {
+    assert.strictEqual(typeof appUpdates, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    settings.getAutoupdatePattern(function (error, result) {
+        if (error) return callback(error);
+
+        // if we are auto updating, then just consider apps that cannot be auto updated
+        if (result !== constants.AUTOUPDATE_PATTERN_NEVER) appUpdates = appUpdates.filter(update => !apps.canAutoupdateApp(update.app, update.updateInfo));
+
+        if (appUpdates.length === 0) return callback();
+
+        forEachAdmin({ skip: [] }, function (admin, done) {
+            mailer.appUpdatesAvailable(admin.email, appUpdates, done);
+        }, callback);
+    });
+}
+
 function boxUpdated(eventId, oldVersion, newVersion, callback) {
     assert.strictEqual(typeof eventId, 'string');
     assert.strictEqual(typeof oldVersion, 'string');
@@ -229,7 +260,7 @@ function boxUpdated(eventId, oldVersion, newVersion, callback) {
     const changes = changelog.getChanges(newVersion);
     const changelogMarkdown = changes.map((m) => `* ${m}\n`).join('');
 
-    actionForAllAdmins([], function (admin, done) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
         add(admin.id, eventId, `Cloudron updated to v${newVersion}`, `Cloudron was updated from v${oldVersion} to v${newVersion}.\n\nChangelog:\n${changelogMarkdown}\n`, done);
     }, callback);
 }
@@ -239,7 +270,7 @@ function boxUpdateError(eventId, errorMessage, callback) {
     assert.strictEqual(typeof errorMessage, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, done) {
+    forEachAdmin({ skip: [] }, function (admin, done) {
         mailer.boxUpdateError(admin.email, errorMessage);
         add(admin.id, eventId, 'Cloudron update failed', `Failed to update Cloudron: ${errorMessage}.`, done);
     }, callback);
@@ -251,7 +282,7 @@ function certificateRenewalError(eventId, vhost, errorMessage, callback) {
     assert.strictEqual(typeof errorMessage, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         mailer.certificateRenewalError(admin.email, vhost, errorMessage);
         add(admin.id, eventId, `Certificate renewal of ${vhost} failed`, `Failed to new certs of ${vhost}: ${errorMessage}. Renewal will be retried in 12 hours`, callback);
     }, callback);
@@ -263,7 +294,7 @@ function backupFailed(eventId, taskId, errorMessage, callback) {
     assert.strictEqual(typeof errorMessage, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         mailer.backupFailed(admin.email, errorMessage, `${settings.adminOrigin()}/logs.html?taskId=${taskId}`);
         add(admin.id, eventId, 'Backup failed', `Backup failed: ${errorMessage}. Logs are available [here](/logs.html?taskId=${taskId}).`, callback);
     }, callback);
@@ -275,11 +306,10 @@ function alert(id, title, message, callback) {
     assert.strictEqual(typeof message, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`alert: id=${id} title=${title}`);
-
     const acknowledged = !message;
+    debug(`alert: id=${id} title=${title} ack=${acknowledged}`);
 
-    actionForAllAdmins([], function (admin, callback) {
+    forEachAdmin({ skip: [] }, function (admin, callback) {
         const data = {
             userId: admin.id,
             eventId: null,

src/paths.js

@@ -16,8 +16,10 @@ exports = module.exports = {
 
     CLOUDRON_DEFAULT_AVATAR_FILE: path.join(__dirname + '/../assets/avatar.png'),
     INFRA_VERSION_FILE: path.join(baseDir(), 'platformdata/INFRA_VERSION'),
+    DASHBOARD_DIR: constants.TEST ? path.join(__dirname, '../../dashboard/src') : path.join(baseDir(), 'box/dashboard/dist'),
 
     PROVIDER_FILE: '/etc/cloudron/PROVIDER',
+    SETUP_TOKEN_FILE: '/etc/cloudron/SETUP_TOKEN',
 
     PLATFORM_DATA_DIR: path.join(baseDir(), 'platformdata'),
     APPS_DATA_DIR: path.join(baseDir(), 'appsdata'),
@@ -35,12 +37,14 @@ exports = module.exports = {
     SNAPSHOT_INFO_FILE: path.join(baseDir(), 'platformdata/backup/snapshot-info.json'),
     DYNDNS_INFO_FILE: path.join(baseDir(), 'platformdata/dyndns-info.json'),
     FEATURES_INFO_FILE: path.join(baseDir(), 'platformdata/features-info.json'),
+    PROXY_AUTH_TOKEN_SECRET_FILE: path.join(baseDir(), 'platformdata/proxy-auth-token-secret'),
     VERSION_FILE: path.join(baseDir(), 'platformdata/VERSION'),
 
     // this is not part of appdata because an icon may be set before install
     APP_ICONS_DIR: path.join(baseDir(), 'boxdata/appicons'),
     PROFILE_ICONS_DIR: path.join(baseDir(), 'boxdata/profileicons'),
     MAIL_DATA_DIR: path.join(baseDir(), 'boxdata/mail'),
+    SFTP_KEYS_DIR: path.join(baseDir(), 'boxdata/sftp/ssh'),
     ACME_ACCOUNT_KEY_FILE: path.join(baseDir(), 'boxdata/acme/acme.key'),
     APP_CERTS_DIR: path.join(baseDir(), 'boxdata/certs'),
     CLOUDRON_AVATAR_FILE: path.join(baseDir(), 'boxdata/avatar.png'),
@@ -56,7 +60,9 @@ exports = module.exports = {
 
     GHOST_USER_FILE: path.join(baseDir(), 'platformdata/cloudron_ghost.json'),
 
+    SWAP_RATIO_FILE: path.join(baseDir(), 'platformdata/swap-ratio'),
+
     // this pattern is for the cloudron logs API route to work
     BACKUP_LOG_FILE: path.join(baseDir(), 'platformdata/logs/backup/app.log'),
-    UPDATER_LOG_FILE: path.join(baseDir(), 'platformdata/logs/updater/app.log')
+    UPDATER_LOG_FILE: path.join(baseDir(), 'platformdata/logs/updater/app.log'),
 };

src/platform.js

@@ -1,32 +1,30 @@
 'use strict';
 
 exports = module.exports = {
-    start: start,
-    stopAllTasks: stopAllTasks,
+    start,
+    stopAllTasks,
 
     // exported for testing
     _isReady: false
 };
 
-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+const apps = require('./apps.js'),
     assert = require('assert'),
     async = require('async'),
     debug = require('debug')('box:platform'),
     fs = require('fs'),
-    graphs = require('./graphs.js'),
     infra = require('./infra_version.js'),
     locker = require('./locker.js'),
     paths = require('./paths.js'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
-    settings = require('./settings.js'),
-    sftp = require('./sftp.js'),
+    services = require('./services.js'),
     shell = require('./shell.js'),
     tasks = require('./tasks.js'),
     _ = require('underscore');
 
-function start(callback) {
+function start(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     if (process.env.BOX_ENV === 'test' && !process.env.TEST_CREATE_INFRA) return callback();
@@ -54,11 +52,9 @@ function start(callback) {
     if (error) return callback(error);
 
     async.series([
-        (next) => { if (existingInfra.version !== infra.version) removeAllContainers(existingInfra, next); else next(); },
-        markApps.bind(null, existingInfra), // mark app state before we start addons. this gives the db import logic a chance to mark an app as errored
-        graphs.startGraphite.bind(null, existingInfra),
-        sftp.startSftp.bind(null, existingInfra),
-        addons.startServices.bind(null, existingInfra),
+        (next) => { if (existingInfra.version !== infra.version) removeAllContainers(next); else next(); },
+        markApps.bind(null, existingInfra, options), // mark app state before we start addons. this gives the db import logic a chance to mark an app as errored
+        services.startServices.bind(null, existingInfra),
         fs.writeFile.bind(fs, paths.INFRA_VERSION_FILE, JSON.stringify(infra, null, 4))
     ], function (error) {
         if (error) return callback(error);
@@ -80,7 +76,7 @@ function onPlatformReady(infraChanged) {
     exports._isReady = true;
 
     let tasks = [ apps.schedulePendingTasks ];
-    if (infraChanged) tasks.push(applyPlatformConfig, pruneInfraImages);
+    if (infraChanged) tasks.push(pruneInfraImages);
 
     async.series(async.reflectAll(tasks), function (error, results) {
         results.forEach((result, idx) => {
@@ -89,14 +85,6 @@ function onPlatformReady(infraChanged) {
     });
 }
 
-function applyPlatformConfig(callback) {
-    settings.getPlatformConfig(function (error, platformConfig) {
-        if (error) return callback(error);
-
-        addons.updateServiceConfig(platformConfig, callback);
-    });
-}
-
 function pruneInfraImages(callback) {
     debug('pruneInfraImages: checking existing images');
 
@@ -115,14 +103,14 @@ function pruneInfraImages(callback) {
             debug(`pruneInfraImages: removing unused image of ${image.repo}: tag: ${parts[1]} id: ${parts[0]}`);
 
             let result = safe.child_process.execSync(`docker rmi ${parts[0]}`, { encoding: 'utf8' });
-            if (result === null) debug(`Erroring removing image ${parts[0]}: ${safe.error.mesage}`);
+            if (result === null) debug(`Error removing image ${parts[0]}: ${safe.error.mesage}`);
         }
 
         iteratorCallback();
     }, callback);
 }
 
-function removeAllContainers(existingInfra, callback) {
+function removeAllContainers(callback) {
     debug('removeAllContainers: removing all containers for infra upgrade');
 
     async.series([
@@ -131,10 +119,14 @@ function removeAllContainers(existingInfra, callback) {
     ], callback);
 }
 
-function markApps(existingInfra, callback) {
+function markApps(existingInfra, options, callback) {
+    assert.strictEqual(typeof existingInfra, 'object');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
     if (existingInfra.version === 'none') { // cloudron is being restored from backup
         debug('markApps: restoring installed apps');
-        apps.restoreInstalledApps(callback);
+        apps.restoreInstalledApps(options, callback);
     } else if (existingInfra.version !== infra.version) {
         debug('markApps: reconfiguring installed apps');
         reverseProxy.removeAppConfigs(); // should we change the cert location, nginx will not start

src/provision.js

@@ -11,6 +11,7 @@ var assert = require('assert'),
     async = require('async'),
     backups = require('./backups.js'),
     BoxError = require('./boxerror.js'),
+    branding = require('./branding.js'),
     constants = require('./constants.js'),
     cloudron = require('./cloudron.js'),
     debug = require('debug')('box:provision'),
@@ -148,16 +149,17 @@ function activate(username, password, email, displayName, ip, auditSource, callb
                 expires: result.expires
             });
 
-            setImmediate(cloudron.onActivated.bind(null, NOOP_CALLBACK)); // hack for now to not block the above http response
+            setImmediate(cloudron.onActivated.bind(null, {}, NOOP_CALLBACK)); // hack for now to not block the above http response
         });
     });
 }
 
-function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, callback) {
+function restore(backupConfig, backupId, version, sysinfoConfig, options, auditSource, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof backupId, 'string');
     assert.strictEqual(typeof version, 'string');
     assert.strictEqual(typeof sysinfoConfig, 'object');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -202,7 +204,10 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
                     (done) => {
                         const adminDomain = settings.adminDomain(); // load this fresh from after the backup.restore
                         async.series([
-                            cloudron.setupDnsAndCert.bind(null, constants.ADMIN_LOCATION, adminDomain, auditSource, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK)),
+                            (next) => {
+                                if (options.skipDnsSetup) return next();
+                                cloudron.setupDnsAndCert(constants.ADMIN_LOCATION, adminDomain, auditSource, (progress) => setProgress('restore', progress.message, NOOP_CALLBACK), next);
+                            },
                             cloudron.setDashboardDomain.bind(null, adminDomain, auditSource)
                         ], done);
                     },
@@ -212,7 +217,7 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
                     gProvisionStatus.restore.active = false;
                     gProvisionStatus.restore.errorMessage = error ? error.message : '';
 
-                    if (!error) cloudron.onActivated(NOOP_CALLBACK);
+                    if (!error) cloudron.onActivated(options, NOOP_CALLBACK);
                 });
             });
         });
@@ -233,8 +238,9 @@ function getStatus(callback) {
                 apiServerOrigin: settings.apiServerOrigin(), // used by CaaS tool
                 webServerOrigin: settings.webServerOrigin(), // used by CaaS tool
                 cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-                footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
+                footer: branding.renderFooter(allSettings[settings.FOOTER_KEY] || constants.FOOTER),
                 adminFqdn: settings.adminDomain() ? settings.adminFqdn() : null,
+                language: allSettings[settings.LANGUAGE_KEY],
                 activated: activated,
                 provider: settings.provider() // used by setup wizard of marketplace images
             }, gProvisionStatus));

src/proxyauth.js

@@ -0,0 +1,266 @@
+'use strict';
+
+// heavily inspired from https://gock.net/blog/2020/nginx-subrequest-authentication-server/ and https://github.com/andygock/auth-server
+
+exports = module.exports = {
+    start,
+    stop
+};
+
+const apps = require('./apps.js'),
+    assert = require('assert'),
+    basicAuth = require('basic-auth'),
+    constants = require('./constants.js'),
+    debug = require('debug')('box:proxyAuth'),
+    ejs = require('ejs'),
+    express = require('express'),
+    fs = require('fs'),
+    hat = require('./hat.js'),
+    http = require('http'),
+    HttpError = require('connect-lastmile').HttpError,
+    HttpSuccess = require('connect-lastmile').HttpSuccess,
+    jwt = require('jsonwebtoken'),
+    middleware = require('./middleware'),
+    path = require('path'),
+    paths = require('./paths.js'),
+    safe = require('safetydance'),
+    speakeasy = require('speakeasy'),
+    translation = require('./translation.js'),
+    users = require('./users.js');
+
+let gHttpServer = null;
+let TOKEN_SECRET = null;
+const EXPIRY_DAYS = 7;
+
+function jwtVerify(req, res, next) {
+    const token = req.cookies.authToken;
+
+    if (!token) return next();
+
+    jwt.verify(token, TOKEN_SECRET, function (error, decoded) {
+        if (error) {
+            debug('clearing token', error);
+            res.clearCookie('authToken');
+            return next(new HttpError(403, 'Malformed token or bad signature'));
+        }
+
+        req.user = decoded.user || null;
+        next();
+    });
+}
+
+function basicAuthVerify(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    const credentials = basicAuth(req);
+    if (!appId || !credentials) return next();
+
+    const api = credentials.name.indexOf('@') !== -1 ? users.verifyWithEmail : users.verifyWithUsername;
+
+    apps.get(appId, function (error, app) {
+        if (error) return next(new HttpError(503, error.message));
+
+        if (!app.manifest.addons.proxyAuth.basicAuth) return next();
+
+        api(credentials.name, credentials.pass, appId, function (error, user) {
+            if (error) return next(new HttpError(403, 'Invalid username or password' ));
+
+            req.user = user;
+            next();
+        });
+    });
+}
+
+function loginPage(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    translation.getTranslations(function (error, translationAssets) {
+        if (error) return next(new HttpError(500, 'No translation found'));
+
+        const raw = safe.fs.readFileSync(path.join(paths.DASHBOARD_DIR, 'templates/proxyauth-login.ejs'), 'utf8');
+        if (raw === null) return next(new HttpError(500, 'Login template not found'));
+
+        const translatedContent = translation.translate(raw, translationAssets.translations || {}, translationAssets.fallback || {});
+        var finalContent = '';
+
+        apps.get(appId, function (error, app) {
+            if (error) return next(new HttpError(503, error.message));
+
+            const title = app.label || app.manifest.title;
+
+            apps.getIconPath(app, {}, function (error, iconPath) {
+                const icon = 'data:image/png;base64,' + safe.fs.readFileSync(iconPath || '', 'base64');
+
+                try {
+                    finalContent = ejs.render(translatedContent, { title, icon });
+                } catch (e) {
+                    debug('Error rendering proxyauth-login.ejs', e);
+                    return next(new HttpError(500, 'Login template error'));
+                }
+
+                res.set('Content-Type', 'text/html');
+                return res.send(finalContent);
+            });
+        });
+    });
+}
+
+// someday this can be more sophisticated and check for a real browser
+function isBrowser(req) {
+    const userAgent = req.get('user-agent');
+    if (!userAgent) return false;
+
+    // https://github.com/docker/engine/blob/master/dockerversion/useragent.go#L18
+    return !userAgent.toLowerCase().includes('docker');
+}
+
+// called by nginx to authorize any protected route. this route must return only 2xx or 401/403 (http://nginx.org/en/docs/http/ngx_http_auth_request_module.html)
+function auth(req, res, next) {
+    if (!req.user) {
+        if (isBrowser(req)) return next(new HttpError(401, 'Unauthorized'));
+
+        // the header has to be generated here and cannot be set in nginx config - https://forum.nginx.org/read.php?2,171461,171469#msg-171469
+        res.set('www-authenticate', 'Basic realm="Cloudron"');
+        return next(new HttpError(401, 'Unauthorized'));
+    }
+
+    // user is already authenticated, refresh cookie
+    const token = jwt.sign({ user: req.user }, TOKEN_SECRET, { expiresIn: `${EXPIRY_DAYS}d` });
+
+    res.cookie('authToken', token, {
+        httpOnly: true,
+        maxAge: EXPIRY_DAYS * 86400 * 1000, // milliseconds
+        secure: true
+    });
+
+    return next(new HttpSuccess(200, {}));
+}
+
+// endpoint called by login page, username and password posted as JSON body
+function passwordAuth(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    if (typeof req.body.username !== 'string') return next(new HttpError(400, 'username must be non empty string' ));
+    if (typeof req.body.password !== 'string') return next(new HttpError(400, 'password must be non empty string' ));
+    if ('totpToken' in req.body && typeof req.body.totpToken !== 'string') return next(new HttpError(400, 'totpToken must be a string' ));
+
+    const { username, password, totpToken } = req.body;
+
+    const api = username.indexOf('@') !== -1 ? users.verifyWithEmail : users.verifyWithUsername;
+
+    api(username, password, appId, function (error, user) {
+        if (error) return next(new HttpError(403, 'Invalid username or password' ));
+
+        if (!user.ghost && !user.appPassword && user.twoFactorAuthenticationEnabled) {
+            if (!totpToken) return next(new HttpError(403, 'A totpToken must be provided'));
+
+            let verified = speakeasy.totp.verify({ secret: user.twoFactorAuthenticationSecret, encoding: 'base32', token: req.body.totpToken, window: 2 });
+            if (!verified) return next(new HttpError(403, 'Invalid totpToken'));
+        }
+
+        req.user = user;
+        next();
+    });
+}
+
+function authorize(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    apps.get(appId, function (error, app) {
+        if (error) return next(new HttpError(403, 'No such app' ));
+
+        apps.hasAccessTo(app, req.user, function (error, hasAccess) {
+            if (error) return next(new HttpError(403, 'Forbidden' ));
+            if (!hasAccess) return next(new HttpError(403, 'Forbidden' ));
+
+            const token = jwt.sign({ user: users.removePrivateFields(req.user) }, TOKEN_SECRET, { expiresIn: `${EXPIRY_DAYS}d` });
+
+            res.cookie('authToken', token, {
+                httpOnly: true,
+                maxAge: EXPIRY_DAYS * 86400 * 1000, // milliseconds
+                secure: true
+            });
+
+            res.redirect(302, '/');
+        });
+    });
+}
+
+function logoutPage(req, res, next) {
+    const appId = req.headers['x-app-id'] || '';
+    if (!appId) return next(new HttpError(503, 'Nginx misconfiguration'));
+
+    apps.get(appId, function (error, app) {
+        if (error) return next(new HttpError(503, error.message));
+
+        res.clearCookie('authToken');
+
+        // when we have no path, redirect to the login page. we cannot redirect to '/' because browsers will immediately serve up the cached page
+        // if a path is set, we can assume '/' is a public page
+        res.redirect(302, app.manifest.addons.proxyAuth.path ? '/' : '/login');
+    });
+}
+
+function logout(req, res, next) {
+    res.clearCookie('authToken');
+    next(new HttpSuccess(200, {}));
+}
+
+// provides webhooks for the auth wall
+function initializeAuthwallExpressSync() {
+    let app = express();
+    let httpServer = http.createServer(app);
+
+    let QUERY_LIMIT = '1mb'; // max size for json and urlencoded queries
+    let REQUEST_TIMEOUT = 10000; // timeout for all requests
+
+    let json = middleware.json({ strict: true, limit: QUERY_LIMIT }); // application/json
+
+    if (process.env.BOX_ENV !== 'test') app.use(middleware.morgan('proxyauth :method :url :status :response-time ms - :res[content-length]', { immediate: false }));
+
+    var router = new express.Router();
+    router.del = router.delete; // amend router.del for readability further on
+
+    app
+        .use(middleware.timeout(REQUEST_TIMEOUT))
+        .use(middleware.cookieParser())
+        .use(router)
+        .use(middleware.lastMile());
+
+    router.get ('/login',     loginPage);
+    router.get ('/auth',      jwtVerify, basicAuthVerify, auth);
+    router.post('/login',     json, passwordAuth, authorize);
+    router.get ('/logout',    logoutPage);
+    router.post('/logout',    json, logout);
+
+    return httpServer;
+}
+
+function start(callback) {
+    assert.strictEqual(typeof callback, 'function');
+    assert.strictEqual(gHttpServer, null, 'Authwall is already up and running.');
+
+    if (!fs.existsSync(paths.PROXY_AUTH_TOKEN_SECRET_FILE)) {
+        TOKEN_SECRET = hat(64);
+        fs.writeFileSync(paths.PROXY_AUTH_TOKEN_SECRET_FILE, TOKEN_SECRET, 'utf8');
+    } else {
+        TOKEN_SECRET = fs.readFileSync(paths.PROXY_AUTH_TOKEN_SECRET_FILE, 'utf8').trim();
+    }
+
+    gHttpServer = initializeAuthwallExpressSync();
+
+    gHttpServer.listen(constants.AUTHWALL_PORT, '127.0.0.1', callback);
+}
+
+function stop(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!gHttpServer) return callback(null);
+
+    gHttpServer.close(callback);
+    gHttpServer = null;
+}

src/reverseproxy.js

@@ -57,6 +57,14 @@ var acme2 = require('./cert/acme2.js'),
 var NGINX_APPCONFIG_EJS = fs.readFileSync(__dirname + '/nginxconfig.ejs', { encoding: 'utf8' }),
     RELOAD_NGINX_CMD = path.join(__dirname, 'scripts/reloadnginx.sh');
 
+function nginxLocation(s) {
+    if (!s.startsWith('!')) return s;
+
+    let re = s.replace(/[\^$\\.*+?()[\]{}|]/g, '\\$&'); // https://github.com/es-shims/regexp.escape/blob/master/implementation.js
+
+    return `~ ^(?!(${re.slice(1)}))`; // negative regex assertion - https://stackoverflow.com/questions/16302897/nginx-location-not-equal-to-regex
+}
+
 function getAcmeApi(domainObject, callback) {
     assert.strictEqual(typeof domainObject, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -109,7 +117,7 @@ function providerMatchesSync(domainObject, certFilePath, apiOptions) {
     const domain = subject.substr(subject.indexOf('=') + 1).trim(); // subject can be /CN=, CN=, CN = and other forms
     const issuer = subjectAndIssuer.match(/^issuer=(.*)$/m)[1];
     const isWildcardCert = domain.includes('*');
-    const isLetsEncryptProd = issuer.includes('Let\'s Encrypt Authority');
+    const isLetsEncryptProd = issuer.includes('Let\'s Encrypt');
 
     const issuerMismatch = (apiOptions.prod && !isLetsEncryptProd) || (!apiOptions.prod && isLetsEncryptProd);
     // bare domain is not part of wildcard SAN
@@ -117,7 +125,9 @@ function providerMatchesSync(domainObject, certFilePath, apiOptions) {
 
     const mismatch = issuerMismatch || wildcardMismatch;
 
-    debug(`providerMatchesSync: ${certFilePath} subject=${subject} domain=${domain} issuer=${issuer} wildcard=${isWildcardCert}/${apiOptions.wildcard} prod=${isLetsEncryptProd}/${apiOptions.prod} match=${!mismatch}`);
+    debug(`providerMatchesSync: ${certFilePath} subject=${subject} domain=${domain} issuer=${issuer} `
+        + `wildcard=${isWildcardCert}/${apiOptions.wildcard} prod=${isLetsEncryptProd}/${apiOptions.prod} `
+        + `issuerMismatch=${issuerMismatch} wildcardMismatch=${wildcardMismatch} match=${!mismatch}`);
 
     return !mismatch;
 }
@@ -160,7 +170,7 @@ function validateCertificate(location, domainObject, certificate) {
 }
 
 function reload(callback) {
-    if (process.env.BOX_ENV === 'test') return callback();
+    if (constants.TEST) return callback();
 
     shell.sudo('reload', [ RELOAD_NGINX_CMD ], {}, function (error) {
         if (error) return callback(new BoxError(BoxError.NGINX_ERROR, `Error reloading nginx: ${error.message}`));
@@ -186,7 +196,8 @@ function generateFallbackCertificateSync(domainObject) {
     opensslConfWithSan = `${opensslConf}\n[SAN]\nsubjectAltName=DNS:${domain},DNS:*.${cn}\n`;
     let configFile = path.join(os.tmpdir(), 'openssl-' + crypto.randomBytes(4).readUInt32LE(0) + '.conf');
     safe.fs.writeFileSync(configFile, opensslConfWithSan, 'utf8');
-    let certCommand = util.format(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 3650 -subj /CN=*.${cn} -extensions SAN -config ${configFile} -nodes`);
+    // the days field is chosen to be less than 825 days per apple requirement (https://support.apple.com/en-us/HT210176)
+    let certCommand = util.format(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 800 -subj /CN=*.${cn} -extensions SAN -config ${configFile} -nodes`);
     if (!safe.child_process.execSync(certCommand)) return { error: new BoxError(BoxError.OPENSSL_ERROR, safe.error.message) };
     safe.fs.unlinkSync(configFile);
 
@@ -246,22 +257,22 @@ function setAppCertificateSync(location, domainObject, certificate) {
     return null;
 }
 
-function getAcmeCertificate(hostname, domainObject, callback) {
-    assert.strictEqual(typeof hostname, 'string');
+function getAcmeCertificate(vhost, domainObject, callback) {
+    assert.strictEqual(typeof vhost, 'string'); // this can contain wildcard domain (for alias domains)
     assert.strictEqual(typeof domainObject, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     let certFilePath, keyFilePath;
 
-    if (hostname !== domainObject.domain && domainObject.tlsConfig.wildcard) { // bare domain is not part of wildcard SAN
-        let certName = domains.makeWildcard(hostname).replace('*.', '_.');
+    if (vhost !== domainObject.domain && domainObject.tlsConfig.wildcard) { // bare domain is not part of wildcard SAN
+        let certName = domains.makeWildcard(vhost).replace('*.', '_.');
         certFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.cert`);
         keyFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.key`);
 
         if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
     } else {
-        certFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.cert`);
-        keyFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.key`);
+        certFilePath = path.join(paths.APP_CERTS_DIR, `${vhost}.cert`);
+        keyFilePath = path.join(paths.APP_CERTS_DIR, `${vhost}.key`);
 
         if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
     }
@@ -333,7 +344,7 @@ function ensureCertificate(vhost, domain, auditSource, callback) {
                     debug(`ensureCertificate: ${vhost} certificate already exists at ${currentBundle.keyFilePath}`);
 
                     if (!isExpiringSync(currentBundle.certFilePath, 24 * 30) && providerMatchesSync(domainObject, currentBundle.certFilePath, apiOptions)) return callback(null, currentBundle, { renewed: false });
-                    debug(`ensureCertificate: ${vhost} cert require renewal`);
+                    debug(`ensureCertificate: ${vhost} cert requires renewal`);
                 } else {
                     debug(`ensureCertificate: ${vhost} cert does not exist`);
                 }
@@ -379,7 +390,8 @@ function writeDashboardNginxConfig(bundle, configFileName, vhost, callback) {
         endpoint: 'admin',
         certFilePath: bundle.certFilePath,
         keyFilePath: bundle.keyFilePath,
-        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n')
+        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n'),
+        proxyAuth: { enabled: false, id: null, location: nginxLocation('/') }
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
     var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, configFileName);
@@ -426,8 +438,9 @@ function writeDashboardConfig(domain, callback) {
     });
 }
 
-function writeAppNginxConfig(app, bundle, callback) {
+function writeAppNginxConfig(app, fqdn, bundle, callback) {
     assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof fqdn, 'string');
     assert.strictEqual(typeof bundle, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -446,20 +459,28 @@ function writeAppNginxConfig(app, bundle, callback) {
     var data = {
         sourceDir: sourceDir,
         adminOrigin: settings.adminOrigin(),
-        vhost: app.fqdn,
+        vhost: fqdn,
         hasIPv6: sysinfo.hasIPv6(),
-        port: app.httpPort,
+        ip: app.containerIp,
+        port: app.manifest.httpPort,
         endpoint: endpoint,
         certFilePath: bundle.certFilePath,
         keyFilePath: bundle.keyFilePath,
         robotsTxtQuoted,
         cspQuoted,
-        hideHeaders
+        hideHeaders,
+        proxyAuth: {
+            enabled: app.sso && app.manifest.addons && app.manifest.addons.proxyAuth,
+            id: app.id,
+            location: nginxLocation(safe.query(app.manifest, 'addons.proxyAuth.path') || '/')
+        },
+        httpPaths: app.manifest.httpPaths || {}
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
 
-    var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, app.id + '.conf');
-    debug('writeAppNginxConfig: writing config for "%s" to %s with options %j', app.fqdn, nginxConfigFilename, data);
+    const aliasSuffix = app.fqdn === fqdn ? '' : `-alias-${fqdn.replace('*', '_')}`;
+    var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, `${app.id}${aliasSuffix}.conf`);
+    debug('writeAppNginxConfig: writing config for "%s" to %s with options %j', fqdn, nginxConfigFilename, data);
 
     if (!safe.fs.writeFileSync(nginxConfigFilename, nginxConf)) {
         debug('Error creating nginx config for "%s" : %s', app.fqdn, safe.error.message);
@@ -485,7 +506,8 @@ function writeAppRedirectNginxConfig(app, fqdn, bundle, callback) {
         keyFilePath: bundle.keyFilePath,
         robotsTxtQuoted: null,
         cspQuoted: null,
-        hideHeaders: []
+        hideHeaders: [],
+        proxyAuth: { enabled: false, id: app.id, location: nginxLocation('/') }
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
 
@@ -505,21 +527,30 @@ function writeAppConfig(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    getCertificate(app.fqdn, app.domain, function (error, bundle) {
-        if (error) return callback(error);
+    let appDomains = [];
+    appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'primary' });
 
-        writeAppNginxConfig(app, bundle, function (error) {
-            if (error) return callback(error);
-
-            async.eachSeries(app.alternateDomains, function (alternateDomain, iteratorDone) {
-                getCertificate(alternateDomain.fqdn, alternateDomain.domain, function (error, bundle) {
-                    if (error) return iteratorDone(error);
-
-                    writeAppRedirectNginxConfig(app, alternateDomain.fqdn, bundle, iteratorDone);
-                });
-            }, callback);
-        });
+    app.alternateDomains.forEach(function (alternateDomain) {
+        appDomains.push({ domain: alternateDomain.domain, fqdn: alternateDomain.fqdn, type: 'alternate' });
     });
+
+    app.aliasDomains.forEach(function (aliasDomain) {
+        appDomains.push({ domain: aliasDomain.domain, fqdn: aliasDomain.fqdn, type: 'alias' });
+    });
+
+    async.eachSeries(appDomains, function (appDomain, iteratorDone) {
+        getCertificate(appDomain.fqdn, appDomain.domain, function (error, bundle) {
+            if (error) return iteratorDone(error);
+
+            if (appDomain.type === 'primary') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alternate') {
+                writeAppRedirectNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alias') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            }
+        });
+    }, callback);
 }
 
 function configureApp(app, auditSource, callback) {
@@ -527,21 +558,30 @@ function configureApp(app, auditSource, callback) {
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    ensureCertificate(app.fqdn, app.domain, auditSource, function (error, bundle) {
-        if (error) return callback(error);
+    let appDomains = [];
+    appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'primary' });
 
-        writeAppNginxConfig(app, bundle, function (error) {
-            if (error) return callback(error);
-
-            async.eachSeries(app.alternateDomains, function (alternateDomain, iteratorDone) {
-                ensureCertificate(alternateDomain.fqdn, alternateDomain.domain, auditSource, function (error, bundle) {
-                    if (error) return iteratorDone(error);
-
-                    writeAppRedirectNginxConfig(app, alternateDomain.fqdn, bundle, iteratorDone);
-                });
-            }, callback);
-        });
+    app.alternateDomains.forEach(function (alternateDomain) {
+        appDomains.push({ domain: alternateDomain.domain, fqdn: alternateDomain.fqdn, type: 'alternate' });
     });
+
+    app.aliasDomains.forEach(function (aliasDomain) {
+        appDomains.push({ domain: aliasDomain.domain, fqdn: aliasDomain.fqdn, type: 'alias' });
+    });
+
+    async.eachSeries(appDomains, function (appDomain, iteratorDone) {
+        ensureCertificate(appDomain.fqdn, appDomain.domain, auditSource, function (error, bundle) {
+            if (error) return iteratorDone(error);
+
+            if (appDomain.type === 'primary') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alternate') {
+                writeAppRedirectNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            } else if (appDomain.type === 'alias') {
+                writeAppNginxConfig(app, appDomain.fqdn, bundle, iteratorDone);
+            }
+        });
+    }, callback);
 }
 
 function unconfigureApp(app, callback) {
@@ -565,7 +605,7 @@ function renewCerts(options, auditSource, progressCallback, callback) {
     apps.getAll(function (error, allApps) {
         if (error) return callback(error);
 
-        var appDomains = [];
+        let appDomains = [];
 
         // add webadmin and mail domain
         if (settings.mailFqdn() === settings.adminFqdn()) {
@@ -575,16 +615,20 @@ function renewCerts(options, auditSource, progressCallback, callback) {
             appDomains.push({ domain: settings.mailDomain(), fqdn: settings.mailFqdn(), type: 'mail' });
         }
 
-        // add app main
         allApps.forEach(function (app) {
             if (app.runState === apps.RSTATE_STOPPED) return; // do not renew certs of stopped apps
 
-            appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'main', app: app, nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, app.id + '.conf') });
+            appDomains.push({ domain: app.domain, fqdn: app.fqdn, type: 'primary', app: app, nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, app.id + '.conf') });
 
             app.alternateDomains.forEach(function (alternateDomain) {
                 const nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, `${app.id}-redirect-${alternateDomain.fqdn}.conf`);
                 appDomains.push({ domain: alternateDomain.domain, fqdn: alternateDomain.fqdn, type: 'alternate', app: app, nginxConfigFilename });
             });
+
+            app.aliasDomains.forEach(function (aliasDomain) {
+                const nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, `${app.id}-alias-${aliasDomain.fqdn.replace('*', '_')}.conf`);
+                appDomains.push({ domain: aliasDomain.domain, fqdn: aliasDomain.fqdn, type: 'alias', app: app, nginxConfigFilename });
+            });
         });
 
         if (options.domain) appDomains = appDomains.filter(function (appDomain) { return appDomain.domain === options.domain; });
@@ -616,10 +660,12 @@ function renewCerts(options, auditSource, progressCallback, callback) {
                         mail.handleCertChanged,
                         writeDashboardNginxConfig.bind(null, bundle, `${settings.adminFqdn()}.conf`, settings.adminFqdn())
                     ], iteratorCallback);
-                } else if (appDomain.type === 'main') {
-                    return writeAppNginxConfig(appDomain.app, bundle, iteratorCallback);
+                } else if (appDomain.type === 'primary') {
+                    return writeAppNginxConfig(appDomain.app, appDomain.fqdn, bundle, iteratorCallback);
                 } else if (appDomain.type === 'alternate') {
                     return writeAppRedirectNginxConfig(appDomain.app, appDomain.fqdn, bundle, iteratorCallback);
+                } else if (appDomain.type === 'alias') {
+                    return writeAppNginxConfig(appDomain.app, appDomain.fqdn, bundle, iteratorCallback);
                 }
 
                 iteratorCallback(new BoxError(BoxError.INTERNAL_ERROR, `Unknown domain type for ${appDomain.fqdn}. This should never happen`));
@@ -632,7 +678,13 @@ function renewCerts(options, auditSource, progressCallback, callback) {
 
             async.series([
                 (next) => { return renewed.includes(settings.mailFqdn()) ? mail.handleCertChanged(next) : next(); },// mail cert renewed
-                reload // reload nginx if any certs were updated but the config was not rewritten
+                reload, // reload nginx if any certs were updated but the config was not rewritten
+                (next) => { // restart tls apps on cert change
+                    const tlsApps = allApps.filter(app => app.manifest.addons && app.manifest.addons.tls && renewed.includes(app.fqdn));
+                    async.eachSeries(tlsApps, function (app, iteratorDone) {
+                        apps.restart(app, auditSource, () => iteratorDone());
+                    }, next);
+                }
             ], callback);
         });
     });
@@ -657,7 +709,8 @@ function writeDefaultConfig(options, callback) {
         debug('writeDefaultConfig: create new cert');
 
         const cn = 'cloudron-' + (new Date()).toISOString(); // randomize date a bit to keep firefox happy
-        if (!safe.child_process.execSync(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 3650 -subj /CN=${cn} -nodes`)) {
+        // the days field is chosen to be less than 825 days per apple requirement (https://support.apple.com/en-us/HT210176)
+        if (!safe.child_process.execSync(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 800 -subj /CN=${cn} -nodes`)) {
             debug(`writeDefaultConfig: could not generate certificate: ${safe.error.message}`);
             return callback(new BoxError(BoxError.OPENSSL_ERROR, safe.error));
         }
@@ -671,11 +724,14 @@ function writeDefaultConfig(options, callback) {
         endpoint: options.activated ? 'ip' : 'setup',
         certFilePath,
         keyFilePath,
-        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n')
+        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n'),
+        proxyAuth: { enabled: false, id: null, location: nginxLocation('/') }
     };
     const nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
     const nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, constants.NGINX_DEFAULT_CONFIG_FILE_NAME);
 
+    debug(`writeDefaultConfig: writing configs for endpoint "${data.endpoint}"`);
+
     if (!safe.fs.writeFileSync(nginxConfigFilename, nginxConf)) return callback(new BoxError(BoxError.FS_ERROR, safe.error));
 
     reload(callback);

src/routes/accesscontrol.js

@@ -1,11 +1,11 @@
 'use strict';
 
 exports = module.exports = {
-    passwordAuth: passwordAuth,
-    tokenAuth: tokenAuth,
+    passwordAuth,
+    tokenAuth,
 
-    authorize: authorize,
-    websocketAuth: websocketAuth
+    authorize,
+    websocketAuth
 };
 
 var accesscontrol = require('../accesscontrol.js'),
@@ -21,17 +21,17 @@ function passwordAuth(req, res, next) {
 
     if (!req.body.username || typeof req.body.username !== 'string') return next(new HttpError(400, 'A username must be non-empty string'));
     if (!req.body.password || typeof req.body.password !== 'string') return next(new HttpError(400, 'A password must be non-empty string'));
+    if ('totpToken' in req.body && typeof req.body.totpToken !== 'string') return next(new HttpError(400, 'totpToken must be a string' ));
 
-    const username = req.body.username;
-    const password = req.body.password;
+    const { username, password, totpToken } = req.body;
 
     function check2FA(user) {
         assert.strictEqual(typeof user, 'object');
 
         if (!user.ghost && !user.appPassword && user.twoFactorAuthenticationEnabled) {
-            if (!req.body.totpToken) return next(new HttpError(401, 'A totpToken must be provided'));
+            if (!totpToken) return next(new HttpError(401, 'A totpToken must be provided'));
 
-            let verified = speakeasy.totp.verify({ secret: user.twoFactorAuthenticationSecret, encoding: 'base32', token: req.body.totpToken, window: 2 });
+            let verified = speakeasy.totp.verify({ secret: user.twoFactorAuthenticationSecret, encoding: 'base32', token: totpToken, window: 2 });
             if (!verified) return next(new HttpError(401, 'Invalid totpToken'));
         }
 
@@ -99,6 +99,7 @@ function tokenAuth(req, res, next) {
         if (error && error.reason === BoxError.INVALID_CREDENTIALS) return next(new HttpError(401, 'Unauthorized'));
         if (error) return next(new HttpError(500, error.message));
 
+        req.access_token = token; // used in logout route
         req.user = user;
 
         next();

src/routes/apps.js

@@ -1,49 +1,51 @@
 'use strict';
 
 exports = module.exports = {
-    getApp: getApp,
-    getApps: getApps,
-    getAppIcon: getAppIcon,
-    install: install,
-    uninstall: uninstall,
-    restore: restore,
-    importApp: importApp,
-    backup: backup,
-    update: update,
-    getLogs: getLogs,
-    getLogStream: getLogStream,
-    listBackups: listBackups,
-    repair: repair,
+    getApp,
+    getApps,
+    getAppIcon,
+    install,
+    uninstall,
+    restore,
+    importApp,
+    exportApp,
+    backup,
+    update,
+    getLogs,
+    getLogStream,
+    listBackups,
+    repair,
 
-    setAccessRestriction: setAccessRestriction,
-    setLabel: setLabel,
-    setTags: setTags,
-    setIcon: setIcon,
-    setMemoryLimit: setMemoryLimit,
-    setCpuShares: setCpuShares,
-    setAutomaticBackup: setAutomaticBackup,
-    setAutomaticUpdate: setAutomaticUpdate,
-    setReverseProxyConfig: setReverseProxyConfig,
-    setCertificate: setCertificate,
-    setDebugMode: setDebugMode,
-    setEnvironment: setEnvironment,
-    setMailbox: setMailbox,
-    setLocation: setLocation,
-    setDataDir: setDataDir,
-    setBinds: setBinds,
+    setAccessRestriction,
+    setLabel,
+    setTags,
+    setIcon,
+    setMemoryLimit,
+    setCpuShares,
+    setAutomaticBackup,
+    setAutomaticUpdate,
+    setReverseProxyConfig,
+    setCertificate,
+    setDebugMode,
+    setEnvironment,
+    setMailbox,
+    setLocation,
+    setDataDir,
+    setMounts,
 
-    stop: stop,
-    start: start,
-    restart: restart,
-    exec: exec,
-    execWebSocket: execWebSocket,
+    stop,
+    start,
+    restart,
+    exec,
+    execWebSocket,
 
-    clone: clone,
+    clone,
 
-    uploadFile: uploadFile,
-    downloadFile: downloadFile,
+    uploadFile,
+    downloadFile,
 
-    load: load
+
+    load
 };
 
 var apps = require('../apps.js'),
@@ -138,12 +140,18 @@ function install(req, res, next) {
         if (data.alternateDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'alternateDomains array must contain objects with domain and subdomain strings'));
     }
 
+    if ('aliasDomains' in data) {
+        if (!Array.isArray(data.aliasDomains)) return next(new HttpError(400, 'aliasDomains must be an array'));
+        if (data.aliasDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'aliasDomains array must contain objects with domain and subdomain strings'));
+    }
+
     if ('env' in data) {
         if (!data.env || typeof data.env !== 'object') return next(new HttpError(400, 'env must be an object'));
         if (Object.keys(data.env).some(function (key) { return typeof data.env[key] !== 'string'; })) return next(new HttpError(400, 'env must contain values as strings'));
     }
 
     if ('overwriteDns' in req.body && typeof req.body.overwriteDns !== 'boolean') return next(new HttpError(400, 'overwriteDns must be boolean'));
+    if ('skipDnsSetup' in req.body && typeof req.body.skipDnsSetup !== 'boolean') return next(new HttpError(400, 'skipDnsSetup must be boolean'));
 
     apps.downloadManifest(data.appStoreId, data.manifest, function (error, appStoreId, manifest) {
         if (error) return next(BoxError.toHttpError(error));
@@ -352,7 +360,13 @@ function setLocation(req, res, next) {
         if (req.body.alternateDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'alternateDomains array must contain objects with domain and subdomain strings'));
     }
 
+    if ('aliasDomains' in req.body) {
+        if (!Array.isArray(req.body.aliasDomains)) return next(new HttpError(400, 'aliasDomains must be an array'));
+        if (req.body.aliasDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'aliasDomains array must contain objects with domain and subdomain strings'));
+    }
+
     if ('overwriteDns' in req.body && typeof req.body.overwriteDns !== 'boolean') return next(new HttpError(400, 'overwriteDns must be boolean'));
+    if ('skipDnsSetup' in req.body && typeof req.body.skipDnsSetup !== 'boolean') return next(new HttpError(400, 'skipDnsSetup must be boolean'));
 
     apps.setLocation(req.resource, req.body, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
@@ -443,6 +457,17 @@ function importApp(req, res, next) {
     });
 }
 
+function exportApp(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+    assert.strictEqual(typeof req.resource, 'object');
+
+    apps.exportApp(req.resource, {}, auditSource.fromRequest(req), function (error, result) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(202, { taskId: result.taskId }));
+    });
+}
+
 function clone(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
     assert.strictEqual(typeof req.resource, 'object');
@@ -455,6 +480,7 @@ function clone(req, res, next) {
     if (('portBindings' in data) && typeof data.portBindings !== 'object') return next(new HttpError(400, 'portBindings must be an object'));
 
     if ('overwriteDns' in req.body && typeof req.body.overwriteDns !== 'boolean') return next(new HttpError(400, 'overwriteDns must be boolean'));
+    if ('skipDnsSetup' in req.body && typeof req.body.skipDnsSetup !== 'boolean') return next(new HttpError(400, 'skipDnsSetup must be boolean'));
 
     apps.clone(req.resource, data, req.user, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
@@ -597,7 +623,7 @@ function getLogs(req, res, next) {
 
         res.writeHead(200, {
             'Content-Type': 'application/x-logs',
-            'Content-Disposition': 'attachment; filename="log.txt"',
+            'Content-Disposition': `attachment; filename="${req.resource.id}.log"`,
             'Cache-Control': 'no-cache',
             'X-Accel-Buffering': 'no' // disable nginx buffering
         });
@@ -766,22 +792,20 @@ function downloadFile(req, res, next) {
     });
 }
 
-function setBinds(req, res, next) {
+function setMounts(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
     assert.strictEqual(typeof req.resource, 'object');
 
-    if (!req.body.binds || typeof req.body.binds !== 'object') return next(new HttpError(400, 'binds should be an object'));
-
-    for (let name of Object.keys(req.body.binds)) {
-        if (!req.body.binds[name] || typeof req.body.binds[name] !== 'object') return next(new HttpError(400, 'each bind should be an object'));
-        if (typeof req.body.binds[name].hostPath !== 'string') return next(new HttpError(400, 'hostPath must be a string'));
-        if (typeof req.body.binds[name].readOnly !== 'boolean') return next(new HttpError(400, 'readOnly must be a boolean'));
+    if (!Array.isArray(req.body.mounts)) return next(new HttpError(400, 'mounts should be an array'));
+    for (let m of req.body.mounts) {
+        if (!m || typeof m !== 'object') return next(new HttpError(400, 'mounts must be an object'));
+        if (typeof m.volumeId !== 'string') return next(new HttpError(400, 'volumeId must be a string'));
+        if (typeof m.readOnly !== 'boolean') return next(new HttpError(400, 'readOnly must be a boolean'));
     }
 
-    apps.setBinds(req.resource, req.body.binds, auditSource.fromRequest(req), function (error, result) {
+    apps.setMounts(req.resource, req.body.mounts, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
-

src/routes/cloudron.js

@@ -20,7 +20,9 @@ exports = module.exports = {
     prepareDashboardDomain,
     renewCerts,
     getServerIp,
-    syncExternalLdap
+    getLanguages,
+    syncExternalLdap,
+    syncDnsRecords
 };
 
 let assert = require('assert'),
@@ -36,6 +38,7 @@ let assert = require('assert'),
     system = require('../system.js'),
     tokendb = require('../tokendb.js'),
     tokens = require('../tokens.js'),
+    translation = require('../translation.js'),
     updater = require('../updater.js'),
     users = require('../users.js'),
     updateChecker = require('../updatechecker.js');
@@ -62,24 +65,11 @@ function login(req, res, next) {
 }
 
 function logout(req, res) {
-    var token;
+    assert.strictEqual(typeof req.access_token, 'string');
 
-    // this determines the priority
-    if (req.body && req.body.access_token) token = req.body.access_token;
-    if (req.query && req.query.access_token) token = req.query.access_token;
-    if (req.headers && req.headers.authorization) {
-        var parts = req.headers.authorization.split(' ');
-        if (parts.length == 2) {
-            var scheme = parts[0];
-            var credentials = parts[1];
+    eventlog.add(eventlog.ACTION_USER_LOGOUT, auditSource.fromRequest(req), { userId: req.user.id, user: users.removePrivateFields(req.user) });
 
-            if (/^Bearer$/i.test(scheme)) token = credentials;
-        }
-    }
-
-    if (!token) return res.redirect('/login.html');
-
-    tokendb.delByAccessToken(token, function () { res.redirect('/login.html'); });
+    tokendb.delByAccessToken(req.access_token, function () { res.redirect('/login.html'); });
 }
 
 function passwordResetRequest(req, res, next) {
@@ -225,7 +215,7 @@ function getLogs(req, res, next) {
 
         res.writeHead(200, {
             'Content-Type': 'application/x-logs',
-            'Content-Disposition': 'attachment; filename="log.txt"',
+            'Content-Disposition': `attachment; filename="${req.params.unit}.log"`,
             'Cache-Control': 'no-cache',
             'X-Accel-Buffering': 'no' // disable nginx buffering
         });
@@ -291,6 +281,8 @@ function prepareDashboardDomain(req, res, next) {
 }
 
 function renewCerts(req, res, next) {
+    if ('domain' in req.body && typeof req.body.domain !== 'string') return next(new HttpError(400, 'domain must be a string'));
+
     cloudron.renewCerts({ domain: req.body.domain || null }, auditSource.fromRequest(req), function (error, taskId) {
         if (error) return next(BoxError.toHttpError(error));
 
@@ -302,7 +294,7 @@ function syncExternalLdap(req, res, next) {
     externalLdap.startSyncer(function (error, taskId) {
         if (error) return next(new HttpError(500, error.message));
 
-        next(new HttpSuccess(202, { taskId: taskId }));
+        next(new HttpSuccess(202, { taskId }));
     });
 }
 
@@ -313,3 +305,25 @@ function getServerIp(req, res, next) {
         next(new HttpSuccess(200, { ip }));
     });
 }
+
+function getLanguages(req, res, next) {
+    translation.getLanguages(function (error, languages) {
+        if (error) return next(new BoxError.toHttpError(error));
+
+        next(new HttpSuccess(200, { languages }));
+    });
+}
+
+function syncDnsRecords(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    if ('domain' in req.body && typeof req.body.domain !== 'string') return next(new HttpError(400, 'domain must be a string'));
+    if ('type' in req.body && typeof req.body.type !== 'string') return next(new HttpError(400, 'type must be a string'));
+
+    cloudron.syncDnsRecords(req.body, function (error, taskId) {
+        if (error && error.reason === BoxError.ACCESS_DENIED) return next(new HttpSuccess(200, { error: { reason: error.reason, message: error.message }}));
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(201, { taskId }));
+    });
+}

src/routes/domains.js

@@ -1,13 +1,13 @@
 'use strict';
 
 exports = module.exports = {
-    add: add,
-    get: get,
-    getAll: getAll,
-    update: update,
-    del: del,
+    add,
+    get,
+    getAll,
+    update,
+    del,
 
-    checkDnsRecords: checkDnsRecords,
+    checkDnsRecords,
 };
 
 var assert = require('assert'),
@@ -99,6 +99,13 @@ function update(req, res, next) {
         if (!req.body.tlsConfig.provider || typeof req.body.tlsConfig.provider !== 'string') return next(new HttpError(400, 'tlsConfig.provider must be a string'));
     }
 
+    if ('wellKnown' in req.body) {
+        if (typeof req.body.wellKnown !== 'object') return next(new HttpError(400, 'wellKnown must be an object'));
+        if (req.body.wellKnown) {
+            if (Object.keys(req.body.wellKnown).some(k => typeof req.body.wellKnown[k] !== 'string')) return next(new HttpError(400, 'wellKnown is a map of strings'));
+        }
+    }
+
     // some DNS providers like DigitalOcean take a really long time to verify credentials (https://github.com/expressjs/timeout/issues/26)
     req.clearTimeout();
 
@@ -107,7 +114,8 @@ function update(req, res, next) {
         provider: req.body.provider,
         config: req.body.config,
         fallbackCertificate: req.body.fallbackCertificate || null,
-        tlsConfig: req.body.tlsConfig || { provider: 'letsencrypt-prod' }
+        tlsConfig: req.body.tlsConfig || { provider: 'letsencrypt-prod' },
+        wellKnown: req.body.wellKnown || null
     };
 
     domains.update(req.params.domain, data, auditSource.fromRequest(req), function (error) {

src/routes/filemanager.js

@@ -4,30 +4,29 @@ exports = module.exports = {
     proxy
 };
 
-var assert = require('assert'),
+const assert = require('assert'),
     BoxError = require('../boxerror.js'),
-    docker = require('../docker.js'),
     middleware = require('../middleware/index.js'),
     HttpError = require('connect-lastmile').HttpError,
-    safe = require('safetydance'),
+    services = require('../services.js'),
     url = require('url');
 
 function proxy(req, res, next) {
     assert.strictEqual(typeof req.params.id, 'string');
 
-    const appId = req.params.id;
+    const id = req.params.id; // app id or volume id
 
     req.clearTimeout();
 
-    docker.inspect('sftp', function (error, result) {
-        if (error)return next(BoxError.toHttpError(error));
+    services.getContainerDetails('sftp', 'CLOUDRON_SFTP_TOKEN', function (error, result) {
+        if (error) return next(BoxError.toHttpError(error));
 
-        const ip = safe.query(result, 'NetworkSettings.Networks.cloudron.IPAddress', null);
-        if (!ip) return next(new BoxError(BoxError.INACTIVE, 'Error getting IP of sftp service'));
+        let parsedUrl = url.parse(req.url, true /* parseQueryString */);
+        parsedUrl.query['access_token'] = result.token;
 
-        req.url = req.originalUrl.replace(`/api/v1/apps/${appId}/files`, `/files/${appId}`);
+        req.url = url.format({ pathname: `/files/${id}/${encodeURIComponent(req.params[0])}`, query: parsedUrl.query }); // params[0] already contains leading '/'
 
-        const proxyOptions = url.parse(`https://${ip}:3000`);
+        const proxyOptions = url.parse(`https://${result.ip}:3000`);
         proxyOptions.rejectUnauthorized = false;
         const fileManagerProxy = middleware.proxy(proxyOptions);
 

src/routes/graphs.js

@@ -1,7 +1,7 @@
 'use strict';
 
 exports = module.exports = {
-    getGraphs: getGraphs
+    getGraphs
 };
 
 var middleware = require('../middleware/index.js'),

src/routes/groups.js

@@ -62,6 +62,7 @@ function updateMembers(req, res, next) {
 
     if (!req.body.userIds) return next(new HttpError(404, 'missing or invalid userIds fields'));
     if (!Array.isArray(req.body.userIds)) return next(new HttpError(404, 'userIds must be an array'));
+    if (req.body.userIds.some((u) => typeof u !== 'string')) return next(new HttpError(400, 'userIds array must contain strings'));
 
     groups.setMembers(req.params.groupId, req.body.userIds, function (error) {
         if (error) return next(BoxError.toHttpError(error));

src/routes/index.js

@@ -24,5 +24,7 @@ exports = module.exports = {
     support: require('./support.js'),
     tasks: require('./tasks.js'),
     tokens: require('./tokens.js'),
-    users: require('./users.js')
+    users: require('./users.js'),
+    volumes: require('./volumes.js'),
+    wellknown: require('./wellknown.js')
 };

src/routes/mail.js

@@ -3,8 +3,6 @@
 exports = module.exports = {
     getDomain,
 
-    setDnsRecords,
-
     getStatus,
 
     setMailFromValidation,
@@ -50,21 +48,6 @@ function getDomain(req, res, next) {
     });
 }
 
-function setDnsRecords(req, res, next) {
-    assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.domain, 'string');
-
-    // can take a setup all the DNS entries. this is mostly because some backends try to list DNS entries (DO)
-    // for upsert and this takes a lot of time
-    req.clearTimeout();
-
-    mail.setDnsRecords(req.params.domain, function (error) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(201));
-    });
-}
-
 function getStatus(req, res, next) {
     assert.strictEqual(typeof req.params.domain, 'string');
 
@@ -196,9 +179,10 @@ function addMailbox(req, res, next) {
     assert.strictEqual(typeof req.params.domain, 'string');
 
     if (typeof req.body.name !== 'string') return next(new HttpError(400, 'name must be a string'));
-    if (typeof req.body.userId !== 'string') return next(new HttpError(400, 'userId must be a string'));
+    if (typeof req.body.ownerId !== 'string') return next(new HttpError(400, 'ownerId must be a string'));
+    if (typeof req.body.ownerType !== 'string') return next(new HttpError(400, 'ownerType must be a string'));
 
-    mail.addMailbox(req.body.name, req.params.domain, req.body.userId, auditSource.fromRequest(req), function (error) {
+    mail.addMailbox(req.body.name, req.params.domain, req.body.ownerId, req.body.ownerType, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(201, {}));
@@ -209,9 +193,10 @@ function updateMailbox(req, res, next) {
     assert.strictEqual(typeof req.params.domain, 'string');
     assert.strictEqual(typeof req.params.name, 'string');
 
-    if (typeof req.body.userId !== 'string') return next(new HttpError(400, 'userId must be a string'));
+    if (typeof req.body.ownerId !== 'string') return next(new HttpError(400, 'ownerId must be a string'));
+    if (typeof req.body.ownerType !== 'string') return next(new HttpError(400, 'ownerType must be a string'));
 
-    mail.updateMailboxOwner(req.params.name, req.params.domain, req.body.userId, auditSource.fromRequest(req), function (error) {
+    mail.updateMailboxOwner(req.params.name, req.params.domain, req.body.ownerId, req.body.ownerType, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(204));

src/routes/mailserver.js

@@ -2,21 +2,28 @@
 
 exports = module.exports = {
     proxy,
+    restart,
 
     getLocation,
     setLocation
 };
 
-var addons = require('../addons.js'),
-    assert = require('assert'),
+const assert = require('assert'),
     auditSource = require('../auditsource.js'),
     BoxError = require('../boxerror.js'),
+    debug = require('debug')('box:routes/mailserver'),
     HttpError = require('connect-lastmile').HttpError,
     HttpSuccess = require('connect-lastmile').HttpSuccess,
     mail = require('../mail.js'),
     middleware = require('../middleware/index.js'),
+    services = require('../services.js'),
     url = require('url');
 
+function restart(req, res, next) {
+    mail.restartMail((error) => debug('Error restarting mail container', error));
+    next();
+}
+
 function proxy(req, res, next) {
     let parsedUrl = url.parse(req.url, true /* parseQueryString */);
     const pathname = req.path.split('/').pop();
@@ -26,7 +33,7 @@ function proxy(req, res, next) {
     delete req.headers['authorization'];
     delete req.headers['cookies'];
 
-    addons.getContainerDetails('mail', 'CLOUDRON_MAIL_TOKEN', function (error, addonDetails) {
+    services.getContainerDetails('mail', 'CLOUDRON_MAIL_TOKEN', function (error, addonDetails) {
         if (error) return next(BoxError.toHttpError(error));
 
         parsedUrl.query['access_token'] = addonDetails.token;
@@ -36,6 +43,7 @@ function proxy(req, res, next) {
         proxyOptions.rejectUnauthorized = false;
         const mailserverProxy = middleware.proxy(proxyOptions);
 
+        req.clearTimeout(); // TODO: add timeout to mail server proxy logic instead of this
         mailserverProxy(req, res, function (error) {
             if (!error) return next();
 

src/routes/provision.js

@@ -1,11 +1,12 @@
 'use strict';
 
 exports = module.exports = {
-    providerTokenAuth: providerTokenAuth,
-    setup: setup,
-    activate: activate,
-    restore: restore,
-    getStatus: getStatus
+    providerTokenAuth,
+    setup,
+    activate,
+    restore,
+    getStatus,
+    setupTokenAuth
 };
 
 var assert = require('assert'),
@@ -15,10 +16,24 @@ var assert = require('assert'),
     debug = require('debug')('box:routes/setup'),
     HttpError = require('connect-lastmile').HttpError,
     HttpSuccess = require('connect-lastmile').HttpSuccess,
+    paths = require('../paths.js'),
     provision = require('../provision.js'),
     request = require('request'),
+    safe = require('safetydance'),
     settings = require('../settings.js');
 
+function setupTokenAuth(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    const setupToken = safe.fs.readFileSync(paths.SETUP_TOKEN_FILE, 'utf8');
+    if (!setupToken) return next();
+
+    if (!req.body.setupToken) return next(new HttpError(400, 'setup token required'));
+    if (setupToken.trim() !== req.body.setupToken) return next(new HttpError(422, 'setup token does not match'));
+
+    return next();
+}
+
 function providerTokenAuth(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
 
@@ -96,7 +111,7 @@ function restore(req, res, next) {
 
     if (!req.body.backupConfig || typeof req.body.backupConfig !== 'object') return next(new HttpError(400, 'backupConfig is required'));
 
-    var backupConfig = req.body.backupConfig;
+    const backupConfig = req.body.backupConfig;
     if (typeof backupConfig.provider !== 'string') return next(new HttpError(400, 'provider is required'));
     if ('password' in backupConfig && typeof backupConfig.password !== 'string') return next(new HttpError(400, 'password must be a string'));
     if (typeof backupConfig.format !== 'string') return next(new HttpError(400, 'format must be a string'));
@@ -106,8 +121,13 @@ function restore(req, res, next) {
     if (typeof req.body.version !== 'string') return next(new HttpError(400, 'version must be a string'));
 
     if ('sysinfoConfig' in req.body && typeof req.body.sysinfoConfig !== 'object') return next(new HttpError(400, 'sysinfoConfig must be an object'));
+    if ('skipDnsSetup' in req.body && typeof req.body.skipDnsSetup !== 'boolean') return next(new HttpError(400, 'skipDnsSetup must be a boolean'));
 
-    provision.restore(backupConfig, req.body.backupId, req.body.version, req.body.sysinfoConfig || { provider: 'generic' }, auditSource.fromRequest(req), function (error) {
+    const options = {
+        skipDnsSetup: req.body.skipDnsSetup || false
+    };
+
+    provision.restore(backupConfig, req.body.backupId, req.body.version, req.body.sysinfoConfig || { provider: 'generic' }, options, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));