WIP: lots more documentation

doc/_index.md

@@ -15,8 +15,8 @@ documentation on the [Overleaf Wiki](https://github.com/overleaf/overleaf/wiki)
 
 ## Using the Toolkit
 
-- The Doctor
+- [The Doctor](./the-doctor.md)
-- Working with Docker-Compose Services
+- [Working with Docker-Compose Services](./docker-compose.md)
 
 
 ## Configuration
@@ -25,7 +25,17 @@ documentation on the [Overleaf Wiki](https://github.com/overleaf/overleaf/wiki)
 - [overleaf.rc](./overleaf-rc.md)
 
 
+## Persistent Data
+
+- [Persistent Data Overview](./persistent-data.md)
+
+
 ## Server Pro
 
 - [Getting Server Pro](./getting-server-pro.md)
 - [Sandboxed Compiles](./sandboxed-compiles.md)
+
+
+## Upgrades
+
+- [Upgrading the Toolkit](./upgrading.md)

doc/configuration.md

@@ -35,9 +35,14 @@ See [The full specification](./overleaf-rc.md) for more details on the supported
 
 ## The `variables.env` File
 
-
+The `config/variables.env` file contains environment variables that are loaded into the overleaf docker container, and used to configure the overleaf microservices. These include the name of the application, as displayed in the header of the web interface, settings for sending emails, and settings for using LDAP with Server Pro.
 
 
 ## The `version` File
 
 The `config/version` file contains the version number of the docker images that will be used to create the running instance of Overleaf.
+
+
+## The `docker-compose.override.yml` File
+
+If present, the `config/docker-compose.override.yml` file will be included in the invocation to `docker-compose`. This is useful for overriding configuration specific to docker-compose.

doc/dependencies.md

@@ -1,6 +1,5 @@
 # Dependencies
 
-
 This project requires a modern unix system as a base (such as Ubuntu Linux).
 It also requires `bash`, `docker`, and `docker-compose`. 
 
@@ -10,3 +9,8 @@ Note: MacOS does not ship with a `realpath` program. In this case we fall
 back to a custom shell function to imitate some of what `realpath` does, but
 it is relatively limited. We recommend users on MacOS install the gnu coreutils
 with `brew install coreutils`, to get a working version of `realpath`.
+
+
+## Host Environment
+
+The Overleaf toolkit is tested on Ubuntu Linux, but we expect that most modern Linux systems will be able to run the toolkit without problems. Although it is possible to run the toolkit on other platforms, they are not officially supported.

doc/docker-compose.md

@@ -0,0 +1,70 @@
+# Working with Docker-Compose Services
+
+The Overleaf toolkit runs Overleaf inside a docker container, plus the
+supporting databases (MongoDB and Redis), in their own containers. All of this
+is orchestrated with `docker-compose`.
+
+Note: for legacy reasons, the main Overleaf container is called `sharelatex`,
+and is based on the `sharelatex/sharelatex` docker image. This is because the
+technology is based on the ShareLaTeX code base, which was merged into Overleaf.
+See [this blog
+post](https://www.overleaf.com/blog/518-exciting-news-sharelatex-is-joining-overleaf)
+for more details. At some point in the future, this will be renamed to match the
+Overleaf naming scheme.
+
+
+## The `bin/docker-compose` Wrapper
+
+The `bin/docker-compose` script is a wrapper around `docker-compose`. It
+loads configuration from the `config/` directory, before invoking
+`docker-compose` with whatever arguments were passed to the script.
+
+You can treat `bin/docker-compose` as a transparent wrapper for the
+`docker-compose` program installed on your machine.
+
+For example, we can check which containers are running with the following:
+
+```
+$ bin/docker-compose ps
+```
+
+
+## Convenience Helpers
+
+In addition to `bin/docker-compose`, the toolkit also provides a collection of
+convenient scripts to automate common tasks:
+
+- `bin/up`: shortcut for `bin/docker-compose up`
+- `bin/start`: shortcut for `bin/docker-compose start`
+- `bin/stop`: shortcut for `bin/docker-compose stop`
+- `bin/shell`: starts a shell inside the main container
+
+
+## Architecture
+
+Inside the overleaf container, the Overleaf software runs as a set of micro-services, managed by `runit`. Some of the more interesting files inside the container are:
+
+- `/etc/service/`: initialisation files for the microservices
+- `/etc/sharelatex/settings.coffee`: unified settings file for the microservices
+- `/var/log/sharelatex/`: logs for each microservice
+- `/var/www/sharelatex/`: code for the various microservices
+- `/var/lib/sharelatex/`: the mount-point for persistent data (corresponds to the directory indicated by `SHARELATEX_DATA_PATH` on the host)
+
+
+## The MongoDB and Redis Containers
+
+Overleaf dedends on two external databases: MongoDB and Redis. By default, the toolkit will provision a container for each of these databases, in addition to the Overleaf container, for a total of three docker containers.
+
+If you would prefer to connect to an existing MongoDB or Redis instance, you can do so by setting the appropriate settings in  the [overleaf.rc](./overleaf-rc.md) configuration file.
+
+
+## Viewing Individual Micro-Service Logs
+
+The `bin/logs` script allows you to select individual log streams from inside the overleaf container.
+For example, if you want to see just the logs for the `web` and `clsi` (compiler) micro-services, run:
+
+```
+$ bin/logs -f web clsi
+```
+
+See the output of `bin/logs --help` for more options.

doc/persistent-data.md

@@ -0,0 +1,26 @@
+# Persistent Data Overview
+
+The Overleaf toolkit needs to store persistent data, such as the files required to compile LaTeX projects, and the contents of the MongoDB database. This is achieved by mounting a few directories from the host machine into the docker containers, and writing the data to those directories.
+
+
+## Data Directories
+
+The Overleaf contaner requires a directory in which to store data relating to LaTeX compiles. This directory is set with the `SHARELATEX_DATA_PATH` variable in `config/overleaf.rc`. 
+
+The MongoDB container, if it is enabled, requires a directory in which to store it's database files, and the same is true of the Redis container. These directories can also be configured in `config/overleaf.rc`.
+
+
+## File Permissions
+
+Because docker runs as `root`, the data directories will end up being owned by the `root` user, even if the toolkit is being used by a non-root user. This is not a problem, but is worth being aware of, if you intend to alter the persistent data from outside of the containers.
+
+
+## Backups
+
+The recommended backup procedure is as follows:
+
+- Shut down the docker services with `bin/stop`
+- For each of the data directories, make a copy of that directory
+  to another safe location
+  - (We recommend using `rsync` for this)
+- Start the services again with `bin/start`

doc/quick-start-guide.md

@@ -141,12 +141,7 @@ We should see some output similar to this:
 ====== Overleaf Doctor ======
 - Host Information
     - Linux
-    - Output of 'lsb_release -a':
+    ...
-            No LSB modules are available.
-            Distributor ID:	Ubuntu
-            Description:	Ubuntu 20.04 LTS
-            Release:	20.04
-            Codename:	focal
 - Dependencies
     - bash
         - status: present
@@ -157,30 +152,9 @@ We should see some output similar to this:
     - docker-compose
         - status: present
         - version info: docker-compose version 1.24.0, build 0aa59064
-    - realpath
+    ...
-        - status: present
-        - version info: realpath (GNU coreutils) 8.30
-    - perl
-        - status: present
-        - version info: 5.030000
-    - awk
-        - status: present
-        - version info: GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)
-- Docker Daemon
-    - status: up
 ====== Configuration ======
-- config/version
+    ...
-    - status: present
-    - version: 2.3.1
-- config/overleaf.rc
-    - status: present
-    - values
-        - SHARELATEX_DATA_PATH: data/sharelatex
-        - SERVER_PRO: false
-        - MONGO_ENABLED: true
-        - REDIS_ENABLED: true
-- config/variables.env
-    - status: present
 ====== Warnings ======
 - None, all good
 ====== End ======

doc/sandboxed-compiles.md

@@ -1,2 +1,20 @@
 # Sandboxed Compiles
 
+In Server Pro, it is possible to have each LaTeX project be compiled in a separate docker container, achieving sandbox isolation between projects. 
+
+
+## How It Works
+
+When sandboxed compiles are enabled, the toolkit will mount the docker socket from the host into the overleaf container, so that the compiler service in the container can create new docker containers on the host. Then for each run of the compiler in each project, the LaTeX compiler service (CLSI) will do the following:
+
+- Write out the project files to a location inside the `SHARELATEX_DATA_PATH`, 
+- Use the mounted docker socket to create a new `texlive` container for the compile run
+- Have the `texlive` container read the project data from the location under `SHARELATEX_DATA_PATH`
+- Compile the project inside the `texlive` container
+
+
+## Enabling Sibling Containers
+
+In `config/overleaf.rc`, set `SIBLING_CONTAINERS_ENABLED=true`, and ensure that the `DOCKER_SOCKET_PATH` setting is set to the location of the docker socket on the host.
+
+The next time you start the docker services (with `bin/up`), the overleaf container will verify that it can communicate with docker on the host machine, and will pull the `texlive` image it requires to create the sandboxed compile containers. This process can take several minutes, and compiles will be un-available during this time.

doc/the-doctor.md

@@ -0,0 +1,139 @@
+# The Doctor
+
+The Overleaf toolkit comes with a handy `doctor` script, to help with debugging. Just run `bin/doctor` and the script will print out information about your host environment, your configuration, and the dependencies the toolkit needs. This output can also help the Overleaf support team to help you figure out what has gone wrong, in the case of a Server Pro installation.
+
+
+## Getting Help
+
+Users of the free Community Edition should [open an issue on github](https://github.com/overleaf/toolkit/issues). 
+
+Users of Server Pro should contact `support@overleaf.com` for assistance.
+
+In both cases, it is a good idea to include the output of the `bin/doctor` script in your message, so the Overleaf team can help debug the problem.
+
+
+## Consulting with the Doctor
+
+Run the doctor script:
+
+```sh
+$ bin/doctor
+```
+
+You will see some output like this:
+
+```
+====== Overleaf Doctor ======
+- Host Information
+    - Linux
+    - Output of 'lsb_release -a':
+            No LSB modules are available.
+            Distributor ID:	Ubuntu
+            Description:	Ubuntu 20.04 LTS
+            Release:	20.04
+            Codename:	focal
+- Dependencies
+    - bash
+        - status: present
+        - version info: 5.0.17(1)-release
+    - docker
+        - status: present
+        - version info: Docker version 19.03.6, build 369ce74a3c
+    - docker-compose
+        - status: present
+        - version info: docker-compose version 1.24.0, build 0aa59064
+    - realpath
+        - status: present
+        - version info: realpath (GNU coreutils) 8.30
+    - perl
+        - status: present
+        - version info: 5.030000
+    - awk
+        - status: present
+        - version info: GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)
+- Docker Daemon
+    - status: up
+====== Configuration ======
+- config/version
+    - status: present
+    - version: 2.3.1
+- config/overleaf.rc
+    - status: present
+    - values
+        - SHARELATEX_DATA_PATH: data/sharelatex
+        - SERVER_PRO: false
+        - MONGO_ENABLED: true
+        - REDIS_ENABLED: true
+- config/variables.env
+    - status: present
+====== Warnings ======
+- None, all good
+====== End ======
+```
+
+
+### Host Information
+
+The `Host Information` section contains information about the machine on which the toolkit is running. This includes information about the type of Linux system being used. 
+
+
+### Dependencies
+
+The `Dependencies` section shows a list of tools which are required for the toolkit to work.
+If the tool is present on the system, it will be listed as `status: present`, along with the version of the tool. For example:
+
+```
+- docker
+    - status: present
+    - version info: Docker version 19.03.6, build 369ce74a3c
+```
+
+However, if the tool is missing, it will be listed as `status: MISSING!`, and a warning will be added to the bottom of the `doctor` output. For example:
+
+```
+- docker
+    - status: MISSING!
+```
+
+If any of the dependencies are missing, the toolkit will almost certainly not work.
+
+
+### Configuration
+
+The `Configuration` section contains information about the files in the `config/` directory. In the case of `config/overleaf.rc`, the doctor also prints out some of the more important values from the file. If any of the files are not present, they will be listed as `status: MISSING!`, and a warning will be added to the bottom of the `doctor` output. For example:
+
+```
+====== Configuration ======
+- config/version
+    - status: present
+    - version: 2.3.1
+- config/overleaf.rc
+    - status: present
+    - values
+        - SHARELATEX_DATA_PATH: /tmp/sharelatex 
+        - SERVER_PRO: false
+        - MONGO_ENABLED: false
+        - REDIS_ENABLED: true
+- config/variables.env
+    - status: MISSING!
+```
+
+The above example shows a few problems:
+
+- The `SHARELATEX_DATA_PATH` variable is set to `/tmp/sharelatex`, which is probably not a safe place to put important data
+- The `MONGO_ENABLED` variable is set to `false`, so the toolkit will not provision it's own MongoDB database. In this case, we had better be sure to set `MONGO_URL` to point to a MongoDB database managed outside of the toolkit
+- the `config/variables.env` file is missing
+
+
+### Warnings
+
+
+The `Warnings` section shows a summary of problems discovered by the doctor script. Or, if there are no problems, this section will say so. For example:
+
+```
+====== Warnings ======
+- configuration file variables.env not found
+- rc file, SHARELATEX_DATA_PATH not set
+====== End =======
+```
+