Orderly supports a centralised workflow, where a central server holds the canonical version of a report. In order to make this workflow work, we need a number of things:
- To be able to run a report on a central server
- To be able to get copies of reports from the central server in order to use them as dependencies of locally run reports
- To be able to share these centrally run reports with someone
To make this workflow work, we have developed some additional software.
First, we have a docker-based server system called OrderlyWeb which can be deployed onto a server. We will communicate with this server over https using a package
orderlyweb which will follow
orderly onto CRAN.
The OrderlyWeb server software provides a number of things
- A persistent copy of R running
orderlythat can run reports as needed using the
- A web server that exposes a user-friendly web portal showing versions of orderly reports and an http api (this application is written in Kotlin)
- A reverse proxy used to secure the application with https
Getting OrderlyWeb installed requires a little work, and you will need access to an appropriate server running docker, our python-based deployment tool, and TLS certificates for the server that you are using. We recommend using vault for storing certificates and any secrets used in deployment.
OrderlyWeb will use GitHub for authentication, the setup for which is described here.
Once OrderlyWeb is running, we can tell
orderly about it by adding an appropriate setting to the
orderly_config.yml file. For example, if you had one testing and one production instance
$GITHUB_TOKEN is an environment variable that holds a GitHub token with “user” scope. The
testing remote is listed first, and so will be the default one used in any remote-using command.
Several commands can interact with the remote server:
orderly::orderly_run_remotewill run a report on a remote server, committing it immediately. This will save the log alongside the report.
orderly::orderly_pull_archivewill pull a version of a report from a remote server
orderly::orderly_pull_dependencieswill pull all the dependencies of a report from the remote server (i.e., versions of reports declared in the
depends:section of a report)
We hope that this system will become easier to deploy in future. We use it internally on two projects with orderly archives that hold gigabytes of analyses, and it has evolved to meet our needs. If you think it might work for you but can’t get started, please get in touch.
We recommend using HashiCorp’s vault for storing secrets (of which several are used here), and our tooling is designed to make this easy.
You will need to set up and/or gather a few things beforehand:
- SSL certificate and private key for the reverse proxy; in the example below we have stored the certificate in the vault
- If the orderly source repository is private, create a deploy key with read-only access to your repository, and add that to the vault also
- Create an OAuth app, following instructions here, somewhat better instructions here
- Create a docker image that has all of your wanted R packages and external software; this image should be based off of
vimc/orderly.server:master, for example the Dockerfile for the coronavirus response and push that to some accessible location (either docker hub or a private registry). You can initially skip this step and just use
vimc/orderly.server:masterbut the set of built-in packages is very limited
vault: addr: https://vault.dide.ic.ac.uk:8200 auth: method: github container_prefix: orderly_web network: orderlyweb_nw volumes: orderly: orderly_volume proxy_logs: orderlyweb_proxy_logs css: orderlywebweb_css orderly: image: repo: yourorg name: orderly tag: latest initial: source: clone url: [email protected]:yourorg/orderly-reports ssh: public: VAULT:secret/deploy-key:public private: VAULT:secret/deploy-key:private web: image: repo: vimc name: orderly-web tag: master migrate: orderlyweb-migrate admin: orderly-web-user-cli css-generator: orderly-web-css-generator url: https://orderly.example.com dev_mode: false port: 8888 name: Outputs email: [email protected] auth: github_org: yourorg github_team: yourteam github_oauth: id: VAULT:secret/oauth/real:id secret: VAULT:secret/oauth/real:secret fine_grained: false montagu: false proxy: enabled: true ssl: certificate: VAULT:secret/proxy/ssl_certificate:value key: VAULT:secret/ncov/ssl_private_key:value hostname: ncov.dide.ic.ac.uk port_http: 80 port_https: 443 image: repo: vimc name: orderly-web-proxy tag: master
You need to install the
orderly-web deployment scripts from PyPi
pip3 install --user orderly-web
With all that done, running
orderly-web start .
should bring everything up!