.. _`ris-docker-workshop`: =================== RIS Docker Workshop =================== .. contents:: :depth: 1 :local: .. Note:: Connecting to get command line access: ``ssh wustlkey@compute1-client-1.ris.wustl.edu`` Queue to use: ``workshop, workshop-interactive`` Group to use: ``compute-workshop`` (if part of multiple groups) .. _`ris-docker-workshop-video`: Docker Workshop Video --------------------- Coming Soon! .. _`ris-docker-workshop-provides`: What will this documentation provide? ------------------------------------- - An introduction to using Docker. - An example of utilizing Docker to create images. - Leveraging Docker images to do analysis on the RIS Compute Platform. .. _`ris-docker-workshop-needed`: What is needed? --------------- - Access to the RIS Compute Platform. - Knowledge of how to submit jobs on the RIS Compute Platform. - Docker installed on your local computer or ability to utilize Docker development on compute, :ref:`documentation found here. ` - An account on Docker Hub. .. _`ris-docker-workshop-what-is-docker`: What is Docker? --------------- - The technical definition of Docker is an open-source project that automates the deployment of software applications inside containers by providing an additional layer of abstraction and automation of OS-level virtualization on Linux. - The simpler definition is that Docker is a tool that allows users to deploy applications within a sandbox (containers) to run on the host operating system (RIS Compute Platform). - This method allows for all dependencies and environments to be able to remain unique to the software without interacting or interfering with other software's environments. .. _`ris-docker-workshop-what-is-container`: What is a container? -------------------- - A container is a virtual machine (VM), that runs software applications. - This means that a container is virtual computer that has an operating system (OS) and whatever software users installed. - Docker has created a syntax language for creating these virtual computers which get referred to then as Docker containers. - You can find more information about why RIS chose Docker and containers :ref:`here. ` .. _`ris-docker-workshop-docker-info`: Where can I find Docker? ------------------------ - You can find official Docker documentation `here. `__ - You can find Docker containers and a place to host Docker containers `here. `__ - You can download Docker `here. `__ .. _`ris-docker-workshop-create`: Creating a Docker Container --------------------------- .. _`ris-docker-workshop-base-container`: 1. Decide The Base Container ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - The first thing you'll want to when creating a docker container, is decide what type of container you want to start with as the base. - You can start with a base container of just an operating system, like Ubuntu, or you can start with a container that already has software installed. - Basic OS Docker Containers (This list is not comprehensive.) - `Ubuntu `__ - `debian `__ - `CentOS `__ - `Alpine `__ - `Windows `__ - Base Software Docker Containers (This list is not comprehensive.) - `R `__ - `Python `__ - `Miniconda `__ - `Jupyter `__ - For our example we are going to start with the ``bionic`` Ubuntu image. - To do that we need to open up a text editor and create the base of our container. .. code:: #Start from bionic base Ubuntu image. FROM ubuntu:bionic .. _`ris-docker-workshop-install-software`: 2. Install Software and Software Dependencies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - The next step is to determine what software we're going to install. - For this example we'll be installing a software called cowsay. - To do this, we'll have to use apt-get to install the software. - First we'll want to do an update using the following command. .. code:: RUN apt-get update - Then we need to actually install cowsay. In the install, since we're installing in a Docker image, we'll want to use some options to make it cleaner. - The command should look like the following. .. code:: RUN apt-get install -y --no-install-recommends cowsay - Once all of the software we want to install has been installed, we will want to run a clean to help keep our image clean and smaller. .. code:: RUN apt-get clean - We can run all the apt-get commands with the same RUN command if we wish, by utilizing ``&&``. Now our Dockerfile should look like the following. .. code:: RUN apt-get update \ && apt-get install -y --no-install-recommends cowsay \ && apt-get clean - We next will need to add the directory where cowsay is installed to the PATH variable so that we can use the software. .. code:: ENV PATH="$PATH:/usr/games" RUN export PATH - Now our Dockerfile should look like the following. .. code:: #Start from bionic base Ubuntu image. FROM ubuntu:bionic #Install cowsay RUN apt-get update \ && apt-get install -y --no-install-recommends cowsay \ && apt-get clean #Add cowsay to the PATH variable ENV PATH="$PATH:/usr/games" RUN export PATH .. _`ris-docker-workshop-build-test-upload`: 3. Build, Test, and Upload An Image ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Once you have your Dockerfile saved within a directory (folder) designed for the image, the next step is to build the container. - The Docker base command to build a Docker container from a Dockerfile, looks like the following. .. code:: docker build -t username/container-name:tag directory - In our case, we'll be using a directory named docker-example and we'll simply call the container docker-example. - ``username`` refers to your Docker Hub username. - So, our Docker build command should look like the following. .. code:: docker build -t username/docker-example:latest docker-example/ - If it builds successfully, you should get output of information about the building process, but at the end you'll see the following. .. image:: images/docker-workshop/build.output.png - Now we can run the Docker image we've created. - The base Docker ``run`` command is as follows. .. code:: docker run username/container-name:tag command - For our example image, this will look like the following. .. code:: docker run username/docker-example:latest cowsay "Hello World!" - Your output should look like the following. .. image:: images/docker-workshop/cowsay.base.png - Once we are certain our Docker image is functioning correctly, we can then push it to Docker Hub. - The basic ``push`` command looks as follows. .. code:: docker push username/container-name:tag - You should see output like the following for the push. .. image:: images/docker-workshop/docker.push.example.png .. _`ris-docker-workshop-expand-image`: 4. Expanding An Image ~~~~~~~~~~~~~~~~~~~~~ - While it's fun to tell our cow what to say, what if we had it say randomly generated fortunes? - We can do this by also installing the fortune library into our docker container. - Luckily, once our base image has been designed this requires changing only 1 line in our Dockerfile. - We will add ``fortune`` and ``fortunes`` to our ``apt-get install`` command, like the following. .. code:: && apt-get install -y --no-install-recommends cowsay fortune fortunes \ - Once that is changed, we can save the Dockerfile and rebuild the image. - Now we can pipe ``fortune`` into ``cowsay`` and create our fortune telling cow. - Unfortunately when running Docker, to be able to use the pipe command we need to add ``/bin/bash -c`` to our command. - So our new Docker run command should look like the following. .. code:: docker run username/docker-example:latest /bin/bash -c "fortune | cowsay" - If everything is working correctly, we should get output like the following. .. image:: images/docker-workshop/cowsay.fortune.png - You can re-upload your image to Docker Hub so that you have the newest image available for the next part. - Your complete Dockerfile should now look like the following. .. code:: #Start from bionic base Ubuntu image. FROM ubuntu:bionic #Install cowsay RUN apt-get update \ && apt-get install -y --no-install-recommends cowsay fortune fortunes \ && apt-get clean #Add cowsay to the PATH variable ENV PATH="$PATH:/usr/games" RUN export PATH Using a Docker Container on the Compute Platform ------------------------------------------------ - Now that we have our docker container created and uploaded to Docker Hub, we can use it to run the software we installed on the RIS Compute Platform. .. Note:: - If you are not knowledgeable on how to use the RIS Compute Platform, you can go over the following documentation. - :ref:`Compute Quick Start ` - :ref:`RIS Compute 101 ` - :ref:`RIS Compute 101 ` - :ref:`RIS Compute 101 ` - :ref:`RIS Compute 101 ` - To get the output we want on the RIS Compute Platform, we will have to use the following commands. .. code:: export LSF_DOCKER_PRESERVE_ENVIRONMENT=false bsub -Is -q workshop-interactive -G compute-workshop -a 'docker(username/docker-example:latest)' /bin/bash -c "fortune | cowsay" - We need to use the ``LSF_DOCKER_PRESERVE_ENVIRONMENT`` variable because we had to set environment variables within our container and we want to use those instead of preserving those on the Compute Platform. - Again, in this case the username is your Docker Hub username. - If everything is working correctly, you should see results like the following. .. image:: images/docker-workshop/cowsay.on.compute.png - ``cowsay`` has a fun little additional aspect. If a fortune telling cow isn't fantastic enough, we can always use a dragon. - If you add the ``-f dragon`` option to the cowsay command, you can turn your cow into a dragon. .. code:: export LSF_DOCKER_PRESERVE_ENVIRONMENT=false bsub -Is -q workshop-interactive -G compute-workshop -a 'docker(username/docker-example:latest)' /bin/bash -c "fortune | cowsay -f dragon" .. image:: images/docker-workshop/cowsay.compute.dragon.png Advanced Container Addition --------------------------- - As an advanced step, we can add another library that will add color to our cows and dragons. - ``lolcat`` adds rainbow coloring to the display. - Once we add it to our container, update, and upload to Docker Hub, we can add the color via ``lolcat`` with the following command. - There are no hints for this step as it is advanced and designed for the user to figure out. .. code:: export LSF_DOCKER_PRESERVE_ENVIRONMENT=false bsub -Is -q workshop-interactive -G compute-workshop -a 'docker(username/docker-example:latest)' /bin/bash -c "fortune | cowsay | lolcat" .. image:: images/docker-workshop/cowsay.compute.color.png Example of Using the Compute Platform for Development ----------------------------------------------------- - As mentioned before you can develop a Docker image on the Compute Platform. - The first thing you'll need to do is let Docker Hub know your credentials by using the following command. .. code:: LSB_DOCKER_LOGIN_ONLY=1 bsub -G compute-workshop -q workshop-interactive -Is -a 'docker_build' -- . - This command only needs to be done once and will ask for your Docker Hub credentials (you'll need an Docker Hub account prior to this step). - The login credentials you'll need to use are those for Docker Hub NOT your WashU credentials. .. image:: images/docker-workshop/dockerhub.login.example.png - Once you have this set, you can start development of your Docker image. - If you wish to use the Compute Platform to develop images or other software, we suggest checking out our documentation on doing so (Coming Soon!). - For this simple example, we'll use the ``vi`` text editor that's available on the Compute Platform. - You need to create a directory called docker-example and then ``cd`` into that directory. - Once in the docker-example directory, you will need to create your Dockerfile via the following command. .. code:: vi Dockerfile - Once in the editor you will need to press the ``I`` key to be able to edit the file. Now you just need to put in the information in the Dockerfile like developed above. - Once you've entered the info, then you press the ``escape`` key to disengage edit mode. - Then you need to press the ``:`` key and type ``wq`` and hit the ``enter`` key. This will write your changes and quit the editor. - Now you can use the ``cat`` command on your file to make sure it contains the correct information. .. image:: images/docker-workshop/creating.dockerfile.on.compute.png - In order to build and push our Docker image, we need to ``cd`` back out of docker-example directory. .. image:: images/docker-workshop/cd.out.of.directory.png - Then we need to use the following command to build and push our Docker image. .. code:: bsub -G compute-workshop -q workshop-interactive -Is -a 'docker_build(username/docker-example)' -- latest docker-example - Where ``username`` is your Docker Hub username. - Once this is entered you will see the normal Docker building information. .. image:: images/docker-workshop/docker.build.compute1.png .. image:: images/docker-workshop/docker.build.compute2.png - Now that the container is built and pushed, we can use the container like we need above in order to have our fortune telling cow. .. code:: export LSF_DOCKER_PRESERVE_ENVIRONMENT=false bsub -Is -q workshop-interactive -G compute-workshop -a 'docker(username/docker-example:latest)' /bin/bash -c "fortune | cowsay" .. image:: images/docker-workshop/build.on.compute.run.png