.. _`ris-compute-102`: =============== RIS Compute 102 =============== .. contents:: :depth: 1 :local: .. Note:: Connecting to get command line access: ``ssh wustlkey@compute1-client-1.ris.wustl.edu`` Queue to use: ``workshop, workshop-interactive`` Group to use: ``compute-workshop`` (if part of multiple groups) .. _`ris-compute-102-video`: Compute 102 Video ----------------- .. raw:: confluence .. raw:: confluence ]]> .. _`ris-compute-102-provided`: What will this documentation provide? ------------------------------------------ - A basic knowledge of some of the options for text editors. - A basic knowledge of shell writing and executing shell scripts. .. _`ris-compute-102-editors-scripts`: Text Editors and Shell Scripts ------------------------------ Shell Scripts and Batch Scripts - In a normal HPC environment, batch scripts tell the system how to process your data. - We will cover shell scripts first and then batch scripts as batch scripts are shell scripts with additional functionality. - This type of setup is typical for the average HPC. - In the RIS HPC environment, you do not need batch scripts, but you can essentially duplicate them for the sake of familiarity. Shell Scripts - These are a text file with a series of Unix/Linux commands. - They are useful for automating a series of commands. - Each shell has its own syntax but they all support programming structures like loops, conditionals, etc. - They are powerful enough that it becomes possible to create what are essentially new commands. - Example of a simple shell script: .. image:: images/compute102/shell.script.example.png Text Editors - Text editors are used to write and edit shell/batch scripts (as well as other files). - There are many types of text editors - some are CLI and some are GUI. - A few text editors vi, vim, emacs, gedit, etc. - Text editors that are CLI based are tricky to learn, so a GUI text editor can be useful for the beginning user. - If you edit any file on Windows and transfer it to the HPC environment, you will have formatting errors. - Windows terminates each line with both a line feed and a carriage return while Unix/Linux just uses a line feed. - X11 is part of the Unix/Linux system and allows for GUIs. - This not only allows for you to run GUIs directly on a Unix system, you can also run GUIs on remote systems and have them display on your local system. - Linux has X11 built into the OS. - Unix has X11 included in OSX 10.5 - 10.7, otherwise you will need to install XQuartz (https://www.xquartz.org/). - Windows, you can use MobaXterm to utilize X11. - SSH can "tunnel" X11 so that it travels through a secure connection. - You just have to add the -Y flag to your connection command when using Linux/Unix. .. code:: ssh -Y username@machine-id ssh -Y wustlkey@compute1-client-1.ris.wustl.edu - MobaXterm has an SSH client that will automatically tunnel X11 so we don't need the -Y flag when using that particular software to connect. - Emacs is a good basic editor. It looks like notepad and has many basic features. .. image:: images/compute102/emacs.gui.png Shell Scripts Workflow - Write the script in a text editor - Make the script executable - Run the script Our First Shell Script - Lines that begin with '#' are usually interpreted as comments. - An important exception to this is the first line of a shell script. .. code:: #!/bin/bash - ``'#!'`` tells the OS what interpreter to run the shell script with, there are multiple. - Bourne: ``#!/bin/sh`` - BASH: ``#!/bin/bash`` - Python: ``#!/usr/bin/python`` - Perl: ``#!/usr/bin/perl -w`` - To execute a Unix/Linux command from a shell script, you just enter the command on a new line in your script. - ``echo`` is used to print text to the screen. .. code:: echo "Hello World!" - Example of writing a simple shell script and executing said script: .. image:: images/compute102/emacs.create.shell.script.gif .. image:: images/compute102/run.shell.script.gif - You can use the chmod command to make your script executable. .. code:: chmod +x my-script-name.sh - In the command line, cat and less are commands that can be used to view the contents of a file. cat will just print the entire contents of the file to the screen while less will allow you to interact with the file and scroll through the contents. .. code:: cat my-script-name.sh less my-script-name.sh .. image:: images/compute102/cat.example.gif .. image:: images/compute102/less.example.gif - When you are viewing the file with less, you can use the spacebar to scroll through the content and 'q' to quit out of the view. File and Directory Permissions - As mentioned in the RIS Compute 101, the command ``ls -lh`` returns the details of all files and directories in the current directory (working directory). .. image:: images/compute102/lslh.permissions.png - The first section of information contains whether or not the item listed is a file or a directory (- = file, d = directory). - It also contains the permissions for 3 groups, the first is the owner of the file, the second is the group the owner of the file is in, and the third is for everyone. - The permission codes are as follows: r (read) / w (write) / x (executable) - The following would represent that only the owner can read, write, and execute the file. .. code:: -rwx------ - The following represents the owner has permissions as above, but the group also has read and write permissions. .. code:: -rwxrw---- - The following represents as above but now everyone can read as well. .. code:: -rwxrw-r-- - We've shown the chmod command before which allowed us to make our shell script executable, the same command can be used to change any of the permissions of a file or directory. - There is a number system associated with the permissions. - Read is 4, Write is 2, and Executable is 1, therefore 4 would mean only read, 6 would mean read and write and 7 would mean all three. - The following would change the permissions so that only the owner can read and write. .. code:: chmod 600 file-name.txt - The following would make it so that the owner and the group can read and write. .. code:: chmod 660 file-name.txt - The following would make it so that everyone can read the file and only the owner and group can write. .. code:: chmod 664 file-name.txt - To run a script you need to enter the path to the script. You can use either the absolute path, or the home alias. .. code:: /home/username/my-script-name.sh ~/my-script-name.sh - And if your script is in the directory you're currently in (working directory) you can use "./" as the "." indicates the current directory (working directory). This is what we used in our example. .. code:: ./my-script-name.txt Shell Scripts - Additional Concepts - In this section, we'll briefly discuss variables, conditionals, and loops. - In shell scripts, traditionally variables are defined using all capital letters. - Putting a dollar sign in front of a variable indicates to give the value of the variable. .. code:: MYVARIABLE = "This is my variable!" $MYVARIABLE .. image:: images/compute102/variable.example.gif - You can use variables within scripts. With this you can do things like only type a long path to a file once and then reference the variable later in the script instead of having to type the long path again. .. code:: MYLONGPATH="/long/path/name/to/file.txt" echo $MYLONGPATH command $MYLONGPATH - Conditionals are where different actions are taken dependent on whether a certain condition is met or not. - An example of this are if/then statements. .. code:: #!/bin/bash T1=5 T2=7 if [[ $T1 -ge $T2 ]]; then echo "expression evaluated as true" else echo "expression evaluated as false" fi .. image:: images/compute102/conditional.example.gif .. Note:: Bash is picky about white spaces so if you don't have the correct white spaces, your script may not behave properly. - Loops execute a command over and over again a specified amount of times. - If you have a defined set of values, you would use a ``for`` loop. - If you need to loop until a specified condition is no longer valid, you would use a ``while`` loop. - The following while loop runs while the counter is less than 10 and simply echos the value of the counter. .. code:: #!/bin/bash COUNTER=0 while [ $COUNTER -lt 10 ] do echo $COUNTER COUNTER=$(expr $COUNTER + 1) done .. image:: images/compute102/while-loop.example.gif - The following ``for`` loops iterates on i with values 1 through 5 and echos ``Welcome $i times`` for each value in the list (1,2,3,4,5). .. code:: #!/bin/bash for i in {1..5} do echo "Welcome $i times" done .. image:: images/compute102/for-loop.example.gif