Warming up with Unix commands

• Prev: LabQSM#Getting Started
• Next: Scripting

Intro

In this lesson, you will learn some basic commands of the Unix operative system. The operating system (OS) is the system software that manages computer hardware, software resources, and provides common services for computer programs, e.g. allowing for the communication between hardware and software. In order to use a computer, you need an OS. At this moment you may be using a Virtual Machine containing the Ubuntu OS, a Linux distribution, a Unix-like OS. The basic commands used in Linux and Unix are essentially the same.

If you open a terminal window a prompt will appear where you can type command lines. A snapshot of the prompt is shown here below:

You can inspect the environment by typing the following commands:

 $> whoami             #  (my username)
 $> hostname [-A]      #  (machine name)
 $> pwd                #  (current directory)
 $> ls [-ltr]          #  list files and dirs
 $> ps                 #  running processes
 $> top                #  more on running procs
 $> date               #  show date and time
 $> cal 2020           #  show 2020 calendar

Need help?

 $> man ls             #  manpage for ls
 $> info ls 
 $> gnuplot -h         #help for larger programs (e.g. gnuplot)

Shell history:

 $> history           # show last commands typed
 $> !s                # last command starting with 's'
 $> !5                # run again command #5 

Command-line tricks:

<tab>                #autocomplete
$ ls D<tab>
$ ls Downloads/

<ctrl>-a #move cursor to beginning of the line
<ctrl>-e #move to end
<ctrl>-c #cancel, quit, kill
<ctrl>-d #logout
<ctrl>-z #exit without quit (suspend)

At a glance, during this lecture we'll cover the following specific topics:

• Filesystem
• Users and Groups
• Processes
• Environment
• Interpreters / Scripting

Filesystem

Navigating in the filesystem

As seen above, the command $pwd (present working directory) shows the directory you are presently working on. Now we want to see how to navigate to other directories. This is done by using the command cd (change directory) followed by a path.

If we want to move for instance to the folder LabQSM we need to provide its path. A path is the address of a file or a folder in the filesystem.

We can either use:

  $> cd /home/max/LabQSM         # absolute path  or
  $> cd ./LabQSM/                # relative path,
  

provided that you are already in /home/max. This avoids typing complete paths.

You can then navigate up on level by typing:

  $> cd ..

and you will be back to the HOME directory. You can also navigate to multiple directories as:

  $> cd ./LabQSM/LAB_1/test_diamond 

More examples:

  $> cd /etc/init.d/          # absolute
  $> cd ../../usr/bin         # relative

Finally the commands

  $> cd     or    $> cd ~    or    $> cd $HOME

take you to the HOME directory, independently of the present working directory. In order to inspect the content of a folder, or the existence of a file, you can use the ls (list) command. Examples follow:

  $>  ls ./LabQSM     or     ls -a      or    ls -ltr

Exercise:

Using the ls command, look at the content of the root folder (the folder labelled / ).

Creating, removing, copying, and moving files and directories

For the sake of the presentation, here we first create a file. A more detail presentation of the topic is given below, Creating and editing text files.

A way to create a new file is the following:

• $> cat > filename
• Enter the content e.g. your name
• Press ctrl + d to return to command prompt.

A file named filename containing a text with your name has been created. You can find it in the list using the ls command, and you can inspect it by typing:

 $> cat filename

We will see later other ways to create and edit files using the editor named vi.

To create a new directory use the command:

 $> mkdir dirname

Files and directory are removed using the command rm, as

 $> rm filename
 $> rm -r dirname

To move a file from a directory to another, use the command mv:

 $> mv filename ./LabQSM

The same syntax applies also for moving directories. The same command is also used to rename (or overwrite) files:

 $> mv filename newfilename

File and directory can be also copied (duplicated) in the same directory or in other directories:

 $> cp filename ./LabQSM/         #copy the file in the LabQSM directory with the same name
 $> cp filename ./LabQSM/newname  #copy the file in the LabQSM directory with a new name
 $> cp -r dirname ./LabQSM/       #copy the directory dirname and all its content in the LabQSM directory
 $> cp -a dirname ./LabQSM/       # as above, but considering the folder as an archive (keeps some extra features like symbolic links)

As mentioned above, instead of copying physically a file or a folder from one location to another, one can create symbolic links (basically filesystem aliases):

 $> mkdir TMP ; cd TMP 
 $> ln -s  ../LabQSM/pseudo ./     # makes a link copy of the pseudo dir in the current directory

In all cases, when the possibility of losing information exists, a confirmation is asked to the user (who has to provide a y/n answer). This behaviour can be switched off by using the flag -f (force) on each of the commands above (rm, mv, cp). Instead the option -i (interactive), makes the command require the user interaction.

Users, Groups & Permissions

Now let's inspect the content of a directory in details, by typing

 $>  ls -al 

You can see the list of the files, containing information for each file including the owner and the group, the permissions (who can read, write, or execute that file), the dates it was modified, the size etc. Here is an example:

Permissions can be changed using the chmod command:

 $>  chmod a+r  file        # makes file readable by everyone
 $>  chmod u+x  file        # makes file executable by the user
 $>  chmod go-rw file       # makes file not readable nor writable by group and others

In the example above, the community of users is described by means of u,g,o,a (user, group, others, all). As an example, users belonging to the same group of the file will have permissions specified for g, etc.

As an example, you can try to create a file in the root directory. You can go to the root directory by typing $cd ../.. from your home (relative path) or directly by typing $cd / (absolute path). The command $cat > filename will not take effect as you do not have permission to write in the root directory.

Exercise:

Create a file in your HOME directory, check out the permissions granted by default, and change them to make the file writable by all users.

Processes

Linux (Unix) is a native multi-tasking and multi-user systems. So, it allows for multiple processes to operate simultaneously without interfering with each other. A process is an executing instance of a program and carry out different tasks within the operating system.

You can monitor the active process by using the command ps

$>  ps      #shows the processes for the current shell
$>  ps -a   #shows all processes not associated with a terminal
$>  ps -x   #shows all process owned by you

The ps command also shows the unique process id (PID), the terminal type that the user is logged into (TTY), the amount of CPU in minutes and seconds that the process has been running for (TIME), and the name of the command that launched the process (CMD).

In order to see all the options available for a specific command, e.g. ps, you can use the man command (manual) e.g.

$>  man ps   #shows all the option for the ps command
$>  man ls   #shows all the option for the ls command

It is possible to kill a process by typing:

$>  kill [-9] <pid>       # <pid> is the process id as given by ps

Processes that are launched interactively can be killed or interrupted by typing

$>  ctrl+c         # kills the process 
$>  ctrl+z         # suspect the process

In the latter case, the process can be sent in background or foreground by using bg and fg

Exercise

• Launch the process bc -l (This is an arbitrary precision calculator language with interactive execution of statements) and perform an arbitrary operation.
• Stop the process
• Send it in foreground (resume the process)
• Stop the process again
• Kill the process
• Verify that the process is not running anymore

Environment

Each shell (bash interpreter, here) comes with environment variables. Variables related to your environment that can be modified and customized.

$> env                      # shows all the defined environment variables
$> echo $<var-name>         # write the content of the variable 

Relevant variables are:

$> echo $HOME       # This variable contains the absolute path or your Home directory 
$> echo $USER       # This variable contains your uername
$> echo $PATH       # This variable specify a set of directories where executable programs are located. 

Executables located in the directories can be run simply by typing the name of the executable.

In general, if you want to execute an executable (binary or script), simply by typing its name you can either copy it in a directory contained in $PATH, or add the directory (e.g. mydir) containing your executable in the $PATH variable. This can be done by using the export command.

$> export PATH=$PATH:mydir

Many environment variables are defined in a file present in your home directory, named .bashrc, that is read ad each login, so it is possible to modify or define new variables permanently by editing this file. In order for the changes to take effect, you need to read and execute the content of the file by using the source command

$> source ~/.bashrc 

Variables can be modified or created from scratch:

$> myvar="This is my new var"
$> echo $myvar

When assigning variables pay attention to quotes:

$> var="$HOME"
$> echo $var   ->   /home/max
$> var='$HOME'
$> echo $var   ->   "$HOME"    
$> var=`hostname`
$> echo $var   ->   qmobile

The last quote pair (back quotes, ``) expands a command before the assignment is performed (very useful for scripting). As an example:

$> list=`ls *.f90`
$> echo $list           # list now contains the list of all f90 files in the current directory       

It is also possible to define new commands: here, for instance, we modify the rm command to make it safer:

$> alias rm='rm -i'

Now the rm command will ask for confirmation before removing a file:

$> touch pippo.txt         #create an empty file 
$> rm pippo.txt            #will ask: remove pippo.txt? (y/n)

One can do the same also with cp and mv.

Another example:

$> alias hello="echo hello $USER"
$> hello  -> hello max

Exercise

In this course, we will make use of the executable pw.x from the Quantum ESPRESSO distribution.

• Verify that the file pw.x is contained in one of the directories listed in $PATH
• Verify also that the file pw.x is an executable file
• Try to execute the code in a dry run (i.e. without input file)
• If needed, kill the process (ctrl+c or send it to bg and use ps + kill commands)

Hint: when executing a file, an explicit path such as ./ or /opt/QE/qe-6.2/bin/ needs to be provided (unless the file is placed directories listed in the $PATH variable).

Creating and editing text files

There are several ways to create files, let's explore some of them:

$> touch myfile.txt  #create an empty file

A way to create a file with some content was already see before:

• $> cat > filename.txt
• Enter the content e.g. your name
• Press ctrl + d to return to command prompt.

or alternatively:

$> cat > filename.txt <<EOF
   >Enter your content
   >EOF

or we can use and editor:

$> vi myfile.txt         #Using the editor vi
$> gedit myfile.txt      #Using the getid editor (Ubuntu)

While gedit is more intuitive, here we suggest using the vi editor, a bit more involved, but very powerful once you learn how to use it. A short guide of the vi editor can be found in ./LabQSM/docs/vi_cheat_sheet.pdf and can be open with the pdf reader evince:

evince ./LabQSM/docs/vi_cheat_sheet.pdf

Let's create a text file named atp.txt with the following content:

 9850 Nadal
 6630 Federer
 3075 Berrettini
12030 Djokovic

To view the file content you can use the following commands:

$> cat atp.txt
$> more atp.txt      # <space> for next page
$> less atp.txt      # <arrows> to move and <q> to exit

or, of course, you can open it with an editor such as vi/gedit.

To view the start/end (first/last lines) of the file you can do:

$> head -1 atp.txt
$> tail -2 atp.txt

Sort, check contents:

$> wc -l atp.txt         # provides the number of lines
$> sort atp.txt          # sort the file according to its first digit
$> sort -n atp.txt       # sort the file according to its numerical value
$> sort -nr atp.txt      # sort the file according to its numerical value in reverse order
$> sort -k2 atp.txt      # sort column 2 

Search for contents: grep: The grep utility searches any given input files, selecting lines that match one or more patterns.

$> grep er atp.txt       # Select lines matching with the string 
$> grep -v er atp.txt    # Selected lines are those not matching

More commands

Compressing files

Create an archive (backup, data transfer), portable across different machines:

 $> tar -cvf  file.tar  <dir>     # create a tar archive (aka tarball)
 $> tar -cvfz  file.tgz <dir>     # compress the resulting file with gzip

Extract the archive:

 $> tar -xvf  file.tar 
 $> tar -xvfz file.tgz

Connecting remotely to a unix machine:

 $> ssh [-X/-Y] -l <username> <machine>          or
 $> ssh [-X/-Y]    <username>@<machine>

You need to have a username and a password to the remote machine.

One can also use a similar syntax to copy a file from one machine to another:

 $> scp <file> <username>@<machine>:./.            #copy a file from the local to the remote machine
 $> scp <username>@<machine>:<path-to-file> ./     #copy a file from the remote to the local machine

Combining commands

A very powerful feature of unix commands is that their input and output can be redirected using the operators <, >, >>.

$> echo "Hello world" > greetings.txt            # output of echo redirected to file
$> echo "Hello world again" >> greetings.txt     # here the output of echo is appended to file
$> ./pw.x < file.in  > file.out                  # input is taken from file.in, output written to file.out

By redirecting command input/output one can combine different commands together (using the operator |), in such a way that each command takes input from the previous command and provides output to the next one. Examples follow.

$> cat greetings.txt  |  cut  -d" " -f1          
$> history | tail -5
$> history | head -1 > my_first_cmd.txt
$> more  my_first_cmd.txt

Command integration can also be combined with variables, see eg:

$> hostname             # complete hostname of the machine, e.g.  bamboo.fisica.unimo.it
$> host_no_domain=`hostname -A | cut -d "." -f1 `
                        # gives the machine name w/o domain, i.e. bamboo

Exercises

Exercise 1

Run our first calculation using pw.x

• Create a new directory (diamond/)
• cp there the pw.x input file LabQSM/LAB_1/test_diamond/scf.diamond.in
• Run pw.x reading the input file using <
• Run pw.x reading the input from command line (pw.x -i file.in)
• Redirect the output to file using >
• Search for the final total energy

Hint:

• pw.x executable is in the folder /usr/local/bin (which is in the $PATH)
• In order to run, you need to cp or link the pseudopotential directory (LabQSM/pseudo) in your local folder.
• Alternatively you can edit the input file providing the right path to the pseudo folder.
• The final total energy is preceded by an exclamation mark "!"

Exercise 2

My favourite tennis player

Here we will play more with info extraction and editing (quite relevant for scripting). Let's reconsider the file we prepared previously (atp.txt).

• add the names of the players as last column and commas between the fields

  9850, Nadal,  Rafael 
  6630, Federer,  Roger
  3075, Berrettini,  Matteo 
 12030, Djokovic,  Novak

• Pick a random integer <int> from 1 to 4
• Print Name, Surname of the player corresponding to the chosen line, and dump it to file

Simple solution

$> name=`cat atp.txt | head -<int> | tail -1 | cut -d"," -f3`  
$> surname=`cat atp.txt | head -<int> | tail -1 | cut -d"," -f2`
$> echo $name, $surname  >  my_favourite_player.txt