Warming up with Unix commands
- Prev: LabQSM#Getting Started
- Next: Scripting
Intro
In this lesson, you will learn some basic commands of the Unix operative system. The operating system (OS) is the system software that manages computer hardware, software resources, and provides common services for computer programs, e.g. allowing for the communication between hardware and software. In order to use a computer, you need an OS. At this moment you may be using a Virtual Machine containing the Ubuntu OS, a Linux distribution, a Unix-like OS. The basic commands used in Linux and Unix are essentially the same.
If you open a terminal window a prompt will appear where you can type command lines. A snapshot of the prompt is shown here below:
You can inspect the environment by typing the following commands:
$> whoami # (my username) $> hostname [-A] # (machine name) $> pwd # (current directory) $> ls [-ltr] # list files and dirs $> ps # running processes $> top # more on running procs $> date # show date and time $> cal 2020 # show 2020 calendar
Need help?
$> man ls # manpage for ls $> info ls $> gnuplot -h #help for larger programs (e.g. gnuplot)
Shell history:
$> history # show last commands typed $> !s # last command starting with 's' $> !5 # run again command #5
Command-line tricks:
<tab> #autocomplete $ ls D<tab> $ ls Downloads/
<ctrl>-a #move cursor to beginning of the line <ctrl>-e #move to end <ctrl>-c #cancel, quit, kill <ctrl>-d #logout <ctrl>-z #exit without quit (suspend)
At a glance, during this lecture we'll cover the following specific topics:
- Filesystem
- Users and Groups
- Processes
- Environment
- Interpreters / Scripting
Filesystem
As seen above, the command $pwd (present working directory) shows the directory you are presently working on. Now we want to see how to navigate to other directories. This is done by using the command cd (change directory) followed by a path.
If we want to move for instance to the folder LabQSM we need to provide its path. A path is the address of a file or a folder in the filesystem.
We can either use:
$> cd /home/max/LabQSM # absolute path or $> cd ./LabQSM/ # relative path,
provided that you are already in /home/max. This avoids typing complete paths.
You can then navigate up on level by typing:
$> cd ..
and you will be back to the HOME directory. You can also navigate to multiple directories as:
$> cd ./LabQSM/LAB_1/test_diamond
More examples:
$> cd /etc/init.d/ # absolute $> cd ../../usr/bin # relative
Finally the commands
$> cd or $> cd ~ or $> cd $HOME
take you to the HOME directory, independently of the present working directory.
In order to inspect the content of a folder, or the existence of a file, you can use the ls (list) command. Examples follow:
$> ls ./LabQSM or ls -a or ls -ltr
Exercise:
Using the ls command, look at the content of the root folder (the folder labelled / ).
Creating, removing, copying, and moving files and directories
For the sake of the presentation, here we first create a file. A more detail presentation of the topic is given below, Creating and editing text files.
A way to create a new file is the following:
$> cat > filename- Enter the content e.g. your name
- Press
ctrl + dto return to command prompt.
A file named filename containing a text with your name has been created.
You can find it in the list using the ls command,
and you can inspect it by typing:
$> cat filename
We will see later other ways to create and edit files using the editor named vi.
To create a new directory use the command:
$> mkdir dirname
Files and directory are removed using the command rm, as
$> rm filename $> rm -r dirname
To move a file from a directory to another, use the command mv:
$> mv filename ./LabQSM
The same syntax applies also for moving directories. The same command is also used to rename (or overwrite) files:
$> mv filename newfilename
File and directory can be also copied (duplicated) in the same directory or in other directories:
$> cp filename ./LabQSM/ #copy the file in the LabQSM directory with the same name $> cp filename ./LabQSM/newname #copy the file in the LabQSM directory with a new name $> cp -r dirname ./LabQSM/ #copy the directory dirname and all its content in the LabQSM directory $> cp -a dirname ./LabQSM/ # as above, but considering the folder as an archive (keeps some extra features like symbolic links)
As mentioned above, instead of copying physically a file or a folder from one location to another, one can create symbolic links (basically filesystem aliases):
$> mkdir TMP ; cd TMP $> ln -s ../LabQSM/pseudo ./ # makes a link copy of the pseudo dir in the current directory
In all cases, when the possibility of losing information exists, a confirmation is asked to the user (who has to provide a y/n answer).
This behaviour can be switched off by using the flag -f (force) on each of the commands above (rm, mv, cp).
Instead the option -i (interactive), makes the command require the user interaction.
Users, Groups & Permissions
Now let's inspect the content of a directory in details, by typing
$> ls -al
You can see the list of the files, containing information for each file including the owner and the group, the permissions (who can read, write, or execute that file), the dates it was modified, the size etc. Here is an example:
Permissions can be changed using the chmod command:
$> chmod a+r file # makes file readable by everyone $> chmod u+x file # makes file executable by the user $> chmod go-rw file # makes file not readable nor writable by group and others
In the example above, the community of users is described by means of u,g,o,a (user, group, others, all). As an example, users belonging to the same group of the file will have permissions specified for g, etc.
As an example, you can try to create a file in the root directory. You can go to the root directory by typing $cd ../.. from your home (relative path) or directly by typing $cd / (absolute path).
The command $cat > filename will not take effect as you do not have permission to write in the root directory.
Exercise:
Create a file in your HOME directory, check out the permissions granted by default, and change them to make the file writable by all users.
Processes
Linux (Unix) is a native multi-tasking and multi-user systems. So, it allows for multiple processes to operate simultaneously without interfering with each other. A process is an executing instance of a program and carry out different tasks within the operating system.
You can monitor the active process by using the command ps
$> ps #shows the processes for the current shell $> ps -a #shows all processes not associated with a terminal $> ps -x #shows all process owned by you
The ps command also shows the unique process id (PID), the terminal type that the user is logged into (TTY),
the amount of CPU in minutes and seconds that the process has been running for (TIME), and the name of the command that launched the process (CMD).
In order to see all the options available for a specific command, e.g. ps, you can use the man command (manual) e.g.
$> man ps #shows all the option for the ps command $> man ls #shows all the option for the ls command
It is possible to kill a process by typing:
$> kill [-9] <pid> # <pid> is the process id as given by ps
Processes that are launched interactively can be killed or interrupted by typing
$> ctrl+c # kills the process $> ctrl+z # suspect the process
In the latter case, the process can be sent in background or foreground by using bg and fg
Exercise
- Launch the process
bc -l(This is an arbitrary precision calculator language with interactive execution of statements) and perform an arbitrary operation. - Stop the process
- Send it in foreground (resume the process)
- Stop the process again
- Kill the process
- Verify that the process is not running anymore
Environment
Each shell (bash interpreter, here) comes with environment variables. Variables related to your environment that can be modified and customized.
$> env # shows all the defined environment variables $> echo $<var-name> # write the content of the variable
Relevant variables are:
$> echo $HOME # This variable contains the absolute path or your Home directory $> echo $USER # This variable contains your uername $> echo $PATH # This variable specify a set of directories where executable programs are located.
Executables located in the directories can be run simply by typing the name of the executable.
In general, if you want to execute an executable (binary or script), simply by typing its name you can either copy it in a directory contained in $PATH, or add the directory (e.g. mydir) containing your executable in the $PATH variable. This can be done by using the export command.
$> export PATH=$PATH:mydir
Many environment variables are defined in a file present in your home directory, named .bashrc, that is read ad each login, so it is possible to modify or define new variables permanently by editing this file. In order for the changes to take effect, you need to read and execute the content of the file by using the source command
$> source ~/.bashrc
Variables can be modified or created from scratch:
$> myvar="This is my new var" $> echo $myvar
When assigning variables pay attention to quotes:
$> var="$HOME" $> echo $var -> /home/max $> var='$HOME' $> echo $var -> "$HOME" $> var=`hostname` $> echo $var -> qmobile
The last quote pair (back quotes, ``) expands a command before the assignment is performed (very useful for scripting). As an example:
$> list=`ls *.f90` $> echo $list # list now contains the list of all f90 files in the current directory
It is also possible to define new commands: here, for instance, we modify the rm command to make it safer:
$> alias rm='rm -i'
Now the rm command will ask for confirmation before removing a file:
$> touch pippo.txt #create an empty file $> rm pippo.txt #will ask: remove pippo.txt? (y/n)
One can do the same also with cp and mv.
Another example:
$> alias hello="echo hello $USER" $> hello -> hello max
Exercise
In this course, we will make use of the executable pw.x from the Quantum ESPRESSO distribution.
- Verify that the file
pw.xis contained in one of the directories listed in $PATH - Verify also that the file
pw.xis an executable file - Try to execute the code in a dry run (i.e. without input file)
- If needed, kill the process (ctrl+c or send it to bg and use ps + kill commands)
Hint: when executing a file, an explicit path such as ./ or /opt/QE/qe-6.2/bin/ needs to be provided (unless the file is placed directories listed in the $PATH variable).
Creating and editing text files
There are several ways to create files, let's explore some of them:
$> touch myfile.txt #create an empty file
A way to create a file with some content was already see before:
$> cat > filename.txt- Enter the content e.g. your name
- Press
ctrl + dto return to command prompt.
or alternatively:
$> cat > filename.txt <<EOF >Enter your content >EOF
or we can use and editor:
$> vi myfile.txt #Using the editor vi $> gedit myfile.txt #Using the getid editor (Ubuntu)
While gedit is more intuitive, here we suggest using the vi editor, a bit more involved, but very powerful once you learn how to use it. A short guide of the vi editor can be found in ./LabQSM/docs/vi_cheat_sheet.pdf and can be open with the pdf reader evince:
evince ./LabQSM/docs/vi_cheat_sheet.pdf
Let's create a text file named atp.txt with the following content:
9850 Nadal 6630 Federer 3075 Berrettini 12030 Djokovic
To view the file content you can use the following commands:
$> cat atp.txt $> more atp.txt # <space> for next page $> less atp.txt # <arrows> to move and <q> to exit
or, of course, you can open it with an editor such as vi/gedit.
To view the start/end (first/last lines) of the file you can do:
$> head -1 atp.txt $> tail -2 atp.txt
Sort, check contents:
$> wc -l atp.txt # provides the number of lines $> sort atp.txt # sort the file according to its first digit $> sort -n atp.txt # sort the file according to its numerical value $> sort -nr atp.txt # sort the file according to its numerical value in reverse order $> sort -k2 atp.txt # sort column 2
Search for contents: grep:
The grep utility searches any given input files, selecting lines that match one or more patterns.
$> grep er atp.txt # Select lines matching with the string $> grep -v er atp.txt # Selected lines are those not matching
More commands
Compressing files
Create an archive (backup, data transfer), portable across different machines:
$> tar -cvf file.tar <dir> # create a tar archive (aka tarball) $> tar -cvfz file.tgz <dir> # compress the resulting file with gzip
Extract the archive:
$> tar -xvf file.tar $> tar -xvfz file.tgz
Connecting remotely to a unix machine:
$> ssh [-X/-Y] -l <username> <machine> or $> ssh [-X/-Y] <username>@<machine>
You need to have a username and a password to the remote machine.
One can also use a similar syntax to copy a file from one machine to another:
$> scp <file> <username>@<machine>:./. #copy a file from the local to the remote machine $> scp <username>@<machine>:<path-to-file> ./ #copy a file from the remote to the local machine
Combining commands
A very powerful feature of unix commands is that their input and output can be redirected using the operators <, >, >>.
$> echo "Hello world" > greetings.txt # output of echo redirected to file $> echo "Hello world again" >> greetings.txt # here the output of echo is appended to file $> ./pw.x < file.in > file.out # input is taken from file.in, output written to file.out
By redirecting command input/output one can combine different commands together (using the operator |), in such a way that each command takes input from the previous command and provides output to the next one. Examples follow.
$> cat greetings.txt | cut -d" " -f1 $> history | tail -5 $> history | head -1 > my_first_cmd.txt $> more my_first_cmd.txt
Command integration can also be combined with variables, see eg:
$> hostname # complete hostname of the machine, e.g. bamboo.fisica.unimo.it
$> host_no_domain=`hostname -A | cut -d "." -f1 `
# gives the machine name w/o domain, i.e. bamboo
Exercises
Exercise 1
Run our first calculation using pw.x
- Create a new directory (diamond/)
- cp there the pw.x input file LabQSM/LAB_1/test_diamond/scf.diamond.in
- Run pw.x reading the input file using <
- Run pw.x reading the input from command line (pw.x -i file.in)
- Redirect the output to file using >
- Search for the final total energy
Hint:
- pw.x executable is in the folder /usr/local/bin (which is in the $PATH)
- In order to run, you need to cp or link the pseudopotential directory (LabQSM/pseudo) in your local folder.
- Alternatively you can edit the input file providing the right path to the pseudo folder.
- The final total energy is preceded by an exclamation mark "!"
Exercise 2
My favourite tennis player
Here we will play more with info extraction and editing (quite relevant for scripting). Let's reconsider the file we prepared previously (atp.txt).
- add the names of the players as last column and commas between the fields
9850, Nadal, Rafael 6630, Federer, Roger 3075, Berrettini, Matteo 12030, Djokovic, Novak
- Pick a random integer <int> from 1 to 4
- Print Name, Surname of the player corresponding to the chosen line, and dump it to file
Simple solution
$> name=`cat atp.txt | head -<int> | tail -1 | cut -d"," -f3` $> surname=`cat atp.txt | head -<int> | tail -1 | cut -d"," -f2` $> echo $name, $surname > my_favourite_player.txt