Introduction to Computing Fall 2015     Syllabus

9. Bash II (scripting & ssh)

Bash scripting

Besides being the environment you can access through a terminal on Linux, the bash is also a programming language. For certain tasks like repeating commands or running a series of programs it can be quite handy to write scripts in bash.

Next week we are going to look at how to use the OIST supercomputer facilities and we only can access them through ssh and using the bash scripting facilities greatly simplifies our life.

Variables

MY_VAR=10
echo $MY_VAR
echo ${MY_VAR}
echo "${MY_VAR}"

# a hash-sign starts a comment

The second and third variant are preferred over the first variant.

Parameter expansion

Tons ans tons of different options, that all allow you to do something with a variable before it is used in a context.

FILE="myfile.csv"
echo ${FILE%%.*}
echo ${FILE#*.}

https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html

Command substitution

HOST=$(hostname)
LONGINPUT="$(timedatectl)"

see http://www.gnu.org/software/bash/manual/bashref.html#Command-Substitution

Arrays

Arrays in bash can get a bit messy, but it boils down to

declare -a my_array
my_array[0]="Hey!"
my_array=("Ho!" "What?!")

echo ${my_array[1]}
echo ${my_array[@]}
echo ${#my_array}

There are also associative arrays (dictionaries) in bash.

http://www.thegeekstuff.com/2010/06/bash-array-tutorial/ and http://www.gnu.org/software/bash/manual/html_node/Arrays.html

Loops and control structure

# Looping through arrays
declare -a files=("1.csv" "2.csv" "3.csv")
for f in ${files[@]}; do
   cp ${f} "${f%%.*}.tsv"
done

for i in `seq 1 10`; do
  echo $i
done

for i in $( ls ); do
  echo item: $i
done

# ifs

if [ "foo" = "foo" ]; then
  echo expression evaluated as true
else
  echo expression evaluated as false
fi

for f in $( ls ); do
  if [ -d ${f} ]; then
    echo "Found directory ${f}"
  fi
done

In the last example I am using a conditional expression. In bash there are several special conditionals that you can use: http://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html

#! shebangs

In order to run you shell scripts like they are native commands put them in a file that looks like this.

#!/bin/bash

# Do stuff
cp a b

and then chmod +x filename. Afterwards you can just enter the path to the command in the terminal in execute it.

If you want to make sure that your scripts runs every where.

#!/usr/bin/env bash

and if you have a python script you can do the same

#!/usr/bin/env python

and to run it in POSIX compatible mode

#!/usr/bin/env sh

A note on file-permissions

You might have noticed me using chmod +x to make a file executable in one of the previous examples. This is based on the Unix file level permission setup.

If you run ls -l you will get in the first column a string like drwxr-xr-x and in the third and fourth user group. What this means is that the file belongs to the specific user and also to other members who are part of the specific group. The first column tells you what different permissions are set for that particular file. The first letter is either d, l or -. d stands for directory and l for link, while - is just a normal file. The next triplet of letters tells you the permission for the user that owns the file, the second triplet for the group that owns the file and the third triplet for everybody else.

r stands for reading, w for writing and x for eXecutable. If the permission is not set it is replaced by -. with the commands chmod and chown you can change the permissions and the user:group who owns the file.

Arithmetic

# $((  )) is an an arithmetic expansion.

x=0
x=$(($x + 1))
x=$((x + 1)) # parameter dereferencing is optional

n=0
echo "n = $n"
(( n += 1 ))
# (( $n += 1 )) is incorrect!
echo "n = $n"

# and now the let operator whoooho
let x=1
# and if I put quotes around it will be actually legible.
let "x += 1"

# and sometimes we just want to confuse people

y=7/3
echo $y
declare -i z
z=7/3
echo $z
# Better to be explicit and use arithmetic expansion (( ))
# But wait isn't 7/3=2.333 ?

let r=3.5
(( r = 3.5 ))

# damn no floating point numbers?
# bc - An arbitrary precision calculator language
r=1.7
s=$(echo "$r + 2.5" | bc) # I kid you not.

# comparing numbers
if (( x < y )); then
  stuff
fi

if [ $x -lt $y ]; then
  stuff
fi

# So you say you want to do numerical computing in bash?
# bc returns 1 or 0 for true and false
if [ $( echo "$t < 3.4" | bc ) -eq 1 ]; then
  stuff
fi

SSH

To securely login to remote systems use the ssh command.

ssh [-p port] [user@]hostname [command]

The default values for port is 22 and for user it is $(whoami). If a command is present that will be executed and the session terminated afterwards.

SSH keys

In order that you don't have to enter your password every time you login to a server or copy a file to and from it, I would recommend creating a private-public key and copying the public part to the server.

EMAIL="your_email@example.com"
USER="user-name"

if [ ! -f "~/.ssh/id_rsa.pub" ]; do
  ssh-keygen -t rsa -b 4096 -C "${EMAIL}"
fi

# ssh-copy-id does not exist on macs
ssh-copy-id ${USER}@login.oist.jp

# on a mac use this
# cat ~/.ssh/id_rsa.pub | ssh user@machine "mkdir -p ~/.ssh; cat >> ~/.ssh/authorized_keys"

SSH config

The configuration file for ssh in ~/.ssh/config allows you to set default configurations and aliases.

Host sango
   HostName sango.oist.jp
   User user-name

ssh sango

scp & rsync

scp and rsync are tools to copy files from and to remote hosts.

# From
scp host:path/to/file local/target
scp local/file host:remote/target

rsync is a bit more powerful, because it allows use to resume transfers when we lose connection and the basic syntax is the same like scp.

See https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories-on-a-vps

stuff for fun

# Execute commnad
cat /var/log/failog
# permission denied, what ?!
sudo !!

# Process id
pidof matlab

# monitor matlab
top -p $(pidof matlab)

# killall matlabs
killall matlab

# Ctrl - r #reverse incremental histroy search
# Tab #auto-completion
# Tab Tab # show possibilities
# Alt + * # show all possible completions
# rm *.csv # -> rm 2.csv 1.csv
# Ctrl - Alt - e # what am I actually doing?
history
!-2 # run the second to last command
sudo !:-2 # get the arguments of the second to last command
echo $RANDOM
SECONDS=0; sleep 5 ; echo "that took approximately $SECONDS seconds"
cd - # go back to the last directory