13. How can I use bash to automate things?#
13.1. What is bash again?#
So far we have used bash commands to navigate our file system as a way to learn about the file system itself. To do this we used commands like:
mvcdpwdls
Bash is a unix shell for the GNU operating system and it has been adopted in other contexts as well. It is the default shell on Ubuntu linux as well for example (and many others). This is why we teach it.
We can describe this relationship with a concept map. Each shape corresponds to a concept and the arrows are labeled to describe the relationship between the two concepts. This can be a helpful way of writing out how you understand things.
A Unix shell is both a command interpreter and a programming language. As a command interpreter, the shell provides the user interface to the rich set of GNU utilities. The programming language features allow these utilities to be combined.
Read the official definition of bash and a shell in the bash manual
We can add this information to the diagram:
13.2. What does this definiton of bash mean?#
Today we will start from the main course directory.
Note on macOS when I run bash to use the bash shell, it reminds me that MacOS prefers a different shell.
Last login: Sat Oct 21 17:29:01 on ttys080
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
We can have different shells in the same terminal in some cases, and all of the different shells on the system have access to the same file system.
Then I will move to my systems dir
cd Documents/inclass/systems/
and look at files
ls
fa23-kwl-brownsarahm github-inclass-fa23-brownsarahm
fall2023 test
ghic tiny-book
13.3. Working from an Example#
Important
Here I used cd fa23-kwl-brownsarahm and gh repo view --web to go to my KWL repo in the browser and then cd .. to go back
In your KWL repo the .github/workflows directory has several .yml files that are github action files.
These files are yaml, with key:value pairs. The values of the run key are commands that are passed to bash. So these are a place where you can see examples of what I did to look for particular syntax.
Learning from examples is a common strategy in CS and how you can learn new technology especially in a job, so today in class, we modeled that by looking for different syntax in the action files and building up what those things do.
13.4. Variables in Bash#
We can create variables like VARNAME=value.
NAME='Sarah'
notice that there are no spaces around the =. spaces in bash separate commands and options, so they cannot be in variable declarations.
A common mistake is to put a space around the = sign, this is actually considered good style in many other languages.
NAME ='Sarah'
-bash: NAME: command not found
In bash, however, this creates an error. When there is a space after NAME, bash tried to interpet NAME as a bash command, but then it does not find it, so it gives an error.
NAME='Sarah'
We can use variables with a $ before the variable name.
echo $NAME
Sarah
Important
This variable is local, in memory, to the current terminal window, so if we open a separate window and try echo $NAME it will not work. We can also see that it does not create any file changes.
echo $NAME
Sarah
We can see that it does not create any files by comparing ls now to what it was before:
ls
fa23-kwl-brownsarahm github-inclass-fa23-brownsarahm
fall2023 test
ghic tiny-book
ls -a
. ghic
.. github-inclass-fa23-brownsarahm
fa23-kwl-brownsarahm test
fall2023 tiny-book
We can, however use the variable at different working directories. So if we move
cd github-inclass-fa23-brownsarahm/
and echo agian
echo $NAME
Sarah
it still works!
The $ is essential syntax for recalling variables in bash. If we forget it, it treats it as a literal
echo NAME
NAME
so we get the variable name out instead of the variable value
We’ll go back up a level now:
cd ..
13.5. Bash Loops#
We can also make loops like
for loopvar in list
do
# loop body
echo $loopvar
done
So for example we can loop over some names
for name in Sarah Amoy Marcin
> do
> echo $name
> done
A few important things, to make note of:
loop variable does not need to be an iterator. the loop variable here
nametakes each value from a list (Sarah Amoy Marcin)lists is bash are defined with no brackets and no commas
Sarah Amoy Marcinis a listwe start the loop body with
doand close it withdonethese are like the{and}in some langauges.after we start the loop with a line starting with
forbash continues that “command” by putting>at the start of each line until we end the loop withdoneand then it runs.
And we see it echos each name:
Sarah
Amoy
Marcin
Note that the loop variable is not scope limited to the loop. After the loop it still has the same value of the last time through the loop.
echo $name
Marcin
When we get the command back with the up arrow key, it puts it all on one line, because it was one command. The ; (semicolon) separates the “lines”
We can add another name in
for name in Sarah Amoy Marcin Osman; do echo $name; done
Sarah
Amoy
Marcin
Osman
the output is like before, but with one more
echo $name
Osman
and this time Osman is the name left in the loop variable later.
13.6. Nesting commands#
There are examples of this in your KWL repo actions. If we want the output of a command to be in a variable this is how to do it.
To do this, let’s move to a folder with some files in it.
cd github-inclass-fa23-brownsarahm/
here we have a number of files in this working directory:
ls
API.md abstract_base_class.py important_classes.py
CONTRIBUTING.md alternative_classes.py setup.py
LICENSE.md docs tests
README.md helper_functions.py
We can run a command to generate the list by putting it inside $() to run that command first. Think kind of like PEMDAS and the $ in bash is for variables.
for FILE in $(ls)
> do
> echo $FILE
> done
API.md
CONTRIBUTING.md
LICENSE.md
README.md
abstract_base_class.py
alternative_classes.py
docs
helper_functions.py
important_classes.py
setup.py
tests
the $() tells bash to run that command first and then hold its output as a variable for use elsewhere
For example, in your getassignment.yml I have a line like this:
pretitle="prepare-"$(sysgetbadgedate --prepare)
it creates a variable that is “prepare-” concatenated with the output of sysgetbadgedate --prepare
echo $pretitle
prepare-2023-10-26
We can also modify our previous loop with an option:
for FILE in $(ls -a); do echo $FILE; done
.
..
.git
.github
API.md
CONTRIBUTING.md
LICENSE.md
README.md
abstract_base_class.py
alternative_classes.py
docs
helper_functions.py
important_classes.py
setup.py
tests
13.7. Conditionals in bash#
We can also do conditional statements
if test -f docs
> then
> echo "file"
> fi
the key parts of this:
testchecks if a file or directory existsthe
-foption makes it check if the item is a filewhat to do if the condition is met goes after a
thenkeywordthe
fi(backwardsif) closes the if statmentthe
>work again like the loop and the\we did in the past
this if does nothing though, because docs is a directory not a file.
If we switch it, we get output:
if test -f API.md ; then echo "file"; fi
file
13.8. Script files#
We can put our script into a file
nano filecheck.sh
and check the content of the file
cat filecheck.sh
for file in $(ls)
do
if test -f $file
then
echo $file
fi
done
and run it with bash <filename>
bash filecheck.sh
API.md
CONTRIBUTING.md
LICENSE.md
README.md
abstract_base_class.py
alternative_classes.py
filecheck.sh
helper_functions.py
important_classes.py
setup.py
And this is as ls except without the folders:
ls
API.md abstract_base_class.py helper_functions.py
CONTRIBUTING.md alternative_classes.py important_classes.py
LICENSE.md docs setup.py
README.md filecheck.sh tests
Important
We also looked at the structure and some examples of the action files in the KWL repo and course website.
I recommend the docs, starting from the quickstart and the understand section
13.9. Using GH#
cd ../fall2023/
gh pr list
Showing 5 of 5 open pull requests in introcompsys/fall2023
#15 Create mermaid.md introcompsys:trevmoy-pat... about 5 days ago
#12 sample proposals for 212... 212build about 10 days ago
#11 Community-typos jacksonarich:main about 16 days ago
#10 Hash notes NoahV17:hash_notes about 18 days ago
#5 Added question to 9 26 n... NoahV17:added-question-t... about 23 days ago
Grep can search content, so with a pipe we can use it to filter results.
gh pr list | grep sample
12 sample proposals for 212 docs build 212build OPEN 2023-10-14 17:02:14 +0000 UTC
13.10. Terminal coloring and customizing your prompt.#
Your path settings, aliases, and the your prompt can be set with a file.
On MacOS the file is called .bash_profile and is stored at your ~ home directory. If you use zsh instead of bash there is a different file, but it is similar.
On linux it is .bashrc it should be the same in WSL on Windows.
Community Badge opportunity
If you use windows, can you PR to update these notes with what file is used and which terminal/shell you are using (bash via GitBash, powershell, bash in WSL, etc. )
cat ~/.bash_profile
I have some aliases set, this way when I use Python, I can use the python command instead of python3.
the PS1 variable describes the main default prompt, according to the spec. this tool can help you customize your prompt
alias pip=pip3
alias python=python3
export PS1="\u@\W $ "
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/brownsarahm/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/Users/brownsarahm/anaconda3/etc/profile.d/conda.sh" ]; then
. "/Users/brownsarahm/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/Users/brownsarahm/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
13.11. Prepare for Next Class#
Read the 3 bulleted examples of why use a cluster from HPC carpentry.
13.12. Review Badge#
Update your KWL Chart learned column with what you’ve learned
Write a bash script that you can run in your group repo to generate a file with a list of all of your PRs and. Save the script as groupcontributions.sh and its output as group_contributions-YYYY-MM-DD.md in your KWL repo.
13.13. Practice Badge#
Update your KWL Chart learned column with what you’ve learned
Write a bash script that you can run in your group repo to generate a file with a list of all of your PRs and PR reviews (you may go in and assign yourself to these PRs to make this easier). Save the script as groupcontributions.sh and its output as group_contributions-YYYY-MM-DD.md in your KWL repo.
13.14. Experience Report Evidence#
13.15. Questions After Today’s Class#
13.15.1. How often will we be writing scripts for this class?#
Just a few times to get practice and exposure, but they will come back in 412 and are handy in general.
13.15.2. Are bash scripts commonly used today in real world applications or is it something that was used more in the 70s?#
They are still super common. GitHub actions can be written using bash scripting as a strategy. Lots of systems have build scripts.
The thing we would not do that often that we did today was write it from nano instead of a full IDE.
13.15.3. What is the scope of loop variables?#
In bash the loop variable has the same scope as any other variable. It is the same in Python. This is one of the many features that can very from language to language.
13.15.4. When would I use bash instead of a different programming language?#
When you need to do things that are file operations or other os level actions, it can be faster to run if it is in bash.
13.15.5. What specific tasks do developers like to use bash to automate?#
build processes, file manipulations, git operations. Anything even just a few steps that would be repetitive and you would have to do often.
13.15.6. do people write full programs in bash using things like nano or directly into the terminal?#
Yes! Some people especialy people who started programming a long time ago still do.
More often, using a terminal text editor is a handy tool to use on occasion, but not for every day work. We will work on a remote server, which is an occasion you would use that, on Thursday.
13.15.7. Is there a way to save files that you create in the shell?#
yes, today we saved the filecheck.sh file.
13.15.8. what else can we do with bash?#
Its mostly for automating things and processing files or outputs.
It can do anything though.
13.15.9. Does running a bash script with `bash `` remove the need to add #!/bin/bash to the top of it and then make it executable prior to running?#
Yes. by running it like bash <filename it works just like passing those lines in the file to bash.
Alternatively you can put the shebang #!/bin/bash at the top to tell the computer what shell to use to run it and then treat it as an executable.
We will learn about making it executable permissions wise on Thursday.
13.15.10. Is it possible to create functions in bash scripts?#
13.15.11. Where would we see bash the most as a developer? Would we use this language more for an embedded swe role or is it something that’s present in every type of programming job.#
I think it’s present in just about any programming job. You will always need to work with files at times, you will alwasy have repetitive things that you do not want to do manually. You may write more or less, but it’s alwasy a handy tool to have.
13.15.12. Where could I learn more about writing something similar to the workflows we saw?#
The github actions documentation has instructions with how tos, API like documentation, and examples.