12. Shell Scripting¶

12.1. Feedback¶

12.2. Request: posted videos¶

videos are a lot of work for them to be high quality and accessible
even just a basic zoom is one more thing to manage
the notes have a complete transcript of every action on the terminal
I will pilot using webcaptioner to produce a transcript of more detail. I will then use these to fill in the notes and/or post them on slack (in a dedicated channel)

12.2.1. Request: more clear instructions¶

Sorry, I’m adapting based on how far we get, so I don’t have time to get iterative feedback on them.
I will try to expand on them and add examples, also some patterns will emerge so you’ll get used to the type of thing that’s expected.
also always feel free to ask more questions (slack or github issue to class repo)
you always can revise your work, so it’s also okay, espeically for more practice to put in the place of a response a question or try re-interpreting the instructions to check if they’re correct.
recent updates commit

12.2.2. Slack can be annoying¶

check your notification settings!!
use e-mail notifications if you prefer

12.2.3. Installations are hard¶

this is sort of generally true, sorry
the download part makes them not good for in class
always ask in slack
we might be done with most for a while
I will try to link to more detaled instructions if i can find them.

12.2.4. Request: More direction for extensions¶

instructions on site
Will start to add notes with a marker

12.2.5. Multiple repositories is hard to manage¶

try to think of them as separate directories and have an offline copy of each. We sort files often.
The reason for multiple is because that’s actually somewhat more realistic, partially and partially becuase of how templates work

12.2.6. This should be a 200 level course/ required¶

the goal is for it to eventually be required and for most students to take it at the same time as 212. I’m glad (and other faculty will be too!) that you seem to agree

12.2.7. Some things only work on mac¶

I think everything essential can be done on any OS, but there are different ways.
I try to check everything and we are through the most tricky parts.
If i miss something

12.2.8. More lecture/slides would be helpful¶

I probably shouldn’t have offered “slides” I will use prismia so that I can stay adaptive and swap the order in response
I will try to prepare diagrams for topics in advance and/or use the prismia drawing tool,
feel free to ask me to draw or explain either by raising your hand or on prismia during class

12.3. Jupyter Book¶

To make your KWL repo a jupyterbook, you need to:

add the required files (_toc.yml and _config.yml)
add your files to _toc.yml
customize meta-data and settings in _config.yml

To add images, you add the file to the repository and mention it in a markdown file using the following syntax.

![mandatory alt text](relative/path/to/image)

also, myst cheatsheet from jupyterbook docs

12.4. Bash has programming language features¶

Bash includes most common basic programming language features:

conditional expressions and constructs
loops
variables and builtin varialbes
comments #

12.4.1. Variables¶

echo $SHELL

/bin/bash

12.4.2. Loops¶

The syntax for a loop is:

for name [ [in [words …] ] ; ] do commands; done

or, with more spacing for readability

for item in [LIST]
do
  [COMMANDS]
done

Note in this syntax, we do not need a counter variable. In each iteration through the loop the next item from [LIST] becomes the value of item.

In our case, we can use this like:

for file in README.md gitoffline.md terminal.md software.md abstraction.md chart.md reorg.md stdinouterr.md hardwaresurvey.md gitlog.txt workflows.md gitplumbing.md gitunderstanding.md idethoughts.md numbers.md hexspeak.md gitstory.md gitstory2.md donotcommit.md jupyterbooktroubleshooting.md templating.md docs.md
> do
> echo $file
> done

To send all of the file names to STDOUT:

README.md
gitoffline.md
terminal.md
software.md
abstraction.md
chart.md
reorg.md
stdinouterr.md
hardwaresurvey.md
gitlog.txt
workflows.md
gitplumbing.md
gitunderstanding.md
idethoughts.md
numbers.md
hexspeak.md
gitstory.md
gitstory2.md
donotcommit.md
jupyterbooktroubleshooting.md
templating.md
docs.md

12.4.3. Conditionals¶

for file in README.md gitoffline.md terminal.md software.md abstraction.md chart.md reorg.md stdinouterr.md hardwaresurvey.md gitlog.txt workflows.md gitplumbing.md gitunderstanding.md idethoughts.md numbers.md hexspeak.md gitstory.md gitstory2.md donotcommit.md jupyterbooktroubleshooting.md templating.md docs.md
> do
> if test -f $file; then
> echo $file
> fi
> done

12.5. Shell Scripts¶

So, now we have checked your repository, but for you to use this on a regular basis there are two challenges.

You would have to type (or copy paste) those lines every time
the list of files will expand but the list in that will not update.

We will address the first with a bash script. Bash scripts allow you to make programs, including small utiltiies that help you with little tasks.

nano checker.sh

And we can paste this into the file:

for file in README.md gitoffline.md terminal.md software.md abstraction.md chart.md reorg.md stdinouterr.md hardwaresurvey.md gitlog.txt workflows.md gitplumbing.md gitunderstanding.md idethoughts.md numbers.md hexspeak.md gitstory.md gitstory2.md donotcommit.md jupyterbooktroubleshooting.md templating.md docs.md
> do
> if ! test -f $file; then
> echo $file
> fi
> done

12.6. Installing from source¶

We have used programs that we installed from their installation tools. Today we will use a small program I wrote for class, that we install from source.

git clone https://github.com/introcompsys/courseutils

Cloning into 'courseutils'...
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 16 (delta 5), reused 6 (delta 2), pack-reused 0
Receiving objects: 100% (16/16), 4.03 KiB | 4.03 MiB/s, done.
Resolving deltas: 100% (5/5), done.

First we can look at what this contains.

cd courseutils/
ls

It is very lightweight:

LICENSE		README.md	kwltracking.py	setup.py

Let’s review the purpose of each

This repo is public, so it has a license to describe how it is pertmitted to be used
The README is information about the repository; currently it has the documentation
kwltracking.py is the actual code of the program
setup.py is instructions for python’s pip on how to install the program.

pip install .

If it works, there will be along output that ends like:

...
Successfully installed syscourseutils-0.1.0

After you install, you will need to open a new terminal window (or otherwise restart bash, which varies on a lot of parameters)

Making it installable changes how we can run and use the program. In particular, it makes it a bash command.

So, on the terminal, we can use it like:

kwlfilecheck

This program scrapes the course website and echos to stdout the list of files expected in the KWL chart repo.

README gitoffline.md terminal.md software.md abstraction.md chart.md reorg.md stdinouterr.md hardwaresurvey.md gitlog.txt workflows.md gitplumbing.md gitunderstanding.md idethoughts.md numbers.md hexspeak.md gitstory.md gitstory2.md donotcommit.md jupyterbooktroubleshooting.md templating.md docs.md

Now, back in your KWL repo, we want to change our script to use this.

cd ..

We can edit the script, with nano

nano checker.sh

We can edit it so that it calls that process, assigns its output to a variable, and then uses that variable as our list output like this:

for file in $(kwlfilechecker)
do
 if ! test -f $file; then
  echo $file
 fi
done

Now you can use:

bash checker.sh

To see what, if any files you are missing.

12.7. Prepare for next class¶

Review the notes (to reinforce ideas and improve your memory of them)
Update your KWL to include all rows listed on the KWL page of the course site. ( because tracking your learning deepens your learning)
priority Make an issue on your grading contract repo that includes: a self-assessment of your progress on your contract so far and a plan going forward. Tag @sp22instructors so that we can help you meet your goals. (so that you get the grade you want)
priority On Windows, install Putty ( we will use this Monday)
Write a bash script that checks only if the number of files in your kwl meets or exceeds the minimum (number from list + 2 for toc and config) OR that does something else of your choosing. Add it to your KWL repo in a utils directory. Include Comments. (to practice so that you remember)

12.8. More Practice¶

priority add or link a glossary or cheatsheet, OR add a box with some historical context or extra reading links to the notes from any past class (in order to practice git/github and using jupyter book features, which are similar to other documentation tools, so even if you do not work in a python centric environment the concepts will translate)

12.9. Extension Ideas¶

document or add course scripts to the courseutils repo
write a similar util for another context and document it
turn your bash script above (if not the toy counting one) into a longer tutorial that explains its components piece by piece, how they work together.
use the jupyter book api referencing capability to document a side project or code from a different course, eg 110 (keep it in a separate private repo so as to not put solutions to a course on the public internet)

12.10. Questions after class¶

12.10.1. Will we create bash programs from scratch?¶

At least small ones, we probably will not write large programs though.

12.10.2. What are some applications of bash scripting?¶

We will see its use in HPC in the next class, but also for small utilities like this.

Introduction to Computer Systems

Shell Scripting

Contents