15. How can I run a job on a remote server ?#

15.1. Logging back in to seawulf#

First, we will log in as we did before :

ssh -l brownsarahm seawulf.uri.edu

brownsarahm@seawulf.uri.edu's password: 
Last login: Thu Oct 26 13:03:10 2023 from 172.20.131.240

15.2. Creating a script#

On a server where we only have terminal access, using a terminal text based editor is what we have. For larger code, we could use git or copy the files, but to edit on the server, we only have a terminal text editor:

nano example-job.sh

We will put the following in the file:

echo -n "this script is running on "
hostname

I used a new option on echo that we have not seen before and we used the documentation to work from there.

man echo

bash example-job.sh 

this script is running on seawulf.uri.edu

./example-job.sh

-bash: ./example-job.sh: Permission denied

ls -l

total 138456
-rw-r--r--. 1 brownsarahm spring2022-csc392 12534006 Apr 18  2021 bash-lesson.tar.gz
drwxr-xr-x. 2 brownsarahm spring2022-csc392     4096 Apr  4  2023 compilec
-rw-r--r--. 1 brownsarahm spring2022-csc392 77426528 Jan 16  2018 dmel-all-r6.19.gtf
-rw-r--r--. 1 brownsarahm spring2022-csc392   721242 Jan 25  2016 dmel_unique_protein_isoforms_fb_2016_01.tsv
drwxr-xr-x. 3 brownsarahm spring2022-csc392       24 Mar 30  2023 example
-rw-r--r--. 1 brownsarahm spring2022-csc392       46 Oct 31 13:04 example-job.sh
-rw-r--r--. 1 brownsarahm spring2022-csc392 25056938 Jan 25  2016 gene_association.fb
-rw-r--r--. 1 brownsarahm spring2022-csc392      447 Oct 26 13:30 linecounts.txt
drwxr-xr-x. 2 brownsarahm spring2022-csc392       10 Mar  8  2022 results
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625262 Jan 25  2016 SRR307023_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625262 Jan 25  2016 SRR307023_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625376 Jan 25  2016 SRR307024_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625376 Jan 25  2016 SRR307024_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625286 Jan 25  2016 SRR307025_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625286 Jan 25  2016 SRR307025_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625302 Jan 25  2016 SRR307026_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625302 Jan 25  2016 SRR307026_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625312 Jan 25  2016 SRR307027_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625312 Jan 25  2016 SRR307027_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625338 Jan 25  2016 SRR307028_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625338 Jan 25  2016 SRR307028_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625390 Jan 25  2016 SRR307029_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625390 Jan 25  2016 SRR307029_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625318 Jan 25  2016 SRR307030_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625318 Jan 25  2016 SRR307030_2.fastq

chmod +x example-job.sh 

ls -l | grep example

drwxr-xr-x. 3 brownsarahm spring2022-csc392       24 Mar 30  2023 example
-rwxr-xr-x. 1 brownsarahm spring2022-csc392       46 Oct 31 13:04 example-job.sh

./example-job.sh

this script is running on seawulf.uri.edu

cp example-job.sh ex2.sh
cp example-job.sh ex3.sh
cp example-job.sh ex4.sh

ls| grep ex

ex2.sh
ex3.sh
ex4.sh
example
example-job.sh

ls -l

total 138468
-rw-r--r--. 1 brownsarahm spring2022-csc392 12534006 Apr 18  2021 bash-lesson.tar.gz
drwxr-xr-x. 2 brownsarahm spring2022-csc392     4096 Apr  4  2023 compilec
-rw-r--r--. 1 brownsarahm spring2022-csc392 77426528 Jan 16  2018 dmel-all-r6.19.gtf
-rw-r--r--. 1 brownsarahm spring2022-csc392   721242 Jan 25  2016 dmel_unique_protein_isoforms_fb_2016_01.tsv
-rwxr-xr-x. 1 brownsarahm spring2022-csc392       46 Oct 31 13:20 ex2.sh
-rwxr-xr-x. 1 brownsarahm spring2022-csc392       46 Oct 31 13:20 ex3.sh
-rwxr-xr-x. 1 brownsarahm spring2022-csc392       46 Oct 31 13:20 ex4.sh
drwxr-xr-x. 3 brownsarahm spring2022-csc392       24 Mar 30  2023 example
-rwxr-xr-x. 1 brownsarahm spring2022-csc392       46 Oct 31 13:04 example-job.sh
-rw-r--r--. 1 brownsarahm spring2022-csc392 25056938 Jan 25  2016 gene_association.fb
-rw-r--r--. 1 brownsarahm spring2022-csc392      447 Oct 26 13:30 linecounts.txt
drwxr-xr-x. 2 brownsarahm spring2022-csc392       10 Mar  8  2022 results
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625262 Jan 25  2016 SRR307023_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625262 Jan 25  2016 SRR307023_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625376 Jan 25  2016 SRR307024_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625376 Jan 25  2016 SRR307024_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625286 Jan 25  2016 SRR307025_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625286 Jan 25  2016 SRR307025_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625302 Jan 25  2016 SRR307026_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625302 Jan 25  2016 SRR307026_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625312 Jan 25  2016 SRR307027_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625312 Jan 25  2016 SRR307027_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625338 Jan 25  2016 SRR307028_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625338 Jan 25  2016 SRR307028_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625390 Jan 25  2016 SRR307029_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625390 Jan 25  2016 SRR307029_2.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625318 Jan 25  2016 SRR307030_1.fastq
-rw-r--r--. 1 brownsarahm spring2022-csc392  1625318 Jan 25  2016 SRR307030_2.fastq

chmod -x ex*.sh

ls -l | grep ex

-rw-r--r--. 1 brownsarahm spring2022-csc392       46 Oct 31 13:20 ex2.sh
-rw-r--r--. 1 brownsarahm spring2022-csc392       46 Oct 31 13:20 ex3.sh
-rw-r--r--. 1 brownsarahm spring2022-csc392       46 Oct 31 13:20 ex4.sh
drwxr-xr-x. 3 brownsarahm spring2022-csc392       24 Mar 30  2023 example
-rw-r--r--. 1 brownsarahm spring2022-csc392       46 Oct 31 13:04 example-job.sh

./example-job.sh

-bash: ./example-job.sh: Permission denied

chmod +x example-job.sh 

./example-job.sh

this script is running on seawulf.uri.edu

nano example-job.sh 

cat example-job.sh 

#!/bin/bash
#SBATCH -J hello-world

echo -n "this script is running on "
hostname

sbatch example-job.sh 

Submitted batch job 27093

squeue -u brownsarahm

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

sbatch example-job.sh 

Submitted batch job 27107

squeue -u brownsarahm

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

whoami

brownsarahm

ls 

bash-lesson.tar.gz                           linecounts.txt     SRR307026_1.fastq
compilec                                     results            SRR307026_2.fastq
dmel-all-r6.19.gtf                           slurm-27093.out    SRR307027_1.fastq
dmel_unique_protein_isoforms_fb_2016_01.tsv  slurm-27107.out    SRR307027_2.fastq
ex2.sh                                       SRR307023_1.fastq  SRR307028_1.fastq
ex3.sh                                       SRR307023_2.fastq  SRR307028_2.fastq
ex4.sh                                       SRR307024_1.fastq  SRR307029_1.fastq
example                                      SRR307024_2.fastq  SRR307029_2.fastq
example-job.sh                               SRR307025_1.fastq  SRR307030_1.fastq
gene_association.fb                          SRR307025_2.fastq  SRR307030_2.fastq

cat slurm-27093.out 

this script is running on n005.cluster.com

cat slurm-27107.out 

this script is running on n005.cluster.com

nano example-job.sh 

logout
Connection to seawulf.uri.edu closed.

ssh -l brownsarahm seawulf.uri.edu

brownsarahm@seawulf.uri.edu's password: 
Permission denied, please try again.
brownsarahm@seawulf.uri.edu's password: 
Last failed login: Tue Oct 31 13:41:13 EDT 2023 from 172.20.131.240 on ssh:notty
There were 3 failed login attempts since the last successful login.
Last login: Tue Oct 31 12:59:02 2023 from 172.20.131.240

cat ex2.sh 
echo -n "this script is running on "
hostname

chmod +x ex2.sh 

sbatch -J hello -o name-%j ex2.sh

sbatch: error: This does not look like a batch script.  The first
sbatch: error: line must start with #! followed by the path to an interpreter.
sbatch: error: For instance: #!/bin/sh

nano ex2.sh 

sbatch -J hello -o name-%j ex2.sh
Submitted batch job 27140

ls |grap name
-bash: grap: command not found

ls |grep name
name-27140

sbatch  ex2.sh
Submitted batch job 27141

sysapprovedbages

-bash: sysapprovedbages: command not found

sysapprovedbadges --help

-bash: sysapprovedbadges: command not found

exit

Connection to seawulf.uri.edu closed.

15.3. Prepare for Next Class#

No separate preparation, focus on badge work and reviewing the notes if you missed class.

Make sure you did the prep for 10-31, we will work from that on 11/2.

15.4. Review today’s class#

  1. File permissions are represented numerically often in octal, by transforming the permissions for each level (owner, group, all) as binary into a number. Add octal.md to your KWL repo and answer the following. Try to think through it on your own, but you may look up the answers, as long as you link to (or ideally cite using jupyterbook citations) a high quality source.

    1. Describe how to transform the permissions [`r--`, `rw-`, `rwx`] to octal, by treating each set as three bits.
1. Transform the permission we changed our script to `rwxr-xr-x` to octal.
1. Which of the following would prevent a file from being edited by other people 777 or 755 and why?

15.5. More Practice#

  1. File permissions are represented numerically often in octal, by transforming the permissions for each level (owner, group, all) as binary into a number. Add octal.md to your KWL repo and answer the following. Try to think through it on your own, but you may look up the answers, as long as you link to (or ideally cite using jupyterbook citations) a high quality source.

    1. Describe how to transform the permissions [`r--`, `rw-`, `rwx`] to octal, by treating each set as three bits.
1. Transform the permission we changed our script to `rwxr-xr-x` to octal.
1. Which of the following would prevent a file from being edited by other people 777 or 755 and why?

  2. Answer the following in hpc.md of your KWL repo: (to think about how the design of the system we used in class impacts programming and connect it to other ideas taught in CS)

    1. What kinds of things would your code need to do if you were going to run it on an HPC system? 
1. What sbatch options seem the most helpful?
1. How might you go about setting the time limits for a script? How could you estimate how long a script will take?

15.6. Experience Report Evidence#

none

15.7. Questions After Today’s Class#

15.7.1. Will we be writing more intricate bash scripts down the line?#

only slightly in this class. In CSC412, you will learn more elaborate scripting, this course is to get you the basics.

15.7.2. why are files not executable by default?#

For safety, an executable could manipulate other files, you do not want to do that by accident.

15.7.3. When looking at the access permission, -rwxrwxrwx, how do you change just one of the levels and not all of them at once? like if i want to change perms for just the lowest level.#

This can be answered using the man file for chmod

15.7.4. When can I use this in practice?#

Whenever you want to execute a file, modifying the permissions is necessary

15.7.5. How can you change the permissions of a file using the chmod command in Bash?#

with chmod

15.7.6. Can bash scripts call other bash scripts?#

Yes!

15.7.7. Would having a file without the read permissions essentially make the file hidden to users?#

Yes

15.7.8. Is seawulf something we just have access to during this semester or will we have access to it after?#

technically, your access may not be revoked, but this server is designed only for teaching. That means that is relatively low-powered and has very minimal data storage.

If you are doing a side project, I recommend using a commerical server system like google cloud or aws.

If you are doing research with a professor, ask them for access to a server that URI has for research.

15.7.9. Do other servers have the ability to create scripts with languages besides Bash?#

Some may have other default shells, but even on a server that uses bash you can run other code. We will do that soon, we will write some C code and then run it from a bash script.

15.7.10. How else can we work in seawulf to improve our skills with CLI tools?#

The badges for thursday and this week’s lab will continue to build on these skills.