7. How do programmers communicate about code?#
Today we are going to pick up from where we left off talking about the conventional commits.
That is a core example of the types of detailed communication we do in programming that is embedded into the work.
7.1. Closing your Codespace#
Use the codespace page to see all of yours.
Check that your github-inclass repo has no ucnommited changes. If it does you need to commit them, there are two strategies:
start the codespace, commit and push
from that page, export the changes to a branch, create a PR and merge it if needed
7.2. Build and Explore Ideas#
I am going to put ideas in the discussions repo
I violated the instructions a little, but I thought it made the most sense.
7.3. Why Documentation#
Today we will talk about documentation, there are several reasons this is important:
using official documentation is the best way to get better at the tools
understanding how documentation is designed and built will help you use it better
writing and maintaining documentation is really important part of working on a team
documentation building tools are a type of developer tool (and these are generally good software design)
Design is best learned from examples. Some of the best examples of software design come from developer tools.
In particular documentation tools are really good examples of:
pattern matching
modularity
automation
building
By the end of today’s class you will be able to:
describe different types of documentation
find different information in a code repo
generate documentation as html
ignore content from a repo
create a repo locally and push to GitHub
7.4. What is documentation#
Note
We opened this page and discussed the different types of documentation in the table in class
from ethnography of documentation data science
Data Science is only one type of programing, but this is still pretty representative. The main difference is that in this case the “target” users are often data scientists and more technically skilled than the “target” users of, for example, a social media app. This would mean that user-facing docs would vary.
7.5. Why is documentation so important?#
we should probably spend more time on it
7.6. What is a docstring#
python
Python 3.11.4 (main, Jul 5 2023, 09:00:44) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> help(print)
>>> def ex():
... ''' doc string
... '''
... padd
...
>>> help(ex)
>>>
>>> # a comment
>>> exit()
7.7. So, how do we do it?#
linux kernel uses sphinx and here is why and how it works
7.8. Install Jupyter-book#
Jupyterbook wraps sphinx and uses markdown instead of restructured text. The project authors note in the documenation that it “can be thought of as an opinionated distribution of Sphinx”. We’re going to use this, so that we can use most of the power of sphinx without having to learn ReStructured Text which is more complex than markdown syntax.
install jupyterbook on Mac or linux those instructions will work on your regular terminal, if you have python installed. On Windows those instructions will work in the Anaconda prompt or any other terminal that is set up with python. If these steps do not make sense see the recommendations in the syllabus for more instructions including videos of the Python install process in both Mac and Windows.
pip install jupyter-book
Note
this command does a lot of stuff, it installs jupyter-book and all of its dependencies, which is a lot. I removed the output from the notes
the thing that matters is that near the end it will say “successfully installed”
Now we move to our inclass repo:
cd Documents/inclass/systems/
ls
github-inclass-fa23-brownsarahm
And note that we can see the other folders we have made so far, but note that we are not inside any of them.
We can create this
jupyter-book create tiny-book
===============================================================================
Your book template can be found at
tiny-book/
===============================================================================
and it tells us the relative path to the new folder, we can use ls again to see that it is there.
ls
github-inclass-fa23-brownsarahm tiny-book
7.9. Parts of a Jupyer book#
First we go in the directory:
cd tiny-book/
Then look
ls
_config.yml logo.png notebooks.ipynb
_toc.yml markdown-notebooks.md references.bib
intro.md markdown.md requirements.txt
7.9.1. starting a git repo locally#
Before we work with the files, let’s commit the current version
git status
fatal: not a git repository (or any of the parent directories): .git
We cannot check status because we are not in a git repo.
Recall a git repo is a folder that contains a .git directory with the versions of our project in it. This directory is typically hidden, so to confirm it is not there we need the -a option on ls
ls -a
. intro.md notebooks.ipynb
.. logo.png references.bib
_config.yml markdown-notebooks.md requirements.txt
_toc.yml markdown.md
We can create a new repo with git init and the path to wher we want it. We want it at our current working directory, which is .
git init .
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /Users/brownsarahm/Documents/inclass/systems/tiny-book/.git/
Here we are faced with a social aspect of computing that is also a good reminder about how git actually works
7.9.2. Retiring racist language#
Historically the default branch was called master.
derived from a master/slave analogy which is not even how git works, but was adopted terminology from other projects
literally the person who chose the names “master” and “origin” regrets that choice the name main is a more accurate and not harmful term and the current convention.
we’ll change our default branch to main
git branch -m main
Now, we can look at the directory to see how git init changed the folder.
ls -a
. _toc.yml markdown.md
.. intro.md notebooks.ipynb
.git logo.png references.bib
_config.yml markdown-notebooks.md requirements.txt
Now we have the .git directory!
and
git status
On branch main
No commits yet
Untracked files:
(use "git add <file>..." to include in what will be committed)
_config.yml
_toc.yml
intro.md
logo.png
markdown-notebooks.md
markdown.md
notebooks.ipynb
references.bib
requirements.txt
nothing added to commit but untracked files present (use "git add" to track)
it works!
and we will commit the template.
git add .
git commit -m 'init jupyterbook'
[main (root-commit) 3fcd849] init jupyterbook
9 files changed, 341 insertions(+)
create mode 100644 _config.yml
create mode 100644 _toc.yml
create mode 100644 intro.md
create mode 100644 logo.png
create mode 100644 markdown-notebooks.md
create mode 100644 markdown.md
create mode 100644 notebooks.ipynb
create mode 100644 references.bib
create mode 100644 requirements.txt
7.9.3. Structure of a Jupyter book#
ls
_config.yml logo.png notebooks.ipynb
_toc.yml markdown-notebooks.md references.bib
intro.md markdown.md requirements.txt
A jupyter book has two required files (_config.yml and _toc.yml), some for content, and some helpers that are common but not required.
the
*.mdfiles are contentthe
.bibfile is bibiolography informationThe other files are optional, but common. Requirements.txt is the format for pip to install python depndencies. There are different standards in other languages for how
the extention (.yml) is yaml, which stands for “YAML Ain’t Markup Language”. It consists of key, value pairs and is deigned to be a human-friendly way to encode data for use in any programming language.
7.9.4. Dev tools mean we do not have to write bibliographies manually#
bibliographies are generated with bibtex which takes structured information from the references in a bibtex file with help from sphinxcontrib-bibtex
For general reference, reference managers like zotero and mendeley can track all of your sources and output the references in bibtex format that you can use anywhere or sync with tools like MS Word or Google Docs.
7.9.5. Configuration options#
The config file template is pretty complete, including the URL of the webpage with all of the detailed setting options.
The configuration file, tells jupyter-book basic iformation about the book, it provides all of the settings that jupyterbook and sphinx need to render the content as whatever output format we want.
cat _config.yml
# Book settings
# Learn more at https://jupyterbook.org/customize/config.html
title: My sample book
author: The Jupyter Book Community
logo: logo.png
# Force re-execution of notebooks on each build.
# See https://jupyterbook.org/content/execute.html
execute:
execute_notebooks: force
# Define the name of the latex output file for PDF builds
latex:
latex_documents:
targetname: book.tex
# Add a bibtex file so that we can create citations
bibtex_bibfiles:
- references.bib
# Information about where the book exists on the web
repository:
url: https://github.com/executablebooks/jupyter-book # Online location of your book
path_to_book: docs # Optional path to your book, relative to the repository root
branch: master # Which branch of the repository should be used when creating links (optional)
# Add GitHub buttons to your book
# See https://jupyterbook.org/customize/config.html#add-a-link-to-your-repository
html:
use_issues_button: true
use_repository_button: true
7.9.6. Content Organization#
The table of contents file describe how to put the other files in order.
cat _toc.yml
# Table of contents
# Learn more at https://jupyterbook.org/customize/toc.html
format: jb-book
root: intro
chapters:
- file: markdown
- file: notebooks
- file: markdown-notebooks
7.9.7. Building a book#
We can transform from raw source to an output by building the book
Building is transforming from input to output format. In this case from markdown, yaml, and notebook files into HTML.
We will see later that building is also what we refer to the whole process from source code to executable.
jupyter-book build .
Running Jupyter-Book v0.15.1
Source Folder: /Users/brownsarahm/Documents/inclass/systems/tiny-book
Config Path: /Users/brownsarahm/Documents/inclass/systems/tiny-book/_config.yml
Output Path: /Users/brownsarahm/Documents/inclass/systems/tiny-book/_build/html
Running Sphinx v4.5.0
making output directory... done
[etoc] Changing master_doc to 'intro'
checking bibtex cache... out of date
parsing bibtex file /Users/brownsarahm/Documents/inclass/systems/tiny-book/references.bib... parsed 5 entries
myst v0.18.1: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=['colon_fence', 'dollarmath', 'linkify', 'substitution', 'tasklist'], disable_syntax=[], all_links_external=False, url_schemes=['mailto', 'http', 'https'], ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, footnote_transition=True, words_per_minute=200, sub_delimiters=('{', '}'), linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area')
myst-nb v0.17.2: NbParserConfig(custom_formats={}, metadata_key='mystnb', cell_metadata_key='mystnb', kernel_rgx_aliases={}, execution_mode='force', execution_cache_path='', execution_excludepatterns=[], execution_timeout=30, execution_in_temp=False, execution_allow_errors=False, execution_raise_on_error=False, execution_show_tb=False, merge_streams=False, render_plugin='default', remove_code_source=False, remove_code_outputs=False, code_prompt_show='Show code cell {type}', code_prompt_hide='Hide code cell {type}', number_source_lines=False, output_stderr='show', render_text_lexer='myst-ansi', render_error_lexer='ipythontb', render_image_options={}, render_figure_options={}, render_markdown_format='commonmark', output_folder='build', append_css=True, metadata_to_fm=False)
Using jupyter-cache at: /Users/brownsarahm/Documents/inclass/systems/tiny-book/_build/.jupyter_cache
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 4 source files that are out of date
updating environment: [new config] 4 added, 0 changed, 0 removed
/Users/brownsarahm/Documents/inclass/systems/tiny-book/markdown-notebooks.md: Executing notebook using local CWD [mystnb]
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
/Users/brownsarahm/Documents/inclass/systems/tiny-book/markdown-notebooks.md: Executed notebook in 1.89 seconds [mystnb]
/Users/brownsarahm/Documents/inclass/systems/tiny-book/notebooks.ipynb: Executing notebook using local CWD [mystnb]
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
/Users/brownsarahm/Documents/inclass/systems/tiny-book/notebooks.ipynb: Executed notebook in 2.56 seconds [mystnb]
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] notebooks
generating indices... genindex done
writing additional pages... search done
copying images... [100%] _build/jupyter_execute/137405a2a8521f521f06724f6d604e5a5544cce7bd94d903975cee58b0605ccb.png
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
The HTML pages are in _build/html.
[etoc] missing index.html written as redirect to 'intro.html'
===============================================================================
Finished generating HTML for book.
Your book's HTML pages are here:
_build/html/
You can look at your book by opening this file in a browser:
_build/html/index.html
Or paste this line directly into your browser bar:
file:///Users/brownsarahm/Documents/inclass/systems/tiny-book/_build/html/index.html
===============================================================================
Try it yourself
Which files created by the template are not included in the rendered output? How could you tell?
ls
_build logo.png references.bib
_config.yml markdown-notebooks.md requirements.txt
_toc.yml markdown.md
intro.md notebooks.ipynb
we note that this made a new folder called _build. we can look inside there.
ls _build/
html jupyter_execute
and in the html folder:
ls _build/html/
_images index.html objects.inv
_sources intro.html search.html
_sphinx_design_static markdown-notebooks.html searchindex.js
_static markdown.html
genindex.html notebooks.html
We got HTML, CSS, and javascript files without having to write them manually.
We can also copy the path to the file and open it in our browser
we can change the size of a browswer window or use the screen size settings in inspect mode to see that this site is responsive.
We didn’t have to write any html and we got a responsive site!
If you wanted to change the styling with sphinx you can use built in
themes which tell sphinx to put different
files in the _static folder when it builds your site, but you don’t have to change any of your content! If you like working on front end things (which is great! it’s just not alwasy the goal) you can even
build your own theme that can work with sphinx.
7.10. Ignoring Built files#
The built site files are compeltey redundant, content wise, to the original markdown files.
We do not want to keep track of changes for the built files since they are generated from the source files. It’s redundant and makes it less clear where someone should update content.
git status
On branch main
Untracked files:
(use "git add <file>..." to include in what will be committed)
_build/
nothing added to commit but untracked files present (use "git add" to track)
ls
_build logo.png references.bib
_config.yml markdown-notebooks.md requirements.txt
_toc.yml markdown.md
intro.md notebooks.ipynb
Git helps us with this with the .gitignore
echo "_build/" >> .gitignore
git status
On branch main
Untracked files:
(use "git add <file>..." to include in what will be committed)
.gitignore
nothing added to commit but untracked files present (use "git add" to track)
now that’s the only new file as far as git is concerned, so we will track this,
git add .
git commit -m 'ignores build'
[main 9c4157f] ignores build
1 file changed, 1 insertion(+)
create mode 100644 .gitignore
We can look at the file’s contents
cat .gitignore
_build/
What it does is, when git goes to add, it skips any specific files that match both the git add <path> AND any of the paths listed in .gitignore
7.11. How do I push a repo that I made locally to GitHub?#
Right now, we do not have any remotes
git remote
If we do the same in another repo that we got by cloing from GitHub, they all have remotes.
cd ../github-inclass-fa23-brownsarahm/
git remote
origin
Typically they all use the default name origin
Back to our tiny-book
cd ../tiny-book/
For today, we will create an empty github repo shared with me using the link shared on prismia.
More generally, you can create a repo
That default page for an empty repo if you do not initiate it with any files will give you the instructions for what remote to add.
Now we add the remote, following the instructions from the blank repo page.
git remote add origin https://github.com/introcompsys/f23-tiny-book-brownsarahm.git
git branch -M main
and finally we can push!
git push -u origin main
Enumerating objects: 14, done.
Counting objects: 100% (14/14), done.
Delta compression using up to 8 threads
Compressing objects: 100% (12/12), done.
Writing objects: 100% (14/14), 16.45 KiB | 8.23 MiB/s, done.
Total 14 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), done.
To https://github.com/introcompsys/f23-tiny-book-brownsarahm.git
* [new branch] main -> main
branch 'main' set up to track 'origin/main'.
git remote
origin
7.12. Prepare for Next Class#
Follow up on your grade plan PR and self reflection that you complete in lab
7.13. Review today’s class#
Review the notes, jupyterbook docs, and experiment with the
jupyter-bookCLI to determine what files are required to makejupyter-book buildrun. Make your kwl repo into a jupyter book. Set it so that the_builddirectory is not under version control.Add
docs.mdto your KWL repo and explain the most important things to know about documentation in your own words using other programming concepts you have learned so far. Include in a markdown (same as HTML<!-- comment -->) comment the list of CSC courses you have taken for context while we give you feedback.
7.14. More Practice#
Review the notes, jupyterbook docs, and experiment with the
jupyter-bookCLI to determine what files are required to makejupyter-book buildrun. Make your kwl repo into a jupyter book. Set it so that the_builddirectory is not under version control.Learn about the documentation ecosystem in another language that you know using at least one official source and additional sources as you find helpful. In
docs.mdinclude a summary of your findings and compare and contrast it to jupyter book/sphinx. Include a bibtex based bibliography of the sources you used. You can use this generator for informal sources and google scholar for formal sources.
7.15. Experience Report Evidence#
Tiny book repo should exist. Append your terminal output or this repo’s git log to your experience report.
7.16. Questions After Today’s Class#
7.16.1. Why would one experience missing .exe files within their python library, even after reinstalling python?#
it is probably a missing path
7.16.2. If I want to track the _build file will it automatically be tracked now or do I have to remove it from gitignore?#
To track the files in the build directory you would need to delete it from the .gitignore file.
Also, of note, git does not track folders, it tracks files, but listing folder paths can help
7.16.3. Will jupyter book look for a universal syntax for documentation or will it look for a language dependent syntax?#
Default it works with Python, but the sphinx docs describe how to work with other languages. The jupyter book docs include a page on how sphinx and jupyter-book are related and a page on how to use other sphinx features in a jupyter book.
Jupyter notebooks also have alternative kernels so notebook styled pages can be written in these languages as well.
7.16.4. Can you create an entire repository with only git?#
By definition git is what makes a git repository.
7.16.5. What is jupyter used for#
Jupyter is a large software project designed to support computational sciences.
it includes the notebook file format specification, the lab interface for working with them, the hub for hosting it on a server.
Jupyter-book is a separate, but related, project that is highly compatible.
7.16.8. Is it possible to edit the Jupyter Book website on a web front site or does it have to be in the files?#
You have to edit the source files not the built ones to keep the changes.
You can however, add custom HTML and/or CSS.
7.16.9. Are their alternatives to Jupyterbook that have similar functionality?#
Yes, this is the practice badge.
7.16.10. If I want to track the _build file will it automatically be tracked now or do I have to remove it from gitignore?#
It has to not be in .gitignore to be tracked.
7.16.11. Will jupyter book look for a universal syntax for documentation or will it look for a language dependent syntax?#
By default it looks for Python, but you can configure it to work with other languages.
Important
This is one thing you could do for a build badge.
7.17. Good Questions for Explore Badges#
7.17.1. What other platforms/libraries are used to generate similar webpages?#
Learning about the landscape of static site generators is a great topic.