workflow-basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(workflow-basics)=\n",
    "# Workflow Basics\n",
    "\n",
    "This chapter will take you through some of the essential parts of a Python workflow.\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "You'll need an installation of Python and Visual Studio Code with the Python extensions to get to grips with this chapter. If you haven't installed those yet, head back to {ref}`code-preliminaries` and follow the instructions there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Python scripts and the interactive window\n",
    "\n",
    "As a reminder, the figure below shows the typical layout of Visual Studio Code.\n",
    "\n",
    "![A typical user view in Visual Studio Code](https://github.com/aeturrell/coding-for-economists/blob/main/img/vscode_layout.png?raw=true)\n",
    "\n",
    "When you create a new script (File->New File->Save as 'your_script_name.py), it will appear in the part of the screen labelled as 3.\n",
    "\n",
    "To run a script, select the code you want to run, right click, and select \"Run Selection/Line in Interactive Window\". You can also hit shift + enter if you set this shortcut up; if you haven't it's well worth doing and you can find the instructions in {ref}`code-preliminaries`.\n",
    "\n",
    "Using the \"Run Selection/Line in Interactive Window\" option or using the shortcut will cause panel 5 in the above diagram (the interactive window) to appear, where you will see the code run and the outputs of your script appear.\n",
    "\n",
    "```{tip}\n",
    "If you have an issue getting the code to run in the interactive window, first check the instructions in {ref}`code-preliminaries`. If you're still having issues, it may be that Visual Studio Code isn't sure which Python to run, or where Python is on your system. To fix the latter problem, hit the \"Select kernel\" button in the top right-hand side of the interactive window.\n",
    "```\n",
    "\n",
    "When you are first writing a script, it's useful to be able to move back and forth between the script and the interactive window. You might execute a line of code (put the cursor on the relevant line and hit shift and enter) in the interactive window, then manually write out some code in the interactive window's execution box (seen at the bottom of panel 5 saying \"Type code here...\"), and then explore some of the variables you've created with the variable explorer (using the button \"Variables\") at the top of the interactive window.\n",
    "\n",
    "But, once you've honed the code in your script, it's good to make the script a complete analytical process that you are happy running end-to-end and that—for production or 'final' work—you would use the \"Run Current File in Interactive Window\" option to run all the way through. This is good practice because *what is in your script is reproducible* but what you've entered manually in the interactive window is not. And you want the outputs from your code to be reproducible and understandable by others (including future you!), but this is hard if there are undocumented extra lines of code that you only did on the fly via the interactive window's execution box."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0154c032",
   "metadata": {},
   "source": [
    "## Using installed packages and modules\n",
    "\n",
    "We already saw how to install packages in {ref}`code-preliminaries`. If you forgot, look back at how to do this now. In short, packages are installed using the command line or, on Windows, the Anaconda prompt. With either of these open, type `conda install packagename` and hit enter to both search for and install the package you need.\n",
    "\n",
    "What about using a package that you've installed? That's what we'll look at now.\n",
    "\n",
    "Let's see an example of using the powerful numerical library **numpy**. There are different ways to import packages to use within a script or notebook; you can import the entire package in one go or just import the functions you need (if you know their names). When an entire package is imported, you can give it any name you like and the convention for **numpy** is to import it as the shortened 'np'. All of the functions and methods of the package can be accessed by typing `np` followed by `.` and then typing the function name. This convention of importing packages with a given name makes your code easier to read, because you know exactly which package is doing what, and avoids any conflicts when functions from different packages have the same name.\n",
    "\n",
    "As well as demonstrating importing the whole package for **numpy**, the example below shows importing just one specific function from **numpy**, `inv`, which does matrix inversion. Note that because `inv` was imported separately it can be used without an `np` prefix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f86893cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.linalg import inv\n",
    "\n",
    "matrix = np.array([[4.0, 2.0, 4.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]])\n",
    "\n",
    "print(\"Matrix:\")\n",
    "print(matrix)\n",
    "\n",
    "inv_mat = inv(matrix)\n",
    "print(\"Inverse:\")\n",
    "print(inv_mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb7ac525",
   "metadata": {},
   "source": [
    "We could have imported all of **numpy** and it used it without extension using `from numpy import *` but this is considered bad practice as it fills our 'namespace' with function names that might clash with other packages and it's less easy to read because you don't know which function came from which package (one of Python's mantras is \"explicit is better than implict\"). However, some packages are *designed* to be used like this, so, for example, you will see `from lets_plot import *` in this book.\n",
    "\n",
    "```{note}\n",
    "If you want to check what packages you have installed in your Python environment, run `conda list` on your computer's command line (aka the *terminal* or *command prompt*).\n",
    "```\n",
    "\n",
    "Sometimes you might forget what a function you have imported does! Or at least, you might not be sure what all of the optional arguments are. In Visual Studio Code, you can just hover your cursor over the name of the function and a box will come up that tells you everything you need to know about it. This box is auto-generated by doc-strings; information that is written in text just under a function's definition (`def` statement).\n",
    "\n",
    "An alternative way to see what a function does is to use a wonderful package called [**rich**](https://github.com/willmcgugan/rich) that does many things including providing an `inspect()` function. You will need to use `pip` to install rich by running `pip install rich` on the command line. Here's an example of using **rich**'s inpsect method on the `inv()` function we imported above (`methods=True` reports all of the functionality of `inv()`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fd14fd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from rich import inspect\n",
    "\n",
    "inspect(inv, help=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "653880d8",
   "metadata": {},
   "source": [
    "```{admonition} Exercise\n",
    "Write a code block that imports the **numpy** function `numpy.linalg.det()` as `det()`. Run `inspect()` on it. Find the determinant of `[[4, 3], [1, 7]]`.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d912c5",
   "metadata": {},
   "source": [
    "### Modules\n",
    "\n",
    "Sometimes, you will want to call in some code from a different script that *you* wrote (rather than from a package provided by someone else). Imagine you have several scripts with code in, a, b, and c, all of which need to use the same underlying function that you have written. What do you do? (Note that \"script with code in\" is just a text file that has a `.py` extension and contains code.)\n",
    "\n",
    "A central tenet of good coding is that you *do not repeat yourself*. Therefore, a bad solution to this problem would be to copy and paste the same code into all three of the scripts. A *good* solution is to write the code that's need just once in a separate 'utility' script and have the other scripts import that one function. This also adheres to another important programming principle: that of writing modular code.\n",
    "\n",
    "This schematic shows the kind of situation we're talking about:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0efeaee9",
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib_inline.backend_inline\n",
    "\n",
    "# Plot settings\n",
    "plt.style.use(\n",
    "    \"https://github.com/aeturrell/coding-for-economists/raw/main/plot_style.txt\"\n",
    ")\n",
    "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")\n",
    "\n",
    "\n",
    "graph = nx.DiGraph()\n",
    "graph.add_edges_from(\n",
    "    [\n",
    "        (\"Utility script\", \"code file a\"),\n",
    "        (\"Utility script\", \"code file b\"),\n",
    "        (\"code file a\", \"code file c\"),\n",
    "        (\"code file b\", \"code file c\"),\n",
    "        (\"Utility script\", \"code file c\"),\n",
    "    ]\n",
    ")\n",
    "colour_node = \"#AFCBFF\"\n",
    "fixed_pos = nx.spring_layout(graph, seed=100)\n",
    "nx.draw(graph, pos=fixed_pos, with_labels=True, node_size=6000, node_color=colour_node)\n",
    "extent = 1.4\n",
    "plt.xlim(-extent, extent)\n",
    "plt.ylim(-extent, extent)\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5df7665d",
   "metadata": {},
   "source": [
    "How can we give code files a, b, and c access to the functions etc in the \"Utility script\"? We would define a file 'utilities.py' that had the following function in that we would like to use in the other code files:\n",
    "\n",
    "```python\n",
    "# Contents of utilities.py file\n",
    "def really_useful_func(number):\n",
    "    return number*10\n",
    "```\n",
    "\n",
    "Then, in 'code_script_a.py', we would write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2e3422c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import utilities as utils\n",
    "\n",
    "print(utils.really_useful_func(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb775643",
   "metadata": {},
   "source": [
    "An alternative is to *just* import the function we want, with the name we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb14c455",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utilities import really_useful_func as ru_fn\n",
    "\n",
    "print(ru_fn(30))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15365d79",
   "metadata": {},
   "source": [
    "Another important example is the case where you want to run 'utilities.py' as a standalone script, but still want to borrow functions from it to run in other scripts. There's a way to do this. Let's change utilities.py to\n",
    "\n",
    "```python\n",
    "# Contents of utilities.py file\n",
    "def really_useful_func(number):\n",
    "    return number*10\n",
    "\n",
    "\n",
    "def default_func():\n",
    "    print('Script has run')\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    default_func()\n",
    "```\n",
    "\n",
    "What this says is that if we call 'utilities.py' from the command line, eg\n",
    "\n",
    "```bash\n",
    "python utilities.py\n",
    "```\n",
    "\n",
    "It will return `Script has run` because, by executing the script alone, we are asking for anything in the `main` block defined at the end of the file to be run. But we can still import anything from utilities into other scripts as before--and in that case it is not the main script, but an import, and so the `main` block will *not* be executed by default.\n",
    "\n",
    "You can important several functions at once from a module (aka another script file) like this:\n",
    "\n",
    "```python\n",
    "from utilities import really_useful_func, default_func\n",
    "```\n",
    "\n",
    "```{admonition} Exercise\n",
    "Write your own `utilities.py` that has a `super_useful_func` that accepts a number and returns the number divided by 10. In another script, `main.py`, try a) importing all of utilities and running `super_useful_func` on a number and, b), importing just `super_useful_func` from utilities and running it on a number.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45bfe197",
   "metadata": {},
   "source": [
    "## Reading and writing files\n",
    "\n",
    "Although most applications in economics will use the **pandas** package to read and write tabular data, it's sometimes useful to know how to read and write arbitrary files using the built-in Python libraries too. To open a file\n",
    "\n",
    "```python\n",
    "open('filename', mode)\n",
    "```\n",
    "\n",
    "where `mode` could be `r` for read, `a` for append, `w` for write, and `x` to create a file. Create a file called `text_example.txt` and write a single line in it, 'hello world'. To open the file and print the text, use:\n",
    "\n",
    "```python\n",
    "with open('text_example.txt') as f:\n",
    "    text_in = f.read()\n",
    "\n",
    "print(text_in)\n",
    "```\n",
    "\n",
    "```python\n",
    "'hello world!\\n'\n",
    "```\n",
    "\n",
    "`\\n` is the new line character. Now let's try adding a line to the file:\n",
    "\n",
    "```python\n",
    "with open('text_example.txt', 'a') as f:\n",
    "    f.write('this is another line\\n')\n",
    "```\n",
    "\n",
    "Writing and reading files using the `with` command is a quick and convenient shorthand for the less concise open, action, close pattern. For example, the above example can also be written as:\n",
    "\n",
    "```python\n",
    "f = open('text_example.txt', 'a')\n",
    "f.write('this is another line\\n')\n",
    "f.close()\n",
    "```\n",
    "\n",
    "Although this short example shows opening and writing a text file, this approach can be used to edit a wide range of file extensions including .json, .xml, .csv, .tsv, and many more, including binary files in addition to plain text files."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.12 ('codeforecon')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "c4570b151692b3082981c89d172815ada9960dee4eb0bedb37dc10c95601d3bd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}