0 Python and JupyterLab

(CSE331) Python for Data Science

Author
Affiliation

Md Rasel Biswas

IASDS, University of Dhaka

1 Introduction to Python

Python is a general-purpose programming language widely used in:

  • data analysis;
  • statistical computing;
  • machine learning;
  • artificial intelligence;
  • web development;
  • automation; and
  • scientific research.

Python was created by Guido van Rossum and first released in 1991. It was initially developed at the Centrum Wiskunde & Informatica (CWI) in the Netherlands.

Python is popular because it has:

  • readable and relatively simple syntax;
  • a large collection of packages;
  • strong support for data science;
  • a large global community;
  • support for Windows, macOS, and Linux; and
  • free and open-source implementations.

The development of Python is supported by the Python Software Foundation (PSF) and a large community of contributors.

1.1 Your First Python Command

A traditional first program is:

print("Hello, Python!")
Hello, Python!

Python can also be used as a calculator:

2 + 3
5
10 * 5
50
(12 + 8) / 4
5.0
Python is case-sensitive

Python treats uppercase and lowercase letters as different characters.

For example, name, Name, and NAME are three different names.

2 Why Learn Python for Data Science?

Python provides tools for almost every stage of a data-science project:

  1. collecting or importing data;
  2. cleaning and transforming data;
  3. exploring data;
  4. visualizing data;
  5. fitting statistical or machine-learning models;
  6. evaluating models; and
  7. communicating or deploying results.

2.1 Important Data-Science Packages

Some packages that we will use in this course are:

Package Main purpose
NumPy Numerical arrays and mathematical operations
pandas Data manipulation using data frames
Matplotlib Basic data visualization
Seaborn Statistical data visualization
Plotly Interactive visualization
SciPy Scientific and numerical computing
statsmodels Statistical models and hypothesis tests
scikit-learn Machine-learning models and utilities

If you have used R, a pandas DataFrame is similar to an R data frame.

3 Python Across the Data-Science Workflow

In some programming workflows, analysts use one language for data exploration and another language for building the final application.

Python can often be used for both:

  • interactive analysis and experimentation;
  • statistical and machine-learning modelling;
  • automation;
  • application development; and
  • production systems.

When additional speed is required, Python packages can use optimized implementations written in languages such as C, C++, or Fortran.

Note

This is sometimes discussed as the two-language problem: one language is used for analysis and another for production. Python can reduce the need to switch between languages.

4 Installing Python

Python can be downloaded directly from:

https://www.python.org/downloads/

However, installing Python alone does not automatically install every package or interface needed for data science.

For this course, we will use Anaconda Distribution because it provides:

  • Python;
  • JupyterLab;
  • Anaconda Navigator;
  • the conda package manager; and
  • many commonly used data-science packages.

Download Anaconda Distribution from:

https://www.anaconda.com/download

4.1 Installation Steps

  1. Download the installer for your operating system.
  2. Run the installer.
  3. Follow the instructions shown on the screen.
  4. Complete the installation using the recommended options.
  5. Open Anaconda Navigator or Anaconda Prompt.

Choose a current version of Python 3. Python 2 is obsolete and should not be used for this course.

4.2 Verify the Installation

Open Anaconda Prompt, Terminal, or Command Prompt and enter:

python --version

Depending on the operating system, one of the following commands may be required:

python3 --version
py -3 --version

You can also check whether JupyterLab is installed:

jupyter lab --version

5 The Python Interpreter and REPL

Python provides an interactive environment in which commands can be entered and executed immediately.

From a terminal, run:

python

On some systems, use:

python3

You should see a prompt similar to:

>>>

This interactive environment is called the Python interpreter, Python shell, or Python REPL.

Python REPL

5.1 What Does REPL Mean?

REPL stands for Read–Evaluate–Print Loop.

  1. Read: Python reads the command.
  2. Evaluate: Python executes the command.
  3. Print: Python displays the result.
  4. Loop: Python waits for the next command.

For example:

>>> 5 + 7
12

To exit the REPL, enter:

exit()

You may also press Ctrl+D on macOS/Linux or Ctrl+Z, followed by Enter, on Windows.

6 Scripts, Consoles, and Notebooks

Python code can be written and executed in several ways.

6.1 Python Script

A Python script is a plain-text file containing Python code. Its filename usually ends with .py.

For example:

analysis.py

A script can contain:

name = "Rasel"
print("Hello,", name)

Popular editors and integrated development environments for Python scripts include:

  • VS Code;
  • PyCharm;
  • Spyder;
  • Thonny; and
  • IDLE.

6.2 Python Console

A console provides an interactive prompt in which commands are executed one at a time.

It is useful for:

  • quick calculations;
  • testing short commands; and
  • checking how a function works.

6.3 Jupyter Notebook

A Jupyter notebook is an interactive document that can contain:

  • Python code;
  • formatted text;
  • mathematical notation;
  • tables;
  • figures; and
  • output produced by the code.

A notebook file normally ends with:

.ipynb

For example:

lecture_01.ipynb

6.4 JupyterLab

JupyterLab is the browser-based interface that we will use to create and manage notebooks.

The distinction is important:

  • JupyterLab is the working interface.
  • A Jupyter notebook is a document opened inside that interface.

7 Launching JupyterLab

7.1 Using Anaconda Navigator

  1. Open Anaconda Navigator.
  2. Find JupyterLab.
  3. Click Launch.

7.2 Using a Terminal

Open Anaconda Prompt or a terminal and enter:

jupyter lab

JupyterLab should open automatically in your default web browser.

The terminal window must normally remain open while JupyterLab is running.

To stop JupyterLab:

  1. return to the terminal;
  2. press Ctrl+C; and
  3. confirm that you want to stop the server, if asked.
Course software

We will primarily use JupyterLab in this course. The classic command

jupyter notebook

opens a different notebook interface.

8 The JupyterLab Interface

JupyterLab interface

The JupyterLab interface usually contains:

  • a file browser on the left;
  • a main work area in the centre;
  • a menu bar;
  • open-document tabs;
  • a status bar; and
  • a launcher.

From the launcher, you can open a:

  • notebook;
  • console;
  • terminal;
  • text file;
  • Markdown file; or
  • Python script.

9 Creating a Notebook

To create a new notebook:

  1. open the JupyterLab launcher;
  2. locate the Notebook section;
  3. click the available Python kernel;
  4. wait for the notebook to open; and
  5. rename the notebook.

JupyterLab initially gives the notebook a name such as:

Untitled.ipynb

Creating a notebook

Rename it to something meaningful, such as:

lecture_01_practice.ipynb

To rename a notebook:

  • right-click its name in the file browser and select Rename; or
  • right-click the notebook tab and select Rename Notebook.

10 The Notebook Kernel

A notebook uses a kernel to execute code.

The kernel:

  • receives the code from a code cell;
  • executes the code;
  • stores variables and other objects in memory; and
  • returns the output to the notebook.

For this course, we will normally use a Python kernel.

The kernel remains active until it is:

  • interrupted;
  • restarted;
  • shut down; or
  • disconnected.

11 Notebook Cells

A Jupyter notebook is divided into rectangular sections called cells.

The two most important cell types are:

  1. Code cells
  2. Markdown cells

11.1 Code Cells

A code cell contains Python code.

For example:

x = 10
y = 5
x + y
15

The result appears directly below the cell.

A code cell may contain:

  • calculations;
  • variable definitions;
  • functions;
  • data analysis;
  • tables;
  • graphs; or
  • package imports.

11.2 Markdown Cells

A Markdown cell contains formatted text rather than Python code.

Markdown cells can be used for:

  • titles and headings;
  • explanations;
  • lists;
  • links;
  • images;
  • tables; and
  • mathematical notation.

For example:

# Main Heading

## Subheading

This is **bold text** and this is *italic text*.

- First item
- Second item

12 Running Notebook Cells

The most common keyboard shortcut is:

  • Shift+Enter: run the current cell and move to the next cell.

Other useful shortcuts are:

Shortcut Action
Shift+Enter Run the cell and move to the next cell
Ctrl+Enter Run the cell and remain in the same cell
Alt+Enter Run the cell and insert a new cell below
Enter Enter edit mode
Esc Enter command mode

In command mode, the following shortcuts are useful:

Shortcut Action
A Insert a cell above
B Insert a cell below
M Change the cell to Markdown
Y Change the cell to code
D, D Delete the selected cell
Z Undo cell deletion
Warning

Some browser or operating-system shortcuts may interfere with JupyterLab shortcuts. Commands are also available from the JupyterLab menus.

13 A First Notebook Example

Create a code cell and enter:

student_name = "Ayesha"
course = "Python for Data Science"

print("Student:", student_name)
print("Course:", course)
Student: Ayesha
Course: Python for Data Science

Now perform a calculation:

quiz_1 = 8
quiz_2 = 9
assignment = 10

total = quiz_1 + quiz_2 + assignment
total
27

A notebook displays the result of the final expression automatically.

Compare:

total
27

with:

print(total)
27

Both display the value, but print() explicitly asks Python to display it.

14 Writing Mathematics in Markdown

Jupyter Markdown cells support mathematical notation written using LaTeX syntax.

Inline mathematics is placed between single dollar signs:

The sample mean is denoted by $\bar{x}$.

It appears as:

The sample mean is denoted by \(\bar{x}\).

Display mathematics is placed between double dollar signs:

$$
\bar{x} = \frac{1}{n}\sum_{i=1}^{n}x_i
$$

It appears as:

\[ \bar{x} = \frac{1}{n}\sum_{i=1}^{n}x_i \]

15 Execution Order Matters

Notebook cells do not necessarily run from top to bottom automatically.

Suppose the following cell is run first:

result = value * 2

Python will produce an error if value has not already been defined.

The required cell must be run first:

value = 10

After that, the calculation will work:

result = value * 2
result

The number beside a code cell, such as [1], [2], or [3], indicates its execution order.

A reproducible notebook

A well-organized notebook should work when its cells are run from the first cell to the last cell in order.

Before submitting a notebook, use:

Kernel → Restart Kernel and Run All Cells

Then check whether every cell runs without an error.

16 Interrupting and Restarting the Kernel

A cell may sometimes take too long to finish.

To stop the current calculation, use:

Kernel → Interrupt Kernel

If the notebook behaves unexpectedly, use:

Kernel → Restart Kernel

Restarting the kernel:

  • clears variables from memory;
  • removes imported packages from the current session; and
  • gives the notebook a fresh Python session.

After restarting, the cells must be run again.

17 Saving a Notebook

JupyterLab usually saves notebooks automatically, but it is good practice to save your work manually.

Use:

  • Ctrl+S on Windows/Linux; or
  • Command+S on macOS.

Good notebook filenames include:

lecture_01_practice.ipynb
assignment_01.ipynb
household_data_analysis.ipynb

Avoid filenames such as:

new final latest notebook 2.ipynb

Prefer:

  • meaningful names;
  • lowercase letters;
  • underscores instead of spaces; and
  • a consistent naming system.

18 Working Directory

The working directory is the folder Python currently treats as its main location for reading and writing files.

For example, when Python reads:

data.csv

it normally searches for that file in the current working directory.

18.1 Checking the Working Directory

Use the os module:

import os

os.getcwd()

The result may look like:

C:\Users\Student\Documents\python_course

or:

/Users/student/Documents/python_course

18.2 Listing Files

To list files and folders in the current directory:

import os

os.listdir()

18.3 Changing the Working Directory

On Windows:

import os

os.chdir(r"C:\Users\Student\Documents\python_course")

On macOS or Linux:

import os

os.chdir("/Users/student/Documents/python_course")

The r before a Windows path creates a raw string, which helps Python interpret backslashes correctly.

Tip

A better habit is to create a separate folder for each project and launch JupyterLab from that folder. This reduces the need to change the working directory repeatedly.

19 Jupyter Magic Commands

IPython and Jupyter provide special commands called magic commands.

These commands usually begin with %.

%pwd

The %pwd command displays the working directory.

%ls

The %ls command lists files in the current directory.

%cd /path/to/folder

The %cd command changes the working directory.

Note

Magic commands are features of IPython and Jupyter. They are not part of standard Python and may not work inside an ordinary .py script.

20 Python Packages and Modules

A module is a file containing reusable Python code.

A package is a collection of related modules.

Packages allow us to use functions and tools developed by other programmers without writing everything ourselves.

Community-contributed Python packages are commonly distributed through the Python Package Index, or PyPI.

https://pypi.org/

21 Importing Packages

A package must usually be imported before it can be used.

import math

We can then use functions from that package:

math.sqrt(25)
5.0

Data-science packages are often imported using standard abbreviations:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Here:

  • np represents NumPy;
  • pd represents pandas;
  • plt represents matplotlib.pyplot; and
  • sns represents Seaborn.

These abbreviations are conventions rather than requirements.

22 Installing Packages

A package must be installed only once in a particular Python environment. However, it must be imported again whenever a new Python session begins.

22.1 Installing from a Notebook

Inside Jupyter, use:

%pip install package_name

For example:

%pip install emoji

The %pip command is generally preferable to writing only pip inside a notebook because it installs the package into the Python environment associated with the active kernel.

22.2 Installing from a Terminal

Using pip:

python -m pip install package_name

On some systems:

python3 -m pip install package_name

Using conda from Anaconda Prompt:

conda install package_name

After installing a package, you may sometimes need to restart the notebook kernel.

23 A Fun Package Test

Install the emoji package:

%pip install emoji

Import and use it:

import emoji

message = emoji.emojize(
    "Python is :thumbs_up:",
    language="alias"
)

print(message)

Expected result:

Python is 👍

24 Understanding Errors

Errors are a normal part of programming. An error message provides information about what went wrong.

For example:

print(student_age)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 print(student_age)

NameError: name 'student_age' is not defined

Because student_age has not been defined, Python returns a NameError.

Read an error message from the bottom upward. The final line normally gives:

  • the type of error; and
  • a short explanation.

Common errors during the first few lectures include:

Error Possible reason
NameError A variable has not been defined
SyntaxError The code does not follow Python syntax
ModuleNotFoundError A required package is not installed
FileNotFoundError Python cannot find the requested file
TypeError An operation was applied to an unsuitable data type

Do not be afraid of error messages. Learning to understand and correct errors is an essential programming skill.

25 Good Notebook Practices

When working with notebooks:

  • give the notebook a meaningful filename;
  • use Markdown headings to organize the analysis;
  • explain the purpose of important code cells;
  • run cells in a logical order;
  • avoid keeping unnecessary outputs;
  • save your work regularly;
  • use clear variable names;
  • keep data and notebooks in an organized project folder; and
  • restart the kernel and run all cells before submitting the notebook.

26 Lecture Summary

In this lecture, we learned that:

  • Python is a general-purpose language widely used in data science.
  • JupyterLab is an interface for working with notebooks and other files.
  • A notebook contains code cells and Markdown cells.
  • A kernel executes code and stores objects in memory.
  • Notebook execution order matters.
  • The working directory determines where Python searches for files.
  • Packages provide additional functionality.
  • %pip can be used to install packages from a notebook.
  • Errors are a normal and useful part of programming.
  • A notebook should run correctly from beginning to end.

27 Before the Next Lecture

Before the next class, make sure that you can:

  • open JupyterLab;
  • create and rename a notebook;
  • create code and Markdown cells;
  • run and save cells;
  • restart the kernel and run all cells;
  • locate your working directory; and
  • import a Python package.