Prerequisites
I. Python Fundamentals
1. List Comprehensions and Conditional Assignment
When generating a sequence of numbers in Python, you can write it as follows:
| |
[0, 2, 4, 6, 8]
List comprehensions can actually simplify this notation: [* for i in *]. Here the first * is the mapping function, whose input is whatever i refers to in the latter part, whereas the second * denotes the object being iterated over.
| |
[0, 2, 4, 6, 8]
List comprehensions also support multiple levels of nesting. In the example below, the first for is the outer loop and the second is the inner loop:
| |
['a_c', 'a_d', 'b_c', 'b_d']
Besides list comprehensions, another handy piece of syntactic sugar is conditional assignment with an if clause, which takes the form value = a if condition else b:
| |
'cat'
This is equivalent to the following:
| |
Here is an example that caps the elements of a list at 5: any value greater than 5 is replaced by 5, while values below 5 keep their original value.
| |
[1, 2, 3, 4, 5, 5, 5]
2. Anonymous Functions and the map Method
Some functions have a clear and simple mapping relationship — such as the my_func above — and in such cases an anonymous function expresses them concisely:
| |
6
| |
3
The usage above, however, actually contradicts the very idea of being “anonymous”. In practice it tends to appear where a function need not be called in several places — as in the list comprehension example above, where the user cares little about the function’s name and only about the mapping itself:
| |
[0, 2, 4, 6, 8]
To handle this kind of anonymous-function mapping inside a list comprehension, Python provides the map function. It returns a map object, which you must convert to a list with list:
| |
[0, 2, 4, 6, 8]
When a function maps multiple input values, you can achieve it by appending further iterable objects:
| |
['0_a', '1_b', '2_c', '3_d', '4_e']
3. zip Objects and the enumerate Method
The zip function packs several iterables into a single iterable of tuples. It returns a zip object, and you can obtain the packed result with tuple or list:
| |
[('a', 'd', 'h'), ('b', 'e', 'i'), ('c', 'f', 'j')]
| |
(('a', 'd', 'h'), ('b', 'e', 'i'), ('c', 'f', 'j'))
The zip function is commonly used when looping over several iterables at once:
| |
a d h
b e i
c f j
The enumerate function is a special kind of packing: while iterating, it binds each element to its traversal index:
| |
0 a
1 b
2 c
3 d
You can implement the same behaviour just as easily with a zip object:
| |
0 a
1 b
2 c
3 d
When you need to build a dictionary mapping from two lists, a zip object comes in handy:
| |
{'a': 'd', 'b': 'e', 'c': 'f'}
Since there is a packing function, Python also offers the * operator to use together with zip for unpacking:
| |
[('a', 'd', 'h'), ('b', 'e', 'i'), ('c', 'f', 'j')]
| |
[('a', 'b', 'c'), ('d', 'e', 'f'), ('h', 'i', 'j')]
II. NumPy Fundamentals
1. Constructing NumPy Arrays
The most general way to construct an array is through array:
| |
array([1, 2, 3])
Let us now look at some ways of generating special arrays:
(a)Arithmetic sequences: np.linspace, np.arange
| |
array([1. , 1.4, 1.8, 2.2, 2.6, 3. , 3.4, 3.8, 4.2, 4.6, 5. ])
| |
array([1, 3])
(b)Special matrices: zeros, eye, full
| |
array([[0., 0., 0.],
[0., 0., 0.]])
| |
array([[1., 0., 0.],
[0., 1., 0.],
[0., 0., 1.]])
| |
array([[0., 1., 0.],
[0., 0., 1.],
[0., 0., 0.]])
| |
array([[10, 10, 10],
[10, 10, 10]])
| |
array([[1, 2, 3],
[1, 2, 3]])
(c)Random matrices: np.random
The most commonly used random generators are rand, randn, randint, and choice; they produce, respectively, an array from a uniform distribution on 0–1, an array from the standard normal distribution, an array of random integers, and a random sample drawn from a list:
| |
array([0.92340835, 0.20019461, 0.40755472])
| |
array([[0.8012362 , 0.53154881, 0.05858554],
[0.13103034, 0.18108091, 0.30253153],
[0.00528884, 0.99402007, 0.36348797]])
To generate values from a uniform distribution over the interval from a to b, you can write:
| |
array([6.59370831, 8.03865138, 9.19172546])
More generally, you can call an existing library function:
| |
array([11.26499636, 13.12311185, 6.00774156])
randn generates samples from the standard normal distribution N(0,I):
| |
array([ 1.87000209, 1.19885561, -0.58802943])
| |
array([[-1.3642839 , -0.31497567],
[-1.9452492 , -3.17272882]])
For a univariate normal distribution with variance $\sigma^2$ and mean $\mu$, samples can be generated as follows:
| |
array([1.56024917, 0.22829486, 7.3764211 ])
Likewise, you may turn to an existing function:
| |
array([3.53517851, 5.3441269 , 3.51192744])
randint lets you specify the minimum and maximum (exclusive) values for the random integers, together with the shape of the result:
| |
array([[ 5, 12],
[14, 9]])
choice draws results from a given list according to specified probabilities and a specified method. When no probabilities are given, sampling is uniform, and the default method is sampling with replacement:
| |
array(['b', 'a'], dtype='<U1')
| |
array([['c', 'b', 'd'],
['d', 'a', 'd'],
['a', 'c', 'd']], dtype='<U1')
When the number of returned elements equals the size of the original list, sampling without replacement is equivalent to calling permutation — in other words, shuffling the original list:
| |
array(['c', 'a', 'd', 'b'], dtype='<U1')
Finally, the random seed deserves a mention: it fixes the output of the random number generator:
| |
0.5488135039273248
| |
0.5488135039273248
2. Reshaping and Combining NumPy Arrays
(a)Transpose: T
| |
array([[0., 0.],
[0., 0.],
[0., 0.]])
(b)Combining operations: r_, c_
For two-dimensional arrays, r_ and c_ perform vertical and horizontal combination, respectively:
| |
array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])
| |
array([[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.]])
When a one-dimensional array is combined with a two-dimensional one, treat the former as a column vector. Provided the lengths match, only the horizontal c_ operation is available:
| |
ValueError('all the input arrays must have same number of dimensions, but the array at index 0 has 1 dimension(s) and the array at index 1 has 2 dimension(s)')
| |
array([0., 0., 0., 0.])
| |
array([[0., 0., 0., 0.],
[0., 0., 0., 0.]])
(c)Dimension transformation: reshape
reshape lets you rearrange an existing array into new dimensions. It offers two modes, C and F, which fill and read the data row by row and column by column, respectively.
| |
array([[0, 1, 2, 3],
[4, 5, 6, 7]])
| |
array([[0, 1],
[2, 3],
[4, 5],
[6, 7]])
| |
array([[0, 2],
[4, 6],
[1, 3],
[5, 7]])
In particular, because the size of the array being reshaped is known, reshape allows one dimension to be left unspecified — simply put -1 there:
| |
array([[0, 1],
[2, 3],
[4, 5],
[6, 7]])
Converting an array of size n*1 into a one-dimensional array is a frequently used operation:
| |
array([[1.],
[1.],
[1.]])
| |
array([1., 1., 1.])
3. Slicing and Indexing NumPy Arrays
Array slicing supports the slice-style start:end:step syntax, and you can also pass a list directly to select indices along a given dimension:
| |
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
| |
array([[0, 2],
[3, 5]])
In addition, np.ix_ allows boolean indexing along specific dimensions, although slice slicing is not permitted in that case:
| |
array([[0, 2],
[6, 8]])
| |
array([[3, 5],
[6, 8]])
When the array is one-dimensional, you can index with a boolean mask directly, without np.ix_:
| |
array([0, 2, 4, 6, 8])
4. Commonly Used Functions
For simplicity, assume that the arrays passed to the functions below are all one-dimensional.
(a)where
where is a conditional function that lets you specify the fill values for positions satisfying a condition and for those that do not:
| |
array([5, 1, 5, 5])
(b)nonzero, argmax, argmin
All three functions return indices: nonzero returns the indices of the nonzero elements, whereas argmax and argmin return the indices of the largest and smallest elements, respectively:
| |
(array([0, 1, 3, 4, 5], dtype=int64),)
| |
4
| |
1
(c)any, all
any returns True when the sequence contains at least one True or nonzero element, and False otherwise
all returns True when the sequence elements are all True or nonzero, and False otherwise
| |
True
| |
False
(d)cumprod, cumsum, diff
cumprod and cumsum compute cumulative products and cumulative sums, returning arrays of the same length. diff computes the difference with the preceding element; because the first element has no predecessor, the result is one element shorter than the original array under the default parameters.
| |
array([1, 2, 6], dtype=int32)
| |
array([1, 3, 6], dtype=int32)
| |
array([1, 1])
(e) Statistical functions
Common statistical functions include max, min, mean, median, std, var, sum, quantile. Quantile computation is a global method, so it cannot be called as array.quantile:
| |
array([0, 1, 2, 3, 4])
| |
4
| |
2.0
On an array containing missing values, however, these functions also return missing values. To skip missing values you must use functions of the nan* family — each of the statistical functions mentioned above has a corresponding nan* version.
| |
array([ 1., 2., nan])
| |
nan
| |
2.0
| |
1.5
Covariance and correlation coefficients can be computed with cov and corrcoef, respectively, as follows:
| |
array([[ 11.66666667, -16.66666667],
[-16.66666667, 38.66666667]])
| |
array([[ 1. , -0.78470603],
[-0.78470603, 1. ]])
Finally, a word about the axis parameter of statistical functions on two-dimensional Numpy arrays. It selects the dimension along which the statistics are computed: with axis=0 the result gives per-column statistics, whereas with axis=1 it gives per-row statistics:
| |
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
| |
array([12, 15, 18])
| |
array([ 6, 15, 24])
5. Broadcasting
Broadcasting handles operations between arrays of differing dimensions. Here we consider only broadcasting between arrays of at most two dimensions.
(a)Operations between a scalar and an array
When a scalar is combined with an array, the scalar is automatically expanded to the array’s size, and the operation is then applied element by element:
| |
array([[4., 4.],
[4., 4.]])
| |
array([[0.25, 0.25],
[0.25, 0.25]])
(b)Operations between two-dimensional arrays
When two arrays have exactly the same dimensions, the operation is applied to corresponding elements; otherwise an error is raised. The exception arises when one array has shape $m\times1$ or $1\times n$: its dimension equal to $1$ is then expanded to the corresponding dimension of the other array. For example, when a $1\times2$ array and a $3\times2$ array are combined element-wise, the first is expanded to $3\times2$, with the corresponding values copied during expansion. Note, however, that if the first array has shape $1\times3$, an error occurs because the sizes along the second dimension do not match and neither of them equals $1$.
| |
array([[1., 1.],
[1., 1.],
[1., 1.]])
| |
array([[2., 3.],
[2., 3.],
[2., 3.]])
| |
array([[2., 2.],
[3., 3.],
[4., 4.]])
| |
array([[2., 2.],
[2., 2.],
[2., 2.]])
(c)Operations between a one-dimensional and a two-dimensional array
When a one-dimensional array $A_k$ is combined with a two-dimensional array $B_{m,n}$, it is treated as the two-dimensional array $A_{1,k}$, and the broadcasting rules are the same as in (b). An error is raised when $k!=n$ and neither $k,n$ is $1$.
| |
array([[2., 2., 2.],
[2., 2., 2.]])
| |
array([[2., 2., 2.],
[2., 2., 2.]])
| |
array([[2., 2., 2.],
[2., 2., 2.]])
6. Vector and Matrix Computations
(a)Vector inner product: dot
| |
22
(b)Vector norms and matrix norms: np.linalg.norm
When computing matrix norms, the most important argument is ord, whose possible values are listed below:
| ord | norm for matrices | norm for vectors |
|---|---|---|
| None | Frobenius norm | 2-norm |
| ‘fro’ | Frobenius norm | / |
| ’nuc' | nuclear norm | / |
| inf | max(sum(abs(x), axis=1)) | max(abs(x)) |
| -inf | min(sum(abs(x), axis=1)) | min(abs(x)) |
| 0 | / | sum(x != 0) |
| 1 | max(sum(abs(x), axis=0)) | as below |
| -1 | min(sum(abs(x), axis=0)) | as below |
| 2 | 2-norm (largest sing. value) | as below |
| -2 | smallest singular value | as below |
| other | / | sum(abs(x)**ord)**(1./ord) |
| |
array([[0, 1],
[2, 3]])
| |
3.7416573867739413
| |
5.0
| |
3.702459173643833
| |
array([0, 1, 2, 3])
| |
3.0
| |
3.7416573867739413
| |
3.3019272488946263
(c)Matrix multiplication: @
| |
array([[0, 1],
[2, 3]])
| |
array([[-4, -3],
[-2, -1]])
| |
array([[ -2, -1],
[-14, -9]])
III. Exercises
Ex1: Writing matrix multiplication with a list comprehension
Following the formula, ordinary matrix multiplication can be written with three nested loops. Rewrite it as a list comprehension.
| |
True
Ex2: Updating a matrix
Given a matrix $A_{m\timesn}$, update every element of $A$ to produce a matrix $B$ according to $B_{ij}=A_{ij}\sum_{k=1}^n\frac{1}{A_{ik}}$. For instance, if $A$ is the matrix below, then $B_{2,2}=5\times(\frac{1}{4}+\frac{1}{5}+\frac{1}{6})=\frac{37}{12}$. Implement this efficiently with Numpy.
Ex3: The chi-squared statistic
Given a matrix $A_{m\times n}$, let $B_{ij} = \frac{(\sum_{i=1}^mA_{ij})\times (\sum_{j=1}^nA_{ij})}{\sum_{i=1}^m\sum_{j=1}^nA_{ij}}$. Define the chi-squared value as follows:
$$\chi^2 = \sum_{i=1}^m\sum_{j=1}^n\frac{(A_{ij}-B_{ij})^2}{B_{ij}}$$Use Numpy to compute, for a given matrix $A$, the value of $\chi^2$
| |
Ex4: Improving the performance of a matrix computation
Let $Z$ be an $m\timesn$ matrix, and let $B$ and $U$ be $m\timesp$ and $p\timesn$ matrices, respectively. Let $B_i$ be a row of $B$, indexed by $i$, and $U_j$ a column of $U$, indexed by $j$; define $\displaystyle R=\sum_{i=1}^m\sum_{j=1}^n\|B_i-U_j\|_2^2Z_{ij}$, where $\|\mathbf{a}\|_2^2$ denotes the sum of squares of the components of the vector $a$, namely $\sum_i a_i^2$.
Someone has computed the value of $R$ from the sample data given below. Make full use of the functions in Numpy to improve the performance of this code for the problem.
| |
100566
Ex5: Maximum length of consecutive integers
Given a Numpy array of integers, return the maximum length of a strictly increasing subarray of consecutive integers, where “forward” means the increasing direction. For example, given
, the subarray
\[5,6,7\]has the maximum length, so the answer is 3; given
\[3,2,1,2,3,4,6\], it is
\[1,2,3,4\], so the answer is 4. Make full use of Numpy’s built-in functions. (Hint: consider the nonzero, diff functions.)
