Pandas Fundamentals
| |
Make sure your pandas version is at least the one shown below; otherwise, please upgrade. Also confirm that the three packages xlrd, xlwt, openpyxl are installed, with xlrd no newer than 2.0.0.
| |
'1.1.5'
I. Reading and Writing Files
1. Reading Files
pandas can read many file formats; here we focus on csv, excel, and txt files.
| |
| col1 | col2 | col3 | col4 | col5 | |
|---|---|---|---|---|---|
| 0 | 2 | a | 1.4 | apple | 2020/1/1 |
| 1 | 3 | b | 3.4 | banana | 2020/1/2 |
| 2 | 6 | c | 2.5 | orange | 2020/1/5 |
| 3 | 5 | d | 3.2 | lemon | 2020/1/7 |
| |
| col1 | col2 | col3 | col4 | |
|---|---|---|---|---|
| 0 | 2 | a | 1.4 | apple 2020/1/1 |
| 1 | 3 | b | 3.4 | banana 2020/1/2 |
| 2 | 6 | c | 2.5 | orange 2020/1/5 |
| 3 | 5 | d | 3.2 | lemon 2020/1/7 |
| |
| col1 | col2 | col3 | col4 | col5 | |
|---|---|---|---|---|---|
| 0 | 2 | a | 1.4 | apple | 2020/1/1 |
| 1 | 3 | b | 3.4 | banana | 2020/1/2 |
| 2 | 6 | c | 2.5 | orange | 2020/1/5 |
| 3 | 5 | d | 3.2 | lemon | 2020/1/7 |
Several common parameters are worth noting. header=None means the first row is not used as column names, whereas index_col designates one or more columns as the index — indexing is covered in detail in Chapter 3. usecols specifies the set of columns to read, with all columns read by default, and parse_dates marks the columns to be converted to datetime; time series are discussed in Chapter 10. Finally, nrows gives the number of data rows to read. All of these parameters work with the three functions above.
| |
| 0 | 1 | 2 | 3 | |
|---|---|---|---|---|
| 0 | col1 | col2 | col3 | col4 |
| 1 | 2 | a | 1.4 | apple 2020/1/1 |
| 2 | 3 | b | 3.4 | banana 2020/1/2 |
| 3 | 6 | c | 2.5 | orange 2020/1/5 |
| 4 | 5 | d | 3.2 | lemon 2020/1/7 |
| |
| col3 | col4 | col5 | ||
|---|---|---|---|---|
| col1 | col2 | |||
| 2 | a | 1.4 | apple | 2020/1/1 |
| 3 | b | 3.4 | banana | 2020/1/2 |
| 6 | c | 2.5 | orange | 2020/1/5 |
| 5 | d | 3.2 | lemon | 2020/1/7 |
| |
| col1 | col2 | |
|---|---|---|
| 0 | 2 | a |
| 1 | 3 | b |
| 2 | 6 | c |
| 3 | 5 | d |
| |
| col1 | col2 | col3 | col4 | col5 | |
|---|---|---|---|---|---|
| 0 | 2 | a | 1.4 | apple | 2020-01-01 |
| 1 | 3 | b | 3.4 | banana | 2020-01-02 |
| 2 | 6 | c | 2.5 | orange | 2020-01-05 |
| 3 | 5 | d | 3.2 | lemon | 2020-01-07 |
| |
| col1 | col2 | col3 | col4 | col5 | |
|---|---|---|---|---|---|
| 0 | 2 | a | 1.4 | apple | 2020/1/1 |
| 1 | 3 | b | 3.4 | banana | 2020/1/2 |
When reading txt files, you often encounter delimiters other than whitespace. read_table has a sep parameter that lets you customise the delimiter used to parse a txt file. For example, the table read below is delimited by ||||:
| |
| col1 |||| col2 | |
|---|---|
| 0 | TS |||| This is an apple. |
| 1 | GQ |||| My name is Bob. |
| 2 | WT |||| Well done! |
| 3 | PT |||| May I help you? |
The result above is clearly not what we want. You can use sep instead, specifying python as the engine:
| |
| col1 | col2 | |
|---|---|---|
| 0 | TS | This is an apple. |
| 1 | GQ | My name is Bob. |
| 2 | WT | Well done! |
| 3 | PT | May I help you? |
[WARNING]sep is a regular expression parameter
When using read_table, keep in mind that sep is interpreted as a regular expression, so you must escape | as \| or the data will not be read correctly. For the basics of regular expressions, see Chapter 8 or other reference material.
[END]
2. Writing Data
When writing data, the most common operation is to set index to False, particularly when the index carries no special meaning; doing so drops the index from the saved file.
| |
pandas does not define a to_table function, but to_csv can write txt files and allows a custom delimiter — a tab \t is commonly used:
| |
To convert a table quickly into markdown or latex, use the to_markdown and to_latex functions; this requires the tabulate package to be installed.
| |
| | col1 | col2 | col3 | col4 | col5 |
|---:|-------:|:-------|-------:|:-------|:---------|
| 0 | 2 | a | 1.4 | apple | 2020/1/1 |
| 1 | 3 | b | 3.4 | banana | 2020/1/2 |
| 2 | 6 | c | 2.5 | orange | 2020/1/5 |
| 3 | 5 | d | 3.2 | lemon | 2020/1/7 |
| |
\begin{tabular}{lrlrll}
\toprule
{} & col1 & col2 & col3 & col4 & col5 \\
\midrule
0 & 2 & a & 1.4 & apple & 2020/1/1 \\
1 & 3 & b & 3.4 & banana & 2020/1/2 \\
2 & 6 & c & 2.5 & orange & 2020/1/5 \\
3 & 5 & d & 3.2 & lemon & 2020/1/7 \\
\bottomrule
\end{tabular}
II. Basic Data Structures
pandas provides two basic data structures: Series, which stores one-dimensional values, and DataFrame, which stores two-dimensional values. Many attributes and methods are defined on these two structures.
1. Series
A Series generally consists of four parts: the sequence’s values data, its index index, its storage type dtype, and the sequence’s name name. The index may also be given a name, which is empty by default.
| |
my_idx
id1 100
20 a
third {'dic1': 5}
Name: my_name, dtype: object
[NOTE]The object type
object represents a mixed type, as illustrated above where integers, strings, and a Python dictionary are all stored together. Moreover, pandas currently treats a sequence of pure strings as an object-typed sequence by default, although it can also be stored with the string type; text sequences are discussed in Chapter 8.
[END]
You can access these attributes with the dot syntax:
| |
array([100, 'a', {'dic1': 5}], dtype=object)
| |
Index(['id1', 20, 'third'], dtype='object', name='my_idx')
| |
dtype('O')
| |
'my_name'
The length of a sequence can be obtained with .shape:
| |
(3,)
Indexing is one of the most important concepts in pandas and is discussed in detail in Chapter 3. To retrieve the value corresponding to a single index, use [index_item].
2. DataFrame
A DataFrame extends Series by adding column indices; a data frame can be constructed from two-dimensional data together with row and column indices:
| |
| col_0 | col_1 | col_2 | |
|---|---|---|---|
| row_0 | 1 | a | 1.2 |
| row_1 | 2 | b | 2.2 |
| row_2 | 3 | c | 3.2 |
In practice, however, data frames are more often built from a mapping of column names to data, supplemented with a row index:
| |
| col_0 | col_1 | col_2 | |
|---|---|---|---|
| row_0 | 1 | a | 1.2 |
| row_1 | 2 | b | 2.2 |
| row_2 | 3 | c | 3.2 |
Thanks to this mapping, [col_name] and [col_list] can be used in a DataFrame to select a single column or a table made up of several columns, yielding a Series and a DataFrame, respectively:
| |
row_0 1
row_1 2
row_2 3
Name: col_0, dtype: int64
| |
| col_0 | col_1 | |
|---|---|---|
| row_0 | 1 | a |
| row_1 | 2 | b |
| row_2 | 3 | c |
As with Series, the corresponding attributes can be retrieved from a data frame:
| |
array([[1, 'a', 1.2],
[2, 'b', 2.2],
[3, 'c', 3.2]], dtype=object)
| |
Index(['row_0', 'row_1', 'row_2'], dtype='object')
| |
Index(['col_0', 'col_1', 'col_2'], dtype='object')
| |
col_0 int64
col_1 object
col_2 float64
dtype: object
| |
(3, 3)
A DataFrame can be transposed with .T:
| |
| row_0 | row_1 | row_2 | |
|---|---|---|---|
| col_0 | 1 | 2 | 3 |
| col_1 | a | b | c |
| col_2 | 1.2 | 2.2 | 3.2 |
III. Common Basic Functions
For the examples in this and the remaining chapters, we will use a fictitious dataset called learn_pandas.csv, which records personal information from the physical fitness tests of students at four schools.
| |
Index(['School', 'Grade', 'Name', 'Gender', 'Height', 'Weight', 'Transfer',
'Test_Number', 'Test_Date', 'Time_Record'],
dtype='object')
These column names denote, in order, school, grade, name, gender, height, weight, whether the student transferred departments, test session, test date, and 1000-metre score. This chapter only needs the first seven columns.
| |
1. Summary Functions
The head and tail functions return the first n and the last n rows of a table or sequence, respectively, with n defaulting to 5:
| |
| School | Grade | Name | Gender | Height | Weight | Transfer | |
|---|---|---|---|---|---|---|---|
| 0 | Shanghai Jiao Tong University | Freshman | Gaopeng Yang | Female | 158.9 | 46.0 | N |
| 1 | Peking University | Freshman | Changqiang You | Male | 166.5 | 70.0 | N |
| |
| School | Grade | Name | Gender | Height | Weight | Transfer | |
|---|---|---|---|---|---|---|---|
| 197 | Shanghai Jiao Tong University | Senior | Chengqiang Chu | Female | 153.9 | 45.0 | N |
| 198 | Shanghai Jiao Tong University | Senior | Chengmei Shen | Male | 175.3 | 71.0 | N |
| 199 | Tsinghua University | Sophomore | Chunpeng Lv | Male | 155.7 | 51.0 | N |
info and describe return an overview of the table and the main statistics of its numeric columns, respectively:
| |
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 200 entries, 0 to 199
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 School 200 non-null object
1 Grade 200 non-null object
2 Name 200 non-null object
3 Gender 200 non-null object
4 Height 183 non-null float64
5 Weight 189 non-null float64
6 Transfer 188 non-null object
dtypes: float64(2), object(5)
memory usage: 11.1+ KB
| |
| Height | Weight | |
|---|---|---|
| count | 183.000000 | 189.000000 |
| mean | 163.218033 | 55.015873 |
| std | 8.608879 | 12.824294 |
| min | 145.400000 | 34.000000 |
| 25% | 157.150000 | 46.000000 |
| 50% | 161.900000 | 51.000000 |
| 75% | 167.500000 | 65.000000 |
| max | 193.900000 | 89.000000 |
[NOTE]A more comprehensive data summary
info and describe reveal only a limited amount of information. For a thorough and effective view of a dataset — especially one with many columns — the pandas-profiling package is recommended; it is mentioned again in Chapter 11.
[END]
2. Statistical Functions for Features
A number of statistical functions are defined on Series and DataFrame, the most common being sum, mean, median, var, std, max, min. Let us select the height and weight columns for a demonstration:
| |
Height 163.218033
Weight 55.015873
dtype: float64
| |
Height 193.9
Weight 89.0
dtype: float64
Three further functions deserve introduction: quantile, count, and idxmax, which return the quantile, the number of non-missing values, and the index of the maximum value, respectively:
| |
Height 167.5
Weight 65.0
Name: 0.75, dtype: float64
| |
Height 183
Weight 189
dtype: int64
| |
Height 193
Weight 2
dtype: int64
All the functions above return scalars, which is why they are also called aggregation functions. They share an axis parameter: the default 0 means aggregating column by column, whereas setting it to 1 means aggregating row by row:
| |
0 102.45
1 118.25
2 138.95
3 41.00
4 124.00
dtype: float64
3. Unique-Value Functions
Applying unique and nunique to a sequence yields, respectively, a list of its unique values and the number of unique values:
| |
array(['Shanghai Jiao Tong University', 'Peking University',
'Fudan University', 'Tsinghua University'], dtype=object)
| |
4
value_counts gives the unique values together with their frequencies:
| |
Tsinghua University 69
Shanghai Jiao Tong University 57
Fudan University 40
Peking University 34
Name: School, dtype: int64
To inspect unique combinations of several columns, use drop_duplicates. Its key parameter is keep: the default first retains the row where each combination first appears, last retains the row where it last appears, and False removes all rows belonging to duplicated combinations.
| |
| Gender | Transfer | Name | |
|---|---|---|---|
| 0 | Female | N | Gaopeng Yang |
| 1 | Male | N | Changqiang You |
| 12 | Female | NaN | Peng You |
| 21 | Male | NaN | Xiaopeng Shen |
| 36 | Male | Y | Xiaojuan Qin |
| 43 | Female | Y | Gaoli Feng |
| |
| Gender | Transfer | Name | |
|---|---|---|---|
| 147 | Male | NaN | Juan You |
| 150 | Male | Y | Chengpeng You |
| 169 | Female | Y | Chengquan Qin |
| 194 | Female | NaN | Yanmei Qian |
| 197 | Female | N | Chengqiang Chu |
| 199 | Male | N | Chunpeng Lv |
| |
| Gender | Transfer | Name | |
|---|---|---|---|
| 0 | Female | N | Gaopeng Yang |
| 1 | Male | N | Changqiang You |
| 2 | Male | N | Mei Sun |
| 4 | Male | N | Gaojuan You |
| 5 | Female | N | Xiaoli Qian |
| |
0 Shanghai Jiao Tong University
1 Peking University
3 Fudan University
5 Tsinghua University
Name: School, dtype: object
The duplicated function is similar to drop_duplicates, but it returns a boolean list indicating whether each element is unique, with the same keep parameter. In the returned sequence, duplicated elements are marked True and the others False. Calling drop_duplicates is equivalent to dropping the rows for which duplicated is True.
| |
0 False
1 False
2 True
3 True
4 True
dtype: bool
| |
0 False
1 False
2 True
3 False
4 True
Name: School, dtype: bool
4. Replacement Functions
Replacement is usually carried out on a single column, so the examples below all use Series. The replacement functions in pandas fall into three categories: mapping replacement, logical replacement, and numeric replacement. Mapping replacement includes the replace method, the str.replace method from Chapter 8, and the cat.codes method from Chapter 9; here we introduce replace.
With replace, you can specify replacements through a dictionary or by passing two lists:
| |
0 0
1 1
2 1
3 0
4 1
Name: Gender, dtype: int64
| |
0 0
1 1
2 1
3 0
4 1
Name: Gender, dtype: int64
In addition, replace supports a special directional mode. Setting method to ffill replaces values with the nearest preceding value that was not replaced, whereas bfill uses the nearest following one. As the examples below show, the two produce different results:
| |
0 a
1 a
2 b
3 b
4 b
5 b
6 a
dtype: object
| |
0 a
1 b
2 b
3 a
4 a
5 a
6 a
dtype: object
[WARNING]Use str.replace for regular-expression replacement
Although replace can perform regular-expression substitution, the current version still has a bug when applying it to string-typed data. If you need this functionality, use str.replace instead; the details are explained in Chapter 8.
[END]
Logical replacement comprises where and mask, which are perfectly symmetrical: where replaces the rows where the supplied condition is False, whereas mask replaces those where it is True. If no replacement value is given, missing values are used.
| |
0 -1.0
1 NaN
2 NaN
3 -50.0
dtype: float64
| |
0 -1.0
1 100.0
2 100.0
3 -50.0
dtype: float64
| |
0 NaN
1 1.2345
2 100.0000
3 NaN
dtype: float64
| |
0 -50.0000
1 1.2345
2 100.0000
3 -50.0000
dtype: float64
Note that the condition you pass need only be a boolean sequence whose index matches that of the Series being called:
| |
0 -50.0000
1 1.2345
2 100.0000
3 -50.0000
dtype: float64
Numeric replacement includes the round, abs, and clip methods, which round to a given precision, take absolute values, and clip values to a range, respectively:
| |
0 -1.00
1 1.23
2 100.00
3 -50.00
dtype: float64
| |
0 1.0000
1 1.2345
2 100.0000
3 50.0000
dtype: float64
| |
0 0.0000
1 1.2345
2 2.0000
3 0.0000
dtype: float64
[Try it yourself]
In clip, values beyond a boundary are simply cut down to that boundary. How would you replace out-of-range values with a custom value instead?
[END]
5. Sorting Functions
There are two ways to sort: by value and by index, implemented by sort_values and sort_index, respectively.
To demonstrate the sorting functions, we first use set_index to turn the grade and name columns into indices. Multi-level indices and the methods for setting indices are explained in detail in Chapter 3.
| |
| Height | Weight | ||
|---|---|---|---|
| Grade | Name | ||
| Freshman | Gaopeng Yang | 158.9 | 46.0 |
| Changqiang You | 166.5 | 70.0 | |
| Senior | Mei Sun | 188.9 | 89.0 |
Sorting by height, the default ascending=True produces ascending order:
| |
| Height | Weight | ||
|---|---|---|---|
| Grade | Name | ||
| Junior | Xiaoli Chu | 145.4 | 34.0 |
| Senior | Gaomei Lv | 147.3 | 34.0 |
| Sophomore | Peng Han | 147.8 | 34.0 |
| Senior | Changli Lv | 148.7 | 41.0 |
| Sophomore | Changjuan You | 150.5 | 40.0 |
| |
| Height | Weight | ||
|---|---|---|---|
| Grade | Name | ||
| Senior | Xiaoqiang Qin | 193.9 | 79.0 |
| Mei Sun | 188.9 | 89.0 | |
| Gaoli Zhao | 186.5 | 83.0 | |
| Freshman | Qiang Han | 185.3 | 87.0 |
| Senior | Qiang Zheng | 183.9 | 87.0 |
Sorting by several columns is a common requirement — for example, sorting by height within groups of equal weight, with height in descending order and weight in ascending order:
| |
| Height | Weight | ||
|---|---|---|---|
| Grade | Name | ||
| Sophomore | Peng Han | 147.8 | 34.0 |
| Senior | Gaomei Lv | 147.3 | 34.0 |
| Junior | Xiaoli Chu | 145.4 | 34.0 |
| Sophomore | Qiang Zhou | 150.5 | 36.0 |
| Freshman | Yanqiang Xu | 152.4 | 38.0 |
Sorting by index works exactly like sorting by value, except that the values to sort reside in the index; you then specify the name or position of the index level through the level parameter. Bear in mind also that strings are ordered alphabetically.
| |
| Height | Weight | ||
|---|---|---|---|
| Grade | Name | ||
| Freshman | Yanquan Wang | 163.5 | 55.0 |
| Yanqiang Xu | 152.4 | 38.0 | |
| Yanqiang Feng | 162.3 | 51.0 | |
| Yanpeng Lv | NaN | 65.0 | |
| Yanli Zhang | 165.1 | 52.0 |
6. The apply Method
The apply method is commonly used to iterate over the rows or columns of a DataFrame. Its axis argument has the same meaning as for the statistical aggregation functions in Section 2, and the argument passed to apply is typically a function that takes a sequence as input. For example, .mean() can be written with apply as follows:
| |
Height 163.218033
Weight 55.015873
dtype: float64
You can likewise use a lambda expression to keep the code concise; here x refers to each successive sequence fed in from the df_demo table:
| |
Height 163.218033
Weight 55.015873
dtype: float64
If you set axis=1, each call passes a Series made up of a row’s elements to the function, and the result agrees with the earlier row-wise means.
| |
0 102.45
1 118.25
2 138.95
3 41.00
4 124.00
dtype: float64
Here is another example. The mad function returns the mean of the absolute deviations from a sequence’s mean. For the sequence 1, 3, 7, 10, the mean is 5.25, the absolute deviations are 4.25, 2.25, 1.75, 4.75, and their mean is 3.25. Let us use apply to compute the mad index for height and weight:
| |
Height 6.707229
Weight 10.391870
dtype: float64
This agrees with the result from the built-in mad function:
| |
Height 6.707229
Weight 10.391870
dtype: float64
[WARNING]Use apply with care
Because it accepts user-defined functions, apply is highly flexible — but that flexibility comes at the cost of performance. In general, handling the same task with a built-in pandas function is considerably faster than using apply, so reach for apply only when you genuinely have custom requirements.
[END]
IV. Window Objects
pandas offers three kinds of windows: the rolling window rolling, the expanding window expanding, and the exponentially weighted window ewm. Note that rolling windows sized by a date offset are discussed in Chapter 10, whereas exponentially weighted windows appear in this chapter’s exercises.
1. Rolling Window Objects
To use rolling-window functions, you must first call .rolling on a sequence to obtain a rolling object, whose most important parameter is the window size window.
| |
Rolling [window=3,center=False,axis=0]
Once you have a rolling object, you can carry out computations with the corresponding aggregation functions. Keep in mind that the window includes the element at the current row: at the fourth position, for instance, the mean is (2+3+4)/3 rather than (1+2+3)/3:
| |
0 NaN
1 NaN
2 2.0
3 3.0
4 4.0
dtype: float64
| |
0 NaN
1 NaN
2 6.0
3 9.0
4 12.0
dtype: float64
Rolling correlation and rolling covariance can be computed as follows:
| |
0 NaN
1 NaN
2 2.5
3 7.0
4 12.0
dtype: float64
| |
0 NaN
1 NaN
2 0.944911
3 0.970725
4 0.995402
dtype: float64
You can also pass a custom function through apply; it receives the Series for the corresponding window. The mean function above, for example, can be expressed equivalently as:
| |
0 NaN
1 NaN
2 2.0
3 3.0
4 4.0
dtype: float64
shift, diff, and pct_change form a group of rolling-like functions. They share the parameter periods=n, which defaults to 1, and return, respectively, the value of the n-th preceding element, the difference from the n-th preceding element (unlike in Numpy, where it denotes an n-th order difference), and the growth rate relative to the n-th preceding element. The value of n may be negative, indicating the analogous operation in the opposite direction.
| |
0 NaN
1 NaN
2 1.0
3 3.0
4 6.0
dtype: float64
| |
0 NaN
1 NaN
2 NaN
3 9.0
4 12.0
dtype: float64
| |
0 NaN
1 2.000000
2 1.000000
3 0.666667
4 0.500000
dtype: float64
| |
0 3.0
1 6.0
2 10.0
3 15.0
4 NaN
dtype: float64
| |
0 -5.0
1 -7.0
2 -9.0
3 NaN
4 NaN
dtype: float64
They are regarded as rolling-like functions because their behaviour can be reproduced equivalently with the rolling method using a window of size n+1:
| |
0 NaN
1 NaN
2 1.0
3 3.0
4 6.0
dtype: float64
| |
0 NaN
1 NaN
2 NaN
3 9.0
4 12.0
dtype: float64
| |
0 NaN
1 2.000000
2 1.000000
3 0.666667
4 0.500000
dtype: float64
[Try it yourself]
The default window direction of a rolling object is forward, but sometimes you need a backward-looking window. For example, applying a backward window of size 2 with sum to 1, 2, 3 gives 3, 5, NaN. How would you implement such a backward rolling operation?
[END]
2. Expanding Windows
An expanding window, also called a cumulative window, can be thought of as a window of variable length: at any given position it spans from the beginning of the sequence up to that position, and the aggregation functions act on these progressively growing windows. Concretely, for a sequence a1, a2, a3, a4, the windows at the four positions are
\[a1\],
\[a1, a2\],
\[a1, a2, a3\], and
\[a1, a2, a3, a4\].
| |
0 1.000000
1 2.000000
2 3.333333
3 5.000000
dtype: float64
[Try it yourself]
The functions cummax, cumsum, cumprod are typical expanding-like functions. Implement each of them using an expanding object.
[END]
V. Exercises
Ex1: The Pokémon dataset
Here is a dataset of Pokémon, with some background information first:
#is the national Pokédex number; repeated numbers in different rows denote different forms of the same creatureA creature may have a single type or two types; for those with one type,
Type 2is missingTotal, HP, Attack, Defense, Sp. Atk, Sp. Def, Speeddenote base stat total, hit points, attack, defense, special attack, special defense, and speed; the base stat total is the sum of the remaining six
| |
| # | Name | Type 1 | Type 2 | Total | HP | Attack | Defense | Sp. Atk | Sp. Def | Speed | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | Bulbasaur | Grass | Poison | 318 | 45 | 49 | 49 | 65 | 65 | 45 |
| 1 | 2 | Ivysaur | Grass | Poison | 405 | 60 | 62 | 63 | 80 | 80 | 60 |
| 2 | 3 | Venusaur | Grass | Poison | 525 | 80 | 82 | 83 | 100 | 100 | 80 |
Sum
HP, Attack, Defense, Sp. Atk, Sp. Def, Speedand verify that the result equals theTotalvalue.Keep only the first record for each creature whose
#is repeated, then answer the following:
- Find how many distinct primary types there are, and identify the three most common ones
- Find the distinct combinations of the primary and secondary types
- Find the type combinations that have never appeared
- Construct
Seriesaccording to the following requirements:
- Take the attack values and replace those above 120 with
high, those below 50 withlow, and the rest withmid - Take the primary type and convert all its letters to uppercase using
replaceand, separately,apply - For each creature, compute the deviation of its six abilities — the value furthest from the median — add it to
df, and sort in descending order
Ex2: Exponentially Weighted Windows
ewmas an expanding window
Within an expanding window you can use various functions to compute cumulative historical metrics, but these built-in statistical functions typically give every element in the window the same weight. In fact, you can assign different weights to the elements; the exponentially weighted window is a special kind of expanding window that does precisely this.
The most important parameter is alpha, which determines the default window weights $w_i=(1-\alpha)^i,i\in\{0,1,...,t\}$, where $i=t$ refers to the current element and $i=0$ to the first element of the sequence.
The weight formula shows that weights decay as elements lie further from the current value. Writing the original sequence as $x$ and the updated current element as $y_t$, normalising the weighted formula gives:
$$ \begin{split}y_t &=\frac{\sum_{i=0}^{t} w_i x_{t-i}}{\sum_{i=0}^{t} w_i} \\ &=\frac{x_t + (1 - \alpha)x_{t-1} + (1 - \alpha)^2 x_{t-2} + ... + (1 - \alpha)^{t} x_{0}}{1 + (1 - \alpha) + (1 - \alpha)^2 + ... + (1 - \alpha)^{t}}\\\end{split} $$For a Series, the exponentially smoothed sequence can be computed with an ewm object as follows:
| |
0 -1
1 -1
2 -2
3 -2
4 -2
dtype: int32
| |
0 -1.000000
1 -1.000000
2 -1.409836
3 -1.609756
4 -1.725845
dtype: float64
Implement this with an expanding window.
ewmas a rolling window
As seen in part 1, ewm is a special case of an expanding window and can only weight elements starting from the first one. Suppose we now want a bounded window n, sliding over the n most recent elements including the current one and smoothing them with weights. Drawing on the rolling-window functions, derive the new update formulas for wi and yt, and implement this behaviour with a rolling window.
