Pandas Categorical Data
| |
I. The cat Object
1. Attributes of the cat Object
pandas provides the category dtype so that users can work with categorical variables; an ordinary series can be turned into a categorical variable with the astype method.
| |
0 Freshman
1 Freshman
2 Senior
3 Sophomore
4 Sophomore
Name: Grade, dtype: category
Categories (4, object): ['Freshman', 'Junior', 'Senior', 'Sophomore']
A categorical Series defines a cat object, much like the str object introduced in the previous chapter. It offers a number of attributes and methods for manipulating categories.
| |
<pandas.core.arrays.categorical.CategoricalAccessor object at 0x0000020F9B7A7108>
A concrete categorical variable has two components: the categories themselves, stored as an Index, and whether they are ordered. Both can be accessed through attributes of cat:
| |
Index(['Freshman', 'Junior', 'Senior', 'Sophomore'], dtype='object')
| |
False
In addition, each series assigns a unique integer code to its categories, determined by the order in cat.categories; this attribute is accessible through codes:
| |
0 0
1 0
2 2
3 3
4 3
dtype: int8
2. Adding, Removing, and Modifying Categories
The categories attribute of the cat object lets you query the categories, so how do you carry out the other three operations — adding, modifying, and deleting?
[NOTE] Categories Cannot Be Modified Directly
As mentioned in Chapter 3, an Index cannot be modified with index_obj[0] = item. Since categories are stored in an Index, pandas defines several methods on the cat attribute to achieve the same end.
[END]
First, categories can be added with add_categories:
| |
Index(['Freshman', 'Junior', 'Senior', 'Sophomore', 'Graduate'], dtype='object')
To remove a category use remove_categories; every element in the original series belonging to that category is then set to missing. For example, remove the freshman category:
| |
Index(['Junior', 'Senior', 'Sophomore', 'Graduate'], dtype='object')
| |
0 NaN
1 NaN
2 Senior
3 Sophomore
4 Sophomore
Name: Grade, dtype: category
Categories (4, object): ['Junior', 'Senior', 'Sophomore', 'Graduate']
You can also use set_categories to assign an entirely new set of categories; elements belonging to categories absent from the new set are set to missing.
| |
Index(['Sophomore', 'PhD'], dtype='object')
| |
0 NaN
1 NaN
2 NaN
3 Sophomore
4 Sophomore
Name: Grade, dtype: category
Categories (2, object): ['Sophomore', 'PhD']
To drop categories that never occur in the series, use remove_unused_categories:
| |
Index(['Sophomore'], dtype='object')
Finally, the remaining operation among add, modify, query, and delete is modification, handled by rename_categories. Note that this method also modifies the corresponding values in the original series. For example, rename Sophomore to Second-year undergraduate:
| |
0 NaN
1 NaN
2 NaN
3 Second-year undergraduate
4 Second-year undergraduate
Name: Grade, dtype: category
Categories (1, object): ['Second-year undergraduate']
II. Ordered Categoricals
1. Establishing an Order
Ordered and unordered categories can be converted into one another with as_unordered and reorder_categories. Note that the argument passed to the latter must be a list made up of the current series’ unordered categories: you may neither add new categories nor omit existing ones, and you must specify ordered=True, or the method has no effect. For example, divide the grades into relative levels and then restore the unordered state:
| |
0 Freshman
1 Freshman
2 Senior
3 Sophomore
4 Sophomore
Name: Grade, dtype: category
Categories (4, object): ['Freshman' < 'Sophomore' < 'Junior' < 'Senior']
| |
0 Freshman
1 Freshman
2 Senior
3 Sophomore
4 Sophomore
Name: Grade, dtype: category
Categories (4, object): ['Freshman', 'Sophomore', 'Junior', 'Senior']
[NOTE] Categories Cannot Be Modified Directly
If you would rather not specify ordered=True, you can first convert with s.cat.as_ordered() and then use reorder_categories to make the precise relative adjustments.
[END]
2. Sorting and Comparison
In Chapter 2 we discussed the sorting of string and numeric series; now consider categorical variables. Once a column’s type has been changed to category and the appropriate ordering assigned, sort_index and sort_values work as usual. For example, sort by grade:
| |
| Grade | Name | Gender | Height | Weight | |
|---|---|---|---|---|---|
| 0 | Freshman | Gaopeng Yang | Female | 158.9 | 46.0 |
| 105 | Freshman | Qiang Shi | Female | 164.5 | 52.0 |
| 96 | Freshman | Changmei Feng | Female | 163.8 | 56.0 |
| 88 | Freshman | Xiaopeng Han | Female | 164.1 | 53.0 |
| 81 | Freshman | Yanli Zhang | Female | 165.1 | 52.0 |
| |
| Name | Gender | Height | Weight | |
|---|---|---|---|---|
| Grade | ||||
| Freshman | Gaopeng Yang | Female | 158.9 | 46.0 |
| Freshman | Qiang Shi | Female | 164.5 | 52.0 |
| Freshman | Changmei Feng | Female | 163.8 | 56.0 |
| Freshman | Xiaopeng Han | Female | 164.1 | 53.0 |
| Freshman | Yanli Zhang | Female | 165.1 | 52.0 |
Once an order exists, comparisons become possible. Comparisons of categorical variables fall into two kinds. The first uses == or !=, with a scalar or a Series (or list) of the same length as the operand. The second uses the four ordering relations >,>=,<,<=; the operands are similar, but every element involved must belong to the original series’ categories and must share its index.
| |
0 False
1 False
2 False
3 True
4 True
Name: Grade, dtype: bool
| |
0 False
1 False
2 False
3 False
4 False
Name: Grade, dtype: bool
| |
0 True
1 True
2 False
3 True
4 True
Name: Grade, dtype: bool
| |
0 True
1 True
2 False
3 True
4 True
Name: Grade, dtype: bool
III. Interval Categories
1. Constructing Intervals with cut and qcut
Intervals are a special kind of category. In practical data analysis, interval series are usually built with cut and qcut, which bin the numeric features of the original series — replacing concrete values with interval positions.
First, the common usage of cut:
The most important parameter is bins. Given an integer n, it divides the whole array into n equal-width segments between the minimum and the maximum. Since intervals are by default left-open and right-closed, the minimum must be brought inside; pandas handles this by subtracting 0.001*(max-min) from the left endpoint of the lowest interval. Thus dividing [1,2] into two bins gives (0.999,1.5] for the first and (1.5,2] for the second. To obtain left-closed, right-open intervals instead, set right to False; the corresponding adjustment adds 0.001*(max-min) to the right endpoint of the highest interval.
| |
0 (0.999, 1.5]
1 (1.5, 2.0]
dtype: category
Categories (2, interval[float64]): [(0.999, 1.5] < (1.5, 2.0]]
| |
0 [1.0, 1.5)
1 [1.5, 2.001)
dtype: category
Categories (2, interval[float64]): [[1.0, 1.5) < [1.5, 2.001)]
Another common use of bins is to pass a list of split points (np.infty can represent infinity):
| |
0 (-inf, 1.2]
1 (1.8, 2.2]
dtype: category
Categories (4, interval[float64]): [(-inf, 1.2] < (1.2, 1.8] < (1.8, 2.2] < (2.2, inf]]
Two other common parameters are labels and retbins, giving the interval names and whether to return the split points (not returned by default):
| |
0 small
1 big
dtype: category
Categories (2, object): ['small' < 'big']
| |
array([0.999, 1.5 , 2. ])
In usage, qcut differs little from cut: the bins parameter simply becomes q, where q stands for quantile. When q is an integer n, the data are binned into n quantile groups; alternatively, you may pass a list of floats for the corresponding quantile boundaries.
| |
0 (33.999, 48.0]
1 (55.0, 89.0]
2 (55.0, 89.0]
3 (33.999, 48.0]
4 (55.0, 89.0]
Name: Weight, dtype: category
Categories (3, interval[float64]): [(33.999, 48.0] < (48.0, 55.0] < (55.0, 89.0]]
| |
0 (44.0, 69.4]
1 (69.4, 89.0]
2 (69.4, 89.0]
3 (33.999, 44.0]
4 (69.4, 89.0]
Name: Weight, dtype: category
Categories (3, interval[float64]): [(33.999, 44.0] < (44.0, 69.4] < (69.4, 89.0]]
2. Constructing General Intervals
A concrete interval has three elements: a left endpoint, a right endpoint, and the openness of the endpoints, where the last can be one of right, left, both, neither:
| |
Interval(0, 1, closed='right')
Its attributes include mid, length, right, left, closed,, denoting the midpoint, length, right endpoint, left endpoint, and closure state respectively.
Use in to test whether an element belongs to an interval:
| |
True
Use overlaps to test whether two intervals intersect:
| |
True
In general, pd.IntervalIndex objects can be created in four ways — from_breaks, from_arrays, from_tuples, interval_range — each suited to different situations:
from_breaks works like cut or qcut, except that the latter two compute the split points whereas this one takes custom ones directly:
| |
IntervalIndex([[1, 3], [3, 6], [6, 10]],
closed='both',
dtype='interval[int64]')
from_arrays takes separate lists of left and right endpoints; it suits cases where intervals overlap and you know the starts and ends:
| |
IntervalIndex([(1, 5), (3, 4), (6, 9), (10, 11)],
closed='neither',
dtype='interval[int64]')
from_tuples takes a list of tuples made up of starts and ends:
| |
IntervalIndex([(1, 5), (3, 4), (6, 9), (10, 11)],
closed='neither',
dtype='interval[int64]')
An equally spaced interval series is determined by a start, an end, a count, and a step. Once any three of these are fixed, the remaining one follows; the start, end, periods, freq parameters of interval_range correspond to the four quantities, so the intervals can be constructed accordingly:
| |
IntervalIndex([(1.0, 1.5], (1.5, 2.0], (2.0, 2.5], (2.5, 3.0], (3.0, 3.5], (3.5, 4.0], (4.0, 4.5], (4.5, 5.0]],
closed='right',
dtype='interval[float64]')
| |
IntervalIndex([(1.0, 1.5], (1.5, 2.0], (2.0, 2.5], (2.5, 3.0], (3.0, 3.5], (3.5, 4.0], (4.0, 4.5], (4.5, 5.0]],
closed='right',
dtype='interval[float64]')
[Exercise]
Both interval_range here and date_range in the next chapter on time series fix three of the four elements of an arithmetic sequence and thereby determine the whole sequence. Recall how the first term, last term, number of terms, and common difference of an arithmetic progression are related, and write down the identity connecting the four parameters of interval_range.
[END]
In addition, if you call pd.IntervalIndex([...], closed=...) directly with a list of Interval objects, all the intervals are forced to the specified closed type, since pd.IntervalIndex only stores Interval objects with the same closure state.
| |
Interval(0, 1, closed='right')
| |
Interval(0.5, 1.5, closed='left')
| |
IntervalIndex([[0.0, 1.0), [0.5, 1.5)],
closed='left',
dtype='interval[float64]')
3. Interval Attributes and Methods
IntervalIndex also defines several useful attributes and methods. Moreover, to analyze the results of cut or qcut in detail, you first need to convert them to this index type:
| |
IntervalIndex([(33.945, 52.333], (52.333, 70.667], (70.667, 89.0]],
closed='right',
name='Weight',
dtype='interval[float64]')
Like a single Interval, IntervalIndex has several common attributes: left, right, mid, length, which give the left and right endpoints, their average, and the interval length respectively.
| |
IntervalIndex([(33.945, 52.333], (52.333, 70.667], (70.667, 89.0], (33.945, 52.333], (70.667, 89.0]],
closed='right',
name='Weight',
dtype='interval[float64]')
| |
Float64Index([33.945, 52.333, 70.667, 33.945, 70.667], dtype='float64')
| |
Float64Index([52.333, 70.667, 89.0, 52.333, 89.0], dtype='float64')
| |
Float64Index([43.138999999999996, 61.5, 79.8335, 43.138999999999996, 79.8335], dtype='float64')
| |
Float64Index([18.387999999999998, 18.334000000000003, 18.333,
18.387999999999998, 18.333],
dtype='float64')
IntervalIndex also has two commonly used methods, contains and overlaps, which test respectively whether each interval contains an element and whether it intersects a pd.Interval object.
| |
array([False, False, False, False, False])
| |
array([ True, True, False, True, False])
IV. Exercises
Ex1: Counting Categories That Do Not Appear
Chapter 5 introduced crosstab, which with default parameters tabulates the frequencies of combinations of two columns:
| |
| B | cat | dog |
|---|---|---|
| A | ||
| a | 2 | 0 |
| b | 1 | 0 |
| c | 0 | 1 |
In fact, some columns store categorical variables that do not necessarily contain every category. If you want these absent categories to appear in the crosstab result as well, set the dropna parameter to False:
| |
| B | cat | dog | sheep |
|---|---|---|---|
| A | |||
| a | 2 | 0 | 0 |
| b | 1 | 0 | 0 |
| c | 0 | 1 | 0 |
Implement a my_crosstab function with a dropna parameter that reproduces the behavior above.
Ex2: The Diamonds Dataset
Here is a dataset about diamonds, in which carat, cut, clarity, price stand for carat weight, cut quality, clarity, and price respectively. A sample follows:
| |
| carat | cut | clarity | price | |
|---|---|---|---|---|
| 0 | 0.23 | Ideal | SI2 | 326 |
| 1 | 0.21 | Premium | SI1 | 326 |
| 2 | 0.23 | Good | VS1 | 327 |
- Apply
nuniquetodf.cutunder theobjecttype and thecategorytype respectively, then compare their performance. - Cut quality has five grades, from worst to best
Fair, Good, Very Good, Premium, Ideal; clarity has eight, from worst to bestI1, SI2, SI1, VS2, VS1, VVS2, VVS1, IF. Sort cut quality from best to worst, and for diamonds of equal cut quality sort from worst to best by clarity. - Using two different methods, map the
cut, claritycolumns from best to worst to integers from 0 to n-1, where n denotes the number of categories. - Bin the price per carat into five categories
Very Low, Low, Mid, High, Very High, using quantiles (q= \[0.2, 0.4, 0.6, 0.8\]) and the split points \[1000, 3500, 5500, 18000\], then append the resultingcategoryseries to the original table in turn. - In the series obtained from integer binning in question 4, do all categories appear? If any category is absent, remove it.
- For the series obtained from quantile binning in question 4, find the left and right endpoints and the length of the interval containing each sample.
