Think of it like an unordered tuple (kind of)

If you frequently use Python sets and tuples, you might be familiar with the frozenset class, which is often underrated.

Python’s built-in frozenset class is an immutable and unordered collection of unique elements. It is similar to a set, but the key difference is that it cannot be modified after creation. These objects are hashable, making them suitable as elements in sets or keys in dictionaries. Their hashable nature also makes them valuable for grouping data or combining Pandas DataFrames based on sets of elements.

Quoting the Python documentation:

“The frozenset type is immutable and hashable — its contents cannot be altered after it is created; it can therefore be used as a dictionary key or as an element of another set.”

The Challenge

In a recent project, the task was to identify projects with the same set of members. Members were listed in a semicolon-separated string format, with names in varying orders across projects. The goal was to:

Identify projects with identical member lists, regardless of the order.
Assign a unique group ID to each unique set of members.
Replace the member lists in the project data with the corresponding group IDs.

For example:

"Doe, Jane; Robinson, George; White, Robert; Moore, Lisa; Brown, Charlie" # should equal "Doe, Jane; Robinson, George; Moore, Lisa; Brown, Charlie; White, Robert"

The Solution

To address this, Python’s frozenset was used to manage the immutability and order-agnostic requirements of member lists.

The Data

Here is a sample dataset:

| id  | project  | members                                                                           |
|-----|----------|-----------------------------------------------------------------------------------|
| 1   | Alpha    | Garcia, Kevin; Taylor, David; Smith, John; Miller, Michael; Martin, Christopher   |
| 2   | Beta     | Johnson, Emily; White, Robert; Martinez, Jennifer; Doe, Jane; Clark, Angela       |
| 3   | Gamma    | Harris, Mary; Wilson, James; White, Robert; Doe, Jane; Clark, Angela              |
| ... | ...      | ...                                                                               |

The Code

import pandas as pd
df = pd.read_csv("./path/to/file.csv")

# define a function to convert the members string to a frozen set
def get_frozen_set(members: str) -> frozenset:
    """
    Convert a semicolon-separated string of member names into a frozenset.
    """
    members_list = [x.strip() for x in members.split(';')]
    return frozenset(members_list)

# apply the function
df['members'] = df['members'].apply(get_frozen_set)

# group by the frozenset column
groups_df = (
    df
    .value_counts('members')
    .reset_index()
    .reset_index()
    .rename(columns={'index': 'group_id'})
)

# merge back to associate each project with its group
final_df = pd.merge(df, groups_df, on='members').drop(columns=['members', 'count'])

The data is now normalized, with each project linked to a group ID.

The frozenset also supports useful set operations such as:

isdisjoint
issubset
union
intersection
difference
and more

For more details, refer to the Python docs.

Conclusion

Incorporating frozenset into your data processing toolkit offers a simple and efficient way to handle unordered collections of unique elements. It enables you to join dataframes on set columns, group dataframes by set columns, and create indices from unordered sets. This hidden gem in Python’s standard library could prove to be an invaluable tool in your repertoire.

Blog