我觉得很奇怪你得到了better性能使用Counter
。这是我的测试结果(n=10000
):
Using Series.mode on Series with nan: 52.41649858
Using Series.mode on Series without nan: 17.186453438
Using Counter on Series with nan: 269.33117825500005
Using Counter on Series without nan: 134.207576572
#-----------------------------------------------------#
Series.mode Counter
----------- -------------
With nan 52.42s 269.33s
Without nan 17.19s 134.21s
测试代码:
import timeit
setup = '''
import pandas as pd
from collections import Counter
def get_most_common(srs):
return srs.mode(dropna=False)[0]
def get_most_common_counter(srs):
x = list(srs)
my_counter = Counter(x)
return my_counter.most_common(1)[0][0]
df = pd.read_csv(r'large.data')
'''
print(f"""Using Series.mode on Series with nan: {timeit.timeit('get_most_common(df["has_nan"])', setup=setup, number=10000)}""")
print(f"""Using Series.mode on Series without nan: {timeit.timeit('get_most_common(df["no_nan"])', setup=setup, number=10000)}""")
print(f"""Using Counter on Series with nan: {timeit.timeit('get_most_common_counter(df["has_nan"])', setup=setup, number=10000)}""")
print(f"""Using Counter on Series without nan: {timeit.timeit('get_most_common_counter(df["no_nan"])', setup=setup, number=10000)}""")
large.data
是 2 x 50000 行DataFrame
随机 2 位数字字符串0
to 99
, where has_nan
has a mode
of nan=551
.
如果有的话,你的if np.nan not in my_counter.keys()
条件总是会被触发,因为np.nan
不在my_counter.keys()
。所以实际上你从未使用过pd.Series.mode
,它一直在使用Counter
。正如另一个问题中提到的,因为你的pandas
对象已经创建了副本np.nan
内Series/DataFrame
, the in
条件永远不会被满足。试一试:
np.nan in pd.Series([np.nan, 1, 2]).to_list()
# False
消除整个复杂性if/else
并坚持使用一种方法。然后比较性能。正如您在其他问题中提到的,pandas 方法几乎总是比任何外部模块/方法更好的方法。如果您仍在观察其他情况,请更新您的问题。