Action将两个 csv(data.csv 和 label.csv)读取到单个数据帧。
df = dd.read_csv(data_files, delimiter=' ', header=None, names=['x', 'y', 'z', 'intensity', 'r', 'g', 'b'])
df_label = dd.read_csv(label_files, delimiter=' ', header=None, names=['label'])
Problem列的串联需要已知的划分。然而,设置索引会对数据进行排序,这是我明确不希望的,因为两个文件的顺序是它们的匹配。
df = dd.concat([df, df_label], axis=1)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-11-e6c2e1bdde55> in <module>()
----> 1 df = dd.concat([df, df_label], axis=1)
/uhome/hemmest/.local/lib/python3.5/site-packages/dask/dataframe/multi.py in concat(dfs, axis, join, interleave_partitions)
573 return concat_unindexed_dataframes(dfs)
574 else:
--> 575 raise ValueError('Unable to concatenate DataFrame with unknown '
576 'division specifying axis=1')
577 else:
ValueError: Unable to concatenate DataFrame with unknown division specifying axis=1
Tried添加一个'id'
column
df['id'] = pd.Series(range(len(df)))
然而,Dataframe 的长度导致 Series 大于内存。
Question显然 Dask 知道两个 Dataframe 具有相同的长度:
In [15]:
df.index.compute()
Out[15]:
Int64Index([ 0, 1, 2, 3, 4, 5, 6,
7, 8, 9,
...
1120910, 1120911, 1120912, 1120913, 1120914, 1120915, 1120916,
1120917, 1120918, 1120919],
dtype='int64', length=280994776)
In [16]:
df_label.index.compute()
Out[16]:
Int64Index([1, 5, 5, 2, 2, 2, 2, 2, 2, 2,
...
3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
dtype='int64', length=280994776)
如何利用这些知识来简单地连接?