我有许多大型(13GB+大小)h5 文件,每个 h5 文件都有两个用 pandas 创建的数据集:
df.to_hdf('name_of_file_to_save', 'key_1',table=True)
df.to_hdf('name_of_file_to_save', 'key_2', table=True) # saved to the same h5 file as above
我在这里看到过一个帖子:
连接两个big pandas.HDFStore HDF5文件 https://stackoverflow.com/questions/28918851/concatenate-two-big-pandas-hdfstore-hdf5-files
关于使用 odo 连接 h5 文件。我想要做的是对于创建的每个 h5 文件,每个文件都有key_1
and key_2
,将它们组合起来,使得所有的key_1
数据位于新 h5 文件中的一个数据集中,并且所有key_2
位于同一个新 h5 文件的另一个数据集中。所有的key_1
具有相同的列数,同样适用于key_2
我试过这个:
from odo import odo
files = ['file1.h5','file2.h5','file3.h5','file4.h5']
for i in files:
odo('hdfstore://path_to_here_h5_files_live/%s::key_1' % i,
'hdfstore://path_store_new_large_h5::key_1')
但是我收到错误:
(tables/hdf5extension.c:7824)
tables.exceptions.HDF5ExtError: HDF5 error back trace
File "H5A.c", line 259, in H5Acreate2
unable to create attribute
File "H5Aint.c", line 275, in H5A_create
unable to create attribute in object header
File "H5Oattribute.c", line 347, in H5O_attr_create
unable to create new attribute in header
File "H5Omessage.c", line 224, in H5O_msg_append_real
unable to create new message
File "H5Omessage.c", line 1945, in H5O_msg_alloc
unable to allocate space for message
File "H5Oalloc.c", line 1142, in H5O_alloc
object header message is too large
End of HDF5 error back trace
Can't set attribute 'non_index_axes' in node:
/key_1 (Group) ''.
Closing remaining open