I have a massive CSV file which I can not fit all into memory at one time. How do I convert it to HDF5?
We are working to make this process an easy one liner. In the meantime, consider this strategy: read the CSV file in chunks, and use vaex to export each chunk to disk. Since all resulting HDF5 files will have the same structure, one can use vaex.open(part*) to open all chunks as a single DataFrame. For a small performance improvement, that DataFrame can be exported to disk in a single large HDF5 file.
Consider the following code example:
for i, chunk in enumerate(vaex.read_csv('/path/to/data/BigData.csv', chunksize=100_000)):
df_chunk = vaex.from_pandas(chunk, copy_index=False)
export_path = f'/path/to/data/part_{i}.hdf5'
df_chunk.export_hdf5(export_path)
df = vaex.open('/path/to/data/part*')
df.export_hdf5('/path/to/data/Final.hdf5')
https://www.leiphone.com/news/201912/pW63YGX6lJapjyf9.html
https://vaex.readthedocs.io/en/latest/faq.html#I-have-a-massive-CSV-file-which-I-can-not-fit-all-into-memory-at-one-time.-How-do-I-convert-it-to-HDF5?
本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)
