我正在尝试使用以下命令写入大小在 1kb 到 10GB 之间的文件ByteArrayOutputStream
但抛出以下异常。我正在使用jdk 6。请建议任何更好的高性能Api。我使用同一个网络盒来读取和写入。
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at java.io.OutputStream.write(Unknown Source)
at
Code:
import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
public class PrepareFile {
/**
* @param args
* @throws Exception
*/
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
new PrepareFile().constructFile("f:\\hello","f:\\output",10000000);
}
//Writes a large file of 10 GB using input file data of small size by duplicating
public void constructFile(String fileName, String outPath, int multiplier) throws Exception {
BufferedOutputStream fos = null;
FileInputStream fis = null;
final File inputFile = new File(fileName);
String path = inputFile.getParent();
if (outPath != null && !outPath.isEmpty()) {
path = outPath;
}
fis = new FileInputStream(fileName);
try {
// read the transactions in the input file.
byte[] txnData = new byte[(int) inputFile.length()];
fis.read(txnData);
final File outFile = new File(path, "Myfile");
fos = new BufferedOutputStream(new FileOutputStream(outFile));
final ByteArrayOutputStream baos = new ByteArrayOutputStream();
final ByteArrayOutputStream baos1 = new ByteArrayOutputStream();
//multiplier if input file size is 1 KB and output file is 10 GB, then multiplier value is (1024*1024)
for (long i = 1; i <= multiplier; i++) {
if(i >=40000 && i % 40000==0){
System.out.println("i value now: "+i);
baos.writeTo(fos);
baos.reset();
//baos.write(txnData);
}
// write transactions
baos.write(txnData);
baos1.write(txnData); //Exception is coming at this line
}
int Padding = myCustomMethod(baos1.toByteArray());
// write all out data to the output stream
baos.writeTo(fos);
baos.flush();
baos1.flush();
} catch(Exception e){
e.printStackTrace();
}finally {
fos.close();
fis.close();
}
}
public int myCustomMethod(byte[] b){
//Need complete bytes to prepare the file trailer
return 0;
}
}
您不能在内存中拥有 2 GB 或更多的缓冲区ByteArrayOutputStream
因为大小是 32 位有符号的。
如果您想要性能,我会逐步处理文件并避免如此大的内存副本,因为它们非常昂贵。
顺便说一句,我有一个图书馆编年史字节 https://github.com/OpenHFT/Chronicle-Bytes它支持大于 2 GB 的缓冲区,可以使用本机内存并映射到文件以避免使用堆,并且可以大于主内存。
但是,如果您逐步处理数据,则不需要这么大的缓冲区。
我还建议您使用 Java 8,因为它比 Java 6(十年前发布)执行 64 位操作更好
编辑根据您的代码,无需使用 ByteArrayOutputStream,您可以逐步准备文件。
//Writes a large file of 10 GB using input file data of small size by duplicating
public void constructFile(String fileName, String outFileName, int multiplier) throws IOException {
byte[] bytes;
try (FileInputStream fis = new FileInputStream(fileName)) {
bytes = new byte[fis.available()];
fis.read(bytes);
}
try (FileOutputStream fos = new FileOutputStream(outFileName)) {
for (int i = 0; i < multiplier; i++) {
fos.write(bytes);
}
}
// now process the file "outFileName"
// how depends on what you are trying to do.
// NOTE: It is entirely possible the file should be processed as it is written.
}
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)