如果没有看到完整的代码,我可以想象你正在调用write_csv
多次,因为您试图将多个对象写入该文件。
问题是你第一次打电话时write_csv
你正在结束writer
,这就是为什么你第二次调用它时会得到:
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
function write_csv(obj) {
writer.pipe(fs.createWriteStream('out.csv', { flags: 'a' }))
writer.write()
writer.end();
}
write_csv({ hello: 'world', foo: 'bar', baz: 'taco'});
// When you call it again, writer.end(); is already closed
// The following line will trigger the error
write_csv({ hello: 'world', foo: 'bar', baz: 'taco'});
相反,您应该做的是仅在完成写入后才关闭写入器。
const writer = csvWriter(); // Create new writer
// open file
writer.pipe(fs.createWriteStream('out.csv', { flags: 'a' }));
for(const obj of objects) // Write as many times as you wish
writer.write(obj);
writer.end(); // I'm done writing.
现在你遇到的问题是,如果你尝试执行多个.writes
你将达到内存限制,因为你没有处理背压正确。
我建议阅读以下问题:
为什么尝试写入大文件会导致 js 堆内存不足
要处理这个问题,您需要等待drain
要发出的事件。
这是一个包装器csvWriter
这将处理背压。
const fs = require('fs');
const csvWriter = require('csv-write-stream');
class Writer {
constructor(file) {
this.writer = csvWriter();
this.writer.pipe(fs.createWriteStream(file, { flags: 'a' }));
}
write(obj) {
// if .write returns false we have to wait until `drain` is emitted
if(!this.writer.write(obj))
return new Promise(resolve => this.writer.once('drain', resolve))
return true;
}
end() {
// Wrap it in a promise if you wish to wait for the callback.
this.writer.end();
}
}
(async() => {
const writer = new Writer('out.csv');
for(let i = 0; i < 1e8; i++) {
const res = writer.write({ hello: 'world', foo: 'bar', baz: 'taco' });
if(res instanceof Promise) {
// You can remove this if, and leave just: await writer.write...
// but the code will be slower
await res; // This will wait for the stream to emit the drain event
}
}
writer.end();
})();
Update:现在有了实际的代码,上面的答案仍然有效,但是因为您在收到请求时正在写入文件。您可以选择是否打开文件一次,并在每个请求上写入,在服务器关闭时(或您选择时)关闭它。或者只是打开文件,写入文件,然后根据每个请求关闭它,
对于前者,您应该使用上面的答案,对于后者,您需要做的就是每次调用时创建一个新的 writerwrite_csv
而不是只有一位全球作家。
function write_csv(obj) {
// Create a new writer every time
const writer = csvWriter({ sendHeaders: false });
writer.pipe(fs.createWriteStream('out.csv', { flags: 'a' }));
writer.write(obj);
writer.end();
};