这是错误消息:
<--- Last few GCs --->
[2383:0x7efe08001450] 6100 ms: Scavenge 30.3 (39.5) -> 30.5 (42.7) MB, 73.5 / 0.0 ms (average mu = 1.000, curr
ent mu = 1.000) allocation failure
[2383:0x7efe08001450] 8464 ms: Scavenge 35.1 (44.5) -> 35.3 (44.8) MB, 2336.2 / 0.0 ms (average mu = 1.000, cu
rrent mu = 1.000) allocation failure
[2383:0x7efe08001450] 32349 ms: Scavenge 36.1 (44.8) -> 36.0 (45.8) MB, 23879.5 / 0.2 ms (average mu = 1.000, c
urrent mu = 1.000) allocation failure
<--- JS stacktrace --->
FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory
1: 0xa2b020 node::Abort() [node]
2: 0x97a467 node::FatalError(char const*, char const*) [node]
3: 0xb9e0ee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb9e467 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd3e875 [node]
6: 0xdab25e [node]
7: 0xdae060 v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) [node]
8: 0xdaeb29 v8::internal::ScavengingTask::ProcessItems() [node]
9: 0xdaecd1 v8::internal::ScavengingTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
10: 0xd590f9 v8::internal::ItemParallelJob::Run() [node]
11: 0xdaf9c5 v8::internal::ScavengerCollector::CollectGarbage() [node]
12: 0xd37aba v8::internal::Heap::Scavenge() [node]
13: 0xd4d37b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [nod
e]
14: 0xd4de65 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReaso
n, v8::GCCallbackFlags) [node]
15: 0xd5082c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal
::AllocationOrigin, v8::internal::AllocationAlignment) [node]
16: 0xd20e82 v8::internal::Factory::AllocateRawWithAllocationSite(v8::internal::Handle<v8::internal::Map>, v8::inte
rnal::AllocationType, v8::internal::Handle<v8::internal::AllocationSite>) [node]
17: 0xd20f38 v8::internal::Factory::NewJSObjectFromMap(v8::internal::Handle<v8::internal::Map>, v8::internal::Alloc
ationType, v8::internal::Handle<v8::internal::AllocationSite>) [node]
18: 0xd2130e v8::internal::Factory::NewJSArrayWithUnverifiedElements(v8::internal::Handle<v8::internal::FixedArrayB
ase>, v8::internal::ElementsKind, int, v8::internal::AllocationType) [node]
19: 0xd24a32 v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllo
cationMode, v8::internal::AllocationType) [node]
20: 0xf6edbb v8::internal::ValueDeserializer::ReadDenseJSArray() [node]
21: 0xf6f1cd v8::internal::ValueDeserializer::ReadObjectInternal() [node]
22: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
23: 0xf6d80d v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>,
v8::internal::SerializationTag, bool) [node]
24: 0xf6ec06 v8::internal::ValueDeserializer::ReadSparseJSArray() [node]
25: 0xf6f118 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
26: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
27: 0xf6e186 v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>,
v8::internal::SerializationTag, bool) [node]
28: 0xf6e959 v8::internal::ValueDeserializer::ReadJSObject() [node]
29: 0xf6f140 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
30: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
31: 0xf6d80d v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>,
v8::internal::SerializationTag, bool) [node]
32: 0xf6e959 v8::internal::ValueDeserializer::ReadJSObject() [node]
33: 0xf6f140 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
34: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
35: 0xba6e15 v8::ValueDeserializer::ReadValue(v8::Local<v8::Context>) [node]
36: 0xa720ff node::worker::Message::Deserialize(node::Environment*, v8::Local<v8::Context>) [node]
37: 0xa72d86 node::worker::MessagePort::ReceiveMessage(v8::Local<v8::Context>, bool) [node]
38: 0xa72ee4 node::worker::MessagePort::OnMessage() [node]
39: 0x13247e9 [node]
40: 0x1336c50 [node]
41: 0x13250f8 uv_run [node]
42: 0xae3533 node::worker::Worker::Run() [node]
43: 0xae3fc8 [node]
44: 0x7eff53bdafa3 [/lib/x86_64-linux-gnu/libpthread.so.0]
45: 0x7eff53b0b4cf clone [/lib/x86_64-linux-gnu/libc.so.6]
GCP 上的节点服务器 - Linux x64、96 核、135GB RAM。我正在处理一个包含 100.000.000 个项目的数组,分为 95 个核心并创建工作线程。
这只是我得到的错误类型之一,我得到了NewArray Allocation failed
以及例如。我在玩弄工人的数量,--max-old-space-size
, --max-semi-space-size
,但没有一个有效。最有趣的是我的 mac 有 6 核,16GB 内存(只有 10 个给节点)运行良好,只是速度慢,随着硬件资源的增加,我期待更快的结果。
这是我第一次尝试处理这么大量的数据,并且使用节点进行多线程,我做错了什么?