我用过这个script https://stackoverflow.com/questions/13121790/using-multiprocessing-manager-list-instead-of-a-real-list-makes-the-calculation(参见最后的代码)来评估当父进程被分叉时全局对象是否被共享或复制。
简而言之,该脚本创建了一个全局data
对象,子进程迭代data
。该脚本还监视内存使用情况,以评估对象是否在子进程中被复制。
结果如下:
-
data = np.ones((N,N))
。子进程中的操作:data.sum()
。结果:data
is shared(无副本)
-
data = list(range(pow(10, 8)))
。子进程中的操作:sum(data)
。结果:data
is copied.
-
data = list(range(pow(10, 8)))
。子进程中的操作:for x in data: pass
。结果:data
is copied.
由于写时复制,结果 1) 是预期的结果。我对结果 2) 和 3) 有点困惑。为什么是data
copied?
Script
source https://stackoverflow.com/questions/13121790/using-multiprocessing-manager-list-instead-of-a-real-list-makes-the-calculation
import multiprocessing as mp
import numpy as np
import logging
import os
logger = mp.log_to_stderr(logging.WARNING)
def free_memory():
total = 0
with open('/proc/meminfo', 'r') as f:
for line in f:
line = line.strip()
if any(line.startswith(field) for field in ('MemFree', 'Buffers', 'Cached')):
field, amount, unit = line.split()
amount = int(amount)
if unit != 'kB':
raise ValueError(
'Unknown unit {u!r} in /proc/meminfo'.format(u = unit))
total += amount
return total
def worker(i):
x = data.sum() # Exercise access to data
logger.warn('Free memory: {m}'.format(m = free_memory()))
def main():
procs = [mp.Process(target = worker, args = (i, )) for i in range(4)]
for proc in procs:
proc.start()
for proc in procs:
proc.join()
logger.warn('Initial free: {m}'.format(m = free_memory()))
N = 15000
data = np.ones((N,N))
logger.warn('After allocating data: {m}'.format(m = free_memory()))
if __name__ == '__main__':
main()
详细结果
运行 1 输出
[WARNING/MainProcess] Initial free: 25.1 GB
[WARNING/MainProcess] After allocating data: 23.3 GB
[WARNING/Process-2] Free memory: 23.3 GB
[WARNING/Process-4] Free memory: 23.3 GB
[WARNING/Process-1] Free memory: 23.3 GB
[WARNING/Process-3] Free memory: 23.3 GB
运行2输出
[WARNING/MainProcess] Initial free: 25.1 GB
[WARNING/MainProcess] After allocating data: 21.9 GB
[WARNING/Process-2] Free memory: 12.6 GB
[WARNING/Process-4] Free memory: 12.7 GB
[WARNING/Process-1] Free memory: 16.3 GB
[WARNING/Process-3] Free memory: 17.1 GB
运行3输出
[WARNING/MainProcess] Initial free: 25.1 GB
[WARNING/MainProcess] After allocating data: 21.9 GB
[WARNING/Process-2] Free memory: 12.6 GB
[WARNING/Process-4] Free memory: 13.1 GB
[WARNING/Process-1] Free memory: 14.6 GB
[WARNING/Process-3] Free memory: 19.3 GB