On 这一页,多处理模块的作者 Jesse Noller 展示了处理的正确方法KeyboardInterrupt
是让子进程返回——而不是重新引发异常。这允许主进程终止池。
然而,正如下面的代码所示,主进程并没有到达except KeyboardInterrupt
阻塞直到after生成的所有任务pool.map
已运行。这就是为什么(我相信)您会看到对工作函数的额外调用,run_nlin
, after Ctrl-C
已被按下。
一种可能的解决方法是测试所有工作函数是否存在multiprocessing.Event
已经设置好了。如果事件已定,则让工人尽早退出,否则继续进行长期计算。
import logging
import multiprocessing as mp
import time
logger = mp.log_to_stderr(logging.WARNING)
def worker(x):
try:
if not terminating.is_set():
logger.warn("Running worker({x!r})".format(x = x))
time.sleep(3)
else:
logger.warn("got the message... we're terminating!")
except KeyboardInterrupt:
logger.warn("terminating is set")
terminating.set()
return x
def initializer(terminating_):
# This places terminating in the global namespace of the worker subprocesses.
# This allows the worker function to access `terminating` even though it is
# not passed as an argument to the function.
global terminating
terminating = terminating_
def main():
terminating = mp.Event()
result = []
pool = mp.Pool(initializer=initializer, initargs=(terminating, ))
params = range(12)
try:
logger.warn("starting pool runs")
result = pool.map(worker, params)
pool.close()
except KeyboardInterrupt:
logger.warn("^C pressed")
pool.terminate()
finally:
pool.join()
logger.warn('done: {r}'.format(r = result))
if __name__ == '__main__':
main()
运行脚本会产生:
% test.py
[WARNING/MainProcess] starting pool runs
[WARNING/PoolWorker-1] Running worker(0)
[WARNING/PoolWorker-2] Running worker(1)
[WARNING/PoolWorker-3] Running worker(2)
[WARNING/PoolWorker-4] Running worker(3)
这里按下 Ctrl-C;每个工人设定terminating
事件。我们实际上只需要一个来设置它,但是尽管效率有点低,但这仍然有效。
C-c C-c[WARNING/PoolWorker-4] terminating is set
[WARNING/PoolWorker-2] terminating is set
[WARNING/PoolWorker-3] terminating is set
[WARNING/PoolWorker-1] terminating is set
现在所有其他任务都在排队pool.map
正在运行:
[WARNING/PoolWorker-4] got the message... we're terminating!
[WARNING/PoolWorker-2] got the message... we're terminating!
[WARNING/PoolWorker-1] got the message... we're terminating!
[WARNING/PoolWorker-2] got the message... we're terminating!
[WARNING/PoolWorker-4] got the message... we're terminating!
[WARNING/PoolWorker-2] got the message... we're terminating!
[WARNING/PoolWorker-1] got the message... we're terminating!
[WARNING/PoolWorker-3] got the message... we're terminating!
最终主流程到达except KeyboardInterrupt
block.
[WARNING/MainProcess] ^C pressed
[WARNING/MainProcess] done: []