我遇到的情况与中描述的情况非常相似这个问题 https://stackoverflow.com/questions/17236456/limiting-the-number-of-subshells-spawned(但是是批量的,不是外壳)。我制作了一个简单的批处理脚本来迭代图块的行并使用 python 脚本从服务器下载数据(该过程本身比简单的下载更复杂,它必须使用 API 进行身份验证并获取多个 URL)。
第一个版本如下:
for /F "tokens=*" %%A in (client_name_list.txt) do python download_metadata.py "%%A"
它的方式是等到每次迭代完成才能继续,所以我将其更新为以下内容:
for /F "tokens=*" %%A in (client_name_list.txt) do start cmd /C python download_metadata.py "%%A"
第二个版本做了我想做的事情,但是,由于文件 client_name_list.txt 大约有 30,000 行长,许多命令提示符开始出现,并且计算机在几秒钟内冻结。
如何限制 CMD 运行实例的数量(例如 10 个)并使脚本等待,直到有“空闲 CMD 插槽”才能进入下一行?
改编自我对“shell进程的并行执行”的回答 https://stackoverflow.com/a/11715437/1012053。请点击链接获取解释。
@echo off
setlocal enableDelayedExpansion
:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
set "lockHandle=1"
set "showOutput=1"
) else (
set "lockHandle=1^>nul 9"
set "showOutput="
)
:: Define the maximum number of parallel processes to run.
set "maxProc=10"
:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
set "lock="
for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
set "lock=%%T"
goto :break
)
:break
set "lock=%temp%\lock%lock%_%random%_"
:: Initialize the counters
set /a "startCount=0, endCount=0"
:: Clear any existing end flags
for /l %%N in (1 1 %maxProc%) do set "endProc%%N="
:: Launch the commands in a loop
set launch=1
for /f "delims=" %%A in (client_name_list.txt) do (
if !startCount! lss %maxProc% (
set /a "startCount+=1, nextProc=startCount"
) else (
call :wait
)
set cmd!nextProc!=%%A
if defined showOutput echo -------------------------------------------------------------------------------
echo !time! - proc!nextProc!: starting %%A
2>nul del %lock%!nextProc!
%= Redirect the lock handle to the lock file. The CMD process will =%
%= maintain an exclusive lock on the lock file until the process ends. =%
start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 python download_metadata.py "%%A"
)
set "launch="
:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
:: redirect stderr to null to suppress any error message if redirection
:: within the loop fails.
for /l %%N in (1 1 %startCount%) do 2>nul (
%= Redirect an unused file handle to the lock file. If the process is =%
%= still running then redirection will fail and the IF body will not run =%
if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
%= Made it inside the IF body so the process must have finished =%
if defined showOutput echo ===============================================================================
echo !time! - proc%%N: finished !cmd%%N!
if defined showOutput type "%lock%%%N"
if defined launch (
set nextProc=%%N
exit /b
)
set /a "endCount+=1, endProc%%N=1"
)
)
if %endCount% lss %startCount% (
1>nul 2>nul ping /n 2 ::1
goto :wait
)
2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Done
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)