当尝试在我的服务器环境中使用 PlaywrightCrawler 包时,我收到以下错误日志。
[32mINFO[39m [33m PlaywrightCrawler:[39m Starting the crawl
[33mWARN[39m [33m PlaywrightCrawler:[39m Reclaiming failed request back to the list or queue. browserType.launchPersistentContext: Failed to launch: Error: spawn /root/.cache/ms-playwright/chromium-1033/chrome-linux/chrome ENOENT
============================================================
at async PlaywrightPlugin._launch (/usr/src/app/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)
at async BrowserPool._launchBrowser (/usr/src/app/node_modules/@crawlee/browser-pool/browser-pool.js:465:29)
at async /usr/src/app/node_modules/@crawlee/browser-pool/browser-pool.js:274:37[90m {"id":"erPiwuCRDH5Dzyr","url":"https://www.google.com.au/search?tbm=shop&hl=en-AU&psb=1&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL&q=Coles+Extra Virgin Avocado Oil 250mL&oq=Coles+Extra Virgin Avocado Oil 250mL&gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&sclient=products-cc&ts=658","method":"GET","uniqueKey":"https://www.google.com.au/search?gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&hl=en-AU&oq=Coles+Extra Virgin Avocado Oil 250mL&psb=1&q=Coles+Extra Virgin Avocado Oil 250mL&sclient=products-cc&tbm=shop&ts=658&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL"}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m All the requests from request list and/or request queue have been processed, the crawler will shut down.
[32mINFO[39m [33m PlaywrightCrawler:[39m Crawl finished. Final request statistics:[90m {"requestsFinished":0,"requestsFailed":1,"retryHistogram":[null,null,null,1],"requestAvgFailedDurationMillis":107,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":5,"requestTotalDurationMillis":107,"requestsTotal":1,"crawlerRuntimeMillis":11078}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m Error analysis:[90m {"totalErrors":1,"uniqueErrors":1,"mostCommonErrors":["1x: browserType.launchPersistentContext: Failed to launch: Error: spawn /root/.cache/ms-playwright/chromium-1033/chrome-linux/chrome ENOENT (/usr/src/app/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)"]}[39m
这是我的 .ebextension/chromium.config:
packages:
yum:
cups-libs: []
dbus-glib: []
libXrandr: []
libXcursor: []
libXinerama: []
cairo: []
cairo-gobject: []
pango: []
libwayland-client: []
libwayland-cursor: []
gtk3: []
gdk-pixbuf2: []
libxkbcommon: []
libXScrnSaver: []
GConf2: []
atk: []
at-spi2-atk: []
at-spi2-core: []
alsa-lib: []
container_commands:
install_playwright:
command: "npx playwright install --with-deps chromium"
以下是我的 yum.log 的输出:
Dec 08 10:06:18 Installed: libwayland-client-1.17.0-1.amzn2.x86_64
Dec 08 10:06:18 Installed: mesa-libglapi-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:18 Installed: atk-2.22.0-3.amzn2.0.2.x86_64
Dec 08 10:06:18 Installed: fontpackages-filesystem-1.44-8.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-fonts-common-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: libusbx-1.0.21-1.amzn2.x86_64
Dec 08 10:06:19 Installed: pixman-0.34.0-1.amzn2.0.2.x86_64
Dec 08 10:06:19 Installed: libxshmfence-1.2-1.amzn2.0.2.x86_64
Dec 08 10:06:19 Installed: 1:libglvnd-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
Dec 08 10:06:19 Installed: libwayland-server-1.17.0-1.amzn2.x86_64
Dec 08 10:06:19 Installed: mesa-libgbm-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:19 Installed: libgusb-0.2.9-1.amzn2.0.2.x86_64
Dec 08 10:06:19 Installed: 1:liberation-mono-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-serif-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-narrow-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-sans-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: dejavu-fonts-common-2.33-6.amzn2.noarch
Dec 08 10:06:19 Installed: dejavu-sans-fonts-2.33-6.amzn2.noarch
Dec 08 10:06:20 Installed: fontconfig-2.13.0-4.3.amzn2.x86_64
Dec 08 10:06:21 Installed: libwayland-cursor-1.17.0-1.amzn2.x86_64
Dec 08 10:06:21 Installed: graphite2-1.3.10-1.amzn2.0.2.x86_64
Dec 08 10:06:21 Installed: harfbuzz-1.7.5-2.amzn2.x86_64
Dec 08 10:06:21 Installed: vulkan-filesystem-1.0.61.1-2.amzn2.noarch
Dec 08 10:06:21 Installed: jasper-libs-1.900.1-33.amzn2.x86_64
Dec 08 10:06:21 Installed: alsa-lib-1.1.4.1-2.amzn2.x86_64
Dec 08 10:06:21 Installed: fribidi-1.0.2-1.amzn2.1.x86_64
Dec 08 10:06:21 Installed: libepoxy-1.3.1-2.amzn2.x86_64
Dec 08 10:06:21 Installed: json-glib-1.4.2-2.amzn2.x86_64
Dec 08 10:06:21 Installed: hicolor-icon-theme-0.12-7.amzn2.noarch
Dec 08 10:06:21 Installed: xkeyboard-config-2.20-1.amzn2.noarch
Dec 08 10:06:21 Installed: libxkbcommon-0.7.1-3.amzn2.x86_64
Dec 08 10:06:22 Installed: dconf-0.28.0-4.amzn2.x86_64
Dec 08 10:06:22 Installed: gsettings-desktop-schemas-3.28.0-3.amzn2.0.1.x86_64
Dec 08 10:06:22 Installed: avahi-libs-0.6.31-20.amzn2.x86_64
Dec 08 10:06:22 Installed: 1:cups-libs-1.6.3-51.amzn2.x86_64
Dec 08 10:06:22 Installed: libthai-0.1.14-9.amzn2.0.2.x86_64
Dec 08 10:06:22 Installed: trousers-0.3.14-2.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: gnutls-3.3.29-9.amzn2.0.1.x86_64
Dec 08 10:06:23 Installed: libXau-1.0.8-2.1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libxcb-1.12-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: vulkan-1.0.61.1-2.amzn2.x86_64
Dec 08 10:06:23 Installed: libX11-common-1.6.7-3.amzn2.0.2.noarch
Dec 08 10:06:23 Installed: libX11-1.6.7-3.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXext-1.3.3-3.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXrender-0.9.10-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXfixes-5.0.3-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXdamage-1.1.4-4.1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXrandr-1.5.1-2.amzn2.0.3.x86_64
Dec 08 10:06:23 Installed: libXi-1.7.9-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXcomposite-0.4.4-4.1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: gdk-pixbuf2-2.36.12-3.amzn2.x86_64
Dec 08 10:06:24 Installed: gtk-update-icon-cache-3.22.30-3.amzn2.x86_64
Dec 08 10:06:24 Installed: libXtst-1.2.3-1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: at-spi2-core-2.22.0-1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: at-spi2-atk-2.22.0-2.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: libXcursor-1.1.15-1.amzn2.x86_64
Dec 08 10:06:24 Installed: libXft-2.3.2-2.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: libXinerama-1.1.3-2.1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: libXxf86vm-1.1.4-1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: 1:libglvnd-egl-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: mesa-libEGL-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: cairo-1.15.12-4.amzn2.x86_64
Dec 08 10:06:24 Installed: pango-1.42.4-4.amzn2.x86_64
Dec 08 10:06:24 Installed: cairo-gobject-1.15.12-4.amzn2.x86_64
Dec 08 10:06:24 Installed: libwayland-egl-1.17.0-1.amzn2.x86_64
Dec 08 10:06:24 Installed: 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch
Dec 08 10:06:24 Installed: desktop-file-utils-0.23-2.amzn2.x86_64
Dec 08 10:06:25 Installed: xdg-utils-1.1.0-0.17.20120809git.amzn2.noarch
Dec 08 10:06:25 Installed: lcms2-2.6-3.amzn2.0.2.x86_64
Dec 08 10:06:25 Installed: colord-libs-1.3.4-1.amzn2.0.2.x86_64
Dec 08 10:06:25 Installed: adwaita-cursor-theme-3.26.0-1.amzn2.noarch
Dec 08 10:06:28 Installed: adwaita-icon-theme-3.26.0-1.amzn2.noarch
Dec 08 10:06:28 Installed: libmodman-2.0.1-8.amzn2.0.2.x86_64
Dec 08 10:06:28 Installed: libproxy-0.4.11-10.amzn2.0.3.x86_64
Dec 08 10:06:28 Installed: glib-networking-2.56.1-1.amzn2.x86_64
Dec 08 10:06:28 Installed: libsoup-2.56.0-6.amzn2.x86_64
Dec 08 10:06:28 Installed: rest-0.8.0-2.amzn2.x86_64
Dec 08 10:06:29 Installed: gtk3-3.22.30-3.amzn2.x86_64
Dec 08 10:06:43 Installed: google-chrome-stable-108.0.5359.98-1.x86_64
即使所有的 chromium 依赖项都已安装,仍然遇到问题。
本地运行就没有这个问题。
希望有人能指教是什么问题。
UPDATE:
经过进一步研究发现问题可能出在 Docker 镜像上。所以我将现有的 Docker 镜像替换为[zenika/alpine-chrome:with-playwright][1]
其中有剧作家和无头铬,并包括一个新的运行命令npx playwright install
。现在出现一个新错误:
[31mERROR[39m[33m PlaywrightCrawler:[39m Request failed and reached maximum retries. browserType.launchPersistentContext: Executable doesn't exist at /home/chrome/.cache/ms-playwright/chromium-1028/chrome-linux/chrome
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just installed or updated. ║
║ Please run the following command to download new browsers: ║
║ ║
║ npx playwright install ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════════════════════════════════════╝
at async PlaywrightPlugin._launch (/usr/src/app/node_modules/crawlee/node_modules/@crawlee/playwright/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)
at async BrowserPool._launchBrowser (/usr/src/app/node_modules/@crawlee/browser/node_modules/@crawlee/browser-pool/browser-pool.js:465:29)
at async /usr/src/app/node_modules/@crawlee/browser/node_modules/@crawlee/browser-pool/browser-pool.js:274:37[90m {"id":"tGOZOvUIBmhYpXW","url":"https://www.google.com.au/search?tbm=shop&hl=en-AU&psb=1&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL&q=Coles+Extra Virgin Avocado Oil 250mL&oq=Coles+Extra Virgin Avocado Oil 250mL&gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&sclient=products-cc&ts=782","method":"GET","uniqueKey":"https://www.google.com.au/search?gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&hl=en-AU&oq=Coles+Extra Virgin Avocado Oil 250mL&psb=1&q=Coles+Extra Virgin Avocado Oil 250mL&sclient=products-cc&tbm=shop&ts=782&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL"}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m All the requests from request list and/or request queue have been processed, the crawler will shut down.
[32mINFO[39m [33m PlaywrightCrawler:[39m Crawl finished. Final request statistics:[90m {"requestsFinished":0,"requestsFailed":1,"retryHistogram":[null,null,null,1],"requestAvgFailedDurationMillis":18,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":5,"requestTotalDurationMillis":18,"requestsTotal":1,"crawlerRuntimeMillis":11089}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m Error analysis:[90m {"totalErrors":1,"uniqueErrors":1,"mostCommonErrors":["1x: browserType.launchPersistentContext: Executable doesn't exist at /home/chrome/.cache/ms-playwright/chromium-1028/chrome-linux/chrome (/usr/src/app/node_modules/crawlee/node_modules/@crawlee/playwright/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)"]}[39m
这是我的 Dockerfile:
# # This file is a template, and might need editing before it works on your project.
FROM zenika/alpine-chrome:with-playwright
WORKDIR /usr/src/app
ENV PORT 3000
COPY package.json /usr/src/app/
RUN npm install --force
RUN npx playwright install
COPY . /usr/src/app
# replace this with your application's default port
EXPOSE 3000
CMD [ "npm", "start" ]