Apify

Apify and Crawlee Official Forum

Home
Members
Stewart McKinlay (CAL)
S
Stewart McKinlay (CAL)
Offline, last seen 4 months ago
Joined August 30, 2024
SStewart McKinlay (CAL)
·

Enqueue Links from new Window

Hi. I'm attempting to scrape data from a website, using crawlee and playwright, that is very JS heavy. The links I'm interested in are created by a JS function that opens the content in a new window. I've implemented the enqueueLinksByClickingElements function with a very specific selector. Playwright reports successfully clicking the links but I suspect the request is not being intercepted.
Plain Text
DEBUG Playwright Click Elements: enqueueLinksByClickingElements: There are 1 elements to click.
DEBUG Playwright Click Elements: enqueueLinksByClickingElements: Successfully clicked 1 elements out of 1
DEBUG PlaywrightCrawler: Crawled 1/2 pages, 0 failed requests.
DEBUG PlaywrightCrawler: Crawled 1/2 pages, 0 failed requests.
INFO  PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.

I've also passed in the transformRequestFunction to set useExtendedUniqueKey to true. Is there a way I can:
  1. Take a screenshot after Playwright clicks the element?
  2. Log the intercepted requests?
Thanks!
4 comments
L
S
o
O
SStewart McKinlay (CAL)
·

request queue data

I'd like to access the request queue data that's stored in memory and being written to the storage directory.
I want to add data to my output such as the url, handledAt, etc. How would I go about that? Thanks.
2 comments
S
o