Apify and Crawlee Official Forum

Updated last month

enqueue urls / request queue not being unique

I'm seeing a lot of the same exact URL's being ran twice? Any ideas?
M
1 comment
Hi. Hm...that is not the default behavior πŸ€”.
The equality of two requests is determined via comparison of their uniqueKey properties. By changing this property value, you could be able to process the same URLs multiple times (if their uniqueKey values were unique ofc). Do you explicitly set this property?

See this.
Add a reply
Sign up and join the conversation on Discord