Apify and Crawlee Official Forum

Updated 11 months ago

How to rotate non-random sessions within a crawler?

At a glance

How to make a crawler use non-random input based session ID while still making it possible to rotate/randomize and persist the session when needed (e.g. bad/blocked proxy)? E.g. first use session ID email and if the session is retired and request retried then use session ID email-randomNumber etc.

3 comments

TTobey

Note that this will hang the process:

Plain Text

createSessionFunction: async sessionPool => await sessionPool.getSession(input.id) || new Session({id: input.id, sessionPool}),

TTobey

and this will create duplicit sessions:

Plain Text

createSessionFunction: sessionPool => new Session({id: input.id, sessionPool}),

TTobey

Let me simplify the question: How to supply a crawler with a custom session after checking it does not already exist?

Add a reply