Apify and Crawlee Official Forum

Updated 3 months ago

How does createSessionFunction create session when parallel requests are being made

I have a custom function which open a browser to get cookies. The problem is my machine is very small and what would happen when multiple sessions are being made it will try to open many browsers at the same time. Can I somehow make the creation of sessions sequential ? so even though I need 1000s of sessions but at any point in time only one session is created and no session can be created in parallel. So only one browser instance will be running at any point in time.

createSessionFunction: async(sessionPool,options) => { var new_session = new Session({ sessionPool }); var proxyurl = await proxyConfigurationDE.newUrl(new_session.id); console.log(proxyurl, new_session.id); var cookies_g = await getCookiesDE(proxyurl,'https://www.example.com'); console.log("cookies from playwright..",cookies_g); new_session.setCookies( cookies_g, 'https://www.example.com' ); console.log("Checking cookies set..", new_session.getCookieString("example.com")) return new_session; }
c
A
2 comments
Solution: you have to use queues for that. I used bullmq and every createSession task would be queued and system resources will not be overwhelmed.
set some low maxConcurrency, i.e. from input, use different values for it at localhost and in Apify cloud
Add a reply
Sign up and join the conversation on Discord