Apify and Crawlee Official Forum

Updated 11 months ago

How to inject storage state into crawler's page

At a glance

How can a storageState be injected into crawler's page before the page is created? Basically, how to reproduce this but using storageState from the session instead:

Plain Text

        prePageCreateHooks: [
            async (pageId, browserController, pageOptions) => {
                const storageState = await KeyValueStore.getValue('storageState');
                if (pageOptions && storageState)
                    pageOptions.storageState = storageState;
            },
        ],

8 comments

AAndrey Bykov

Hey ! Long time no see 🙂 Could you elaborate more on what exactly are you trying to achieve? To be frank - I am not really sure what's happening there :/

TTobey

Hey ! Indeed. I need to restore storageState from a session and don't know how to do that in a crawler... Ideally, I need access to browserPools context before it is created and before it calls newPage but haven't found a way. I want to pass options to the call to newPage which is the only way to inject the storageState in PW.

TTobey

https://playwright.dev/docs/api/class-browsercontext#browser-context-storage-state

sstrajk

Another apify alumni, bring beers 😄
I was trying to hijack newPage too recently, and found this old discussion mentioning that modifying finalPageOptions works only with useIncognitoPages: true

And indeed, it works!
(didn't have time to investigate why it is, but that's the world we live in 😄 )

sstrajk

https://discord.com/channels/801163717915574323/1018881795791532093/1019020911468359731

AAndrey Bykov

and the thread you pointed brings another apify alumni 😄

TTobey

Cheers! PW ignores the options passed from browserPool unless in incognito mode. Not sure why, looks like a bug to me... the code there is quite messy. I think there should be a way to intercept the browserPool calls to newContext() and inject the options before it's created but didn't find any.

TTobey

Do you know of any way?

Add a reply