Sign up for Apify Platform here
Star Crawlee on GitHub
Star Crawlee for Python on GitHub
Apify and Crawlee Official Forum
New post
View all posts
Related posts
Was this helpful?
๐
๐
๐
Powered by
Hall
Active
Updated 10 hours ago
0
Follow
How to implement persistent login with crawlee-js/playwright?
How to implement persistent login with crawlee-js/playwright?
Active
0
Follow
o
osenvosem
2 weeks ago
ยท
I need to scrape content on multiple pages in one social network (x.com) that requires auth. Where to implement the login mechanism in order to it happened before following urls and persisted to use it until it is valid?
!
C
O
5 comments
Share
Open in Discord
!
! Spooky
2 weeks ago
await page.context().storageState({ path: authFilePath })
Look up storageState() on the playwright docs
C
Crafty
13 hours ago
that gets the cookie and localstorage state but how do you load it into a new session. im facing the same dilema
O
Oleg V.
edited 11 hours ago
You can store you cookies in named KV store and then modify Your session with function:
https://crawlee.dev/api/next/core/interface/SessionPoolOptions#createSessionFunction
or You can do the same (update your requests with those cookies) in preNavigationHooks (
https://crawlee.dev/api/next/browser-crawler/interface/BrowserCrawlerOptions#preNavigationHooks
)
C
Crafty
10 hours ago
what about local storage? im supprised there seems to be no easy way to seed a session with local data
C
Crafty
10 hours ago
and also, setCookie wants a raw cookie string and a url rather than the format the getState() gives you
Add a reply
Sign up and join the conversation on Discord
Join on Discord