RequestRouter not using the `log` instance passed in its respective Playwright crawler
RequestRouter not using the `log` instance passed in its respective Playwright crawler
At a glance
The community member has a use case where they need to scrape multiple sites, and the issue they are facing is that the logging gets mixed up when using a single scraper. They tried creating multiple loggers, but the issue is that the log from crawlee is shared between all the places that have imported it, and they have a custom logging implementation that is different for each site.
The community member looked at the documentation and found that the createPlaywrightCrawler has an option to pass in a custom logger, but this only affects the internal logging of crawlee and not the logs they explicitly log in their routeHandler.
The community members tried to get the log from the crawlerContext as suggested by another community member, but the issue is that the internal logs of crawlee are still not respecting the logger they passed in.
The community member eventually found the RequstQueue export in crawlee and was able to fix the issue by setting a global log config in their main.ts file and then spawning new log objects for each crawler separately.
There is no explicitly marked answer in the comments.
Hey folks, I have a bit of a weird use case here, I need to scrape multiple sites and I know the recommended approach is to use a single scraper and pass in the urls to the starting queue
But issue with that is, the logging gets messed up with all the different site logs getting mixed together, to fix that I created multiple loggers for it but the issue I am facing is,
when we import log from crawlee it looks like its shared between all the places that has imported log too, and I have a custom logging implementation which is different for each site (mainly in the format of logs), and I need crawlee to respect that.
So to fix this I looked at the docs and createPlaywrightCrawler has an option for log where you can pass in your own logger to it, and while it uses the correct format for crawlee's internal logging, it does nothing to the logs which I explicitly log in my routehandler.
so How do I either separate log instances for each instance, or make my route handler respect the logger i have passed when I created the crawler
