Apify and Crawlee Official Forum

Updated 4 weeks ago

await a promise set in a pre navigation hook

Hi All,

I have a pre navigation hook that listens for requests and if they return images saves them to the cloud

Plain Text
  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud


This works in 95% of cases however there are some where the main request hander completes before each res.response() can be collected. This causes an error since the browser context closes with the main request handler. My question is how can i best get around this? One idea I had was that I could put a promise in the page userdata that resolves when the page has no outstanding images. After reading the docs however im not sure if this is possible since userdata needs to be serialisable? Has anyone else encountered this type of issue and how have they got around it?
P
C
2 comments
Hi @Crafty If I understand it correctly, the simplest solution would be to have global variable and match it with the request.

Plain Text
const requestPromises = new Map();

Plain Text
// preNavHook
async (context) => {
   requestPromises.set(context.request.id, [ /* ... Promises ... */ ]);
}

Plain Text
...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(requestPromises.get(context.request.id));
  requestPromises.delete(context.request.id);
}
...



Sometimes I see people passing their own variables to the context which is then passed to the routeHandler

Plain Text
// preNavHook
async (context) => {
   context.myPromises = [ ... ];
}

Plain Text
...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(context.myPromises);
}
...
Hi, thanks for this. It should solve my problem nicely. πŸ™‚
Add a reply
Sign up and join the conversation on Discord