await a promise set in a pre navigation hook

At a glance

The community member has a pre-navigation hook that listens for requests and saves images to the cloud. However, in some cases, the main request handler completes before the image responses can be collected, causing an error. The community member suggests using a promise in the page userdata to resolve when the page has no outstanding images, but is unsure if this is possible since userdata needs to be serializable.

Another community member suggests a solution using a global variable to store the request promises, and then awaiting them in the request handler. Alternatively, they suggest passing the promises to the context and awaiting them in the request handler.

The original community member indicates that the suggested solution should solve their problem nicely.

CCrafty

Hi All,

I have a pre navigation hook that listens for requests and if they return images saves them to the cloud

Plain Text

  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud

This works in 95% of cases however there are some where the main request hander completes before each res.response() can be collected. This causes an error since the browser context closes with the main request handler. My question is how can i best get around this? One idea I had was that I could put a promise in the page userdata that resolves when the page has no outstanding images. After reading the docs however im not sure if this is possible since userdata needs to be serialisable? Has anyone else encountered this type of issue and how have they got around it?

2 comments

PPepa J

Hi @Crafty If I understand it correctly, the simplest solution would be to have global variable and match it with the request.

Plain Text

const requestPromises = new Map();

Plain Text

// preNavHook
async (context) => {
   requestPromises.set(context.request.id, [ /* ... Promises ... */ ]);
}

Plain Text

...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(requestPromises.get(context.request.id));
  requestPromises.delete(context.request.id);
}
...

Sometimes I see people passing their own variables to the context which is then passed to the routeHandler

Plain Text

// preNavHook
async (context) => {
   context.myPromises = [ ... ];
}

Plain Text

...
requestHandler: async (context) => {
  // my routeHandler code

  await Promise.all(context.myPromises);
}
...

CCrafty

Hi, thanks for this. It should solve my problem nicely. 🙂

Add a reply

Apify Discord Mirror

await a promise set in a pre navigation hook