isFinishedFunction, check other crawler?

bbmax

Hello,

two questions.

Is there a way to call this.isFinishedFunction so it calls the original function but also just add another web_crawler_queue isFinished on top of it? the uncommented out function I tried, worked somewhat, but, after a long running web_crawler_queue finished, it just kept giving me a stalled error, and, this crawler that this function belongs to never finished.

Plain Text

    autoscaledPoolOptions: {
      isFinishedFunction: async () => {
        const web_crawler_queue = await RequestQueue.open(place_id)
//        return this.isFinishedFunction() && await web_crawler_queue.isFinished()
        return await request_queue.isFinished() && await web_crawler_queue.isFinished()

      }
    },

8 comments

AApifyBot

just advanced to level 3! Thanks for your contributions! 🎉

LLukas Krivka

There is no original function since you are putting your own there in options. If you would want to use original, you would have to monkeypatch it after defining.

Plain Text

// not sure about the .bind
const origFn = crawler.autoscaledPool.isFinishedFunction.bind(crawler)
crawler.autoscaledPool.isFinishedFunction = async () => {
    const orig = origFn()
    // etc.
}

bbmax

got it, will try this

bbmax

my code above almost works but for some reason doesn't all the time..

bbmax

which is basically is my current request_queue finished and is my other crawler finished

bbmax

but sometimes it doesn't pick up and I'll get an log along the lines of : "x, y, z ahs been stalled for 350 seconds"

bbmax

Plain Text

{
  "isSystemIdle": false,
  "memInfo": {
    "isOverloaded": true,
    "limitRatio": 0.2,
    "actualRatio": 1
  },
  "eventLoopInfo": {
    "isOverloaded": false,
    "limitRatio": 0.6,
    "actualRatio": 0.057
  },
  "cpuInfo": {
    "isOverloaded": true,
    "limitRatio": 0.4,
    "actualRatio": 1
  },
  "clientInfo": {
    "isOverloaded": false,
    "limitRatio": 0.3,
    "actualRatio": 0
  }
}

bbmax

this: RNING","msg":"RequestQueue: The request queue seems to be stuck for 370s, resetting internal state.","inProgress":[]}

Add a reply

Join on Discord

Apify and Crawlee Official Forum

isFinishedFunction, check other crawler?