Apify

Apify and Crawlee Official Forum

b
F
A
J
A
Members
Jimpex
J
Jimpex
Offline, last seen last month
Joined August 30, 2024
JJimpex
·

Is it possible to run a crawl within a crawl?

I have a crawler I have setup (x) that collects links from a page. I want to run another crawler(y) within x crawl to collect more data and return it with the rest of x crawl data. Is it possible to do this?

(I know you can add requests to the queue from a crawl but I want to keep the data together)

example data structure:
Plain Text
const data = [{
    // from "x" crawl
    title: 'a page',
    links: [
        {
            // from "y" crawl
            title: 'a page',
            link: 'link'
        }
    ]
}]
5 comments
J
P
A
JJimpex
·

Is it possible to run a crawl within a crawl?

I have a crawler I have setup (x) that collects links from a page. I want to run another crawler(y) within x crawl to collect more data and return it with the rest of x crawl data. Is it possible to do this?

(I know you can add requests to the queue from a crawl but I want to keep the data together)

example data structure:
Plain Text
const data = [{
    // from "x" crawl
    title: 'a page',
    links: [
        {
            // from "y" crawl
            title: 'a page',
            link: 'link'
        }
    ]
}]
5 comments
J
P
A
JJimpex
·

Set cookies with cheerio

I switched from playwright to cheerio because of familiarity and cannot figure out how to set cookies (specifically for auth).

In playwright I did ontext.page.context().addCookies. How do I do this in cheerio?
2 comments
J
L
JJimpex
·

Set cookies with cheerio

I switched from playwright to cheerio because of familiarity and cannot figure out how to set cookies (specifically for auth).

In playwright I did ontext.page.context().addCookies. How do I do this in cheerio?
2 comments
J
L
JJimpex
·

Bug with typescript

I installed using templates (typescript example and playwright typescript example) on multiple operating systems (windows & ubuntu via vmware) and ran into the error "state.conditions.includes is not a function). I tested the basic js template and had no issue.

I went back to the ts project and updated to latest typescript(5.3.2) and tried again, same error. I then installed 5.1.6 and it worked.
1 comment
A
JJimpex
·

Bug with typescript

I installed using templates (typescript example and playwright typescript example) on multiple operating systems (windows & ubuntu via vmware) and ran into the error "state.conditions.includes is not a function). I tested the basic js template and had no issue.

I went back to the ts project and updated to latest typescript(5.3.2) and tried again, same error. I then installed 5.1.6 and it worked.
1 comment
A