Apify and Crawlee Official Forum

K
Kirada
Offline, last seen 2 months ago
Joined August 30, 2024
So I turned my apify account to an organization and it suddenly reduced my number of users : I don''t know if it's related but I think it is
2 comments
K
O
K
Kirada
·

Bug reporting.

There is this bug in the payout section.
1 comment
o
Plain Text
"platform ": {
            "title": "Search engine",
            "type": "string",
            "description": "Select the search engine you want your websites from.",
            "editor": "select",
            "default": "Duckduckgo",
            "enum": ["ddg", "ggl", "serp"],
            "enumTitles": ["Duckduckgo", "Google", "Serpapi"]
            },

I have this in my actor's input schema. Then I get the content of this
Plain Text
 actor_input = await Actor.get_input() or {}
            raw_query = actor_input.get('query')
            query = raw_query.replace(" ", "+")
            numbers_of_results = actor_input.get('results')
            engine = actor_input.get('platform')

but when I print the engine's content, it says
None
1 comment
R
Hello, my scraper goes in an infinite run on some url for a reason even tho I put a timeout time
Plain Text
 request = r.get(url=url, timeout=1.5)
. Any ideas why?
18 comments
K
A
R
Hello, I can't find the type of this schema, could anyone tell me?
4 comments
D
m
K
R
Hello. I got an issue that I asked to fix a few weeks ago in bug report channel that seem to have been deleted. I tried to run a scrapy script from local using apify run and it worked fine. I pushed it and when I run it from the platform I get this error. It's kinda an "urgent" issue because I was planning to deploy my Actor and try to monetize it. Is it possible to fix this issue ? Thanks
6 comments
K
v
Hello, so I did a script in vsc locally (apify run) and it worked fine. Then, I pushed it and tried to run it from the apify platform and it returns this error I downloaded the reactor like the template asked to install the reactor which I did
Plain Text
install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')
and defined it aswell
Plain Text
settings = get_project_settings()
        settings['TWISTED_REACTOR'] = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'
4 comments
K
L
Plain Text
async def main():


    async with Actor:
        input_data = await Actor.get_input() 
        urls = input_data['websites']
        print(urls)




    async with aiohttp.ClientSession() as session:
        tasks = [fetch_emails(session, website) for website in urls]
        results = await asyncio.gather(*tasks)
        print('here')
    dataset = apify_client.dataset()
    for email in website_list:
        print(email)
        dataset.push_data({"email": email})

    print(website_list)
    print(f"tried to scrape {len(urls)}")
    print(f"there is {invalid_url}")
    print(f'successfully scraped {len(website_list)} emails ! ')

asyncio.run(main())
the
Plain Text
async with Actor:
        input_data = await Actor.get_input() 
        urls = input_data['websites']
        print(urls)
does print the url. But after that, the code stops. The scripts work fine in vsc. I'm trying to integrate it with apify but I'm facing some issues.
3 comments
A
K
R
Hello, I got a little problem. As I said in the title, my script does not execute the close_spider function and when the scraping finishes, it goes in an infinite loops. I guess that's why close_spider doesn't get executed. Can anyone help?
33 comments
K
V
A
O