Apify

Apify Crawlee GitHub

Apify Discord Mirror

Updated 6 months ago

url not timing out for a reason.

url not timing out for a reason.

At a glance

The community member is experiencing an issue with their web scraper, where it appears to be running indefinitely despite setting a timeout. The comments suggest that the issue may be platform-related, as some URLs take a long time to process (1-2 minutes or even 5 minutes). The community members discuss potential causes, such as the script timing out after 1.5 seconds for a maximum of 21 requests, which should not exceed 31.5 seconds. However, there is no explicitly marked answer in the comments.

Useful resources

·

Hello, my scraper goes in an infinite run on some url for a reason even tho I put a timeout time

Plain Text

 request = r.get(url=url, timeout=1.5)

. Any ideas why?

K

R

A

18 comments

I figured that it doesn't go in an infinite run but rather takes 1-2 minute to process the url

or 5 minute sometime

Plain Text

2024-03-14T07:05:07.582Z https://dunamisenergy.com is an invalid url
2024-03-14T07:05:07.603Z Processing https://broadsign.com...
2024-03-14T07:07:38.962Z Processing https://heartlandchargingservices.com...

Plain Text

2024-03-14T06:59:42.263Z Processing https://shinetsua.com...
2024-03-14T07:04:00.486Z Processing https://noralta.com...

Hello, is this a platform related issue or is it caused by some of your own logic how you process the request?

I believe it's platform related

could you give me id of the run

https://console.apify.com/view/runs/OmAMon9hYXW2CpIyS

just advanced to level 6! Thanks for your contributions! 🎉

Attachment

here yahoo.com takes 1M30 second

and yama-group takes 1m

in my script, there is at maximum in total 21 requests that times out after 1.5s

which is at maximum 31,5 second

and if the first request times out

Attachment

it should skip it

Attachment

here there is a time out time aswell

Add a reply

Sign up and join the conversation on Discord