Apify and Crawlee Official Forum

Updated 3 months ago

Scrape not writing data to dataset

I am a noob with JS and creating a scraper. I am not sure this is the correct place to post a question but it wasn't getting answered from the issue section of the scrapper itself.

I am using Puppeteer Scraper made by Apify and can print to screen the scrapped data but I can't get it to write to the dataset. I have tried multiple different methods but no success. Any help would be appreciated or a point in the correct direction would be appreciated.

I know this is something simple and am going to kick myself when it is figured out! lol
k
H
I
5 comments
import dataset like this and just push any object and that's it.
Plain Text
import { Dataset, createPuppeteerRouter } from 'crawlee';
await Dataset.pushData(data);
data returned by pageFunction are pushed to the dataset, see the docs part Results here https://console.apify.com/actors/YJCnS9qogi9XxDgLB/information/version-3/readme
so if you add to the end of the function for example this
Plain Text
return {
  title: "Web Scraping, Data Extraction and Automation - Apify"
}

you will see that in the dataset after the run
Thank you! That was a simple thing and I found out that I was trying to return inside the for loop before. I brought the data outside the for loop into an array and now I have the data.

Just by chance and I am going to look into it the documentation now, you wouldn't know how to format the CSV file for this would you?

Looks great in the JSON format but CSV is not putting the data into columns but instead just throwing each into a row.

Am I supposed to append the data? I think I would do this if I was running locally from something like node.js so I wasn't sure if this was something I would need to look at for the actor.
Attachment
image.png
If you want it in one column then it must not be array,
That makes sense. I was thinking an array because of the dynamic ability. Thanks again for the info.
Add a reply
Sign up and join the conversation on Discord