Scrape not writing data to dataset

IInvictus806

I am a noob with JS and creating a scraper. I am not sure this is the correct place to post a question but it wasn't getting answered from the issue section of the scrapper itself.

I am using Puppeteer Scraper made by Apify and can print to screen the scrapped data but I can't get it to write to the dataset. I have tried multiple different methods but no success. Any help would be appreciated or a point in the correct direction would be appreciated.

I know this is something simple and am going to kick myself when it is figured out! lol

5 comments

kkprocks

import dataset like this and just push any object and that's it.

Plain Text

import { Dataset, createPuppeteerRouter } from 'crawlee';
await Dataset.pushData(data);

HHonzaS

data returned by pageFunction are pushed to the dataset, see the docs part Results here https://console.apify.com/actors/YJCnS9qogi9XxDgLB/information/version-3/readme
so if you add to the end of the function for example this

Plain Text

return {
  title: "Web Scraping, Data Extraction and Automation - Apify"
}

you will see that in the dataset after the run

IInvictus806

Thank you! That was a simple thing and I found out that I was trying to return inside the for loop before. I brought the data outside the for loop into an array and now I have the data.

Just by chance and I am going to look into it the documentation now, you wouldn't know how to format the CSV file for this would you?

Looks great in the JSON format but CSV is not putting the data into columns but instead just throwing each into a row.

Am I supposed to append the data? I think I would do this if I was running locally from something like node.js so I wasn't sure if this was something I would need to look at for the actor.

Attachment

HHonzaS

If you want it in one column then it must not be array,

IInvictus806

That makes sense. I was thinking an array because of the dynamic ability. Thanks again for the info.

Add a reply

Apify and Crawlee Official Forum

Scrape not writing data to dataset