Apify and Crawlee Official Forum

Updated 3 months ago

sending a json response and saving it

in my code i want to send a json response to a file and save it
how do i install the dependencies needed (mongoose, next, etc.) and without any errors when sending the json and use the fetchAPI or whatever method that can work w/ apify, and save the data do a mongodb database
heres the code:
h
A
6 comments
Plain Text
        // Read the existing data from the rawData.json file
        let rawData: any = {};
        try {
            const rawDataStr = fs.readFileSync('rawData.json', 'utf8');
            rawData = JSON.parse(rawDataStr);
        } catch (error) {
            console.log('Error reading rawData.json:', error);
        }

        // Append the new data to the existing data
        if (rawData.productList) {
            rawData.productList.push(productData);
        } else {
            rawData.productList = [productData];
        }

        // Write the updated data back to the rawData.json file
        fs.writeFileSync('rawData.json', JSON.stringify(rawData, null, 2));


        const response = await fetch('/db', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
          },
          body: JSON.stringify({ rawData }), // Send the search query in the request body
          
        });

        if (!response.ok) {
          throw new Error('Query failed');
        }
       
Plain Text
import mongoose from 'mongoose';
//import { NextApiRequest, NextApiResponse } from 'next';

const dbURI = 'mongodb+srv://harishs:k0IzPKGQpvZNmcRq@eco-cluster.jiamihu.mongodb.net/'//'mongodb+srv://harishs:k0IzPKGQpvZNmcRq@cluster01.nx3olbr.mongodb.net/?retryWrites=true&w=majority';


const Schema = mongoose.Schema;

const productSchema = new Schema({
  storeName: {
    type: String,
    required: true,
  },
  image: String,
  title: String,
  url: String,
  description: String,
  salePrice: String,
  originalPrice: String,
  reviewScore: String,
  shippingInfo: String,
  reviewNumber: Number,
  ECOscore: Number,
}, { timestamps: true });

productSchema.index({
  title: 'text',
  description: 'text'
});

const Product = mongoose.models.Product || mongoose.model('Product', productSchema);


const connection: { isConnected?: number } = {};

async function dbConnect() {
  if (connection.isConnected) {
    return;
  }

  try {
    mongoose.connect(dbURI)
      .then(() => console.log('Connected to DB and listening on port 5500'))
      .catch((err) => console.log(err));
  } catch (error) {
    console.error('Error connecting to MongoDB:', error);
  }
}

export default async function handler (req, res) {
  if (req.method !== 'POST') {
    return res.status(405).json({ message: 'Method Not Allowed' });
  }

  const { products } = req.body;

  try {
    // Establish a connection to the MongoDB database
    await dbConnect();
    const rawProducts = await Product.insertMany(products);


  } catch (error) {
    console.error('Error querying products:', error);
    return res.status(500).json({ error: 'Internal Server Error' });
  }
}
AND how can i format my data to send a json response
Plain Text
2023-08-06T05:04:53.344Z  ERR! code EJSONPARSE
2023-08-06T05:04:53.346Z npm ERR! path /home/myuser/package.json
2023-08-06T05:04:53.349Z 
2023-08-06T05:04:53.351Z npm ERR!
2023-08-06T05:04:53.353Z  JSON.parse Unexpected token "}" (0x7D) in JSON at position 346 while parsing near "...\": \"^7.4.2\",\n\n\n\n    },\n    \"devDependenc..."
2023-08-06T05:04:53.355Z 
2023-08-06T05:04:53.357Z npm ERR! JSON.parse Failed to parse JSON data.
2023-08-06T05:04:53.359Z npm ERR! JSON.parse Note: package.json must be actual JSON, not just JavaScript.
2023-08-06T05:04:53.361Z 
2023-08-06T05:04:53.364Z 
2023-08-06T05:04:53.366Z 
2023-08-06T05:04:53.368Z npm ERR! A complete log of this run can be found in:
2023-08-06T05:04:53.370Z npm ERR!     /home/myuser/.npm/_logs/2023-08-06T05_04_53_138Z-debug-0.log
2023-08-06T05:04:53.373Z 
2023-08-06T05:04:54.448Z Removing intermediate container 21434274a5c4
Its out of SDK functionality, use whatever recommended for reading files or interact with db for nodeJs apps. However your error above related to wrong format of package.json and its npm error, not in your code. As a quick solution please consider to use apify create - it will generate correct dependencies by template
so where do i run
Plain Text
apify create-
or similar commands in the console to install dependencies
Add a reply
Sign up and join the conversation on Discord