Category Archives: Technology

Overview

If you have large records of data, entering each record into cloud Firestore database manually is time consuming. Not to mention that it is going to be a tedious as well. In this tutorial you will learn an easy method to import large CSV file to Google Cloud Firestore using GCP and Node.js. Therefore you need Google account and valid CSV format file.  Before that, let’s know what GCP and Cloud Firestore is all about.

let’s start…

GCP

Google provide seamless cloud computing services to various clients across the world using Google Cloud Platform (GCP). They provide series of modular services such as hosting services, data storage, data analytics and Machine Learning (ML) and many more which use Google Hardware infrastructure to run.

Cloud Firestore

When you look for a fast NoSQL document database, then there is nothing better than Cloud Firestore provided by Google. It is serverless, meaning it simplifies data storing, syncing, and data querying for developers. It also supports real time synchronization and offline support.  Security is another feature where developers spend less time on security of the application, therefore time requires to develop a mobile application is significantly reduced.

GCP setup

Create new project in Google Cloud Platform (GCP). In this case concilsVote (you can give a name as your wish). This is how the main menu looks like. Click Firestore and then Data.

Then create new project by clicking NEW PROJECT or CREATE PROJECT as shown. If you have existing project you may search the project and then use it.

Fill the form and enter CREATE. (You need to enter suitable project name and Location).

Now, select the project you just created from the drop down and then click OPEN.

You will see the following screen. Choose SELECT NATIVE MODE from the options given.

Then choose, the nearest location where to store the data and CREATE DATABASE. Please select the location carefully because we are unable to change the location once it’s being created.

You will see process of database creation process running in the background.

Now you will see a screen like this. It says Your database is ready to go. Just add data. It is time to Activate Cloud Shell.

Enter the following command below to check whether the project is configured or not.

gcloud config list project

Now set the project ID using the below command. You may see the project ID from the list.

gcloud config set project PROEJCTID
gcloud config set project concilsVote

Now write some logic to read CSV data and import it to Firestore.

Create a new directory named concilsVote in the terminal and then change the directory.

mkdir concilsVote
cd concilsVote

Initialize npm using the below command in the terminal and fill the details.

npm init

Press Yes, and enter for all the options, until package.json file is get created with the details below.

{
"name": "rainfallcsvexport",
"version": "1.0.0",
"description": "Rainfall Data conversion",
"main": "index.js",
"scripts": { "test": "echo "Error: no test specified" && exit 1" },
"author": "Your name",
"license": "ISC"
}

Now you need to install the following dependencies. Just run the following npm commands.

npm install @google-cloud/firestore
npm install csv-parse

Create a new file named concilVote.js using the command in the terminal (Any name is fine). Copy the below code and paste in the file.

const {readFile}  = require('fs').promises;
const {promisify} = require('util');
const parse       = promisify(require('csv-parse'));
const {Firestore} = require('@google-cloud/firestore');
if (process.argv.length < 3) {
  console.error('Please include a path to a csv file');
  process.exit(1);
}
const db = new Firestore();
function writeToFirestore(records) {
  const batchCommits = [];
  let batch = db.batch();
  records.forEach((record, i) => {
    var docRef = db.collection('rainfall').doc(record.SUBDIVISION);
    batch.set(docRef, record);
    if ((i + 1) % 500 === 0) {
      console.log(`Writing record ${i + 1}`);
      batchCommits.push(batch.commit());
      batch = db.batch();
    }
  });
  batchCommits.push(batch.commit());
  return Promise.all(batchCommits);
}
async function importCsv(csvFileName) {
  const fileContents = await readFile(csvFileName, 'utf8');
  const records = await parse(fileContents, { columns: true });
  try {
    await writeToFirestore(records);
  }
  catch (e) {
    console.error(e);
    process.exit(1);
  }
  console.log(`Wrote ${records.length} records`);
}
importCsv(process.argv[2]).catch(e => console.error(e));

Now we will upload the CSV file. Upload the CSV file to the concilVote folder by right-clicking the folder name.

To import data to Firestore simply run the following command.

node concilVote.js concilVote.csv

Click Enter, and now you are able to see the data getting imported. If everything goes well, you will see a message like below, in the console. Now just refresh the Firestore console page and you will see the data.

Wrote 5302 records

Congratulations