javascript - Google BigQuery Node library: how to stream data in batches? -

i'm using google cloud node libraries bigquery. i'm using createquerystream method stream data bigquery:

var query = 'select transfer_date, price, postcode '; query += '[table] order transfer_date limit 10000'; bigquery.createquerystream(query)   .on('error', console.error)   .on('data', function(row) {     console.log(row);   })   .on('end', function() {     // rows retrieved.   }); 

this outputs each row individually console. however, update application in batches, every 10,000 results. how amend query stream data in chunks of 10,000?

the query method has autopaginate option, don't understand how use it.

or need write function manually fires each 10,000 rows? seems inefficient.

bigquery node.js api takes care of pagination automatically when execute query, don't have anything. api method using stream result record 1 one. if want 10,000 records using such method might like:

    var rows = [];     var query = 'select transfer_date, price, postcode ';     query += '[table] order transfer_date limit 10000';      var bigquery = require('@google-cloud/bigquery')();     bigquery.createquerystream(query,{'autopaginate':false,'maxapicalls':10,'maxresults':1000})     .on('error', console.error)     .on('data', function(row) {         //either use stream record or keep pushing stack.         rows.push(row);     })     .on('end', function() {         //if want process results @ end.         rows.foreach(row=>console.log(row));     }); 

options object in createquerystream method same query method. however, if want 10k records per request, use batch results using "query" method following:

    var query = 'select transfer_date, price, postcode ';     query += '[table] order transfer_date limit 10000';      var bigquery = require('@google-cloud/bigquery')();     bigquery.query(query, function(err, rows) {         if (err) {             // handle error here.         }else{             // handle results here.          rows.foreach(row=>console.log(row));         }     }); 

autopaginate option set true default. documents, here link