[CONJS-60] handling pipe error for stream Created: 2019-01-30  Updated: 2019-01-31  Resolved: 2019-01-30

Status: Closed
Project: MariaDB Connector/node.js
Component/s: other
Affects Version/s: None
Fix Version/s: 2.0.3-GA

Type: Bug Priority: Major
Reporter: Diego Dupin Assignee: Diego Dupin
Resolution: Fixed Votes: 0
Labels: None


 Description   

see https://github.com/MariaDB/mariadb-connector-nodejs/issues/40

The problem is when "piping", to handle the stream when writable stream fails

How to handle writable that stop for any reason ( for example a ClientRequest that has ended):
stream is then in pause mode, waiting for Writable to consume feed. All subsequent request are stopped until stream finished.

I don't see any other solution but to document piping properly to indicate to resume.
Probably a good solution for pool state is that streaming not consumed must be handled when connection is given back to pool.

example :

pool
  .getConnection()
  .then(conn => {
    const someWriterStream = fs.createWriteStream("./tmp.file")
      .on("error", err => {
        //export has ended, resume stream
        queryStream.resume();
      });
 
    const transform = new Transform({
      transform: function transformer(chunk, encoding, callback) {
        callback(null, JSON.stringify(chunk));
      },
      objectMode: true
    });
 
    const queryStream = conn.queryStream("SELECT seq as val FROM seq_1_to_10000");
    queryStream
      .pipe(transform)
      .pipe(someWriterStream)
      .on('close', () => {
        conn.end();
      });
 
    //forcing a WRITER end to simulate this error
    setTimeout(someWriterStream.destroy.bind(someWriterStream), 2);
  })
  .catch(err => {
    //error
  });



 Comments   
Comment by Diego Dupin [ 2019-01-30 ]

Correction is done with https://github.com/MariaDB/mariadb-connector-nodejs/commit/ddb208003cd942b3551cc0cd7cd28df29163db3e, simplifying implementation, using basic TCP Back Pressure.
This ensures connection state in any case and avoids lots of edge case.

Generated at Thu Feb 08 03:22:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.