Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
None
Description
see https://github.com/MariaDB/mariadb-connector-nodejs/issues/40
The problem is when "piping", to handle the stream when writable stream fails
How to handle writable that stop for any reason ( for example a ClientRequest that has ended):
stream is then in pause mode, waiting for Writable to consume feed. All subsequent request are stopped until stream finished.
I don't see any other solution but to document piping properly to indicate to resume.
Probably a good solution for pool state is that streaming not consumed must be handled when connection is given back to pool.
example :
pool
|
.getConnection()
|
.then(conn => {
|
const someWriterStream = fs.createWriteStream("./tmp.file") |
.on("error", err => { |
//export has ended, resume stream |
queryStream.resume();
|
});
|
|
|
const transform = new Transform({ |
transform: function transformer(chunk, encoding, callback) { |
callback(null, JSON.stringify(chunk)); |
},
|
objectMode: true |
});
|
|
|
const queryStream = conn.queryStream("SELECT seq as val FROM seq_1_to_10000"); |
queryStream
|
.pipe(transform)
|
.pipe(someWriterStream)
|
.on('close', () => { |
conn.end();
|
});
|
|
|
//forcing a WRITER end to simulate this error |
setTimeout(someWriterStream.destroy.bind(someWriterStream), 2);
|
})
|
.catch(err => { |
//error |
});
|
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
see https://github.com/MariaDB/mariadb-connector-nodejs/issues/40
The problem is when "piping", to handle the stream when writable stream fails How to handle writable that stop for any reason ( in your case a ClientRequest that has been ended): stream is then in pause mode, waiting for Writable to consume feed. All subsequent request are stopped until stream finished. I don't see any other solution but to document piping properly to indicate to resume. Probably a good solution for pool state is that streaming not consumed must be handled when connection is given back to pool. example : {code:javascript} pool .getConnection() .then(conn => { const someWriterStream = fs.createWriteStream("./tmp.file") .on("error", err => { //export has ended, resume stream queryStream.resume(); }); const transform = new Transform({ transform: function transformer(chunk, encoding, callback) { callback(null, JSON.stringify(chunk)); }, objectMode: true }); const queryStream = conn.queryStream("SELECT seq as val FROM seq_1_to_10000"); queryStream .pipe(transform) .pipe(someWriterStream) .on('close', () => { conn.end(); }); //forcing an WRITER end to simulate this error setTimeout(someWriterStream.destroy.bind(someWriterStream), 2); }) .catch(err => { //error }); {code} |
see https://github.com/MariaDB/mariadb-connector-nodejs/issues/40
The problem is when "piping", to handle the stream when writable stream fails How to handle writable that stop for any reason ( for example a ClientRequest that has ended): stream is then in pause mode, waiting for Writable to consume feed. All subsequent request are stopped until stream finished. I don't see any other solution but to document piping properly to indicate to resume. Probably a good solution for pool state is that streaming not consumed must be handled when connection is given back to pool. example : {code:javascript} pool .getConnection() .then(conn => { const someWriterStream = fs.createWriteStream("./tmp.file") .on("error", err => { //export has ended, resume stream queryStream.resume(); }); const transform = new Transform({ transform: function transformer(chunk, encoding, callback) { callback(null, JSON.stringify(chunk)); }, objectMode: true }); const queryStream = conn.queryStream("SELECT seq as val FROM seq_1_to_10000"); queryStream .pipe(transform) .pipe(someWriterStream) .on('close', () => { conn.end(); }); //forcing an WRITER end to simulate this error setTimeout(someWriterStream.destroy.bind(someWriterStream), 2); }) .catch(err => { //error }); {code} |
| Description |
see https://github.com/MariaDB/mariadb-connector-nodejs/issues/40
The problem is when "piping", to handle the stream when writable stream fails How to handle writable that stop for any reason ( for example a ClientRequest that has ended): stream is then in pause mode, waiting for Writable to consume feed. All subsequent request are stopped until stream finished. I don't see any other solution but to document piping properly to indicate to resume. Probably a good solution for pool state is that streaming not consumed must be handled when connection is given back to pool. example : {code:javascript} pool .getConnection() .then(conn => { const someWriterStream = fs.createWriteStream("./tmp.file") .on("error", err => { //export has ended, resume stream queryStream.resume(); }); const transform = new Transform({ transform: function transformer(chunk, encoding, callback) { callback(null, JSON.stringify(chunk)); }, objectMode: true }); const queryStream = conn.queryStream("SELECT seq as val FROM seq_1_to_10000"); queryStream .pipe(transform) .pipe(someWriterStream) .on('close', () => { conn.end(); }); //forcing an WRITER end to simulate this error setTimeout(someWriterStream.destroy.bind(someWriterStream), 2); }) .catch(err => { //error }); {code} |
see https://github.com/MariaDB/mariadb-connector-nodejs/issues/40
The problem is when "piping", to handle the stream when writable stream fails How to handle writable that stop for any reason ( for example a ClientRequest that has ended): stream is then in pause mode, waiting for Writable to consume feed. All subsequent request are stopped until stream finished. I don't see any other solution but to document piping properly to indicate to resume. Probably a good solution for pool state is that streaming not consumed must be handled when connection is given back to pool. example : {code:javascript} pool .getConnection() .then(conn => { const someWriterStream = fs.createWriteStream("./tmp.file") .on("error", err => { //export has ended, resume stream queryStream.resume(); }); const transform = new Transform({ transform: function transformer(chunk, encoding, callback) { callback(null, JSON.stringify(chunk)); }, objectMode: true }); const queryStream = conn.queryStream("SELECT seq as val FROM seq_1_to_10000"); queryStream .pipe(transform) .pipe(someWriterStream) .on('close', () => { conn.end(); }); //forcing a WRITER end to simulate this error setTimeout(someWriterStream.destroy.bind(someWriterStream), 2); }) .catch(err => { //error }); {code} |
| Component/s | other [ 14408 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Closed [ 6 ] |
| Workflow | MariaDB v3 [ 92146 ] | MariaDB v4 [ 135252 ] |
Correction is done with https://github.com/MariaDB/mariadb-connector-nodejs/commit/ddb208003cd942b3551cc0cd7cd28df29163db3e, simplifying implementation, using basic TCP Back Pressure.
This ensures connection state in any case and avoids lots of edge case.