[CONPY-252] No proper way to close result set from server side cursor select that doesn't hang the program. Created: 2023-02-23  Updated: 2023-11-06

Status: Needs Feedback
Project: MariaDB Connector/Python
Component/s: Other
Affects Version/s: 1.1.5post1
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Michael Conrad Assignee: Georg Richter
Resolution: Unresolved Votes: 0
Labels: None
Environment:
  • lib version: 1.1.5post3, not 1.15post1
  • conda python environment
  • Ubuntu 22.04 LTS, mate-desktop
  • PyCharm Professional
  • SqlAlchemy

Python Version: 3.10.9

 Description   

I'm running a query on millions of records and need to use server side cursors. I'm able to get streaming working, but when I close the result set either via exiting the context manager block or via an explicit #close() everything hangs while it waits for the driver to pull in and discard the remaining data associated with the server side cursor.

The mariadb client code indicates that a #close() on a cursor is supposed to:
```python
def close(self):
"""
Closes the cursor.

If the cursor has pending or unread results, .close() will cancel them
so that further operations using the same connection can be executed.

The cursor will be unusable from this point forward; an Error
(or subclass) exception will be raised if any operation is attempted
with the cursor."
"""

  1. CONPY-231: fix memory leak
    if self._data:
    del self._data

if not self.connection._closed:
super().close()
```

Shouldn't the #close() on a result object stop the steaming of data from the server instead of causing a potentially many hour or longer hang waiting on the result set to be streamed over the network then discarded? Is this incomplete or incorrect documentation?

I did find in the pymysql lib where it explicitly exhausts the cursor on close with a comment that there isn't a way to stop the stream of data.



 Comments   
Comment by Georg Richter [ 2023-02-24 ]

Hi,

could you please provide a sample program which demonstate this behavior? When using a server side cursor it shouldn't block (see https://github.com/mariadb-corporation/mariadb-connector-python/blob/1.1/testing/test/integration/test_cursor.py#L395)

Comment by Michael Conrad [ 2023-02-24 ]

What do you mean by "blocking"? And how would I demonstrate the #close() hanging in a test? Timing?

Comment by Georg Richter [ 2023-02-25 ]

The sample program should contain some information: e.g. How was the cursor created, what was executed, were data fetched etc.

About blocking/unbuffered cursors:

cursor1= connection.cursor(buffered=False)
cursor2= connection.cursor(buffered=False)
cursor1.execute("SELECT 1,2")
 
# this fails until cursor1 was closed, or all resultsets from cursor1 were fetched
#cursor2.execute("SELECT 2,3")

Generated at Thu Feb 08 03:31:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.