The Storage-Engine Independent Column Compression (MDEV-11371) does not call deflateEnd() when deflate() does not return Z_STREAM_END.

https://github.com/MariaDB/server/blob/10.5/sql/field_comp.cc#L70-L72

According to the zlib manual deflate() can also return other values:

If the parameter flush is set to Z_FINISH, pending input is processed, pending output is flushed and deflate returns with Z_STREAM_END if there was enough output space. If deflate returns with Z_OK or Z_BUF_ERROR, this function must be called again with Z_FINISH and more output space (updated avail_out) but no more input data, until it returns with Z_STREAM_END or an error. After deflate has returned Z_STREAM_END, the only possible operations on the stream are deflateReset or deflateEnd.

This for instance happens when the data is already (externally) compressed and deflate() needs more space than the original data. There is no use in enlarging the output buffer and calling deflate() again, but memory still needs to be freed by calling deflateEnd().

When running the following query on an existing table with ~100.000 records MariaDB quickly uses up several gigabytes of memory (without returning it):
ALTER TABLE test MODIFY content longblob compressed not null
(the content column contains mostly externally gzipped data with an average length of ~1.500 bytes)

Tried on MariaDB 10.4.8, 10.4.11 and 10.5