Patch looks good. Needs a file_size < 0 test for read errors which means file_size should be ssize_t.

Unrelated note: Apologises for quoting the wrong this MDEV for travis IPv6 things which are obviously unrelated.

Unassigned this one for now. The patch will need updating against the latest version and test cases added. Marked as "beginner-friendly".

Let me try to fix it

Haha , PR is avalilable https://github.com/MariaDB/server/pull/2267 .
I just follow the code patch above and add two test case .

Would someone take a look for PR ?

pr now https://github.com/MariaDB/server/pull/2297

Branch bb-11.8-MDEV-9158-file-key-manager

sergI'm not seeing any system variable named 'MAX_KEY_FILE_SIZE' but there is this variable 'key_buffer_size'. So as a user should I check 'key_buffer_size' for the max key size value?

Also, is the same max key size value applicable for named pipes?

Did you try to grep?

$ grep -wrI MAX_KEY_FILE_SIZE plugin/file_key_management/

plugin/file_key_management/parser.cc:#define MAX_KEY_FILE_SIZE 1024*1024

plugin/file_key_management/parser.cc:  if (file_size > MAX_KEY_FILE_SIZE)

sergYes, my grep shows that MAX_KEY_FILE_SIZE is hardcoded to 1MB. But I have the following two concerns,
1. The error message is misleading when the file size exceeds MAX_KEY_FILE_SIZE

11.8.0-opt>INSTALL SONAME 'file_key_management';

ERROR 2 (HY000): Invalid key at /test/MDEV-9158-MD150125-mariadb-11.8.0-linux-x86_64-opt/mariadb-test/var/tmp/key.txt line 14926, column 7

11.8.0-opt>

log:

2025-01-21 21:53:36 4 [Note] mariadbd: Read from /test/MDEV-9158-MD150125-mariadb-11.8.0-linux-x86_64-opt/mariadb-test/var/tmp/key.txt, read bytes: 1048576, max key file size: 1048576 bytes

2025-01-21 21:53:36 4 [ERROR] mariadbd: Invalid key at /test/MDEV-9158-MD150125-mariadb-11.8.0-linux-x86_64-opt/mariadb-test/var/tmp/key.txt line 14926, column 7

2025-01-21 21:53:36 4 [ERROR] Plugin 'file_key_management' init function returned error.

Here the errors (both SQL and log) say 'Invalid key' which is not right. It failed due the file size exceeds MAX_KEY_FILE_SIZE. Also in such case why should we read partial key file data because anyway we error out eventually?

2. Named pipe max file size is 64KB which is different from that of a regular file (1MB)

11.8.0-opt>INSTALL SONAME 'file_key_management';

ERROR 2 (HY000): Invalid key at /tmp/fifo.key line 952, column 25

11.8.0-opt>

log:

2025-01-22 12:35:12 5 [Note] mariadbd: Read from /tmp/fifo.key, read bytes: 65536, max key file size: 1048576 bytes

2025-01-22 12:35:12 5 [ERROR] mariadbd: Invalid key at /tmp/fifo.key line 952, column 25

2025-01-22 12:35:12 5 [ERROR] Plugin 'file_key_management' init function returned error.

2025-01-22 12:35:12 5 [ERROR] Plugin 'file_key_management' registration as a ENCRYPTION failed.

Please see if the above two need to be addressed.

A key file is a small file with the list of keys, it's not supposed to be more than 1Mb, and if it is — it's considered invalid, that's what the error says.

May be it could be optimized by not reading the file at all in this case, but it's not the code path we need to optimize, so either behavior is fine.

as for 2. — did you try to pipe a >64K file? because the message says "read bytes: 65536, max key file size: 1048576 bytes", so max size is 1Mb.

did you try to pipe a >64K file?
sergYes, I tried piping a ~1MB file. But it reads only 64K bytes and then errors out.

okay. 64K is a default fifo buffer size. I'll do something about it, thanks.

pushed fixes for both 1 and 2

I'm closing testing after covering the following test cases,
1. Pipe valid key:value data
2. Multiple writes into a single pipe
3. Pipe key:value data without the mandatory key 1
4. Pipe key:value data exceeding max size of 1MB
5. Pipe with no data
6. Use a key not present in the piped data
7. Install with non-existing pipe
8. Install with already pipe by another program
9. CRATE TABLE/ALTER TABLE with valid encryption key
10. CRATE TABLE/ALTER TABLE with invalid encryption key

The test results are looking good. OK to push from QA.