Got it, thanks! So I built bb-11.6-MDEV-32887-vector, and the build process seemed to go fine. In the log I can see at startup:

2024-07-03 15:45:53 0 [Note] Starting MariaDB 11.6.0-MariaDB source revision 77a016686ec2a2617dd6489a756b1f9f11a78d9f as process 27924

Which seems to be the latest commit on that branch as far as I can tell, so it looks like I've gotten the proper source. But when I run "ALTER TABLE data ADD COLUMN embedding VECTOR(100);", I get "SQL Error (4161): Unknown data type: 'VECTOR'". Is there something else I need to enable to test?

No, nothing. VECTOR data type is MDEV-33410, and it's open no work done on it yet.

We're going to implement it, of course, but it's not the first priority — it's a convenience feature that helps to avoid mistakes, but an application does not really need it, one can store and search embedding without a dedicated data type. We're prioritizing features that an application cannot work without. Functions VEC_FromText() and VEC_AsText() are also not a priority.

See the test file mysql-test/main/vector.test — that's how one uses it now, store in blob, insert as binary.
In python I do it like

c.execute('INSERT kb (emb) VALUES (?)', array.array('f',resp.data.embedding).tobytes())

Great, that works, so I'll start testing my real workload against it. Thanks!

A word of caution, most performance optimizations, even if implemented, haven't been pushed into this branch yet.

Hi serg, I summarize some findings regarding the scalar quantization using _Float16 that we discussed during our meeting today.

Draft commit to test _Float16 (2-byte float) in HNSW index: https://github.com/HugoWenTD/server/commit/9656b6c0d

Benchmarks indicate that using _Float16 instead of floats results in a 40-60% reduction in insertion speed and a 15-20% reduction in search speed. There is also a minor decrease in recall of less than 1%.

There are two issues with this solution (more research needed):

1. Converting 4-byte floats to 2-byte floats results in precision loss and a reduced range. Proportional scaling is necessary, but there is no simple method to define a proportion that works for all cases. Scaling must be done during transformation, and the best approach depends on the specific dataset and range of values.
   • _Float16 range is -65504 ~ 65504
   • If original floats and distance squares are all below this value the direct transform from float to _Float16 will work perfectly. e.g. [1, 2, 222], [0, -1, 0]
   • However if original floats or distance squares are all out of the range, then scaling must be done during transformation. Otherwise the distance makes no sense at all as they are out of range. e.g. [6789, 1234], [-6789, -1234]
     • for dataset of mnist-784-euclidean, without scaling, the distance are bigger than FLT16_MAX and recall is 0. If divided the float by 1000 during transformation, then the recall becomes 0.978.
   • One possible solution could be to allow for configuring a "proportion" parameter when they choose scalar quantization, which would enable the user to specify the appropriate scaling factor for their specific use case.
   • Another possible solution might be define the corresponding data type (half-vector) and let the users to do the scaling before inserting the data.
2. _Float16 requires CPU instruction set support, otherwise it will revert to float and not utilize SIMD, leading to performance issues. In the commit I’m using -mf16c but looks it could be improved further.