You're right, this should to be documented in the manual.

The short version is here: if you look at the original ed25519 from SUPERCOP, it has a crypto_sign_keypair() function that generates a pair of keys, pk and sk. It starts from generating 32 random bytes — this is the sk. And doing SHA512 on that.

In our case, we use the user password instead of random bytes, and start from SHA512("user password").

The unfortunate consequences of this design is that neither libsodium nor SUPERCOP work for you out of the box, it basically needs a patched crypto_sign_keypair() unless the password is exactly 32 bytes long.

yes I tried to understand the impact of SHA512 but it seems like these libraries are already doing a lot of SHA512 internally on the key as well, so I didn't see a way of following that without understanding the whole thing: https://github.com/pyca/pynacl/blob/master/src/libsodium/src/libsodium/crypto_sign/ed25519/ref10/sign.c#L65

The quickest way this could be done would be if mariadb could allow their crypto_sign() function to be part of the client API, then we could build a Python binding on that and import it from PyMySQL. OR if you could convince libsodium to add a new function (and that's where we'd get to see if they have some kind of issue with this change).

that is, PyMySQL can import some other package we put on pypi that has the thing here written natively, but PyMySQL itself can't package any C code directly.

I can try to provide crypto_sign() or something, but how would it help you? PyMySQL is a pure-python library, you don't link with libmariadb.

I can do a pull request for python-ed25519, but I don't know if there's any chance of it being accepted.

if you look in https://github.com/PyMySQL/PyMySQL/blob/master/pymysql/_auth.py, PyMySQL imports native crypro libraries to help with auth. If you provided some kind of C function, we'd have to look into publishing an all-new package on pypi that provides just this function. I would try to see if this could be maintained underneath the PyMySQL organization on github of which I am a member. This new library would be an optional-only dependency for PyMySQL for the small portion of users that want to use mariadb ed25519 authentication.

Alternatively I noticed there's a mariadb-connector-python apparently, maybe having it exposed there, that would be weird though if PyMySQL relied on a different DBAPI in order to do auth

Agree I don't know the status of python-ed25519. having it be possible via pynacl/libsodium would be the best, if it could occur via some esoteric combination of lower level crypto functions present there, however per my colleague he claims this recent change https://github.com/jedisct1/libsodium/commit/d3976446a0c19aa9fe6f66523741126869a4990e means it is even less possible .

mariadb's decision to fork the algo makes this a tough call and I think having it be a public API function that we can publish Python bindings towards in an entirely standalone library might be the only real solution that has zero chance of breaking.

we may have come up with a patch for libsodium that can potentially get this working using that library directly, if libsodium accepts our patches. this would be a much better path to travel since no new libraries need to be written.

err libsodum -> pynacl

I've done more testing, and starting version 1.0.17,libsodium provides a new low-level crypto API that exposes the functions which are called by crypto_sign_keypair (scalar multiplication on the Ed25519 curve and modular arithmetic).

That means that we can now implement our own Ed25519 signature with arbitrary length passwords like what MariaDB auth_ed25519 does. I published a gist [1] to show what my libsodium-based implementation looks like.

I think we can close this issue now that we have a clean path forward with libsodium 1.0.17.

[1] https://gist.github.com/dciabrin/1295fa9900147ae1de0df9d9e106a278

Thanks for the info!

My current thinking (over the last couple of weeks) is that it was a mistake to "optimize" the plugin in a way that made it incompatible with the standard ed25519 implementations. But unfortunately I still don't see any solution for this that wouldn't invalidate all current users' passwords.

It's great to know that libsodium provided a workaround.

dciabrin where are we going to put that C code ?

Hey Mike, that was just for the sake of the example (and to convince myself that libsodium has what we need to implement auth_ed25519).
For our immediate use in OpenStack we'll probably do a similar python implementation that uses PyNaCl (py binding to libsodium)

I acknowledge, it was a mistake to deviate from the "standard" ed25519, the minor benefit wasn't worth the broken compatibility with 3rd party libraries.

Unfortunately, I don't see anything we can do now. In the next authentication plugin we'll know to avoid this.

If you have any idea how we can fix it without forcing all users to recreate their passwords — please do tell and we'll reopen this issue.