Okay, I see.
After you got this error, without further uninstalling or removing anything, please find the option innodb_flush_method = O_DIRECT in your config files and comment it.
Then run sudo mysql_install_db.
And then try to start the server the way you would normally start it.

i commented out
innodb_flush_method = O_DIRECT inside my.cnf
root@s6:/etc/mysql# mysql_install_db

WARNING: The host 's6' could not be looked up with /usr/bin/resolveip.
This probably means that your libc libraries are not 100 % compatible
with this binary MariaDB version. The MariaDB daemon, mysqld, should work
normally with the exception that host name resolving will not work.
This means that you should use IP addresses instead of hostnames
when specifying MariaDB privileges !
Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
2017-11-16 8:52:50 140192713385856 [ERROR] InnoDB: The Auto-extending innodb_system data file './ibdata1' is of a different size 0 pages than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages!
2017-11-16 8:52:50 140192713385856 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2017-11-16 8:52:51 140192713385856 [ERROR] Plugin 'InnoDB' init function returned error.
2017-11-16 8:52:51 140192713385856 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-11-16 8:52:51 140192713385856 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-11-16 8:52:51 140192713385856 [ERROR] Aborting

Installation of system tables failed! Examine the logs in
/var/lib/mysql for more information.

Okay, that's a bit confusing. Just in case the directory got damaged by the previous failed attempt, keep the option commented, and run

sudo rm -rf /var/lib/mysql/*

sudo mysql_install_db

If it doesn't work this way, then you probably have permission problems on /var/lib/mysql.

But if it helps and mysql_install_db works, please let us know what df -kT /var/lib/mysql say.

root@s6:/rm -rf /var/lib/mysql/*
root@s6:/mysql_install_db
root@s6:/systemctl status mariadb.service
root@s6:/var/lib/mysql# systemctl status mariadb.service
● mariadb.service - MariaDB database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Thu 2017-11-16 10:49:48 CST; 39s ago

yes that worked

root@s6:/var/lib/mysql# df -kT /var/lib/mysql
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/xvda2 ext3 102832424 1536596 96065492 2% /

Thank you

Thanks. ext3 confirms that you've encountered the bug MDEV-14244 . It's already fixed, the fix will be released with the next release 10.2.11.