[MDEV-6521] HELP Contents need to use MariaDB Knowledgebase content Created: 2014-08-03  Updated: 2022-05-31  Resolved: 2019-07-01

Status: Closed
Project: MariaDB Server
Component/s: Documentation
Fix Version/s: 10.4.7

Type: Task Priority: Major
Reporter: Ian Gilfillan Assignee: Ian Gilfillan
Resolution: Fixed Votes: 3
Labels: None

Issue Links:
PartOf
includes MDEV-5738 online help for CHANGE MASTER TO not ... Closed
includes MDEV-7120 MariaDB 10 help tables are obsolete Closed
includes MDEV-8145 repair view missing from scripts/fill... Closed
includes MDEV-16687 former MySQL doc links in help tables Closed
Relates
relates to MDEV-28701 Update server HELP contents Closed
relates to MDEV-19890 ANALYZE documentation Closed
relates to MDEV-23095 No help topics for prepared statement... Open

 Description   

The problem

Currently, the HELP command in any MariaDB client will return data that is generated from the MySQL documentation. The URL's point to the MySQL documentation instead of MariaDB's, and the text generated is in some cases not accurate. The HELP content is generated from the mysql_install_db and the fill_help_table.sql scripts, which are the MySQL versions.

There is also scope to greatly improve the existing HELP, for example including variables in the help contents.

The solution

In 2013, I wrote a script that would replace all MySQL URL's with MariaDB ones. This worked well and would be relatively easy to maintain, but was not a complete solution, as it did not solve the problem of the inaccurate text.

The ideal solution as I can see it would be to have the HELP contents all automatically generated from the MariaDB Knoweldgebase.

Not all content from the Knowledgebase needs to be included (blog posts, news articles and case studies spring to mind, so the Knowledgebase would need to be amended to cater for selection)

The existing structure

The HELP contents are in the following format. There are four relevant tables in the mysql database: help_topic, help_category, help_keyword and help_relation.

DESC help_keyword;
+-----------------+------------------+------+-----+---------+-------+
| Field           | Type             | Null | Key | Default | Extra |
+-----------------+------------------+------+-----+---------+-------+
| help_keyword_id | int(10) unsigned | NO   | PRI | NULL    |       |
| name            | char(64)         | NO   | UNI | NULL    |       |
+-----------------+------------------+------+-----+---------+-------+

The help_keyword table consists of individual keywords, such as "SHA" or "INNODB". Entering "HELP xxx" will return

DESC help_topic;
+------------------+----------------------+------+-----+---------+-------+
| Field            | Type                 | Null | Key | Default | Extra |
+------------------+----------------------+------+-----+---------+-------+
| help_topic_id    | int(10) unsigned     | NO   | PRI | NULL    |       |
| name             | char(64)             | NO   | UNI | NULL    |       |
| help_category_id | smallint(5) unsigned | NO   |     | NULL    |       |
| description      | text                 | NO   |     | NULL    |       |
| example          | text                 | NO   |     | NULL    |       |
| url              | char(128)            | NO   |     | NULL    |       |
+------------------+----------------------+------+-----+---------+-------+

help_topic contains the final description that is displayed, as well as the URL.

DESC help_category;
+--------------------+----------------------+------+-----+---------+-------+
| Field              | Type                 | Null | Key | Default | Extra |
+--------------------+----------------------+------+-----+---------+-------+
| help_category_id   | smallint(5) unsigned | NO   | PRI | NULL    |       |
| name               | char(64)             | NO   | UNI | NULL    |       |
| parent_category_id | smallint(5) unsigned | YES  |     | NULL    |       |
| url                | char(128)            | NO   |     | NULL    |       |
+--------------------+----------------------+------+-----+---------+-------+

help_category contains a list of categories, for example "Geographic Features". Searching for help by category returns a list of sub-categories and related topics, for example:

HELP Geographic Features;
You asked for help about help category: "Geographic Features"
For more information, type 'help <item>', where <item> is one of the following
topics:
   GEOMETRY
   GEOMETRY HIERARCHY
   SPATIAL
categories:
   Geometry constructors
   Geometry properties
   Geometry relations
   LineString properties
   MBR
   Point properties
   Polygon properties
   WKB
   WKT

DESC help_relation;
+-----------------+------------------+------+-----+---------+-------+
| Field           | Type             | Null | Key | Default | Extra |
+-----------------+------------------+------+-----+---------+-------+
| help_topic_id   | int(10) unsigned | NO   | PRI | NULL    |       |
| help_keyword_id | int(10) unsigned | NO   | PRI | NULL    |       |
+-----------------+------------------+------+-----+---------+-------+

help_relation describes a relation between a topic and a keyword. For example, topic "CREATE TABLE" is related to keyword "ROW_FORMAT". If one searches for "HELP ROW_FORMAT", the content for the topic is returned, as if one had searched for "HELP CREATE TABLE".

Generating the content

To generate the content using the same structure,

In the Knowledgebase tool (admin tool, or simply when editing), add a checkbox labeled "Include in HELP contents", default unchecked. If checked, it becomes mandatory to link the page to a Category (the Category could also be automatically generated from the page's Knowledgebase Category), and optional to link to one or more Keywords. The keyword must be unique to that page, the category can be shared by other pages. A Knowledgebase page that is itself category can also be linked to another category as a child category.

With this structure in place, a script then runs to generate the contents of fill_help_table.sql. Here's a broad spec of what the script needs to:

  • empty each of the four tables, then include a top level "Contents" category
  • traverse each page of the knowledgebase
  • if "Include in HELP contents" is false, move on to the next page, otherwise continue as below
  • insert a new record into help_topic (see below), using the title of the page as the name, and the page URL as url with the remaining fields as determined below.
  • see if the category the page has been linked to exists in the help_category table.
  • If not, insert a record into help_category table, including the parent category if applicable (which may itself need to be inserted here). The url field is always kept empty in the MySQL version of the page. This can be populated or left empty, but I am not sure what effect this has on anything.
  • if the page is linked to a keyword, insert the keyword and relationship in the help_keyword and help_relation tables.

When adding a record to the help_topic table, the Knowledgebase formatting has to be sensibly parsed out. Here are some (I'm sure by no means complete) considerations:

  • Many pages contain a Syntax/Description/Examples template. These are headings. All headings should be included, with macros (e.g. contents, product macro) and macro formatting removed.
  • I am very keen on the idea of including HELP content for each of the variables. Deep links exist so they can be accessed directly, but I haven't yet given this much thought on how this could be automatically generated.
  • To be fleshed out.


 Comments   
Comment by Bryan Alsdorf [ 2014-08-05 ]

It is easy to add new fields to the admin page for the an article / category. More complicated is the script that actually turns this into fill_help_table.sql. If we want it to run as part of the KB I can write it to directly access the database and generate the content, otherwise I can make the needed data available via json.

The creoleparser we use has a method of exporting to text which is fairly good about what it strips and what it leaves. We can change this as needed.

For deep linking variables, we possibly could surround variables or other content in a macro and assign a keyword that way? We would have to discuss exactly how this works.

I'd be happy to work on this, I just need to sort out with Rasmus when I can do this. Unfortunately I have a decent queue of items in front of me at the moment.

Comment by Ian Gilfillan [ 2014-08-07 ]

OK great. If adding new fields is relatively trivial, and you're happy that's the best way to go ahead, then feel free to go ahead and add these - it'll take a while to correctly tag everything in the knowledgebase, and I can get started with that in the meantime.

Adding a macro around each variable description would work, but it would be quite tedious to add for every variable. What about getting the script to use the table on https://mariadb.com/kb/en/mariadb/documentation/variables-and-modes/full-list-of-mariadb-options-system-and-status-variables/ - all variables will always be listed there. If we want to add more descriptions that are parts of other pages later (I can't think of any besides the variables for now), we could add this to the script rather than tag on the knowledgebase.

A final issue is linkrot. At the moment, the KB gracefully handles page renames, but can't handle cases where variables may move. For example, the sole OQGRAPH status variable sits at https://mariadb.com/kb/en/mariadb/documentation/optimization-and-tuning/system-variables/server-status-variables/#oqgraph_boost_version If, in the future, there are more OQGRAPH status variables, these will move to their own page, and the URL will change to something like:
https://mariadb.com/kb/en/mariadb/documentation/storage-engines/oqgraph/oqgraph-system-variables/oqgraph_boost_version

This has already happened on some of the KB pages, and as more forums and external sites, along with the MariaDB client HELP, link to our KB, it would be good not to lose these. Is there a way to do this currently? If not, perhaps either the table or macro around the subsection idea could also be used to maintain these links.

Comment by Bryan Alsdorf [ 2014-09-10 ]

Hi Ian,

Unfortunately I am not going to be able to work on this for a little while. I have some other projects that Rasmus wants to take priority.

I think I should be able to tackle this middle of October.

Comment by Ian Gilfillan [ 2014-11-13 ]

Hi Bryan, any news on this one? If you're too busy, I can take a look at it, but will need help with access and adding the new fields to the KB.

Comment by Bryan Alsdorf [ 2014-11-15 ]

No news until you reminded me. I had a few things come up that took my attention.

I'm going to start this just by adding the new fields and seeing where we can go from there. Instead of adding the fields to the nodes I'm actually going to create separate tables so we have more flexibility and this can be developed / updated without impacting anything else.

For the main table I believe we would need:

  • Node (links to the node)
  • Category
  • Keywords

Do you want me to just pull the list of categories from mysql.help_category or do you want to manage this through the admin interface? Same thing with keywords.

Comment by Sergei Golubchik [ 2014-11-15 ]

bryan, doesn't KB already have it? It has links and categories already. Keywords — I'd say it has them too, that's what search prompts show.

On a more general note, instead of export from KB to help tables and to any other format separately, it'd be easier to export the manual pages from KB to, say, docbook. And all other formats can be generated from it. Only one conversion from KB. And more consistent output in all generated formats, whatever they can be in the future.

Comment by Bryan Alsdorf [ 2014-11-17 ]

All the data is there, but it isn't linked in a formal way where we can generate the help files I don't believe. We need a way to mark which articles should be exported. While we can re-arrange the KB as needed what we want for the help files might be different then what we want for the help database.

As far as the exporting goes, my plan was to just export title and text contents so Ian can handle the rest but if we can easily do docbook that sounds good.

Comment by Ian Gilfillan [ 2014-11-18 ]

I seem to remember that Daniel was also working on the reverse, using docbook content to generate KB content. I'll ping him so that he can give input just to avoid any duplication.

There are regular requests for documentation in pdf and other downloadable formats too, so solving this in one consistent way would be great.

But at the very least as Bryan says there does need to be a way to mark that content should be in the help contents.

I would prefer to manage things from the admin interface rather than just using the mysql categories, as these could be improved or changed in future.

Be sure to take note of the point above about including variables though, that's an important point. I would really like to be able to type "HELP aria_block_size", for example, and only get content from https://mariadb.com/kb/en/mariadb/documentation/storage-engines/aria/aria-system-variables/#aria_block_size section. That would be a big usability gain.

Comment by Bryan Alsdorf [ 2014-11-19 ]

Okay, so I'm going make a new admin section for the help tables. In addition to the table I described above I'll make category and keyword tables populated initially from the system tables.

For the variables, yes I should be able to extract the content from that table and the linked content. How do we want them included in the export? Do they all belong to a category of "Variables" with the variable name as the keyword? Or do they need to be in a sub category?

Comment by Ian Gilfillan [ 2014-11-19 ]

Category would be "Variables", sub-categories would be "Aria System Variables", "Aria Status Variables", "MyISAM System Variables", "General System Variables" etc. and then the individual variable keywords.

Comment by Bryan Alsdorf [ 2014-11-20 ]

So we will need a way to organize these in the admin or split https://mariadb.com/kb/en/mariadb/documentation/variables-and-modes/full-list-of-mariadb-options-system-and-status-variables/ up into different sections.

Comment by Ian Gilfillan [ 2014-12-03 ]

Suggestion from Monty based on how MySQL used to do it. Add tags in the article text to allow portions of an article to be included, in cases where we don't want the entire article to be included.

Comment by Elena Stepanova [ 2015-06-24 ]

Help contents have not been updated for nearly three years. One or another way, it needs to be done.
Also, I added 10.0 to the list of 'Fix version/s' – while 5.5 might be more or less okay, since it does not differ that much from MySQL, for 10.0 it's certainly a problem.

Comment by Ian Gilfillan [ 2015-06-24 ]

This is stalled at the moment as there's been no progress on generating the contents from the Knowledge Base. I have given Monty an updated version of the fill_help_tables.sql script that I made in December, which keeps the same contents, but at least links to the MariaDB KB, but this hasn't been uploaded yet.

Bryan, has there been any progress on the changes above?

Comment by Ian Gilfillan [ 2015-08-18 ]

This was updated with this commit: https://github.com/MariaDB/server/commit/d983565

Still mostly generated from the old MySQL content, but at least it points to the updated documentation on the MariaDB KB now. I have access to the KB db now, and will look at generating the content automatically at some point.

Comment by Ian Gilfillan [ 2016-08-22 ]

As per Federico's suggestion on the mailing list, HELP should generate a warning until this is fixed.

Comment by Gisbert W. Selke [ 2019-07-01 ]

Thank you, all of you who were involved in fixing this!

Generated at Thu Feb 08 07:12:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.