Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27142

Windows - Disable text mode when opening files

Details

    Description

      By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

      This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since Notepad gained ability to read text files with whatever ending , none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

      Yet, the added CRLF adds some obscurity whenever it comes to MariaDB (search for 'Win' in this test ported from MySQL.

      Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
      https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

      Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the source code , without much explanation, and then many extra lines were added in the test case to compensate for platform differences (I actually doubt that --binary-mode works on Windows, at all)

      The plan

      • set binary mode on stdin,stdout,stderr, as well as set_fmode() to prevent text-mode by default, already during my_init.
      • remove questionable code fragments (basically, search for '\r' mentioned in code, in conjunction with Windows)
      • fix questionable tests that assume Windows would be different and handle \r in its special way.

      Attachments

        Activity

          wlad Vladislav Vaintroub created issue -
          wlad Vladislav Vaintroub made changes -
          Field Original Value New Value
          Description By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB (just look at this [test case|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences.
          By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB ([search for 'Win' in this test |https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences.
          wlad Vladislav Vaintroub made changes -
          Description By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB ([search for 'Win' in this test |https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences.
          By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB ([search for 'Win' in this test |https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences.

          The plan
          - set binary mode on stdin,stdout,stderr, as well as set_fmode() to prevent text-mode by default, already during my_init.
          - remove questionable code fragments.
          wlad Vladislav Vaintroub made changes -
          Description By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB ([search for 'Win' in this test |https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences.

          The plan
          - set binary mode on stdin,stdout,stderr, as well as set_fmode() to prevent text-mode by default, already during my_init.
          - remove questionable code fragments.
          By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB ([search for 'Win' in this test |https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences.

          The plan
          - set binary mode on stdin,stdout,stderr, as well as set_fmode() to prevent text-mode by default, already during my_init.
          - remove questionable code fragments (basically, search for '\r' mentioned in code, in conjunction with Windows)
          - fix questionable tests that assume Windows would be different and handle \r in its special way.
          wlad Vladislav Vaintroub made changes -
          Description By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB ([search for 'Win' in this test |https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences.

          The plan
          - set binary mode on stdin,stdout,stderr, as well as set_fmode() to prevent text-mode by default, already during my_init.
          - remove questionable code fragments (basically, search for '\r' mentioned in code, in conjunction with Windows)
          - fix questionable tests that assume Windows would be different and handle \r in its special way.
          By default, C runtime when creating or opening files would use so-called text mode , which means translating LF to CRLF when writing, and CRLF to LF when reading.

          This is because traditionally, Windows text files used CRLF to indicate end of line. For a while already, since [Notepad gained ability to read text files with whatever ending | https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/], none of this is necessary. CRT and other libraries have supported reading and writing text files with LF, since very long time.

          Yet, the added CRLF adds some obscurity whenever it comes to MariaDB ([search for 'Win' in this test |https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0] ported from MySQL.

          Moreover, there had been subtle bugs in text mode over the years in the C runtime. MDEV-9409 had been quit a pain, and recent one I came across,
          https://developercommunity.visualstudio.com/t/-write-fails-to-write-non-ascii-characters-to-cons-3/248245

          Nobody benefits from having a extra character inserted and removed every line, at least not since Notepad was fixed. But I think MariaDB would benefit very much from removing obscure code fragments, where it reads "remove \r unconditionally on Windows" in the [source code|https://github.com/MariaDB/server/commit/2db4392bf4cb0447fd8f09fe85997f34b9c697d0#diff-d2b16182d64c1e3f0d662420a2b3bb1b055dad4afcea0094141935fe505154e4R73] , without much explanation, and then many extra lines were [added|https://github.com/MariaDB/server/blob/10.8/mysql-test/main/mysql_binary_mode.test#L61] in the test case to compensate for platform differences (I actually doubt that --binary-mode works on Windows, at all)

          The plan
          - set binary mode on stdin,stdout,stderr, as well as set_fmode() to prevent text-mode by default, already during my_init.
          - remove questionable code fragments (basically, search for '\r' mentioned in code, in conjunction with Windows)
          - fix questionable tests that assume Windows would be different and handle \r in its special way.
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 128040 ] MariaDB v4 [ 131604 ]
          wlad Vladislav Vaintroub made changes -
          Fix Version/s 10.9 [ 26905 ]
          Fix Version/s 10.8 [ 26121 ]
          wlad Vladislav Vaintroub made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          wlad Vladislav Vaintroub made changes -
          issue.field.resolutiondate 2022-07-21 19:42:51.0 2022-07-21 19:42:51.677
          wlad Vladislav Vaintroub made changes -
          Fix Version/s 10.9.2 [ 27115 ]
          Fix Version/s 10.9 [ 26905 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]
          julien.fritsch Julien Fritsch made changes -
          Summary Windows - Disable text mode when opening files Windows - Disable text mode when opening files

          People

            wlad Vladislav Vaintroub
            wlad Vladislav Vaintroub
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.