Hi,

I am not sure what your eventual process will include, but I have successfully made my own local git repos from launchpad using the following method (Debian Wheezy with git 1.8 from wheezy-backports)

apt-get install git-bzr
git clone bzr::/filesystem/path/to/my-maria-tree-branch

The clone about three hours on my PC which is a quad-core phenom and produced a working copy where the .git directory was > 5 Gigabytes... but I could then visualise the history with gitg and see all the maria tags and all the merged past branches, so it did a good job of preserving history.

I could then merge the latest trunk in my bzr branch and then pull those changes into my git tree.

I could also push changes back to my local bzr, which can then be pushed back to launchpad, I haven't tested this recently but I have done it successfully in the past.

So I guess you could run both in parallel for a while, with some delay, and then one day just make github the official version and drop synchronisation from launchpad...

HTH
Andrew

Any move forward ?
Hope to see mariadb on github. what's a good platform for Open Source working together.
You may find the large amount of open source project list on github.
The most important repository:

Apache family

Linus Torwards kernel

Postgres repo

FreeBSD

anything else i missed?

I can try to do some stuff on this, put a repo on github and maybe integrate something with Buildbot.

I tried converting the 10.0 repo:

git init --bare
bzr fast-export ../bzr-10.0 - | git fast-import

It takes around 2 hours, seems to go well even though there are some warnings. This could be used as the basis for a proper switch from bzr to git (so that we set a point in time at which we migrate, after that all work is done in git, and all bzr trees are kept only as read-only mirrors, or just dropped).

It seems there is another solution proposed, which is to use a git-bzr plugin to allow to push from a git repo into a bzr repo. There was even talk of letting 5.1 .. 10.0 trees remain bzr forever and only using git for 10.1... (I think this is likely to cause more problems than it solves).

I can try using git-bzr-ng to clone 10.1. And then test working with ... 10.0 in bzr and 10.1 ... in git.

Ouch, I couldn't even clone the 10.1 repo with git-bzr-ng, I got a bzr
internal error

git bzr clone lp:~maria-captains/maria/10.1 git-10.1

bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

missing chk node(s) for parent_id_basename_to_file_id maps

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 930, in exception_to_return_code

    return the_callable(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1141, in run_bzr

    ret = run(*run_argv)

  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 673, in run_argv_aliases

    return self.run(**all_cmd_args)

  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 697, in run

    return self._operation.run_simple(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 136, in run_simple

    self.cleanups, self.func, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups

    result = func(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1508, in run

    source_branch=br_from)

  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 366, in sprout

    create_tree_if_local=create_tree_if_local)

  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 132, in run

    self.cleanups, self.func, self, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups

    result = func(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 416, in _sprout

    result_repo.fetch(source_repository, fetch_spec=fetch_spec)

  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 1268, in fetch

    find_ghosts=find_ghosts, fetch_spec=fetch_spec)

  File "/usr/lib/python2.7/dist-packages/bzrlib/decorators.py", line 218, in write_locked

    result = unbound(self, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2585, in fetch

    find_ghosts=find_ghosts)

  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 77, in __init__

    self.__fetch()

  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 104, in __fetch

    self._fetch_everything_for_search(search_result)

  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 132, in _fetch_everything_for_search

    stream, from_format, [])

  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2051, in insert_stream

    hint = self.target_repo.commit_write_group()

  File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 633, in commit_write_group

    result = self._commit_write_group()

  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1727, in _commit_write_group

    hint = self._pack_collection._commit_write_group()

  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1607, in _commit_write_group

    "Cannot add revision(s) to repository: " + problems_summary)

BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

missing chk node(s) for parent_id_basename_to_file_id maps

bzr 2.6.0dev2 on python 2.7.3 (Linux-3.13.0-kn-x86_64-with-debian-7.4)

arguments: ['/usr/bin/bzr', 'branch', 'lp:~maria-captains/maria/10.1',

    '/home/knielsen/my/10.0/git-ng/git-10.1/.git/bzr/repo/master']

plugins: bash_completion[2.6.0dev2], bzr_merge_into[unknown], bzrtools[2.5.0],

    changelog_merge[2.6.0dev2], email[unknown], fastimport[0.13.0],

    gtk[0.99.1], launchpad[2.6.0dev2], netrc_credential_store[2.6.0dev2],

    news_merge[2.6.0dev2], po_merge[2.6.0dev2], qbzr[0.22.2], rewrite[0.6.3],

    search[1.7.0dev], weave_fmt[2.6.0dev2]

encoding: 'utf-8', fsenc: 'UTF-8', lang: 'en_DK.UTF-8'

*** Bazaar has encountered an internal error.  This probably indicates a

    bug in Bazaar.  You can help us fix it by filing a bug report at

        https://bugs.launchpad.net/bzr/+filebug

    including this traceback and a description of the problem.

ERROR:root:Command "bzr branch lp:~maria-captains/maria/10.1 /home/knielsen/my/10.0/git-ng/git-10.1/.git/bzr/repo/master" failed.

Hi Kristian,

I had more luck using git with bzr transport - see my comment dated 25 January

I documented some experiments I conducted converting between bzr and git on my blog - see http://blog.oldcomputerjunk.net/2014/launchpad-to-github-round-trip-save-5-giabytes-on-the-way/ - although the latter part that will not be directly relevant

Re options for git migration related to your previous comment

You may wish to consider 'chopping' the history available in the main working Github to a few years, just to make it more manageable from a size perspective. Obviously retaining the prior history is important for a number of reasons, so one solution might be to have two repos:

• a 'MariaDB - Full History' repo
• a MariaDB active repo (say, >= 2007 to pick a completely arbitrary number)

Development would continue in the active repo, and anyone wanting to do digital archeaology on mysql + mariadb could analyse the Full History repo. I believe that theoretically it would be possibly to regularly / automatically "top up" the full history wth new work from the active repo, to satisfy the needs of any future historians...
(Real world example of digital archealogy: Dave Chinners analysis of the history of xfs - http://mirror.linux.org.au/linux.conf.au/2014/Wednesday/47-Linux_Filesystems_Where_did_they_come_from_-_Dave_Chinner.mp4 )

This is just an idea, I have no idea if it is desirable...

It's true that the immediate result from fast-import is around 5 GB. But after "git repack -a -d --depth=250 --window=250 -f" it's down to less than 300MB, which should be fine (the checked-out source tree is larger than that)

Thats actually quite an impressive reduction. I got 223 MB just now ... I guess I never thought to go try that on the full tree once I worked out how to extract just the storage engine code! (I think I was a bit over waiting for git to do its thing by then)
(Still I'm glad I did go through the exercise because there are many many commits in between just the ones I wanted to see...)

This is good, cloning is way faster now.

It actually made gitg load a little faster which is quite interesting. It still takes several seconds to count through 80000 commits though which is a little annoying.

But I think for me it makes maintaining a separate storage engine repo more of an intellectual curiosity now!

Looking forward to a mariadb github becoming reality...

I tried converting first 10.0 and then 10.1 with fast-export |fast-import. It seems to work well.
I was able to push the 10.1 into the repo as a new branch, it worked.
I was then also able to merge 10.0->10.1 (inside git), also seemed to work well.

I can maybe try something in buildbot next.

I think I got most of the pieces in place.

I created a git repo on https://github.com/knielsen/mariadb. I was able to
eg. merge 10.0 -> 10.1 by re-converting 10.0 to git, push that to the 10.0
branch in github, merge that into the master branch in github (so all merging
can be done using only git).

We will want to convert as many branches as possible to git eventually, as the
conversion is rather heavy - it takes 2 hours.

git init --bare
bzr fast-export ../bzr-10.0 - | git fast-import
git repack -a -d --depth=250 --window=250 -f

I was able to get Buildbot integration working. Github does a HTTP POST
request to Buildbot when a new change is pushed. The request can be manually
resent if it gets lots in Buildbot for some reason.

Once it is received by Buildbot, the push is just another change, same as the
existing bzr changes.

We need to install git on all buildslaves that do source checkout from
repositories. We also need to update all builders in the Buildbot config to
use git for checkouts. The required changes are not hard (I did a couple
already). It is also possible to conditionally do a bzr or git checkout,
depending on which branch it is.

Your github seems to have lost all the past tags - is that what you were intending?

The tags are in my local repo. Maybe they need to be pushed to github explicitly (I have not worked much with tags in git before)

Right, it seems so:

https://help.github.com/articles/pushing-to-a-remote#pushing-tags

note the github release functionality can also be used to tag and release at the same time however pushing tags is definitely appropriate for older releases.

Stewart Smith just pointed me to the scripts he used to convert Percona repos to git.
It would seem useful to check and try to do the same thing they did:

https://github.com/stewartsmith/bzr-to-git-conversion-scripts

Hi, I just tried these scripts and it died 15 minutes in, so will need to dive in and see what broke.

Some stuff probably unrelated to why it failed but important to maria:

• there is a hack with sed in the middle that rewrites someones email address
• these is a text file with user to email mappings in the repository that also gets referenced
  Both the above are certainly specific to Percona and need to be removed (or replaced) for Maria as needed

The script seems to make a second branch of maria bzr for some reason before performing a bzr fast-export and git fast-import that

I made some tweaks to the script tp be more generic and fixed and documented missing dependencies; am trying again. Will push changes to my github

Ok, dependencies sorted and running a test now, modified script at https://github.com/andymc73/bzr-to-git-conversion-scripts/tree/maria-10