Blog.

Merge multiple gradle projects including git history

MF

Marco Franssen /

10 min read1952 words

Cover Image for Merge multiple gradle projects including git history

As we started 2 years ago with a micro-service architecture we faced some issues in the productivity of the teams a while ago. This made me think what we could do about that from an architecture and development point of view. As we used different Git repos for the various components of our micro services in the long run it became harder and harder for developers to work with those components. At that point in time I decided to simplify the development workflow to merge the different Git repos of one functional bounded context into one Git repository.

Finally I found some time to document this process in the form of a blogpost for my own reference, but also to share this with the community. In case you are thinking, I'm not using Gradle and are at the point to quite reading, please don't! Only last bit goes Gradle specific. The first 75% you can reuse for whatever kind of project you are running.

To keep impact for development team low to non-existing I wrote a migration script in bash to be able to do the migration within a couple of minutes and do some dry runs to verify if everything was still working. By everything I mean the following parts:

  • Do the projects still compile?
  • Do the unit and integration test still succeed?
  • Does the continuous integration still work on Jenkins?
  • Does the continuous deployment still work on Kubernetes?

Lets start with a folder structure example which represents an similar situation we faced. In every folder we had a Git repository containing the Gradle project including a Jenkinsfile and some other common files for CI and CD purposes.

$ tree code-folder
code-folder
├── domain-a-api
├── domain-a-commands
├── domain-a-consumer
├── domain-a-events
├── domain-b-api
├── domain-b-commands
├── domain-b-consumer
└── domain-b-events

8 directories, 0 files

I'll zoom in for the remainder of this article on domain-a. The goal is to end up with a structure like the following.

$ tree -a code-folder/domain-a
code-folder/domain-a
├── .git
├── Jenkinsfile
├── api
│   └── src
├── build.gradle
├── commands
│   └── src
├── consumer
│   └── src
└── events
    └── src

9 directories, 2 files

The first step is to move all files in the existing repositories into a subfolder. For that I wrote a bash function to be reused in all the projects I had to migrate.

function move_into_subfolder () {
    local subfolder=$1
    if [ ! -z "$2" ] ; then
        subfolder=$2
    fi

    pushd $1
    mkdir $subfolder
    git mv `ls -1 | grep -v $subfolder` $subfolder
    git mv .gitignore $subfolder
    git commit -a -m "Moving files into subfolder $subfolder"

    # navigate back to previous folder
    popd
}

This function can be used as following, move_into_subfolder "domain-a-api" "api", which will result in all files within the folder domain-a-api to be moved into domain-a-api/api. The move of files will also be committed in the existing repository.

As I was doing this for many repos and I had to make sure I was merging the latest develop branches of these repositories I also wrote a small bash function to clone or pull the latest develop branches from the repositories I was about to merge.

function clone_or_pull_develop() {
    if [ ! -d "${1}" ] ; then
        git clone -b develop [email protected]:marcofranssen/${1}.git
    else
        pushd ${1}
        git reset HEAD~1 --hard
        git pull
        popd
    fi
}

This function clones get's the latest changes from the develop branch and removes the last move-into-subfolder commit so you can run the function over and over again for testing if all comes allong nicely.

Now we have a way to have all the repositories prepared to be merged as one repository without loosing the commit history. Also for this action I wrote a small bash function to be easily reused on the various repositories I wanted to merge.

function merge_old_repo_commits () {
    git remote add $1 ../$1
    git fetch $1
    git merge --no-edit --allow-unrelated-histories $1/develop
    git remote rm $1
}

To start we will have to prepare a new git repository first and then I can call my new bash function to merge the repositories in one.

rm -rf domain-a #So we can easily restart from scratch
mkdir -p domain-a
cd domain-a

git init
git remote add origin [email protected]:marcofranssen/domain-a.git

merge_old_repo_commits "commands"
merge_old_repo_commits "events"
merge_old_repo_commits "consumer"
merge_old_repo_commits "api"

The result is we got all the 4 separate repository commit histories merged together in this single repository. Last but not least we will do a few additional commits on top of the merged repositories to make it a fully working multi-module gradle repository. This is probably where you would to go your own route to do your project specific modifications. As a reference I will show you below what I did.

First off all I made a new commit adding a new README.md file in the root of the project which links to the README.md files in my subprojects.

touch README.md
echo '# Domain A' > README.md
echo '' >> README.md
echo '## Api' >> README.md
echo '' >> README.md
echo '[README.md](api/README.md)' >> README.md
echo '## Consumer' >> README.md
echo '' >> README.md
echo '[README.md](consumer/README.md)' >> README.md
echo '## Commands' >> README.md
echo '' >> README.md
echo '[README.md](commands/README.md)' >> README.md
echo '## Events' >> README.md
echo '' >> README.md
echo '[README.md](events/README.md)' >> README.md
git add README.md
git commit -m "Add readme to root of project, linking to the module README.md files"

Then I made a commit where I add a new .gitignore file which I manualy merged and remove the original .gitignore files.

cp ../.gitignore .
git add .gitignore
git rm api/.gitignore consumer/.gitignore events/.gitignore commands/.gitignore
git commit -m "Add .gitignore to root of project and remove .gitignore files from subfolders"

Same I did for my Gradle files so I have one file to manage the Gradle build from the root of the new project folder.

gradle init
cp ../gradle* ../settings.gradle .
git add .
git update-index --chmod=+x gradlew
git commit -m "Initialized gradle project"

cp ../build.gradle .
echo "include 'api', 'consumer', 'events', 'commands'" >> settings.gradle
git add build.gradle settings.gradle
git rm -r api/build.gradle api/settings.gradle api/gradle*
git rm -r consumer/build.gradle consumer/settings.gradle consumer/gradle*
git rm -r events/build.gradle events/settings.gradle events/gradle*
git rm -r commands/build.gradle commands/settings.gradle commands/gradle*

git commit -m "Add new gradle configuration and remove the old ones"

Last but not least I also put in place new docker-compose setup and remove the old files. And put in place my updated Jenkinsfile which I also merged manually.

cp ../docker-compose.yaml .
git add docker-compose.yaml
git rm api/docker-compose.yml consumer/docker-compose.yml
git commit -m "Add docker-compose file in the root of project replacing the specific module ones"

cp ../ApiDockerfile api/Dockerfile
cp ../ConsumerDockerfile consumer/Dockerfile
git add api/ consumer/
git commit -m "Updated Dockerfiles"

# Put in place new Jenkins config
cp ../Jenkinsfile .
git add Jenkinsfile
git rm api/Jenkinsfile
git rm consumer/Jenkinsfile
git rm events/Jenkinsfile
git rm commands/Jenkinsfile
git commit -m "Add new Jenkinsfile and remove the old ones"

As you can imagine I had to run my script of few times from scratch to fix the bugs and mistakes. So in the end I tested all by running the gradle build and pushing it to my repo to trigger the Jenkins build.

./gradlew build

Once all succeeded I planned with the impacted team a 10 minute code freeze (grabbed some coffee) and then we continued our work on the new repository. Once the first one finished the other ones where more easy as most our projects use a similar setup.

TL;DR

Here the whole summary in one script. Including my folder layout from where I executed this script.

$ tree -a my-merge-folder
my-merge-folder
├── .gitignore
├── ApiDockerfile
├── ConsumerDockerfile
├── Jenkinsfile
├── build.gradle
├── docker-compose.yml
├── gradle.proeprties
├── merge-repos.sh
└── settings.gradle

0 directories, 9 files

I omit all my project specific files that I merged manually, but will provide you with the full script below. It is up to you to put all together.

#!/bin/bash

function move_into_subfolder () {
    local subfolder=$1
    if [ ! -z "$2" ] ; then
        subfolder=$2
    fi

    pushd domain-a-$1
    mkdir $subfolder
    # git mv !($1) $1 # this bashism doesn't seem to work on windows git and is replaced by following 2 lines
    git mv `ls -1 | grep -v $subfolder` $subfolder
    git mv .gitignore $subfolder
    git commit -a -m "Moving files into subfolder $subfolder"

    # navigate back to previous folder
    popd
}

function merge_old_repo_commits () {
    git remote add $1 ../$1
    git fetch $1
    git merge --no-edit --allow-unrelated-histories $1/develop
    git remote rm $1
}

function clone_or_pull_develop() {
    if [ ! -d "${1}" ] ; then
        git clone -b develop [email protected]:marcofranssen/${1}.git
    else
        pushd ${1}
        git reset HEAD~1 --hard
        git pull
        popd
    fi
}

# Pull the latest develop branches and move the files in a subfolder as preparation for the merge
clone_or_pull_develop "domain-a-api"
move_into_subfolder "domain-a-api" "api"
clone_or_pull_develop "domain-a-commands"
move_into_subfolder "domain-a-commands" "commands"
clone_or_pull_develop "domain-a-consumer"
move_into_subfolder "domain-a-consumer" "consumer"
clone_or_pull_develop "domain-a-events"
move_into_subfolder "domain-a-events" "events"

# Prepare a new git repo to execute the merge
rm -rf domain-a
mkdir -p domain-a
cd domain-a

git init
git remote add origin [email protected]:marcofranssen/domain-a.git

merge_old_repo_commits "commands"
merge_old_repo_commits "events"
merge_old_repo_commits "consumer"
merge_old_repo_commits "api"

#######################################################
# Below you would like to customize to your own needs #
#######################################################

# Add a new README.md to the root of the project.

touch README.md
echo '# Domain A' > README.md
echo '' >> README.md
echo '## Api' >> README.md
echo '' >> README.md
echo '[README.md](api/README.md)' >> README.md
echo '## Consumer' >> README.md
echo '' >> README.md
echo '[README.md](consumer/README.md)' >> README.md
echo '## Commands' >> README.md
echo '' >> README.md
echo '[README.md](commands/README.md)' >> README.md
echo '## Events' >> README.md
echo '' >> README.md
echo '[README.md](events/README.md)' >> README.md
git add README.md
git commit -m "Add readme to root of project, linking to the module README.md files"

# Put in place the manually merged .gitignore file and remove the old ones

cp ../.gitignore .
git add .gitignore
git rm api/.gitignore consumer/.gitignore event/.gitignore command/.gitignore
git commit -m "Add .gitignore to root of project and remove .gitignore files from subfolders"

# Put in place the manually merged gradle file and remove the old ones
gradle init
cp ../gradle* ../settings.gradle .
git add .
git update-index --chmod=+x gradlew
git commit -m "Initialize gradle project"

cp ../build.gradle .
echo "include 'api', 'consumer', 'events', 'commands'" >> settings.gradle
git add build.gradle settings.gradle
git rm -r api/build.gradle api/settings.gradle api/gradle*
git rm -r consumer/build.gradle consumer/settings.gradle consumer/gradle*
git rm -r events/build.gradle events/settings.gradle events/gradle*
git rm -r commands/build.gradle commands/settings.gradle commands/gradle*

git commit -m "Add new gradle configuration and remove the old ones"

# Add a new enhanced docker composer for better dev experience
cp ../docker-compose.yaml .
git add docker-compose.yaml
git rm api/docker-compose.yml consumer/docker-compose.yml
git commit -m "Add docker-compose file in the root of project replacing the specific module ones"

cp ../ApiDockerfile api/Dockerfile
cp ../ConsumerDockerfile consumer/Dockerfile
git add api/ consumer/
git commit -m "Update Dockerfiles"

# Put in place new Jenkins config
cp ../Jenkinsfile .
git add Jenkinsfile
git rm api/Jenkinsfile
git rm consumer/Jenkinsfile
git rm events/Jenkinsfile
git rm commands/Jenkinsfile
git commit -m "Add new Jenkinsfile and remove the old ones"

./gradlew build

git push -f -u origin master
git co -b develop
git push -f -u origin develop

I got inspired by an nice post on Medium.com from an ex colleague of mine. Thanks Fred! :) Please share this article if you liked it and as always I would love your feedback in the comments below.

You have disabled cookies. To leave me a comment please allow cookies at functionality level.

More Stories

Cover Image for Start on your first Golang project

Start on your first Golang project

MF

Marco Franssen /

A couple of months ago I started to do some coding in Go a.k.a Golang. Not only because they have an awesome logo ;-). My main reason was because I wanted to have something running as bare metal as possible on my Raspberry Pi and I wanted to have it available for different platforms to be easy to install. Some other reasons are the ease of creating async code by using Go in front of your methods and the unique approach of channels to sync between go routines (threads). I have been reading a lot…

Cover Image for How to add network driver to Windows 10 PE

How to add network driver to Windows 10 PE

MF

Marco Franssen /

Very recently I have been trying to reinstall my Laptop using my WinPE approach as it didn't have a optical drive anymore. However my problem was that the WinPE image I created was lacking the network driver for my laptop. So then I recreated a WinPE USB drive using the Windows 10 ADK, hoping it would include the required driver. However unlucky me I still had no network when I booted by new laptop using my new Windows 10 PE USB drive. Therefore I had to add the network driver for my laptop to t…

Cover Image for Responsive Adaptive Progressive impressive webpages

Responsive Adaptive Progressive impressive webpages

MF

Marco Franssen /

In the last couple of years web applications technologies and frameworks went through a fast paced transformation and evolution. For all these evolutions there was coined a marketing term which (by coincidence) all end on …ive. So lets coin another one. In this article I'm going to explain you the basic concepts of all these principles which combined allow you to build an impressive web application. Responsive Web Design It started all back in (from top of my head) late 2010, with the idea of…

Cover Image for Upgrade Raspbian Jessie to RaspbianStretch

Upgrade Raspbian Jessie to RaspbianStretch

MF

Marco Franssen /

Very recently I have upgraded my Raspberry 3 to the new Raspbian OS, code named "Stretch". Due to some security issues in the chipset of the Raspberry Pi 3 and Raspberry zero, I decided to upgrade mine to Raspbian Stretch, which resolves these security issues. Before you get yourself into any trouble make sure to have a backup of any important data and also please note I can't be responsible for any data loss. In this guide I also assume you know how to connect to your raspberry using ssh and h…