Synchronising GitHub and an Internal Git server

Note: I found this mini How-To while having a clean-up of my GitHub repositories. I figured it would be worth sharing on my blog. Hopefully it is of use to someone. If you want to play around with the steps, but don’t want to use one of your existing projects, you can use this repository.

The Problem

  1. I have my repository hosted on GitHub
  2. I have an internal Git server used for deployments
  3. I want to keep these synchronised using my normal workflow

Getting Started

Both methods I’ll describe need a “bare” version of the GitHub repository on your internal server. This worked best for me:

[code lang=bash]
cd ~/projects/repo-sync-test/
scp -r .git [email protected]:/path/to/sync.git

Here, I’m changing to my local working directory, then using scp to copy the .git folder to the internal server over ssh.

More information and examples this can be found in the online Git Book:

4.2 Git on the Server – Getting Git on a Server

Once the internal server version of the repository is ready, we can begin!

The Easy, Safe, But Manual Method:

[code lang=text]
+———+ +———-+ /——>
| GitHub | | internal | — deploy –>
+———+ +———-+ \——>
^ ^
| |
| +———+ |
\—–| ME! | —-/

This one I have used before, and is the least complex. It needs the least setup, but doesn’t sync the two repositories automatically. Essentially we are going to add a second Git Remote to the local copy, and push to both servers in our workflow:

In your own local copy of the repository, checked out from GitHub, add a new remote a bit like this:

[code lang=bash]
git remote add internal [email protected]:/path/to/sync.git

This guide on has a bit more information about adding Remotes.

You can change the remote name of “internal” to whatever you want. You could also rename the remote which points to GitHub (“origin”) to something else, so it’s clearer where it is pushing to:

[code lang=bash]
git remote rename origin github

With your remotes ready, to keep the servers in sync you push to both of them, one after the other:

[code lang=bash]
git push github master
git push internal master

  • Pros: Really simple
  • Cons: It’s a little more typing when pushing changes

The Automated Way:

[code lang=text]
+———+ +———-+ /——>
| GitHub | ======> | internal | — deploy –>
+———+ +———-+ \——>
| +———+
L————- | ME! |

The previous method is simple and reliable, but it doesn’t really scale that well. Wouldn’t it be nice if the internal server did the extra work?

The main thing to be aware of with this method is that you wouldn’t be able to push directly to your internal server – if you did, then the changes would be overwritten by the process I’ll describe.


One problem I had in setting this up initially, is the local repositories on my PC are cloned from GitHub over SSH, which would require a lot more setup to allow the server to fetch from GitHub without any interaction. So what I did was remove the existing remote, and add a new one pointing to the https link:

[code lang=bash]
(on the internal server)
cd /path/to/repository.git
git remote rm origin
git remote add origin
git fetch origin

You might not have to do this, but I did, so best to mention it!

At this point, you can test everything is working OK. Create or modify a file in your local copy, and push it to GitHub. On your internal server, do a git fetch origin to sync the change down to the server repository. Now, if you were to try and do a normal git merge origin at this point, it would fail, because we’re in a “bare” repository. If we were to clone the server repository to another machine, it would reflect the previous commit.

Instead, to see our changes reflected, we can use git reset (I’ve included example output messages):

[code lang=bash]
git reset refs/remotes/origin/master

Unstaged changes after reset:
M testfile1.txt
M testfile2.txt
M testfile3.txt

Now if we were to clone the internal server’s repository, it would be fully up to date with the repository on GitHub. Great! But so far it’s still a manual process, so lets add a cron task to stop the need for human intervention.

In my case, adding a new file to /etc/cron.d/, with the contents below was enough:

[code lang=bash]
*/30 * * * * user cd /path/to/sync.git && git fetch origin && git reset refs/remotes/origin/master > /dev/null

What this does is tell cron that every 30 minutes it should run our command as the user user. Stepping through the command, we’re asking to:

  1. cd to our repository
  2. git fetch from GitHub
  3. git reset like we did in our test above, while sending the messages to /dev/null

That should be all we need to do! Our internal server will keep itself up-to-date with our GitHub repository automatically.

  • Pros: It’s automated; only need to push changes to one server.
  • Cons: If someone mistakenly pushes to the internal server, their changes will be overwritten


Learning New Stuff: Vim

It started as a throwaway comment:

But like all interesting ideas, it took root in my brain, and I started to wonder “no, really, what if…”

This morning I woke up deciding to finally learn Vim. It’s been on my “to-do” list for quite a while – years, in fact – my old boss used to be a bit of a Vim guru, who could edit file on the Linux boxes 10x faster than me using GEdit, which I used to marvel at as a junior.

Beyond the “code from my iPad” idea above, I have a couple more reasons for learning Vim:

  • Vim is near universal. It runs on pretty much any platform, and it’s standard on virtually any Linux distro
  • Since switching back to Linux, I spend half my time in the CLI anyway – Vim cuts down the need to switch away to something like Sublime
  • Having a CLI-based workflow reduces the amount of “stuff” I need to set-up on a computer to write some code

But coming back to the iPad, I’m quite keen to get this working because I’ll be moving house soon, and it’s more than likely I won’t have space for my current PC set-up, and a new laptop is out of my budget for now. So if this works it will let me keep tinkering on projects even without a “proper” computer. I’ve been in this position before, so I know the limitations. This time, however, I’m using an iPad Mini and a Logitech Bluetooth keyboard case.

My Set-up/Learning Resources

To get myself up and running, I’ve installed and configured Vim on my PC, and will install GVim on my work laptop. I’m going to try and use it in place of Sublime Text Editor as much as I can, but give no promises when it comes to anything work-related, as that generally needs done fast.

I’ve fired up a Ubuntu box on Joyent (starting with the first 5-minutes post I linked to last week), and replicated my PC’s Vim configuration, so I can start working from the iPad straight away. I installed Git, and authorised the box with my Github account, so I can push and pull to my heart’s content.

As I’m just starting out, the configuration I’ve gone for is very basic and minimal, but giving me room to grow as I get more experienced. Essentially, I’ve installed Pathogen, and applied their recommended beginner defaults.

For learning resources, I’ve found there’s a heck of a lot out there for Vim. So much so, I’ve chosen to limit myself for now so I don’t get overwhelmed.

  • The Pragmatic Programmers books are somewhere I always check when I want to learn something new. I find their books to be practical, well written, and informative. True to form, Practical Vim (from what I’ve read so far) is an excellent introduction to “real world” Vim.
  • Vim Adventures is a neat, interactive learning tool, which turns learning Vim into an 16-bit adventure game.
  • VimCasts has 50 free screencasts, dealing – unsurprisingly – with learning Vim

Feel free to suggest some more good resources though – as I get more experience I’ll need to branch out into other areas!