Creating repositories with ease

My second holiday project this year is publishing this module. With RevDB, we have to manage dozens if not hundreds of Github repositories, in many different organizations with a variety of contributors.

Find the referenced modules at the Github repository and the Terraform registry for some insights.

There is nothing surprising in the fact that we use Terraform to do that. However, creating and managing so many repositories, forced us to get creative around them.

Generally, we have to create 3 different kind of repositories:

  • Python
  • Terraform
  • Empty (with defaults)

No two repositories are identical, but there are a lot of shared properties. For exmple

  • .gitignore file
  • editor config
  • administrators and maintainers
  • README, Changelog
  • Terraform repositories have to have at least a main.tf, variables.tf and an output.tf
  • Python repositories have the same structure with setup.py, click  and testing configured

Every time a new repository is created, these have to be added manually or even worst, copied over. Not anymore.

Using Cookiecutters with Terraform

Our very own Aleks is keep surprising me. He came up with the idea of putting together a shell script that runs, every time the github_repository resource is created. This bash script would take care of checking out the cookiecutter repository and push the first commit, using provisioner { local-exec}! If that doesn’t tell you much, don’t worry, I didn’t understand it either at first.

First of all, we have a bash script template, you can find here. It would be a way too much to include here.

The above bashscript will be used as a template for the following Terraform template file resource.

data "template_file" "init_repo" {
  template = file("${path.module}/init_repo.sh")
  vars = {
    ssh_key_path              = var.ssh_key_path
    default_branch            = var.default_branch
    organization              = var.organization
    repo_name                 = var.name
    repo_kind                 = var.repo_kind
    project_name              = var.name
    project_short_description = var.description
    states_bucket             = var.states_bucket
    email                     = var.email
    github_username           = var.github_username
    trigger_cookiecutter      = var.trigger_cookiecutter
    setup_travis              = var.setup_travis
  }
}

Alright, but how is that going to be executed, every time a repo is created? Well, that’s what the provisioner-local-exec is for.

resource "github_repository" "repo" {
  name               = var.name
  description        = var.description
  homepage_url       = var.homepage_url
  private            = var.private
  allow_rebase_merge = 
  provisioner "local-exec" {
    command = data.template_file.init_repo.rendered
  }
}

Isn’t it just brilliant?

Create a Github repository, with cookicutters, using Terraform

There is just no easier way to say this, but spell out, that we are creating github repositories with cookiecutters using Terraform. However, I tried to make the module as easy and simple to use as possible.

There is a checklist you need to get going, setup with github:

  • Github token
  • SSH key pair
  • Cookiecutter must be installed

Once these are set, the absolute minimum code, looks like this:

module "python-playground" {
  source                = "revenants-cie/revdb-repository-manager/github"
  name                  = "python-demo-playground_6"
  organization          = "revdb-test"
  owner_team_id         = github_team.committers.id
  ssh_key_path          = abspath("/Users/istvan/.ssh/id_rsa")
  admin_team_id         = github_team.admins.id
  repo_kind             = "python"
}

That’s it, as long as the provider is set, teams are created and the SSH key exists, this would just simply work. However, there are dependencies.

For the SSH key, I would recommend reading Github’s very own documentation and make sure, the path is pointing right at it. Read the docs here.

Add a provider, this is where you will need your github token.

provider "github" {
  token        = "my-very-private-token"
  organization = "revdb-test"
}

And the teams I’m using at last.

resource "github_team" "committers" {
  name        = "committers"
  description = "RevDb developers"
  privacy     = "closed"
}

resource "github_team" "admins" {
  name        = "admins"
  description = "RevDb administrators"
  privacy     = "closed"
}

resource "github_team_membership" "member_committers" {
  username = "ipodorrcie"
  team_id  = github_team.committers.id
}

resource "github_team_membership" "admins" {
  username = "ipodorrcie"
  team_id  = github_team.admins.id
  role     = "maintainer"
}

Creating multiple repositories of different kinds and types

This bit of example file, will create three different repositories. One Python, one Terraform and one empty. The empty repository will be open and not protected, while the other two will be private and branch protected. For each repo, I added some tinkering with the settings as well.

provider "github" {
  token        = "very secret token"
  organization = "revdb-test"
}

module "python-playground" {
  source                = "revenants-cie/revdb-repository-manager/github"
  name                  = "python-demo-playground"
  organization          = "revdb-test"
  owner_team_id         = github_team.committers.id
  ssh_key_path          = abspath("/Users/istvan/.ssh/id_rsa")
  admin_team_id         = github_team.admins.id
  repo_kind             = "python"
  has_branch_protection = true
}

module "terraform-playground" {
  source                = "revenants-cie/revdb-repository-manager/github"
  name                  = "terraform-demo-playground"
  organization          = "revdb-test"
  owner_team_id         = github_team.committers.id
  ssh_key_path          = abspath("/Users/istvan/.ssh/id_rsa")
  admin_team_id         = github_team.admins.id
  repo_kind             = "terraform"
  has_branch_protection = true
}

module "empty-playground" {
  source                = "revenants-cie/revdb-repository-manager/github"
  name                  = "empty-demo-playground"
  organization          = "revdb-test"
  owner_team_id         = github_team.committers.id
  ssh_key_path          = abspath("/Users/istvan/.ssh/id_rsa")
  admin_team_id         = github_team.admins.id
  repo_kind             = "empty"
  has_branch_protection = false
  private               = false
  default_branch        = "main"
  has_issues            = true
  has_downloads         = true
  email                 = "public_dev@revdb.io"
}

resource "github_team" "committers" {
  name        = "committers"
  description = "RevDb developers"
  privacy     = "closed"
}

resource "github_team" "admins" {
  name        = "admins"
  description = "RevDb administrators"
  privacy     = "closed"
}

resource "github_team_membership" "member_committers" {
  username = "ipodorrcie"
  team_id  = github_team.committers.id
}

resource "github_team_membership" "admins" {
  username = "ipodorrcie"
  team_id  = github_team.admins.id
  role     = "maintainer"
}

Conclusion

Infrastructure as code is a way of doing things and creating unified repositories is no exception. This way, we are able to have dozens of repositories and still be sure that they all have the same settings. Not mentioning how much easier it is to quickly jump onto creating a new python module. Not just that our new python modules will be created, but the testing will already be failing! With automatic branch protection and in our own case, travis configured, when we add a new module, it must come with passing tests.

I hope you find this useful even if it’s only an example to your own implementation. Feedback and contribution is highly appreciated!


0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *