Static IP addresses in a GCP Managed Instance Group?

Can you assign static IPs for Managed Instance Group members in Google Cloud Platform?

Well, maybe..

• Is auto-scaling turned off in your Managed Instance Group?
• Do you have a way of identifying each instance uniquely from an instance startup script?

You may be using Managed Instance Groups with or without load balancers for a variety of use cases:

1. Creating application clusters that communicate and work in a primary + multiple secondaries method with automatic fail over.
2. Spreading identical resources across multiple zones.
3. Stateless virtual machine groups (customization done via startup scripts) that can be rebuilt on demand for testing or development
4. Virtual machine sets that can be updated or upgraded with minimal to no down time for services.
5. Auto-scaling groups that allow you to increase or decrease the number of machines serving a set of identical services behind a load balancer.

In some of these use cases, you may encounter applications that require you to assign a static IP address to the nodes either for communication between the nodes, DNS resolution or other legacy application needs.

In a scenario where you might set up auto-healing for an instance that was corrupted in some way, you might need the instance to come back up with the same IP address it had before.

If you use the relatively new “stateful” Managed Instance Groups, your data disk might have references to IP addresses of that machine or other machines in the group — bad application design decisions are not that uncommon.

There are still legacy applications out there that depend on static addresses for their cluster members, and they need to run in the cloud and be able to utilize the advanced protections like auto-healing. I came across one such application in a client engagement that caused us to get creative with the instance group configuration.

So, how does it work?

This utilizes a Google Cloud Compute feature known as ‘alias IP ranges’ along with reserved static internal IP addresses. Screenshot below shows this option when creating a new instance:

Alias IP ranges for Compute Instance Networking

There needs to be a way to uniquely identify each instance from it’s startup scripts for this process to work. In our scenario we had a managed instance group with 3 nodes, one in each zone of a region, predefined during the instance group creation. So I chose to use the zone name to uniquely identify the instance.

Sample configuration details below:

• An instance group is created with 3 nodes in us-central1 region with predefined zone list of us-central1-a, us-central1-b and us-central1-c.
• Auto-scaling is disabled on this group with 3 nodes as the minimum and maximum.
• Auto-healing is enabled with a health check utilizing a specific TCP port.
• The VPC subnet that is used by the machine has 3 reserved static IPs named with the zone and the application name to identify them:myapp-us-central1-a-ip, myapp-us-central1-b-ip and myapp-us-central1-c-ip.

The instance template contains a startup script that would identify the zone that the machine was deployed in, using the instance metadata. It will then use that to retrieve the static IP corresponding to that zone, and runs a gcloud command to assign this IP address as an alias IP to the specific instance.

Sample code and explanation below for a Linux instance:

• Get the instance Zone information from the metadata server and save it to an environment variable.

export INSTZONE=$(curl http://metadata.google.internal/computeMetadata/v1/instance/zone -H "Metadata-Flavor: Google")

• The zone name is returned as projects/<project>/zones/us-central1-c. We are only interested in the final part. So, do a little bash trick to get the last part of the zone name returned —basename returns the leaf node in a path and is perfect for what we need here — the returned value looks like a file system path and we need the leaf node of it.

export ZONE=`basename $INSTZONE`

• Use the gcloud command to get the static address named myapp-<zone>-ip. Format the output using the--format option to obtain the IP address.

export INSTIP=`gcloud compute addresses list --filter=name:myapp-$ZONE-ip --format="value(address:label=address)"` 

• Retrieve the instance name with another call to the metadata server.

export INSTNAME=$(curl http://metadata.google.internal/computeMetadata/v1/instance/name -H "Metadata-Flavor: Google")

• Assign the IP address as an alias address (with a CIDR suffix of 32 indicating a single IP) to the network interface on the named instance.

gcloud compute instances network-interfaces update $INSTNAME --zone=$ZONE --aliases="$INSTIP/32"

• Add the IP address as a secondary address to your network interface using the ip addr command.

ip addr add $INSTIP/32 dev eth0

Note: The gcloud command for assigning the alias IP requires that the service account assigned to the instance to have the compute-rw permission.

Including this script in the startup script metadata field on the instance template will automatically assign the static IP to the 3 instances when they are created, based on their zone. You may have to get creative with how you can uniquely identify a machine in order to assign it’s static address. You will also need to create the required static addresses with matching names, before the instance group is created.

Following is a set of sample Terraform code snippets showing how an instance group can be created using the process above with static IP assignments.

main.tf

resource "google_compute_instance_template" "mig-template" {
name         = "mig-template"
  machine_type = var.mig-machine-type
  project      = var.mig-project
  region       = var.mig-cluster-region
disk {
    boot         = true
    device_name  = "mig-template"
    disk_size_gb = 20
    disk_type    = "pd-standard"
    source_image = var.mig-os-image
  }
network_interface {
    subnetwork         = var.mig-subnetwork
    subnetwork_project = var.mig-subnetwork-project
  }
metadata_startup_script = var.startup-script
service_account {
    scopes = [
      "service-control",
      "compute-rw",
      "storage-rw",
      "monitoring-write",
      "logging-write",
      "service-management"
    ]
  }
}
resource "google_compute_region_instance_group_manager" "mig-cluster-ig" {
name                      = "mig-sample"
  project                   = var.mig-project
  base_instance_name        = "myapp"
  region                    = var.mig-cluster-region
  distribution_policy_zones = var.mig-cluster-zones
version {
    instance_template = google_compute_instance_template.mig-template.id
  }
target_size = 3
}

variables.tf

variable "mig-project" {}
variable "mig-machine-type" {}
variable "mig-os-image" {}
variable "mig-cluster-region" {}
variable "mig-subnetwork" {}
variable "mig-subnetwork-project" {}
variable "startup-script" {}
variable "mig-cluster-zones" {
  type = list(string)
}

terraform.tfvars

mig-project            = "project-name"
mig-machine-type       = "custom-1-1024"
mig-subnetwork         = "subnet-name"
mig-subnetwork-project = "sharedvpc-host-project"
mig-os-image           = "rhel-7-v20200811"
mig-cluster-region     = "us-central1"
mig-cluster-zones      = ["us-central1-a", "us-central1-b", "us-central1-c", ]
startup-script = <<EOF
export INSTZONE=$(curl http://metadata.google.internal/computeMetadata/v1/instance/zone -H "Metadata-Flavor: Google")
export ZONE=`basename $INSTZONE`
export INSTIP=`gcloud compute addresses list --filter=name:mq01-$ZONE-ip --format="value(address:label=address)"`
export INSTNAME=$(curl http://metadata.google.internal/computeMetadata/v1/instance/name -H "Metadata-Flavor: Google")
gcloud compute instances network-interfaces update $INSTNAME --zone=$ZONE --aliases="$INSTIP/32"
ip addr add $INSTIP/32 dev eth0
EOF

Fill out the proper values for your environment, and test it out with Terraform. Please leave comments on any other methods that may be useful to uniquely identify the instances in your own Managed Instance Groups.

Note: In the event that auto-healing causes an instance to be rebuilt, the alias IP may already be assigned to the machine as part of the instance metadata. That could cause the startup script to throw an error for that line of code. The IP address assignment to your network interface using ip addr add would still work as the alias IP remains the same.