Using Google Container Registry with Kubernetes

 

Alternative Text by

I recently got into orchestrating my Docker containers with Kubernetes. For one of our projects, I needed to pull docker images from the Google Container Registry (GCR).
When using the Google Kubernetes Engine with the GCR everything works out of the box, but to run the containers locally with docker, I had to install and configure docker-credential-gcr. (Go to the GitHub repository for guidance.)
However, when I tried to run a deployment on my local Kubernetes cluster I encountered some errors

Kubernetes does not use the docker client to log in and pull images which is why there are no valid GCR credentials configured.

To pull images from the GCR, you can use Kubernetes’ ImagePullSecrets concept.
Secrets can be assigned to single pods or a service account, which then adds the secret to any new pod created in its namespace. Whenever someone or something accesses the Kubernetes cluster, the API server authenticates them as a specific account type. Simply put, User accounts are for real-life humans, service accounts are for processes that run inside pods.

Whether you are using minikube, docker-for-mac edge (which comes with native Kubernetes support) or a native Kubernetes cluster, this guide applies to all Kubernetes environments.
I will show you two ways to setup Kubernetes ImagePullSecrets for GCR, but the principles are applicable to any private registry.

Push Image to GCR

Start with enabling the Container Registry API by logging into Google Cloud and navigating to Container Registry on your project

Click Enable Container Registry API

Then build a Docker image and push it to your project’s GCR

All following kubectl commands should run in a specific namespace. Namespaces are virtual clusters, running in the same physical cluster and are an excellent way to keep your project environments separate. Define a namespace localdev for which you want to set up ImagePullSecrets.

If you do not want to add --namespace=localdev to all kubectl commands, you can set this as the default namespace for your current context

Now it is time to create some GCR credentials.

Create & use GCR credentials

JSON Key

JSON Keys are valid until the key is removed from the Google Cloud.
Google uses its own service accounts to allow automated access to its platform services.
Log into your Google Cloud Console, navigate to APIs and Services -> Credentials and create a new Service account key.

In the drop-down menu for service accounts, pick New service account and give it a name. Since you are only using this account for pulling images from the registry, you can give it the Project -> Viewer role to start.
From the key types choose JSON and click Create.

The JSON key file is automatically downloaded and will kind of look like this:

Now you need to tell Kubernetes to use the JSON key file when pulling from GCR by creating a secret named gcr-json-key.

Replace ~/json-key-file.json with the path to your json key file. The docker-server value has to match the hostname of your registry exactly. So outside of the EU, you might want to use https://gcr.io instead.
Also note that docker-registry is a necessary keyword whereas gcr-json-key is a freely customizable name.

kubectl should answer with

If you get

instead, you might want to think of another name or delete the existing secret first

Alternatively, you can add this little hack to the above command, which updates the old secret

Finally, you have to add the secret to your default service account as ImagePullSecrets, so it will actually be used, when Kubernetes spins up a new pod with this service account.

You can check the setup of the default service account with

by adding the -o flag, you can format the output to JSON or YAML

Now you should be able to pull from GCR without any problems

Take a look at your pods to verify that pulling was successful.

(Note: If you still have an old version of this deployment running, kubectl does not notice any changes and outputs

You need to delete the old deployment and apply again.)

Access token

For keys that are short-lived for one-time and instant usage, you can create a GCR access token for your ImagePullSecrets. For this, you need to have gcloud installed and correctly configured.

Creating a secret is virtually the same as with JSON keys

except using oauth2accesstoken instead of _json_key as username and the output of gcloud auth print-access-token as password.

Don’t forget to patch your service account with the new secret.

A look at the service account yaml should tell you if you configured everything correctly

That is it! You should now be able to pull from the GCR.

ImagePullSecrets for single pods

In case you want to use the secret for one specific pod only, all you need to do is add the secret to your pod.yaml instead of patching the service account with your secret.

Looking for a new challenge in a learning organisation? We’re hiring!

C

10 Comments

  1. The JSON option worked perfectly! I don’t understand why this isn’t clearly documented on the GCR documentation, it was certainly doing my head in until I found this post. Worked within seconds after renaming those variables and downloading their JSON file.

  2. Excellent explanation!

    I’m doing an online course covering K8s, and it assumes we’re using the standard Docker Registry, so this was invaluable!

    Thanks!

  3. I get : error: unable to parse “‘{imagePullSecrets:[{name:”: yaml: found unexpected end of stream
    when i run
    $ kubectl patch serviceaccount default \
    -p ‘{“imagePullSecrets”: [{“name”: “gcr-json-key”}]}’

    I see the yaml for serviceaccount default has no imagePullSecrets object

    • The error message indicates, that there’s an invalid yaml syntax. How does the yaml output of the serviceaccount manifest look like?

      kubectl get serviceaccount default -o yaml

    • From what I understand, it is a “keyword” that specifies that this user is not an actual user, but using a json key as password.

  4. Thanks so much mate! Spent so much time trying to understand why Docker was able to pull images and not kubelet!

Leave a Reply

Your email address will not be published. Required fields are marked *