From Pet to Cattle – Running Sonar on Kubernetes
by Ádám Sándor
Recently we ran into a bit of trouble with our pet CI server. It was being used as a Jenkins server and a Postgres database and Sonar was also running on it but it wasn’t evident as it was just a manually ran docker compose setup on the machine. Sure enough someone turned off the server not knowing that Sonar is also running there. As Sonar is part of our minimesos build our builds stopped working.
I thought about the best way to move Sonar and Postgres off the Jenkins server. They were already “containerized” – started as a Docker Compose setup so I though let’s keep that but run the containers on a proper cluster which will keep them alive and takes the management of the underlying VMs out of our hands. For this I looked to Google Container Engine which is a managed Kubernetes setup. I can use the gcloud client for Google Cloud to create a cluster and do a standard Kubernetes deployment to run the application there.
Stuff I want to create
- One Sonar container. I used
sonarqube:5.3. - One Postgres container. I used
postgres:9.5.3. - Kubernetes
deploymentfiles for both services. - Kubernetes
servicefiles for both services. - Persistent storage for the database.
- A Kubernetes cluster. Easiest way to get that one is to use Google Container Engine.
- Secret for storing the database password.
- Certificate for the DNS name sonar.infra.container-solutions.com.
- Loadbalancer and DNS in GCE.
A bit about pets and cattle
I refer to the analogy often used in #DevOps circles to distinguish servers that are created by SSH-ing to the machine from those that are created by a fully automated process. This former is called is a pet – it has a name (in our case jenkins-ci-4) and it doesn’t have a reproducible way to create it. Killing it is considered extreme cruelty. As opposed to pets, cattle can be killed with impunity – because as we all know cows just get resurrected if you shoot them in the head. Let’s say these analogies are not perfect but I hope you get what I mean. What I wanted to achieve with the new Sonar deployment was that the whole infrastructure would be automated so we no longer have to worry about accidental shutdowns or unclear ways of configuring stuff.
How it’s done
Services
I started out by converting the original docker-compose.yml file to two Kubernetes deployments. A deployment in Kubernetes describes how a Pod (in this case Pod=DockerContainer) should be created, including it’s:
- Image name
- Exposed ports
- Environment variables
- Volumes
Here is what I ended up with for the Sonar container:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sonar spec: replicas: 1 template: metadata: name: sonar labels: name: sonar spec: containers: - image: sonarqube:5.3 args: - -Dsonar.web.context=/sonar name: sonar env: - name: SONARQUBE_JDBC_PASSWORD valueFrom: secretKeyRef: name: postgres-pwd key: password - name: SONARQUBE_JDBC_URL value: jdbc:postgresql://sonar-postgres:5432/sonar ports: - containerPort: 9000 name: sonar |
This defines that we want a single replica of the sonarqube:5.3 image running connecting to the Postgres database. I didn’t feel like running several instances because Kubernetes will always restart this one if it fails and that’s enough for such a rarely used internal service. What’s really nice here is that I can reference the Postgres server by an internal DNS name sonar-postgres. Another nice thing is that I can reference a secret defined in Kubernetes for the database password, so I get secret management out of the box.
Additionally I need to define a service for Sonar. Adding Type: LoadBalancer to the service definition tells Kubernetes to use the underlying platform (GCE in this case) to create a load balancer to expose our service over the internet.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
apiVersion: v1 kind: Service metadata: labels: name: sonar name: sonar spec: ports: - port: 80 targetPort: 9000 name: sonarport selector: name: sonar type: LoadBalancer |
The Postgres deployment looks like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sonar-postgres spec: replicas: 1 template: metadata: name: sonar-postgres labels: name: sonar-postgres spec: containers: - image: postgres:9.5.3 name: sonar-postgres env: - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: postgres-pwd key: password - name: POSTGRES_USER value: sonar ports: - containerPort: 5432 name: postgresport volumeMounts: # This name must match the volumes.name below. - name: data-disk mountPath: /var/lib/postgresql/data volumes: - name: data-disk gcePersistentDisk: # This disk must already exist. pdName: minimesos-sonar-postgres-disk fsType: ext4 |
You can see that I’m referencing the same password as in the Sonar deployment. Additionally I’m attaching a GCE persistent disk to keep our DB data safe. I’ll get back to creating this disk a bit later.
The service definition for Postgres doesn’t contain a LoadBalancer entry as it doesn’t need to be publicly accessible. It just makes port 5432 accessible inside the cluster under the DNS name sonar-postgres.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
apiVersion: v1 kind: Service metadata: labels: name: sonar-postgres name: sonar-postgres spec: ports: - port: 5432 selector: name: sonar-postgres |
Persistent storage
Getting persistent storage is a point where we have to get out of Kubernetes-world and use GCE. Kubernetes doesn’t support creating persistent volumes but it does play nice with ones created in GCE (see the gcePersistentDisk entry above). Creating the volume in GCE is very simple:
|
1 2 |
gcloud compute disks create --size 200GB minimesos-sonar-postgres-disk |
I also had to migrate the existing data. For that I mounted the new volume to the old pet machine:
gcloud compute instances attach-disk jenkins-ci-4 --disk minimesos-sonar-postgres-disk --device-name postgresdisk
mount and format it:
/usr/share/google/safe_format_and_mount /dev/disk/by-id/google-postgresdisk /postgresdisk
…copy the files…
then detach the volume:
gcloud compute instances detach-disk jenkins-ci-4 --disk minimesos-sonar-postgres-disk
Creating the Cluster
Creating a new Kubernetes cluster is of course super-easy, as that’s the point of Google Container Engine. You use gcloud but the second parameter is container instead of compute to dive into container-land. ok
gcloud container clusters create minimesos-sonar --machine-type n1-standard-2 --zone europe-west1-d
You can of course specify a lot more parameters here.
Secrets
Creating secrets is a really cool Kubernetes feature that takes care of an annoying problem for us – distributing the database password to two separate containers. There are of course more sophisticated solutions out there like Hashicorp’s Vault but for this simple setup Kubernetes’s secret support is great. We first create a secret with this command:
kubectl create secret generic postgres-pwd --from-file=./password
The ./password is a file on the disk that contains nothing else then the password itself. I then use the password by injecting it as and environment variable into the containers. You can see that in the deployment definition files above.
This is the first time we used the kubectl command. kubectl is the client to control your Kuberntes cluster. gcloud is the client for the Google Cloud service while kubectl controls a single Kubernetes cluster by communicating with the API server of that cluster. kubectl is independent of Google Cloud and is used to control any Kubernetes cluster including ones that weren’t created using gcloud. It is important to point your kubectl to the newly created cluster by running gcloud container clusters get-credentials minimesos-sonar-cluster. This will create and entry in your ~/.kube/config file and possibly set the new cluster as the current one. You can verify this using kubectl cluster-info. If you don’t see the IPs of your cluster listed then you’ll have to use kubectl config set-cluster minimesos-sonar-cluster to set it as the current one.
Fire engines.. blast off!
We can now launch both services by passing the 4 yaml files to kubectl apply. This command will compare the current state of the cluster with the desired state described in the config files and make any necessary changes. In this case it means Kubernetes needs to start everything up.
kubectl apply -f sonar-postgres-deployment.yaml -f sonar-deployment.yaml -f sonar-postgres-service.yaml -f sonar-service.yaml
Run kubectl get deployments,services,pods to view the state of all our freshly started Kubernetes things:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
$ kubectl get deployments,services,pods NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sonar 1 1 1 1 1m sonar-postgres 1 1 1 1 1m NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.139.240.1 <none> 443/TCP 9d sonar 10.139.246.151 104.155.45.237 80/TCP 1m sonar-postgres 10.139.241.166 <none> 5432/TCP 1m NAME READY STATUS RESTARTS AGE sonar-117517980-fgkqx 1/1 Running 0 1m sonar-postgres-176201253-2hu79 1/1 Running 0 1m |
This shows that all the deployments, services and pods are working. It’s a really cool feature of kubectl that it allows fetching data about multiple types of resources in one command. We can see that the sonar service has an external IP. This is because we defined a load-balancer in the service definition. I can now navigate to http://104.155.45.237/sonar to see the Sonar web UI.
Let’s do a simple exercise to see how cattle-ish our new deployment really is: kubectl delete -f sonar-postgres-deployment.yaml -f sonar-deployment.yaml -f sonar-postgres-service.yaml -f sonar-service.yaml Just swapping kubectl apply for kubectl delete will tear down both services as if they never existed and of course we can start them again without any hassle. It’s also easy to make changes to the configuration and do another kubectl apply to apply the changes.
To be continued
This post is running quite long and there is still work left to do. We need to expose the service over DNS, a temporary IP that changes with ever restart of the Pod won’t do. We also want https access to the service for which we’ll need to get certificate and set up automatic renewal for it. We’ll also need a proxy in front of the Sonar service to terminate the https connection. It will be quite some work to achieve this, so I’ll leave it for a new post.
On the topic of Pet & Cattle and PostgreSQL. I think Patroni is very interesting. Josh Berkus recently did a presentation and called it: Bots, Not Cattle. So cattle but smarter (with a state machine).
That sounds similar to what Joyent is doing with Autopilot (https://www.joyent.com/blog/dbaas-simplicity-no-lock-in). Moving towards more intelligent components is a very interesting trend. But that’s one side of the analogy, it’s still very important to emphasise why pets are a bad idea (please don’t quote this sentence out of context :D)
It is no surprise that Joyent would do the same thing, because the basic idea behind Patroni actually (at least partially) came from what Joyent was already doing. 🙂
If you look at the source code of flynn.io you’ll notice something similar: “This design is heavily based on the prior work done by Joyent on the [Manatee state machine](https://github.com/joyent/manatee-state-machine).”
If you haven’t checked out flynn.io or Patroni you really should.
Just checked and seems others have written about Mantl and ContainerPilot on this blog before. 🙂
I noticed you haven’t attached any volumes to the sonar container. How are you finding config persistence following restarts?
Honestly this never went into production because I had to figure out the domain name stuff with Letsencrypt which never happened. So yeah.. I didn’t think of configurations that aren’t saved to the database, thought Sonar keeps all moving parts there, and I would just put any other configs into the Docker image (which I obviously also didn’t do yet).
Do you know about configurations for Sonar that are not static (can be changed from the UI) but are not kept in the database? In that case we’ll need a volume for Sonar too.
How can you use the sonarqube to connect to a remote postgres server?
I’m not sure what you mean by remote. In this case the Postgres server is running on the same Kubernetes cluster but it is remote from the Sonarqube container. It might run on another Node. The jdbc:postgresql://sonar-postgres:5432/sonar URL has
sonar-postgresas hostname which is a DNS name managed by Kubernetes and will be assigned to the dynamic IP of the Pod on the cluster. Does this answer your question?You did mention that this blog post will be continued with details about DNS, https, certs and proxy. I am actually waiting for that eagerly(I havent been able to figure the puzzle on my own as Kubernetes is fairly new to me . I am looking to setup a Highly Available SonarQube app in a kubernetes cluster in a production environment. Any idea when you will be able to complete this post? Or any other blog post with something similar?
Hi Prateek, now that you poked me I might get back to that 🙂 Projects steered me away from this post but I’d love to continue it sometime soon. However I can’t give you any ETA as it’s currently not my focus point.
Is there a way to keep the configs for Quality Gates, Profiles etc as source so you have Infrastructure as Code or do you have to fiddle around in the Sonarqube GUI every time?
Hi David, that’s a good question and I don’t know the answer. My preference would also be to keep them as code and put them in the Docker container during build. However Sonar is very much geared to be managed from the UI so I wouldn’t be surprised if this was not possible.
Hi Adam, was setting up Sonarqube on K8s and did everything but for the persistent data part. Your instructions helped me a lot in setting up that part as well.
When all done, there seems to be an issue when sonarqube is trying to come up, but not able to due to connection issues with DB. Below are the logs for the same. Had tried using the password the secret way and also by providing it on the yaml files as well, but with the same result.
Not sure how to check the sonar.jdbc values after create, as the pod never comes up and not able to validate the values. Just wanted to check if you had come across anything like this. Thanks for your article.
java.lang.IllegalStateException: Can not connect to database. Please check connectivity and settings (see the properties prefixed by ‘sonar.jdbc.’).
at org.sonar.db.DefaultDatabase.checkConnection(DefaultDatabase.java:108)
Hi Satya, I can see your pod is not able to connect to the database. If you specify the username and password straight in the yaml file there is no chance those values could change before getting to Sonarqube, so I would guess the problem is elsewhere. I would try to run another Pod and run telnet from that Pod to check if the database is listening on the specified port. You can use my swissarmyknife container for this, as it has telnet and other networking tools installed. Use this command:
kubectl run -ti --image=adamsandor83/swissarmyknife --rmThis will run the container and give you control of it’s internal console. You can then run telnet inside the container and target the host and port of the database server to try if TCP connections work. That’s the best I can tell you as the range of issues here is really wide. You could also create your own Java-based container and use the same JDBC configuration to be able to see where the connection goes wrong.