Jekyll2021-05-08T19:48:48+00:00http://zoltanaltfatter.com/feed.xmlZoltan AltfatterSoftware EngineerHello World with Knative Serving2021-03-14T00:00:00+00:002021-03-14T00:00:00+00:00http://zoltanaltfatter.com/2021/03/14/hello-world-with-knative-serving<p><a href="https://knative.dev/">Knative</a> goal is to make developers more productive by providing higher level abstractions (<a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CRDs</a>) on top of Kubernetes.
It solves common problems like stand up a scalable, secure, stateless service in seconds, connecting disparate systems together.
<code class="language-plaintext highlighter-rouge">Knative</code> also brings serverless deployment models to Kubernetes.</p>
<p>There are two major subprojects of Knative: <code class="language-plaintext highlighter-rouge">Serving</code> and <code class="language-plaintext highlighter-rouge">Eventing</code>. <code class="language-plaintext highlighter-rouge">Serving</code> is responsible
for deploying containers on Kubernetes, taking care of the details of networking, autoscaling (even to zero), upgrading, routing.
Eventing is responsible for connecting disparate systems. In early blog posts about Knative it was mentioned <code class="language-plaintext highlighter-rouge">Build</code> as a third subproject. <code class="language-plaintext highlighter-rouge">Build</code> became and independent project which is known under the name <a href="https://tekton.dev/">Tekton</a>.</p>
<p>In this blog post we will look into <code class="language-plaintext highlighter-rouge">Knative Serving</code>. The three main components of <code class="language-plaintext highlighter-rouge">Knative Serving</code> are good represented on this diagram
(taken from <a href="https://twitter.com/jacques_chester">Jacques Chester</a>’s excellent book <a href="https://livebook.manning.com/book/knative-in-action/chapter-1/v-6/184">Knative in Action</a>):</p>
<p><img src="/images/2021-03-14/Knative.png" alt="Knative" /></p>
<p>I like also the Knative definition from <a href="https://twitter.com/piotr_minkowski">Piotr Mińkowski</a> :</p>
<p><img src="/images/2021-03-14/Knative-CloudFoundry.png" alt="CloudRun" /></p>
<h3 id="local-setup">Local setup</h3>
<p>In this example we use the single-node Kubernetes cluster provided with <a href="https://www.docker.com/products/docker-desktop">Docker Desktop</a>.
We also configured <code class="language-plaintext highlighter-rouge">Docker Desktop</code> with at least 4 CPUs and 8 GB memory to make sure everything is running smoothly.</p>
<p>We need <code class="language-plaintext highlighter-rouge">kubectl</code> which can be easily installed following <a href="https://kubernetes.io/docs/tasks/tools/">https://kubernetes.io/docs/tasks/tools/</a>
and we need also the <code class="language-plaintext highlighter-rouge">kn</code>, the Knative CLI, which can be easily installed following <a href="https://knative.dev/v0.19-docs/install/install-kn/">https://knative.dev/docs/install/install-kn/</a></p>
<p>Next we can start with Knative installation following <a href="https://knative.dev/docs/install/any-kubernetes-cluster/">https://knative.dev/docs/install/any-kubernetes-cluster/</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">--filename</span> https://github.com/knative/serving/releases/download/v0.21.0/serving-crds.yaml
<span class="nv">$ </span>kubectl apply <span class="nt">--filename</span> https://github.com/knative/serving/releases/download/v0.21.0/serving-core.yaml
</code></pre></div></div>
<p>For networking we use Istio:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">--filename</span> https://github.com/knative/net-istio/releases/download/v0.21.0/istio.yaml
<span class="nv">$ </span>kubectl apply <span class="nt">--filename</span> https://github.com/knative/net-istio/releases/download/v0.21.0/net-istio.yaml
</code></pre></div></div>
<p>For DNS magic we configure Knative Serving to use <a href="http://xip.io/">xip.io</a> as the default DNS suffix.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">--filename</span> https://github.com/knative/serving/releases/download/v0.21.0/serving-default-domain.yaml
</code></pre></div></div>
<p>This is a service that reflects back IP addresses that we submitted as domain names. For example</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nslookup 10.1.2.3.xip.io
Server: 192.168.0.254
Address: 192.168.0.254#53
Non-authoritative answer:
Name: 10.1.2.3.xip.io
Address: 10.1.2.3
</code></pre></div></div>
<p>We can use this to send traffic to endpoints for which we haven’t configured a domain name.</p>
<p>After successful installation we have something like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods <span class="nt">--all-namespaces</span>
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-system istio-ingressgateway-7f6b78d5b7-2fzcg 1/1 Running 0 10m
istio-system istiod-7fcb569c74-8bgs8 1/1 Running 0 10m
istio-system istiod-7fcb569c74-mkhcm 1/1 Running 0 9m38s
istio-system istiod-7fcb569c74-t9mb9 1/1 Running 0 9m38s
knative-serving activator-86956bbd6f-8vzz8 1/1 Running 0 10m
knative-serving autoscaler-54cbd576f6-rnvpd 1/1 Running 0 10m
knative-serving controller-79c9cccd6f-648nn 1/1 Running 0 10m
knative-serving default-domain-rfrfd 0/1 Completed 0 9m58s
knative-serving istio-webhook-56748b47-wlp7g 1/1 Running 0 10m
knative-serving networking-istio-5db557d5c4-xmj86 1/1 Running 0 10m
knative-serving webhook-5fd484cf4-qgcll 1/1 Running 0 10m
kube-system coredns-f9fd979d6-2vd95 1/1 Running 3 4d16h
kube-system coredns-f9fd979d6-jjbsm 1/1 Running 3 4d16h
kube-system etcd-docker-desktop 1/1 Running 3 4d16h
kube-system kube-apiserver-docker-desktop 1/1 Running 3 4d16h
kube-system kube-controller-manager-docker-desktop 1/1 Running 3 4d16h
kube-system kube-proxy-cpccf 1/1 Running 3 4d16h
kube-system kube-scheduler-docker-desktop 1/1 Running 3 4d16h
kube-system storage-provisioner 1/1 Running 5 4d16h
kube-system vpnkit-controller 1/1 Running 3 4d16h
</code></pre></div></div>
<h3 id="example-application">Example application</h3>
<p>We use a <a href="https://github.com/altfatterz/cloud-run-demos/tree/master/hello-world-knative">simple</a> Spring Boot application using the <a href="https://spring.io/blog/2021/03/11/announcing-spring-native-beta">recently released</a> <code class="language-plaintext highlighter-rouge">Spring Native Beta</code>.
After building the image (note it will take longer, around 5 minutes)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mvn spring-boot:build-image
</code></pre></div></div>
<p>we can start our application using</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">--rm</span> <span class="nt">-p</span> 8080:8080 altfatterz/hello-world-knative:0.0.1-SNAPSHOT
...
2021-03-15 20:05:47.331 INFO 1 <span class="nt">---</span> <span class="o">[</span> main] c.example.HelloWorldKnativeApplication : Started HelloWorldKnativeApplication <span class="k">in </span>0.054 seconds <span class="o">(</span>JVM running <span class="k">for </span>0.057<span class="o">)</span>
</code></pre></div></div>
<p>Note, the application starts up super fast only 0.054 seconds.</p>
<p>Next, we need to push our image into a Docker Registry. We use <a href="https://hub.docker.com/">Dockerhub</a> in this example.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker login
<span class="nv">$ </span>docker push altfatterz/hello-world-knative:0.0.1-SNAPSHOT
</code></pre></div></div>
<h3 id="knative-serving">Knative Serving</h3>
<p>We create a Knative Service using:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kn service create hello-world-knative <span class="nt">--image</span><span class="o">=</span>docker.io/altfatterz/hello-world-knative:0.0.1-SNAPSHOT <span class="nt">--env</span> <span class="nv">TARGET</span><span class="o">=</span><span class="s2">"Knative"</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Creating service <span class="s1">'hello-world-knative'</span> <span class="k">in </span>namespace <span class="s1">'default'</span>:
0.049s The Route is still working to reflect the latest desired specification.
0.085s ...
0.094s Configuration <span class="s2">"hello-world-knative"</span> is waiting <span class="k">for </span>a Revision to become ready.
15.504s ...
15.561s Ingress has not yet been reconciled.
15.645s Waiting <span class="k">for </span>load balancer to be ready
15.835s Ready to serve.
Service <span class="s1">'hello-world-knative'</span> created to latest revision <span class="s1">'hello-world-knative-00001'</span> is available at URL:
http://hello-world-knative.default.127.0.0.1.xip.io
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">kn</code> creates a <code class="language-plaintext highlighter-rouge">Knative Service</code>, which causes the creation of a <code class="language-plaintext highlighter-rouge">Route</code>, a <code class="language-plaintext highlighter-rouge">Revision</code> and a <code class="language-plaintext highlighter-rouge">Configuration</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kn service list
NAME URL LATEST AGE CONDITIONS READY REASON
hello-world-knative http://hello-world-knative.default.127.0.0.1.xip.io hello-world-knative-00001 45m 3 OK / 3 True
<span class="nv">$ </span>kn routes list
NAME URL READY
hello-world-knative http://hello-world-knative.default.127.0.0.1.xip.io True
<span class="nv">$ </span>kn revisions list
NAME SERVICE TRAFFIC TAGS GENERATION AGE CONDITIONS READY REASON
hello-world-knative-00001 hello-world-knative 100% 1 2m19s 3 OK / 4 True
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">3 OK / 4 </code> value in the the <code class="language-plaintext highlighter-rouge">Route</code> conditions column could be more understood if we ask for the details of the revision</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kn revision hello-world-knative-00001
Name: hello-world-knative-00001
Namespace: default
Age: 51m
Image: docker.io/altfatterz/hello-world-knative:0.0.1-SNAPSHOT <span class="o">(</span>pinned to 913d61<span class="o">)</span>
Env: <span class="nv">TARGET</span><span class="o">=</span>Knative
Service: hello-world-knative
Conditions:
OK TYPE AGE REASON
++ Ready 51m
++ ContainerHealthy 51m
++ ResourcesAvailable 51m
I Active 3m NoTraffic
</code></pre></div></div>
<p>In the <code class="language-plaintext highlighter-rouge">Conditions</code> the <code class="language-plaintext highlighter-rouge">OK</code> column <code class="language-plaintext highlighter-rouge">++</code> means everything is ok, <code class="language-plaintext highlighter-rouge">!!</code> means that something is bad, <code class="language-plaintext highlighter-rouge">??</code> when Knative has no clue what is happening.
The <code class="language-plaintext highlighter-rouge">Active</code> condition is very important, it tells us if an instance of the <code class="language-plaintext highlighter-rouge">Revision</code> is running. In this case not, indeed if we use:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods
No resources found <span class="k">in </span>default namespace.
</code></pre></div></div>
<p>As soon as, we send an HTTP request via <code class="language-plaintext highlighter-rouge">curl http://hello-world-knative.default.127.0.0.1.xip.io</code> we will see a pod is started</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-world-knative-00001-deployment-548f9f94c-zm4cp 2/2 Running 0 18s
</code></pre></div></div>
<p>and also the <code class="language-plaintext highlighter-rouge">Active</code> conditions status is changed</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Conditions:
OK TYPE AGE REASON
++ Ready 58m
++ ContainerHealthy 58m
++ ResourcesAvailable 58m
++ Active 23s
</code></pre></div></div>
<p>By default, the <code class="language-plaintext highlighter-rouge">Autoscaler</code> makes a decision based on the past 60 seconds, so if there is no traffic the pod is terminated.</p>
<p>The <code class="language-plaintext highlighter-rouge">Configuration</code> object is not exposed by the <code class="language-plaintext highlighter-rouge">kn</code> CLI, we can access it using <code class="language-plaintext highlighter-rouge">kubectl</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get config
NAME LATESTCREATED LATESTREADY READY REASON
hello-world-knative hello-world-knative-00001 hello-world-knative-00001 True
<span class="nv">$ </span>kubectl describe config hello-world-knative
Name: hello-world-knative
Namespace: default
Labels: serving.knative.dev/service<span class="o">=</span>hello-world-knative
serving.knative.dev/serviceUID<span class="o">=</span>87232544-1140-464d-a1ef-421cf38a3916
Annotations: serving.knative.dev/creator: docker-for-desktop
serving.knative.dev/lastModifier: docker-for-desktop
serving.knative.dev/routes: hello-world-knative
API Version: serving.knative.dev/v1
Kind: Configuration
Metadata:
Creation Timestamp: 2021-03-14T20:02:27Z
Generation: 1
Managed Fields:
API Version: serving.knative.dev/v1
Fields Type: FieldsV1
Manager: controller
Operation: Update
Time: 2021-03-14T20:02:37Z
Spec:
Template:
Spec:
Containers:
Env:
Name: TARGET
Value: Knative
Image: docker.io/altfatterz/hello-world-knative:0.0.1-SNAPSHOT
Status:
Conditions:
Last Transition Time: 2021-03-14T20:02:37Z
Status: True
Type: Ready
Latest Created Revision Name: hello-world-knative-00001
Latest Ready Revision Name: hello-world-knative-00001
Observed Generation: 1
Events:
Type Reason Age From Message
<span class="nt">----</span> <span class="nt">------</span> <span class="nt">----</span> <span class="nt">----</span> <span class="nt">-------</span>
Normal Created 17m configuration-controller Created Revision <span class="s2">"hello-world-knative-00001"</span>
Normal ConfigurationReady 17m configuration-controller Configuration becomes ready
Normal LatestReadyUpdate 17m configuration-controller LatestReadyRevisionName updated to <span class="s2">"hello-world-knative-00001"</span>
</code></pre></div></div>
<p>Let’s update the service by changing the value of the <code class="language-plaintext highlighter-rouge">TARGET</code> environment variable.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kn service update hello-world-knative <span class="nt">--image</span><span class="o">=</span>docker.io/altfatterz/hello-world-knative:0.0.1-SNAPSHOT <span class="nt">--env</span> <span class="nv">TARGET</span><span class="o">=</span><span class="s2">"Knative community"</span>
</code></pre></div></div>
<p>Under the hood, the <code class="language-plaintext highlighter-rouge">kn</code> CLI updates the YAML document and sends it to the Kubernetes API server.
We see that a new <code class="language-plaintext highlighter-rouge">Revision</code> is created and the existing <code class="language-plaintext highlighter-rouge">Configuration</code> is pointing to this newly created <code class="language-plaintext highlighter-rouge">Revision</code>. There is a parent-child relationship between a <code class="language-plaintext highlighter-rouge">Configuration</code> and a <code class="language-plaintext highlighter-rouge">Revision</code>.<br />
We can also use the <code class="language-plaintext highlighter-rouge">kubectl</code> CLI to access the Knative objects:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get revisions
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
hello-world-knative-00001 hello-world-knative hello-world-knative-00001 1 True
hello-world-knative-00002 hello-world-knative hello-world-knative-00002 2 True
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get configurations
NAME LATESTCREATED LATESTREADY READY REASON
hello-world-knative hello-world-knative-00002 hello-world-knative-00002 True
</code></pre></div></div>
<h3 id="running-on-cloud-run">Running on Cloud Run</h3>
<p><a href="https://cloud.google.com/run">Cloud Run</a> is one of the fully managed <a href="https://knative.dev/docs/knative-offerings/">Knative offerings</a> available which we are going to use in this blog post.
It implements most parts of the Knative Serving API. There is another version <a href="https://cloud.google.com/anthos/run">Cloud Run for Anthos</a> which provides Knative installation on your existing Kubernetes/GKE cluster.</p>
<p><img src="/images/2021-03-14/CloudRun.png" alt="CloudRun" /></p>
<p>First let’s configure our environment with <code class="language-plaintext highlighter-rouge">gcloud</code> the <a href="https://cloud.google.com/sdk/docs/install">Google Cloud SDK</a>: extract the Google Cloud project id into an environment variable (to be referenced later), enable the <code class="language-plaintext highlighter-rouge">Cloud Run</code> API, set the <code class="language-plaintext highlighter-rouge">Cloud Run</code> version to be managed, and set a default region for <code class="language-plaintext highlighter-rouge">Cloud Run</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ PROJECT_ID=$(gcloud config get-value project)
$ gcloud services enable run.googleapis.com
$ gcloud config set run/platform managed
$ gcloud config set run/region europe-west6
</code></pre></div></div>
<p>Using <code class="language-plaintext highlighter-rouge">Cloud Run</code> we can deploy a container image stored in <a href="https://cloud.google.com/container-registry">Container Registry</a> or <a href="https://cloud.google.com/artifact-registry">Artifact Registry</a>. We are going to use here <code class="language-plaintext highlighter-rouge">Container Registry</code>.
First, we tag our image:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker tag altfatterz/hello-world-knative:0.0.1-SNAPSHOT eu.gcr.io/<span class="k">${</span><span class="nv">PROJECT_ID</span><span class="k">}</span>/hello-world-knative:0.0.1-SNAPSHOT
</code></pre></div></div>
<p>Next, we configure docker authentication for <code class="language-plaintext highlighter-rouge">gcloud</code> and push the image:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud auth configure-docker
<span class="nv">$ </span>docker push eu.gcr.io/<span class="k">${</span><span class="nv">PROJECT_ID</span><span class="k">}</span>/hello-world-knative:0.0.1-SNAPSHOT
</code></pre></div></div>
<p>We can verify that the image was uploaded using:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud container images list-tags eu.gcr.io/<span class="k">${</span><span class="nv">PROJECT_ID</span><span class="k">}</span>/hello-world-knative
DIGEST TAGS TIMESTAMP
1f2a539da7d5 0.0.1-SNAPSHOT 2021-03-14T08:40:25
</code></pre></div></div>
<p>Finally, we deploy our service:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud run deploy hello-world-knative <span class="nt">--allow-unauthenticated</span> <span class="se">\</span>
<span class="nt">--cpu</span><span class="o">=</span>2 <span class="nt">--memory</span><span class="o">=</span>1G <span class="nt">--set-env-vars</span><span class="o">=</span><span class="s2">"SPRING_PROFILES_ACTIVE=prod"</span> <span class="se">\</span>
<span class="nt">--image</span><span class="o">=</span>eu.gcr.io/<span class="k">${</span><span class="nv">PROJECT_ID</span><span class="k">}</span>/hello-world-knative:0.0.1-SNAPSHOT
</code></pre></div></div>
<p>After a minute or so we have our service deployed:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Deploying container to Cloud Run service <span class="o">[</span>hello-world-knative] <span class="k">in </span>project <span class="o">[</span>cloud-run-demos-306217] region <span class="o">[</span>europe-west6]
✓ Deploying new service... Done.
✓ Creating Revision...
✓ Routing traffic...
✓ Setting IAM Policy...
Done.
Service <span class="o">[</span>hello-world-knative] revision <span class="o">[</span>hello-world-knative-00001-pet] has been deployed and is serving 100 percent of traffic.
Service URL: https://hello-world-knative-edbd2pdvbq-oa.a.run.app
</code></pre></div></div>
<p>For more information look into the official documentation of <a href="https://cloud.google.com/run/docs">Cloud Run</a>. The <a href="https://github.com/ahmetb/cloud-run-faq">cloud-run-faq</a> community-maintained informal knowledge base is also very useful.</p>
<h3 id="conclusion">Conclusion</h3>
<p>We have seen how to set up a basic local <code class="language-plaintext highlighter-rouge">Knative Serving</code> environment, deploy a <a href="https://github.com/altfatterz/cloud-run-demos/tree/master/hello-world-knative">simple</a> Spring Boot application on it. We have looked into the core concepts of <code class="language-plaintext highlighter-rouge">Knative Serving</code> and also we have seen how we can deploy our application to a fully managed <code class="language-plaintext highlighter-rouge">Knative Serving</code> offering like <code class="language-plaintext highlighter-rouge">Cloud Run</code> from Google</p>Knative goal is to make developers more productive by providing higher level abstractions (CRDs) on top of Kubernetes. It solves common problems like stand up a scalable, secure, stateless service in seconds, connecting disparate systems together. Knative also brings serverless deployment models to Kubernetes.Kubernetes PersistentVolume Subsystem2021-01-17T00:00:00+00:002021-01-17T00:00:00+00:00http://zoltanaltfatter.com/2021/01/17/kubernetes-persistentvolume-subystem<p>Container storage is <code class="language-plaintext highlighter-rouge">ephemeral</code>, it goes away when the container does.
Because of this, storage needs to be independent of the container in order to live beyond the container.
In Kubernetes, <code class="language-plaintext highlighter-rouge">volumes</code> provide the abstraction to decouple storage from the pod’s containers.
When we attach a volume to a pod it provides a directory mounted inside the pod’s containers so that we can access files.
There are many different volume types.</p>
<h3 id="emptydir">emptyDir</h3>
<p>The <a href="https://kubernetes.io/docs/concepts/storage/volumes/#emptydir">emptyDir</a> volume type is the simplest. It is backed by the host currently running the pod. When the pod is scheduled on the node, it creates an empty directory on the node.
It is not completely permanent, if the pod is removed from the node and moved to another node, the data is deleted. However, it will persist beyond the life of the container.
It allows multiple containers within a pod to have both read and write access to the files in the <code class="language-plaintext highlighter-rouge">emptyDir</code> volume.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Pod</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">my-pod</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">busybox1</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">busybox</span>
<span class="na">command</span><span class="pi">:</span> <span class="pi">[</span> <span class="s2">"</span><span class="s">/bin/sh"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">-c"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">while</span><span class="nv"> </span><span class="s">true;</span><span class="nv"> </span><span class="s">do</span><span class="nv"> </span><span class="s">sleep</span><span class="nv"> </span><span class="s">3600;</span><span class="nv"> </span><span class="s">done"</span> <span class="pi">]</span>
<span class="na">volumeMounts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/data</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">data</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">busybox2</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">busybox</span>
<span class="na">command</span><span class="pi">:</span> <span class="pi">[</span> <span class="s2">"</span><span class="s">/bin/sh"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">-c"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">while</span><span class="nv"> </span><span class="s">true;</span><span class="nv"> </span><span class="s">do</span><span class="nv"> </span><span class="s">sleep</span><span class="nv"> </span><span class="s">3600;</span><span class="nv"> </span><span class="s">done"</span> <span class="pi">]</span>
<span class="na">volumeMounts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/data</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">data</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">data</span>
<span class="na">emptyDir</span><span class="pi">:</span> <span class="pi">{</span> <span class="pi">}</span>
</code></pre></div></div>
<p>After creating the above pod in our cluster, we can validate that a file created with the first container can be accessed by the second container.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> my-pod <span class="nt">-c</span> busybox1 <span class="nt">--</span> /bin/sh
<span class="nv">$ </span><span class="nb">cd</span> /data
<span class="nv">$ </span><span class="nb">echo</span> <span class="s2">"hello"</span> <span class="o">></span> hello.txt
<span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> my-pod <span class="nt">-c</span> busybox2 <span class="nt">--</span> /bin/sh
<span class="nv">$ </span><span class="nb">cat</span> /data/hello.txt
hello
</code></pre></div></div>
<h3 id="persistentvolume-and-persistentvolumeclaim">PersistentVolume and PersistentVolumeClaim</h3>
<p><code class="language-plaintext highlighter-rouge">PersistentVolume</code> or <code class="language-plaintext highlighter-rouge">PV</code> represents a storage resource. If a <code class="language-plaintext highlighter-rouge">Node</code> in Kubernetes represents CPU and memory resources for a pod, a <code class="language-plaintext highlighter-rouge">PersistentVolume</code> represents storage resource to a pod.
A <code class="language-plaintext highlighter-rouge">PersistentVolumeClaim</code> or <code class="language-plaintext highlighter-rouge">PVC</code> is an abstraction between the pod and the PV. When you create your pod you don’t need to worry about where the storage is located, how is implemented, only you need to specify the <code class="language-plaintext highlighter-rouge">PVC</code> with the required <code class="language-plaintext highlighter-rouge">storageClass</code> and <code class="language-plaintext highlighter-rouge">accessModes</code>. (more about these later)</p>
<p>First create our local Kubernetes cluster. With <a href="https://kind.sigs.k8s.io/">kind</a> is very easy:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kind delete cluster <span class="nt">--name</span> kubernetes-storage-demo
</code></pre></div></div>
<p>Now we are ready to create a <code class="language-plaintext highlighter-rouge">PV</code> and consume that storage resource within a pod.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">PersistentVolume</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">mysql-pv</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">accessModes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ReadWriteOnce</span>
<span class="na">capacity</span><span class="pi">:</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s">1Gi</span>
<span class="na">storageClassName</span><span class="pi">:</span> <span class="s">localdisk</span>
<span class="na">hostPath</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">/mnt/data</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">hostPath</code> will allocate storage on an actual node in the cluster where the pod is running.
After creating the <code class="language-plaintext highlighter-rouge">PV</code> in our cluster, we can see that its status is <code class="language-plaintext highlighter-rouge">Available</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
mysql-pv 1Gi RWO Retain Available localdisk 3s
</code></pre></div></div>
<p>Next we create a <code class="language-plaintext highlighter-rouge">PVC</code>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">PersistentVolumeClaim</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">mysql-pvc</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">accessModes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ReadWriteOnce</span>
<span class="na">storageClassName</span><span class="pi">:</span> <span class="s">localdisk</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">requests</span><span class="pi">:</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s">500Mi</span>
</code></pre></div></div>
<p>After creating the <code class="language-plaintext highlighter-rouge">PVC</code> in our cluster, we can see that both the status of the <code class="language-plaintext highlighter-rouge">PV</code> and <code class="language-plaintext highlighter-rouge">PVC</code> is BOUND.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pvc Bound mysql-pv 1Gi RWO localdisk 7s
<span class="nv">$ </span>kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
mysql-pv 1Gi RWO Retain Bound default/mysql-pvc localdisk 91s
</code></pre></div></div>
<p>Finally, we create a pod referencing the <code class="language-plaintext highlighter-rouge">PVC</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Pod</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">mysql-pod</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">mysql</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">mysql:5.6</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">3306</span>
<span class="na">env</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">MYSQL_ROOT_PASSWORD</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">password</span>
<span class="na">volumeMounts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">mysql-volume</span>
<span class="na">mountPath</span><span class="pi">:</span> <span class="s">/var/lib/mysql</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">mysql-volume</span>
<span class="na">persistentVolumeClaim</span><span class="pi">:</span>
<span class="na">claimName</span><span class="pi">:</span> <span class="s">mysql-pvc</span>
</code></pre></div></div>
<p>After creating the pod in our cluster, we can verify that the storage was created on the Node. With our one node Kubernetes cluster is easy:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker <span class="nb">exec</span> <span class="nt">-it</span> kubernetes-storage-demo-control-plane <span class="nb">du</span> <span class="nt">-sh</span> /mnt/data/mysql
7.0M /mnt/data/mysql
</code></pre></div></div>
<p>In order to delete our local 1 node Kubernetes cluster we can use the following command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kind delete cluster <span class="nt">--name</span> kubernetes-storage-demo
</code></pre></div></div>
<h3 id="gcepersistentdisk">gcePersistentDisk</h3>
<p>Since pods come and go, they are scheduled on different nodes, we need a more robust solution for our data persistence. In this section we are going to use a GKE cluster and delegate the storage to a <code class="language-plaintext highlighter-rouge">PersistentDisk</code>.</p>
<p>First lets create a GKE cluster:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud container clusters create kubernetes-storage-demo-cluster
<span class="nv">$ </span>kubectl get nodes
gke-kubernetes-storage-d-default-pool-378cb5bc-5tq5 Ready <none> 7m14s v1.16.15-gke.6000
gke-kubernetes-storage-d-default-pool-378cb5bc-dq78 Ready <none> 7m14s v1.16.15-gke.6000
gke-kubernetes-storage-d-default-pool-378cb5bc-z1dm Ready <none> 7m14s v1.16.15-gke.6000
</code></pre></div></div>
<p>Next we create a GCE Persistent Disk:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud compute disks create <span class="nt">--size</span><span class="o">=</span>10Gi <span class="nt">--zone</span><span class="o">=</span>europe-west6-a mongodb
</code></pre></div></div>
<p>We create a <code class="language-plaintext highlighter-rouge">PV</code> using the previously created <code class="language-plaintext highlighter-rouge">PersistentDisk</code> and reference it using <code class="language-plaintext highlighter-rouge">gcePersistentDisk</code>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">PersistentVolume</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">mongodb-pv</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">accessModes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ReadWriteOnce</span>
<span class="na">capacity</span><span class="pi">:</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s">1Gi</span>
<span class="na">storageClassName</span><span class="pi">:</span> <span class="s">ssd-disk</span>
<span class="na">gcePersistentDisk</span><span class="pi">:</span>
<span class="na">pdName</span><span class="pi">:</span> <span class="s">mongodb</span>
</code></pre></div></div>
<p>After this object is created in our cluster we have a 1Gi <code class="language-plaintext highlighter-rouge">PersistentVolume</code> available in our cluster for consumption that will outlive the lifecycle of any pod that use it.
To consume the volume we create a <code class="language-plaintext highlighter-rouge">PVC</code>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">PersistentVolumeClaim</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">mongodb-pvc</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">accessModes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ReadWriteOnce</span>
<span class="na">storageClassName</span><span class="pi">:</span> <span class="s">ssd-disk</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">requests</span><span class="pi">:</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s">500Mi</span>
</code></pre></div></div>
<p>Kubernetes will match our claim to an available <code class="language-plaintext highlighter-rouge">PV</code> that meets our requirements (<code class="language-plaintext highlighter-rouge">accessMode</code>, <code class="language-plaintext highlighter-rouge">storageClassName</code>, <code class="language-plaintext highlighter-rouge">resource.requests.storage</code>). For example if the claim is asking for more storage capacity than the <code class="language-plaintext highlighter-rouge">PV</code>, it will not bind and it will sit there pending. Once a claim is bound, nothing else can claim the same <code class="language-plaintext highlighter-rouge">PV</code> object unless we free it up.</p>
<p>Finally, we need to consume the <code class="language-plaintext highlighter-rouge">PVC</code> in our pod:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Pod</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">mongodb</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">mongo</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">mongo</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">27017</span>
<span class="na">volumeMounts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">mongodb-persitent-storage</span>
<span class="na">mountPath</span><span class="pi">:</span> <span class="s">/data/db</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">mongodb-persitent-storage</span>
<span class="na">persistentVolumeClaim</span><span class="pi">:</span>
<span class="na">claimName</span><span class="pi">:</span> <span class="s">mongodb-pvc</span>
</code></pre></div></div>
<p>After the pod is created on the cluster, let’s create some data, for example a document using the mongodb shell:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> mongodb <span class="nt">--</span> mongo
<span class="o">></span> use mydb
switched to db mydb
<span class="o">></span> db.customers.insert<span class="o">({</span>name:<span class="s1">'John Doe'</span><span class="o">})</span>
WriteResult<span class="o">({</span> <span class="s2">"nInserted"</span> : 1 <span class="o">})</span>
<span class="o">></span> db.customers.find<span class="o">()</span>
<span class="o">{</span> <span class="s2">"_id"</span> : ObjectId<span class="o">(</span><span class="s2">"6002be707b7ff91b50cb9fe7"</span><span class="o">)</span>, <span class="s2">"name"</span> : <span class="s2">"John Doe"</span> <span class="o">}</span>
</code></pre></div></div>
<p>We verify on which node is the pod is running:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods <span class="nt">-o</span> wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mongodb 1/1 Running 0 3m11s 10.100.1.8 gke-kubernetes-storage-d-default-pool-378cb5bc-dq78 <none> <none>
</code></pre></div></div>
<p>Next we would like to see if the pod is scheduled on another node, we can still access the data. For this first we mark the node as <code class="language-plaintext highlighter-rouge">unschedulable</code> using</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl cordon gke-kubernetes-storage-d-default-pool-378cb5bc-dq78
</code></pre></div></div>
<p>Then we delete and recreate the pod.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl delete <span class="nt">-f</span> mongodb-pod.yaml
<span class="nv">$ </span>kubectl create <span class="nt">-f</span> mongodb-pod.yaml
</code></pre></div></div>
<p>Next, we verify that the pod is running on a different node now:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods <span class="nt">-o</span> wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mongodb 1/1 Running 0 52s 10.100.0.3 gke-kubernetes-storage-d-default-pool-378cb5bc-5tq5 <none> <none>
</code></pre></div></div>
<p>Finally, we see that we still have our data:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> mongodb <span class="nt">--</span> mongo
<span class="o">></span> use mydb
switched to db mydb
<span class="o">></span> db.customers.find<span class="o">()</span>
<span class="o">{</span> <span class="s2">"_id"</span> : ObjectId<span class="o">(</span><span class="s2">"600361e62ba042906cb98af1"</span><span class="o">)</span>, <span class="s2">"name"</span> : <span class="s2">"John Doe"</span> <span class="o">}</span>
</code></pre></div></div>
<h3 id="accessmodes">AccessModes</h3>
<p>There are 3 ways a pod can access a volume. Not all volumes support all these access modes, generally block based volumes support <code class="language-plaintext highlighter-rouge">ReadWriteOnce</code> and <code class="language-plaintext highlighter-rouge">ReadOnlyMany</code>, and file based volumes can support even <code class="language-plaintext highlighter-rouge">ReadWriteMany</code> access mode. The <code class="language-plaintext highlighter-rouge">PersistentVolume</code> in GCP does not support for example <code class="language-plaintext highlighter-rouge">ReadWriteMany</code> access mode. A single volume can only be opened in one mode at a time.</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">ReadWriteOnce</code> (RWO) - can only be mounted as read-write by one pod in the cluster.</li>
<li><code class="language-plaintext highlighter-rouge">ReadOnlyMany</code> (ROM) - many pods can mount it, but only in read only mode.</li>
<li><code class="language-plaintext highlighter-rouge">ReadWriteMany</code> (RWM) - many pods can mount it in read write mode</li>
</ol>
<p>Worth to note that all replicas included in a <code class="language-plaintext highlighter-rouge">Deployment</code> will share the same <code class="language-plaintext highlighter-rouge">PVC</code>. The only way we can support this is by keeping the volume in <code class="language-plaintext highlighter-rouge">ReadOnlyMany</code> access mode. Even if there is a single pod in our deployment with <code class="language-plaintext highlighter-rouge">ReadWriteOnce</code> can be problematic depending on the rollout strategy to update a deployment. An updated replica set is not going to be able to mount a volume in read-write mode while the previous replica set still exists.</p>
<h3 id="reclaim-policy">Reclaim Policy</h3>
<p>A reclaim policy specifies what happens when a claim on a volume is released. It can be <code class="language-plaintext highlighter-rouge">Delete</code>(default) or <code class="language-plaintext highlighter-rouge">Retain</code>. In case of <code class="language-plaintext highlighter-rouge">Delete</code> when the claim is released the data is removed, however with <code class="language-plaintext highlighter-rouge">Retain</code> when we can keep the volume and its content, when the claim is released, or even when the pod is failed or deleted.</p>
<h3 id="storageclasses">StorageClasses</h3>
<p>In the previous example we needed first to create manually the <code class="language-plaintext highlighter-rouge">PersistentDisk</code> in and create also the <code class="language-plaintext highlighter-rouge">PV</code> in order to use reference it in our <code class="language-plaintext highlighter-rouge">PVC</code>. Luckily storage classes make this way more dynamic.<br />
<code class="language-plaintext highlighter-rouge">StorageClasses</code> enable dynamic provisioning of volumes. Like everything in Kubernetes, a storage class is an API resource defined in a YAML file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast
provisioner: kubernetes.io/gce-pd
parameters:
<span class="nb">type</span>: pd-ssd
</code></pre></div></div>
<p>The <a href="https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner">provisioner</a> describers what volume plugin is used for provisioning PVs.
In the above example the storage class will provision a zonal sdd disk with ext4 filesystem type.</p>
<p>Our GKE cluster has already a default storage class:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get sc
NAME PROVISIONER AGE
standard <span class="o">(</span>default<span class="o">)</span> kubernetes.io/gce-pd 91s
</code></pre></div></div>
<p>which is using non sdd disk</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl describe sc standard
Name: standard
IsDefaultClass: Yes
Annotations: storageclass.kubernetes.io/is-default-class<span class="o">=</span><span class="nb">true
</span>Provisioner: kubernetes.io/gce-pd
Parameters: <span class="nb">type</span><span class="o">=</span>pd-standard
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
</code></pre></div></div>
<p>After creating the <code class="language-plaintext highlighter-rouge">fast</code> StorageClass in our GKE cluster let’s use it in PVC:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">PersistentVolumeClaim</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">mongodb-pvc</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">accessModes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ReadWriteOnce</span>
<span class="na">storageClassName</span><span class="pi">:</span> <span class="s">fast</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">requests</span><span class="pi">:</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s">1Gi</span>
</code></pre></div></div>
<p>After creating the <code class="language-plaintext highlighter-rouge">mongodb-pvc</code> we can see the that the status is <code class="language-plaintext highlighter-rouge">BOUND</code> as opposed to <code class="language-plaintext highlighter-rouge">AVAILABLE</code> and is using our <code class="language-plaintext highlighter-rouge">fast</code> storage class.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mongodb-pvc Bound pvc-c0719ec2-9fbd-4d56-8650-7bba6890fabc 1Gi RWO fast 3s
</code></pre></div></div>
<p>Behind the scene we can see that both the <code class="language-plaintext highlighter-rouge">PersistentVolume</code> and the underlying <code class="language-plaintext highlighter-rouge">GCP Persistent Disk</code> were created:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-c0719ec2-9fbd-4d56-8650-7bba6890fabc 1Gi RWO Delete Bound default/mongodb-pvc fast 4m30s
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud compute disks list
gke-kubernetes-storage-pvc-c0719ec2-9fbd-4d56-8650-7bba6890fabc europe-west6-a zone 1 pd-ssd READY
</code></pre></div></div>
<p>Storage can be a pretty dry topic, but hopefully with these examples (which you can also try) it is easier to understand. All the examples used in this blog post can be found in my github <a href="https://github.com/altfatterz/learning-kubernetes/tree/master/volume-demo">repo</a>.</p>Container storage is ephemeral, it goes away when the container does. Because of this, storage needs to be independent of the container in order to live beyond the container. In Kubernetes, volumes provide the abstraction to decouple storage from the pod’s containers. When we attach a volume to a pod it provides a directory mounted inside the pod’s containers so that we can access files. There are many different volume types.Kubernetes Network Policies2021-01-10T00:00:00+00:002021-01-10T00:00:00+00:00http://zoltanaltfatter.com/2021/01/10/kubernetes-network-policies<p><a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">Network policies</a> allow to secure a
Kubernetes network. They allow you to control what traffic is allowed to come in or out from your pod. The Kubernetes
network model specifies that every pods gets its own IP address, containers within a pod share the IP address and can
freely communicate with each other. Pods can communicate with all other pods in the cluster using pod IP addresses (without NAT). This style of network is sometimes referred to as a “flat network”.</p>
<p>This approach simplifies the network and allows new workloads scheduled dynamically in the cluster with no dependency on
the network design. The network policy represents an important evolution of network security, not just because it handles the dynamic nature of modern
microservices, but because it empowers dev and devops engineers to easily define network security themselves, rather
than needing to learn low-level networking details or raise tickets with a separate team responsible for managing
firewalls.</p>
<p>Every Kubernetes <code class="language-plaintext highlighter-rouge">NetworkPolicy</code> resource is namespaced and has a pod selector that defines the pods the policies
applies to. (in the below example with label <code class="language-plaintext highlighter-rouge">app: account-service</code>)
Then it has a series of <code class="language-plaintext highlighter-rouge">ingress</code> or <code class="language-plaintext highlighter-rouge">egress</code> rules or both, which defines again with selectors other pods
in the cluster.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.k8s.io/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">NetworkPolicy</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">account-service-policy</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">production</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">podSelector</span><span class="pi">:</span>
<span class="na">matchLabels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">account-service</span>
<span class="na">policyTypes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Ingress</span>
<span class="pi">-</span> <span class="s">Egress</span>
<span class="na">ingress</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">from</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">podSelector</span><span class="pi">:</span>
<span class="na">matchLabels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">account-service-client</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">80</span>
<span class="na">egress</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">to</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">podSelector</span><span class="pi">:</span>
<span class="na">matchLabels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">account-service-db</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">5432</span>
</code></pre></div></div>
<p>Kubernetes itself does not enforce network policies (just stores them) and instead delegates their enforcement to
network plugins.</p>
<p>Kubernetes built in network
support, <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#kubenet">kubenet</a>
does not implement network policies. It is common to use third party network implementations which plug into Kubernetes
using the <a href="https://www.cni.dev/">CNI</a> (Container Network Interface) API.</p>
<p>The most popular CNI plugins are: <a href="https://github.com/coreos/flannel">flannel</a>
, <a href="https://github.com/projectcalico/calico">calico</a>, <a href="https://github.com/weaveworks/weave">weave</a>
and <a href="https://docs.projectcalico.org/getting-started/kubernetes/flannel/flannel">canal</a> (technically a combination of
multiple plugins). <a href="https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/">Here</a>
you can find a good article which compares them.</p>
<p>In this blog post we are going to use Canal, but first lets create our Kubernetes cluster.</p>
<h4 id="create-our-kubernetes-cluster">Create our Kubernetes cluster</h4>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kind create cluster <span class="nt">--name</span> network-policy-demo
Creating cluster <span class="s2">"network-policy-demo"</span> ...
✓ Ensuring node image <span class="o">(</span>kindest/node:v1.19.1<span class="o">)</span> 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to <span class="s2">"kind-network-policy-demo"</span>
</code></pre></div></div>
<p>Let’s check the running pods in all namespaces by default.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods <span class="nt">--all-namespaces</span>
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-f9fd979d6-dxbs2 1/1 Running 0 73s
kube-system coredns-f9fd979d6-kl77w 1/1 Running 0 73s
kube-system etcd-network-policy-demo-control-plane 1/1 Running 0 85s
kube-system kindnet-vwnbz 1/1 Running 0 73s
kube-system kube-apiserver-network-policy-demo-control-plane 1/1 Running 0 85s
kube-system kube-controller-manager-network-policy-demo-control-plane 1/1 Running 0 85s
kube-system kube-proxy-ck6tp 1/1 Running 0 73s
kube-system kube-scheduler-network-policy-demo-control-plane 1/1 Running 0 85s
local-path-storage local-path-provisioner-78776bfc44-vvhjj 1/1 Running 0 73s
</code></pre></div></div>
<h4 id="install-calico-for-policy-and-flannel-aka-canal-for-networking">Install Calico for policy and flannel (aka Canal) for networking.</h4>
<p>Next, we install the network policy support.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl https://docs.projectcalico.org/manifests/canal.yaml <span class="nt">-O</span>
<span class="nv">$ </span>kubectl apply <span class="nt">-f</span> canal.yaml
configmap/canal-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/canal-flannel created
clusterrolebinding.rbac.authorization.k8s.io/canal-calico created
daemonset.apps/canal created
serviceaccount/canal created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
</code></pre></div></div>
<p>Verify that two more pods were created in the <code class="language-plaintext highlighter-rouge">kube-system</code> namespace</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods <span class="nt">--all-namespaces</span>
...
kube-system calico-kube-controllers-744cfdf676-g82hg 1/1 Running 0 102s
kube-system canal-rb8d6 2/2 Running 0 102s
</code></pre></div></div>
<p>Next we define two pods <code class="language-plaintext highlighter-rouge">server</code> and <code class="language-plaintext highlighter-rouge">client</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Pod</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">server</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">secure-pod</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">nginx</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">nginx</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">80</span>
</code></pre></div></div>
<p>This is a simple <code class="language-plaintext highlighter-rouge">nginx</code> pod running on port 80 with label <code class="language-plaintext highlighter-rouge">app=secure-pod</code>. We will try to access this pod from a <code class="language-plaintext highlighter-rouge">client</code> pod.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Pod</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">client</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">busybox</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">radial/busyboxplus:curl</span>
<span class="na">command</span><span class="pi">:</span> <span class="pi">[</span> <span class="s2">"</span><span class="s">/bin/sh"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">-c"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">while</span><span class="nv"> </span><span class="s">true;</span><span class="nv"> </span><span class="s">do</span><span class="nv"> </span><span class="s">sleep</span><span class="nv"> </span><span class="s">3600;</span><span class="nv"> </span><span class="s">done"</span> <span class="pi">]</span>
</code></pre></div></div>
<p>We create both pods on the cluster.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> server.yaml
<span class="nv">$ </span>kubectl apply <span class="nt">-f</span> client.yaml
<span class="nv">$ </span>kubectl get pods <span class="nt">-o</span> wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
client 1/1 Running 0 10m 10.244.0.7 network-policy-demo-control-plane <none> <none>
server 1/1 Running 0 10m 10.244.0.6 network-policy-demo-control-plane <none> <none>
</code></pre></div></div>
<p>Then from the <code class="language-plaintext highlighter-rouge">client</code> pod we can see that we can access the <code class="language-plaintext highlighter-rouge">server</code> pod using the <code class="language-plaintext highlighter-rouge">server</code> pod’s IP address:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> client <span class="nt">--</span> curl 10.244.0.6
<<span class="o">!</span>DOCTYPE html>
<html>
<<span class="nb">head</span><span class="o">></span>
<title>Welcome to nginx!</title>
<style>
body <span class="o">{</span>
width: 35em<span class="p">;</span>
margin: 0 auto<span class="p">;</span>
font-family: Tahoma, Verdana, Arial, sans-serif<span class="p">;</span>
<span class="o">}</span>
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a <span class="nv">href</span><span class="o">=</span><span class="s2">"http://nginx.org/"</span><span class="o">></span>nginx.org</a>.<br/>
Commercial support is available at
<a <span class="nv">href</span><span class="o">=</span><span class="s2">"http://nginx.com/"</span><span class="o">></span>nginx.com</a>.</p>
<p><em>Thank you <span class="k">for </span>using nginx.</em></p>
</body>
</html>
</code></pre></div></div>
<p>Next we can create a network policy selecting our <code class="language-plaintext highlighter-rouge">server</code> pod with label <code class="language-plaintext highlighter-rouge">app=secure-pod</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: simple-network-policy
spec:
podSelector:
matchLabels:
app: secure-pod
ingress:
- from:
- podSelector:
matchLabels:
allow-access: <span class="s2">"true"</span>
ports:
- protocol: TCP
port: 80
</code></pre></div></div>
<p>As soon as a network policy applies to a pod that pod is completely locked down, meaning nothing can talk to it and it cannot talk to anything until we provide some rules that whitelist some specific traffic.
The above network policy will only apply to incoming traffic (pods with label <code class="language-plaintext highlighter-rouge">allow-access: true</code>) and the traffic leaving the pod will be completely open.</p>
<p>After applying our network policy on the cluster we can see that the <code class="language-plaintext highlighter-rouge">client</code> pod cannot access anymore the <code class="language-plaintext highlighter-rouge">server</code> pod.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> simple-network-policy
<span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> client <span class="nt">--</span> curl 10.244.0.6
^C
</code></pre></div></div>
<p>After adding the <code class="language-plaintext highlighter-rouge">allow-access=true</code> label to the <code class="language-plaintext highlighter-rouge">client</code> pod it will meet the <code class="language-plaintext highlighter-rouge">matchLabels</code> whitelist condition in our network policy and the access will be allowed:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl label pod client allow-access<span class="o">=</span><span class="nb">true</span>
<span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> client <span class="nt">--</span> curl 10.244.0.6
<<span class="o">!</span>DOCTYPE html>
<html>
<<span class="nb">head</span><span class="o">></span>
<title>Welcome to nginx!</title>
...
</code></pre></div></div>
<p>Hopefully with this example is easier to understand network policies and you can play with it yourself. The resources in this example can be found in my github <a href="https://github.com/altfatterz/learning-kubernetes/tree/master/network-policy-demo">repo</a>.
Next to <code class="language-plaintext highlighter-rouge">podSelector</code> there is also <code class="language-plaintext highlighter-rouge">namespaceSelector</code> and <code class="language-plaintext highlighter-rouge">ipBlock</code> which you can use in the <code class="language-plaintext highlighter-rouge">ingress</code> or <code class="language-plaintext highlighter-rouge">egress</code> rules.
More details you can find in the official <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">documentation</a>.</p>Network policies allow to secure a Kubernetes network. They allow you to control what traffic is allowed to come in or out from your pod. The Kubernetes network model specifies that every pods gets its own IP address, containers within a pod share the IP address and can freely communicate with each other. Pods can communicate with all other pods in the cluster using pod IP addresses (without NAT). This style of network is sometimes referred to as a “flat network”.Build images with Cloud Native Buildpacks2020-12-26T00:00:00+00:002020-12-26T00:00:00+00:00http://zoltanaltfatter.com/2020/12/26/build-images-with-cloud-native-buildpacks<p>Cloud Native Buildpacks (CNB) is a <a href="https://github.com/buildpacks/spec/blob/main/buildpack.md">specification</a> to transform your application source code into an <a href="https://github.com/opencontainers/image-spec/blob/master/spec.md">OCI</a> image.
It was initiated by Pivotal and Heroku in 2018, and recently moved from Sandbox to Incubation by the <a href="https://www.cncf.io/blog/2020/11/18/toc-approves-cloud-native-buildpacks-from-sandbox-to-incubation/">CNCF</a>.
If you are interested in how it compares to other popular alternatives like <a href="https://github.com/GoogleContainerTools/jib">Jib</a> or <a href="https://github.com/openshift/source-to-image">s2i</a> head over to <a href="https://buildpacks.io/features/">https://buildpacks.io/features/</a></p>
<p>A <strong>buildpack</strong> inspects your application code and transforms the source code into a runnable app image.
It has an <em>auto-detection</em> feature, for example a <strong>Maven buildpack</strong> looks for pom.xml, an <strong>NPM buildpack</strong> looks for a package.json, etc.
There is always a list of ordered buildpacks which are applied on your application source code, which are encapsulated in an image called the <strong>builder</strong>.
The <strong>builder</strong> contains the ordered list of buildpacks together with the base image to use for the app container image.</p>
<p>The fastest way to get started with CNB is with the <a href="https://github.com/buildpacks/pack">pack</a> reference CLI.
<code class="language-plaintext highlighter-rouge">Pack</code> uses Docker to run the build process in isolated environment.</p>
<p>After <a href="https://buildpacks.io/docs/tools/pack/">installing</a> the pack CLI let’s use it on one example:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git clone http://github.com/altfatterz/buildpacks-demo
<span class="nv">$ </span><span class="nb">cd </span>buildpacks-demo
<span class="nv">$ </span>pack build buildpacks-demo <span class="nt">--builder</span> paketobuildpacks/builder:base
</code></pre></div></div>
<p>By default, the image is uploaded to the local Docker daemon, but with <code class="language-plaintext highlighter-rouge">--publish</code> parameter we can specify a registry.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
paketobuildpacks/run base-cnb 4347963b10a1 5 days ago 106MB
paketobuildpacks/builder base 03aa716c9552 41 years ago 558MB
buildpacks-demo latest a29f2833a7a1 41 years ago 278MB
</code></pre></div></div>
<p>We can see that the <code class="language-plaintext highlighter-rouge">paketobuildpacks/builder</code> and the run-image <code class="language-plaintext highlighter-rouge">paketobuildpacks/run</code> were pulled from the <a href="https://hub.docker.com/search?q=paketobuildpacks">dockerhub</a>.
If you are confused by the <code class="language-plaintext highlighter-rouge">41 years ago</code> timestamp it has to do with to allow reproducible builds. Here you can find more information about the details: <a href="https://medium.com/buildpacks/time-travel-with-pack-e0efd8bf05db">Time Travel with Pack</a></p>
<p><a href="https://paketo.io/">Packeto Buildpacks</a> is an implementation of <a href="https://buildpacks.io/">Cloud Native Buildpacks</a>.</p>
<p>With <code class="language-plaintext highlighter-rouge">pack</code> we can inspect the built app image. Here we can see that with the used builder which buildpacks were applied to our source code and which was the base image.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pack inspect-image buildpacks-demo
...
Base Image:
Reference: 4347963b10a16e72afa854b12834d41835f4347d7a92f5428bb47c0238464c95
Top Layer: sha256:8cc74c487427203dd7ff8b5a615b63f9178117a42a4a966f3606fd51746e0fc5
Run Images:
index.docker.io/paketobuildpacks/run:base-cnb
gcr.io/paketo-buildpacks/run:base-cnb
Buildpacks:
ID VERSION
paketo-buildpacks/ca-certificates 1.0.1
paketo-buildpacks/bellsoft-liberica 6.0.0
paketo-buildpacks/maven 3.2.1
paketo-buildpacks/executable-jar 3.1.3
paketo-buildpacks/apache-tomcat 3.1.0
paketo-buildpacks/dist-zip 2.2.2
paketo-buildpacks/spring-boot 3.5.0
...
</code></pre></div></div>
<p><a href="https://github.com/wagoodman/dive">Dive</a> is also a great tool to get a view into the file system of each layer.</p>
<p><img src="/images/2020-12-26/dive.png" alt="Dive" /></p>
<p>We can inspect also the builder using</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pack inspect-builder paketobuildpacks/builder:base
</code></pre></div></div>
<p>where we can see that it supports multiple programming languages.</p>
<p>Next to <code class="language-plaintext highlighter-rouge">detect</code> and <code class="language-plaintext highlighter-rouge">build</code> there are 3 more stages in the lifecycle:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">detector</code> - Detects the app type and produces a build plan.</li>
<li><code class="language-plaintext highlighter-rouge">analyzer</code> - Restores layer metadata from the previous image and from the cache.</li>
<li><code class="language-plaintext highlighter-rouge">restorer</code> - Restores cached layers.</li>
<li><code class="language-plaintext highlighter-rouge">builder</code> - Executes buildpacks.</li>
<li><code class="language-plaintext highlighter-rouge">exporter</code> - Creates an image and caches layers.</li>
</ol>
<p>This ensures that after the first build the builds are much faster, but also safer since only the layers that need to change are replaced.</p>
<p>Spring Boot also has support for the <a href="https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-container-images-buildpacks">CNB</a>. The <a href="https://docs.spring.io/spring-boot/docs/2.4.1/maven-plugin/reference/htmlsingle/#build-image">Maven</a> and <a href="https://docs.spring.io/spring-boot/docs/2.4.1/gradle-plugin/reference/htmlsingle/#build-image">Gradle</a> plugins are using the Packeto Buildpacks implementation.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mvn spring-boot:build-image
</code></pre></div></div>
<p>Below you can explore what is happening in each lifecycle: <code class="language-plaintext highlighter-rouge">DETECTING</code>, <code class="language-plaintext highlighter-rouge">ANALYZING</code>, <code class="language-plaintext highlighter-rouge">RESTORING</code>, <code class="language-plaintext highlighter-rouge">BUILDING</code> and <code class="language-plaintext highlighter-rouge">EXPORTING</code>.
The image is based on the <code class="language-plaintext highlighter-rouge">docker.io/paketobuildpacks/run:base-cnb</code> run image, which is a minimal Paketo stack based on Ubuntu 18.04.
From the logs we can see that from the 18 available buildpacks 5 are applied to the app source code. The
<code class="language-plaintext highlighter-rouge">paketo-buildpacks/ca-certificates</code> adds CA certificates to the system truststore at build and runtime, the <code class="language-plaintext highlighter-rouge">paketo-buildpacks/bellsoft-liberica</code> requests that a JRE be installed.
In the end <code class="language-plaintext highlighter-rouge">paketo-buildpacks/spring-boot</code> contributes Spring Boot dependency information and slices an application into multiple layers.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>INFO] Building image <span class="s1">'docker.io/library/buildpacks-demo:0.0.1-SNAPSHOT'</span>
<span class="o">[</span>INFO]
<span class="o">[</span>INFO] <span class="o">></span> Pulling builder image <span class="s1">'docker.io/paketobuildpacks/builder:base'</span> 100%
<span class="o">[</span>INFO] <span class="o">></span> Pulled builder image <span class="s1">'paketobuildpacks/builder@sha256:984a3684db80a6d53214b81a9f21c31529bede5b447d6d6d82d94cd6734d2424'</span>
<span class="o">[</span>INFO] <span class="o">></span> Pulling run image <span class="s1">'docker.io/paketobuildpacks/run:base-cnb'</span> 100%
<span class="o">[</span>INFO] <span class="o">></span> Pulled run image <span class="s1">'paketobuildpacks/run@sha256:f393fa2927a2619a10fc09bb109f822d20df909c10fed4ce3c36fad313ea18e3'</span>
<span class="o">[</span>INFO] <span class="o">></span> Executing lifecycle version v0.10.1
<span class="o">[</span>INFO] <span class="o">></span> Using build cache volume <span class="s1">'pack-cache-89b2691a0bb8.build'</span>
<span class="o">[</span>INFO]
<span class="o">[</span>INFO] <span class="o">></span> Running creator
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="o">===></span> DETECTING
<span class="o">[</span>INFO] <span class="o">[</span>creator] 5 of 18 buildpacks participating
<span class="o">[</span>INFO] <span class="o">[</span>creator] paketo-buildpacks/ca-certificates 1.0.1
<span class="o">[</span>INFO] <span class="o">[</span>creator] paketo-buildpacks/bellsoft-liberica 6.0.0
<span class="o">[</span>INFO] <span class="o">[</span>creator] paketo-buildpacks/executable-jar 3.1.3
<span class="o">[</span>INFO] <span class="o">[</span>creator] paketo-buildpacks/dist-zip 2.2.2
<span class="o">[</span>INFO] <span class="o">[</span>creator] paketo-buildpacks/spring-boot 3.5.0
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="o">===></span> ANALYZING
<span class="o">[</span>INFO] <span class="o">[</span>creator] Previous image with name <span class="s2">"docker.io/library/buildpacks-demo:0.0.1-SNAPSHOT"</span> not found
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="o">===></span> RESTORING
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="o">===></span> BUILDING
<span class="o">[</span>INFO] <span class="o">[</span>creator]
<span class="o">[</span>INFO] <span class="o">[</span>creator] Paketo CA Certificates Buildpack 1.0.1
<span class="o">[</span>INFO] <span class="o">[</span>creator] https://github.com/paketo-buildpacks/ca-certificates
<span class="o">[</span>INFO] <span class="o">[</span>creator] Launch Helper: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing profile.d/helper
<span class="o">[</span>INFO] <span class="o">[</span>creator]
<span class="o">[</span>INFO] <span class="o">[</span>creator] Paketo BellSoft Liberica Buildpack 6.0.0
<span class="o">[</span>INFO] <span class="o">[</span>creator] https://github.com/paketo-buildpacks/bellsoft-liberica
<span class="o">[</span>INFO] <span class="o">[</span>creator] Build Configuration:
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="nv">$BP_JVM_VERSION</span> 11.<span class="k">*</span> the Java version
<span class="o">[</span>INFO] <span class="o">[</span>creator] Launch Configuration:
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="nv">$BPL_JVM_HEAD_ROOM</span> 0 the headroom <span class="k">in </span>memory calculation
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="nv">$BPL_JVM_LOADED_CLASS_COUNT</span> 35% of classes the number of loaded classes <span class="k">in </span>memory calculation
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="nv">$BPL_JVM_THREAD_COUNT</span> 250 the number of threads <span class="k">in </span>memory calculation
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="nv">$JAVA_TOOL_OPTIONS</span> the JVM launch flags
<span class="o">[</span>INFO] <span class="o">[</span>creator] BellSoft Liberica JRE 11.0.9: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Downloading from https://github.com/bell-sw/Liberica/releases/download/11.0.9.1+1/bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz
<span class="o">[</span>INFO] <span class="o">[</span>creator] Verifying checksum
<span class="o">[</span>INFO] <span class="o">[</span>creator] Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding 138 container CA certificates to JVM truststore
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/BPI_APPLICATION_PATH.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/BPI_JVM_CACERTS.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/BPI_JVM_CLASS_COUNT.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/JAVA_HOME.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/MALLOC_ARENA_MAX.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Launch Helper: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/active-processor-count
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/java-opts
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/link-local-dns
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-configurer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-classpath-9
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing profile.d/helper
<span class="o">[</span>INFO] <span class="o">[</span>creator] JVMKill Agent 1.16.0: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Downloading from https://github.com/cloudfoundry/jvmkill/releases/download/v1.16.0.RELEASE/jvmkill-1.16.0-RELEASE.so
<span class="o">[</span>INFO] <span class="o">[</span>creator] Verifying checksum
<span class="o">[</span>INFO] <span class="o">[</span>creator] Copying to /layers/paketo-buildpacks_bellsoft-liberica/jvmkill
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/JAVA_TOOL_OPTIONS.append
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/JAVA_TOOL_OPTIONS.delim
<span class="o">[</span>INFO] <span class="o">[</span>creator] Java Security Properties: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/JAVA_SECURITY_PROPERTIES.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/JAVA_TOOL_OPTIONS.append
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/JAVA_TOOL_OPTIONS.delim
<span class="o">[</span>INFO] <span class="o">[</span>creator]
<span class="o">[</span>INFO] <span class="o">[</span>creator] Paketo Executable JAR Buildpack 3.1.3
<span class="o">[</span>INFO] <span class="o">[</span>creator] https://github.com/paketo-buildpacks/executable-jar
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/CLASSPATH.delim
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/CLASSPATH.prepend
<span class="o">[</span>INFO] <span class="o">[</span>creator] Process types:
<span class="o">[</span>INFO] <span class="o">[</span>creator] executable-jar: java org.springframework.boot.loader.JarLauncher
<span class="o">[</span>INFO] <span class="o">[</span>creator] task: java org.springframework.boot.loader.JarLauncher
<span class="o">[</span>INFO] <span class="o">[</span>creator] web: java org.springframework.boot.loader.JarLauncher
<span class="o">[</span>INFO] <span class="o">[</span>creator]
<span class="o">[</span>INFO] <span class="o">[</span>creator] Paketo Spring Boot Buildpack 3.5.0
<span class="o">[</span>INFO] <span class="o">[</span>creator] https://github.com/paketo-buildpacks/spring-boot
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating slices from layers index
<span class="o">[</span>INFO] <span class="o">[</span>creator] dependencies
<span class="o">[</span>INFO] <span class="o">[</span>creator] spring-boot-loader
<span class="o">[</span>INFO] <span class="o">[</span>creator] snapshot-dependencies
<span class="o">[</span>INFO] <span class="o">[</span>creator] application
<span class="o">[</span>INFO] <span class="o">[</span>creator] Launch Helper: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing profile.d/helper
<span class="o">[</span>INFO] <span class="o">[</span>creator] Web Application Type: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Servlet web application detected
<span class="o">[</span>INFO] <span class="o">[</span>creator] Writing env.launch/BPL_JVM_THREAD_COUNT.default
<span class="o">[</span>INFO] <span class="o">[</span>creator] Spring Cloud Bindings 1.7.0: Contributing to layer
<span class="o">[</span>INFO] <span class="o">[</span>creator] Downloading from https://repo.spring.io/release/org/springframework/cloud/spring-cloud-bindings/1.7.0/spring-cloud-bindings-1.7.0.jar
<span class="o">[</span>INFO] <span class="o">[</span>creator] Verifying checksum
<span class="o">[</span>INFO] <span class="o">[</span>creator] Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
<span class="o">[</span>INFO] <span class="o">[</span>creator] 4 application slices
<span class="o">[</span>INFO] <span class="o">[</span>creator] Image labels:
<span class="o">[</span>INFO] <span class="o">[</span>creator] org.opencontainers.image.title
<span class="o">[</span>INFO] <span class="o">[</span>creator] org.opencontainers.image.version
<span class="o">[</span>INFO] <span class="o">[</span>creator] org.springframework.boot.spring-configuration-metadata.json
<span class="o">[</span>INFO] <span class="o">[</span>creator] org.springframework.boot.version
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="o">===></span> EXPORTING
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/ca-certificates:helper'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/bellsoft-liberica:helper'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/bellsoft-liberica:java-security-properties'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/bellsoft-liberica:jre'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/bellsoft-liberica:jvmkill'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/executable-jar:class-path'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/spring-boot:helper'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/spring-boot:spring-cloud-bindings'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'paketo-buildpacks/spring-boot:web-application-type'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding 5/5 app layer<span class="o">(</span>s<span class="o">)</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'launcher'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'config'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding layer <span class="s1">'process-types'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding label <span class="s1">'io.buildpacks.lifecycle.metadata'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding label <span class="s1">'io.buildpacks.build.metadata'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding label <span class="s1">'io.buildpacks.project.metadata'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding label <span class="s1">'org.opencontainers.image.title'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding label <span class="s1">'org.opencontainers.image.version'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding label <span class="s1">'org.springframework.boot.spring-configuration-metadata.json'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Adding label <span class="s1">'org.springframework.boot.version'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] Setting default process <span class="nb">type</span> <span class="s1">'web'</span>
<span class="o">[</span>INFO] <span class="o">[</span>creator] <span class="k">***</span> Images <span class="o">(</span>54bf7c7f30b1<span class="o">)</span>:
<span class="o">[</span>INFO] <span class="o">[</span>creator] docker.io/library/buildpacks-demo:0.0.1-SNAPSHOT
<span class="o">[</span>INFO]
<span class="o">[</span>INFO] Successfully built image <span class="s1">'docker.io/library/buildpacks-demo:0.0.1-SNAPSHOT'</span>
<span class="o">[</span>INFO]
<span class="o">[</span>INFO] <span class="nt">------------------------------------------------------------------------</span>
<span class="o">[</span>INFO] BUILD SUCCESS
<span class="o">[</span>INFO] <span class="nt">------------------------------------------------------------------------</span>
<span class="o">[</span>INFO] Total <span class="nb">time</span>: 29.156 s
<span class="o">[</span>INFO] Finished at: 2020-12-25T19:10:19+01:00
<span class="o">[</span>INFO] <span class="nt">------------------------------------------------------------------------</span>
</code></pre></div></div>
<p>Now we can run our application using:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">--rm</span> <span class="nt">-p</span> 8080:8080 buildpacks-demo:0.0.1-SNAPSHOT
<span class="nv">$ </span>http :8080
Hello, Buildpacker!
</code></pre></div></div>
<p>With <code class="language-plaintext highlighter-rouge">pack</code> is possible to <code class="language-plaintext highlighter-rouge">rebase</code> the image to a version pinned run image. This is useful for example when there is a security issue with the run image and with this command we can change only that OS layer in the image.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pack rebase buildpacks-demo:latest <span class="nt">--run-image</span> paketobuildpacks/run:1.0.11-base-cnb
1.0.11-base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:f393fa2927a2619a10fc09bb109f822d20df909c10fed4ce3c36fad313ea18e3
Status: Image is up to <span class="nb">date </span><span class="k">for </span>paketobuildpacks/run:1.0.11-base-cnb
Rebasing buildpacks-demo:latest on run image paketobuildpacks/run:1.0.11-base-cnb
<span class="k">***</span> Images <span class="o">(</span>7c8abb9b40d2<span class="o">)</span>:
buildpacks-demo:latest
Rebased Image: 7c8abb9b40d2f50b556efa003642ca5d00a24ec76fb91efc82c52d37887401a5
Successfully rebased image buildpacks-demo:latest
</code></pre></div></div>
<p>There is another tool <code class="language-plaintext highlighter-rouge">kpack</code> which running as a service on Kubernetes could do this <code class="language-plaintext highlighter-rouge">rebase</code> on all of your affected images, but that is a topic for another blog post :)</p>Cloud Native Buildpacks (CNB) is a specification to transform your application source code into an OCI image. It was initiated by Pivotal and Heroku in 2018, and recently moved from Sandbox to Incubation by the CNCF. If you are interested in how it compares to other popular alternatives like Jib or s2i head over to https://buildpacks.io/features/Getting started with kustomize2020-10-10T00:00:00+00:002020-10-10T00:00:00+00:00http://zoltanaltfatter.com/2020/10/10/getting-started-with-kustomize<p><a href="https://kustomize.io/">Kustomize</a> is a command line tool that lets you customize application configuration in a template free way.
In Kubernetes world the declarative way is the recommended approach to create the resources. However, it is difficult to use only <code class="language-plaintext highlighter-rouge">kubectl</code>
to follow the declarative way, another tools are required like, like <a href="https://helm.sh/">Helm</a>, <a href="https://github.com/deepmind/kapitan">Kapitan</a>, <a href="https://github.com/jimmycuadra/ktmpl">ktmpl</a>.
The full list of these tools you can find <a href="https://docs.google.com/spreadsheets/d/1FCgqz1Ci7_VCz_wdh8vBitZ3giBtac_H8SBw4uxnrsE/edit#gid=0">here</a></p>
<p>The drawback of these tools is that you have to learn new complicated DSLs, they use templating which can only override parameterized config, they provide multiple features
like package or dependency management, come with dashboards which live in your cluster, they allow to manage the lifecycle of specific version when to rollback specific version, and come with customization features too, when you deploy to different environments</p>
<p><code class="language-plaintext highlighter-rouge">Kustomize</code> instead is focusing only on the <code class="language-plaintext highlighter-rouge">customization</code> domain and allows you to tailor your YAML files to a specific environment.
It is using <code class="language-plaintext highlighter-rouge">overlay</code> approach and exposes and teaches the native k8s APIs, not trying to hide them. This makes sure, that the user gets a deeper understanding about Kubernetes.</p>
<p><code class="language-plaintext highlighter-rouge">Kustomize</code> is available as standalone executable (<code class="language-plaintext highlighter-rouge">kustomize</code>) and since 1.14 is part of <code class="language-plaintext highlighter-rouge">kubectl</code> using the <code class="language-plaintext highlighter-rouge">kubectl apply</code> with <code class="language-plaintext highlighter-rouge">-k</code> flag.
In this blog post we will use the standalone executable, which can be easily installed following the <a href="https://kubernetes-sigs.github.io/kustomize/installation/">guide</a> specific to your platform.</p>
<p>For Mac users is straightforward:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>brew <span class="nb">install </span>kustomize
</code></pre></div></div>
<p>In this article we assume you have already a <code class="language-plaintext highlighter-rouge">Docker</code> environment and <code class="language-plaintext highlighter-rouge">kubectl</code> is already installed. To create a Kubernetes cluster we use <a href="https://kind.sigs.k8s.io/docs/user/quick-start/">kind</a>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kind create cluster <span class="nt">--config</span><span class="o">=</span>kind-cluster-config.yaml
</code></pre></div></div>
<p>where the <code class="language-plaintext highlighter-rouge">kind-cluster-config.yaml</code> specifies a 3 node cluster.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># three node (two workers) cluster config</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Cluster</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">kind.x-k8s.io/v1alpha4</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">my-cluster</span>
<span class="na">nodes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">control-plane</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">worker</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">worker</span>
</code></pre></div></div>
<p>The command also creates a <code class="language-plaintext highlighter-rouge">Kubernetes</code> <code class="language-plaintext highlighter-rouge">context</code> and switches to it. <code class="language-plaintext highlighter-rouge">Kind</code> is a great tool to run multiple-node Kubernetes clusters locally. With <code class="language-plaintext highlighter-rouge">kind get clusters</code> we can query the created clusters.</p>
<p>With <code class="language-plaintext highlighter-rouge">docker ps</code> you can view the Kubernetes nodes</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
067290147e33 kindest/node:v1.19.1 "/usr/local/bin/entr…" 4 days ago Up 31 minutes my-cluster-worker
c1ab152ccf6e kindest/node:v1.19.1 "/usr/local/bin/entr…" 4 days ago Up 31 minutes 127.0.0.1:61807->6443/tcp my-cluster-control-plane
9d64cefcd1e7 kindest/node:v1.19.1 "/usr/local/bin/entr…" 4 days ago Up 31 minutes my-cluster-worker2
</code></pre></div></div>
<p>Next, we are going to use the <a href="https://github.com/altfatterz/kustomize-demo">kustomize-demo</a> application which is very simple Spring Boot application in order to showcase <code class="language-plaintext highlighter-rouge">kustomize</code> features.
After cloning the repository and building the example we have a docker image <code class="language-plaintext highlighter-rouge">kustomize-demo:0.0.1-SNAPSHOT</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git clone https://github.com/altfatterz/kustomize-demo
$ cd kustomize-demo
$ mvn clean package
</code></pre></div></div>
<p>Next we need to load the created Docker image into our Kubernetes cluster.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kind load docker-image kustomize-demo:0.0.1-SNAPSHOT <span class="nt">--name</span> my-cluster
altfatterz@zoltan-altfatter-zrh:~/projects/demos/kustomize-demo|master⚡ ⇒ kind load docker-image kustomize-demo:0.0.1-SNAPSHOT <span class="nt">--name</span> my-cluster
Image: <span class="s2">"kustomize-demo:0.0.1-SNAPSHOT"</span> with ID <span class="s2">"sha256:adb9bc431439dea21dc31155587da79183bda9f8636d80127d08d0dbdce6d4c7"</span> not yet present on node <span class="s2">"my-cluster-worker"</span>, loading...
Image: <span class="s2">"kustomize-demo:0.0.1-SNAPSHOT"</span> with ID <span class="s2">"sha256:adb9bc431439dea21dc31155587da79183bda9f8636d80127d08d0dbdce6d4c7"</span> not yet present on node <span class="s2">"my-cluster-control-plane"</span>, loading...
Image: <span class="s2">"kustomize-demo:0.0.1-SNAPSHOT"</span> with ID <span class="s2">"sha256:adb9bc431439dea21dc31155587da79183bda9f8636d80127d08d0dbdce6d4c7"</span> not yet present on node <span class="s2">"my-cluster-worker2"</span>, loading...
</code></pre></div></div>
<p>We can verify that the image is present on Kubernetes nodes using</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker exec -it 067290147e33 crictl images | grep kustomize-demo
docker.io/library/kustomize-demo 0.0.1-SNAPSHOT adb9bc431439d 259MB
</code></pre></div></div>
<p>Finally, we create two namespaces where we will deploy the application with different configurations:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl create namespace dev
<span class="nv">$ </span>kubectl create namespace prod
</code></pre></div></div>
<p>In the <code class="language-plaintext highlighter-rouge">kustomize-demo/ops</code> folder we can find the traditional way of declarative Kubernetes configuration duplicating the resource definitions for <code class="language-plaintext highlighter-rouge">dev</code> and <code class="language-plaintext highlighter-rouge">prod</code> environments.</p>
<p><code class="language-plaintext highlighter-rouge">Kustomize</code> allows to specify the resource definitions without duplicating common elements. It does this Kubernetes way, using to use Custom Resource Definitions (CRDs) to configure the differences, rather than variable-replacement. <br />
We move the common yaml configuration into a <code class="language-plaintext highlighter-rouge">base</code> directory and create two <code class="language-plaintext highlighter-rouge">overlays</code> representing <code class="language-plaintext highlighter-rouge">dev</code> and <code class="language-plaintext highlighter-rouge">prod</code> environments using the following directory structure:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── base
│ ├── deployment.yaml
│ └── service.yaml
└── overlays
├── dev
│ └── kustomization.yaml
└── prod
└── kustomization.yaml
</code></pre></div></div>
<p>where the <code class="language-plaintext highlighter-rouge">dev/kustomization.yaml</code> content is</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/deployment.yaml
- ../../base/service.yaml
namespace: dev
</code></pre></div></div>
<p>In order to see the yaml generated for the <code class="language-plaintext highlighter-rouge">dev</code> environment we can use:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kustomize build <span class="nt">--load_restrictor</span> none k8s/overlays/dev
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">--load_restrictor</code> flag set to <code class="language-plaintext highlighter-rouge">none</code>, allows that customizations may load files from outside their root.</p>
<p>To create the resource in the dev namespace then we can pipe the output of the <code class="language-plaintext highlighter-rouge">kustomize</code> command to the <code class="language-plaintext highlighter-rouge">kubectl apply</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kustomize build <span class="nt">--load_restrictor</span> none k8s/overlays/dev | kubectl apply <span class="nt">-f</span> -
service/kustomize-demo-service created
deployment.apps/kustomize-demo-deployment created
</code></pre></div></div>
<p>To delete the resources we can use:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kustomize build <span class="nt">--load_restrictor</span> none k8s/overlays/dev | kubectl delete <span class="nt">-f</span> -
</code></pre></div></div>
<p>In production environment is good approach to add a prefix like <code class="language-plaintext highlighter-rouge">prod-</code> to all product resource names in order to avoid
modifying or deleting these resources by mistake. With <code class="language-plaintext highlighter-rouge">Kustomize</code> we can easily achieve this adding the following snippet
to <code class="language-plaintext highlighter-rouge">prod/kustomization.yaml</code> configuration file</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">namePrefix</span><span class="pi">:</span> <span class="s">prod-</span>
</code></pre></div></div>
<p>We want also that resources in production environment to have certain labels so that we can query them by label selector:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">commonLabels</span><span class="pi">:</span>
<span class="na">env</span><span class="pi">:</span> <span class="s">prod</span>
</code></pre></div></div>
<p>We also want to increase the replicas count in production environment (<code class="language-plaintext highlighter-rouge">prod/increase_replicas_patch.yaml</code>)</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kustomize-demo</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">replicas</span><span class="pi">:</span> <span class="s">3</span>
</code></pre></div></div>
<p>Another important configuration in production is to set the resource constraints (<code class="language-plaintext highlighter-rouge">prod/resource_constraints_patch.yaml</code>)</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kustomize-demo</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kustomize-demo</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">requests</span><span class="pi">:</span>
<span class="na">memory</span><span class="pi">:</span> <span class="s">512Mi</span>
<span class="na">cpu</span><span class="pi">:</span> <span class="s">256m</span>
<span class="na">limits</span><span class="pi">:</span>
<span class="na">memory</span><span class="pi">:</span> <span class="s">1Gi</span>
<span class="na">cpu</span><span class="pi">:</span> <span class="s">512m</span>
</code></pre></div></div>
<p>and health check configuration (<code class="language-plaintext highlighter-rouge">prod/health_check_patch.yaml</code>)</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kustomize-demo</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kustomize-demo</span>
<span class="na">readinessProbe</span><span class="pi">:</span>
<span class="na">httpGet</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">/actuator/health/readiness</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">8080</span>
<span class="na">initialDelaySeconds</span><span class="pi">:</span> <span class="m">5</span>
<span class="na">periodSeconds</span><span class="pi">:</span> <span class="m">5</span>
<span class="na">livenessProbe</span><span class="pi">:</span>
<span class="na">httpGet</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">/actuator/health/liveness</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">8080</span>
<span class="na">initialDelaySeconds</span><span class="pi">:</span> <span class="m">5</span>
<span class="na">periodSeconds</span><span class="pi">:</span> <span class="m">5</span>
</code></pre></div></div>
<p>We need to reference all these patches in <code class="language-plaintext highlighter-rouge">prod/kustomization.yaml</code> under the <code class="language-plaintext highlighter-rouge">patchesStrategicMerge</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">patchesStrategicMerge</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">resource_constraints_patch.yaml</span>
<span class="pi">-</span> <span class="s">environment_patch.yaml</span>
<span class="pi">-</span> <span class="s">health_check_patch.yaml</span>
<span class="pi">-</span> <span class="s">increase_replicas_patch.yaml</span>
</code></pre></div></div>
<p>To create the resources in <code class="language-plaintext highlighter-rouge">prod</code> environment we use</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kustomize build <span class="nt">--load_restrictor</span> none k8s/overlays/prod | kubectl apply <span class="nt">-f</span> -
</code></pre></div></div>
<p>Indeed, the pods where created</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods <span class="nt">-n</span> prod
NAME READY STATUS RESTARTS AGE
prod-kustomize-demo-548ccc8db9-dtc52 1/1 Running 0 25s
prod-kustomize-demo-548ccc8db9-m9xzb 1/1 Running 0 25s
prod-kustomize-demo-548ccc8db9-qjhxw 1/1 Running 0 25s
</code></pre></div></div>
<p>The source code for this blog post can be found here <a href="https://github.com/altfatterz/kustomize-demo">https://github.com/altfatterz/kustomize-demo</a></p>Kustomize is a command line tool that lets you customize application configuration in a template free way. In Kubernetes world the declarative way is the recommended approach to create the resources. However, it is difficult to use only kubectl to follow the declarative way, another tools are required like, like Helm, Kapitan, ktmpl. The full list of these tools you can find hereSemantic versioning with jgitver2020-04-10T00:00:00+00:002020-04-10T00:00:00+00:00http://zoltanaltfatter.com/2020/04/10/semantic-versioning-with-jgitver<p><a href="https://jgitver.github.io/">jgitver</a> is a very useful tool to automatically compute the version of your project leveraging the git history.
It does not pollute the project’s git history like the <a href="http://maven.apache.org/maven-release/maven-release-plugin/">maven release plugin</a></p>
<p><code class="language-plaintext highlighter-rouge">jgitver</code> has support for both Maven and Gradle. In this blog post we are going to use Gradle.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>plugins <span class="o">{</span>
<span class="nb">id</span> <span class="s1">'fr.brouillard.oss.gradle.jgitver'</span> version <span class="s1">'0.9.1'</span>
<span class="o">}</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">jgitver</code> comes with many good defaults which follow best practices and conventions which can be customised if needed.</p>
<p>It automatically registers a task <code class="language-plaintext highlighter-rouge">version</code>. When executing it on a non git repository then we get the following result:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>./gradlew version
Version: 0.0.0-NOT_A_GIT_REPOSITORY
</code></pre></div></div>
<p>And indeed after <code class="language-plaintext highlighter-rouge">./gradlew build</code> we have an <code class="language-plaintext highlighter-rouge"><artifact-id>.0.0.0-NOT_A_GIT_REPOSITORY.jar</code> in the <code class="language-plaintext highlighter-rouge">build/libs</code> folder.</p>
<p>Let’s make it a git repository with <code class="language-plaintext highlighter-rouge">git init</code> and verify that <code class="language-plaintext highlighter-rouge">./gradlew version</code> returns <code class="language-plaintext highlighter-rouge">0.0.0-EMPTY_GIT_REPOSITORY</code>.</p>
<p>Next, adding our fist commit we have the version <code class="language-plaintext highlighter-rouge">0.0.0-0</code>. When no tags can be found in the commit history, <code class="language-plaintext highlighter-rouge">jgitver</code> defaults to a virtual lightweight git tag <code class="language-plaintext highlighter-rouge">0.0.0</code> on the first commit. Any additional commits will increment the last number, <code class="language-plaintext highlighter-rouge">0.0.0-1</code>, <code class="language-plaintext highlighter-rouge">0.0.0-2</code>, etc.</p>
<p>The last number signals how far (how many commits) are we from the last lightweight git tag. (in the above case the <code class="language-plaintext highlighter-rouge">0.0.0</code> virtual lightweight tag)</p>
<h3 id="lightweight-git-tag">Lightweight git tag</h3>
<p>Let’s create a lightweight git tag and check the project version:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git tag 0.0.1
<span class="nv">$ </span>./gradlew version
Version: 0.0.1-0
</code></pre></div></div>
<h3 id="annotated-git-tag">Annotated git tag</h3>
<p>Let’s create an annotated git tag and check the project version:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git tag <span class="nt">-a</span> 0.1.0 <span class="nt">-m</span> <span class="s2">"first stable version"</span>
<span class="nv">$ </span>./gradlew version
Version: 0.1.0
</code></pre></div></div>
<p>As we see with annotated git tag the calculated version is the tag name. Any further commits create versions like <code class="language-plaintext highlighter-rouge">0.1.1-1</code>, <code class="language-plaintext highlighter-rouge">0.1.1-2</code>, etc. Beside incrementing the last number, it also moved the first part of the version from <code class="language-plaintext highlighter-rouge">0.1.0</code> (based on our exiting annotated git tag) to <code class="language-plaintext highlighter-rouge">0.1.1</code>.<br />
And again if we create another annotated git tag, be it <code class="language-plaintext highlighter-rouge">0.1.1</code> (minor fixes) or <code class="language-plaintext highlighter-rouge">0.2.0</code> (new features but still backwards compatible) or <code class="language-plaintext highlighter-rouge">1.0.0</code> (incompatible API change) then the calculated version will be always equal to the annotated git tag.</p>
<h3 id="branching">Branching</h3>
<p>Let’s say we are on the <code class="language-plaintext highlighter-rouge">0.2.0</code> annotated git tag from where we create our <code class="language-plaintext highlighter-rouge">awesome-feature</code> branch then the calculated project version will be the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git checkout <span class="nt">-b</span> awesome-feature
<span class="nv">$ </span>./gradle version
<span class="nv">$ </span>Version: 0.2.0-awesome_feature
</code></pre></div></div>
<p>Any further commits on the <code class="language-plaintext highlighter-rouge">awesome-feature</code> branch will produce the following versions: <code class="language-plaintext highlighter-rouge">0.2.1-1-awesome_feature</code>, <code class="language-plaintext highlighter-rouge">0.2.1-2-awesome_feature</code>, etc.</p>
<h3 id="expose-version">Expose version</h3>
<p>It comes handy when the service exposes its version number. Using Spring Boot this is very handy, we just need to include</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>springBoot <span class="o">{</span>
buildInfo<span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The above configuration makes sure that build information is exposed via the <code class="language-plaintext highlighter-rouge">/info</code> actuator endpoint.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>http :8080/actuator/info
<span class="o">{</span>
<span class="s2">"app"</span>: <span class="o">{</span>
<span class="s2">"description"</span>: <span class="s2">"jgitver calculates a project semver compatible version from a git repository"</span>,
<span class="s2">"name"</span>: <span class="s2">"jgitver-demo"</span>
<span class="o">}</span>,
<span class="s2">"build"</span>: <span class="o">{</span>
<span class="s2">"artifact"</span>: <span class="s2">"jgitver-demo"</span>,
<span class="s2">"group"</span>: <span class="s2">"com.example"</span>,
<span class="s2">"name"</span>: <span class="s2">"jgitver-demo"</span>,
<span class="s2">"time"</span>: <span class="s2">"2020-04-09T20:50:58.276Z"</span>,
<span class="s2">"version"</span>: <span class="s2">"0.0.2-1"</span>
<span class="o">}</span>,
<span class="s2">"git"</span>: <span class="o">{</span>
<span class="s2">"branch"</span>: <span class="s2">"master"</span>,
<span class="s2">"commit"</span>: <span class="o">{</span>
<span class="s2">"id"</span>: <span class="s2">"de5248a"</span>,
<span class="s2">"time"</span>: <span class="s2">"2020-03-07T21:46:14Z"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In the above example git information is also exposed using the <a href="https://plugins.gradle.org/plugin/com.gorylenko.gradle-git-properties">gradle-git-properties</a> plugin.</p>
<h3 id="docker">Docker</h3>
<p>Since nowadays the build artifact is a docker image we want to make sure the image is tagged with calculated project version.
Here we are using <a href="https://github.com/GoogleContainerTools/jib">Jib</a> to dockerize our Spring Boot application. If you are new to Jib have a look to my previous blog post <a href="https://zoltanaltfatter.com/2019/08/16/dockerizing-spring-boot-apps-with-jib/">https://zoltanaltfatter.com/2019/08/16/dockerizing-spring-boot-apps-with-jib/</a></p>
<p>To build to a Docker daemon, we can use</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>./gradlew jibDockerBuild
</code></pre></div></div>
<p>And indeed the docker image was created with the tag equal to the project’s version calculated by <code class="language-plaintext highlighter-rouge">jgitver</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker images | <span class="nb">grep </span>jgitver-demo
jgitver-demo 0.0.2-1 a5418196da55 2 minutes ago 143MB
</code></pre></div></div>
<h3 id="pipeline">Pipeline</h3>
<p>But we don’t create docker images locally, we do that with a build pipeline. We are going to use <a href="https://github.com/features/actions">GitHub Actions</a><br />
to build the artifact and publish it to <a href="https://github.com/features/packages">GitHub Packages</a></p>
<p>The workflow is <a href="https://help.github.com/en/actions/reference/events-that-trigger-workflows">triggered</a> on <code class="language-plaintext highlighter-rouge">push</code> event.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Build</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">push</span><span class="pi">:</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">build</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
</code></pre></div></div>
<p>We use standard actions for <a href="https://github.com/actions/checkout">checking out</a> the repository and <a href="https://github.com/actions/setup-java">setting up java version</a>.
The checkout action only fetches a single commit, but in order to <code class="language-plaintext highlighter-rouge">jgitver</code> to calculate the project version from git history we need all the tags.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout latest code</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">fetch-depth</span><span class="pi">:</span> <span class="m">0</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Fetch all tags</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">git fetch --depth=1 origin +refs/tags/*:refs/tags/*</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up JDK </span><span class="m">1.8</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/setup-java@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">java-version</span><span class="pi">:</span> <span class="m">1.8</span>
</code></pre></div></div>
<p>Then after restoring the Gradle build cache we build the project.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup build cache</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/cache@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">~/.gradle/caches</span>
<span class="na">key</span><span class="pi">:</span> <span class="s">${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}</span>
<span class="na">restore-keys</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">${{ runner.os }}-gradle-</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build with Gradle</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">./gradlew build</span>
</code></pre></div></div>
<p>In the next step we build the docker image and push it to <code class="language-plaintext highlighter-rouge">GitHub Packages</code> registry.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Compute version</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">compute_version</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo "::set-output name=version::$(./gradlew version | grep Version | awk '{ print $2 }')"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build and Upload Docker image</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo ${{ secrets.GITHUB_TOKEN }}</span>
<span class="s">./gradlew jib \</span>
<span class="s">-Djib.to.image=docker.pkg.github.com/altfatterz/jgitver-demo/jgitver-demo:${{ steps.compute_version.outputs.version }} \</span>
<span class="s">-Djib.to.auth.username=altfatterz \</span>
<span class="s">-Djib.to.auth.password=${{ secrets.GITHUB_TOKEN }}</span>
</code></pre></div></div>
<p>As you can see our pipeline is pretty fast, it took only 41 seconds.</p>
<p><img src="/images/2020-04-10/github-actions.png" alt="graphql-actions" /></p>
<p>And the produced docker image is in the GitHub Package Registry:</p>
<p><img src="/images/2020-04-10/github-packages.png" alt="github-packages" /></p>
<p>If we click on the version we can see the details how to pull the image and also some download statistics and previous versions.<br />
You cannot remove packages from GitHub Package Registry.</p>
<p><img src="/images/2020-04-10/github-packages-details.png" alt="github-packages-details" /></p>
<p>The example source code can be found here <a href="https://github.com/altfatterz/jgitver-demo">https://github.com/altfatterz/jgitver-demo</a></p>jgitver is a very useful tool to automatically compute the version of your project leveraging the git history. It does not pollute the project’s git history like the maven release plugin jgitver has support for both Maven and Gradle. In this blog post we are going to use Gradle.GraphQL Server and Client2020-01-25T00:00:00+00:002020-01-25T00:00:00+00:00http://zoltanaltfatter.com/2020/01/25/graphql-server-and-client<p><a href="https://graphql.github.io/">GraphQL</a> is a query language for your API. It is also a specification which determines the validity schema on the server.
It is strongly typed, the schema defines the GraphQL API’s type system. It defines the possible objects that a client can access. A client can find information
about the schema via introspection.</p>
<p>In this blog post we are going to develop a simple GraphQL server and client in Kotlin/Java. First we define the GraphQL schema
with the following root types.</p>
<div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span><span class="w"> </span><span class="n">Query</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">bookById</span><span class="p">(</span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="nb">ID</span><span class="p">!):</span><span class="w"> </span><span class="n">Book</span><span class="w">
</span><span class="n">books</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">Book</span><span class="p">!]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="k">type</span><span class="w"> </span><span class="n">Mutation</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">addReview</span><span class="p">(</span><span class="n">input</span><span class="p">:</span><span class="w"> </span><span class="n">AddReviewInput</span><span class="p">!):</span><span class="w"> </span><span class="nb">Boolean</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>We will be able to retrieve books by their id, retrieve all books and to add a review to a book. All types in GraphQL
are nullable by default, with ! we can declare that the field is non nullable.</p>
<div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span><span class="w"> </span><span class="n">Book</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="nb">ID</span><span class="p">!</span><span class="w">
</span><span class="n">title</span><span class="p">:</span><span class="w"> </span><span class="nb">String</span><span class="p">!</span><span class="w">
</span><span class="n">pageCount</span><span class="p">:</span><span class="w"> </span><span class="nb">Int</span><span class="w">
</span><span class="n">publishedDate</span><span class="p">:</span><span class="w"> </span><span class="n">Date</span><span class="w">
</span><span class="n">author</span><span class="p">:</span><span class="w"> </span><span class="n">Author</span><span class="p">!</span><span class="w">
</span><span class="n">reviews</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">Review</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="k">type</span><span class="w"> </span><span class="n">Author</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">firstName</span><span class="p">:</span><span class="w"> </span><span class="nb">String</span><span class="p">!</span><span class="w">
</span><span class="n">lastName</span><span class="p">:</span><span class="w"> </span><span class="nb">String</span><span class="p">!</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="k">type</span><span class="w"> </span><span class="n">Review</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">stars</span><span class="p">:</span><span class="w"> </span><span class="nb">Int</span><span class="p">!</span><span class="w">
</span><span class="n">comment</span><span class="p">:</span><span class="w"> </span><span class="nb">String</span><span class="p">!</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="k">input</span><span class="w"> </span><span class="n">AddReviewInput</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">bookId</span><span class="p">:</span><span class="w"> </span><span class="nb">ID</span><span class="p">!</span><span class="w">
</span><span class="n">stars</span><span class="p">:</span><span class="w"> </span><span class="nb">Int</span><span class="p">!</span><span class="w">
</span><span class="n">comment</span><span class="p">:</span><span class="w"> </span><span class="nb">String</span><span class="p">!</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="k">scalar</span><span class="w"> </span><span class="n">Date</span><span class="w">
</span></code></pre></div></div>
<h2 id="server">Server</h2>
<p>For the GraphQL Server we use the Java implementation <a href="https://www.graphql-java.com/">https://www.graphql-java.com/</a>
For the <code class="language-plaintext highlighter-rouge">bookById</code>, <code class="language-plaintext highlighter-rouge">books</code> and <code class="language-plaintext highlighter-rouge">addReview</code> fields we define a <code class="language-plaintext highlighter-rouge">DataFetcher</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Configuration</span>
<span class="kd">class</span> <span class="nc">GraphqlConfiguration</span><span class="p">(</span><span class="k">private</span> <span class="kd">val</span> <span class="py">graphqlDataFetchers</span><span class="p">:</span> <span class="nc">GraphqlDataFetchers</span><span class="p">,</span>
<span class="nd">@Value</span><span class="p">(</span><span class="s">"classpath:graphql/*.graphqls"</span><span class="p">)</span>
<span class="k">private</span> <span class="kd">val</span> <span class="py">resources</span><span class="p">:</span> <span class="nc">Array</span><span class="p"><</span><span class="nc">Resource</span><span class="p">>)</span> <span class="p">{</span>
<span class="nd">@Bean</span>
<span class="k">fun</span> <span class="nf">graphQL</span><span class="p">():</span> <span class="nc">GraphQL</span><span class="p">?</span> <span class="p">{</span>
<span class="k">return</span> <span class="nc">GraphQL</span><span class="p">.</span><span class="nf">newGraphQL</span><span class="p">(</span><span class="nf">buildGraphQLSchema</span><span class="p">()).</span><span class="nf">build</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">fun</span> <span class="nf">buildGraphQLSchema</span><span class="p">():</span> <span class="nc">GraphQLSchema</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">schemaParser</span> <span class="p">=</span> <span class="nc">SchemaParser</span><span class="p">()</span>
<span class="kd">val</span> <span class="py">typeDefinitionRegistry</span> <span class="p">=</span> <span class="nc">TypeDefinitionRegistry</span><span class="p">()</span>
<span class="k">for</span> <span class="p">(</span><span class="n">resource</span> <span class="k">in</span> <span class="n">resources</span><span class="p">)</span> <span class="p">{</span>
<span class="n">typeDefinitionRegistry</span><span class="p">.</span><span class="nf">merge</span><span class="p">(</span><span class="n">schemaParser</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">resource</span><span class="p">.</span><span class="n">file</span><span class="p">))</span>
<span class="p">}</span>
<span class="kd">val</span> <span class="py">schemaGenerator</span> <span class="p">=</span> <span class="nc">SchemaGenerator</span><span class="p">()</span>
<span class="k">return</span> <span class="n">schemaGenerator</span><span class="p">.</span><span class="nf">makeExecutableSchema</span><span class="p">(</span><span class="n">typeDefinitionRegistry</span><span class="p">,</span> <span class="nf">buildRuntimeWiring</span><span class="p">())</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">fun</span> <span class="nf">buildRuntimeWiring</span><span class="p">():</span> <span class="nc">RuntimeWiring</span> <span class="p">{</span>
<span class="k">return</span> <span class="nc">RuntimeWiring</span><span class="p">.</span><span class="nf">newRuntimeWiring</span><span class="p">()</span>
<span class="p">.</span><span class="nf">scalar</span><span class="p">(</span><span class="nc">ExtendedScalars</span><span class="p">.</span><span class="nc">Date</span><span class="p">)</span>
<span class="p">.</span><span class="nf">type</span><span class="p">(</span><span class="nc">TypeRuntimeWiring</span><span class="p">.</span><span class="nf">newTypeWiring</span><span class="p">(</span><span class="s">"Query"</span><span class="p">)</span>
<span class="p">.</span><span class="nf">dataFetcher</span><span class="p">(</span><span class="s">"bookById"</span><span class="p">,</span> <span class="n">graphqlDataFetchers</span><span class="p">.</span><span class="n">bookByIdDataFetcher</span><span class="p">)</span>
<span class="p">.</span><span class="nf">dataFetcher</span><span class="p">(</span><span class="s">"books"</span><span class="p">,</span> <span class="n">graphqlDataFetchers</span><span class="p">.</span><span class="n">books</span><span class="p">))</span>
<span class="p">.</span><span class="nf">type</span><span class="p">(</span><span class="nc">TypeRuntimeWiring</span><span class="p">.</span><span class="nf">newTypeWiring</span><span class="p">(</span><span class="s">"Mutation"</span><span class="p">)</span>
<span class="p">.</span><span class="nf">dataFetcher</span><span class="p">(</span><span class="s">"addReview"</span><span class="p">,</span> <span class="n">graphqlDataFetchers</span><span class="p">.</span><span class="nf">addReview</span><span class="p">()))</span>
<span class="p">.</span><span class="nf">build</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We can put a GraphQL API on any persistence layer. In this example we use <a href="https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#reference">Spring Data JPA</a>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Component</span>
<span class="kd">class</span> <span class="nc">GraphqlDataFetchers</span><span class="p">(</span><span class="k">private</span> <span class="kd">val</span> <span class="py">bookRepository</span><span class="p">:</span> <span class="nc">BookRepository</span><span class="p">,</span>
<span class="k">private</span> <span class="kd">val</span> <span class="py">reviewRepository</span><span class="p">:</span> <span class="nc">ReviewRepository</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">val</span> <span class="py">bookByIdDataFetcher</span><span class="p">:</span> <span class="nc">DataFetcher</span><span class="p"><</span><span class="nc">Book</span><span class="err">?</span><span class="p">></span>
<span class="k">get</span><span class="p">()</span> <span class="p">=</span> <span class="nc">DataFetcher</span> <span class="p">{</span> <span class="n">env</span><span class="p">:</span> <span class="nc">DataFetchingEnvironment</span> <span class="p">-></span>
<span class="kd">val</span> <span class="py">id</span> <span class="p">=</span> <span class="n">env</span><span class="p">.</span><span class="n">getArgument</span><span class="p"><</span><span class="nc">String</span><span class="p">>(</span><span class="s">"id"</span><span class="p">)</span>
<span class="n">bookRepository</span><span class="p">.</span><span class="nf">findById</span><span class="p">(</span><span class="nc">UUID</span><span class="p">.</span><span class="nf">fromString</span><span class="p">(</span><span class="n">id</span><span class="p">))</span>
<span class="p">}</span>
<span class="o">..</span><span class="p">.</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We define repositories for the following entities: <code class="language-plaintext highlighter-rouge">Book</code>, <code class="language-plaintext highlighter-rouge">Author</code> and <code class="language-plaintext highlighter-rouge">Review</code>.</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">interface</span> <span class="nc">BookRepository</span> <span class="p">:</span> <span class="nc">Repository</span><span class="p"><</span><span class="nc">Book</span><span class="p">,</span> <span class="nc">UUID</span><span class="p">></span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">save</span><span class="p">(</span><span class="n">book</span><span class="p">:</span> <span class="nc">Book</span><span class="p">):</span> <span class="nc">Book</span>
<span class="k">fun</span> <span class="nf">findById</span><span class="p">(</span><span class="n">id</span><span class="p">:</span> <span class="nc">UUID</span><span class="p">):</span> <span class="nc">Book</span><span class="p">?</span>
<span class="nd">@EntityGraph</span><span class="p">(</span><span class="n">value</span> <span class="p">=</span> <span class="s">"books"</span><span class="p">)</span> <span class="k">fun</span> <span class="nf">findAll</span><span class="p">()</span> <span class="p">:</span> <span class="nc">List</span><span class="p"><</span><span class="nc">Book</span><span class="p">></span>
<span class="p">}</span>
<span class="kd">interface</span> <span class="nc">AuthorRepository</span> <span class="p">:</span> <span class="nc">Repository</span><span class="p"><</span><span class="nc">Author</span><span class="p">,</span> <span class="nc">UUID</span><span class="p">></span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">save</span><span class="p">(</span><span class="n">author</span><span class="p">:</span> <span class="nc">Author</span><span class="p">):</span> <span class="nc">Author</span>
<span class="p">}</span>
<span class="kd">interface</span> <span class="nc">ReviewRepository</span> <span class="p">:</span> <span class="nc">Repository</span><span class="p"><</span><span class="nc">Review</span><span class="p">,</span> <span class="nc">UUID</span><span class="p">></span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">save</span><span class="p">(</span><span class="n">review</span><span class="p">:</span> <span class="nc">Review</span><span class="p">):</span> <span class="nc">Review</span>
<span class="p">}</span>
</code></pre></div></div>
<p>where the entities have the following definitions:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Entity</span>
<span class="nd">@Table</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"books"</span><span class="p">)</span>
<span class="nd">@NamedEntityGraph</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"books"</span><span class="p">,</span> <span class="n">attributeNodes</span> <span class="p">=</span> <span class="p">[</span><span class="nc">NamedAttributeNode</span><span class="p">(</span><span class="s">"author"</span><span class="p">),</span><span class="nc">NamedAttributeNode</span><span class="p">(</span><span class="s">"reviews"</span><span class="p">)])</span>
<span class="kd">class</span> <span class="nc">Book</span><span class="p">(</span>
<span class="kd">var</span> <span class="py">title</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="nd">@Column</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"page_count"</span><span class="p">)</span> <span class="kd">var</span> <span class="py">pageCount</span><span class="p">:</span> <span class="nc">Int</span><span class="p">,</span>
<span class="nd">@Column</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"published_date"</span><span class="p">)</span> <span class="kd">var</span> <span class="py">publishedDate</span><span class="p">:</span> <span class="nc">LocalDate</span><span class="p">,</span>
<span class="nd">@ManyToOne</span><span class="p">(</span><span class="n">fetch</span> <span class="p">=</span> <span class="nc">FetchType</span><span class="p">.</span><span class="nc">LAZY</span><span class="p">)</span> <span class="kd">var</span> <span class="py">author</span><span class="p">:</span> <span class="nc">Author</span><span class="p">,</span>
<span class="nd">@OneToMany</span> <span class="nd">@JoinColumn</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"book_id"</span><span class="p">)</span> <span class="kd">var</span> <span class="py">reviews</span><span class="p">:</span> <span class="nc">MutableSet</span><span class="p"><</span><span class="nc">Review</span><span class="p">></span> <span class="p">=</span> <span class="nc">HashSet</span><span class="p">(),</span>
<span class="nd">@Id</span> <span class="nd">@GeneratedValue</span><span class="p">(</span><span class="n">generator</span> <span class="p">=</span> <span class="s">"UUID"</span><span class="p">)</span> <span class="nd">@Type</span><span class="p">(</span><span class="n">type</span> <span class="p">=</span> <span class="s">"uuid-char"</span><span class="p">)</span> <span class="kd">var</span> <span class="py">id</span><span class="p">:</span> <span class="nc">UUID</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">fun</span> <span class="nf">addReview</span><span class="p">(</span><span class="n">review</span><span class="p">:</span> <span class="nc">Review</span><span class="p">)</span> <span class="p">{</span>
<span class="n">reviews</span><span class="p">.</span><span class="nf">add</span><span class="p">(</span><span class="n">review</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nd">@Entity</span>
<span class="nd">@Table</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"authors"</span><span class="p">)</span>
<span class="kd">class</span> <span class="nc">Author</span><span class="p">(</span>
<span class="nd">@Column</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"first_name"</span><span class="p">)</span>
<span class="kd">var</span> <span class="py">firstName</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="nd">@Column</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"last_name"</span><span class="p">)</span>
<span class="kd">var</span> <span class="py">lastName</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="nd">@Id</span> <span class="nd">@GeneratedValue</span><span class="p">(</span><span class="n">generator</span> <span class="p">=</span> <span class="s">"UUID"</span><span class="p">)</span> <span class="nd">@Type</span><span class="p">(</span><span class="n">type</span> <span class="p">=</span> <span class="s">"uuid-char"</span><span class="p">)</span> <span class="kd">var</span> <span class="py">id</span><span class="p">:</span> <span class="nc">UUID</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span><span class="p">)</span>
<span class="nd">@Entity</span>
<span class="nd">@Table</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"reviews"</span><span class="p">)</span>
<span class="kd">class</span> <span class="nc">Review</span><span class="p">(</span>
<span class="kd">var</span> <span class="py">stars</span><span class="p">:</span> <span class="nc">Int</span><span class="p">,</span>
<span class="kd">var</span> <span class="py">comment</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span>
<span class="nd">@Id</span> <span class="nd">@GeneratedValue</span><span class="p">(</span><span class="n">generator</span> <span class="p">=</span> <span class="s">"UUID"</span><span class="p">)</span> <span class="nd">@Type</span><span class="p">(</span><span class="n">type</span> <span class="p">=</span> <span class="s">"uuid-char"</span><span class="p">)</span> <span class="kd">var</span> <span class="py">id</span><span class="p">:</span> <span class="nc">UUID</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span>
<span class="p">)</span>
</code></pre></div></div>
<p>A simple GraphQL query</p>
<div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">query</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">books</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">title</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>could return a response like</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"data"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"books"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The Da Vinci Code"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Memoirs of a Geisha"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The Alchemist"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Behind the scene it will fire the following SQL query to the database</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Hibernate</span><span class="p">:</span>
<span class="k">select</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id1_1_</span><span class="p">,</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">author_id</span> <span class="k">as</span> <span class="n">author_i4_1_</span><span class="p">,</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">page_count</span> <span class="k">as</span> <span class="n">page_cou2_1_</span><span class="p">,</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">title</span> <span class="k">as</span> <span class="n">title3_1_</span>
<span class="k">from</span>
<span class="n">book</span> <span class="n">book0_</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">@NamedEntityGraph</code> and <code class="language-plaintext highlighter-rouge">@EntityGraph</code> make sure to avoid the N+1 query problem. For the query</p>
<div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">query</span><span class="w"> </span><span class="n">Books</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">books</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">title</span><span class="w">
</span><span class="n">author</span><span class="p">{</span><span class="w">
</span><span class="n">firstName</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">reviews</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">stars</span><span class="w">
</span><span class="n">comment</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>instead of having many queries on the <code class="language-plaintext highlighter-rouge">authors</code> and <code class="language-plaintext highlighter-rouge">reviews</code> tables we have only one query:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id1_1_0_</span><span class="p">,</span>
<span class="n">author1_</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id1_0_1_</span><span class="p">,</span>
<span class="n">reviews2_</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id1_2_2_</span><span class="p">,</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">author_id</span> <span class="k">as</span> <span class="n">author_i5_1_0_</span><span class="p">,</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">page_count</span> <span class="k">as</span> <span class="n">page_cou2_1_0_</span><span class="p">,</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">published_date</span> <span class="k">as</span> <span class="n">publishe3_1_0_</span><span class="p">,</span>
<span class="n">book0_</span><span class="p">.</span><span class="n">title</span> <span class="k">as</span> <span class="n">title4_1_0_</span><span class="p">,</span>
<span class="n">author1_</span><span class="p">.</span><span class="n">first_name</span> <span class="k">as</span> <span class="n">first_na2_0_1_</span><span class="p">,</span>
<span class="n">author1_</span><span class="p">.</span><span class="n">last_name</span> <span class="k">as</span> <span class="n">last_nam3_0_1_</span><span class="p">,</span>
<span class="n">reviews2_</span><span class="p">.</span><span class="k">comment</span> <span class="k">as</span> <span class="n">comment2_2_2_</span><span class="p">,</span>
<span class="n">reviews2_</span><span class="p">.</span><span class="n">stars</span> <span class="k">as</span> <span class="n">stars3_2_2_</span><span class="p">,</span>
<span class="n">reviews2_</span><span class="p">.</span><span class="n">book_id</span> <span class="k">as</span> <span class="n">book_id4_2_0__</span><span class="p">,</span>
<span class="n">reviews2_</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id1_2_0__</span>
<span class="k">from</span>
<span class="n">books</span> <span class="n">book0_</span>
<span class="k">left</span> <span class="k">outer</span> <span class="k">join</span>
<span class="n">authors</span> <span class="n">author1_</span>
<span class="k">on</span> <span class="n">book0_</span><span class="p">.</span><span class="n">author_id</span><span class="o">=</span><span class="n">author1_</span><span class="p">.</span><span class="n">id</span>
<span class="k">left</span> <span class="k">outer</span> <span class="k">join</span>
<span class="n">reviews</span> <span class="n">reviews2_</span>
<span class="k">on</span> <span class="n">book0_</span><span class="p">.</span><span class="n">id</span><span class="o">=</span><span class="n">reviews2_</span><span class="p">.</span><span class="n">book_id</span>
</code></pre></div></div>
<p>One more thing to note here that we didn’t define a controller with <code class="language-plaintext highlighter-rouge">/graphql</code> endpoint. A <code class="language-plaintext highlighter-rouge">GraphQLController</code> is defined by using the following dependency.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dependencies <span class="o">{</span>
implementation<span class="o">(</span><span class="s2">"com.graphql-java:graphql-java-spring-boot-starter-webmvc:1.0"</span><span class="o">)</span>
...
<span class="o">}</span>
</code></pre></div></div>
<h2 id="client">Client</h2>
<p>With curl this is how we make a simple GraphQL request:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl <span class="nt">-i</span> <span class="nt">-X</span> POST http://localhost:8080/graphql <span class="nt">-H</span> <span class="s2">"Content-Type:application/json"</span> <span class="nt">-d</span> <span class="s1">'{"query":"query { books {title} } "}'</span>
</code></pre></div></div>
<p>In GraphQL the request methods is always <code class="language-plaintext highlighter-rouge">POST</code>, wheather it is a query or a mutation.</p>
<p>With Java we can use Spring’s RestTemplate to send a GraphQL POST request.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ResponseEntity</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">response</span> <span class="o">=</span> <span class="n">restTemplate</span><span class="o">.</span><span class="na">exchange</span><span class="o">(</span><span class="n">graphqlEndpointUrl</span><span class="o">,</span> <span class="nc">HttpMethod</span><span class="o">.</span><span class="na">POST</span><span class="o">,</span> <span class="n">request</span><span class="o">,</span> <span class="nc">String</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</code></pre></div></div>
<p>We can load the GraphQL query request from the classpath. A mutation query would look like this:</p>
<div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">mutation</span><span class="w"> </span><span class="n">AddReview</span><span class="p">(</span><span class="nv">$input</span><span class="p">:</span><span class="w"> </span><span class="n">AddReviewInput</span><span class="p">!)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">addReview</span><span class="p">(</span><span class="n">input</span><span class="p">:</span><span class="nv">$input</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>where we externalised the <code class="language-plaintext highlighter-rouge">AddReviewInput</code> into a json file.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"input"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"bookId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"b41c35c8-3b39-428d-a84b-97aebc7a1602"</span><span class="p">,</span><span class="w">
</span><span class="nl">"stars"</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nl">"comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Its all about following your dream and about taking the risk of following your dreams"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"input"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"bookId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"c5ec9fb6-b586-449f-b869-bfe4057a4f0e"</span><span class="p">,</span><span class="w">
</span><span class="nl">"stars"</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nl">"comment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"It is like eating fancy dessert at a gourmet restaurant"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>We can build multipe GraphQL queries with the externalised inputs using the <code class="language-plaintext highlighter-rouge">variables</code> field</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="nf">createJsonQueries</span><span class="o">(</span><span class="nc">String</span> <span class="n">graphql</span><span class="o">,</span> <span class="nc">String</span> <span class="n">operationName</span><span class="o">,</span> <span class="nc">String</span> <span class="n">variables</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">JsonProcessingException</span> <span class="o">{</span>
<span class="nc">ObjectNode</span> <span class="n">wrapper</span> <span class="o">=</span> <span class="n">objectMapper</span><span class="o">.</span><span class="na">createObjectNode</span><span class="o">();</span>
<span class="n">wrapper</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"query"</span><span class="o">,</span> <span class="n">graphql</span><span class="o">);</span>
<span class="n">wrapper</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"operationName"</span><span class="o">,</span> <span class="n">operationName</span><span class="o">);</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">queries</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>();</span>
<span class="nc">JsonNode</span> <span class="n">jsonNode</span> <span class="o">=</span> <span class="n">objectMapper</span><span class="o">.</span><span class="na">readTree</span><span class="o">(</span><span class="n">variables</span><span class="o">);</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">jsonNode</span><span class="o">.</span><span class="na">size</span><span class="o">();</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">wrapper</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">"variables"</span><span class="o">,</span> <span class="n">jsonNode</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">i</span><span class="o">));</span>
<span class="n">queries</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">objectMapper</span><span class="o">.</span><span class="na">writeValueAsString</span><span class="o">(</span><span class="n">wrapper</span><span class="o">));</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">queries</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>GraphQL is introspecitve. With the <code class="language-plaintext highlighter-rouge">__schema</code> query we can list all types in the schema</p>
<div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">query</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">__schema</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">types</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">name</span><span class="w">
</span><span class="n">kind</span><span class="w">
</span><span class="n">description</span><span class="w">
</span><span class="n">fields</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">name</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>With the <code class="language-plaintext highlighter-rouge">__type</code> query we can get details about a type</p>
<div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">query</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">__type</span><span class="p">(</span><span class="n">name</span><span class="p">:</span><span class="w"> </span><span class="s2">"Book"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">name</span><span class="w">
</span><span class="n">kind</span><span class="w">
</span><span class="n">description</span><span class="w">
</span><span class="n">fields</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">name</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p><a href="https://github.com/graphql/graphiql"><code class="language-plaintext highlighter-rouge">GraphiQL</code></a> is a great tool to explore a GraphQL API. Normally you include a pre-bundled version into your GraphQL server.
Then based on the schema definition which it gets from the GraphQL Server and you can start playing with the API.
Here is live demo <a href="https://graphql.github.io/swapi-graphql/">https://graphql.github.io/swapi-graphql/</a></p>
<p><a href="https://github.com/prisma-labs/graphql-playground"><code class="language-plaintext highlighter-rouge">GraphQL Playground</code></a> is another popular GraphQL tool. It can be used even
as a desktop app. It has query history, configuration of HTTP headers and tabs.
<img src="/images/2020-01-25/graphql-playground.png" alt="graphql-playground" /></p>
<h2 id="conclusion">Conclusion</h2>
<p>In this blog post we saw how easy to create a simple GraphQL server and client.
The code examples can be found in this repository <a href="https://github.com/altfatterz/graphql-demos">https://github.com/altfatterz/graphql-demos</a></p>GraphQL is a query language for your API. It is also a specification which determines the validity schema on the server. It is strongly typed, the schema defines the GraphQL API’s type system. It defines the possible objects that a client can access. A client can find information about the schema via introspection.Schema Evolution with Confluent Schema Registry2020-01-02T00:00:00+00:002020-01-02T00:00:00+00:00http://zoltanaltfatter.com/2020/01/02/schema-evolution-with-confluent-registry<p>In this blog post we are looking into schema evolution with <a href="https://docs.confluent.io/current/schema-registry/index.html">Confluent Schema Registry</a>. Kakfa doesn’t do any data verification it just accepts bytes as input without even loading into memory.
The consumers might break if the producers send wrong data, for example by renaming a field. The Schema Registry takes the responsibility to validate the data. It is a separate component to which both the consumers and producers talk to. It supports <a href="https://avro.apache.org/">Apache Avro</a> as the data format.</p>
<h4 id="start-kafka">Start Kafka</h4>
<p>We are starting Kafka with <code class="language-plaintext highlighter-rouge">docker-compose</code> using the <a href="https://hub.docker.com/r/lensesio/fast-data-dev">lensesio/fast-data-dev</a> image. This will also startup other services like <code class="language-plaintext highlighter-rouge">Confluent Schema Registry</code> which we will need later on.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">kafka-cluster</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">lensesio/fast-data-dev</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">fast-data-dev</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">ADV_HOST</span><span class="pi">:</span> <span class="s">127.0.0.1</span> <span class="c1"># Change to 192.168.99.100 if using Docker Toolbox</span>
<span class="na">RUNTESTS</span><span class="pi">:</span> <span class="m">0</span> <span class="c1"># Disable Running tests so the cluster starts faster</span>
<span class="na">FORWARDLOGS</span><span class="pi">:</span> <span class="m">0</span> <span class="c1"># Disable running 5 file source connectors that bring application logs into Kafka topics</span>
<span class="na">SAMPLEDATA</span><span class="pi">:</span> <span class="m">0</span> <span class="c1"># Do not create sea_vessel_position_reports, nyc_yellow_taxi_trip_data, reddit_posts topics with sample Avro records.</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">2181:2181</span> <span class="c1"># Zookeeper</span>
<span class="pi">-</span> <span class="s">3030:3030</span> <span class="c1"># Lensesio UI</span>
<span class="pi">-</span> <span class="s">8081-8083:8081-8083</span> <span class="c1"># REST Proxy, Schema Registry, Kafka Connect ports</span>
<span class="pi">-</span> <span class="s">9581-9585:9581-9585</span> <span class="c1"># JMX Ports</span>
<span class="pi">-</span> <span class="s">9092:9092</span> <span class="c1"># Kafka Broker</span>
</code></pre></div></div>
<p>Before you start up the services make sure you increase the RAM for at least 4GB in your Docker for Mac/Windows settings.</p>
<p>To start up the services use this command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker-compose up <span class="nt">-d</span>
</code></pre></div></div>
<p>Open the Lensesio UI in a browser <code class="language-plaintext highlighter-rouge">http://localhost:3030</code></p>
<p><img src="/images/2020-01-02/lenses-ui.png" alt="lenses-ui" /></p>
<h4 id="avro">Avro</h4>
<p>Avro is a language independent, schema-based data serialization library. The schema is specified in a JSON format.</p>
<p>It has the following primitive data types:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">null</code> - No value.</li>
<li><code class="language-plaintext highlighter-rouge">boolean</code> - A binary value.</li>
<li><code class="language-plaintext highlighter-rouge">int</code> - A 32-bit signed integer.</li>
<li><code class="language-plaintext highlighter-rouge">long</code> - A 64-bit signed integer.</li>
<li><code class="language-plaintext highlighter-rouge">float</code> - A single precision (32 bit) IEEE 754 floating-point number</li>
<li><code class="language-plaintext highlighter-rouge">double</code> - A double precision (64-bit) IEEE 754 floating-point number.</li>
<li><code class="language-plaintext highlighter-rouge">byte</code> - A sequence of 8-bit unsigned bytes.</li>
<li><code class="language-plaintext highlighter-rouge">string</code> - A Unicode character sequence.</li>
</ul>
<p>Complex types:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">record</code></li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"namespace"</span><span class="p">:</span><span class="w"> </span><span class="s2">"com.github.altfatterz.avro"</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Customer"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"record"</span><span class="p">,</span><span class="w">
</span><span class="nl">"fields"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"first_name"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc"</span><span class="p">:</span><span class="w"> </span><span class="s2">"the first name of the customer"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"last_name"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc"</span><span class="p">:</span><span class="w"> </span><span class="s2">"the last name of the customer"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">enum</code></li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"enum"</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Colors"</span><span class="p">,</span><span class="w">
</span><span class="nl">"symbols"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"WHITE"</span><span class="p">,</span><span class="w"> </span><span class="s2">"BLUE"</span><span class="p">,</span><span class="w"> </span><span class="s2">"GREEN"</span><span class="p">,</span><span class="w"> </span><span class="s2">"RED"</span><span class="p">,</span><span class="w"> </span><span class="s2">"BLACK"</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">array</code></li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"array"</span><span class="p">,</span><span class="w"> </span><span class="nl">"items"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">map</code></li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"map"</span><span class="p">,</span><span class="w"> </span><span class="nl">"values"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"int"</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">union</code> - they are represented as JSON array, and indicate that a field may have more than one type</li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"phone_number"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"null"</span><span class="p">,</span><span class="s2">"string"</span><span class="p">],</span><span class="w">
</span><span class="nl">"default"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"doc"</span><span class="p">:</span><span class="w"> </span><span class="s2">"the phone number of the customer"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>More information about <code class="language-plaintext highlighter-rouge">Avro</code> you can find <a href="https://avro.apache.org/docs/current/">here</a></p>
<h4 id="avro-tools">Avro Tools</h4>
<p><code class="language-plaintext highlighter-rouge">Avro Tools</code> is a command line utility to work with avro data.</p>
<p>Install</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>brew <span class="nb">install </span>avro-tools
</code></pre></div></div>
<p>Convert from <code class="language-plaintext highlighter-rouge">json</code> to <code class="language-plaintext highlighter-rouge">avro</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>avro-tools fromjson <span class="nt">--schema-file</span> src/main/resources/avro/schema.avsc customer.json <span class="o">></span> customer.avro
</code></pre></div></div>
<p>A <code class="language-plaintext highlighter-rouge">customer.avro</code> should be created.</p>
<p>Get the schema from <code class="language-plaintext highlighter-rouge">customer.avro</code> file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>avro-tools getschema customer.avro
</code></pre></div></div>
<p>The schema will be printed to the standard output.</p>
<p>Convert from <code class="language-plaintext highlighter-rouge">avro</code> to <code class="language-plaintext highlighter-rouge">json</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>avro-tools tojson <span class="nt">--pretty</span> customer.avro
</code></pre></div></div>
<h4 id="confluent-schema-registry">Confluent Schema Registry</h4>
<p>Avro schemas are stored in the <code class="language-plaintext highlighter-rouge">Confluent Schema Registry</code>. A <code class="language-plaintext highlighter-rouge">producer</code> sends <code class="language-plaintext highlighter-rouge">avro</code> content to <code class="language-plaintext highlighter-rouge">Kafka</code> and the schema to <code class="language-plaintext highlighter-rouge">Schema Registry</code>, similarly a <code class="language-plaintext highlighter-rouge">consumer</code> will get the schema from <code class="language-plaintext highlighter-rouge">Schema Registry</code> and will read the <code class="language-plaintext highlighter-rouge">avro</code> content from <code class="language-plaintext highlighter-rouge">Kafka</code>.</p>
<h4 id="kafka-avro-console-producer-and-consumer">Kafka Avro Console Producer and Consumer</h4>
<p>These tools come with the <code class="language-plaintext highlighter-rouge">Confluent Schema Registry</code> and allow to send <code class="language-plaintext highlighter-rouge">avro</code> data to <code class="language-plaintext highlighter-rouge">Kafka</code>. The fastest way to get the binaries is using <code class="language-plaintext highlighter-rouge">docker run</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">--net</span><span class="o">=</span>host <span class="nt">-it</span> confluentinc/cp-schema-registry:5.3.2 /bin/bash
</code></pre></div></div>
<p>Let’s start first the <code class="language-plaintext highlighter-rouge">kafka-avro-console-consumer</code> with the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-avro-console-consumer <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--topic</span> mytopic <span class="se">\</span>
<span class="nt">--from-beginning</span> <span class="se">\</span>
<span class="nt">--property</span> schema.registry.url<span class="o">=</span>http://localhost:8081
</code></pre></div></div>
<p>Then in another terminal start the <code class="language-plaintext highlighter-rouge">kafka-avro-console-producer</code> with the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-avro-console-producer <span class="nt">--broker-list</span> localhost:9092 <span class="nt">--topic</span> mytopic <span class="se">\</span>
<span class="nt">--property</span> schema.registry.url<span class="o">=</span>http://localhost:8081 <span class="se">\</span>
<span class="nt">--property</span> value.schema<span class="o">=</span><span class="s1">'{"type":"record", "name":"Customer", "fields":[{"name":"first_name", "type":"string"}] }'</span>
</code></pre></div></div>
<p>And now we can send data:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">{</span><span class="s2">"first_name"</span>:<span class="s2">"Zoltan"</span><span class="o">}</span>
<span class="o">{</span><span class="s2">"first_name"</span>:<span class="s2">"Kasia"</span><span class="o">}</span>
</code></pre></div></div>
<p>Now if we check the <code class="language-plaintext highlighter-rouge">kafka-avro-console-consumer</code> it should have received the data. In the <a href="http://localhost:3030/kafka-topics-ui/#/">kafka-topics-ui</a> we can see that the <code class="language-plaintext highlighter-rouge">mytopic</code> topic was created and the data inside the topic has <code class="language-plaintext highlighter-rouge">avro</code> data type.</p>
<p><img src="/images/2020-01-02/kafka-avro-console-producer.png" alt="kafka-avro-console-producer" /></p>
<p>If we send the wrong field name or wrong data type as the field value we get an error:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">{</span><span class="s2">"name"</span>:<span class="s2">"Zoltan"</span><span class="o">}</span>
org.apache.kafka.common.errors.SerializationException: Error deserializing json <span class="o">{</span><span class="s2">"name"</span>:<span class="s2">"Zoltan"</span><span class="o">}</span> to Avro of schema <span class="o">{</span><span class="s2">"type"</span>:<span class="s2">"record"</span>,<span class="s2">"name"</span>:<span class="s2">"Customer"</span>,<span class="s2">"fields"</span>:[<span class="o">{</span><span class="s2">"name"</span>:<span class="s2">"first_name"</span>,<span class="s2">"type"</span>:<span class="s2">"string"</span><span class="o">}]}</span>
Caused by: org.apache.avro.AvroTypeException: Expected field name not found: first_name
</code></pre></div></div>
<h4 id="kafka-avro-producer-and-consumer-with-java">Kafka Avro Producer and Consumer with Java</h4>
<p>Compared to the simple <code class="language-plaintext highlighter-rouge">KafkaProducer</code> and <code class="language-plaintext highlighter-rouge">KafkaConsumer</code> discussed in a previous blog post <a href="https://zoltanaltfatter.com/2019/11/23/kafka-basics/">https://zoltanaltfatter.com/2019/11/23/kafka-basics/</a> we need to configure a few things:</p>
<h5 id="kafkaconsumer">KafkaConsumer</h5>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>properties.setProperty<span class="o">(</span>ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class.getName<span class="o">())</span><span class="p">;</span>
properties.setProperty<span class="o">(</span><span class="s2">"specific.avro.reader"</span>, <span class="s2">"true"</span><span class="o">)</span><span class="p">;</span>
properties.setProperty<span class="o">(</span><span class="s2">"schema.registry.url"</span>, <span class="s2">"http://localhost:8081"</span><span class="o">)</span><span class="p">;</span>
</code></pre></div></div>
<h5 id="kafkaproducer">KafkaProducer</h5>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">properties</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">VALUE_SERIALIZER_CLASS_CONFIG</span><span class="o">,</span> <span class="nc">KafkaAvroSerializer</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getName</span><span class="o">());</span>
<span class="n">properties</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"schema.registry.url"</span><span class="o">,</span> <span class="s">"http://localhost:8081"</span><span class="o">);</span>
</code></pre></div></div>
<p>The complete source code can be found <a href="https://github.com/altfatterz/learning-kafka/tree/master/avro-examples/src/main/java/com/github/altfatterz">here</a>.</p>
<p>With the <code class="language-plaintext highlighter-rouge">Kafka Topics UI</code> we can inspect the produced message:</p>
<p><img src="/images/2020-01-02/customer-topic.png" alt="customer-topic" /></p>
<h4 id="schema-evolution">Schema evolution</h4>
<p>We get errors if we try to evolve a schema in a non-compatible way. For example let’s try to register another schema to the previous <code class="language-plaintext highlighter-rouge">mytopic</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-avro-console-producer <span class="nt">--broker-list</span> localhost:9092 <span class="nt">--topic</span> mytopic <span class="se">\</span>
<span class="nt">--property</span> schema.registry.url<span class="o">=</span>http://localhost:8081 <span class="se">\</span>
<span class="nt">--property</span> value.schema<span class="o">=</span><span class="s1">'{"type":"string"}'</span>
</code></pre></div></div>
<p>As soon as we type a string value we will get the following error:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: <span class="s2">"string"</span>
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema<span class="p">;</span> error code: 409
</code></pre></div></div>
<p>Let’s evolve the schema by adding another field which has a default value.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-avro-console-producer <span class="nt">--broker-list</span> localhost:9092 <span class="nt">--topic</span> mytopic <span class="se">\</span>
<span class="nt">--property</span> schema.registry.url<span class="o">=</span>http://localhost:8081 <span class="se">\</span>
<span class="nt">--property</span> value.schema<span class="o">=</span><span class="s1">'{"type":"record", "name":"Customer", "fields":[{"name":"first_name", "type":"string"}, {"name":"last_name", "type":"string", "default":"" }] }'</span>
<span class="o">{</span><span class="s2">"first_name"</span>:<span class="s2">"John"</span>, <span class="s2">"last_name"</span>:<span class="s2">"Doe"</span><span class="o">}</span>
</code></pre></div></div>
<p>We should not get error this time and we should see another version of the schema registered:</p>
<p><img src="/images/2020-01-02/schema-evolution.png" alt="schema-evolution" /></p>
<h4 id="conclusion">Conclusion</h4>
<p>In this blog post we looked into</p>
<ol>
<li>Apache Avro</li>
<li>Confluent Container Registry</li>
<li>Kafka Avro Console Producer and Consumer</li>
<li>Kafka Avro Producer and Consumer with Java</li>
<li>Schema Evolution</li>
</ol>
<p>The source code of the examples can be found here: <a href="https://github.com/altfatterz/learning-kafka/tree/master/schema-registry/avro-examples">https://github.com/altfatterz/learning-kafka/tree/master/schema-registry/avro-examples</a></p>In this blog post we are looking into schema evolution with Confluent Schema Registry. Kakfa doesn’t do any data verification it just accepts bytes as input without even loading into memory. The consumers might break if the producers send wrong data, for example by renaming a field. The Schema Registry takes the responsibility to validate the data. It is a separate component to which both the consumers and producers talk to. It supports Apache Avro as the data format. Start KafkaKafka Basics2019-11-23T00:00:00+00:002019-11-23T00:00:00+00:00http://zoltanaltfatter.com/2019/11/23/kafka-basics<p>In this post we are looking into Apache Kafka.</p>
<h4 id="install-kafka">Install Kafka</h4>
<p>There are many different ways to install Kafka. We can get the binary from <a href="https://kafka.apache.org/downloads">https://kafka.apache.org/downloads</a>
or we can use Docker. The fastest way on Mac is installing with Homebrew.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>brew <span class="nb">install </span>kafka
</code></pre></div></div>
<p>The above will also install Zookeeper.</p>
<p>Homebrew will install the configuration files into <code class="language-plaintext highlighter-rouge">/usr/local/etc/kafka/</code> directory. I prefer to create my own copy of these files.
They are included into the <a href="https://github.com/altfatterz/learning-kafka">https://github.com/altfatterz/learning-kafka</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git clone https://github.com/altfatterz/learning-kafka
</code></pre></div></div>
<p>After cloning the repository we need to change the followings:</p>
<ul>
<li>the <code class="language-plaintext highlighter-rouge">log.dirs</code> property in the <code class="language-plaintext highlighter-rouge">config/server.properties</code> to the <code class="language-plaintext highlighter-rouge"><learning-kafka-path>/data/kafka</code></li>
<li>the <code class="language-plaintext highlighter-rouge">dataDir</code> property in the <code class="language-plaintext highlighter-rouge">config/zookeeper.properties</code> to the value <code class="language-plaintext highlighter-rouge"><learning-kafka-path>/data/zookeeper</code></li>
</ul>
<h4 id="start-zookeeper-and-kafka">Start Zookeeper and Kafka</h4>
<p>Now we can start Zookeeper and Kafka in separate terminals</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>zookeeper-server-start config/zookeeper.properties
<span class="nv">$ </span>kafka-server-start config/server.properties
</code></pre></div></div>
<p>In the logs we should see</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>2019-12-23 16:09:43,145] INFO <span class="o">[</span>KafkaServer <span class="nb">id</span><span class="o">=</span>0] started <span class="o">(</span>kafka.server.KafkaServer<span class="o">)</span>
</code></pre></div></div>
<h4 id="kafka-topics">Kafka Topics</h4>
<p>We create a <code class="language-plaintext highlighter-rouge">demo-topic</code> with 3 partitions and 1 replication factor.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-topics <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--topic</span> demo-topic <span class="nt">--create</span> <span class="nt">--partitions</span> 3 <span class="nt">--replication-factor</span> 1
</code></pre></div></div>
<p>We can verify the result using:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-topics <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--topic</span> demo-topic <span class="nt">--describe</span>
Topic:demo-topic PartitionCount:3 ReplicationFactor:1 Configs:segment.bytes<span class="o">=</span>1073741824
Topic: demo-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: demo-topic Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: demo-topic Partition: 2 Leader: 0 Replicas: 0 Isr: 0
</code></pre></div></div>
<p>To delete a topic we can use:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-topics <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--topic</span> demo-topic <span class="nt">--delete</span>
</code></pre></div></div>
<p>And to see all the topics we use:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-topics <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--topic</span> demo-topic <span class="nt">--list</span>
</code></pre></div></div>
<p>Note that the previously used <code class="language-plaintext highlighter-rouge">--zookeeper</code> option is deprecated now, instead we can use the <code class="language-plaintext highlighter-rouge">--bootstrap-server</code> option.</p>
<h4 id="kafka-console-consumer">Kafka Console Consumer</h4>
<p>To start a simple consumer we can use the <code class="language-plaintext highlighter-rouge">kafka-console-consumer</code> command</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-console-consumer <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--topic</span> demo-topic
</code></pre></div></div>
<p>It does not print anything yet since there are no messages in the topic.</p>
<h4 id="kafka-console-producer">Kafka Console Producer</h4>
<p>In another terminal we produce some messages with the <code class="language-plaintext highlighter-rouge">kafka-console-producer</code> command</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-console-producer <span class="nt">--broker-list</span> 127.0.0.1:9092 <span class="nt">--topic</span> demo-topic
<span class="o">></span>first message
<span class="o">></span>second message
<span class="o">></span>third message
<span class="o">></span>^C
</code></pre></div></div>
<p>In the terminal where the <code class="language-plaintext highlighter-rouge">kafka-console-consumer</code> is started we should now see the messages.</p>
<h4 id="kafka-consumer-groups">Kafka Consumer Groups</h4>
<p>By default if we don’t specify a <code class="language-plaintext highlighter-rouge">group</code> for the consumer a <code class="language-plaintext highlighter-rouge">consumer group</code> will be generated with a single member</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-consumer-groups <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--list</span>
console-consumer-75696
</code></pre></div></div>
<p>We can specify a group when defining the consumer.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-console-consumer <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--topic</span> demo-topic <span class="nt">--group</span> demo-app
</code></pre></div></div>
<p>Let’s start another <code class="language-plaintext highlighter-rouge">kafka-console-consumer</code> with the above command in another terminal with the same <code class="language-plaintext highlighter-rouge">demo-app</code> group.</p>
<p>We can monitor the <code class="language-plaintext highlighter-rouge">current offset</code> and <code class="language-plaintext highlighter-rouge">lag</code> of the consumers connected to the partitions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kafka-consumer-groups <span class="nt">--bootstrap-server</span> localhost:9092 <span class="nt">--group</span> demo-app <span class="nt">--describe</span>
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
demo-app demo-topic 0 1 1 0 consumer-1-7e3aa047-c981-4be1-bfac-c0e90305de12 /192.168.1.6 consumer-1
demo-app demo-topic 1 2 2 0 consumer-1-7e3aa047-c981-4be1-bfac-c0e90305de12 /192.168.1.6 consumer-1
demo-app demo-topic 2 2 2 0 consumer-1-a1cf8bb1-5aef-4e09-8d88-d3bdc2ff6f33 /192.168.1.6 consumer-1
</code></pre></div></div>
<p>Here we can see that one consumer is connected to the first two partitions while the other consumer is connected to the third partition.</p>
<h4 id="kafkaconsumer">KafkaConsumer</h4>
<p>To connect with Java to a topic we need the following dependency:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.apache.kafka<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>kafka-clients<span class="nt"></artifactId></span>
<span class="nt"><version></span>${kafka.version}<span class="nt"></version></span>
<span class="nt"></dependency></span>
</code></pre></div></div>
<p>We set the following properties to create a <code class="language-plaintext highlighter-rouge">KafkaConsumer</code> instance</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Properties properties <span class="o">=</span> new Properties<span class="o">()</span><span class="p">;</span>
properties.setProperty<span class="o">(</span>ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, <span class="s2">"localhost:9092"</span><span class="o">)</span><span class="p">;</span>
properties.setProperty<span class="o">(</span>ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName<span class="o">())</span><span class="p">;</span>
properties.setProperty<span class="o">(</span>ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName<span class="o">())</span><span class="p">;</span>
properties.setProperty<span class="o">(</span>ConsumerConfig.GROUP_ID_CONFIG, <span class="s2">"demo-app"</span><span class="o">)</span><span class="p">;</span>
KafkaConsumer<String, String> consumer <span class="o">=</span> new KafkaConsumer<<span class="o">>(</span>properties<span class="o">)</span><span class="p">;</span>
</code></pre></div></div>
<p>Then we subscribe to the <code class="language-plaintext highlighter-rouge">demo-topic</code> topic and poll the consumer for a duration and try again.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// subscribe consumer to topic</span>
<span class="n">consumer</span><span class="o">.</span><span class="na">subscribe</span><span class="o">(</span><span class="nc">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">"demo-topic"</span><span class="o">));</span>
<span class="c1">// poll for new data</span>
<span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">ConsumerRecords</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">></span> <span class="n">records</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="na">poll</span><span class="o">(</span><span class="nc">Duration</span><span class="o">.</span><span class="na">ofMillis</span><span class="o">(</span><span class="mi">200</span><span class="o">));</span>
<span class="k">for</span> <span class="o">(</span><span class="nc">ConsumerRecord</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">></span> <span class="n">record</span> <span class="o">:</span> <span class="n">records</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"Key: {}, Value:{}, Partition: {}, Offset: {}"</span><span class="o">,</span> <span class="n">record</span><span class="o">.</span><span class="na">key</span><span class="o">(),</span> <span class="n">record</span><span class="o">.</span><span class="na">value</span><span class="o">(),</span> <span class="n">record</span><span class="o">.</span><span class="na">partition</span><span class="o">(),</span> <span class="n">record</span><span class="o">.</span><span class="na">offset</span><span class="o">());</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Next if we produce the <code class="language-plaintext highlighter-rouge">Kafka rocks!</code> message with the <code class="language-plaintext highlighter-rouge">kafka-console-producer</code> we can see in the logs of our Java based Kafka consumer:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>12:04:53.221 <span class="o">[</span>main] INFO c.g.altfatterz.KafkaConsumerDemo - Key: null, Value:Kafka rocks!, Partition: 2, Offset: 4
</code></pre></div></div>
<h4 id="kafkaproducer">KafkaProducer</h4>
<p>We can also produce messages with Java using the <code class="language-plaintext highlighter-rouge">KafkaProducer</code> API</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// producer properties</span>
<span class="nc">Properties</span> <span class="n">properties</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Properties</span><span class="o">();</span>
<span class="n">properties</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">BOOTSTRAP_SERVERS_CONFIG</span><span class="o">,</span> <span class="no">BOOTSTRAP_SERVERS</span><span class="o">);</span>
<span class="n">properties</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">KEY_SERIALIZER_CLASS_CONFIG</span><span class="o">,</span> <span class="nc">StringSerializer</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getName</span><span class="o">());</span>
<span class="n">properties</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">VALUE_SERIALIZER_CLASS_CONFIG</span><span class="o">,</span> <span class="nc">StringSerializer</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getName</span><span class="o">());</span>
<span class="c1">// producer</span>
<span class="nc">KafkaProducer</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">String</span><span class="o">></span> <span class="n">producer</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KafkaProducer</span><span class="o"><>(</span><span class="n">properties</span><span class="o">);</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// create a producer record
ProducerRecord<String, String> record <span class="o">=</span> new ProducerRecord<<span class="o">>(</span>TOPIC, <span class="s2">"learning kafka"</span><span class="o">)</span><span class="p">;</span>
logger.info<span class="o">(</span><span class="s2">"send message asynchronously"</span><span class="o">)</span><span class="p">;</span>
producer.send<span class="o">(</span>record<span class="o">)</span><span class="p">;</span>
logger.info<span class="o">(</span><span class="s2">"flushing and closing the producer"</span><span class="o">)</span><span class="p">;</span>
producer.close<span class="o">()</span><span class="p">;</span>
</code></pre></div></div>
<h4 id="conclusion">Conclusion</h4>
<p>In this post we looked into</p>
<ol>
<li>Starting Zookeeper and Kafka</li>
<li>Useful CLI commands: <code class="language-plaintext highlighter-rouge">kafka-topics</code>, <code class="language-plaintext highlighter-rouge">kafka-console-consumer</code>, <code class="language-plaintext highlighter-rouge">kafka-console-producer</code>, <code class="language-plaintext highlighter-rouge">kafka-consumer-groups</code></li>
<li>The Kafka Java API using <code class="language-plaintext highlighter-rouge">KafkaConsumer</code> and <code class="language-plaintext highlighter-rouge">KafkaProducer</code></li>
</ol>
<p>Source code is available here: <a href="https://github.com/altfatterz/learning-kafka">https://github.com/altfatterz/learning-kafka</a></p>In this post we are looking into Apache Kafka.Signed URLs with Google Cloud Storage bucket2019-09-08T00:00:00+00:002019-09-08T00:00:00+00:00http://zoltanaltfatter.com/2019/09/08/signed-urls-with-google-cloud-storage-bucket<p>Let’s say you have an object in a <a href="https://cloud.google.com/storage/">Google Cloud Storage</a> bucket which is set to be private. You want to share it with people who have no Google Cloud account, for example, subscribed visitors to your website.
This can be a video course that only paying users can access, or an E-book that requires subscription.</p>
<p><a href="https://cloud.google.com/storage/docs/access-control/signed-urls">Signed URLs</a> are good solution for this problem. They give a time-limited resource access to anyone in possession of the URL, regardless of whether they have a Google account or not.</p>
<p>The signed URL contains authentication information in its query string allowing users without credentials to perform specific actions on a resource.</p>
<p>In this blog post we show how to provide users a time-limited read access to an object inside a Google Cloud Storage bucket.</p>
<h2 id="google-cloud-storage-bucket">Google Cloud Storage bucket</h2>
<p>We create a <code class="language-plaintext highlighter-rouge">bucket</code> and upload a file to it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gsutil mb gs://signed-url-demo
<span class="nv">$ </span><span class="nb">echo</span> <span class="s2">"Since you are a subscribed user, you get this resource."</span> <span class="o">></span> demo.txt
<span class="nv">$ </span>gsutil <span class="nb">cp </span>demo.txt gs://signed-url-demo
</code></pre></div></div>
<p>We make sure the file was uploaded.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gsutil <span class="nb">ls</span> <span class="nt">-r</span> gs://signed-url-demo
gs://signed-url-demo/demo.txt
</code></pre></div></div>
<p>We verify that the file is private.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>http https://storage.googleapis.com/signed-url-demo/demo.txt
HTTP/1.1 403 Forbidden
Alt-Svc: <span class="nv">quic</span><span class="o">=</span><span class="s2">":443"</span><span class="p">;</span> <span class="nv">ma</span><span class="o">=</span>2592000<span class="p">;</span> <span class="nv">v</span><span class="o">=</span><span class="s2">"46,43,39"</span>
Cache-Control: private, max-age<span class="o">=</span>0
Content-Length: 216
Content-Type: application/xml<span class="p">;</span> <span class="nv">charset</span><span class="o">=</span>UTF-8
Date: Sun, 08 Sep 2019 10:48:40 GMT
Expires: Sun, 08 Sep 2019 10:48:40 GMT
Server: UploadServer
X-GUploader-UploadID: AEnB2UpTLJO2792GLnoNGHnyhuEiLOyoal4ibgogvHAX-6dyO0glTU6uF8wmmnliF7repZbKGOQArmz2V52vSppuOFYmTqCjkg
<?xml <span class="nv">version</span><span class="o">=</span><span class="s1">'1.0'</span> <span class="nv">encoding</span><span class="o">=</span><span class="s1">'UTF-8'</span>?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous <span class="nb">caller </span>does not have storage.objects.get access to signed-url-demo/demo.txt.</Details></Error>
</code></pre></div></div>
<h2 id="create-a-service-account">Create a service account</h2>
<p>Next we create a <code class="language-plaintext highlighter-rouge">service account</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud iam service-accounts create signed-url-demo <span class="nt">--display-name</span> <span class="s2">"Signed URL demo"</span>
</code></pre></div></div>
<p>We verify that the <code class="language-plaintext highlighter-rouge">service account</code> was created.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud iam service-accounts list
NAME EMAIL DISABLED
...
Signed URL demo signed-url-demo@altfatterz.iam.gserviceaccount.com False
...
</code></pre></div></div>
<h2 id="grant-the-storage-object-viewer-role-to-the-service-account">Grant the <code class="language-plaintext highlighter-rouge">Storage Object Viewer</code> role to the service account</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud projects add-iam-policy-binding altfatterz <span class="se">\</span>
<span class="nt">--member</span> serviceAccount:signed-url-demo@altfatterz.iam.gserviceaccount.com <span class="se">\</span>
<span class="nt">--role</span> roles/storage.objectViewer
</code></pre></div></div>
<p>In the response we see the updated <code class="language-plaintext highlighter-rouge">IAM policy</code> for the <code class="language-plaintext highlighter-rouge">altfatterz</code> project</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Updated IAM policy <span class="k">for </span>project <span class="o">[</span>altfatterz].
bindings:
...
- members:
...
- serviceAccount:signed-url-demo@altfatterz.iam.gserviceaccount.com
role: roles/storage.objectViewer
</code></pre></div></div>
<h2 id="create-a-service-account-key">Create a service account key</h2>
<p>Next, we create a <code class="language-plaintext highlighter-rouge">key</code> for the <code class="language-plaintext highlighter-rouge">service account</code> and store it in the <code class="language-plaintext highlighter-rouge">key.json</code> file</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcloud iam service-accounts keys create key.json <span class="se">\</span>
<span class="nt">--iam-account</span> signed-url-demo@altfatterz.iam.gserviceaccount.com
</code></pre></div></div>
<h2 id="use-the-signurl-command">Use the <code class="language-plaintext highlighter-rouge">signurl</code> command</h2>
<p>And finally we can use the <code class="language-plaintext highlighter-rouge">signurl</code> command to generate a signed URL that embeds authentication data so the URL can be used by someone who does not have a Google account.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gsutil signurl <span class="nt">-d</span> 1m key.json gs://signed-url-demo/demo.txt
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">signurl</code> command uses the private key (<code class="language-plaintext highlighter-rouge">key.json</code>) of the service account to generate the cryptographic signature for the generated URL.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>URL HTTP Method Expiration Signed URL
gs://signed-url-demo/demo.txt GET 2019-09-08 12:24:28 https://storage.googleapis.com/signed-url-demo/demo.txt?x-goog-signature<span class="o">=</span>8f4afba14b4ef24c2af1975bee453383c7a290bca3ca3db2e11350889caa4e48cc11cccfd099a1ccf6763185f36a33862802502362aa9fdc0095dfd7075bd274ce0b99c42b7e4a2a30e7c247d55195b83d1bb45ea390ab947ea51e086c0d948fc61ee2dcfe9304f8affdc267bee05dd45daf6e279da259c1c574a00d94908e707acec1641bec3c1f88058c063affd96c3aa4431f961f622e88d964ae6243239c9e4ecdecd0153e39bb5a6e806c5c0ff6c3da26f3377fbae9fd39451553469bca0f0b9281fc8103ff450d36848298bd49605fd76dcba6465a52ecfc125952f9642902ae332ad7c0914b2a56e2c1a54f20ab937429adc340b2c54f6437a5ad4a9a&x-goog-algorithm<span class="o">=</span>GOOG4-RSA-SHA256&x-goog-credential<span class="o">=</span>signed-url-demo%40altfatterz.iam.gserviceaccount.com%2F20190908%2Fus%2Fstorage%2Fgoog4_request&x-goog-date<span class="o">=</span>20190908T102328Z&x-goog-expires<span class="o">=</span>60&x-goog-signedheaders<span class="o">=</span>host
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">signurl</code> command requires the <code class="language-plaintext highlighter-rouge">pyopenssl</code> library, which you can easily install with <code class="language-plaintext highlighter-rouge">pip install pyopenssl</code></p>
<p>And now we access the file using the signed URL.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>http https://storage.googleapis.com/signed-url-demo/demo.txt<span class="se">\?</span>x-goog-signature<span class="se">\=</span>8f4afba14b4ef24c2af1975bee453383c7a290bca3ca3db2e11350889caa4e48cc11cccfd099a1ccf6763185f36a33862802502362aa9fdc0095dfd7075bd274ce0b99c42b7e4a2a30e7c247d55195b83d1bb45ea390ab947ea51e086c0d948fc61ee2dcfe9304f8affdc267bee05dd45daf6e279da259c1c574a00d94908e707acec1641bec3c1f88058c063affd96c3aa4431f961f622e88d964ae6243239c9e4ecdecd0153e39bb5a6e806c5c0ff6c3da26f3377fbae9fd39451553469bca0f0b9281fc8103ff450d36848298bd49605fd76dcba6465a52ecfc125952f9642902ae332ad7c0914b2a56e2c1a54f20ab937429adc340b2c54f6437a5ad4a9a<span class="se">\&</span>x-goog-algorithm<span class="se">\=</span>GOOG4-RSA-SHA256<span class="se">\&</span>x-goog-credential<span class="se">\=</span>signed-url-demo%40altfatterz.iam.gserviceaccount.com%2F20190908%2Fus%2Fstorage%2Fgoog4_request<span class="se">\&</span>x-goog-date<span class="se">\=</span>20190908T102328Z<span class="se">\&</span>x-goog-expires<span class="se">\=</span>60<span class="se">\&</span>x-goog-signedheaders<span class="se">\=</span>host
HTTP/1.1 200 OK
Accept-Ranges: bytes
Alt-Svc: <span class="nv">quic</span><span class="o">=</span><span class="s2">":443"</span><span class="p">;</span> <span class="nv">ma</span><span class="o">=</span>2592000<span class="p">;</span> <span class="nv">v</span><span class="o">=</span><span class="s2">"46,43,39"</span>
Cache-Control: private, max-age<span class="o">=</span>0
Content-Language: en
Content-Length: 55
Content-Type: text/plain
Date: Sun, 08 Sep 2019 10:23:44 GMT
ETag: <span class="s2">"ef8836fdf24851afb3ea40377e4fed52"</span>
Expires: Sun, 08 Sep 2019 10:23:44 GMT
Last-Modified: Sun, 08 Sep 2019 10:06:51 GMT
Server: UploadServer
X-GUploader-UploadID: AEnB2UqiI6vBXIybnBx9P80LSNlmxARPcXlJjA-XY8ErQyumQ3rSuh_kefOYbvFYpVoRaD0oh12uQYMx9sKGpNJuj9dSVVQWDA
x-goog-generation: 1567937211730069
x-goog-hash: <span class="nv">crc32c</span><span class="o">=</span><span class="nv">eDfjkw</span><span class="o">==</span>
x-goog-hash: <span class="nv">md5</span><span class="o">=</span>74g2/fJIUa+z6kA3fk/tUg<span class="o">==</span>
x-goog-metageneration: 1
x-goog-storage-class: STANDARD
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 55
Since you are a subscribed user, you get this resource.
</code></pre></div></div>
<p>With the <code class="language-plaintext highlighter-rouge">-d</code> parameter we specified that duration until the signed url should be valid (default is 1 hour).</p>
<p>In this case after a minute we get <code class="language-plaintext highlighter-rouge">400 Bad Request</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>http https://storage.googleapis.com/signed-url-demo/demo.txt<span class="se">\?</span>x-goog-signature<span class="se">\=</span>8f4afba14b4ef24c2af1975bee453383c7a290bca3ca3db2e11350889caa4e48cc11cccfd099a1ccf6763185f36a33862802502362aa9fdc0095dfd7075bd274ce0b99c42b7e4a2a30e7c247d55195b83d1bb45ea390ab947ea51e086c0d948fc61ee2dcfe9304f8affdc267bee05dd45daf6e279da259c1c574a00d94908e707acec1641bec3c1f88058c063affd96c3aa4431f961f622e88d964ae6243239c9e4ecdecd0153e39bb5a6e806c5c0ff6c3da26f3377fbae9fd39451553469bca0f0b9281fc8103ff450d36848298bd49605fd76dcba6465a52ecfc125952f9642902ae332ad7c0914b2a56e2c1a54f20ab937429adc340b2c54f6437a5ad4a9a<span class="se">\&</span>x-goog-algorithm<span class="se">\=</span>GOOG4-RSA-SHA256<span class="se">\&</span>x-goog-credential<span class="se">\=</span>signed-url-demo%40altfatterz.iam.gserviceaccount.com%2F20190908%2Fus%2Fstorage%2Fgoog4_request<span class="se">\&</span>x-goog-date<span class="se">\=</span>20190908T102328Z<span class="se">\&</span>x-goog-expires<span class="se">\=</span>60<span class="se">\&</span>x-goog-signedheaders<span class="se">\=</span>host
HTTP/1.1 400 Bad Request
Alt-Svc: <span class="nv">quic</span><span class="o">=</span><span class="s2">":443"</span><span class="p">;</span> <span class="nv">ma</span><span class="o">=</span>2592000<span class="p">;</span> <span class="nv">v</span><span class="o">=</span><span class="s2">"46,43,39"</span>
Cache-Control: private, max-age<span class="o">=</span>0
Content-Length: 202
Content-Type: application/xml<span class="p">;</span> <span class="nv">charset</span><span class="o">=</span>UTF-8
Date: Sun, 08 Sep 2019 10:25:23 GMT
Expires: Sun, 08 Sep 2019 10:25:23 GMT
Server: UploadServer
X-GUploader-UploadID: AEnB2UpomRy06euJeNlmt-nMWJAxCIbGJUJCmaz2t08s4KXvy2hXUySRS62J77d2JV07o5hmRmxqeVuJMTytQU3KVyPq-Bd6gQ
<?xml <span class="nv">version</span><span class="o">=</span><span class="s1">'1.0'</span> <span class="nv">encoding</span><span class="o">=</span><span class="s1">'UTF-8'</span>?><Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Details>Request signature expired at: 2019-09-08T10:24:28+00:00</Details></Error>
</code></pre></div></div>Let’s say you have an object in a Google Cloud Storage bucket which is set to be private. You want to share it with people who have no Google Cloud account, for example, subscribed visitors to your website. This can be a video course that only paying users can access, or an E-book that requires subscription.