In this blog post we will create an Apache Ignite cluster with Kubernetes and interact with it through Ignite’s REST API. We will create a cache, put and get records into/from that cache.
I won’t get into the details of concepts like pods, nodes, services etc. If you are not familier with those you can always refer to official documentation which is really very well written. Let’s get started with installing Kubernetes. I will work with a MacBook Pro running macOS High Sierra but it will be easy to follow on other environments.
Installing Kubernetes
We will first install kubectl which is command line tools to interact with kubernetes:
$ brew install kubectl
Then we will install minikube which enables us to run kubernetes locally:
$ brew cask install minikube
Now we can start our local Kubernetes cluster via running:
$ minikube start
Starting local Kubernetes v1.9.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.
Let’s check if everything is OK:
$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Great! Now our Kubernetes cluster is up and running. You can run:
$ minikube dashboard
to see the Kubernetes Dashboard which is a web-based Kubernetes user interface and a great way to monitor and interact with your cluster.
Deploying Apache Ignite nodes
The next step is to deploy Apache Ignite instances to our cluster. For this I created a yaml configuration file as follows:
apiVersion: v1
kind: Service
metadata:
name: ignite
spec:
ports:
- port: 8080
selector:
app: ignite
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ignite
spec:
selector:
matchLabels:
app: ignite
replicas: 4
template:
metadata:
labels:
app: ignite
spec:
containers:
- name: ignite
image: alpert/ignite
ports:
- containerPort: 8080
env:
- name: CONFIG_URI
value: https://raw.githubusercontent.com/apache/ignite/master/examples/config/example-cache.xml
- name: OPTION_LIBS
value: ignite-rest-http
Gist link is here: https://gist.github.com/alpert/9c937ef8806280e964249050648829e2
The configuration defines a service and a deployment. Service is a NodePort service which enables us to access 8080 port of Kubernetes node. For deployment I didn’t use official Apache Ignite docker image because it does not expose port 8080 which is the rest port we will use. Other than that the image is same with the offical one. We use default example cache configuration and enabled the rest module to be able to use it.
So create a file ignite.yaml (or whatever you like) with the above content. Then run the following:
$ kubectl create -f ignite.yaml
service "ignite" created
deployment "ignite" created
That creates the deployment and the service which we defined in our ignite.yaml file. To see the details of our deployment we can run:
$ kubectl describe deployment ignite
Name: ignite
Namespace: default
CreationTimestamp: Fri, 16 Mar 2018 00:43:03 +0300
Labels: app=ignite
Annotations: deployment.kubernetes.io/revision=1
Selector: app=ignite
Replicas: 4 desired | 4 updated | 4 total | 4 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=ignite
Containers:
ignite:
Image: alpert/ignite
Port: 8080/TCP
Environment:
CONFIG_URI: https://raw.githubusercontent.com/apache/ignite/master/examples/config/example-cache.xml
OPTION_LIBS: ignite-rest-http
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: ignite-66bb989784 (4/4 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 4s deployment-controller Scaled up replica set ignite-66bb989784 to 4
As you can see we have 4 replications of our application. Also you can see pod details by running:
$ kubectl describe pods ignite
Name: ignite-66bb989784-68qb5
Namespace: default
Node: minikube/192.168.99.100
Start Time: Fri, 16 Mar 2018 00:43:03 +0300
Labels: app=ignite
pod-template-hash=2266545340
Annotations: <none>
Status: Running
IP: 172.17.0.4
Controlled By: ReplicaSet/ignite-66bb989784
Containers:
ignite:
Container ID: docker://907cdf40c560f01a7bc7501dc0e06c63e56d295b32f4de0c1ee402a7775a7b09
Image: alpert/ignite
Image ID: docker://sha256:1d00e63317f649c592686c063cb8626821278b141327b3752790b672c6e1ad22
Port: 8080/TCP
State: Running
Started: Fri, 16 Mar 2018 19:11:27 +0300
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Fri, 16 Mar 2018 00:43:04 +0300
Finished: Fri, 16 Mar 2018 09:36:11 +0300
Ready: True
Restart Count: 1
Environment:
CONFIG_URI: https://raw.githubusercontent.com/apache/ignite/master/examples/config/example-cache.xml
OPTION_LIBS: ignite-rest-http
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-zrsd5 (ro)
............................
which has a very long and detailed output so I skip it. You can also describe service details as:
$ kubectl describe service ignite
Name: ignite
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=ignite
Type: NodePort
IP: 10.107.88.245
Port: <unset> 8080/TCP
TargetPort: 8080/TCP
NodePort: <unset> 32036/TCP
Endpoints: 172.17.0.4:8080,172.17.0.5:8080,172.17.0.6:8080 + 1 more...
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
You can also see all these and much more information on the dashboard. Everything seems OK on Kubernetes side, now we can check if our Ignite cluster is operating well. To see the logs first we will get the names of pods:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
ignite-66bb989784-68qb5 1/1 Running 1 18h
ignite-66bb989784-7cqpv 1/1 Running 1 18h
ignite-66bb989784-8z4v5 1/1 Running 1 18h
ignite-66bb989784-sw7hk 1/1 Running 1 18h
And then see the logs:
$ kubectl logs ignite-66bb989784-sw7hk
[16:11:51] __________ ________________
[16:11:51] / _/ ___/ |/ / _/_ __/ __/
[16:11:51] _/ // (7 7 // / / / / _/
[16:11:51] /___/\___/_/|_/___/ /_/ /___/
[16:11:51]
[16:11:51] ver. 2.4.0#20180305-sha1:aa342270
[16:11:51] 2018 Copyright(C) Apache Software Foundation
[16:11:51]
[16:11:51] Ignite documentation: http://ignite.apache.org
[16:11:51]
[16:11:51] Quiet mode.
[16:11:51] ^-- Logging to file '/opt/ignite/apache-ignite-fabric-2.4.0-bin/work/log/ignite-d3d4519a.0.log'
[16:11:51] ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[16:11:51] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[16:11:51]
[16:11:51] OS: Linux 4.9.64 amd64
[16:11:51] VM information: OpenJDK Runtime Environment 1.8.0_151-8u151-b12-1~deb9u1-b12 Oracle Corporation OpenJDK 64-Bit Server VM 25.151-b12
[16:11:53] Configured plugins:
[16:11:53] ^-- None
[16:11:53]
[16:11:53] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[16:11:53] Security status [authentication=off, tls/ssl=off]
[16:12:12] Nodes started on local machine require more than 20% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=1481MB, available=2000MB]
[16:12:20] Performance suggestions for grid (fix if possible)
[16:12:20] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[16:12:20] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
[16:12:20] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[16:12:20] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
[16:12:20] ^-- Decrease number of backups (set 'backups' to 0)
[16:12:20] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning
[16:12:20]
[16:12:20] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[16:12:20]
[16:12:20] Ignite node started OK (id=d3d4519a)
[16:12:20] Topology snapshot [ver=4, servers=4, clients=0, CPUs=8, offheap=1.6GB, heap=4.0GB]
[16:12:20] Data Regions Configured:
[16:12:20] ^-- default [initSize=256.0 MiB, maxSize=400.0 MiB, persistenceEnabled=false]
Everything seems OK here, too. In the logs above there is a line: [16:11:51] ^-- Logging to file '/opt/ignite/apache-ignite-fabric-2.4.0-bin/work/log/ignite-d3d4519a.0.log' which specify the actual log file location. You can also tail it by running:
$ kubectl exec -it ignite-66bb989784-sw7hk -- tail /opt/ignite/apache-ignite-fabric-2.4.0-bin/work/log/ignite-d3d4519a.0.log
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=d3d4519a, uptime=00:29:00.388]
^-- H/N/C [hosts=4, nodes=4, CPUs=8]
^-- CPU [cur=0.33%, avg=1.25%, GC=0%]
^-- PageMemory [pages=1232]
^-- Heap [used=63MB, free=93.58%, comm=981MB]
^-- Non heap [used=54MB, free=96.44%, comm=55MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
Here, from the metrics information, you can see that we have 4 nodes in our Ignite cluster: ^-- H/N/C [hosts=4, nodes=4, CPUs=8]
Using Ignite REST API
To access our Ignite cluster, we need the service endpoint. We will get the endpoint for our service as:
$ minikube service ignite --url
http://192.168.99.100:32036
$ export IGNITE_EP=$(minikube service ignite --url)
We will use that endpoint to make rest calls to Ignite. You can check REST API documentation for Ignite from here:https://apacheignite.readme.io/docs/rest-api. Let’s begin with a simple one and get topology information:
$ curl $IGNITE_EP/ignite\?cmd\=top
{
"successStatus":0,
"error":null,
"sessionToken":null,
"response":[
{
"nodeId":"8c6d5b27-6195-46f8-8afa-87d7245ee846",
"consistentId":"127.0.0.1,172.17.0.2:47500",
"tcpHostNames":[
"ignite-66bb989784-8z4v5"
],
"tcpPort":11211,
"metrics":null,
"caches":[
{
"name":"default",
"mode":"PARTITIONED",
"sqlSchema":null
}
],
"order":1,
"attributes":null,
"tcpAddresses":[
"172.17.0.2",
"127.0.0.1"
]
},
{
"nodeId":"17da1e88-27f8-43a2-954d-d345c90e561e",
"consistentId":"127.0.0.1,172.17.0.7:47500",
"tcpHostNames":[
"ignite-66bb989784-7cqpv"
],
"tcpPort":11211,
"metrics":null,
"caches":[
{
"name":"default",
"mode":"PARTITIONED",
"sqlSchema":null
}
],
"order":2,
"attributes":null,
"tcpAddresses":[
"172.17.0.7",
"127.0.0.1"
]
},
{
"nodeId":"2d578550-2898-4449-8c37-9fc7e2022c5d",
"consistentId":"127.0.0.1,172.17.0.4:47500",
"tcpHostNames":[
"ignite-66bb989784-68qb5"
],
"tcpPort":11211,
"metrics":null,
"caches":[
{
"name":"default",
"mode":"PARTITIONED",
"sqlSchema":null
}
],
"order":3,
"attributes":null,
"tcpAddresses":[
"172.17.0.4",
"127.0.0.1"
]
},
{
"nodeId":"d3d4519a-8543-45b9-9bc8-71dc896987ad",
"consistentId":"127.0.0.1,172.17.0.6:47500",
"tcpHostNames":[
"ignite-66bb989784-sw7hk"
],
"tcpPort":11211,
"metrics":null,
"caches":[
{
"name":"default",
"mode":"PARTITIONED",
"sqlSchema":null
}
],
"order":4,
"attributes":null,
"tcpAddresses":[
"127.0.0.1",
"172.17.0.6"
]
}
]
}
As you can see we have a default cache named default. Lets create another one named test:
$ curl $IGNITE_EP/ignite\?cmd\=getorcreate\&cacheName\=test
{"successStatus":0,"error":null,"sessionToken":null,"response":null}%
and put and get a value:
$ curl $IGNITE_EP/ignite\?cmd\=put\&key\=key\&val\=value\&cacheName\=test
{"successStatus":0,"affinityNodeId":"d3d4519a-8543-45b9-9bc8-71dc896987ad","error":null,"sessionToken":null,"response":true}%
$ curl $IGNITE_EP/ignite\?cmd\=get\&key\=key\&cacheName\=test
{"successStatus":0,"affinityNodeId":"d3d4519a-8543-45b9-9bc8-71dc896987ad","error":null,"sessionToken":null,"response":"value"}%
So far so good. From the documentation above you can see all posible REST commands. Happy hacking!