Kubernetes is one of the popular container orchestration tools. It automates the deployment, scaling and management of containerized (usually docker) applications. It’s so flexible that it let you run your apps on-premises, hybrid, public/private cloud infrastructure (e.g. google cloud platform, GCP or Amazon Web Services, AWS). It’s developed and open-sourced by google based on their decade’s worth of experience building container-management.

Kubernetes is still under active development. Currently its latest release is 1.5.1. One of most important beta feature for this release is the StatefulSets (previously known for PetSets). StatefulSets wass developed due to the need to persist the data storage even if a pod die/restarted, new pod which replaces the old pod should use back the same persistent data storage. There is a unique identity between the pod and the persistent data that the kubernetes is aware of. This aid in the term of scaling the cassandra instance without the need to keep track of the storage attached to it.

For this post, we will try to build a multi data centers cluster of cassandra using both statefulsets and replication controls. The replications controls will build a cluster which only consists of a single node for ease of data backup purpose. So it’s not strictly required and you can use just StatefulSets.

Let set up the service + Replication control which will serve as seed nodes to bootstrap another Cassandra data center. We will change from replication controller example given by the kubernetes

cassandra-controller.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandrabk
  name: cassandrabk    ## this name will be used as the seed ip in another cluster
spec:
  clusterIP: None
  ports:
    - port: 9042
      port: 7000        ## we added this for intra-node communication
  selector:
    app: cassandrabk
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: cassandrabk
  # The labels will be applied automatically
  # from the labels in the pod template, if not set
  # labels:
    # app: cassandra
spec:
  replicas: 1
  # The selector will be applied automatically
  # from the labels in the pod template, if not set.
  # selector:
      # app: cassandra
  template:
    metadata:
      labels:
        app: cassandrabk
    spec:
      containers:
        - command:
            - /run.sh
          resources:
            limits:
              cpu: 100m
              memory: 768Mi
          env:
            - name: MAX_HEAP_SIZE
              value: 256M
            - name: HEAP_NEWSIZE
              value: 100M
            - name: CASSANDRA_SERVICE
              ## need to set this value as we are not using default "cassandra" for the end points to populate ip     
              value: cassandrabk
            - name: CASSANDRA_SEED_PROVIDER
              value: "io.k8s.cassandra.KubernetesSeedProvider"
            - name: CASSANDRA_CLUSTER_NAME
              value: "mycluster-demo"
            - name: CASSANDRA_DC
              value: "dc-bk"
            - name: CASSANDRA_RACK
              value: "rack01"
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          image: gcr.io/google-samples/cassandra:v11
          name: cassandrabk
          ports:
            - containerPort: 7000
              name: intra-node
            - containerPort: 7001
              name: tls-intra-node
            - containerPort: 7199
              name: jmx
            - containerPort: 9042
              name: cql
          volumeMounts:
          - name: cs-host-bk
            mountPath: /cassandra_data
      volumes:
        - name: cs-host-bk
          hostPath:
            path: /home/cassandra/cs-bk

You may notice that we are setting the memory and cpu to a lower value as we try to put everything into a single node for this demo. You should use the recommended value. Let create the resources.

kubectl create -f cassandra-controller.yaml

Now, you may now start populating data into this new C* node.

Let set up the service + StatefulSets yaml file. Again, We will change from the sample given by the kubernetes. A few things to take note:

  • we will first set the auto-bootstrap to “false”. Once the node has restored the data successfully, we will set it to “true”
  • set the seed to service name to the one we just set up (i.e. cassandrabk)

cassandra-statefulsets.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
    - port: 9042
      port: 7000
  selector:
    app: cassandra
---
apiVersion: "apps/v1beta1"
kind: StatefulSet
metadata:
  name: cassandra
spec:
  serviceName: cassandra
  replicas: 1
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v11
        # imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: 100m
            memory: 768Mi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        env:
          - name: MAX_HEAP_SIZE
            value: 256M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            # value: "cassandra-0.cassandra.default.svc.cluster.local"
            value: cassandrabk
          - name: CASSANDRA_CLUSTER_NAME
            value: "mycluster-demo"
          - name: CASSANDRA_DC
            value: "dc01"
          - name: CASSANDRA_RACK
            value: "rack01"
          - name: CASSANDRA_AUTO_BOOTSTRAP
            value: "false"  # when insert into new center
            # value: "true"     # after data has been populated
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 300
          timeoutSeconds: 30
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1G

Once the StatefulSets cassandra data center is live, we can create the database schema for it.

CREATE KEYSPACE mydatabase WITH replication = {'class': 'NetworkTopologyStrategy', 'dc01': 1, 'dc-bk': 1}

Remember to do the same for the cassandrabk cluster.

You can check the

nodetool status

to make sure you see both data centers can discover each other.

[email protected]:/# nodetool status
Datacenter: dc-bk
============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID     Rack
UN  10.0.2.10  241.81 KiB  32           100.0%            xxxx-xxxx  rack01

Datacenter: dc01
===============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID     Rack
UN  10.0.2.11 427.32 KiB  32           100.0%            yyyy-yyyy  rack01

Now you are run the

nodetool rebuild -- cassandrabk

to get your data populated!

If you met error “unable to source for streaming range”, make sure to check on this page. Altering the keyspace to use network topology with proper replication factor will fix it.

We hope you have learned something and enjoyed this post. Do follow our Facebook (update more often) and our Twitter . We will continue to share more on latest technology update & usage besides showcasing our work.

References:

Posted in Cassandra, Docker, Kubernetes, web development on Dec 22, 2016

0 Comment

Leave a Comment

Your Email address will not be published. Required fields are marked *

About IT Wonders:

IT Wonders is a web agency based in Johor Bahru (JB), Malaysia and Singapore. We provide responsive and custom websites unique to your needs. If you have any inquiries about your website, do not hesitate to contact us.

YOU MIGHT BE INTERESTED:
TOPICS