It goes without saying that if you want to orchestrate containers at this point, Kubernetes is what you use to do it. Sure, there be a few Docker Swarm holdouts still around, but for the most part, K8s has cemented itself as the industry standard for container orchestration solutions. As Kubernetes matures, the tools that embody its landscape begin to mature along with it. One of the areas we have seen some optimization, in particular, is in cloud-native storage solutions. may What is Rook? Rook is an open-source , providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments. It turns storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling, and orchestration platform to perform its duties. It supports a variety of different , and type storage. Rook integrates deeply into cloud-native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience. cloud-native storage orchestrator block object, file What is Ceph? Ceph is a distributed storage system that is massively scalable and high-performing with no single point of failure. Ceph is a Software Distributed System (SDS), meaning it can be run on any hardware that matches its requirements. Ceph consists of multiple components: are responsible for forming cluster quorums. All the cluster nodes report to monitor nodes and share information about every change in their state. Ceph Monitors (MON) are responsible for storing objects on local file systems and providing access to them over the network. Usually, one OSD daemon is tied to one physical disk in your cluster. Ceph Object Store Devices (OSD) Ceph clients interact with OSDs directly. provides additional monitoring and interfaces to external monitoring and management systems. Ceph Manager (MGR) are at the core of Ceph storage clusters. This layer makes sure that stored data always remains consistent and performs data replication, failure detection, and recovery among others. Reliable Autonomic Distributed Object Stores (RADOS) To read/write data from/to a Ceph cluster, a client will first contact Ceph MONs to obtain the most recent copy of their . The cluster map contains the cluster topology as well as the data storage locations. Ceph clients use the cluster map to figure out which OSD to interact with and initiate a connection with the associated OSD. cluster map Rook enables Ceph storage systems to run on Kubernetes using Kubernetes primitives. The following image illustrates how Ceph Rook integrates with Kubernetes: With Ceph running in the Kubernetes cluster, Kubernetes applications can mount block devices and filesystems managed by Rook, or can use the S3/Swift API for object storage. The Rook operator automates configuration of storage components and monitors the cluster to ensure the storage remains available and healthy. The Rook operator is a simple container that has all that is needed to bootstrap and monitor the storage cluster. The operator will start and monitor , the Ceph OSD daemons to provide RADOS storage, as well as start and manage other Ceph daemons. The operator manages CRDs for pools, object stores (S3/Swift), and filesystems by initializing the pods and other artifacts necessary to run the services. Ceph monitor pods The operator will monitor the storage daemons to ensure the cluster is healthy. Ceph mons will be started or failed over when necessary, and other adjustments are made as the cluster grows or shrinks. The operator will also watch for desired state changes requested by the api service and apply the changes. The image includes all necessary tools to manage the cluster – there are no changes to the data path. Rook does not attempt to maintain full fidelity with Ceph. Many of the Ceph concepts like placement groups and crush maps are hidden so you don’t have to worry about them. Instead, Rook creates a much simplified UX for admins that is in terms of physical resources, pools, volumes, filesystems, and buckets. At the same time, an advanced configuration can be applied when needed with the Ceph tools. rook/ceph Rook is implemented in Golang. Ceph is implemented in C++ where the data path is highly optimized. We believe this combination offers the best of both worlds. Getting Started This guide will walk you through the basic setup of a Ceph cluster and enable you to consume block, object, and file storage from other pods running in your cluster. Minimum Version Kubernetes or higher is supported by Rook. v1.10 Prerequisites To make sure you have a Kubernetes cluster that is ready for , you can . Rook follow these instructions If you are using to persist rook data on Kubernetes hosts, make sure your host has at least 5GB of space available on the specified path. dataDirHostPath TL;DR If you’re feeling lucky, a simple Rook cluster can be created with the following kubectl commands and . For the more detailed install, skip to the next section to . example yaml files deploy the Rook operator git --single-branch --branch release-1.2 https://github.com/rook/rook.git cluster/examples/kubernetes/ceph
kubectl create -f common.yaml
kubectl create -f operator.yaml
kubectl create -f cluster-test.yaml clone cd After the cluster is running, you can create storage to be consumed by other applications in your cluster. block, object, or file Production Environments For production environments, it is required to have local storage devices attached to your nodes. In this walkthrough, the requirement of local storage devices is relaxed so you can get a cluster up and running as a “test” environment to experiment with Rook. A Ceph filestore OSD will be created in a instead of requiring a device. directory For production environments, you will want to follow the example in instead of in order to configure the devices instead of test directories. See the for more details. cluster.yaml cluster-test.yaml Ceph examples Deploy the Rook Operator The first step is to deploy the Rook operator. Check that you are using the that correspond to your release of Rook. For more options, see the . example yaml files examples documentation cluster/examples/kubernetes/ceph
kubectl create -f common.yaml
kubectl create -f operator.yaml kubectl -n rook-ceph get pod cd ## verify the rook-ceph-operator is in the `Running` state before proceeding You can also deploy the operator with the . Rook Helm Chart Create a Rook Ceph Cluster Now that the Rook operator is running we can create the Ceph cluster. For the cluster to survive reboots, make sure you set the property that is valid for your hosts. For more settings, see the documentation on . dataDirHostPath configuring the cluster Save the cluster spec as : cluster-test.yaml apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion: image: ceph/ceph:v14.2.5
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
  dashboard:
    enabled: storage:
    useAllNodes: useAllDevices: directories:
    - path: /var/lib/rook # For the latest ceph images, see https://hub.docker.com/r/ceph/ceph/tags true true false # Important: Directories should only be used in pre-production environments Create the cluster: kubectl create -f cluster-test.yaml Use to list pods in the namespace. You should be able to see the following pods once they are all running. The number of OSD pods will depend on the number of nodes in the cluster and the number of devices and directories configured. If you did not modify the above, it is expected that one OSD will be created per node. kubectl rook-ceph cluster-test.yaml The and pods are also optional depending on your settings. rook-ceph-agent rook-discover $ kubectl -n rook-ceph get pod
NAME                                   READY   STATUS      RESTARTS   AGE
rook-ceph-agent-4zkg8                  1/1     Running     0          140s
rook-ceph-mgr -d9dcf5748-5s9ft        1/1     Running     0          77s
rook-ceph-mon -7d8f675889-nw5pl       1/1     Running     0          105s
rook-ceph-mon-b-856fdd5cb9-5h2qk       1/1     Running     0          94s
rook-ceph-mon-c-57545897fc-j576h       1/1     Running     0          85s
rook-ceph-operator-6c49994c4f-9csfz    1/1     Running     0          141s
rook-ceph-osd-0-7cbbbf749f-j8fsd       1/1     Running     0          23s
rook-ceph-osd-1-7f67f9646d-44p7v       1/1     Running     0          24s
rook-ceph-osd-2-6cd4b776ff-v4d68       1/1     Running     0          25s
rook-ceph-osd-prepare-node1-vx2rz      0/2     Completed   0          60s
rook-ceph-osd-prepare-node2-ab3fd      0/2     Completed   0          60s
rook-ceph-osd-prepare-node3-w4xyz      0/2     Completed   0          60s
rook-discover-dhkb8                    1/1     Running     0          140s -a -a To verify that the cluster is in a healthy state, connect to the and run the command. Rook toolbox ceph status All mons should be in quorum A should be active mgr At least one OSD should be active If the health is not , the warnings or errors should be investigated HEALTH_OK $ ceph status
  cluster:
    id:     a0452c76-30d9-4c1a-a948-5d8405f19a7c
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 3m)
    mgr: a(active, since 2m)
    osd: 3 osds: 3 up (since 1m), 3 (since 1m)
... in Create a Block Storage Block storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook. Prerequisites This guide assumes a Rook cluster as explained in the . Quickstart Provision Storage Before Rook can provision storage, a and need to be created. This will allow Kubernetes to interoperate with Rook when provisioning persistent volumes. StorageClass CephBlockPool Save this definition as : StorageClass storageclass.yaml apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block provisioner: rook-ceph.rbd.csi.ceph.com
parameters: clusterID: rook-ceph pool: replicapool imageFormat: imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph csi.storage.k8s.io/fstype: xfs reclaimPolicy: Delete # Change "rook-ceph" provisioner prefix to match the operator namespace if needed # clusterID is the namespace where the rook cluster is running # Ceph pool into which the RBD image shall be created # RBD image format. Defaults to "2". "2" # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature. # The secrets contain Ceph admin credentials. # Specify the filesystem type of the volume. If not specified, csi-provisioner # will set default as `ext4`. # Delete the rbd volume when a PVC is deleted If you’ve deployed the Rook operator in a namespace other than “rook-ceph” as is common change the prefix in the provisioner to match the namespace you used. For example, if the Rook operator is running in “rook-op” the provisioner value should be “rook-op.rbd.csi.ceph.com”. Create the storage class. kubectl create -f cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml Consume the storage: Wordpress sample We create a sample app to consume the block storage provisioned by Rook with the classic Wordpress and Mysql apps. Both of these apps will make use of block volumes provisioned by Rook. Start Mysql and Wordpress from the folder: cluster/examples/kubernetes kubectl create -f mysql.yaml
kubectl create -f wordpress.yaml Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following: $ kubectl get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE
mysql-pv-claim   Bound     pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee   20Gi       RWO           1m
wp-pv-claim      Bound     pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee   20Gi       RWO           1m Once the Wordpress and Mysql pods are in the state, get the cluster IP of the Wordpress app, and enter it in your browser: Running $ kubectl get svc wordpress
NAME        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
wordpress   10.3.0.155   <pending>     80:30841/TCP   2m You should see the Wordpress app running. If you are using Minikube, the Wordpress URL can be retrieved with this one-line command: http://$(minikube ip):$(kubectl get service wordpress -o jsonpath= ) echo '{.spec.ports[0].nodePort}' : When running in a vagrant environment, there will be no external IP address to reach Wordpress with. You will only be able to reach Wordpress via the from inside the Kubernetes cluster. NOTE CLUSTER-IP Moving On Now that you've had a chance to play around a bit with using Rook with Ceph, let's move on to the next part and learn how to set up Minio object storage with Rook! About the author - Sudip is a Solution Architect with more than 15 years of working experience, and is the founder of FlyBHP . He likes sharing his knowledge by regularly writing for Hackernoon , DZone , Appfleet and many more. And while he is not doing that, he must be fishing or playing chess. Previously posted at https://appfleet.com/ .

The Tech Writer

Read My Stories

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

How To Deploy A Scalable Cloud Storage Using Rook And Ceph Storage [Part 1]

Too Long; Didn't Read

Company Mentioned

Sudip Sengupta

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

How To Deploy A Scalable Cloud Storage Using Rook And Ceph Storage [Part 1]

Too Long; Didn't Read

Company Mentioned

Sudip Sengupta

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES