Storage for RHV and OCP: Two Glusters on one platform

Architecture is an interesting discipline. There are whitepapers and best practices and reference architectures to offer pristine views of what your perfect deployment should look like. And then there are budgets and timelines and business requirements to derail all of that. It’s what makes this job so interesting and challenging—hacking together the best pieces of disparate and often seemingly unrelated systems to meet goals driven by six leaders whose bonuses are met by completely different metrics.

A recent project has involved combining OpenShift Container Platform (OCP), Red Hat Virtualization (RHV), and Red Hat Gluster Storage (Gluster) into a unified system with common lifecycle operations, minimized management points, and the lowest overall footprint in terms of both capital cost and TCO. The primary storage challenge here is in creating a Gluster environment to support both RHV and its VMs as well as OCP container persistent volume requirements.

Our architectural goals include:

  • Purchase a single flexible hardware platform to serve all the storage needs
  • Segregate Gluster for RHV and Gluster for OCP into separate pools for resource allocation and to avoid possible administration snafus (such as we experienced in early testing)
  • Maintain a single-point and single-method of management—one Heketi server to rule them all
  • Containerize as much as possible to keep lifecycle maintenance atomic

Our early version of the architecture had Gluster running as container-native storage (CNS) for OCP on top of RHV while also serving storage to RHV, but this proved to introduce a chicken-and-egg problem where a single failure (such as an etcd crash) could cause a cascading outage. So our redesign involved splitting Gluster off from OCP as a stand-alone system while still being a unified storage provider and leveraging container atomicity.

The approach we wanted involved containerized Gluster running on bare-metal container hosts. Fundamentally, this is actually pretty straightforward today with pre-build Gluster containers available from the Red Hat registry. What complicated this was our desire to run two separate containerized Gluster pools on the same hardware nodes.


There’s a pretty good chance that this architecture is not explicitly supported by Red Hat. While all the components we use here are definitely supported, this particular combination is untested by our engineering, QE, and performance teams. Don’t consider anything here a recommendation for how you should run your environment, only an academic study of a possible approach to solving an interesting challenge. If you have any questions, please reach out to your Red Hat sales and support teams.

The platform

We initially wanted to build this on top of Red Hat Enterprise Linux Atomic Host, but our lab environment wasn’t setup to provision this build on our systems, so we had to go forward with RHEL plus the docker packages. For a production build, we would return to using Atomic.


Gluster containers are usually configured with host networking because they need to communicate freely with each other and need to serve storage out to other systems and containers. However, with host networking, the Gluster ports are bound to all interfaces, so it is not possible to run two Gluster containers in this mode due to port conflicts. To solve this, the networks for each Gluster pool had to be segregated.

First, a VLAN sub-interface was created on each Gluster node for the storage network interface and using VLAN ID 199. There are ifcfg files to make these persistent. So each node includes a IP on the primary interface and a IP on a VLAN sub-interface. The Switch ports for the storage network interfaces have been configured for the tagged VLAN ID 199. The 802.1q kernel module (for VLANs) was set to load at boot time on each node with a /etc/modules-load.d/8021q.conf file.

Containerized Gluster


Each Gluster container needs to exist on its own interface and subnet. So leveraging the system-level network stuff done above, the two interfaces were each attached to a docker macvlan network on each node.

docker network create -d macvlan --subnet= \

-o parent=eth1 gluster-rhv-net
docker network create -d macvlan --subnet= \

-o parent=eth1.199 gluster-ocp-net


The containers were pulled down from the Red Hat registry.

docker pull
docker pull

The Gluster containers need to be privileged in order to access the /dev/sdX block devices. They also need a number of local persistent volume stores in order to ensure they start up properly each time.

The container fstab file needs a persistent mount. So first we should touch these files, otherwise the gluster-startup command in the container will fail.

touch /var/lib/heketi-{rhv,ocp}/fstab

Then we can run the containers.

docker run -d --privileged=true --net=gluster-rhv-net \

--ip=  --name=gluster-rhv-1 -v /run \

-v /home/gluster-rhv-1-root:/root:z \

-v /etc/glusterfs-rhv:/etc/glusterfs:z \

-v /var/lib/glusterd-rhv:/var/lib/glusterd:z \

-v /var/log/glusterfs-rhv:/var/log/glusterfs:z \

-v /var/lib/heketi-rhv:/var/lib/heketi:z \

-v /sys/fs/cgroup:/sys/fs/cgroup:ro \

-v /dev:/dev rhgs3/rhgs-server-rhel7
docker run -d --privileged=true --net=gluster-ocp-net \

--ip= --name=gluster-ocp-1 -v /run \

-v /home/gluster-ocp-1-root:/root:z \

-v /etc/glusterfs-ocp:/etc/glusterfs:z \

-v /var/lib/glusterd-ocp:/var/lib/glusterd:z \

-v /var/log/glusterfs-ocp:/var/log/glusterfs:z \

-v /var/lib/heketi-ocp:/var/lib/heketi:z \

-v /sys/fs/cgroup:/sys/fs/cgroup:ro \

-v /dev:/dev rhgs3/rhgs-server-rhel7

Block device assignments

Running the containers in privileged mode allows them to access all system block devices. For our particular architectural needs, we intend to use from each node only one SSD for the gluster-rhv pool and the remaining five SSDs for the gluster-ocp pool.

 Gluster Pool  Block Devices
 gluster-rhv  sdb
 gluster-ocp  sdc, sdd, sde, sdf, sdg



The persistent Heketi config is being stored in the /etc/heketi directory on one of the nodes (we’ll call it node1). First, an ssh keypair is created and placed there.

ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''

Next, the heketi.json file is created. Right now, no auth is being used — obviously don’t do this in production. Note the ssh port is 2222, which is what the Gluster containers are configured to listen on.

  "_port_comment": "Heketi Server Port Number",
  "port": "8080",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": false,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "My Secret"
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "My Secret"

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    "executor": "ssh",

    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/etc/heketi/heketi_key",
      "user": "root",
      "port": "2222"

    "_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",

    "_loglevel_comment": [
      "Set log level. Choices are:",
      "  none, critical, error, warning, info, debug",
      "Default is warning"
    "loglevel" : "debug"

SSH access

The Heketi server needs passwordless SSH access to all Gluster containers on port 2222. The public key generated above needs to be added to the authorized_keys for all of the Gluster containers. Note that we have a local persistent volume (PV) for each Gluster container’s /root directory, so this authorized_key entry was simply added to each one of those.

cat /etc/heketi/ >> \


NOTE: This needs to be done for each of the root home directories for each Gluster container


The single Heketi container will run on node1. It needs access to both of the subnets, so the best thing to do is run the container in host networking mode. It also needs a few persistent volumes.

docker run -d --net=host --name=gluster-heketi \

-v /etc/heketi:/etc/heketi:z -v /var/lib/heketi:/var/lib/heketi:z \



Since we are running heketi-cli on the same node that we are running the Heketi container, there is a security issue we have to work through. By default, the container host cannot directly access the local container via the IP assigned to its macvlan network interface. So on the container host node1 we need to create local macvlan interfaces for each of the subnets. Use this at runtime and the /etc/rc.d/rc.local file:

/usr/sbin/ip link add macvlan0 link eth1 type macvlan mode bridge
/usr/sbin/ip addr add dev macvlan0
/usr/sbin/ifconfig macvlan0 up

/usr/sbin/ip link add macvlan1 link eth1.199 type macvlan mode bridge
/usr/sbin/ip addr add dev macvlan1
/usr/sbin/ifconfig macvlan1 up

The rc.local file in RHEL is for legacy support, so it has to be made executable and its systemd service has to be enabled.

chmod 755 /etc/rc.d/rc.local
systemctl enable rc-local.service

Heketi CLI

The heketi-cli needs to run $somewhere. For simplicity, the RPM is installed on node1. With the container running with networking in host mode, heketi is listening on localhost port 8080. Export the environment variable in order to be able to run heketi-cli commands.

export HEKETI_CLI_SERVER=http://localhost:8080

Setting up the Heketi clusters

A JSON file is populated at /root/heketi-rhv-plus-ocp-topology.json on node1. This file defines two separate Heketi clusters with their respective Gluster nodes (containers) and block devices.

    "clusters": [
            "nodes": [
                    "node": {
                        "hostnames": {
                            "manage": [
                            "storage": [
                        "zone": 1
                    "devices": [
                    "node": {
                        "hostnames": {
                            "manage": [
                            "storage": [
                        "zone": 2
                    "devices": [
                    "node": {
                        "hostnames": {
                            "manage": [
                            "storage": [
                        "zone": 3
                    "devices": [

            "nodes": [
                    "node": {
                        "hostnames": {
                            "manage": [
                            "storage": [
                        "zone": 1
                    "devices": [
                    "node": {
                        "hostnames": {
                            "manage": [
                            "storage": [
                        "zone": 2
                    "devices": [
                    "node": {
                        "hostnames": {
                            "manage": [
                            "storage": [
                        "zone": 3
                    "devices": [

This file is passed (once) to Heketi to setup the two clusters.

heketi-cli topology load --json=heketi-rhv-plus-ocp-topology.json

It’s important to note the two different clusters. It’s not (AFAIK) possible to “name” the clusters, so we have to reference them by their UUIDs. The Gluster volumes for RHV will be created on one cluster, and those orchestrated for OCP PVs will be created on a different cluster.

RHV Gluster volumes

For the purposes of RHV, two volumes were requested—one for the Hosted Engine and one for the VM storage. These were created via heketi-cli. Note the cluster ID passed to the commands.

heketi-cli volume create --size 100 --name rhv-hosted-engine \

--clusters ae2a309d02781816adfed567693221a9
heketi-cli volume create --size 1024 --name rhv-virtual-machines \

--clusters ae2a309d02781816adfed567693221a9

These can be mounted to the RHV nodes via the subnet using the Gluster native client. Example fstab entries:      /100g   glusterfs       backupvolfile-server= 0 0      /1t   glusterfs       backupvolfile-server= 0 0

OCP PV Gluster volumes

Our OCP pods are attached to the subnet to communicate with the storage. First on node1 the Heketi API port (8080) needs to be opened in the firewall.

firewall-cmd --add-port 8080/tcp
firewall-cmd --add-port 8080/tcp --permanent

Then the storage class for OCP is defined with the below YAML. Note that we aren’t currently doing any authentication (but obviously we should). You see here that we explicitly define the Heketi cluster ID for this class in order to ensure that all volumes for PVCs are created only on the Gluster pool we have identified for OCP use.

kind: StorageClass
 name: gluster-dyn
 resturl: ""
 restauthenabled: "false"
 clusterid: "74edade536c80f14486edfbabd204151"

Then the storage class is added to OCP on the master.

oc create -f glusterfs-storageclass.yaml

From this point, PVCs (persistent volume claims) made against this storage class will interface with Heketi to dynamically provision Gluster volumes to match the claim.


Auto-start containers

Docker container systemd init scripts are tricky. I’ve found that every example on the internet is either wrong, outdated, or uses an approach I don’t like.

Below is an example systemd service file for the Heketi container, which is simple and works the way we expect it to with the docker run command in the ExecStart (/etc/systemd/system/docker-container-gluster-heketi.service). NOTE: Do not daemonize (-d) the docker run command in the init script. Also, the SuccessExitStatus is important here.

Description=Gluster Heketi Container

SuccessExitStatus=0 137
ExecStartPre=-/usr/bin/docker stop gluster-heketi
ExecStartPre=-/usr/bin/docker rm gluster-heketi
ExecStart=/usr/bin/docker run --net=host --name=gluster-heketi -v /etc/heketi:/etc/heketi:z -v /var/lib/heketi:/var/lib/heketi:z rhgs3/rhgs-volmanager-rhel7
ExecStop=/usr/bin/docker stop gluster-heketi


Reload the systemd daemon:

systemctl daemon-reload

Enable and start the service

systemctl enable docker-container-gluster-heketi

systemctl start docker-container-gluster-heketi

Known issues and TODOs

  • Security needs to be taken into account. We’ll set up appropriate key-based authentication and JWT for Heketi. We’d also like to use role-based auth. Hopefully we’ll cover this in a future blog post.
  • Likely $other_things I haven’t realized yet, or better ways of approaching this. I’d love to hear your comments.

Introducing Red Hat Hyperconverged Infrastructure 1.0

By Steve Bohac, Red Hat Storage

Today we’re proud to announce Red Hat Hyperconverged Infrastructure 1.0! By combining Red Hat virtualization and storage technologies with a stable, proven operating platform, Red Hat Hyperconverged Infrastructure is designed to help enterprises bring datacenter capabilities into locations with limited space, such as branch offices and other remote facilities.

Built on Red Hat Virtualization and Red Hat Gluster Storage, Red Hat Hyperconverged Infrastructure provides simplified planning and procurement, streamlined deployment and management, and a single support stack for virtual compute and virtual storage resources. Red Hat Hyperconverged Infrastructure is an ideal solution for remote/branch office or edge computing needs. Deployment is enabled by Ansible by Red Hat, and Red Hat CloudForms can be used to manage all the Red Hat Hyperconverged Infrastructure installations in your enterprise via a single application.

Customers have been asking us for this type of an integrated solution, so we’re happy to offer this hyperconverged combination in a single SKU to satisfy that request.

Organizations with distributed operations, such as those in the banking, energy, or retail industries, can benefit from offering the same infrastructure services in remote and branch offices as they run in their datacenters. However, remote and branch offices can have unique challenges: less space and power/cooling and fewer (or no) technical staff on site. Organizations in this situation need powerful services, integrated on a single server that allow them to keep their key applications local to the remote site.

Red Hat Hyperconverged Infrastructure addresses these challenges for remote installations. The following figure depicts the benefits that consolidation with a hyperconverged infrastructure provides:

  • Eliminate storage as a discrete tier
  • Easily virtualize business applications, maximizing resource utilization
  • Single budget for compute and storage
  • Single team managing infrastructure
  • Simplified planning and procurement
  • Streamlined deployment and management
  • Single support stack for compute and storage

Removing the storage tier by consolidating compute and storage onto a single server platform/tier offers streamlined deployment and management (enabled by Ansible by Red Hat and Red Hat CloudForms), a single support stack (one vendor to call now instead of two), and simplified planning and procurement (reducing the number of vendors to source from).

For more information on Red Hat Hyperconverged Infrastructure, click here.

For an on-demand webinar discussing Red Hat Hyperconverged Infrastructure in more detail, click here.

Storage can make your digital transformation—or break it

By Ross Turk, Red Hat Storage

The following chart might look familiar, especially if you’ve ever studied patterns of online behavior.

Like all the best charts, this one has a glorious up-and-to-the-right shape. But each year at the end of December, when much of the world goes offline for a few quiet days, there’s a characteristic drop. This chart—from Google Trends—represents a phrase that’s growing in prominence: “digital transformation.”

Digital transformation is everywhere

When I noticed—with distinct déjà vu—the industry using this phrase, I admit I was somewhat taken aback. Many of us live in a world dominated by technology. I can’t remember the last time I paid for fast-food tacos with actual money, but I do know I stopped carrying cash completely when the taco shops began accepting credit cards. Every time I need to mail a letter now, it’s a huge production! I’m just not prepared for that kind of task anymore.

Imagine a world without digital technology…. See?! You can’t do it.

Not all digital transformation is equal

Another case in point. I renewed my driver’s license recently and found myself wondering: Now that the DMV is doing business using modern technology, who’s left to transform? If you live your life in an ivory tower made of wifi and capacitive touch screens—like me—a phrase like “digital transformation” can seem obsolete. It can throw you off the scent a bit. And, indeed, I was missing the point. Sure, even taxi companies embrace digital technologies these days…but are they any good at it? Do they enjoy the same efficiencies as a digitally native service like Uber?

Technology is now serious business

Digital transformation isn’t about using technology—or even offering digital services. It’s about redefining a business in technology terms, putting the modern technology experience first. It’s about businesses coming to terms with the truth: Technology can’t be a hobby for them anymore. They’re going to have a ton of applications and data, and they need to get really good at managing all of it. That means having solid priorities; agility and elasticity are a great place to start.

Modern storage can transform your business

Speaking of great places to start, there’s no better example of the challenges of digital transformation than storage. The amount of data that enterprises need to maintain is growing at a steady clip, and their customers expect all that data to be available instantly. Access patterns change as frequently as customer behaviors. Data is getting bigger, analytics are getting even more sophisticated. The traditional storage appliances that do a lot of the heavy lifting today are convenient, but at petabyte scale they show their inflexibility and limitation.

That’s where Ceph and Gluster come in. They’re flexible, scale-out, software-defined storage technologies built for those who don’t think storage is a hobby.

Learn how storage can make your digital transformation

If you want to learn more about modern approaches to storage—and Red Hat Storage, of course!—join me on June 22 for a 45-minute webinar. Register here.

Going public with Red Hat Ceph Storage 2.3

By Daniel Gilfix, Red Hat Storage

You may not have heard a lot about Red Hat Ceph Storage lately, but that doesn’t mean the product hasn’t been active. News in conjunction with Red Hat OpenStack Platform 11 and 10, technology partners such as Rackspace and Micron, and customer adoption at places like Massachusetts Open Cloud, UKCloud, and CLIMB have reinforced the product’s role as a cornerstone of the Red Hat portfolio. But the advances of the product, itself, have been relatively under wraps, with versions 2.2 and 2.1 carefully monitored by existing fans and loyal software-defined storage blog readers. We don’t expect the announcement of Red Hat Ceph Storage 2.3 to shake mountains with seismic impact, but we do expect it to inspire our user community with the doors the product opens today and what might be possible long term.

Source: Sheri Terris, from June 1, 2007,

Greater versatility

Red Hat Ceph Storage 2.3 takes aim at extending the versatility of object storage so that users can connect more successfully to traditional workloads and link them effectively to modern ones. One way is through our new Network File System (NFS) gateway into the product’s S3-compatible object interface. The gateway facilitates the adoption of Ceph Storage as a target for file-based data without requiring changes in data access protocols or the management of data caching semantics. It means Red Hat Ceph Storage users can access the same data set from both object and file interfaces and gradually transition between them based on business need. It also means they can extend the multi-site capabilities of Red Hat Ceph Storage to enable global clusters and data access with the NFS protocol.

Improved connectivity

Red Hat Ceph Storage 2.3 is laying the groundwork for improved connectivity with analytics engines. By conducting validation tests for compatibility with the Hadoop S3A plugin, Red Hat is extending financial and operational benefits of a scale-out, software-defined object storage platform to analytics workloads and data-driven applications leveraging tools like Hadoop, Hive, and Spark. With analytics data stored in an S3-compatible object store, developers have access to a broader ecosystem of tools and language bindings, no longer forced to use a bulky HDFS client. By concentrating data sets in a common object store, data duplication and data lineage challenges are simplified. Multiple ephemeral instances of elastic analytics clusters can reference a single source of truth.

Deployment flexibility

A final capability targets the highly coveted customer requirement of running Ceph in a containerized format. Red Hat Ceph Storage 2.3 includes a single container image that delivers the same capabilities as in the traditional package format. With a Red Hat Ansible-based deployment tool, users can perform installations, upgrades, and updates with atomicity for reduced complexity, easier management, and faster deployment. This supports customers in areas like telecommunications seeking standardized orchestration and deployment of infrastructure software in containers with Kubernetes by adding a cloud-native storage service to this orchestration methodology.

A new chapter

These new capabilities demonstrate the new heights Red Hat Ceph Storage aims to scale, supported by a company firmly committed to real-world deployment of the future of storage. By combining massive scalability with multi-protocol access to highly available clusters, Red Hat Ceph Storage is moving object storage up the mountain to help unleash its power for modern workloads. For more information, check out this short video:

Red Hat Ceph Storage 2.3 is expected to be available later this month.

Storage for the modern enterprise

In early May, Red Hat put on its annual customer and community conference in Boston—Red Hat Summit—centered around a common theme: the power of the individual. Now more than ever, open source is seen as the most viable option as we enter the next phase of IT delivery, planning, and deployment.

The power of the individual on display

Red Hat has assumed the role of de facto open source leader, driving and nurturing hundreds of communities across the world. One could argue that, at the core, Red Hat isn’t a software company at all. In fact, our best asset is our ability to curate open source communities, bringing to bear the efforts of thousands of contributors, committers, and testers to enterprises in a reliable, secure package that can solve some of the most demanding IT challenges.

The end of planning as we know it

Red Hat CEO, Jim Whitehurst, made a pertinent point in his keynote about the changing face of IT planning. “Planning harder” in an environment full of unknowns is complex and fraught with error. CIOs struggle to balance predictability with the inherent flexibility needed to maintain smooth IT operations.

Building IT infrastructure with open source and industry-standard constructs helps alleviate some of the planning challenges while addressing today’s pain points, much like interchangeable Lego® pieces can be used to build everything from two-story houses to skyscrapers, still withstanding the shock of an earthquake when needed.

Storage for the modern enterprise

With each passing year, it becomes apparent that traditional storage just isn’t going to cut it as enterprises look to create flexible, scalable, and cost-effective IT platforms for cloud-native applications. Simply put, legacy storage systems have failed to keep up with the way customers want to consume storage.

A key piece of the value proposition of software-defined storage is the hardware choice available to customers. For the second year in a row, a Storage Ecosystem Showcase in the partner pavilion of Red Hat Summit featured seven Technology Partners that complete or enhance Red Hat’s software-defined storage offering. Cisco, NGD Systems, Permabit, QCT, Seagate, Storage Made Easy, and Supermicro all demonstrated their wares.

In addition, several other storage partners, such as Dell EMC, Mellanox, and Penguin Computing, chose to sponsor their own booths. The solid upstream ecosystem combines with a growing downstream array of partners to truly differentiate Red Hat Storage.

Learn more

Storage was featured prominently both on the expo floor and in the news from the event. In addition to breakout sessions, Red Hat Storage engineers and consultants held a number of hands-on labs that were very well received. You can access similar self-paced material on the online AWS test drives (Gluster test drive and Ceph test drive) or at an upcoming webinar.

For a review of all the 2017 Red Hat Summit videos, click here. For a video recap with some of the Red Hat Storage team, watch this:

Library of Ceph and Gluster reference architectures – Simplicity on the other side of complexity

The Storage Solution Architectures team at Red Hat develops reference architectures, performance and sizing guides, and test drives for Gluster- and Ceph-based solutions. We’re a group of architects who perform lab validation, tuning, and interoperability development for composable storage services with target workloads on optimized server and network configurations. We seek simplicity on the other side of complexity.

At the end of this blog entry is a full library of our current publications and test drives.

In our modern era, a top company asset is pivotability. Pivotability based on external market changes. Pivotability after unknowns become known. Pivotability after golden ideas become dark alleys. For most enterprises, pivotability requires a composable technology infrastructure for shifting resources to meet changing needs. Composable storage services, such as those provided by Ceph and Gluster, are part of many companies’ composable infrastructures.

Composable technology infrastructures are most frequently described by the following attributes:

  • Open source v. closed development.
  • On-demand architectures v. fixed architectures.
  • Commodity hardware v. proprietary appliances.
  • Cross-industry collaboration v. isolated single-vendor silos.

As noted in the following figure, a few companies with large staffs of in-house experts can create composable infrastructures from raw technologies. Their large investments in in-house expertise allows them to convert raw technologies into solutions with limited pre-integration by technology suppliers. AWS, Google, and Azure are all examples of DIY businesses. A larger number of other companies, also needing composable infrastructures, rely on technology suppliers and the community for solution pre-integration and guidance to reduce their in-house expertise costs. We’ll label them “Assisted DIY.” Finally, the majority of global enterprises lack the in-house expertise for deploying these composable infrastructures. They rely on public cloud providers and pre-packaged solutions for their infrastructure needs. We’ll call them “Pre-packaged.”


The reference architectures, performance and sizing guides, and test drives produced by our team are primarily focused on the “Assisted DIY” segment of companies. Additionally, we strive to make Gluster and Ceph composable storage services available to the “Pre-packaged” segment of companies by using what we learn to produce pre-packaged combinations of Red Hat software with partner hardware targeting specific workload use cases.

We enjoy our roles at Red Hat because of the many of you with whom we collaborate to produce value.  We hope you find these guides useful.

Team-produced with partner collaboration:

Partner-produced with team collaboration:

Pre-packaged solutions:

Hands-on test drives:

Struggling to containerize stateful applications in the cloud? Here’s how.

The newest release of Red Hat’s Reference Architecture “OpenShift Container Platform 3.5 on Amazon Web Services” now incorporates container-native storage, a unique approach based on Red Hat Gluster Storage to avoid lock-in, enable stateful applications, and simplify those applications’ high availability.

In the beginning, everything was so simple. Instead of going through the bureaucracy and compliance-driven process of requesting compute, storage, and networking resources, I would pull out my corporate credit card and register at the cloud provider of my choice. Instead of spending weeks forecasting the resource needs and costs of my newest project, I would get started in less than 1 hour. Much lower risk, virtually no capital expenditure for my newest pet project. And seemingly endless capacity—well, as long as my credit card was covered. If my project didn’t turn out to be a thing, I didn’t end up with excess infrastructure, either.

Until I found out that basically what I was doing was building my newest piece of software against a cloud mainframe. Not directly, of course. I was still operating on top of my operating system with the libraries and tools of my choice, but essentially I spend significant effort getting to that point with regards to orchestration and application architecture. And these are not easily ported to another cloud provider.

I realize that cloud providers are vertically integrated stacks, just as mainframes were. Much more modern and scalable with an entirely different cost structure—but, still, eventually and ultimately, lock-in.

Avoid provider lock-in with OpenShift Container Platform

This is where OpenShift comes in. I take orchestration and development cycles to a whole new level when I stop worrying about operating system instances, storage capacity, network overlays, NAT gateways, firewalls—all the things I need to make my application accessible and provide value.

Instead, I deal with application instances, persistent volumes, services, replication controllers, and build configurations—things that make much more sense to me as an application developer as they are closer to what I am really interested in: deploying new functionality into production. Thus, OpenShift offers abstraction on top of classic IT infrastructure and instead provides application infrastructure. The key here is massive automation on top of the concept of immutable infrastructure, thereby greatly enhancing the capability to bring new code into production.

The benefit is clear: Once I have OpenShift in place, I don’t need to worry about any of the underlying infrastructure—I don’t need to be aware of whether I am actually running on OpenStack, VMware, Azure, Google Cloud, or Amazon Web Services (AWS). My new common denominator is the interface of OpenShift powered by Kubernetes, and I can forget about what’s underneath.

Well, not quite. While OpenShift provides a lot of drivers for various underlying infrastructure, for instance storage, they are all somewhat different. Their availability, performance, and feature set is tied to the underlying provider, for instance Elastic Block Storage (EBS) on AWS. I need to make sure that critical aspects of the infrastructure below OpenShift are reflected in OpenShift topology. A good example are AWS availability zones (AZs): They are failure domains in a region across which an application instance should be distributed to avoid downtime in the event a single AZ is lost. So OpenShift nodes need to be deployed in multiple AZs.

This is where another caveat comes in: EBS volumes are present only inside a single AZ. Therefore, my application must replicate the data across other AZs if it uses EBS to store it.

So there are still dependencies and limitations a developer or operator must be aware of, even if OpenShift has drivers on board for EBS and will take care about provisioning.

Introducing container-native storage

With container-native storage (CNS), we now have a robust, scalable, and elastic storage service out-of-the-box for OpenShift Container Platform—based on Red Hat Gluster Storage. The trick: GlusterFS runs containerized on OpenShift itself. Thus, it runs on any platform that OpenShift is supported on—which is basically everything, from bare metal, to virtual, to private and public cloud.

With CNS, OpenShift gains a consistent storage feature set across, and independent of, all supported cloud providers. It’s deployed with native OpenShift / Kubernetes resources, and GlusterFS ends up running in pods as part of a DaemonSet:

[ec2-user@ip-10-20-4-55 ~]$ oc get pods
NAME              READY     STATUS    RESTARTS   AGE
glusterfs-0bkgr   1/1       Running   9          7d
glusterfs-4fmsm   1/1       Running   9          7d
glusterfs-bg0ls   1/1       Running   9          7d
glusterfs-j58vz   1/1       Running   9          7d
glusterfs-qpdf0   1/1       Running   9          7d
glusterfs-rkhpt   1/1       Running   9          7d
heketi-1-kml8v    1/1       Running   8          7d

The pods are running in privileged mode to access the nodes’ block device directly. Furthermore, for optimal performance, the pods are using host-networking mode. This way, OpenShift nodes are running a distributed, software-defined, scale-out file storage service, just as any distributed micro-service application.

There is an additional pod deployed that runs heketi—a RESTful API front end for GlusterFS. OpenShift natively integrates via a dynamic storage provisioner plug-in with this service to request and delete storage volumes on behalf of the user. In turn, heketi controls one or more GlusterFS Trusted Storage Pools.

Container-native storage on Amazon Web Services

The EBS provisioner has been available for OpenShift for some time. To understand what changes with CNS on AWS, a closer look at how EBS is accessible to OpenShift is in order.

  1. Dynamic provisioning
    EBS volumes are dynamically created and deleted as part of storage provisioning requests (PersistentVolumeClaims) in OpenShift.
  2. Local block storage
    EBS appears to the EC2 instances as a local block device. Once provisioned, it is attached to the EC2 instance, and a PCI interrupt is triggered to inform the operating system.

    NAME                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    xvda                                  202:0    0   15G  0 disk
    ├─xvda1                               202:1    0    1M  0 part
    └─xvda2                               202:2    0   15G  0 part /
    xvdb                                  202:16   0   25G  0 disk
    └─xvdb1                               202:17   0   25G  0 part
      ├─docker_vol-docker--pool_tmeta     253:0    0   28M  0 lvm
      │ └─...                             253:2    0 23.8G  0 lvm
      │   ├─...                           253:8    0    3G  0 dm
      │   └─...                           253:9    0    3G  0 dm
      └─docker_vol-docker--pool_tdata     253:1    0 23.8G  0 lvm
        └─docker_vol-docker--pool         253:2    0 23.8G  0 lvm
          ├─...                           253:8    0    3G  0 dm
          └─...                           253:9    0    3G  0 dm
    xvdc                                  202:32   0   50G  0 disk 
    xvdd                                  202:48   0  100G  0 disk

    OpenShift on AWS also uses EBS to back local docker storage. EBS storage is formatted with a local filesystem like XFS..

  3. Not shared storage
    EBS volumes cannot be attached to more than one EC2 instance. Thus, all pods mounting an EBS-based PersistentVolume in OpenShift must run on the same node. The local filesystem on top of the EBS block device does not support clustering either.
  4. AZ-local storage
    EBS volumes cannot cross AZs. Thus, OpenShift cannot failover pods mounting EBS storage into different AZs. Basically, an EBS volume is a failure domain.
  5. Performance characteristics
    The type of EBS storage, as well as capacity, must be selected up front. Specifically, for fast storage a certain minimum capacity must be requested to have a minimum performance level in terms of IOPS.

This is the lay of the land. While these characteristics may be acceptable for stateless applications that only need to have local storage, they become an obstacle for stateful applications.

People want to containerize databases, as well. Following a micro-service architecture where every service maintains its own state and data model, this request will become more common. The nature of these databases differs from the classic, often relational, database management system IT organizations have spent millions on: They are way smaller and store less data than their big brother from the monolithic world. Still, with the limitations of EBS, I would need to architect replication and database failover around those just to deal with a simple storage failure.

Here is what changes with CNS:

  1. Dynamic provisioning
    The user experience actually doesn’t change. CNS is represented like any storage provider in OpenShift, by a StorageClass. PersistentVolumeClaims (PVCs) are issued against it, and the dynamic provisioner for GlusterFS creates the volume and returns it as a PersistentVolume (PV). When the PVC is deleted, the GlusterFS volume is deleted, as well.
  2. Distributed file storage on top of EBS
    CNS volumes are basically GlusterFS volumes, managed by heketi. The volumes are built out of local block devices of the OpenShift nodes backed by EBS. These volumes provide shared storage and are mounted on the OpenShift nodes with the GlusterFS FUSE client.

    [ec2-user@ip-10-20-5-132 ~]$ mount
    ... on /var/lib/origin/openshift.local.volumes/pods/80e27364-2c60-11e7-80ec-0ad6adc2a87f/volumes/ type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
  3. Container-shared storage
    Multiple pods can mount and write to the same volume. The access mode for the corresponding node is known as “RWX”—read-write many. The containers can run on different OpenShift nodes, and the dynamic provisioner will mount the GlusterFS volume on the right nodes accordingly. Then, this local mount directory is bind-mounted to the container.
  4. Cross-availability zone
    CNS is deployed across AWS AZs. The integrated, synchronous replication of GlusterFS will mirror every write 3 times. GlusterFS is deployed across OpenShift nodes in at least different AZs, and thus the storage is available in all zones. The failure of a single GlusterFS pod, an OpenShift node running the pod, or a block device accessed by the pod will have no impact. Once the failed resources come back, the storage is automatically re-replicated. CNS is actually aware of the failure zones as part of the cluster topology and will schedule new volumes, as well as recovery, so that there is no single point of failure.
  5. Predictable performance
    CNS storage performance is not tied to the size of storage request by the user in OpenShift. It’s the same performance whether 1GB or 100GB PVs are requested.
  6. Storage performance tiers
    CNS allows for multiple GlusterFS Trusted Storage Pools to be managed at once. Each pool consists of at least 3 OpenShift nodes running GlusterFS pods. While the OpenShift nodes belong to a single OpenShift cluster, the various GlusterFS pods form their own Trusted Storage Pools. An administrator can use this to equip the nodes with different kinds of storage and offer their pools with CNS as distinct storage tiers in OpenShift, via its own StorageClass. An administrator instance might, for example, run CNS on 3 OpenShift nodes with SSD (e.g., EBS gp2) storage and call it “fast,” whereas another set of OpenShift nodes with magnetic storage (e.g., EBS st1) runs a separate set of GlusterFS pods as an independent Trusted Storage Pool, represented with a StorageClass called “capacity.”

This is a significant step toward simplifying and abstracting provider infrastructure. For example, a MySQL database service running on top of OpenShift is now able to survive the failure of an AWS AZ, without needing to set up MySQL Master-Slave replication or change the micro-service to replicate data on its own.

Storage provided by CNS is efficiently allocated and provides performance with the first Gigabyte provisioned, thereby enabling storage consolidation. For example, consider six MySQL database instances, each in need of 25 GiB of storage capacity and up to 1500 IOPS at peak load. With EBS, I would create six EBS volumes, each with at least 500 GiB capacity out of the gp2 (General Purpose SSD) EBS tier, in order to get 1500 IOPS guaranteed. Guaranteed performance is tied to provisioned capacity with EBS.
With CNS, I can achieve the same using only 3 EBS volumes at 500 GiB capacity from the gp2 tier and run these with GlusterFS. I would create six 25 GiB volumes and provide storage to my databases with high IOPS performance, provided they don’t peak all at the same time.

Doing that, I would halve my EBS cost and still have capacity to spare for other services. My read IOPS performance is likely even higher because in CNS with 3-way replication I would read from data distributed across 3×1500 IOPS gp2 EBS volumes.

Finally, the setup for CNS is very simple and can run on any OpenShift installation based on version 3.4 or newer.

This way, no matter where I plan to run OpenShift (i.e., which cloud provider currently offers lowest prices), I can rely on the same storage features and performance. Furthermore, the Storage Service grows with the OpenShift cluster but still provides elasticity. Only a subset of OpenShift nodes must run CNS, at least 3 ideally across 3 AZs.

Deploying container-native storage on AWS

Installing OpenShift on AWS is dramatically simplified based on the OpenShift on Amazon Web Services Reference Architecture. A set of Ansible playbooks augments the existing openshift-ansible installation routine and creates all the required AWS infrastructure automatically.

A simple python script provides a convenient wrapper to the playbooks found in the openshift-ansible-contrib repository on GitHub for deploying on AWS.

All the heavy lifting of setting up Red Hat OpenShift Container Platform on AWS is automated with best practices incorporated.

The deployment finishes with an OpenShift Cluster with 3 master nodes, 3 infrastructure nodes, and 2 application nodes deployed in a highly available fashion across AWS AZs. The external and internal traffic is load balanced, and all required network, firewall, and NAT resources are stood up.

Since version 3.5, the reference architecture playbooks now ship with additional automation to make deployment of CNS as easy. Through additional AWS CloudFormation templates and Ansible playbook tasks, the additional, required infrastructure is stood up. This mainly concerns provisioning of additional OpenShift nodes with an amended firewall configuration, additional EBS volumes, and then joining them to the existing OpenShift cluster.

In addition, compared to previous releases, the CloudFormation templates now emit more information as part of the output. These are picked up by the playbooks to further reduce the information needed from the administrator. They will simply get the right information from the existing CloudFormation stack to retrieve the proper integration points.

The result is AWS infrastructure ready for the administrator to deploy CNS. Most of the manual steps of this process can therefore be avoided. Three additional app nodes are deployed with configurable instance type and EBS volume type. Availability zones of the selected AWS region are taken into account.

Subsequent calls allow for provisioning of additional CNS pools. The reference architecture makes reasonable choices for the EBS volume type and the EC2 instance with a balance between running costs and initial performance. The only thing left for the administrator to do is to run the cns-deploy utility and create a StorageClass object to make the new storage service accessible to users.

At this point, the administrator can choose between labeling the nodes as regular application nodes or provide a storage-related label that would initially exclude them from the OpenShift scheduler for regular application pods.

Container-ready storage

The reference architecture also incorporates the concept of Container-Ready Storage (CRS). In this deployment flavor, GlusterFS runs on dedicated EC2 instances with a heketi-instance deployed separately, both running without containers as ordinary system services. The difference is that these instances are not part of the OpenShift cluster. The storage service is, however, made available to, and used by, OpenShift in the same way. If the user, for performance or cost reasons, wants the GlusterFS storage layer outside of OpenShift, this is made possible with CRS. For this purpose, the reference architecture ships to automate the deployment in the same way as for CNS.


CNS provides further means of OpenShift Container Platform becoming an equalizer for application development. Consistent storage services, performance, and management are provided independently of the underlying provider platform. Deployment of data-driven applications is further simplified with CNS as the backend. This way, not only stateless but also stateful applications become easy to manage.

For developers, nothing changes: The details of provisioning and lifecycle of storage capacity for containerized applications is transparent to them, thanks to CNS’s integration with native OpenShift facilities.

For administrators, achieving cross-provider, hybrid-cloud deployments just became even easier with the recent release of the OpenShift Container Platform 3.5 on Amazon Web Service Reference Architecture. With just two basic commands, an elastic and fault-tolerant foundation for applications can be deployed. Once set up, growth becomes a matter of adding nodes.

It is now possible to choose the most suitable cloud provider platform without worrying about various tradeoffs between different storage feature sets or becoming too close to one provider’s implementation, thereby avoiding lock-in long term.

The reference architecture details the deployment and resulting topology. Access the document here.

Moving the ball forward for object storage with Red Hat Ceph Storage 2.2

We are happy to announce that Red Hat Ceph Storage 2.2 is now Generally Available (GA). It is based on the open source community version of Ceph Storage, specifically version 10.2.5, the Jewel release stream. Similar to Red Hat Ceph Storage 2 announced last summer, Red Hat Ceph Storage 2.2 has a heavy focus on object storage deployments. In addition to following a new, more predictable release process, Red Hat Ceph Storage 2.2 offers a number of enhancements, including:

Global clusters

The object access method for Red Hat Ceph Storage (aka the RADOS Gateway, or “RGW”) now supports up to three sites for global cluster configurations. This means that customers can deploy an active-active global cluster across three geographically distributed sites with data replication and consistency across all three. Alternatively, the RGW multi-site capability can be used in disaster recovery configurations for data protection against site disaster using an active-passive deployment.

Better security and encryption support

Red Hat Ceph Storage now has native support for the Secure Sockets Layer (SSL) in RGW. This is a good option for small to medium size installations that require data encryption between the object storage client and RGW. Because SSL and encryption can have a performance impact, we still recommend terminating SSL at the load balancer or HA layer for large-scale installations.

S3 API enhancements

Red Hat Ceph Storage supports the S3 object API for easy application mobility across private and public clouds. Red Hat Ceph Storage 2.2 now adds support for the new multipart upload copy-part S3 API.

Swift enhancements

Red Hat Ceph Storage 2.2 also has important developments for customers using the Swift object API that demonstrate our continued focus on object storage deployments, including:

  • Support for Swift object versioning functionality.
  • Full testing and compliance of Red Hat Ceph Storage 2.2 object storage with the Swift API tests in the Tempest test suite from RefStack toolset for OpenStack. For customers, this translates into interoperability between applications and services using the Swift API and Red Hat Ceph Storage.

For a recent example of a customer success with Red Hat Ceph Storage at scale for object storage, check out our recently published success story with the CLIMB project in the U.K.

For general information on object storage features in Red Hat Ceph Storage, read this blog.

For general information on Red Hat Ceph Storage, visit this page.

The value of on-premise object storage

The recent outage of the Amazon Web Services (AWS) S3 offering has likely reminded customers of the old “too many eggs in one basket” adage, sparking reviews of where and how they deploy and manage their storage. Outages of this magnitude illustrate how much data lives in object storage these days.

Object storage has come to be the foundation for web-scale architectures in public or private clouds, as we’ve previously discussed. Its allure is due to its potential to handle massive scale while minimizing complexity and cost. Customers struggling with large-scale data storage deployments are turning to object storage to overcome the limitations that legacy storage systems face at scale (typically degradation of performance).

Object storage allows application developers and users to focus more on their workflow and logic and not worry about handling file storage and file locations. However, the large extent of outages due to the unavailability of AWS S3 shows the dependence that so many businesses have on public-cloud-based object storage today.

Ceph is an object store at its core and was designed for web-scale applications. Running Red Hat Ceph Storage on premises offers customers a more secure, cost-effective, and performant way to manage large amounts of data without the risks that can be associated with public-cloud-based solutions.

The S3 API has become the de facto standard for accessing data in object stores, and organizations now develop and demand applications that “speak” S3. Applications that use the S3 API can more easily migrate between on-premise, private clouds built on Red Hat Ceph Storage and public clouds like AWS just by changing the storage endpoint network address. This enables them to operate on both datasets that are present in private clouds and those stored in the public cloud. A common API for object storage means that applications can move between these two cloud deployment models, managing data consistently wherever they go.

For a recent customer example using Red Hat Ceph Storage at scale for object storage, check out our recently published success story with the CLIMB project in the U.K.

Hopefully, most of you reading this were not stung by the recent outage. Regardless, now is as good a time as any to review your infrastructure to determine if an on-premise object storage approach with Red Hat Ceph Storage makes sense. We think it does!

For more information on Red Hat Ceph Storage for object storage, check out this page.

For more information on Red Hat Ceph Storage for cloud Infrastructures, check out this page.

Cue the drumroll…Red Hat Gluster Storage 3.2 releases today

Faster metadata operations, deeper integration with OpenShift, smaller hardware footprint

It’s here! Red Hat today announced the general availability of Red Hat Gluster Storage 3.2. Please join the live launch event at 12pm EDT today to hear from customers, partners, experts, and the thriving open source GlusterFS community. Can’t make it? No sweat. The content will be available for your viewing pleasure for 90 days.

The best yet. And then some.

This release is a significant milestone in the life of the product. It brings to bear a number of key enhancements but also represents a solid, stable release that is a culmination of much of the innovation over the past 18 months.

Red Hat Gluster Storage is highly scalable, software-defined storage that can be deployed anywhere – from bare metal, to virtual machines, to containers, and across all three major public cloud providers (Microsoft Azure, Google Cloud Platform, and Amazon AWS).

Red Hat Gluster Storage 3.2 is geared toward cloud-native applications and modern workloads that demand a level of agility and scale that traditional storage appliances cannot offer. Read on to find out how this release delivers performance and cost improvements along with a seamless user experience.

Deeper integration with OpenShift Container Platform

Container-native storage for OpenShift Container Platform enables developers to provision and manage storage as easily as they can applications. In addition, improvements in small-file performance in Red Hat Gluster Storage will help container registry workloads – the heart of the container platform itself – to persist Docker images for containerized applications.

In addition to small file performance, there are a number of scale improvements to the management plane that allow for a larger number of persistent volumes (OpenShift PVs) that can be hosted in a single cluster. Also, the container for Red Hat Gluster Storage has been configured to run the sshd service for supporting async geo-replication between remote sites and has been instrumented to hold the SSL certificates for supporting in-flight encryption to be leveraged by container-native storage in upcoming releases.

Storage administrators will also find that better small file performance allows for a better user experience with day-to-day operations, such as directory listings, up to 8x faster. The secret sauce, client-side caching, that facilitates this performance improvement will benefit both Linux (Fuse) and Windows (SMB) users.

The improvement in small file performance will be particularly relevant to use cases with a large number of files under a few MBs in size. A number of use cases, such as hosting Git repositories and electronic design automation, will benefit as well.

Lower hardware footprint and costs

Most companies look to 3-way replication for enterprise-grade data integrity. What if we told you that you could benefit from the same grade of data integrity while lowering your hardware footprint and cost?

This is possible in Red Hat Gluster Storage 3.2 through a new type of volume known as an arbiter volume that does just that – arbitrates between two nodes that may be out of sync. This is particularly useful for remote office-branch office (ROBO) scenarios where remote locations are interested in saving space, power, and cost through hyperconvergence.

Additionally, faster self-healing of sharded storage volumes will benefit customers running hyperconverged configurations, such as Red Hat Virtualization with Red Hat Gluster Storage.

Performance enhancements

Multi-threaded applications can perform faster by parallelizing I/O requests by enabling client-io threads. This feature is activated automatically during heavy loads and dials back during idle state. There are significant performance gains with more number of threads with client io threads enabled.

Our internal test results have shown up to 250% performance increase with the dispersed volumes with client-io-threads enabled. There is a near linear performance gain with the increase in the number of threads.

Native eventing and integration with Nagios

Red Hat Gluster Storage 3.2 features a new native push-based notification framework that can send out asynchronous alerts. As with earlier releases, full integration with the open source Nagios framework is supported.

Learn more. Speak to Red Hat experts and customers.

Find out more about the benefits of the new feature-rich Red Hat Gluster Storage at our launch event happening today at 12pm EDT. Sign up for free, and interact directly with our experts who will be standing by to answer your questions.

Hear from customers like Brinker International who are running containers on Red Hat Gluster Storage in production and have some great results to share. Hear also about great things happening in the GlusterFS community, which includes contributors like Facebook. We hope to see you there.

Cloud native app developers delight: Container storage just got a whole lot easier!

The new Red Hat OpenShift Container Platform offers a rich user experience with dynamic provisioning of storage volumes, automation, and much more


By Michael Adam, Engineering Lead, Container Native Storage, and Sayan Saha, Head of Product, Red Hat Gluster Storage


Today, Red Hat announced general availability of Red Hat OpenShift Container Platform 3.4, which includes key features such as enhanced multi-tenancy and streamlined deployment for hybrid clouds. In addition, a number of open source storage innovations have been included in this release, which enable easier storage management and provisioning across the lifecycle of containers.

The story so far

Containers were built to be ephemeral and stateless. However, stateful applications running in containers need enterprise-grade persistent storage. Over the past 18 months, Red Hat has delivered a continuum of innovation around persistent storage for containers, leading the charge on both fronts – the open source communities and enterprise products. Red Hat offers container-native storage – durable, distributed, software-defined storage integrated deeply into the Red Hat OpenShift Container Platform, managed by Kubernetes.


Rich developer and management experience

In the latest release, Red Hat OpenShift Container Platform 3.4 offers dynamic provisioning of persistent volumes, allowing for a much richer developer experience, addressing annoying delays due to lengthy storage provisioning cycles needed by traditional storage platforms.

Storage administrators can expect to find that easier volume management with dynamic provisioning frees them up for more value-added tasks. Developers building cloud-native apps deployed in containers can benefit from faster storage provisioning and a better user experience.

DevOps managers can relish the automation and integration through a new deployment tool included with the subscription that can deploy container-native storage with push-button simplicity.

Dynamic provisioning for persistent volume claims

Prior to this release, storage administrators and application developers were limited to a static provisioning model where persistent volumes (PVs) of fixed capacity had to be pre-provisioned manually to be consumed by applications running in Kubernetes pods.

Persistent volume claims (PVCs) are used to consume storage resources in Kubernetes like pods that consume compute resources. When new PVCs were received, an attempt was made to match the PVC request with the closest available PV in terms of capacity, and if one was found the claim would be bound to it. This scheme is inefficient.

Consider a situation where 10, 100 GB PVs have been pre-provisioned and made available. A request for 50 GB of storage would be matched to one of the available 100 GB PVs. This is wasteful as storage is over-committed. On the other hand, a request for 150 GB of storage would go unsatisfied as there is no close match, even though there is unused storage capacity.

The new dynamic provisioning feature fixes that issue by automating the provisioning of storage volumes. For instance, a 50 GB PVC request is addressed using a 50 GB PV that is dynamically provisioned for developers requiring zero admin intervention. In other words, users can expect to get exactly what they asked for as long as the underlying storage platform has available capacity.

Note that dynamic provisioning is supported even when Red Hat Gluster Storage serves out storage from a dedicated storage cluster in addition to container-native storage. This demo shows how container-native storage can be dynamically provisioned in OpenShift Container Platform.


Dynamic provisioning using storage classes

Dynamic provisioning is enabled by a new feature in OpenShift called storage classes. Storage classes enable storage admins to describe as well as classify their various storage implementations that are available to be used by the OpenShift cluster, and they enable developers to configure specific parameters when requesting storage on demand. Container-native storage can be configured as a storage class, which allows OpenShift developers to dynamically provision storage when submitting claims against the storage class, as seen below.


Faster and easier storage deployments using Kubernetes daemon sets

Container-native storage now ships with a deployment tool that will deploy the whole system in an already installed OpenShift cluster. The deployment tool is flexible in that it can easily be used in Ansible playbooks. The administrator only needs to prepare a topology file, a JSON-formatted file describing the nodes and storage devices to be used. Based on that, the deployment of the Gluster storage cluster and the management server as pods in the OpenShift cluster is achieved with the invocation of just a single command. Once deployment is completed, the Gluster storage is ready for both manual and dynamic provisioning with an appropriate storage class. In case of any errors encountered during deployment, the tool supports an abort operation that undoes the failed partial deployment, so that it can be started from scratch. This demo shows the deployment tool in action.


GID level security and endpoints

Several features have been added to Red Hat OpenShift Container Platform 3.4 to create a more secure storage environment. The first of these is the addition of system-controlled, preallocated GIDs for the Red Hat Gluster Storage container. This enables the container to run as a non-root user, permitting only allowed users to access the data.

Second, usability with endpoints has been resolved with the deployment of a service and endpoint for each dynamically provisioned volume. This allows PVs to be specific to the requestors namespace without the added steps of manually creating these resources.

The most comprehensive persistent storage for containers

Red Hat continues to be a major contributor to the Docker and Kubernetes communities. In fact, as of today, Red Hat has the second-most contributors in each, second only to Docker and Google, respectively. Much of the innovation happening upstream is focused on solving the persistent storage challenge for stateful applications. Red Hat has contributed a number of volume plugins for a variety of protocols. Learn more about the latest innovations from Red Hat during the virtual event on January 19 or in a webinar with container storage experts on January 24. Learn more at

A happy new year with Red Hat Ceph Storage!

By Daniel Gilfix, Red Hat Storage

Now that a somewhat tumultuous 2016 is in our rearview mirror, what better way to kick off the new year than with a couple of major endorsements for one of the key emerging businesses of the world’s leading provider of open source solutions? That’s right: Last week alone, over a scant 48 hours, Red Hat watched as Red Hat Ceph Storage 2 earned distinction by CRN as one of the 10 coolest open source products of 2016 and by TechTarget as one of its 12 finalists for 2016 product of the year in server-based storage. Two major endorsements.

Commitment upstream and down

The dual recognition is testament not only to the open source community but also to Red Hat and its valued customers spanning industries like telco, financial services, retail, and the public sector for the advancement of software-defined storage, which has become increasingly indispensable for workloads like cloud infrastructure, data lakes, backup and recovery, media repository, surveillance, and containers. Some of these customers have the necessary in-house skills, but many others rely on the expertise of Red Hat Storage Consulting and training for access to the best practices in architectural design, solution implementation, and knowledge transfer typically required when graduating from upstream to downstream deployment.

Always breaking new ground

Red Hat Ceph Storage 2 was announced at Red Hat Summit last May and began shipping in August. The first major release since the acquisition of Inktank by Red Hat in 2014, it marked the introduction of a user-friendly interface and an integrated lifecycle management system called Red Hat Console 2. Besides the ability to manage and monitor cluster activity in terms of health, performance, and capacity, Console empowered users to install Ceph in under an hour and to grow clusters graphically using Ansible. Red Hat Ceph Storage 2 also marked a departure from an exclusive “we’re for OpenStack” philosophy to a broader emphasis on offering object storage capabilities that are vital for managing vast quantities of unstructured data efficiently with emerging workloads.

Not just for OpenStack anymore

This is not to say that Red Hat Ceph Storage isn’t for OpenStack anymore—far from it. Ceph remains the overwhelmingly preferred storage backend for OpenStack workloads ( user survey, April 2016), and Red Hat Ceph Storage continues to tighten its integration with Red Hat OpenStack Platform. This is evident by Red Hat OpenStack Platform Director’s ability to automate upgrades from Red Hat Ceph Storage 1.3 to 2, manage object and block deployment, and leverage OpenStack’s shared filesystem service (Manila) with the CephFS driver. In effect, Red Hat customers can now fully customize their OpenStack deployment architectures with one unified storage platform. OpenStack Platform customers even receive a complimentary 64 TBs of Red Hat Ceph Storage for proof of concept.

Something more for everyone

But the focus of Red Hat Ceph Storage 2 that caught the attention of CRN and TechTarget was the retransformation of Red Hat’s storage product as an easier-to-use, more versatile product for everyone. The new management interface expanded user appeal for those without advanced Linux expertise or prior familiarity with Ceph. Features like support for Active Directory and LDAP authentication, integration with the S3 protocol of Amazon Web Services, and disaster-recovery options that include remote workloads all added to the product’s strength as an object storage platform. Red Hat Ceph Storage 2 now handles petabyte-scale deployments with the flexibility required by next-generation software-defined datacenters and the ease and enhanced cost efficiency required by today’s business.

Onward to new accomplishments

So, as we embark on new adventures and roller-coaster rides in 2017, let us all take note that software-defined storage on industry-standard hardware has now matured to be a platform of stability for the general populace needing to store enterprise data. Red Hat Ceph Storage has garnered well-deserved recognition for its advancement as a leader of the pack. Stay tuned for things to come this year.