That’s the upside to the CSP scale-out model, but there’s also a notable downside: Compute and storage have to scale in lockstep, because they come as a package via a hyperconverged implementation. That means that when you need to add more storage to accommodate data growth, you have to also add more compute, whether you need the processing capacity or not, and vice-versa.
This reality of the scale-out data center can lead to overprovisioning and higher costs—but it doesn’t have to be the case. Quanta Cloud Technology (QCT) and Intel® recently worked with a large CSP to come up with a novel architecture that solves the scale-out problem while addressing other technology goals.
This groundbreaking architecture enables the CSP to:
• Establish a multi-tenant platform capable of running any workload
• Disaggregate storage and compute, so each can be scaled independently
• Standardize on open source Ceph distributed block store system as the backend for Hadoop
• Maintain storage performance levels in a hybrid storage solution by adding Solid State Drives (SSDs), from the Intel® Solid-State Drive Data Center P3700 Series, and by leveraging Intel® Cache Acceleration Software (Intel® CAS)
CSP’s typically run a variety of workloads and inevitably generate an incredible amount of valuable data. In order to meet different purposes and criteria, CSP’s have traditionally used various storage architectures to process and store these diversified workloads and data. With a distributed and consistent storage architecture in mind as a goal, QCT and Intel utlize open source storage technology, Ceph, with feature-rich QCT server and storage portfolio to develop a Hadoop-optimized Ceph storage solution. Now Cloud Service Provider’s, can use the new QCT and Intel storage solution across multiple workloads due to the flexibility in Ceph.
Intel® CAS, combined with high-performance SSDs, increases performance via intelligent caching through unique I/O classification functionality allowing users to optimize and prioritize their workloads caching the most often used and hot data. By using Intel® Solid State Drives in conjunction with Intel® CAS in a hybrid disk array, data center operators can overcome traditional storage media latency hurdles and open the door to truly disaggregated storage solutions, delivering reduced operational complexity and saving costs.
The use of non-volatile media and Intel® CAS were critical components of the architecture. That’s because disaggregation moves the data farther away from the processors, which creates latencies in the transmission of data from the storage media to the processors. In this architecture, the latency is offset by the increased speed from caching the data by using Intel® Solid State Drives and Intel® Cache Acceleration Software avoiding unacceptable latencies and slow application response times.
With this innovative architecture, our CSP customer is able to reduce data center costs, such as power and datacenter foot print, by scaling storage and compute independently of one another, while avoiding the I/O bottlenecks that come with converged storage and compute. The CSP can now run a disaggregated Hadoop environment and meet its customers’ service-level agreements with confidence. At the same time, the architecture provides scalability, allowing growth for new services, agility, to respond quickly to new customer requests, and efficiency, to grow in a cost-effective manner.
The architecture was developed by QCT and Intel® in close collaboration with the CSP, and then thoroughly validated on a 7 node Hadoop and 7 node Ceph storage cluster. This large-scale proof of concept provided a real life scale testing environment. The CSP is now making plans for deployment of this new and innovative storage architecture on a broad scale.
If you have the opportunity to attend the Intel Developer Forum this week in San Francisco, you can catch demos of the solution in the QCT booth (#197 & #525).