Cloudera Data Platform Data Center With IBM 7.1.1
Migration paths from earlier versions of Hortonworks Data Platform and Cloudera Enterprise Data Hub platforms
This is a Press Release edited by StorageNewsletter.com on July 22, 2020 at 2:14 pmCloudera Data Platform Data Center with IBM combines many of the capabilities of Cloudera Enterprise Data Hub and Hortonworks Data Platform, providing a data management and analytics platform for on-premises environments.
In addition to several enhancements to administration, stability, security, and performance, this release offers a migration path from Hortonworks Data Platform 2.6.5 or Cloudera Enterprise Data Hub 5.13, 5.14, 5.15, or 5.16 to Cloudera Data Platform Data Center with IBM 7.1.1.
Version 7.1.1 of Cloudera Data Platform Data Center with IBM brings the following key updates:
- Support for Cloudera Enterprise Data Hub to Cloudera Data Platform Data Center with IBM in-place upgrade from Cloudera Enterprise Data Hub 5.13, 5.14, 5.15, or 5.16 clusters to Cloudera Runtime 7.1 and upgrade from Cloudera Manager 5.13, 5.14, 5.15, or 5.16 to Cloudera Manager 7.1 in Cloudera Data Platform Data Center 7.1.
- Support for Migration from Cloudera Enterprise Data Hub to Cloudera Data Platform Data Center with IBM upgrade is provided by Cloudera Manager Replication Manager which enables data copy from a Cloudera Enterprise Data Hub 5 or Cloudera Enterprise Data Hub 6 cluster to a Cloudera Runtime 7.1 cluster.
- Support for Hortonworks Data Platform to Cloudera Data Platform Data Center with IBM in-place upgrade path consists of upgrading from Hortonworks Data Platform 2.6.5.x to Cloudera Runtime 7.1 by using Ambari and then migrating the management platform from Ambari to Cloudera Manager. AM2CM tool captures Hortonworks Data Platform Cluster’s blueprint from Ambari and converts it to a Cloudera Manager deployment template that is deployed to Cloudera Manager 7.1 and activates the Cloudera Runtime 7.1 parcels.
- Support for Sentry to Ranger Migration converts Sentry privileges into their equivalents within Ranger service policies. Tool also supports migration of Kafka policies from Sentry to Ranger.
- Support for Navigator to Atlas Migration converts Navigator content from business metadata such as tags, custom properties (definitions and entity assignments), managed metadata properties (definitions and entity assignments), original and updated entity names and descriptions and technical metadata from Hive, Impala, Spark, Referenced HDFS/S3 to Atlas content.
- Streams messaging. This capability is designed to help organizations to achieve a Kafka streaming experience, boost Kafka connectivity, and improve operational efficiency. It enables organizations to monitor and manage Kafka clusters. The Streams messaging function also provides BC with Streams Replication Manager and the capability to scale up messaging through the use of Schema Registry.
Hortonworks DataFlow enhancements
Hortonworks DataFlow is an enterprise-ready open source streaming data platform that collects, curates, analyzes, and acts on data in the data center and cloud. It is powered by key open sourced projects, which include Apache NiFi, Apache Kafka, and Apache Storm. IBM has 3 offerings that are based on HDF: Cloudera Flow Management with IBM for Ambari, Cloudera Enterprise Stream Processing with IBM for Ambari, and Hortonworks DataFlow Enterprise Management Services.
Version 3.5.1 of Hortonworks DataFlow brings the following key updates:
- Apache NiFi is upgraded from 1.9.0 to 1.11.4. New features and improvements in Apache NiFi include:
-
- Parameters. They are a new concept in NiFi that enables users to define parameter contexts that are attached to process groups so that any property (including sensitive properties and properties that do not support expression language) can be parameterized. This is particularly useful in continuous integration and continuous delivery (CI/CD) pipelines when deploying workflows across multiple environments. Parameters are designed to be used as replacements of the variables.
- Predictive monitoring. NiFi has an internal analytics framework that can be enabled to predict back pressure occurrence when provided with the configured settings for threshold on a queue. It uses recent observations from a queue (either number of objects or content size over time) and calculates predictions based on a model to anticipate back pressure.
- SQL reporting task. The Query NiFi Reporting Task enables users to run SQL queries vs. tables that contain information about connection status, processor status, bulletins, process group status, JVM metrics, provenance, and connection status predictions. In combination with site to site, it can be useful to define fine-grained monitoring capabilities simultaneously with running workflows.
- Kafka is upgraded from 2.1.0 to 2.3.1.
- Schema Registry is upgraded from 0.7.0 to 0.8.1.
- Workload Experience Manager
Cloudera Workload Experience Manager for IBM is a tool that provides insights to help you gain in-depth understanding of the workloads you send to clusters managed by Cloudera Manager. In addition, it provides information that can be used for troubleshooting failed jobs and optimizing slow jobs that run on those clusters. After a job ends, information about job execution is sent to Cloudera Workload Experience Manager with IBM with the Telemetry Publisher, a role in the Cloudera Manager Management Service.
Cloudera Workload Experience Manager with IBM uses the information to display metrics about the performance of a job. Additionally, it compares the current run of a job to previous runs of the same job by creating baselines. You can use the knowledge gained from this information to identify and address abnormal or degraded performance or potential performance improvements.
Previously available as a cloud service, Workload XM is now available on-premises.
Planned availability date: July 14, 2020
Subscribe to our free daily newsletter
