Infrastructure at your Service

Mehdi Bada

Creating and Using a Parcel Repository for Cloudera Manager

By July 11, 2018 Big Data No Comments

This blog post describes how to create a hosted Cloudera repository and use it in your Cloudera Manager deployment.

The first step is to install a web server, which will host RPM packages and repodata. The common way, is to use an Apache web server.

Installing Apache HTTPD service

[cdhtest@edge ]$ sudo yum install httpd -y

 

Starting Apache HTTPD service

[cdhtest@edge ]$ sudo systemctl start httpd

Verify that the service has been started properly.

[cdhtest@master html]$ sudo systemctl status httpd
* httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-07-11 09:16:45 UTC; 1h 26min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 53284 (httpd)
   Status: "Total requests: 40; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           |-53284 /usr/sbin/httpd -DFOREGROUND
           |-53285 /usr/sbin/httpd -DFOREGROUND
           |-53286 /usr/sbin/httpd -DFOREGROUND
           |-53287 /usr/sbin/httpd -DFOREGROUND
           |-53288 /usr/sbin/httpd -DFOREGROUND
           |-53289 /usr/sbin/httpd -DFOREGROUND
           |-53386 /usr/sbin/httpd -DFOREGROUND
           |-53387 /usr/sbin/httpd -DFOREGROUND
           |-53388 /usr/sbin/httpd -DFOREGROUND
           `-58024 /usr/sbin/httpd -DFOREGROUND

Jul 11 09:16:45 master systemd[1]: Starting The Apache HTTP Server...
Jul 11 09:16:45 master httpd[53284]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.2.6. Set the 'ServerName' directive globally to suppress this message
Jul 11 09:16:45 master systemd[1]: Started The Apache HTTP Server.

 

Downloading Parcels for CDH5 and Publishing files

Download parcels according to your OS distribution for CDH5 (latest release) from the below link:

https://archive.cloudera.com/cdh5/parcels/latest/

Download the 2 files:

  • .parcels
  • manifest.json

Before downloading the files, create the CDH parcel directory tree in your web server.

[cdhtest@master html]$ cd /var/www/html/
[cdhtest@master html]$ sudo mkdir -p cdh5.15/
[cdhtest@master html]$ sudo chmod -R ugo+rX /var/www/html/cdh5.15/
[cdhtest@master html]$ cd /var/www/html/cdh5.15/
[cdhtest@master cdh5.15]$ sudo wget https://archive.cloudera.com/cdh5/parcels/latest/CDH-5.15.0-1.cdh5.15.0.p0.21-el5.parcel https://archive.cloudera.com/cdh5/parcels/latest/manifest.json
--2018-07-11 12:16:04--  https://archive.cloudera.com/cdh5/parcels/latest/CDH-5.15.0-1.cdh5.15.0.p0.21-el5.parcel
Resolving archive.cloudera.com (archive.cloudera.com)... 151.101.32.167
Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.32.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1675168741 (1.6G) [binary/octet-stream]
Saving to: 'CDH-5.15.0-1.cdh5.15.0.p0.21-el5.parcel'

100%[==================================================================================================================================================================================================================================================================================================>] 1,675,168,741 53.2MB/s   in 29s

2018-07-11 12:16:32 (56.0 MB/s) - 'CDH-5.15.0-1.cdh5.15.0.p0.21-el5.parcel' saved [1675168741/1675168741]

--2018-07-11 12:16:32--  https://archive.cloudera.com/cdh5/parcels/latest/manifest.json
Reusing existing connection to archive.cloudera.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 74072 (72K) [application/json]
Saving to: 'manifest.json'

100%[====================================================================================================================================================================================================================================================================================================>] 74,072      --.-K/s   in 0s

2018-07-11 12:16:32 (225 MB/s) - 'manifest.json' saved [74072/74072]

FINISHED --2018-07-11 12:16:32--
Total wall clock time: 29s
Downloaded: 2 files, 1.6G in 29s (56.0 MB/s)
[cdhtest@master cdh5.15]$
[cdhtest@master cdh5.15]$ ll
total 1635984
-rw-r--r-- 1 root root 1675168741 Jun 14 18:06 CDH-5.15.0-1.cdh5.15.0.p0.21-el5.parcel
-rw-r--r-- 1 root root      74072 Jun 14 18:08 manifest.json

 

Your Parcels remote repository is now available

CM_Parcels8

 

Configuring the Cloudera Manager Server to Use the Parcel URL for Hosted Repositories

1. In the Cluster Installation – Select Repository step

Click on More Options.

CM_Parcels3

Add your Remote Parcel Repository URL .

CM_Parcels4

Then Cloudera Manager will download, distribute, unpack and activate parcels for all cluster hosts.

CM_Parcels5

 

2. You can also configure your local parcels repository in the Cloudera Manager Configuration Menu.

Click on Administration Menu > Settings

Click on Parcels category > Add your Remote Parcel Repository URL here

CM_Parcels6

Click Save Changes to commit the changes.

 

Activate Parcels

In the Cloudera Manager Parcels page,

Click on Check for New Parcels

Click on Download, Distribute, Activate buttons for the parcels found.

CM_Parcels7

Leave a Reply

Mehdi Bada
Mehdi Bada

Consultant