系統(tǒng)環(huán)境
? 操作系統(tǒng):3臺(tái) CentOS 7.3 x64
? Cloudera Manager:6.0.1
? CDH: 6.0.1
超過10多年行業(yè)經(jīng)驗(yàn),技術(shù)領(lǐng)先,服務(wù)至上的經(jīng)營(yíng)模式,全靠網(wǎng)絡(luò)和口碑獲得客戶,為自己降低成本,也就是為客戶降低成本。到目前業(yè)務(wù)范圍包括了:做網(wǎng)站、成都網(wǎng)站制作,成都網(wǎng)站推廣,成都網(wǎng)站優(yōu)化,整體網(wǎng)絡(luò)托管,微信平臺(tái)小程序開發(fā),微信開發(fā),重慶APP開發(fā),同時(shí)也可以讓客戶的網(wǎng)站和網(wǎng)絡(luò)營(yíng)銷和我們一樣獲得訂單和生意!
VM Name | VM IP Adress | Centos_HostName | Role |
---|---|---|---|
Centos_wjg1 | 192.168.1.101 | hadoop01 | Cloudera Manager Server + Cloudera Manager Agent |
Centos_wjg2 | 192.168.1.102 | hadoop02 | Cloudera Manager Agent |
Centos_wjg6 | 192.168.1.106 | hadoop06 | Cloudera Manager Agent |
一、配置網(wǎng)絡(luò)環(huán)境
192.168.1.101 hadoop01
192.168.1.102 hadoop02
192.168.1.106 hadoop06
關(guān)閉防火墻
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
systemctl stop firewalld.service
systemctl disable firewalld.service
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hadoop01'"
and check to make sure that only the key(s) you wanted were added.
在每臺(tái)主機(jī)上重復(fù)以上步驟,使任意兩臺(tái)主機(jī)間都能相互連通。
至此,免密碼登錄已經(jīng)設(shè)定完成。
將當(dāng)前時(shí)間和日期寫入BIOS,避免重啟后失效
hwclock –w
在配置之前,先使用ntpdate手動(dòng)同步一下時(shí)間,免得本機(jī)與對(duì)時(shí)中心時(shí)間差距太大,使得ntpd不能正常同步。這里選用65.55.56.206作為對(duì)時(shí)中心,ntpdate -u ntp1.aliyun.com
vi /etc/ntp.conf
#Hosts on local network are less restricted.
取消下面一行的注釋,并將其中IP和掩碼修改為真實(shí)環(huán)境IP和掩碼,此行配置為允許ntp客戶端連接的配置
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
找到server 0.centos.pool.ntp.org iburst,并將所有server配置進(jìn)行注釋
添加下面兩行內(nèi)容
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
啟動(dòng)NTP服務(wù)
設(shè)置后,重啟ntpd服務(wù),用ntpstat來檢查效果【大概幾秒鐘就可以看到效果】
#systemctl restart ntpd
#ntpstat
vi /etc/sysconfig/ntpdate
#Set to 'yes' to sync hw clock after successful ntpdate
SYNC_HWCLOCK=yes
9.2 Agent節(jié)點(diǎn)配置
修改/etc/ntp.conf
對(duì)該文件進(jìn)行以下內(nèi)容的修改:
1、注釋所有restrict和server配置
2、添加下面注釋,需要修改以下的IP為NTP服務(wù)器的IP(在本案例中就是server主機(jī)的IP)
server 192.168.1.100
將所有主機(jī)的ntp服務(wù)設(shè)置為開機(jī)自動(dòng)啟動(dòng)
chkconfig ntpd on
systemctl enable ntpd.service
二、 配置163 yum源
1)下載repo文件
cd /etc/yum.repos.d/
wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
2)備份并替換系統(tǒng)的repo文件
mv CentOS-Base.repo CentOS-Base.repo.bak
cp CentOS7-Base-163.repo CentOS-Base.repo
3)執(zhí)行yum源更新命令
yum clean all
yum makecache
yum update
yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb –y
yum install httpd mod_ssl openssl-devel python-psycopg2 –y
yum install MySQL-python
yum install perl
yum install autoconf
三、安裝MySQL:
查看并卸載CentOS系統(tǒng)自帶的MySQL和Mariadb
rpm -qa | grep MySQL
rpm -qa | grep mariadb
rpm -e --nodeps MySQL-python-1.2.5-1.el7.x86_64
rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
下載并上傳以下rpm包到主節(jié)點(diǎn)
MySQL-client-5.6.42-1.el7.x86_64.rpm
MySQL-devel-5.6.42-1.el7.x86_64.rpm
MySQL-server-5.6.42-1.el7.x86_64.rpm
在主節(jié)點(diǎn)上執(zhí)行以下命令安裝MySQL
rpm -ivh MySQL*
安裝完成后啟動(dòng)MySQL:
service mysql start
systemctl start mysql.service
修改MySQL的配置文件
vi /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
#Disabling symbolic-links is recommended to prevent assorted security risks;
#to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
#InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
獲取系統(tǒng)初始密碼:
[root@hadoop01 wjg]# cat /root/.mysql_secret
#The random password set for the root user at Tue Dec 18 20:37:59 2018 (local time): pP4xR06U9nPyWrhA
登錄MySQL:
[root@hadoop01 wjg]# mysql -uroot -ppP4xR06U9nPyWrhA
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.42
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('xxxxxxx')
grant all on . to root@"%" identified by " xxxxxxx "
設(shè)置MySQL開機(jī)自啟動(dòng)
systemctl enable mysql.service
初始化MySQL
/usr/bin/mysql_secure_installation
下載mysql-connector-java-5.1.46.tar.gz并解壓
在解壓目錄下找到mysql-connector-java.jar, 復(fù)制到所有主機(jī)的/usr/share/java目錄下
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar
[root@hadoop01 usr]# ls -l /usr/share/java/mysql-connector-java.jar
-rw-r--r-- 1 root root 1004838 Dec 19 20:45 /usr/share/java/mysql-connector-java.jar
下載Oracle JDK包oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm上傳到所有主機(jī)并安裝
下載地址:https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
Creating Databases for Cloudera Software
需要建的庫(kù)有
scm、amon、rman、hue、metastore、sentry、nav、navms、oozie
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm. TO 'scm'@'%' IDENTIFIED BY 'scm@123';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman. TO 'rman'@'%' IDENTIFIED BY 'rman@123';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue. TO 'hue'@'%' IDENTIFIED BY 'hue@123';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore. TO 'metastore'@'%' IDENTIFIED BY 'metastore@123';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry. TO 'sentry'@'%' IDENTIFIED BY 'sentry@123';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav. TO 'nav'@'%' IDENTIFIED BY 'nav@123';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms. TO 'navms'@'%' IDENTIFIED BY 'navms@123';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie. TO 'oozie'@'%' IDENTIFIED BY 'oozie@123';
初始化cloudera manager數(shù)據(jù)庫(kù)
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
四、下載并安裝Cloudera Manager
https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
在主節(jié)點(diǎn)hadoop01服務(wù)器上安裝1,2,3;
在Agent服務(wù)器上安裝1,3;
在主節(jié)點(diǎn)hadoop01上執(zhí)行:
systemctl start cloudera-scm-server
systemctl restart cloudera-scm-server
查看日志:
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
顯示如下表示服務(wù)已經(jīng)正常起來
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
Cloudera Manager將自動(dòng)下載文件到/opt/cloudera/parcel-repo目錄下
CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcel
CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcel.sha
CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcel.torrent
manifest.json
在所有Agent節(jié)點(diǎn)hadoop01, hadoop02, hadoop06上執(zhí)行:
配置Agent的Server指向
vi /etc/cloudera-scm-agent/config.ini
[General]
#Hostname of the CM server.
server_host=hadoop01
[Security]
#Use TLS and certificate validation when connecting to the CM server.
use_tls=1
啟動(dòng)cloudera-scm-agent服務(wù):
systemctl status cloudera-scm-agent
systemctl start cloudera-scm-agent
systemctl restart cloudera-scm-agent
Agent Log文件位置:/var/log/cloudera-scm-agent
cloudera-scm-agent.log
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log
登錄Cloudera Manager管理界面
http://192.168.1.101:7180
用戶名和密碼都是admin
Cloudera 安裝Parcel提示主機(jī)運(yùn)行狀況不良
解決方法
刪除agent目錄下面的cm_guid文件,并重啟失敗節(jié)點(diǎn)的agent服務(wù)。
[root@hadoop06 ~]# cd /var/lib/cloudera-scm-agent/
[root@hadoop06 cloudera-scm-agent]# ll
total 32
-rw------- 1 root root 2 Dec 28 14:27 active_parcels.json
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Dec 28 23:01 agent-cert
-rw-r--r-- 1 root root 36 Dec 28 14:25 cm_guid
-rw------- 1 root root 13813 Dec 29 07:10 response.avro
-rw-r--r-- 1 root root 36 Dec 28 14:25 uuid
[root@hadoop06 cloudera-scm-agent]# rm -rf cm_guid
[root@hadoop06 cloudera-scm-agent]# systemctl restart cloudera-scm-agent
在所有節(jié)點(diǎn)上刪除cm_guid以后,cloudera manager會(huì)自動(dòng)繼續(xù)進(jìn)行安裝。
稍微等一段時(shí)間后就會(huì)看到parcel已經(jīng)安裝到所有主機(jī)上并全部激活。
點(diǎn)擊繼續(xù)就會(huì)進(jìn)入檢查主機(jī)正確性的界面
第1個(gè)警告:Cloudera 建議將 /proc/sys/vm/swappiness 設(shè)置為最大值 10。當(dāng)前設(shè)置為 30。使用 sysctl 命令在運(yùn)行時(shí)更改該設(shè)置并編輯 /etc/sysctl.conf,以在重啟后保存該設(shè)置。您可以繼續(xù)進(jìn)行安裝,但 Cloudera Manager 可能會(huì)報(bào)告您的主機(jī)由于交換而運(yùn)行狀況不良。以下主機(jī)將受到影響:
查看詳細(xì)信息
hadoop[01-02, 06]
解決辦法:
臨時(shí)調(diào)整: sysctl vm.swappiness=10
永久調(diào)整: vi /etc/sysctl.conf
增加下面一行
vm.swappiness=10
第2個(gè)警告:已啟用透明大頁(yè)面壓縮,可能會(huì)導(dǎo)致重大性能問題。請(qǐng)運(yùn)行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此設(shè)置,然后將同一命令添加到 /etc/rc.local 等初始化腳本中,以便在系統(tǒng)重啟時(shí)予以設(shè)置。以下主機(jī)將受到影響:
查看詳細(xì)信息
hadoop[01-02, 06]
解決辦法:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
修改/etc/rc.local將上面兩行添加進(jìn)去
vi /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
第3個(gè)警告:psycopg2版本太低的處理
安裝pip
yum -y install epel-release
yum -y install python-pip
pip install --upgrade pip
yum install postgresql-devel
安裝psycopg2
下載psycopg2-2.7.6.1.tar.gz并解壓
[root@hadoop01 psycopg2-2.7.6.1]# python setup.py build
[root@hadoop01 psycopg2-2.7.6.1]# python setup.py install
重新運(yùn)行檢查主機(jī),所有項(xiàng)目檢查通過
點(diǎn)擊繼續(xù)進(jìn)入選擇安裝產(chǎn)品界面
選擇所有服務(wù)然后點(diǎn)擊繼續(xù),進(jìn)入主機(jī)角色分配界面
點(diǎn)擊按主機(jī)查看對(duì)主機(jī)分配的角色會(huì)看得更清楚一些
點(diǎn)擊繼續(xù)進(jìn)入數(shù)據(jù)庫(kù)設(shè)置界面,填寫先前創(chuàng)建的數(shù)據(jù)庫(kù)名稱、用戶名和密碼
點(diǎn)擊繼續(xù)進(jìn)入審核更改界面
點(diǎn)擊繼續(xù)開始進(jìn)行安裝
安裝完成后點(diǎn)擊繼續(xù),進(jìn)入Summary界面
https://192.168.1.101:7183/cmf/home
本文題目:CDH6.0.1詳細(xì)安裝步驟
網(wǎng)址分享:http://vcdvsql.cn/article12/gjeddc.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供手機(jī)網(wǎng)站建設(shè)、品牌網(wǎng)站建設(shè)、移動(dòng)網(wǎng)站建設(shè)、小程序開發(fā)、定制開發(fā)、網(wǎng)站建設(shè)
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來源: 創(chuàng)新互聯(lián)