大数据之----部署安装编译打包hadoop终极篇-大数据-互联网-天达云

大数据之----部署安装编译打包hadoop终极篇
更新：HHH 时间：2023-1-7

------------------------------软件版本--------------------------------------
RHEL6.8 hadoop2.8.1 apache-maven-3.3.9

findbugs-1.3.9 protobuf-2.5.0.tar.gz jdk-8u45

------------------------------软件版本---------------------------------------

大数据概述可以用“5V + 1C”来概括：

Variety 多样化
Volume 海量
Velocity 快速
Vitality 灵活
Value 价值性
Complexity 复杂

1.Hadoop生态
宏观: Hadoop为主的生态圈 hadoop flume........
狭义: apache hadoop hadoop.apache.org

2.Hadoop(存储+计算+资源和作业调度)
hadoop1.x
    HDFS      存储
    MapReduce 计算+资源和作业调度

hadoop2.x 企业正在用
    HDFS      存储
    MapReduce 计算
    YARN      资源和作业调度平台计算组件都会on yarn

hadoop3.x ???

EC技术：Erasure Encoding 简称EC，是Hadoop3给HDFS拓展的一种新特性，用来解决存储空间文件。
YARN：提供YARN的时间轴服务V.2，以便用户和开发人员可以对其进行测试，并提供反馈意见。
优化Hadoop Shell脚本
重构Hadoop Client Jar包
支持随机Container
MapReduce任务级本地优化
支持多个NameNode
部分默认服务端口被改变
支持文件系统连接器
DataNode内部添加了负载均衡
重构后台程序和任务对管理

2.Maven部署

blog
2.1解压

[root@hadoop1 softwore]# pwd
/opt/softwore
[root@hadoop1 softwore]# ls
apache-maven-3.3.9-bin.zip hadoop-2.8.1-src.tar.gz jdk-8u45-linux-x64.gz
findbugs-1.3.9.zip hadoop-2.8.1.tar.gz protobuf-2.5.0.tar.gz
[root@hadoop1 softwore]# unzip apache-maven-3.3.9-bin.zip

--配置环境变量

[root@hadoop1 softwore]#

vi /etc/profile
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 apache-maven-3.3.9]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

Maven部署为二进制部署，解压配置环境变量

2.2配置mavne目录
2.3查看配置文件和解压我们准备好的仓库文件

3.Hadoop编译

编译hadoop 通过Requirements我看可以知道需要安装jdk maven findbugs protocolBuffer cmake Zlib openssl-devel 等软件
3.1解压
3.2查看pom.xml
3.3查看BUILDING.txt
Requirements:编译软件环境要求

* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

3.4 JDK部署

[root@hadoop1 softwore]# tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java[root@hadoop1 softwore]# ls -ld /usr/java/*
drwxr-xr-x 8 root root 4096 12月 15 2016 /usr/java/djdk1.7.0_79
drwxr-xr-x 8 uucp 143 4096 4月 11 2015 /usr/java/jdk1.8.0_45
drwxr-xr-x 8 uucp 143 4096 10月 7 2015 /usr/java/jdk1.8.0_65
[root@hadoop1 softwore]# vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45
export JRE_HOME=/usr/java/jdk1.8.0_45/jre

[root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

二进制版本解压，配置环境变量即可

3.5 MAVEN

上面我们已经安装maven和jdk检测版本是否满足我们编译软件要求
[root@hadoop1 softwore]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
[root@hadoop1 softwore]#

3.6 FINDBUGS

[root@hadoop1 softwore]# unzip findbugs-1.3.9.zip
[root@hadoop1 softwore]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH [root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# findbugs -version
1.3.9

二进制安装解压，配置环境变量即可

3.7 PROTOCBUF编译安装

[root@hadoop1 software]# tar xvf protobuf-2.5.0.tar.gz
[root@hadoop1 yum.repos.d]# yum install -y gcc gcc-c++ make cmake ---编译环境
[root@hadoop1 yum.repos.d]#cd /opt/software/protobuf-2.5.0
[root@hadoop1 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop1 protobuf-2.5.0]# make && make install
[root@hadoop1 protobuf-2.5.0]# vim /etc/profile
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop1 software]# source /etc/profile
[root@hadoop1 software]# protoc --version
libprotoc 2.5.0
[root@hadoop000 local]#

编译安装完成，配置环境变量

3.8 OTHER 依赖包

yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel \n
zo lzo-devel lzop autoconf automake

其他依赖包在系统安装ios都包含，执行yum安装即可

---------------------------------------------------------------------------------------------------------------------------------------

至此，编译hadoop需要的编译环境全部准备好，下面开启编译之路

---------------------------------------------------------------------------------------------------------------------------------------

3.9 编译

[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]#
mvn clean package -Pdist,native -DskipTests –Dtar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

----编译好软件包保存位置

直接编译需要联网找依赖包，一般网络情况较好，需要几个小时

提前准备好依赖包编译方式

[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src

[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# ls pom.xml ---根据这个文件下载jar
pom.xml
[root@hado

-rw-r--r-- 1 root root 96721446 4月 25 09:57 .m2.tar.gz ---提前下好的仓库软件包

[root@hadoop1 software]#tar xvf .m2.tar.gz –C /root ---不想download，使用提前准备好的仓库软件--->解压----->root家目录

[root@hadoop1 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests –Dtar

……..

[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.562 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 13.396 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.969 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.050 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.110 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 3.862 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.029 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:01 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:45 min
[INFO] Finished at: 2018-04-25T14:03:20+08:00
[INFO] Final Memory: 161M/479M
[INFO] ------------------------------------------------------------------------

[root@hadoop1 hadoop-2.8.1-src]# 编译成功后的软件保存路径

[root@hadoop1 hadoop-2.8.1-src]# ls /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-*
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-javadoc.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-test-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1:
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
[root@hadoop1 hadoop-2.8.1-src]# du -sh /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
186M /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

总结：完成源码的编译好打包，得到安装hadoop安装包，难点在于软件包的装备和仓库软件的准备，接下来准备开始hadoop安装之路。


返回大数据教程...

新手上路

产品管理

支付方式

关于我们