1.准备环境
1.1配置dns
# cat /etc/hosts 172.27.133.60 hadoop-01 172.27.133.61 hadoop-02 172.27.133.62 hadoop-03 172.27.133.63 hadoop-04
1.2配置免密登陆
# ssh-keygen# ssh-copy-id root@hadoop-02/03/04
1.3关闭防火墙
# cat /etc/selinux/config SELINUX=disabled# systemctl stop firewalld# systemctl disable firewalld
1.4配置Java环境,Hadoop环境
# tar -xf /data/software/jdk-8u171-linux-x64.tar.gz -C /usr/local/java# tar -xf /data/software/hadoop-3.2.0.tar.gz -C /data/hadoop# cat /etc/profile export HADOOP_HOME=/data/hadoop/hadoop-3.2.0 export JAVA_HOME=/usr/local/java/jdk1.8.0_171 export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop# source /etc/profile# java -version java version "1.8.0_171" Java(TM) SE Runtime Environment (build 1.8.0_171-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode) 2.配置Hadoop
2.1配置Hadoop环境脚本文件中的JAVA_HOME参数
# cd /data/hadoop/hadoop-3.2.0/etc/hadoop# vim hadoop-env.sh 和 mapred-env.sh 和 yarn-env.sh，向脚本添加JAVA_HOME export JAVA_HOME=/usr/local/java/jdk1.8.0_171# hadoop version Hadoop 3.2.0 Source code repository https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf Compiled by sunilg on 2019-01-08T06:08Z Compiled with protoc 2.5.0 From source with checksum d3f0795ed0d9dc378e2c785d3668f39 This command was run using /data/hadoop/hadoop-3.2.0/share/hadoop/common/hadoop-common-3.2.0.jar
2.2修改Hadoop配置文件
在hadoop-3.2.0/etc/hadoop目录下，修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、workers文件，具体参数按照实际情况修改。
2.2.1修改core-site.xml
（文件系统）
fs.defaultFS hdfs://hadoop-01:9000 hadoop.tmp.dir /data/hadoop/tmp
# mkdir /data/hadoop/tmp
2.2.2修改hdfs-site.xml
(副本数)
dfs.namenode.http-address hadoop-01:50070 dfs.secondary.http.address hadoop-02:50070 dfs.namenode.name.dir file:/data/hadoop/dfs/name dfs.datanode.data.dir file:/data/hadoop/dfs/data dfs.replication 2 dfs.webhdfs.enabled true dfs.permissions false 配置为false后，可以允许不要检查权限就生成dfs上的文件，方便倒是方便了，但是你需要防止误删除.
2.2.3修改mapred-site.xml
（资源调度框架）
mapreduce.framework.name yarn mapreduce.jobhistory.address hadoop-01:10020 mapreduce.jobhistory.webapp.address hadoop-01:19888 mapreduce.application.classpath $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
2.2.4修改yarn-site.xml
yarn.resourcemanager.hostname hadoop-01 yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.webapp.address 172.27.133.60:8088 配置外网只需要替换外网ip为真实ip，否则默认为 localhost:8088 yarn.scheduler.maximum-allocation-mb 2048 每个节点可用内存,单位MB,默认8182MB yarn.nodemanager.vmem-check-enabled false 忽略虚拟内存的检查，如果你是安装在虚拟机上，这个配置很有用，配上去之后后续操作不容易出问题。 yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
2.2.5修改workers
添加从节点地址，IP或者hostname
hadoop-02hadoop-03hadoop-04
2.3将配置好的文件夹拷贝到其他从节点
# scp -r /data/hadoop root@hadoop-02:/data/hadoop# scp -r /data/hadoop root@hadoop-03:/data/hadoop# scp -r /data/hadoop root@hadoop-04:/data/hadoop
2.4配置启动脚本Yarn权限
2.4.1添加hdfs权限
在第二行空白位置添加hdfs权限# cat sbin/start-dfs.sh# cat sbin/stop-dfs.sh #!/usr/bin/env bash HDFS_DATANODE_USER=root HDFS_DATANODE_SECURE_USER=hdfs HDFS_NAMENODE_USER=root HDFS_SECONDARYNAMENODE_USER=root ···
2.4.2添加Yarn权限
在第二行空白位置添加Yarn权限# cat sbin/start-yarn.sh# cat sbin/stop-yarn.sh #!/usr/bin/env bash YARN_RESOURCEMANAGER_USER=root HDFS_DATANODE_SECURE_USER=yarn YARN_NODEMANAGER_USER=root ···
如果不添加权限，在启动时候会报错
ERROR: Attempting to launch hdfs namenode as rootERROR: but there is no HDFS_NAMENODE_USER defined. Aborting launch.Starting datanodesERROR: Attempting to launch hdfs datanode as rootERROR: but there is no HDFS_DATANODE_USER defined. Aborting launch.Starting secondary namenodes [localhost.localdomain]ERROR: Attempting to launch hdfs secondarynamenode as rootERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting launch.
3.初始化并启动
3.1格式化hdfs
# bin/hdfs namenode -format
3.2启动
3.2.1一步启动
# sbin/start-all.sh
3.2.2分布启动
# sbin/start-dfs.sh# sbin/start-yarn.sh
4.验证
4.1列出进程
主节点hadoop-01
# jps32069 SecondaryNameNode2405 NameNode2965 ResourceManager3645 Jps
从节点hadoop-02
# jps25616 NodeManager25377 DataNode25508 SecondaryNameNode25945 Jps
从节点hadoop-03
# jps14946 NodeManager15207 Jps14809 DataNode
从节点hadoop-04
# jps14433 Jps14034 DataNode14171 NodeManager4
4.2访问页面
http://hadoop-01:8088打开ResourceManager页面
http://hadoop-01:50070打开Hadoop Namenode页面
