Hadoop运行环境搭建( 四 )


1)配置core-site.xml
[haoll@hadoop102 ~]$ cd $HADOOP_HOME/etc/hadoop[haioll@hadoop102 hadoop]$ vim core-site.xml文件内容如下:
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href=https://www.isolves.com/it/rj/jy/2023-08-21/"configuration.xsl"?>fs.defaultFShdfs://hadoop01:8020hadoop.tmp.dir/opt/module/hadoop-3.1.3/datahadoop.http.staticuser.userhll(2)HDFS配置文件
配置hdfs-site.xml
[hall@hadoop102 hadoop]$ vim hdfs-site.xml文件内容如下:
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href=https://www.isolves.com/it/rj/jy/2023-08-21/"configuration.xsl"?>dfs.namenode.http-addresshadoop01:9870dfs.namenode.secondary.http-addresshadoop03:98683)YARN配置文件
配置yarn-site.xml
[haoll@hadoop102 hadoop]$ vim yarn-site.xml文件内容如下:
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href=https://www.isolves.com/it/rj/jy/2023-08-21/"configuration.xsl"?>yarn.nodemanager.aux-servicesmapreduce_shuffleyarn.resourcemanager.hostnamehadoop02yarn.nodemanager.env-whitelistJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME4)MapReduce配置文件
配置mapred-site.xml
[haoll@hadoop102 hadoop]$ vim mapred-site.xml文件内容如下:
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href=https://www.isolves.com/it/rj/jy/2023-08-21/"configuration.xsl"?>mapreduce.framework.nameyarn4)在集群上分发配置好的Hadoop配置文件
[hll@hadoop01 hadoop]$ xsync /opt/module/hadoop-3.1.3/etc/hadoop/
5)去02和03上查看文件分发情况
[hll@hadoop02 ~]$ cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml
[hll@hadoop03 ~]$ cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml
群起集群1)配置workers
[hall@hadoop102 hadoop]$ vim /opt/module/hadoop-3.1.3/etc/hadoop/workers在该文件中增加如下内容:
hadoop01
hadoop02
hadoop03
注意:该文件中添加的内容结尾不允许有空格,文件中不允许有空行 。
同步所有节点配置文件
[hall@hadoop01 hadoop]$ xsync /opt/module/hadoop-3.1.3/etc
2)启动集群
(1)如果集群是第一次启动,需要在hadoop01节点格式化NameNode(注意:格式化NameNode,会产生新的集群id,导致NameNode和DataNode的集群id不一致,集群找不到已往数据 。如果集群在运行过程中报错,需要重新格式化NameNode的话,一定要先停止namenode和datanode进程,并且要删除所有机器的data和logs目录,然后再进行格式化 。)
[hll@hadoop01 hadoop-3.1.3]$ hdfs namenode -format
(2)启动HDFS
[hll@hadoop01 hadoop-3.1.3]$ sbin/start-dfs.sh
(3)在配置了ResourceManager的节点(hadoop02)启动YARN
[hll@hadoop02 hadoop-3.1.3]$ sbin/start-yarn.sh
(4)Web端查看HDFS的NameNode
(a)浏览器中输入:http://hadoop01:9870
(b)查看HDFS上存储的数据信息
(5)Web端查看YARN的ResourceManager
(a)浏览器中输入:http://hadoop02:8088
(b)查看YARN上运行的Job信息
3)集群基本测试
(1)上传文件到集群
上传小文件
[haoll@hadoop01 ~]$ hadoop fs -mkdir /opt/module/hadoop-3.1.3/input[haoll@hadoop01 ~]$ hadoop fs -put /wcinput/word.txt /input上传大文件[haoll@hadoop01 ~]$ hadoop fs -put/opt/software/jdk-8u212-linux-x64.tar.gz/(2)上传文件后查看文件存放在什么位置
查看HDFS文件存储路径
[haoll@hadoop01 subdir0]$ pwd/opt/module/hadoop-3.1.3/data/dfs/data/current/BP-1436128598-192.168.10.102-1610603650062/current/finalized/subdir0/subdir0查看HDFS在磁盘存储文件内容[haoll@hadoop01 subdir0]$ cat blk_1073741825hadoop yarnhadoop mapreduce haollhaoll


推荐阅读