Hadoop - 설정 파일

hadoop-env.sh

• Hadoop을 실행하는 쉘스크립트 파일, JDK 경로, Classpath, 데몬 옵션 등 설정

[root@server01 ~]# vi hadoop-env.sh

--------------------------------------------------------------------------  수정
export JAVA_HOME=/usr/local/jdk1.8.0_221
---------------------------------------------------------------------------------

--------------------------------------------------------------------------  추가
export HADOOP_HOME=/usr/local/hadoop-2.9.2
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}
---------------------------------------------------------------------------------

---

Slaves

• Data node 들의 서버 지정

------- Data node 들의 서버 지정
------- 기존에 있는 로컬호스트는 삭제


[root@server01 ~]# vi slaves

----- localhost 삭제
--------------------------------------------------------------------------  추가
server02
server03
--------------------------------------------------------------------------

---

core-site.xml

• HDFS와 Mapreduce에서 공통적으로 사용할 정보들을 설정, hdfs-site와 mapred-site의 공통 설정 부분

[root@server01 ~]# vi core-site.xml

<configuration>
--------------------------------------------------------------------------  추가
        <property>
                <name>fs.default.name</name>
                <value>hdfs://server01:9000</value>     : 마스터 서버의 이름 
        </property>
        <property>
                 <name>hadoop.tmp.dir</name>
                 <value>/usr/local/hadoop-2.9.2/tmp</value> : 임시 디렉토리
         </property>
---------------------------------------------------------------------------------
</configuration>

---

hdfs-site.xml

• 하둡 파일시스템(HDFS)과 관련된 환경 정보를 설정

[root@server01 ~]# vi hdfs-site.xml

<configuration>
--------------------------------------------------------------------------  추가
       <property>
                <name>dfs.replication</name>
                ---- 데이터를 1개만 복사:가상분산모드, 3일경우:완전분산모드
                <value>3</value> 
        </property>
        <property>
                <name>dfs.permissions.enabled</name>
                <value>false</value>
        </property>
        <property>
                <name>dfs.webhdfs.enabled</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.namenode.http.address</name>
                <value>server01:50070</value>
        </property>
        <property>
                <name>dfs.secondary.http.address</name>
                <value>server01:50090</value>
        </property>

---------------------------------------------------------------------------------
</configuration>

---

mapred-site.xml

• MapReduce의 어플리케이션 정보를 설정

 

---- mapred-site.xml 기본적으로는 해당 파일이 없다.
---- mapred-site.xml이 존재하지 않을 경우 mapred-site.xml.template를 복사하여 사용

[root@server01 ~]# cp mapred-site.xml.template mapred-site.xml
[root@server01 ~]# vi mapred-site.xml

<configuration>
--------------------------------------------------------------------------  추가
     <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
   <property>
        <name>mapred.job.tracker</name>
        <value>server01:9001</value>
    </property>

---------------------------------------------------------------------------------
</configuration>

---

yarn-site.xml

• Resource Manager, Node Manager 정보를 설정

[root@server01 ~]# vi yarn-site.xml

<configuration>
--------------------------------------------------------------------------  추가
    <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
    </property>
    <property>
            <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
            <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
            <value>server01:8025</value>
    </property>
    <property>
            <name>yarn.resourcemanager.scheduler.address</name>
            <value>server01:8030</value>
    </property>
    <property>
            <name>yarn.resourcemanager.address</name>
            <value>server01:8040</value>
    </property>

---------------------------------------------------------------------------------
</configuration>

 

출처 : https://wikidocs.net/63337