Hadoop伪分布式安装

By | 04月23日
Advertisement

卸载JDK:

[[email protected] mzsx]# rpm -qa | grep java
java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.x86_64
tzdata-java-2012c-1.el6.noarch
[[email protected] mzsx]# rpm -e --nodeps tzdata-java-2012c-1.el6.noarch
[[email protected] mzsx]# rpm -e --nodeps  java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.x86_64

安装JDK:

[[email protected] mzsx]# chmod 777 jdk-7u45-linux-x64.rpm
[[email protected] mzsx]# rpm -ivh jdk-7u45-linux-x64.rpm
Preparing...                ########################################### [100%]
   1:jdk                    ########################################### [100%]
Unpacking JAR files...
    rt.jar...
    jsse.jar...
    charsets.jar...
    tools.jar...
    localedata.jar...
    jfxrt.jar...

配置JDK环境:

[[email protected] bin]#vim  /etc/profile
export JAVA_HOME=/usr/local/jdk
export JAVA_BIN=/usr/local/jdk/bin
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/usr/local/hadoop
export PATH=/usr/local/hadoop/bin:$PATH
export HADOOP_HOME_WARN_SUPPRESS=1
[[email protected] bin]#source /etc/profile

关闭防火墙的方法为:

. 永久性生效
开启:chkconfig iptables on
关闭:chkconfig iptables off
[[email protected] ~]# service network restart
关闭环回接口:                                             [确定]
弹出环回接口:                                             [确定]
[[email protected] ~]#

免密钥登录,注:一直回车

[[email protected] jdk1.7.0_45]# cd /root/
[[email protected] ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
51:19:ce:ed:ae:f1:57:d4:30:c8:a2:a3:d7:8d:e1:2d [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
|          o+ .   |
|         +o.o o  |
|        ..o..  o.|
|        o...    o|
|       .S+ =.  . |
|      . . E.o   .|
|       .  ...  . |
|           +  .  |
|          . ..   |
+-----------------+
[[email protected] ~]# cd .ssh
[[email protected] .ssh]# ll -a
总用量 16
drwx------.  2 root root 4096 11月  8 03:07 .
dr-xr-x---. 26 root root 4096 11月  8 03:03 ..
-rw-------.  1 root root 1675 11月  8 03:07 id_rsa
-rw-r--r--.  1 root root  408 11月  8 03:07 id_rsa.pub
[[email protected] .ssh]# cp id_rsa.pub authorized_keys
[[email protected] .ssh]# serv
servertool   service      serviceconf
[[email protected] .ssh]# serv
servertool   service      serviceconf
[[email protected] .ssh]# service sshd restart
停止 sshd:                                                [确定]
正在启动 sshd:                                            [确定]
[[email protected] .ssh]#

HADOOP配置:

core-site.xml

<configuration>
    <property>
        <name>hadoop.tem.dir</name>
        <value>/home/hadoop/tem</value>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://192.168.197.131:9000</value>
    </property>
</configuration>

hdfs-site.xml

<configuration>
    <property>
        <name>dfs.data.dir</name>
        <value>/home/hadoop/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
</configuration>

mapred-site.xml

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>192.168.197.131:9001</value>
    </property>
</configuration>

hadoop-env.sh

# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr/local/jdk

格式化分布式文件系统

[[email protected] bin]# ./hadoop namenode -format
Warning: $HADOOP_HOME is deprecated.
13/11/08 03:30:03 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.0
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473; compiled by 'hortonfo' on Mon May  6 06:59:37 UTC 2013
STARTUP_MSG:   java = 1.7.0_45
************************************************************/
13/11/08 03:30:03 INFO util.GSet: Computing capacity for map BlocksMap
13/11/08 03:30:03 INFO util.GSet: VM type       = 64-bit
13/11/08 03:30:03 INFO util.GSet: 2.0% max memory = 1013645312
13/11/08 03:30:03 INFO util.GSet: capacity      = 2^21 = 2097152 entries
13/11/08 03:30:03 INFO util.GSet: recommended=2097152, actual=2097152
13/11/08 03:30:03 INFO namenode.FSNamesystem: fsOwner=root
13/11/08 03:30:04 INFO namenode.FSNamesystem: supergroup=supergroup
13/11/08 03:30:04 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/11/08 03:30:04 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/11/08 03:30:04 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/11/08 03:30:04 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
13/11/08 03:30:04 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/11/08 03:30:04 INFO common.Storage: Image file of size 110 saved in 0 seconds.
13/11/08 03:30:04 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/tmp/hadoop-root/dfs/name/current/edits
13/11/08 03:30:04 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/tmp/hadoop-root/dfs/name/current/edits
13/11/08 03:30:04 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
13/11/08 03:30:04 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
************************************************************/

启动Hadoop

[[email protected] bin]# ./start-all.sh
starting namenode, logging to /home/mzsx/hadoop-0.20.2/bin/../logs/hadoop-root-namenode-hadoop.out
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 0f:68:b7:e0:2d:13:38:4b:44:ac:ec:a3:7f:b8:f6:7e.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
localhost: starting datanode, logging to /home/mzsx/hadoop-0.20.2/bin/../logs/hadoop-root-datanode-hadoop.out
localhost: starting secondarynamenode, logging to /home/mzsx/hadoop-0.20.2/bin/../logs/hadoop-root-secondarynamenode-hadoop.out
starting jobtracker, logging to /home/mzsx/hadoop-0.20.2/bin/../logs/hadoop-root-jobtracker-hadoop.out
localhost: starting tasktracker, logging to /home/mzsx/hadoop-0.20.2/bin/../logs/hadoop-root-tasktracker-hadoop.out

查看与Java相关的线程信息

[[email protected] bin]# jps
14828 DataNode
15600 Bootstrap
23062 Bootstrap
15232 TaskTracker
18636 Jps
2935 Bootstrap
14982 SecondaryNameNode
23445 Bootstrap
14652 NameNode
15080 JobTracker

fs操作

查看目录和文件
[[email protected] bin]# ./hadoop fs -lsr /
drwxr-xr-x   - root supergroup          0 2014-04-23 17:57 /tmp
drwxr-xr-x   - root supergroup          0 2014-04-23 17:57 /tmp/hadoop-root
drwxr-xr-x   - root supergroup          0 2014-04-23 18:38 /tmp/hadoop-root/mapred
drwxr-xr-x   - root supergroup          0 2014-04-23 18:38 /tmp/hadoop-root/mapred/staging
drwxr-xr-x   - root supergroup          0 2014-04-23 18:38 /tmp/hadoop-root/mapred/staging/root
drwx------   - root supergroup          0 2014-04-23 18:38 /tmp/hadoop-root/mapred/staging/root/.staging
drwx------   - root supergroup          0 2014-04-23 18:38 /tmp/hadoop-root/mapred/system
-rw-------   1 root supergroup          4 2014-04-23 17:57 /tmp/hadoop-root/mapred/system/jobtracker.info
上传文件
[[email protected] bin]# ./hadoop fs -put /home/hadoop/aoman.txt ./
执行wordcount
[[email protected] bin]# bin/hadoop jar hadoop-examples-1.2.1.jar  wordcount /user/root/aoman.txt ./in

下载文件

[[email protected] bin]# ./hadoop fs -get /user/root/in/part-r-00000 /home/hadoop/wordcount.txt

Similar Posts:

  • Hadoop伪分布式安装流程

    伪分布式模式的安装步骤: 1.关闭防火墙 如果是对外网提供的,没有必要关闭防火墙 Hadoop集群一般是公司内部,多台几点之间要进行通信,需要占用一些端口 但是这些端口有可能被防火墙保护起来,被拦截,这时我们需要关闭,其目的是就是减少出现不必要的麻烦 2.修改IP 每一台机器都必须要有一个固定的IP 3.修改Hostname 主机名可以方便定位错误源 4.设置ssh自动登录 5.安装JDK 6.安装Hadoop 0.使用host-only方式 将Windows上的虚拟网卡改成跟Linux上的网卡

  • Centos 7中的HaDoop伪分布式安装

    简介: windows 7中安装VMware,VMware中安装Centos 7,Centos 7中安装伪分布式HaDoop 请提前下载jdk和hadoop 至于如何在windows中安装VMware以及在VMware中Centos 7, 请参照博主的博客: Windows7下用VMware安装Centos7 1. 安装前的重要事情 (1)添加root用户 su---使用su命令进入root(需要roo密码) vi /etc/sudoers---–使用此命令修改文件sudoers中的内容 [关于

  • hadoop伪分布式安装步骤

    1.设置静态ip(例子用 host-only模式) 1)在linux系统桌面右上角的图标上,右键修改 2)重启网卡 看看是否修改成功,执行命令service network restart 验证 执行命令 ifconfig 2 修改主机名 1)修改当前会话中的主机名 执行命令 hostname 主机名 2)修改配置文件的主机名 vi /etc/sysconfig/network 验证重启机器 reboot -h now 3 把hostname和ip绑定 执行命令 vi /etc/hosts 增加

  • centos 7下Hadoop 2.7.2 伪分布式安装

    centos 7 下Hadoop 2.7.2 伪分布式安装,安装jdk,免密匙登录,配置mapreduce,配置YARN.详细步骤如下: 1.0 安装JDK 1.1 查看是否安装了openjdk [[email protected] ~]$ java -version openjdk version "1.8.0_65" OpenJDK Runtime Environment (build 1.8.0_65-b17) OpenJDK 64-Bit Server VM (build 25.65-b01,

  • 【hadoop学习】--(2)安装和配置hadoop伪分布式

    先上一张成功的截图 下面是安装和配置的过程 ----------------------------------- 接上文 [hadoop学习]--(1)hadoop安装前准备 9 安装hadoop <1>安装 tar -zvxf hadoop-1.1.2.tar.gz mv hadoop-1.1.2 /usr/lib/hadoop <2>环境变量 执行gedit /etc/profile添加和修改后为: export JAVA_HOME=/usr/lib/jvm export H

  • hadoop单机伪分布式安装

    转自:http://www.powerxing.com/install-hadoop/ 这是比较详细的手把手的教程了.我安装时,修改安装路径到当前登录用户下面,避免了一些因为对linux不熟悉导致的权限问题. 当开始着手实践 Hadoop 时,安装 Hadoop 往往会成为新手的一道门槛.尽管安装其实很简单,书上有写到,官方网站也有 Hadoop 安装配置教程,但由于对 Linux 环境不熟悉,书上跟官网上简略的安装步骤新手往往 Hold 不住.加上网上不少教程也甚是坑,导致新手折腾老几天愣是没

  • Hadoop入门--hadoop2伪分布式安装

    伪分布式安装: 1.java环境变量配置 在.bash_profile文件中加入如下环境变量: ####################JAVAHOME#################### exportJAVA_HOME=/usr/lib/jvm/jre-1.6.0-openjdk.x86_64 exportJRE_HOME=$JAVA_HOME/jre exportCLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib exportPATH=$JAVA_

  • ubuntu12.04(64位)下hadoop1.0.4 伪分布式安装

    主要分为以下几个步骤: 安装java环境 配置环境变量 配置ssh 配置hadoop为单节点[伪分布式]模式A 一.安装JDK 1 下载并安装 sudo apt-get install openjdk-6-jdk ( 安装JDK7为:sudo apt-get install openjdk-7-jdk) 2 在命令行下输入java -version 查看是否安装成功 [email protected]:/home/ubuntu/hadoop-1.0.4# java -version java version

  • 超详细解说Hadoop伪分布式搭建--实战验证【转】

    超详细解说Hadoop伪分布式搭建 原文http://www.tuicool.com/articles/NBvMv2 原原文 http://wojiaobaoshanyinong.iteye.com/blog/1946817 单节点伪分布式Hadoop配置 (声明:文档里面需要用户输入的均已斜体表示) 第一步: 安装JDK 因为 Hadoop 运行必须安装 JDK 环境,因此在安装好 Linux 后进入系统的第一步 便是安装 JDK ,安装过程和在 Windows 环境中的安装步骤很类似,首先去

  • 【Spark】Spark一:Spark伪分布式安装

    Spark1.2.0伪分布式安装 JDK1.7.0_67 Haddoop2.5.2(伪分布式安装) Scala2.11.4 Spark1.2.0(伪分布式安装) 下载并配置Scala 1. 下载Scala2.11.4 wget http://downloads.typesafe.com/scala/2.11.4/scala-2.11.4.tgz?_ga=1.10268349.906591.1419820693 2. 改名 mv scala-2.11.4.tgz?_ga=1.10268349.90

Tags: