远程连接hadoop分布式环境
1、确保分布式环境版本与eclipse插件版本要一致(0.20.205.0),否则连接是提示:
2、插件重新打包,需要把
lib/jackson-core-asl-1.8.8.jar,lib/jackson-mapper-asl-1.8.8.jar,lib/commons-configuration-1.6.jar,lib/commons-lang-2.4.jar,lib/commons-httpclient-3.0.1.jar,lib/commons-cli-1.2.jar,打进插件包中,如下:
MANIFEST.MF修改:
Bundle-ClassPath: classes/,lib/hadoop-core.jar
,lib/jackson-core-asl-1.8.8.jar,lib/jackson-mapper-asl-1.8.8.jar,lib/commons-configuration-1.6.jar,lib/commons-lang-2.4.jar,lib/commons-httpclient-3.0.1.jar,lib/commons-cli-1.2.jar
否则出现:
关于eclipse无法连接报错:
"Map/Reduce location status updater". org/codehaus/jackson/map/JsonMappingException
经过查询,是由于hadoop的eclipse 插件里面缺少了包
3、准备测试类
package com.hadoop.learn.test;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.log4j.Logger;
public class WordCountTest {
private static final Logger log = Logger.getLogger(WordCountTest.class);
public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
log.info("Map key : " + key);
log.info("Map value : " + value);
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
String wordStr = itr.nextToken();
word.set(wordStr);
log.info("Map word : " + wordStr);
context.write(word, one);
}
}
}
public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
log.info("Reduce key : " + key);
log.info("Reduce value : " + values);
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
log.info("Reduce sum : " + sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration config = new Configuration();
String[] otherArgs = new GenericOptionsParser(config, args).getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: WordCountTest <in> <out>");
System.exit(2);
}
Job job = new Job(config, "word count test");
job.setJarByClass(WordCountTest.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
4、配置host
192.168.197.131 hadoop-namenode
否则入参的地址已ip会出错:
java.lang.IllegalArgumentException: Wrong FS: hdfs://192.186.54.1:8020/user/hadoop/test.txt, expected: hdfs://hadoop1
正确如下:
5、hadoop-core-0.20.205.0.jar重新编译
运行是,可能报错:
12/04/24 15:32:44 ERROR security.UserGroupInformation: PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator-519341271\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator-519341271\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
这是由于Windows下文件权限问题,在Linux下可以正常运行,不存在这样的问题。
解决方法是修改 F:\编程开发\hadoop\older\hadoop-0.20.203.0rc1\hadoop-0.20.203.0\src\core\org \apache\hadoop\fs\RawLocalFileSystem.java ,注释掉 checkReturnValue(有些粗暴,在Window下,可以不用检查)
然后重新编译,编译可能出错
(1)ant开始下载依赖和编译文件。 我在编译的时候编译错误。经查,是$hadoop_home/src/saveVersion.sh生成的package-info.java有问题,导致无法编译过去。将saveVersion.sh修改一下:
(2):/hadoop/mapred/gridmix/Gridmix.java:396: 错误: 类型参数? extends T不在类型变量E的范围内
这个问题则需要修改/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java这个文件。
原: private <T> String getEnumValues(Enum<? extends T>[] e) {
+改: private String getEnumValues(Enum<?>[] e) {
StringBuilder sb = new StringBuilder();
String sep = "";
-原: for (Enum<? extends T> v : e) {
+改: for (Enum<?> v : e) {
sb.append(sep); sb.append(v.name()); sep = "|";
以上准备完成后,将执行成功
- 大小: 11.1 KB
- 大小: 11 KB
- 大小: 89.1 KB
- 大小: 88.8 KB
- 大小: 9.7 KB
- 大小: 27.1 KB
- 大小: 82.2 KB
分享到:
相关推荐
Eclipse中远程调试Hadoop必备资料:hadoop-eclipse-plugin-1.1.1和hadoop-core-1.0.2-modified;已经在eclipse-jee-juno-SR1-win32-x86_64和hadoop1.1.1 下测试过。
eclipse远程调试出现Exception in thread “main” java.lang.UnsatisfiedLinkError,eclipse4.5远程调试hadoop2.7.4依赖包 ,方法 将匹配的hadoop.dll、winutils.exe、libwinutils.lib拷贝到C:\Windows\System32 详情...
windows Eclipse远程调用Hadoop
windows下 eclipse操作hadoop集群 插件
用Windows下的Eclipse远程连接Hadoop的插件,适用于64位操作系统,hadoop2.6.0版本
Hadoop伪分布式部署文档(包括服务器伪分布式部署,本地hadoop开发环境部署,eclipse远程连接Hadoop服务器,实测无数遍,综合网上无数文档)
eclipse远程调试hadoop,报Could not locate executable null\bin\winutils.exe ,将winutils.exe放入到hadoop的bin目录下,报UnsatisfiedLinkError错,将 hadoop.dll放入C:\Windows\System32下
hadoop-eclipse-plugin-2.8.2.jar hadoop eclipse环境开发调试所用
eclipse远程调试hadoop时 报出eclipse Hadoop Failed to set permissions of path错误 修改hadoop core包中FileUtil java文件 里面有checkReturnValue方法 将代码throw new IOException "Failed to set ...
1、使用hadoop-eclipse-plugin-2.9.2.jar该eclipse插件,可以访问远程的hdfs 2、使用hadoop-eclipse-plugin-2.9.2-local.jar,访问本地的文件系统 3、wintuils_hadoop.zip本地eclipse运行需要的包。 另外2.9.2hadoop...
eclipse连接远程hadoop集群开发时权限不足问题解决方案 (2).pdfeclipse连接远程hadoop集群开发时权限不足问题解决方案 (2).pdf
hadoop-eclipse-plugin-2.7.2.jar,hadoop远程调试eclipse插件。
主要介绍了windows 32位eclipse远程hadoop开发环境搭建的相关资料,需要的朋友可以参考下
idea hadoop-hdfs插件,和eclipse上一样的Hadoop hdfs的插件功能一样; 原生集群端口分别为50020和9000 cdh 集群 端口分别为8020和9000 ,不用点测试直接点应用即可
在win7上eclipse远程链接hadoop需要的插件
主要介绍了eclipse/intellij idea 远程调试hadoop 2.6.0的相关资料,需要的朋友可以参考下
eclipse远程连接hadoop-2.7.0
针对windows下eclipse远程调试hadoop的权限问题而重新编译的hadoop-core-1.1.1.jar文件,已经测试通过。 原错误提示为:12/04/24 15:32:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for ...
Windows下Eclispe远程开发Mapreduce程序
window 下eclipse远程访问伪分布式hadoop 运行wordcount