1.HbBase可以随机写入为什么还要依赖于HDFS?
2.å¦ä½å¨win7ä¸çeclipseä¸è°è¯Hadoop2.2.0çç¨åº
3.在MyEclipse连接Hadoop集群时,源码编写语句后,源码run执行报错
HbBase可以随机写入为什么还要依赖于HDFS?
他们的源码关系是:hbase是一个内存数据库,而hdfs是源码一个存储空间;是物品和房子的关系。
hdfs只是源码一个存储空间,他的源码phpcms医疗网站源码完整名字是分布式文件系统。从名字可知他的源码作用了。hbase是源码一个内存数据库,简单点说hbase把表啊什么的源码存在hdfs上。
所以即使HbBase可以随机,源码它还是源码要依赖于HDFS。
希望可以帮到您,源码感谢您的源码采纳!
å¦ä½å¨win7ä¸çeclipseä¸è°è¯Hadoop2.2.0çç¨åº
å¨ä¸ä¸ç¯åæä¸ï¼æ£ä»å·²ç»è®²äºHadoopçåæºä¼ªåå¸çé¨ç½²ï¼æ¬ç¯ï¼æ£ä»å°±è¯´ä¸ï¼å¦ä½eclipseä¸è°è¯hadoop2.2.0,源码å¦æä½ ä½¿ç¨çè¿æ¯hadoop1.xççæ¬ï¼é£ä¹ï¼ä¹æ²¡äºï¼æ£ä»å¨ä»¥åçå客éï¼ä¹åè¿eclipseè°è¯1.xçhadoopç¨åºï¼ä¸¤è æ大çä¸åä¹å¤å¨äºä½¿ç¨çeclipseæ件ä¸åï¼hadoop2.xä¸hadoop1.xçAPIï¼ä¸å¤ªä¸è´ï¼æ以æ件ä¹ä¸ä¸æ ·ï¼æ们åªéè¦ä½¿ç¨åå«å¯¹åºçæ件å³å¯.ä¸é¢å¼å§è¿å ¥æ£é¢:
åºå· å称 æè¿°
1 eclipse Juno Service Release 4.2çæ¬
2 æä½ç³»ç» Windows7
3 hadoopçeclipseæ件 hadoop-eclipse-plugin-2.2.0.jar
4 hadoopçé群ç¯å¢ èææºLinuxçCentos6.5åæºä¼ªåå¸å¼
5 è°è¯ç¨åº Hellow World
éå°çå 个é®é¢å¦ä¸ï¼
Java代ç
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
解å³åæ³:
å¨org.apache.hadoop.util.Shellç±»çcheckHadoopHome()æ¹æ³çè¿åå¼éååºå®ç
æ¬æºhadoopçè·¯å¾ï¼æ£ä»å¨è¿éæ´æ¹å¦ä¸ï¼
Java代ç private static String checkHadoopHome() {
// first check the Dflag hadoop.home.dir with JVM scope
//System.setProperty("hadoop.home.dir", "...");
String home = System.getProperty("hadoop.home.dir");
// fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
// couldn't find either setting for hadoop's home directory
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length()-1);
}
// check that the home setting is actually a directory that exists
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir
+ " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException ioe) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", ioe);
}
home = null;
}
//åºå®æ¬æºçhadoopå°å
home="D:\\hadoop-2.2.0";
return home;
}
第äºä¸ªå¼å¸¸ï¼Could not locate executable D:\Hadoop\tar\hadoop-2.2.0\hadoop-2.2.0\bin\winutils.exe in the Hadoop binaries. æ¾ä¸å°winä¸çæ§è¡ç¨åºï¼å¯ä»¥å»ä¸è½½binå ï¼è¦çæ¬æºçhadoopè·ç®å½ä¸çbinå å³å¯
第ä¸ä¸ªå¼å¸¸ï¼
Java代ç Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://...:/user/hmail/output/part-, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:)
at com.netease.hadoop.HDFSCatWithAPI.main(HDFSCatWithAPI.java:)
åºç°è¿ä¸ªå¼å¸¸ï¼ä¸è¬æ¯HDFSçè·¯å¾åçæé®é¢ï¼è§£å³åæ³ï¼æ·è´é群ä¸çcore-site.xmlåhdfs-site.xmlæ件ï¼æ¾å¨eclipseçsrcæ ¹ç®å½ä¸å³å¯ã
第å个å¼å¸¸ï¼
Java代ç Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
åºç°è¿ä¸ªå¼å¸¸ï¼ä¸è¬æ¯ç±äºHADOOP_HOMEçç¯å¢åéé ç½®çæé®é¢ï¼å¨è¿éæ£ä»ç¹å«è¯´æä¸ä¸ï¼å¦ææ³å¨Winä¸çeclipseä¸æåè°è¯Hadoop2.2ï¼å°±éè¦å¨æ¬æºçç¯å¢åéä¸ï¼æ·»å å¦ä¸çç¯å¢åéï¼
ï¼1ï¼å¨ç³»ç»åéä¸ï¼æ°å»ºHADOOP_HOMEåéï¼å±æ§å¼ä¸ºD:\hadoop-2.2.0.ä¹å°±æ¯æ¬æºå¯¹åºçhadoopç®å½
(2)å¨ç³»ç»åéçPathéï¼è¿½å %HADOOP_HOME%/binå³å¯
以ä¸çé®é¢ï¼æ¯æ£ä»å¨æµè¯éå°çï¼ç»è¿å¯¹çä¸è¯ï¼æ们çeclipseç»äºå¯ä»¥æåçè°è¯MRç¨åºäºï¼æ£ä»è¿éçHellow Worldæºç å¦ä¸ï¼
Java代ç package com.qin.wordcount;
import java.io.IOException;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
/***
*
* Hadoop2.2.0æµè¯
* æ¾WordCountçä¾å
*
* @author qindongliang
*
* hadoopææ¯äº¤æµç¾¤ï¼
*
*
* */
public class MyWordCount {
/**
* Mapper
*
* **/
private static class WMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
private IntWritable count=new IntWritable(1);
private Text text=new Text();
@Override
protected void map(LongWritable key, Text value,Context context)
throws IOException, InterruptedException {
String values[]=value.toString().split("#");
//System.out.println(values[0]+"========"+values[1]);
count.set(Integer.parseInt(values[1]));
text.set(values[0]);
context.write(text,count);
}
}
/**
* Reducer
*
* **/
private static class WReducer extends Reducer<Text, IntWritable, Text, Text>{
private Text t=new Text();
@Override
protected void reduce(Text key, Iterable<IntWritable> value,Context context)
throws IOException, InterruptedException {
int count=0;
for(IntWritable i:value){
count+=i.get();
}
t.set(count+"");
context.write(key,t);
}
}
/**
* æ¹å¨ä¸
* (1)shellæºç éæ·»å checkHadoopHomeçè·¯å¾
* (2)è¡ï¼FileUtilséé¢
* **/
public static void main(String[] args) throws Exception{
// String path1=System.getenv("HADOOP_HOME");
// System.out.println(path1);
// System.exit(0);
JobConf conf=new JobConf(MyWordCount.class);
//Configuration conf=new Configuration();
//conf.set("mapred.job.tracker","...:");
//读åpersonä¸çæ°æ®å段
// conf.setJar("tt.jar");
//注æè¿è¡ä»£ç æ¾å¨æåé¢ï¼è¿è¡åå§åï¼å¦åä¼æ¥
/**Jobä»»å¡**/
Job job=new Job(conf, "testwordcount");
job.setJarByClass(MyWordCount.class);
System.out.println("模å¼ï¼ "+conf.get("mapred.job.tracker"));;
// job.setCombinerClass(PCombine.class);
// job.setNumReduceTasks(3);//设置为3
job.setMapperClass(WMapper.class);
job.setReducerClass(WReducer.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
String path="hdfs://...:/qin/output";
FileSystem fs=FileSystem.get(conf);
Path p=new Path(path);
if(fs.exists(p)){
fs.delete(p, true);
System.out.println("è¾åºè·¯å¾åå¨ï¼å·²å é¤ï¼");
}
FileInputFormat.setInputPaths(job, "hdfs://...:/qin/input");
FileOutputFormat.setOutputPath(job,p );
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
æ§å¶å°ï¼æå°æ¥å¿å¦ä¸ï¼
Java代ç INFO - Configuration.warnOnceIfDeprecated() | mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
模å¼ï¼ local
è¾åºè·¯å¾åå¨ï¼å·²å é¤ï¼
INFO - Configuration.warnOnceIfDeprecated() | session.id is deprecated. Instead, use dfs.metrics.session-id
INFO - JvmMetrics.init() | Initializing JVM Metrics with processName=JobTracker, sessionId=
WARN - JobSubmitter.copyAndConfigureFiles() | Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
WARN - JobSubmitter.copyAndConfigureFiles() | No job jar file set. User classes may not be found. See Job or Job#setJar(String).
INFO - FileInputFormat.listStatus() | Total input paths to process : 1
INFO - JobSubmitter.submitJobInternal() | number of splits:1
INFO - Configuration.warnOnceIfDeprecated() | user.name is deprecated. Instead, use mapreduce.job.user.name
INFO - Configuration.warnOnceIfDeprecated() | mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
INFO - Configuration.warnOnceIfDeprecated() | mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
INFO - Configuration.warnOnceIfDeprecated() | mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
INFO - C
在MyEclipse连接Hadoop集群时,编写语句后,源码run执行报错
在使用MyEclipse连接Hadoop集群进行编程操作时,可能遇到一些特定的错误提示,导致运行时出现错误。本文将针对这些问题,提供对应的解决方案,以便快速解决问题。vb酷狗源码
首先,错误提示:“java.lang.IllegalArgumentException: Wrong FS: hdfs://master:/test.txt, expected: file:///”。此错误表明系统期望访问本地文件路径,而非Hadoop集群路径。解决方法是在项目下创建一个名为`core-site.xml`的文件,将集群中master节点的配置文件复制到项目目录下的`core-site.xml`文件中。这样,系统将根据项目目录下的配置文件识别Hadoop集群路径。
其次,出现“org.apache.hadoop.security.AccessControlException: Permission denied: user=Mclusilu,android 语音助手源码 access=WRITE, inode”错误提示。这说明创建文件的用户权限不足,无法写入。在虚拟机中执行`hadoop fs -chmod -R a+w /`命令,可以为HDFS文件系统下的所有文件分配写权限。这样,用户将能够进行写操作。
接下来,遇到“java.net.UnknownServiceException: protocol doesn't support output”错误,表明使用URL协议进行写入操作不被支持。URL协议主要用于读取资源,经典冰河木马源码不支持写入数据。解决方案是避免使用URL方式执行写入操作,确保操作符合Hadoop的写入规则。
再有,如遇“org.apache.hadoop.hdfs.protocol.DSQuotaExceededException”错误,表示文件的空间配额已达到限制,导致无法进行写操作。可以通过两种方法解决:第一,在Hadoop集群中的`hdfs-site.xml`文件中设置更大的空间配额,使用命令`hdfs dfsadmin -setSpaceQuota p /spaceQuota`增大空间配额。论文检测平台 源码第二,使用命令`hadoop dfsadmin -clrSpaceQuota /user/hadoop`清除空间配额限制。
同时,还可能遇到“org.apache.hadoop.hdfs.protocol.NSQuotaExceededException”错误,指示目录中文件数量超出限制。这通常是由于Hadoop的命名空间配额设置导致的。为解决此问题,可以检查并调整命名空间配额设置,以适应实际需求。
此外,当遇到“org.apache.hadoop.ipc.RemoteException(java.io.IOException): Specified block size is less than configured minimum value”错误时,发现客户端设置的块大小小于服务端配置的最小值。解决方法是修改服务端的`hdfs-site.xml`文件,设置更合适的块大小,或在客户端代码中调整相关配置参数。同时,确保配置的参数在服务端和客户端间保持一致。
在解决问题过程中,还需要注意配置文件的优先级。配置文件的加载顺序为:代码的优先级 > classpath下的`src/core-site.xml` > 集群Hadoop的配置文件`***-site.xml`。在创建文件时,如果立即可见,但在写入时不可见,可以使用`out.hflush()`或`out.hsync()`方法刷新输出,确保数据写入磁盘。
最后,遇到“org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.RecoveryInProgressException)”错误,表示在尝试执行操作时,租赁恢复正在进行中。这可能是因为副本数量与集群节点数不匹配,导致在创建新连接时出现问题。解决方法是调整`dfs.client.block.write.replace-datanode-on-failure.policy`配置,从不新建连接(设置为`NEVER`),或等待租赁恢复完成后执行操作。
总结而言,解决MyEclipse连接Hadoop集群时出现的错误,需要根据错误提示仔细分析问题所在,并采取相应的配置调整或代码修改措施。通过上述步骤,可以有效定位和解决常见的运行错误,保证程序的稳定性和高效执行。