|
Hello ,please help me see this error: /opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop jar ./wordcount.jar com.felix.WordCount /mt /mt/output 12/02/11 08:50:44 INFO fs.JobTrackerWatcher: Current running JobTracker is: node1/192.168.172.139:9001 12/02/11 08:50:44 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 12/02/11 08:50:44 WARN snappy.LoadSnappy: Snappy native library is not available 12/02/11 08:50:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library 12/02/11 08:50:44 WARN snappy.LoadSnappy: Snappy native library not loaded 12/02/11 08:50:44 INFO mapred.FileInputFormat: Total input paths to process : 1 12/02/11 08:50:46 INFO mapred.JobClient: Running job: job_201202110652_0003 12/02/11 08:50:47 INFO mapred.JobClient: map 0% reduce 0% 12/02/11 08:50:56 INFO mapred.JobClient: Task Id : attempt_201202110652_0003_m_000000_0, Status : FAILED on node node1 Error: Java heap space 12/02/11 08:50:57 INFO mapred.JobClient: Task Id : attempt_201202110652_0003_m_000001_0, Status : FAILED on node node1 Error: Java heap space 12/02/11 08:51:01 INFO mapred.JobClient: Task Id : attempt_201202110652_0003_m_000000_1, Status : FAILED on node node1 Error: Java heap space 12/02/11 08:51:02 INFO mapred.JobClient: Task Id : attempt_201202110652_0003_m_000001_1, Status : FAILED on node node1 Error: Java heap space 12/02/11 08:51:05 INFO mapred.JobClient: Task Id : attempt_201202110652_0003_m_000000_2, Status : FAILED on node node1 Error: Java heap space 12/02/11 08:51:08 INFO mapred.JobClient: Task Id : attempt_201202110652_0003_m_000001_2, Status : FAILED on node node1 Error: Java heap space 12/02/11 08:51:12 INFO mapred.JobClient: Job complete: job_201202110652_0003 12/02/11 08:51:12 INFO mapred.JobClient: Counters: 7 12/02/11 08:51:12 INFO mapred.JobClient: Job Counters 12/02/11 08:51:12 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=10509 12/02/11 08:51:12 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 12/02/11 08:51:12 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 12/02/11 08:51:12 INFO mapred.JobClient: Launched map tasks=8 12/02/11 08:51:12 INFO mapred.JobClient: Data-local map tasks=8 12/02/11 08:51:12 INFO mapred.JobClient: Failed map tasks=1 12/02/11 08:51:12 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 12/02/11 08:51:12 INFO mapred.JobClient: Job Failed: NA Exception in thread "main" java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1280) at com.felix.WordCount.main(WordCount.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) |
|
Looks like you don't have enough memory: "Java heap space" to run tasks. I am having the same problem. How do I allocate more memory for the tasks?
(06 Mar, 17:22)
tc_dev
/opt/mapr/conf/warden.conf has memory settings for various services. Memory for tasks is allocated based on the memory not used by other services on TaskTracker running nodes. You may try to tweak those settings to reduce memory allocated for other services to "free up" more memory for tasks.
(06 Mar, 18:45)
yufeldman ♦
Yes, I found warden.conf, but changing any settings there did not help. In fact increasing any of them (and restarting warden service) did not even cause more RAM usage. What specific setting should I have modified? I did try to re-install everything and now getting a different error trying to run this MapReduce job - 12/03/07 02:12:05 INFO mapred.JobClient: Task Id : attempt_201203061900_0002_m_000002_0, Status : FAILED on node HOSTNAME java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:259) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:246) Stderr file for hadoop userlog/attempt has this: Invalid maximum heap size: -Xmx0m Could not create the Java virtual machine. How could this be? There is nothing like "mx0m" in warden.conf
(07 Mar, 02:23)
tc_dev
You probably have a 0 value for one of the heap settings. Can you post your warden.conf ?
(07 Mar, 03:06)
Nabeel ♦♦
This is just a standard MapR post-install config that was not modified. It does not fit in full here (900 char limit in comments). I searched through the warden.conf for the word "heap" and all lines including it have non-zero value. Where else to look?
(07 Mar, 11:34)
tc_dev
Tasks use the memory that is close enough to
(07 Mar, 11:44)
yufeldman ♦
The problem is I never see the memory usage reach the total of the whole box! I am testing MapR on a single-node cluster with 4Gb box, yet when I run the MapReduce it fails when usage goes up to about 60% in the "Cluster Utilization" section on the dashboard. What could explain such behavior? P.S. To be clear to give more memory to MapReduce I need to increase these, right?
(07 Mar, 12:42)
tc_dev
decrease, not increase. Memory for tasks is different from memory for TaskTracker. Regarding overall cluster memory utilization - if you specify -Xmx100m it does not mean that that process will use 100m - it just means it won't use more then that. So to your point by setting %, max and min we just configure memory limits for majority of the services, we do not promise those services will utilize all of it, we just prevent the situation that when all of the services reach their max memory utilization node will not "die".
(07 Mar, 13:13)
yufeldman ♦
So if I have service.command.tt.heapsize.percent=2 then on my box Task Tracker would never exceed 4G * 2% = 80M? Does this % setting takes precedence over service.command.tt.heapsize.max=325? Are you also saying the memory for tasks is different from the memory for task tracker? I am just running hadoop from command line per tutorial - hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvolume/in /myvolume/out Should I specify task memory requirements in hadoop command line? How?
(07 Mar, 14:17)
tc_dev
showing 5 of 9
show all
|
|
If you just want to set heapsize for job's map task and reduce task please use mapred.map.child.java.opts and mapred.reduce.child.java.opts. For more details please see mapreduce configuration I added -Xmx500m to these options and the task stopped failing on start. However now instead of failing the task just hangs in "pending" state and even the first attempt never ends. Here is the tail of the attempt's syslog: 2012-03-07 19:03:23,373 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@2a0ab444 What could be the problem here?
(07 Mar, 19:13)
tc_dev
P.S. The task is listed as "Pending" under UI
(08 Mar, 00:17)
tc_dev
Anything in stderr and stdout ?
(08 Mar, 00:22)
Nabeel ♦♦
Both stderr and stdout are empty. The task just hangs as pending forever...
(08 Mar, 00:37)
tc_dev
|
|
Pls execute the support collect script under /opt/mapr/support/tools and send the tar file to support@mapr.com. |
