2
1

Hello guys: I have setup a 6 nodes hosts with MapR, I want to see is it really 3 times fater than hadoop. But when I run the 1TB tera sort. THere is only one reducer.So it tasks me a whole day to complete my job. I am using redhat 5 , and each node with an additional 7200 900GB disk. I got the rpms from http://package.mapr.com/redhat. Do you test tera sort, which part MapR is 3 times faster than hadoop?

Regards, xxqonline

asked 16 Jul '11, 19:58

xxqonline's gravatar image

xxqonline
55336
accept rate: 20%

edited 16 Jul '11, 20:46

TedDunning's gravatar image

TedDunning ♦♦
3.6k322


The variable that controls when reducers start with respect to mappers is controlled by mapred.reduce.slowstart.completed.maps, which is a number between 0 and 1. At 0, the reducers start when mappers start. At 0.75, reducers start when 75% of mappers have finished. In stock hadoop, it is set to 0.2, while MapR sets it to 0.95. You can find the setting in mapred-site.xml, and change the behavior via the command-line as follows:

hadoop jar ... -Dmapred.reduce.slowstart.completed.maps=0.95  ...

In particular, the command line switches need to come before the input and output dirs are specified, eg,

hadoop jar examples.jar terasort -Dmapred.reduce.tasks=30  /teragen_input_dir  /teragen_output_dir

BTW, the terasort benchmark as run by Yahoo! sets output replication to 1. In order to do that with MapR, you need to make /teragen_output_dir into its own volume, and set the volume's replication factor to 1.

As far as I know, the only new thing in MapR's hadoop-0.20.0-dev-examples.jar is the additional file system type called "maprfs" that's recognized. Would be happy to send you the source code, but to where?

link

answered 16 Jul '11, 22:10

MC%20Srivas's gravatar image

MC Srivas ♦♦
2.8k1519
accept rate: 32%

Thanks MC Srivas, your method work. Just put the -Dmapred.reduce.tasks=30 before the path. hadoop jar examples.jar terasort -Dmapred.reduce.tasks=30 /teragen_input_dir /teragen_output_dir I would like to see your exmaple source code. Great thankd

(16 Jul '11, 22:45) xxqonline

The number of reducers is specified on the command line. By default it is set to 1. Take a look at the job-trackers web page when the system is idle, and it will list the total number of reduce slots that are available across the cluster. Then run the terasort with the following option on the command-line:

bin/hadoop jar ... -Dmapred.reduce.tasks=<num reduce tasks>

replacing < num reduce tasks > with the total number of reduce slots in the cluster.

link

answered 16 Jul '11, 20:46

MC%20Srivas's gravatar image

MC Srivas ♦♦
2.8k1519
accept rate: 32%

I have tried your suggestion, but still 1 reducer

(16 Jul '11, 22:08) xxqonline

Yes. Terasort is part of our standard testing. You can see some results in Srivas' slides from the Hadoop Summit.

Do you have a list of the commands that you used to run tera gen and terasort?

Try using the -Dmapred.reduce.tasks= option to use more reducers. The best number to use depends on the size of the data you are sorting.

The basic issue here is that Hadoop's default is to use only one reducer and we preserve that behavior to stay compatible.

link

answered 16 Jul '11, 20:51

TedDunning's gravatar image

TedDunning ♦♦
3.6k322
accept rate: 23%

edited 16 Jul '11, 20:56

I am using -Dmapred.reduce.tasks=30 for testing now. I have another querstion, in standard hadoop, the reducer is started when mapper has some out put, but in mapr, the reducer is started after mapper all finish, what is the reason for this part?

link

answered 16 Jul '11, 20:55

xxqonline's gravatar image

xxqonline
55336
accept rate: 20%

Whether the reducers start before the mappers depends on the nature of the computation. When input is unsorted, a sort key is specified and no combiners are used, then it is not possible to start any reducers early because the first element of any reducer could come from the last mapper to run. Obviously, this applies with terasort.

Again, MapR tracks Hadoop behavior in this regard.

(16 Jul '11, 20:58) TedDunning ♦♦

Is there soure code for MapR hadoop examples(hadoop-0.20.2-dev-examples.jar)? I have seen it is 187526 Bytes, but the standandard hadoop-0.20.2-example.jar is 142466 bytes. And in standard hadoop running time, I do see the reducer started before mapper finished. I think you guys change the hadoop. Because I use the standard hadoop-0.20.2-example.jar run in mapR, it does not start reducer until mapper finish. The reason I use the standard hadoop example is for fair play, I need to ensure the test app is the same, else the result is useless for comparation. My command is 1. Generate with teragen 1TB data in /teragen_1TB 2. bin/hadoop jar hadoop-0.20.2-example.jar terasort /teragen_1TB /terasort_1TB_formal -Dmapred.reduce.tasks=30 But the reducer is still 1.

Do I have to use your hadoop-0.20.2-dev-examples.jar?

link

answered 16 Jul '11, 21:14

xxqonline's gravatar image

xxqonline
55336
accept rate: 20%

edited 16 Jul '11, 21:15

Do you really test it 3 times faster than Hadoop? My hadoop can finish in 2 hours. But we with -Dmapred.reduce.slowstart.completed.maps=0.95 enabled, the mapper become very slow after it reach 95%. And it is more than 2 hours. Now I remove the -Dmapred.reduce.slowstart.completed.maps=0.95.

The only enhance is the dfs io, it really improved a lot in writting. But for tera sort, what is your test result?

link

answered 17 Jul '11, 00:35

xxqonline's gravatar image

xxqonline
55336
accept rate: 20%

Did you look at the slides I pointed you at?

(17 Jul '11, 00:38) TedDunning ♦♦

I will have a reading, thanks

(17 Jul '11, 00:39) xxqonline

You mentioned that apache hadoop performs better by starting reducers earlier (mapred.reduce.slowstart.completed.maps=0.05). I'll suggest you to use this setting for your evaluation. For MapR you can use the same setting or you can also try setting value to 0.95 (very minimum overlap between mappers and reducers). In case of terasort on MapR, we have seen results are almost the same in both cases.

(17 Jul '11, 01:40) amit ♦

The work hang there. And the reducer copy thread is haning at for 1 hour task_201107151459_0022_r_000000 33.07% reduce > copy (3697 of 3726 at 8.59 MB/s) >

Also an exception: 1/07/17 03:41:04 INFO mapred.JobClient: map 95% reduce 0% 11/07/17 03:41:22 INFO mapred.JobClient: Task Id : attempt_201107151459_0022_m_002799_0, Status : FAILED on node db06b12 java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:261) Caused by: java.io.IOException: Task process exit with nonzero status of 137. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:248)

link

answered 17 Jul '11, 03:07

xxqonline's gravatar image

xxqonline
55336
accept rate: 20%

Could you share job configuration(especially memory given for map and reduce tasks) and node resources(memory, cpus, swap, disks)? Exit error 137 means task was forcefully killed. This can happen if node is running low on resources or limits set (http://linux.die.net/man/5/limits.conf) are very low.

(17 Jul '11, 03:24) amit ♦

I just use the default setting. The command is hadoop jar examples.jar terasort -Dmapred.reduce.tasks=30 /teragen_input_dir /teragen_output_dir

If I do not set the reduce task number, it can finish job, but only one reducer.

(17 Jul '11, 03:28) xxqonline

Node is powerful, 24 core cpu, Mem: 32949076k total

(17 Jul '11, 03:29) xxqonline

How many slots per node? BTW, Is job stuck or it is making progress? JobTracker should reschedule failed tasks somewhere else.

(17 Jul '11, 03:36) amit ♦

After one day running, it still hang there, there are 10 reducer slots per host. Could you send me your example source code. I have asked yesterday. Or some sample code?

(17 Jul '11, 19:45) xxqonline

terasort example is unchanged from the one in apache. You can use hadoop-examples.jar from apache hadoop distribution to run terasort on MapR.

(18 Jul '11, 16:32) amit ♦
showing 5 of 6 show all
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or __italic__
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×44
×3

Asked: 16 Jul '11, 19:58

Seen: 11,142 times

Last updated: 18 Jul '11, 16:32

powered by OSQA