Just upgraded my cluster to v2.1.1 and started adding additional nodes on a private network. The JobTracker has an interface on two networks - the "old" public network, and the new private. The new nodes are only on the private network are configuring and MFS is running OK.

But after some time, the TaskTrackers are failing, and looking at the TT logs I can see they are trying to contact the JT on the "old" or public IP address. I have looked all over to see where this is set, but cannot find it in the config files?

A little help here please?

Thanks

asked 01 Mar '13, 10:53

thealy's gravatar image

thealy
375464855
accept rate: 20%

I should have mentioned that we are running M3.

(01 Mar '13, 10:54) thealy

You will not find it in config files. warden on the node where JT is running is creating an entry for it in Zookeeper and this is the place from where all interested in JT parties pick it up. As Zookeeper entry is hostname and not an ip - most likely it will resolve to your public ip and not private. At least see on your failing TT nodes what your JT hostname resolves to.

link

answered 01 Mar '13, 11:43

yufeldman's gravatar image

yufeldman ♦♦
2.6k39
accept rate: 25%

Thanks, but how do I correct it so that is advertising the private network address? If I turn off the public interface, I fear that I will be unable to submit jobs.

All I can think of is to hack the /etc/hosts file on the private network, but this seems very nasty.

(01 Mar '13, 11:52) thealy

You can overrule all this by defining JT ip, port from mapred-site.xml on your "failing" TT nodes. Just don't forget to reconfigure if you will change JT location later on.

(01 Mar '13, 11:55) yufeldman ♦♦

That sounds perfect, but I find no property specifying the jobtracker in mapred-site.xml

(01 Mar '13, 12:57) thealy

mapred.jobtracker.port (if need to change default port)

mapred.job.tracker (if need to specify particular ip)

Try to use "hadoop conf -dump" to see all config options available. It will save you time

(01 Mar '13, 13:12) yufeldman ♦♦

Thanks very much. After adding the private (non-routed) JT interface address to mapred-site.xml on the nodes, it is now working. I guess I was hoping for the lazy man's -J command line option for configure.sh....

(01 Mar '13, 16:13) thealy
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or __italic__
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×62
×50
×44
×2

Asked: 01 Mar '13, 10:53

Seen: 960 times

Last updated: 01 Mar '13, 16:13

powered by OSQA