|
I'm having a peculiar problem with a permission denied error. I'm trying to run a map-reduce job over a set of input files. I have checked that I can view the files on every node with the user I am submitting the job as. Yet, the map tasks can't get started due to the following: Is the Child process getting run as a different user? I'm able to catch some of the Child java processes on the tasktracker nodes using 'ps' and they appear to be getting executed as my user. That said, my user id's and groups come from openldap, could that be a problem for map reduce? As I said, I can view the files just fine in either the shell or using 'hadoop fs -cat' |
|
Hi Matt, If I get this right all tasks are failing with permission denied? All mapreduce task run as user who submitted job. maprfs supports unix uid/gid and permission checks are based on uid/gid. The most common issue we see is that a username is present on all nodes but uid is different and hence task fail to access a file. user->uid/gid mapping on all nodes must be consistent. They are indeed consistent across all nodes. All of the uid/gid values are from openldap. Furthermore, I've verified that the user I am submitting the job as can ssh to the node, has the same output from the 'id' command and can use the 'hadoop fs' tool to view/manipulate the files and can also use the NFS mounted path to see the files.
(20 Oct '11, 17:03)
Matt
Your setup looks correct to me. Could you send us hadoop logs, hadoop mfs -lsr on the directory on which it failed (without 777). Also, if its possible for you to run it again could you run job with fs.mapr.trace=debug set in core-site.xml on jobclient and log4j.logger.org.apache.hadoop=DEBUG in /opt/mapr/hadoop/hadoop-0.20.2/conf/log4j.properties on one of the nodes where tasks fail.
(21 Oct '11, 11:18)
amit ♦
@amit I added a comment above since I've had a chance to work on this again. I have upgraded to version 1.2.3, are the instructions you gave me back in October still valid for collecting debugging info for this job? Is there somewhere I can email you the logs?
(01 May '12, 06:30)
Matt
Yes, it should be the same. I will send you a separate email for Amit's email address.
(01 May '12, 07:18)
MC Srivas ♦♦
Unfortunately I left this job without the opportunity to fully work out this problem, but I wanted to close this for posterity.
(24 Aug '12, 13:17)
Matt
|
I should add that everything works if I chmod 777 the directory tree, but I don't understand why I'm getting permission denied on files that I can definitely read in a non-mapreduce context.
I'm getting back to testing this now after living with the workaround for a long time, but now I need to get a real permissions system working soon. I've narrowed down some new behavior. It seems that in order to process a file with MapReduce I have to either be the owner, or the file has to have at least 755 permissions. I have tried 750 with no success despite being a member of the file's group ownership. I have also test 705 permissions with no success.