tag:blogger.com,1999:blog-8716531089719420013.post1646164473067968703..comments2024-01-15T11:45:52.649+05:30Comments on Big Data and Cloud Tips: Integrating R and Hadoop using rmr2/rhdfs packages from Revolution AnalyticsPraveen Sripatihttp://www.blogger.com/profile/11782284194201977787noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-8716531089719420013.post-5298583901132462892017-12-28T01:48:50.268+05:302017-12-28T01:48:50.268+05:30hdfs.init()
sh: 1: classpath: not found
Error in s...hdfs.init()<br />sh: 1: classpath: not found<br />Error in system(command, intern = TRUE) : error in running commandAjinkzhttps://www.blogger.com/profile/11121943088190326696noreply@blogger.comtag:blogger.com,1999:blog-8716531089719420013.post-13088072524274238672017-08-19T16:15:49.847+05:302017-08-19T16:15:49.847+05:30I have integrated R with Hadoop using RHadoop and ... I have integrated R with Hadoop using RHadoop and running belwo program from R.<br /><br />library(rmr2)<br />library(rhdfs)<br />hdfs.init()<br />ints = to.dfs(1:100)<br />calc = mapreduce(input = ints, map = function(k, v) cbind(v, 2*v))<br />from.dfs(calc)<br /><br />Map reduce job runs successfully but it's giving null output as below.<br /><br />$key<br />NULL<br />$val<br />NULL<br /><br />However it I use the option rmr.options(backend="local") then it's giving proper output.<br /><br />It will be a great help if you can please help me on this and provide the solution.surenderhttps://www.blogger.com/profile/02822260891001731409noreply@blogger.comtag:blogger.com,1999:blog-8716531089719420013.post-87784271176874014512014-11-07T12:37:19.701+05:302014-11-07T12:37:19.701+05:30I got the following error at the end of the execut...I got the following error at the end of the execution. Can you please help us:<br /><br />14/03/25 10:52:37 INFO streaming.StreamJob: Running job: job_1395719087611_0009<br />14/03/25 10:52:37 INFO streaming.StreamJob: Job running in-process (local Hadoop)<br />14/03/25 10:52:38 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:54:04 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:54:06 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:54:47 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:54:49 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:55:35 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:55:36 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:56:18 INFO streaming.StreamJob: map 50% reduce 0%<br />14/03/25 10:56:20 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:56:25 INFO streaming.StreamJob: Job running in-process (local Hadoop)<br />14/03/25 10:56:25 ERROR streaming.StreamJob: Job not successful. Error: Task failed task_1395719087611_0009_m_000000<br />Job failed as tasks failed. failedMaps:1 failedReduces:0<br /><br />14/03/25 10:56:25 INFO streaming.StreamJob: killJob...<br />14/03/25 10:56:25 INFO impl.YarnClientImpl: Killing application application_1395719087611_0009<br />Streaming Command Failed!<br />Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, : <br />hadoop streaming failed with error code 1<br />14/03/25 10:57:09 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.<br />Deleted /tmp/file81d5d54392b<br /><br />Kindly give me any solutionExPeRiMENThttps://www.blogger.com/profile/08088291259533935060noreply@blogger.comtag:blogger.com,1999:blog-8716531089719420013.post-70082295741433880102014-03-25T11:30:10.002+05:302014-03-25T11:30:10.002+05:30Hi, Thanks for the nice article. I got the followi...Hi, Thanks for the nice article. I got the following error at the end of the execution. Can you please help us:<br /><br />14/03/25 10:52:37 INFO streaming.StreamJob: Running job: job_1395719087611_0009<br />14/03/25 10:52:37 INFO streaming.StreamJob: Job running in-process (local Hadoop)<br />14/03/25 10:52:38 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:54:04 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:54:06 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:54:47 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:54:49 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:55:35 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:55:36 INFO streaming.StreamJob: map 0% reduce 0%<br />14/03/25 10:56:18 INFO streaming.StreamJob: map 50% reduce 0%<br />14/03/25 10:56:20 INFO streaming.StreamJob: map 100% reduce 0%<br />14/03/25 10:56:25 INFO streaming.StreamJob: Job running in-process (local Hadoop)<br />14/03/25 10:56:25 ERROR streaming.StreamJob: Job not successful. Error: Task failed task_1395719087611_0009_m_000000<br />Job failed as tasks failed. failedMaps:1 failedReduces:0<br /><br />14/03/25 10:56:25 INFO streaming.StreamJob: killJob...<br />14/03/25 10:56:25 INFO impl.YarnClientImpl: Killing application application_1395719087611_0009<br />Streaming Command Failed!<br />Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, : <br /> hadoop streaming failed with error code 1<br />14/03/25 10:57:09 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.<br />Deleted /tmp/file81d5d54392b<br />Jyothihttps://www.blogger.com/profile/13657100964412118316noreply@blogger.comtag:blogger.com,1999:blog-8716531089719420013.post-81903636750172238122014-01-14T03:26:13.731+05:302014-01-14T03:26:13.731+05:30Hi there, nice tutorial!
When you paste the code ...Hi there, nice tutorial! <br />When you paste the code for install rhdfs, you did the jar rhbase_1.2.0.tar.gz I think you mean rhdfs*tar.gz. I used this jar and everything works great.<br /><br />Cheers!mattiehttps://www.blogger.com/profile/02827145522146105390noreply@blogger.com