Monday, August 25, 2014

Spark on HDP2

http://hortonworks.com/hdp/downloads/
http://spark.apache.org/docs/latest/
http://spark.apache.org/docs/latest/quick-start.html
http://spark.apache.org/examples.html
https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples
http://repo1.maven.org/maven2/org/apache/spark/
http://repo1.maven.org/maven2/org/scala-lang/
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_reference/content/reference_chap2.html
http://spark.apache.org/docs/latest/scala-programming-guide.html#transformations
http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-common/FileSystemShell.html


$ cd spark-0.9.1-bin-hadoop2
$ hdfs dfs -put README.md /tmp
$ bin/spark-shell
scala> val textFile = sc.textFile("hdfs://sandbox.hortonworks.com:8020/tmp/README.md")
scala> textFile.count
....
...
or
bin/run-example org.apache.spark.examples.HdfsTest local[2] hdfs://sandbox.hortonworks.com:8020/tmp/README.md

[Note] Check port 50070 for NameNode's URI

No comments:

Related Posts Plugin for WordPress, Blogger...