running scala scripts in spark

demos

1. wordcount example

   1. wc.scala

       $ pico wc.scala
       val text = sc.textFile("hdfs://localhost:9000/user/hqlgree2/linkage/block_1.csv")
       val counts = text.flatMap(line => line.split(",")).map(word => (word, 1)).reduceByKey(_+_)
       counts.collect
       System.exit(0)
      

   2. firing up shell using the -i flag

       $ spark-shell -i wc.scala
      

2. spark how to run spark file from spark shell

   1. use :load

       scala> :load wc.scala
      

   2. use -i flag

       $ spark-shell -i wc.scala
      

3. how to package a scala project to a jar with sbt

        # 1. a jar of just the project classes
        $ sbt package
        # target/scala-2.x.y/projectname_2.x.y-z.jar
   
        # 2. continuously building the package
        $ sbt ~package
   
        # 3. standalone executable jar with dependencies
        $ sbt assembly
   
        $ java -jar projectname_2.x.y-z.jar [class.with.main.function]