当前位置:首页 > 开发 > 开源软件 > 正文

【Spark七十九】Spark RDD API一

发表于: 2015-02-25   作者:bit1129   来源:转载   浏览:
摘要: aggregate package spark.examples.rddapi import org.apache.spark.{SparkConf, SparkContext} //测试RDD的aggregate方法 object AggregateTest { def main(args: Array[String]) { val conf = new Spar

aggregate

package spark.examples.rddapi

import org.apache.spark.{SparkConf, SparkContext}

//测试RDD的aggregate方法
object AggregateTest {
  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local").setAppName("AggregateTest_00")
    val sc = new SparkContext(conf);
    val z1 = sc.parallelize(List(1, 3, 5, 7, 7, 5, 3, 3, 79), 2)
    /**
     * Aggregate the elements of each partition, and then the results for all the partitions, using
     * given combine functions and a neutral "zero value". This function can return a different result
     * type, U, than the type of this RDD, T. Thus, we need one operation for merging a T into an U
     * and one operation for merging two U's, as in scala.TraversableOnce. Both of these functions are
     * allowed to modify and return their first argument instead of creating a new U to avoid memory
     * allocation.
     */

    // def aggregate[U: ClassTag](zeroValue: U)(seqOp: (U, T) => U, combOp: (U, U) => U): U
    //T是RDD中的元素类型,U是aggregate方法自定义的泛型参数,aggregate返回U(而不一定是T)
    //两个分区取最大值,然后相加
    //math.max(_, _)表示针对每个partition实施的操作, _ + _表示combiner

    val r1 = z1.aggregate(0)(math.max(_, _), _ + _)
    println(r1) //86

    //RDD元素类型字符串,aggregate的返回类型同样为String
    val z2 = sc.parallelize(List("a", "b", "c", "d", "e", "f"), 2)
    val r2 = z2.aggregate("xx")(_ + _, _ + _)
    println(r2) //连接操作,结果xxxxabcxxdef,每个分区计算时,加上xx,最后两个分区计算时,继续把xx加上

    //_ + _的道理也是(x,y) => x + y
    //(x,y)=>math.max是做两两比较吗?
    val z3 = sc.parallelize(List("12", "23", "345", "4567"), 2)
    val r3 = z3.aggregate("")((x, y) => math.max(x.length, y.length).toString, (x, y) => x + y)
    println(r3)   ///结果24,表示两个分区的字符串长度最长的长度转成String后,做拼接

    //结果为什么是11?
    val r4 = sc.parallelize(List("12", "23", "345", "4567"), 2).aggregate("")((x, y) => math.min(x.length, y.length).toString, (x, y) => x + y)
    println(r4)
  }
}

cartesian

 

package spark.examples.rddapi

import org.apache.spark.rdd.{CartesianRDD, RDD}
import org.apache.spark.{SparkContext, SparkConf}


object CartesianTest_01 {
  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local").setAppName("AggregateTest_00")
    val sc = new SparkContext(conf);
    val z1 = sc.parallelize(List(2, 3, 4, 5, 6), 2)
    val z2 = sc.parallelize(List("A", "B", "C", "D", "E", "F", "G", "H", "I", "J"), 3)

    /**
     * Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of
     * elements (a, b) where a is in `this` and b is in `other`.
     */

    //def cartesian[U: ClassTag](other: RDD[U]): RDD[(T, U)] = new CartesianRDD(sc, this, other)
    //z1 和 z2集合的元素类型可以不同,并且cartesian是个转换算子,
    //调用z.collect触发作业
    val z = z1.cartesian(z2)
    println("Number of partitions: " + z.partitions.length) //6
    var count = 0

    z.collect().foreach(x  => {println(x._1 + "," + x._2); count = count + 1}) //

   println("count =" + count) //50

 

 

checkpoint

 

注意点:

Checkpointed RDDs are stored as a binary file within the checkpoint directory which can be specied using the Spark context. (Warning: Spark applies lazy evaluation. Checkpointing will not occur until an action is invoked.) Important note: the directory "my directory name" should exist in all slaves. As an alternative you could use an HDFS directory URL as well.

 

package spark.examples.rddapi

import org.apache.spark.{SparkContext, SparkConf}

object CheckpointTest {
  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local").setAppName("AggregateTest_00")
    val sc = new SparkContext(conf);
    val z = sc.parallelize(List(3, 6, 7, 9, 11))
    sc.setCheckpointDir("file:///d:/checkpoint")
    /**
     * Mark this RDD for checkpointing. It will be saved to a file inside the checkpoint
     * directory set with SparkContext.setCheckpointDir() and all references to its parent
     * RDDs will be removed. This function must be called before any job has been
     * executed on this RDD. It is strongly recommended that this RDD is persisted in
     * memory, otherwise saving it on a file will require recomputation.
     */
    z.checkpoint()
    println("length: " + z.collect().length) //rdd存入目录
    println("count: " + z.count()) //5
  }
}

 

 

d:\checkpoint>tree /f
文件夹 PATH 列表
卷序列号为 EA23-0890
D:.
└─9b0ca0d9-f7fb-46bb-84dc-097d95b9e7b8
    └─rdd-0
            .part-00000.crc
            part-00000

1. 运行过程中发现,checkpoint目录会自动创建,无需预创建

2.程序运行结束后,checkpoint目录并没有删除,上面这些属于checkpoint目录下的目录和文件也没有删除,再次运行会产生新的目录

 

Repartition

 

package spark.examples.rddapi

import org.apache.spark.{SparkContext, SparkConf}

object RepartitionTest_04 {
  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local").setAppName("RepartitionTest_04")
    val sc = new SparkContext(conf);
    val z1 = sc.parallelize(List(3, 9, 18, 22, 11, 9, 8), 3)
    //z1.coalesce(5, true)的效果一样,开启shuffle
    /**
     * Return a new RDD that has exactly numPartitions partitions.
     *
     * Can increase or decrease the level of parallelism in this RDD. Internally, this uses
     * a shuffle to redistribute data.
     *
     * If you are decreasing the number of partitions in this RDD, consider using `coalesce`,
     * which can avoid performing a shuffle.
     */
    val r1 = z1.repartition(5)
     r1.collect().foreach(println)
  }
}

 

coalesce

package spark.examples.rddapi

import org.apache.spark.{SparkContext, SparkConf}

//coalesce:合并
object CoalesceTest_03 {
  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local").setAppName("CoalesceTest_03")
    val sc = new SparkContext(conf);
    val z = sc.parallelize(List(3, 9, 18, 22, 11, 9, 8), 3)

    /**
     * Return a new RDD that is reduced into `numPartitions` partitions.
     *
     * This results in a narrow dependency, e.g. if you go from 1000 partitions
     * to 100 partitions, there will not be a shuffle, instead each of the 100
     * new partitions will claim 10 of the current partitions.
     *
     * However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
     * this may result in your computation taking place on fewer nodes than
     * you like (e.g. one node in the case of numPartitions = 1). To avoid this,
     * you can pass shuffle = true. This will add a shuffle step, but means the
     * current upstream partitions will be executed in parallel (per whatever
     * the current partitioning is).
     *
     * Note: With shuffle = true, you can actually coalesce to a larger number
     * of partitions. This is useful if you have a small number of partitions,
     * say 100, potentially with a few partitions being abnormally large. Calling
     * coalesce(1000, shuffle = true) will result in 1000 partitions with the
     * data distributed using a hash partitioner.
     */
    //shuffle默认为false
    //将分区数由3变成2,大变小使用narrow dependency
    val zz = z.coalesce(2, false)
    println("Partitions length: " + zz.partitions.length) //2
    println(zz.collect()) //结果是[I@100498c?
    zz.collect().foreach(println)

    //将分区数由3变成6,少变多必须使用shuffle=true
    //在单机上没有发现有问题
    //在cluster环境下,为了保证新的分区分布到不同的节点,应该使用shuffle为true
    //也就是说,少变多也可以使用shuffle为false,但是达不到分区数据进行重新分布的目的
    val z2 = z.coalesce(6, false)
    z2.collect().foreach(println)

    //分区扩大,同时设置shuffle为true
    val z3 = z.coalesce(6, true)
    z3.collect().foreach(println)




  }
}

 

 

 

 

【Spark七十九】Spark RDD API一

  • 0

    开心

    开心

  • 0

    板砖

    板砖

  • 0

    感动

    感动

  • 0

    有用

    有用

  • 0

    疑问

    疑问

  • 0

    难过

    难过

  • 0

    无聊

    无聊

  • 0

    震惊

    震惊

编辑推荐
本文对Sogou的日志进行分析,Sogou日志下载地址. http://download.labs.sogou.com/dl/sogoulabdown/
本文对Sogou的日志进行分析,Sogou日志下载地址. http://download.labs.sogou.com/dl/sogoulabdown/
什么是RDD Spark是围绕着RDD(Resilient Distributed Dataset,弹性分布式数据集)建立起来的,也就是
什么是RDD Spark是围绕着RDD(Resilient Distributed Dataset,弹性分布式数据集)建立起来的,也就是
1背景介绍 现今分布式计算框架像MapReduce和Dryad都提供了高层次的原语,使用户不用操心任务分发和
本文目的 最近在使用Spark进行数据清理的相关工作,初次使用Spark时,遇到了一些挑(da)战(ken)
http://www.cnblogs.com/cenyuhai/p/3779125.html 最近在阅读源码,发现这篇博客内容非常好,有助于
从一个简单的例子,来看rdd的演化,和stage是如何生成的(过程灰常之复杂和抽象,请参考附件的图来
与许多专有的大数据处理平台不同,Spark建立在统一抽象的RDD之上,使得它可以以基本一致的方式应对
与许多专有的大数据处理平台不同,Spark建立在统一抽象的RDD之上,使得它可以以基本一致的方式应对
版权所有 IT知识库 CopyRight © 2009-2015 IT知识库 IT610.com , All Rights Reserved. 京ICP备09083238号