SparkSQL綜合案例SparkSQL綜合案例1）

文本文件讀取，示例：Michael,3000Andy,4500Justin,3500Berta,4000Grace,8700SparkSQL綜合案例2）運用SparkSQL解決實際問題(1)查詢所有數(shù)據(jù)，并去除重復(fù)的數(shù)據(jù)(2)篩選salary>3500的記錄(3)將數(shù)據(jù)按name升序排列(4)查詢工資salary的平均值和最小值(5)利用SparkSQL分析員工信息SparkSQL綜合案例(1)查詢所有數(shù)據(jù)，并去除重復(fù)的數(shù)據(jù)scala>importorg.apache.spark.sql.SparkSessionscala>valspark=SparkSession.builder().getOrCreate()scala>importspark.implicits._scala>valdf=spark.read.json("file:///usr/local/spark/employees.json")scala>df.distinct().show()SparkSQL綜合案例(2)篩選salary>3500的記錄scala>df.filter(df("salary")>3500).show()

(3)將數(shù)據(jù)按工資升序排列scala>df.sort(df("salary").asc).show()(4)查詢工資的平均值和最小值scala>df.agg("salary"->"avg")scala>df.agg("salary"->"min")SparkSQL綜合案例(5)利用SparkSQL分析員工信息scala>importorg.apache.spark.sql.Rowscala>importorg.apache.spark.sql.types._scala>valschemaString=("namesalary")scala>valfields=schemaString.split("").map(fieldName=>StructField(fieldName,StringType,nullable=true))SparkSQL綜合案例scala>valschema=StructType(fields)scala>valrowRDD=empRDD.map(_.split(",")).map(attributes=>Row(attributes(0),attributes(1).trim))scala>valpeopleDF=spark.createDataFrame(rowRDD,schema)scala>peopleDF.createOrReplaceTempView("people")SparkSQL綜合案例scala>valresults=spark.sql("SELECTname,salaryFROMpeople")==》選擇合適的