i use spark-connector like this in spark-streaming appliction:
kafkaDirectStream.foreachRDD(rdd => {
try {
if (!rdd.isEmpty) {
val rowRdds = rdd.mapPartitions(partitions => {
partitions.map(x => {
......
})
})
val schema = StructType(Seq(StructField("Name", StringType, false),
StructField("Age", IntegerType, false)))
val df = sqlContext.createDataFrame(rdd, schema)
df
.write
.format("com.memsql.spark.connector")
.mode("error")
.save("people.students")
} catch {
case e: Exception => {
throw new RuntimeException(e)
}
}
})
and result is there is only a little of data has been writed into memsql , and the speed is very slow. then i killed the application and find some error logs in then yarn application logs as follow. how can i resolve this issue.
