How to convert a dataframe to a dataset in Apache spark in Java?

I can easily convert a dataframe to a Scala dataset:

case class Person(name:String,age:Long)
val df = ctx.read.json("/tmp/persons.json")
val ds = df.as[Person]
ds.printSchema

But in the Java version, I don't know how to convert dataframe to dataset? Any ideas?

My efforts are:

DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = new Encoder<>();
Dataset<Person> ds = new Dataset<Person>(ctx,df.logicalPlan(),encoder);
ds.printSchema();

But the compiler says:

Error:(23,27) java: org.apache.spark.sql.Encoder is abstract; cannot be instantiated

Edit (solution):

@ leet Falcon based solution:

DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = Encoders.bean(Person.class);
Dataset<Person> ds = new Dataset<Person>(ctx,encoder);

Solution

The official spark document recommends the following in the dataset API:

List<String> data = Arrays.asList("abc","abc","xyz");
Dataset<String> ds = context.createDataset(data,Encoders.STRING());

The encoder can form tuples:

Encoder<Tuple2<Integer,String>> encoder2 = Encoders.tuple(Encoders.INT(),Encoders.STRING());
List<Tuple2<Integer,String>> data2 = Arrays.asList(new scala.Tuple2(1,"a");
Dataset<Tuple2<Integer,String>> ds2 = context.createDataset(data2,encoder2);

Or build from Java Bean to encoders #b bean:

Encoders.bean(MyClass.class);
The content of this article comes from the network collection of netizens. It is used as a learning reference. The copyright belongs to the original author.
THE END
分享
二维码
< <上一篇
下一篇>>