Java – spark structured streaming automatically converts timestamps to local time

I have UTC and ISO 8601 timestamps, but using structured streams, it will be automatically converted to local time Is there any way to stop this conversion? I want to use it in UTC

I'm reading JSON data from Kafka and using from_ The JSON spark function parses them

Input:

{"Timestamp":"2015-01-01T00:00:06.222Z"}

Flow:

SparkSession
  .builder()
  .master("local[*]")
  .appName("my-app")
  .getOrCreate()
  .readStream()
  .format("kafka")
  ... //some magic
  .writeStream()
  .format("console")
  .start()
  .awaitTermination();

framework:

StructType schema = DataTypes.createStructType(new StructField[] {
        DataTypes.createStructField("Timestamp",DataTypes.TimestampType,true),});

Output:

+--------------------+
|           Timestamp|
+--------------------+
|2015-01-01 01:00:...|
|2015-01-01 01:00:...|
+--------------------+

As you can see, the number of hours increases automatically

PS: I try to use from_ utc_ Timestamp spark function, but no luck

Solution

It worked for me:

spark.conf.set("spark.sql.session.timeZone","UTC")

It tells spark SQL to use UTC as the default time zone for timestamp I use it in spark SQL, for example:

select *,cast('2017-01-01 10:10:10' as timestamp) from soMetable

I know it's in 2.0 1 does not work But applicable to spark 2.2 I also used it in sqltransformer

I'm not sure about streaming