Java – Avro architecture does not respect backward compatibility

I have this Avro architecture

{
 "namespace": "xx.xxxx.xxxxx.xxxxx","type": "record","name": "MyPayLoad","fields": [
     {"name": "filed1","type": "string"},{"name": "filed2","type": "long"},{"name": "filed3","type": "boolean"},{
          "name" : "metrics","type": 
          {
             "type" : "array","items": 
             { 
                 "name": "MyRecord","fields" : 
                     [                         
                       {"name": "min",{"name": "max",{"name": "sum",{"name": "count","type": "long"}
                     ]
             } 
          }
     }
  ]
}

This is the code we use to parse the data

public static final MyPayLoad parseBinaryPayload(byte[] payload) {
        DatumReader<MyPayLoad> payloadReader = new SpecificDatumReader<>(MyPayLoad.class);
        Decoder decoder = DecoderFactory.get().binaryDecoder(payload,null);
        MyPayLoad myPayLoad = null;
        try {
            myPayLoad = payloadReader.read(null,decoder);
        } catch (IOException e) {
            logger.log(Level.SEVERE,e.getMessage(),e);
        }

        return myPayLoad;
    }

Now I want to add a field to the schema, so the schema is as follows

{
 "namespace": "xx.xxxx.xxxxx.xxxxx","type": "long"}
                     ]
             } 
          }
     }
     {"name": "agentType","type": ["null","string"],"default": "APP_AGENT"}
  ]
}

Note that the added fields also define default values The problem is, if we receive data written using the old schema, I will receive this error

java.io.EOFException: null
    at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148) ~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139) ~[avro-1.7.4.jar:1.7.4]
    at com.appdynamics.blitz.shared.util.XXXXXXXXXXXXX.parseBinaryPayload(BlitzAvroSharedUtil.java:38) ~[blitz-shared.jar:na]

I learned from this file that this should be backward compatible, but somehow it doesn't seem so You know what I did wrong?

Solution

At last I got the job I need to give two patterns in the specificdatumreader

public static final MyPayLoad parseBinaryPayload(byte[] payload) {
        DatumReader<MyPayLoad> payloadReader = new SpecificDatumReader<>(SCHEMA_V1,SCHEMA_V2);
        Decoder decoder = DecoderFactory.get().binaryDecoder(payload,e);
        }

        return myPayLoad;
    }