Detailed analysis of several schemes of audio and video synthesis in Android

preface

Recently, the demand of audio-video processing has been met in the work. Under the Android video and video synthesis, there are three main methods in the current research plan: MediaMux hard decode, mp4parser, FFmepg. The three methods can be implemented, but they also have different limitations and problems. First record the implementation and problems here for later summary and learning. I won't say much now. Let's take a look at the detailed introduction.

Method 1 (fail)

Use mediamux to realize audio and video synthesis.

Effect: audio and video can be combined. Using Android's native videoview and surfaceview can play normally, and most players can play normally. However, there will be problems when uploading YouTube: the audio is discontinuous. The analysis is that it will be compressed again when uploading youtube, and there may be problems with the audio frame rate in the process of compression.

Analysis: during the processing of mediacodec.bufferinfo, there was a problem with the timestamp presentationtimeus, resulting in the compression of youtube and the disorder of audio.

public static void muxVideoAndAudio(String videoPath,String audioPath,String muxPath) {
 try {
  MediaExtractor videoExtractor = new MediaExtractor();
  videoExtractor.setDataSource(videoPath);
  MediaFormat videoFormat = null;
  int videoTrackIndex = -1;
  int videoTrackCount = videoExtractor.getTrackCount();
  for (int i = 0; i < videoTrackCount; i++) {
  videoFormat = videoExtractor.getTrackFormat(i);
  String mimeType = videoFormat.getString(MediaFormat.KEY_MIME);
  if (mimeType.startsWith("video/")) {
   videoTrackIndex = i;
   break;
  }
  }
  MediaExtractor audioExtractor = new MediaExtractor();
  audioExtractor.setDataSource(audioPath);
  MediaFormat audioFormat = null;
  int audioTrackIndex = -1;
  int audioTrackCount = audioExtractor.getTrackCount();
  for (int i = 0; i < audioTrackCount; i++) {
  audioFormat = audioExtractor.getTrackFormat(i);
  String mimeType = audioFormat.getString(MediaFormat.KEY_MIME);
  if (mimeType.startsWith("audio/")) {
   audioTrackIndex = i;
   break;
  }
  }
  videoExtractor.selectTrack(videoTrackIndex);
  audioExtractor.selectTrack(audioTrackIndex);
  MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
  MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
  MediaMuxer mediaMuxer = new MediaMuxer(muxPath,MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
  int writeVideoTrackIndex = mediaMuxer.addTrack(videoFormat);
  int writeAudioTrackIndex = mediaMuxer.addTrack(audioFormat);
  mediaMuxer.start();
  ByteBuffer byteBuffer = ByteBuffer.allocate(500 * 1024);
  long sampleTime = 0;
  {
  videoExtractor.readSampleData(byteBuffer,0);
  if (videoExtractor.getSampleFlags() == MediaExtractor.SAMPLE_FLAG_SYNC) {
   videoExtractor.advance();
  }
  videoExtractor.readSampleData(byteBuffer,0);
  long secondTime = videoExtractor.getSampleTime();
  videoExtractor.advance();
  long thirdTime = videoExtractor.getSampleTime();
  sampleTime = Math.abs(thirdTime - secondTime);
  }
  videoExtractor.unselectTrack(videoTrackIndex);
  videoExtractor.selectTrack(videoTrackIndex);
  while (true) {
  int readVideoSampleSize = videoExtractor.readSampleData(byteBuffer,0);
  if (readVideoSampleSize < 0) {
   break;
  }
  videoBufferInfo.size = readVideoSampleSize;
  videoBufferInfo.presentationTimeUs += sampleTime;
  videoBufferInfo.offset = 0;
  //noinspection WrongConstant
  videoBufferInfo.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;//videoExtractor.getSampleFlags()
  mediaMuxer.writeSampleData(writeVideoTrackIndex,byteBuffer,videoBufferInfo);
  videoExtractor.advance();
  }
  while (true) {
  int readAudioSampleSize = audioExtractor.readSampleData(byteBuffer,0);
  if (readAudioSampleSize < 0) {
   break;
  }
  audioBufferInfo.size = readAudioSampleSize;
  audioBufferInfo.presentationTimeUs += sampleTime;
  audioBufferInfo.offset = 0;
  //noinspection WrongConstant
  audioBufferInfo.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;// videoExtractor.getSampleFlags()
  mediaMuxer.writeSampleData(writeAudioTrackIndex,audioBufferInfo);
  audioExtractor.advance();
  }
  mediaMuxer.stop();
  mediaMuxer.release();
  videoExtractor.release();
  audioExtractor.release();
 } catch (IOException e) {
  e.printStackTrace();
 }
 }