Java – convert the 3D world (arcore anchor / pose) to the corresponding 2D screen coordinates

I am trying to achieve this transformation Given the anchor pose in arcore, how can the corresponding 2D coordinates be obtained in the screen?

Solution

Finally, after several days of investigation and obtaining information from different resources, I can make it work normally The following is a code snippet (based on the arcore sample Java application) that can be converted from world coordinates (posture in arcore) to 2D screen coordinates:

First, we need to calculate the matrix converted from the world – > Screen:

public float[] calculateWorld2CameraMatrix(float[] modelmtx,float[] viewmtx,float[] prjmtx) {

    float scaleFactor = 1.0f;
    float[] scaleMatrix = new float[16];
    float[] modelXscale = new float[16];
    float[] viewXmodelXscale = new float[16];
    float[] world2screenMatrix = new float[16];

    Matrix.setIdentityM(scaleMatrix,0);
    scaleMatrix[0] = scaleFactor;
    scaleMatrix[5] = scaleFactor;
    scaleMatrix[10] = scaleFactor;

    Matrix.multiplyMM(modelXscale,modelmtx,scaleMatrix,0);
    Matrix.multiplyMM(viewXmodelXscale,viewmtx,modelXscale,0);
    Matrix.multiplyMM(world2screenMatrix,prjmtx,viewXmodelXscale,0);

    return world2screenMatrix;

}

Once we have this matrix, we can project points from the 3D world to 2D, but in this projection, we must convert from NDC coordinates to the screen Here is how to perform this transformation:

double[] world2Screen(int screenWidth,int screenHeight,float[] world2cameraMatrix)
  {
    float[] origin = {0f,0f,1f};
    float[] ndcCoord = new float[4];
    Matrix.multiplyMV(ndcCoord,world2cameraMatrix,origin,0);

    ndcCoord[0] = ndcCoord[0]/ndcCoord[3];
    ndcCoord[1] = ndcCoord[1]/ndcCoord[3];

    double[] pos_2d = new double[]{0,0};
    pos_2d[0] = screenWidth  * ((ndcCoord[0] + 1.0)/2.0);
    pos_2d[1] = screenHeight * (( 1.0 - ndcCoord[1])/2.0);

    return pos_2d;
  }

Finally, a simple usage example:

DisplayMetrics displayMetrics = new DisplayMetrics();
        getWindowManager().getDefaultDisplay().getMetrics(displayMetrics);
        int height = displayMetrics.heightPixels;
        int width = displayMetrics.widthPixels;

        // Get projection matrix.
        float[] projmtx = new float[16];
        camera.getProjectionMatrix(projmtx,0.1f,100.0f);

        // Get camera matrix and draw.
        float[] viewmtx = new float[16];
        camera.getViewMatrix(viewmtx,0);

        float[] anchorMatrix = new float[16];
        anchor.getPose().toMatrix(anchorMatrix,0);
        float[] world2screenMatrix =    
  virtualObject.calculateWorld2CameraMatrix(anchorMatrix,projmtx);
        double[] anchor_2d =  world2Screen(width,height,world2screenMatrix);