android - opencv4android template matching using camera

Question

I have downloaded and successfully run the example provided in opencv4android sdk.

I am able to simply display the camera frames without any processing,

 public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
     return inputFrame.rgba();
 }

I want to process live frame with some predefined image template to recognize that template. I have taken reference from this post and implemented accordingly. But I get black screen only.

private Mat mCameraMat = new Mat();
private Mat mTemplateMat;

 public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    mCameraMat = inputFrame.rgba();
    initialize();

     int match_method = Imgproc.TM_SQDIFF;

        // Create the result matrix
        int result_cols = mCameraMat.cols() - mTemplateMat.cols() + 1;
        int result_rows = mCameraMat.rows() - mTemplateMat.rows() + 1;
        Log.d(TAG, " mCameraMat cols "+mCameraMat.cols());
        Log.d(TAG, " mCameraMat rows "+mCameraMat.rows());
        Log.d(TAG, " mTemplateMat cols "+mTemplateMat.cols());
        Log.d(TAG, " mTemplateMat rows "+mTemplateMat.rows());

       Mat result = new Mat(result_rows, result_cols, CvType.CV_32F);

        // Do the Matching and Normalize
        Imgproc.matchTemplate(mCameraMat, mTemplateMat, result, match_method);



        Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());

        // Localizing the best match with minMaxLoc
        MinMaxLocResult mmr = Core.minMaxLoc(result);

        Point matchLoc;
        if (match_method == Imgproc.TM_SQDIFF || match_method == Imgproc.TM_SQDIFF_NORMED) {
            matchLoc = mmr.minLoc;
        } else {
            matchLoc = mmr.maxLoc;
        }

        Rect roi = new Rect((int) matchLoc.x, (int) matchLoc.y, mTemplateMat.cols(), mTemplateMat.rows());
        Core.rectangle(mCameraMat, new Point(roi.x, roi.y), new Point(roi.width - 2, roi.height - 2), new Scalar(255, 0, 0, 255), 2);           
 return result;
}

public void initialize(){

    try {
        if (mCameraMat.empty())
            return;
        if(mTemplateMat == null){
            Mat temp = Utils.loadResource(Tutorial1Activity.this, R.drawable.icon);
            mTemplateMat = new Mat(temp.size(), CvType.CV_32F);
            Imgproc.cvtColor(temp, mTemplateMat, Imgproc.COLOR_BGR2RGBA);
            Log.d(TAG, "initialize mTemplateMat cols "+mTemplateMat.cols());
            Log.d(TAG, "initialize mTemplateMat rows "+mTemplateMat.rows());
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}

Note:

My ultimate goal is to recognize the playing cards from live camera. Kindly suggest best approach. Should I use image templates or any other thing to make things faster?

This is how I want to recognize multiple cards from live camera:

Result should be: ?A ?K ?Q ?J ?10 when camera preview seems like below

Answer 1

Template matching is unlikely to be the best approach here.

1. Try aSIFT to do an affine invariant SIFT matching or a normal SIFT (OpenCV implementation exists). However, since these are in C++, you may want to use JNI to make calls to it from Java on an Android device. This is probably the best way to detect the suit of the card from the 4 symbols.
2. Another option to detect and recognize the numbers/alphabets on the cards is to use a text detector like MSER and then a text recognizer on the regions of interest indicated by the MSER filter.

In any case, you are unlikely to be able produce the best of results from the kind of image you've shown in the image. You may be able to get acceptable performance for full frontal, upright images with the first method.