image - How to compress Bitmap as JPEG with least quality loss on Android?

Question

This is not a straightforward problem, please read through!

I want to manipulate a JPEG file and save it again as JPEG. The problem is that even without manipulation there's significant (visible) quality loss. Question: what option or API am I missing to be able to re-compress JPEG without quality loss (I know it's not exactly possible, but I think what I describe below is not an acceptable level of artifacts, especially with quality=100).

Control

I load it as a Bitmap from the file:

BitmapFactory.Options options = new BitmapFactory.Options();
// explicitly state everything so the configuration is clear
options.inPreferredConfig = Config.ARGB_8888;
options.inDither = false; // shouldn't be used anyway since 8888 can store HQ pixels
options.inScaled = false;
options.inPremultiplied = false; // no alpha, but disable explicitly
options.inSampleSize = 1; // make sure pixels are 1:1
options.inPreferQualityOverSpeed = true; // doesn't make a difference
// I'm loading the highest possible quality without any scaling/sizing/manipulation
Bitmap bitmap = BitmapFactory.decodeFile("/sdcard/image.jpg", options);

Now, to have a control image to compare to, let's save the plain Bitmap bytes as PNG:

bitmap.compress(PNG, 100/*ignored*/, new FileOutputStream("/sdcard/image.png"));

I compared this to the original JPEG image on my computer and there's no visual difference.

I also saved the raw int[] from getPixels and loaded it as a raw ARGB file on my computer: there's no visual difference to the original JPEG, nor the PNG saved from Bitmap.

I checked the Bitmap's dimensions and config, they match the source image and the input options: it's decoded as ARGB_8888 as expected.

The above to control checks prove that the pixels in the in-memory Bitmap are correct.

Problem

I want to have JPEG files as a result, so the above PNG and RAW approaches wouldn't work, let's try to save as JPEG 100% first:

// 100% still expected lossy, but not this amount of artifacts
bitmap.compress(JPEG, 100, new FileOutputStream("/sdcard/image.jpg"));

I'm not sure its measure is percent, but it's easier to read and discuss, so I'm gonna use it.

I'm aware that JPEG with the quality of 100% is still lossy, but it shouldn't be so visually lossy that it's noticeable from afar. Here's a comparison of two 100% compressions of the same source.

Open them in separate tabs and click back and forth between to see what I mean. The difference images were made using Gimp: original as bottom layer, re-compressed middle layer with "Grain extract" mode, top layer full white with "Value" mode to enhance badness.

The below images are uploaded to Imgur which also compresses the files, but since all of the images are compressed the same, the original unwanted artifacts remain visible the same way I see it when opening my original files.

Original [560k]: Imgur's difference to original (not relevant to problem, just to show that it's not causing any extra artifacts when uploading the images): IrfanView 100% [728k] (visually identical to original): IrfanView 100%'s difference to original (barely anything) Android 100% [942k]: Android 100%'s difference to original (tinting, banding, smearing)

In IrfanView I have to go below 50% [50k] to see remotely similar effects. At 70% [100k] in IrfanView there's no noticable difference, but the size is 9th of Android's.

Background

I created an app that takes a picture from Camera API, that image comes as a byte[] and is an encoded JPEG blob. I saved this file via OutputStream.write(byte[]) method, that was my original source file. decodeByteArray(data, 0, data.length, options) decodes the same pixels as reading from a File, tested with Bitmap.sameAs so it's irrelevant to the issue.

I was using my Samsung Galaxy S4 with Android 4.4.2 to test things out. Edit: while investigating further I also tried Android 6.0 and N preview emulators and they reproduce the same issue.

Answer 1

After some investigation I found the culprit: Skia's YCbCr conversion. Repro, code for investigation and solutions can be found at TWiStErRob/AndroidJPEG.

Discovery

After not getting a positive response on this question (neither from http://b.android.com/206128) I started digging deeper. I found numerous half-informed SO answers which helped me tremendously in discovering bits and pieces. One such answer was https://stackoverflow.com/a/13055615/253468 which made me aware of YuvImage which converts an YUV NV21 byte array into a JPEG compressed byte array:

YuvImage yuv = new YuvImage(yuvData, ImageFormat.NV21, width, height, null);
yuv.compressToJpeg(new Rect(0, 0, width, height), 100, jpeg);

There's a lot of freedom going into creating the YUV data, with varying constants and precision. From my question it's clear that Android uses an incorrect algorithm. While playing around with the algorithms and constants I found online I always got a bad image: either the brightness changed or had the same banding issues as in the question.

Digging deeper

YuvImage is actually not used when calling Bitmap.compress, here's the stack for Bitmap.compress:

• libjpeg/jpeg_write_scanlines(jcapistd.c:77)
• skia/rgb2yuv_32(SkImageDecoder_libjpeg.cpp:913)
• skia/writer(=Write_32_YUV).write(SkImageDecoder_libjpeg.cpp:961)
  [WE_CONVERT_TO_YUV is unconditionally defined]
• SkJPEGImageEncoder::onEncode(SkImageDecoder_libjpeg.cpp:1046)
• SkImageEncoder::encodeStream(SkImageEncoder.cpp:15)
• Bitmap_compress(Bitmap.cpp:383)
• Bitmap.nativeCompress(Bitmap.java:1573)
• Bitmap.compress(Bitmap.java:984)
• app.saveBitmapAsJPEG()

and the stack for using YuvImage

• libjpeg/jpeg_write_raw_data(jcapistd.c:120)
• YuvToJpegEncoder::compress(YuvToJpegEncoder.cpp:71)
• YuvToJpegEncoder::encode(YuvToJpegEncoder.cpp:24)
• YuvImage_compressToJpeg(YuvToJpegEncoder.cpp:219)
• YuvImage.nativeCompressToJpeg(YuvImage.java:141)
• YuvImage.compressToJpeg(YuvImage.java:123)
• app.saveNV21AsJPEG()

By using the constants in rgb2yuv_32 from the Bitmap.compress flow I was able to recreate the same banding effect using YuvImage, not an achievement, just a confirmation that it's indeed the YUV conversion that is messed up. I double-checked that the problem is not during YuvImage calling libjpeg: by converting the Bitmap's ARGB to YUV and back to RGB then dumping the resulting pixel blob as a raw image, the banding was already there.

While doing this I realized that the NV21/YUV420SP layout is lossy as it samples the color information every 4th pixel, but it keeps the value (brightness) of each pixel which means that some color info is lost, but most of the info for people's eyes are in the brightness anyway. Take a look at the example on wikipedia, the Cb and Cr channel makes barely recognisable images, so lossy sampling on it doesn't matter much.

Solution

So, at this point I knew that libjpeg does the right conversion when it is passed the right raw data. This is when I set up the NDK and integrated the latest LibJPEG from http://www.ijg.org. I was able to confirm that indeed passing the RGB data from the Bitmap's pixels array yields the expected result. I like to avoid using native components when not absolutely necessary, so aside of going for a native library that encodes a Bitmap I found a neat workaround. I've essentially taken the rgb_ycc_convert function from jcolor.c and rewrote it in Java using the skeleton from https://stackoverflow.com/a/13055615/253468. The below is not optimized for speed, but readability, some constants were removed for brevity, you can find them in libjpeg code or my example project.

private static final int JSAMPLE_SIZE = 255 + 1;
private static final int CENTERJSAMPLE = 128;
private static final int SCALEBITS = 16;
private static final int CBCR_OFFSET = CENTERJSAMPLE << SCALEBITS;
private static final int ONE_HALF = 1 << (SCALEBITS - 1);

private static final int[] rgb_ycc_tab = new int[TABLE_SIZE];
static { // rgb_ycc_start
    for (int i = 0; i <= JSAMPLE_SIZE; i++) {
        rgb_ycc_tab[R_Y_OFFSET + i] = FIX(0.299) * i;
        rgb_ycc_tab[G_Y_OFFSET + i] = FIX(0.587) * i;
        rgb_ycc_tab[B_Y_OFFSET + i] = FIX(0.114) * i + ONE_HALF;
        rgb_ycc_tab[R_CB_OFFSET + i] = -FIX(0.168735892) * i;
        rgb_ycc_tab[G_CB_OFFSET + i] = -FIX(0.331264108) * i;
        rgb_ycc_tab[B_CB_OFFSET + i] = FIX(0.5) * i + CBCR_OFFSET + ONE_HALF - 1;
        rgb_ycc_tab[R_CR_OFFSET + i] = FIX(0.5) * i + CBCR_OFFSET + ONE_HALF - 1;
        rgb_ycc_tab[G_CR_OFFSET + i] = -FIX(0.418687589) * i;
        rgb_ycc_tab[B_CR_OFFSET + i] = -FIX(0.081312411) * i;
    }
}

static void rgb_ycc_convert(int[] argb, int width, int height, byte[] ycc) {
    int[] tab = LibJPEG.rgb_ycc_tab;
    final int frameSize = width * height;

    int yIndex = 0;
    int uvIndex = frameSize;
    int index = 0;
    for (int y = 0; y < height; y++) {
        for (int x = 0; x < width; x++) {
            int r = (argb[index] & 0x00ff0000) >> 16;
            int g = (argb[index] & 0x0000ff00) >> 8;
            int b = (argb[index] & 0x000000ff) >> 0;

            byte Y = (byte)((tab[r + R_Y_OFFSET] + tab[g + G_Y_OFFSET] + tab[b + B_Y_OFFSET]) >> SCALEBITS);
            byte Cb = (byte)((tab[r + R_CB_OFFSET] + tab[g + G_CB_OFFSET] + tab[b + B_CB_OFFSET]) >> SCALEBITS);
            byte Cr = (byte)((tab[r + R_CR_OFFSET] + tab[g + G_CR_OFFSET] + tab[b + B_CR_OFFSET]) >> SCALEBITS);

            ycc[yIndex++] = Y;
            if (y % 2 == 0 && index % 2 == 0) {
                ycc[uvIndex++] = Cr;
                ycc[uvIndex++] = Cb;
            }
            index++;
        }
    }
}

static byte[] compress(Bitmap bitmap) {
    int w = bitmap.getWidth();
    int h = bitmap.getHeight();
    int[] argb = new int[w * h];
    bitmap.getPixels(argb, 0, w, 0, 0, w, h);
    byte[] ycc = new byte[w * h * 3 / 2];
    rgb_ycc_convert(argb, w, h, ycc);
    argb = null; // let GC do its job
    ByteArrayOutputStream jpeg = new ByteArrayOutputStream();
    YuvImage yuvImage = new YuvImage(ycc, ImageFormat.NV21, w, h, null);
    yuvImage.compressToJpeg(new Rect(0, 0, w, h), quality, jpeg);
    return jpeg.toByteArray();
}

The magic key seems to be ONE_HALF - 1 the rest looks an awful lot like the math in Skia. That's a good direction for future investigation, but for me the above is sufficiently simple to be a good solution for working around Android's builtin weirdness, albeit slower. Note that this solution uses the NV21 layout which loses 3/4 of the color info (from Cr/Cb), but this loss is much less than the errors created by Skia's math. Also note that YuvImage doesn't support odd-sized images, for more info see NV21 format and odd image dimensions.