How to display a raw YUV frame in a Cocoa OpenGL program

Question

Hello everyone,

I have been assigned wit the task to write a program that takes a sample raw YUV file and display it in a Cocoa OpenGL program.

I am an intern at my job and I have little or no clue how to start. I have been reading wikipedia & articles on YUV, but I couldn't find any good source code on how to open a raw YUV file, extract the data and convert it into RGB and display it in the view window.

Essentially, I need help with the following aspects of the task -how to extract the YUV data from the sample YUV file -how to convert the YUV data into RGB color space -how to display the RGB color space in OpenGL. (This one I think I can figure out with time, but I really need help with the first two points)

please either tell me the classes to use, or point me to places where i can learn about YUV graphic/video display

Answer 1

This answer is not correct, see the other answers and comments. Original answer left below for posterity.

---

You can't display it directly. You'll need to convert it to an RGB texture. As you may have gathered from Wikipedia, there are a bunch of variations on the YUV color space. Make sure you're using the right one.

For each pixel, the conversion from YUV to RGB is a straightforward linear transformation. You just do the same thing to each pixel independently.

Once you've converted the image to RGB, you can display it by creating a texture. You need to call glGenTextures() to allocate a texture handle, glBindTexture() to bind the texture to the render context, and glTexImage2D() to upload the texture data to the GPU. To render it, you again call glBindTexture(), followed by the rendering of a quad with texture coordinates set up properly.

// parameters: image:  pointer to raw YUV input data
//             width:  image width (must be a power of 2)
//             height: image height (must be a power of 2)
// returns: a handle to the resulting RGB texture
GLuint makeTextureFromYUV(const float *image, int width, int height)
{
    float *rgbImage = (float *)malloc(width * height * 3 * sizeof(float));  // check for NULL
    float *rgbImagePtr = rgbImage;

    // convert from YUV to RGB (floats used here for simplicity; it's a little
    // trickier with 8-bit ints)
    int y, x;
    for(y = 0; y < height; y++)
    {
        for(x = 0; x < width; x++)
        {
            float Y = *image++;
            float U = *image++;
            float V = *image++;
            *rgbImagePtr++ = Y                + 1.13983f * V;  // R
            *rgbImagePtr++ = Y - 0.39465f * U - 0.58060f * V;  // G
            *rgbImagePtr++ = Y + 2.03211f * U;                 // B
        }
    }

    // create texture
    GLuint texture;
    glGenTextures(1, &texture);

    // bind texture to render context
    glBindTexture(GL_TEXTURE_2D, texture);

    // upload texture data
    glTexImage2D(GL_TEXTURE_2D, 0, 3, width, height, 0, GL_RGB, GL_FLOAT, rgbImage);

    // don't use mipmapping (since we're not creating any mipmaps); the default
    // minification filter uses mipmapping.  Use linear filtering for minification
    // and magnification.
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

    // free data (it's now been copied onto the GPU) and return texture handle
    free(rgbImage);
    return texture;
}

To render:

glBindTexture(GL_TEXTURE_2D, texture);

glBegin(GL_QUADS);
    glTexCoord2f(0.0f, 0.0f); glVertex3f( 0.0f,  0.0f, 0.0f);
    glTexCoord2f(1.0f, 0.0f); glVertex3f(64.0f,  0.0f, 0.0f);
    glTexCoord2f(1.0f, 1.0f); glVertex3f(64.0f, 64.0f, 0.0f);
    glTexCoord2f(0.0f, 1.0f); glVertex3f( 0.0f, 64.0f, 0.0f);
glEnd();

And don't forget to call glEnable(GL_TEXTURE_2D) at some point during initialization, and call glDeleteTextures(1, &texture) during shutdown.

Answer 2

Adam Rosenfield’s comment is incorrect. On Macs, you can display YCbCr (the digital equivalent to YUV) textures using the GL_YCBCR_422_APPLE texture format, as specified in the APPLE_ycbcr_422 extension.

Answer 3

I've done this with YUV frames captured from a CCD camera. Unfortunately, there are a number of different YUV formats. I believe the one that Apple uses for the GL_YCBCR_422_APPLE texture format is technically 2VUY422. To convert an image from a YUV422 frame generated by an IIDC Firewire camera to 2VUY422, I've used the following:

void yuv422_2vuy422(const unsigned char *theYUVFrame, unsigned char *the422Frame, const unsigned int width, const unsigned int height) 
{
    int i =0, j=0;
    unsigned int numPixels = width * height;
    unsigned int totalNumberOfPasses = numPixels * 2;
    register unsigned int y0, y1, y2, y3, u0, u2, v0, v2;

    while (i < (totalNumberOfPasses) )
    {
     u0 = theYUVFrame[i++]-128;
     y0 = theYUVFrame[i++];
     v0 = theYUVFrame[i++]-128;
     y1 = theYUVFrame[i++];
     u2 = theYUVFrame[i++]-128;
     y2 = theYUVFrame[i++];
     v2 = theYUVFrame[i++]-128;
     y3 = theYUVFrame[i++];

     // U0 Y0 V0 Y1 U2 Y2 V2 Y3

     // Remap the values to 2VUY (YUYS?) (Y422) colorspace for OpenGL
     // Y0 U Y1 V Y2 U Y3 V

     // IIDC cameras are full-range y=[0..255], u,v=[-127..+127], where display is "video range" (y=[16..240], u,v=[16..236])

     the422Frame[j++] = ((y0 * 240) / 255 + 16);
     the422Frame[j++] = ((u0 * 236) / 255 + 128);
     the422Frame[j++] = ((y1 * 240) / 255 + 16);
     the422Frame[j++] = ((v0 * 236) / 255 + 128);
     the422Frame[j++] = ((y2 * 240) / 255 + 16);
     the422Frame[j++] = ((u2 * 236) / 255 + 128);
     the422Frame[j++] = ((y3 * 240) / 255 + 16);
     the422Frame[j++] = ((v2 * 236) / 255 + 128);
    }
}

For efficient display of a YUV video source, you may wish to use Apple's client storage extension, which you can set up using something like the following:

glEnable(GL_TEXTURE_RECTANGLE_EXT);
glBindTexture(GL_TEXTURE_RECTANGLE_EXT, 1);

glTextureRangeAPPLE(GL_TEXTURE_RECTANGLE_EXT, videoImageWidth * videoImageHeight * 2, videoTexture);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_STORAGE_HINT_APPLE , GL_STORAGE_SHARED_APPLE);
glPixelStorei(GL_UNPACK_CLIENT_STORAGE_APPLE, GL_TRUE);

glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glPixelStorei(GL_UNPACK_ROW_LENGTH, 0);

glTexImage2D(GL_TEXTURE_RECTANGLE_EXT, 0, GL_RGBA, videoImageWidth, videoImageHeight, 0, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_REV_APPLE, videoTexture);

This lets you quickly change out the data stored within your client-side video texture before each frame to be displayed on the screen.

To draw, you could then use code like the following:

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);   
glEnable(GL_TEXTURE_2D);

glViewport(0, 0, [self frame].size.width, [self frame].size.height);

glMatrixMode(GL_PROJECTION);
glLoadIdentity();
NSRect bounds = NSRectFromCGRect([self bounds]);
glOrtho( (GLfloat)NSMinX(bounds), (GLfloat)NSMaxX(bounds), (GLfloat)NSMinY(bounds), (GLfloat)NSMaxY(bounds), -1.0, 1.0);

glBindTexture(GL_TEXTURE_RECTANGLE_EXT, 1);
glTexSubImage2D (GL_TEXTURE_RECTANGLE_EXT, 0, 0, 0, videoImageWidth, videoImageHeight, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_REV_APPLE, videoTexture);

glMatrixMode(GL_TEXTURE);
glLoadIdentity();

glBegin(GL_QUADS);
 glTexCoord2f(0.0f, 0.0f);
 glVertex2f(0.0f, videoImageHeight);

 glTexCoord2f(0.0f, videoImageHeight);
 glVertex2f(0.0f, 0.0f);

 glTexCoord2f(videoImageWidth, videoImageHeight);
 glVertex2f(videoImageWidth, 0.0f);

 glTexCoord2f(videoImageWidth, 0.0f);
 glVertex2f(videoImageWidth, videoImageHeight);  
glEnd();