Finding Pixels for Bodies in Kinect for Windows v2

In this post, I’m using Depth and BodyIndex data to create a depth visualization that highlights the active users.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

This post illustrates a few breaking changes from v1 of the Kinect for Windows API.

First off, the event signature to receive all data from the sensor has changed. Rather than getting data from the AllFramesReady event of the sensor, version 2 now uses a MultiSourceFrameReader that has a MultiSourceFrameArrived event. The MultiSourceFrameReader allows you to specify which data streams you wish to receive. This is a relatively minor change.

The other minor change this post illustrates is that the depth data no longer contains a byte to indicate the player index. Rather than applying a bitmask to the depth data, it is now exposed as its own data stream.

this.sensor = KinectSensor.Default;
this.sensor.Open();
this.reader = sensor.OpenMultiFrameSourceReader(
                        FrameSourceTypes.BodyIndex |
                        FrameSourceTypes.Depth
                        );
this.reader.MultiSourceFrameArrived += MultiSourceFrameArrived;

DepthFrameDescription desc = sensor.DepthFrameSource.FrameDescription;

this.bytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
this.depthFrameData = new ushort[desc.Width * desc.Height];
this.bodyIndexData = new byte[desc.Width * desc.Height];

this.pixels = new byte[desc.Width * desc.Height * this.bytesPerPixel];
this.bitmap = new WriteableBitMap(desc.Width, desc.Height, 96, 96, PixelFormat.Bgr32, null);

When the frame arrives, I fill both the depth and body index arrays with the available data. All that's left is to loop through the depth data and color the pixels. In this case, I'm using the GradientColorMap from my last post to decorate color pixels differently than background pixels.

void MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    MultiSourceFrameReference frameReference = e.FrameReference;

    MutliSourceFrame multiSourceFrame = null;
    DepthFrame depthFrame = null;
    BodyIndexFrame bodyIndexFrame = null;

    try
    {
        multiSourceFrame = frameReference.AcquireFrame();
        if (multiSourceFrame != null)
        {
            var depthFrameReference = multiSourceFrame.DepthFrameReference;
            var bodyIndexFrameReference = multiSourceFrame.BodyIndexFrameReference;

            depthFrame = depthFrameReference.AcquireFrame();
            bodyIndexFrame = bodyIndexFrameReference.AcquireFrame();

            if ((depthFrame != null) && (bodyIndexFrame != null))
            {
                depthFrame.CopyFrameDataToArray(this.depthFrameData);
                bodyIndexFrame.CopyFrameDataToArray(this.bodyIndexFrameData);

                int pixelIndex = 0;
                for (int i = 0; i < depthFrameData.Length; ++i)
                {
                    ushort depth = depthFrameData[i];
                    byte player = this.bodyIndexFrameData[i];
                    Color pixelColor;

                    if (player == 0xff) // nobody here
                    {
                        pixelColor = this.grayscaleMap.GetColorForDepth(depth);
                    }
                    else
                    {
                        pixelcolor = this.playerMap.GetColorForDepth(depth);
                    }
                }
                
                pixels[pixelIndex++] = pixelColor.B;
                pixels[pixelIndex++] = pixelColor.G;
                pixels[pixelIndex++] = pixelColor.R;
                pixelIndex++;
             }

             this.bitmap.WritePixels(
                 new Int32Rect(0, 0, depthWidth, depthHeight),
                 this.pixels,
                 depthWidth * this.cbytesPerPixel,
                 0);
            }

        }

    }
    catch
    {
    }
    finally
    {
        // MultiSourceFrame, DepthFrame, ColorFrame, BodyIndexFrame are IDispoable
        if (depthFrame != null)
        {
           depthFrame.Dispose();
           depthFrame = null;
        }

        if (bodyIndexFrame != null)
        {
            bodyIndexFrame.Dispose();
            bodyIndexFrame = null;
        }

        if (multiSourceFrame != null)
        {
            multiSourceFrame.Dispose();
            multiSourceFrame = null;
        }
    }

}

Easy peasy! Happy Coding!