Monday, February 10, 2014

PhotoBooth (Free Code)

I’ve been blogging recently about some of my experiments with the Kinect for Windows v2 developer preview. Most of my experiments have been based on the supplied samples, with a lot of copy and paste action. I’ve pulled some of the repeated code into some re-usable classes.

Here’s some code that I’ve moved out of the samples into a reusable component that takes a screen shot of the Kinect output. As an added bonus, I’ve added a count-down timer. The code provided here is based on the original samples, modified slightly. Free as in beer.

Here’s an example of how to use it.

First, add the visual for the count-down timer into your UI. I’m putting mine below the image that shows my Kinect output.

<ViewBox>
    <Image Source="{Binding ImageSource}" Stretch="UniformToFill" />
    <TextBlock 
        Text="{Binding CountDownTimer}"
        HorizontalAlignment="Center"
        VerticalAlignment="Center"
        FontSize="96"
        Foreground="White" />
</ViewBox>

<Button 
    Style="{StaticResource ScreenshotButton}"
    Content="Screenshot"
    Click="ScreenshotButton_Click" />

Next, initialize the PhotoBooth in your ViewModel. Note that I’m being very lazy and I’m using the code-behind of the XAML as my ViewModel. You know I’m not a big fan of this, I’m just following the lead of the Kinect examples so let’s pretend I didn’t do that. Do proper MVVM, kids.

public class MainWindow : Window, INotifyPropertyChanged
{
    private KinectSensor _sensor;
    private DepthFrameReader _reader;
    private WriteableBitmapImage _bmp;

    // see my last post on how to use the framecounter!
    private FrameCounter _frameCounter;
    private PhotoBooth _photoBooth;

    private string _statusText;
    private string _countDownTimer;


    public MainWindow()
    {
        InitializeComponent();

        _sensor = KinectSensor.Default;
        _sensor.Open();
        _reader = _sensor.DepthFrameSource.OpenReader();
        _reader.FrameArrived += FrameArrived;

        _photoBooth = new PhotoBooth();
        _photoBooth.PropertyChanged += (o,e) => 
            CountDownTimer = _photoBooth.TimeDisplay;       

        _frameCounter = new FrameCounter();
        _frameCounter.PropertyChanged += (o,e) =>
            StatusText = String.Format("{0} FPS", _frameCounter.FramesPerSecond);

        DataContext = this;
    }

    public string StatusText { /* get/set property changed omitted */ }
    public string CountDownTimer { /* get/set property changed omitted */ }
    public BitmapImage ImageSource { /* get/set property changed omitted */ }

    async void ScreenshotButton_Click(object sender, EventArgs e)
    {

        string fileName = await _photoBooth.TakePhoto(_bmp);

        if (!String.IsNullOrEmpty(fileName)
        {
            StatusText = string.Format("Snapshot created: {0}", fileName);
        }
        else
        {
            StatusText = "Couldn't create snapshot :(";
        }

        // allow our message to show for 5 seconds...
        _frameCounter.DeferFrameCount(TimeSpan.FromSeconds(5));
    }

    void FrameArrived(object sender, DepthFrameSourceEventArgs e)
    {
        // create bitmap, write depth data to bitmap
    }
}

By default, the Photobooth will create a new PNG in your Pictures folder with a random guid as the file name. You can change both of these to suit your needs.

// default directory is My Pictures
_photoBooth.BaseDirectory = "C:\Snaps";

// specify the file naming convention. will add .png to the end...
_photoBooth.CreateFileName = () => {
    string time = System.DateTime.Now.ToString("hh'-'mm'-'ss", CultureInfo.CurrentUICulture.DateTimeFormat);
    return String.Format("KinectScreenshot-PhotoBoothSample-{0}", time);    
};

Click to expand to get the full source.

public class PhotoBooth : INotifyPropertyChanged
{
    private int _timeLeft;
    private Func<string> _createFileName;
    private string _baseDirectory;
    private TimeSpan _waitInterval;

    public PhotoBooth()
    {
        BaseDirectory = Environment.GetFolderPath(Environment.SpecialFolder.MyPictures);
        WaitInterval = TimeSpan.FromSeconds(5);
    }

    public string BaseDirectory
    {
        get { return _baseDirectory; }
        set { _baseDirectory = value; }
    }

    public TimeSpan WaitInterval
    {
        get { return _waitInterval; }
        set { _waitInterval = value; }
    }

    public string TimeDisplay
    {
        get
        {
            if (TimeLeft > 0)
            {
                return TimeLeft.ToString();
            }

            return String.Empty;
        }
    }

    public int TimeLeft
    {
        get { return _timeLeft; }
        protected set
        {
            if (_timeLeft != value)
            {
                _timeLeft = value;
                var handler = PropertyChanged;
                if (handler != null)
                {
                    handler(this, new PropertyChangedEventArgs("TimeLeft"));
                    handler(this, new PropertyChangedEventArgs("TimeDisplay"));
                }
            }
        }
    }

    public async Task<string> TakePhoto(WriteableBitmap bitmap, string path = null, int seconds = -1)
    {
        if (string.IsNullOrEmpty(path))
        {
            path = BaseDirectory;
        }

        if (seconds == -1)
        {
            seconds = (int)WaitInterval.TotalSeconds;
        }

        if (seconds > 0)
        {
            TimeLeft = seconds;
            while (TimeLeft > 0)
            {
                await Task.Delay(1000);
                TimeLeft = TimeLeft - 1;
            }
        }

        string fileName = Path.Combine(path, CreateFileName() + ".png");

        bool success = SaveBitmap(fileName, bitmap);

        return success ? fileName : String.Empty;
    }

    public Func<string> CreateFileName
    {
        get
        {
            if (_createFileName == null)
            {
                _createFileName = () => Guid.NewGuid().ToString();
            }
            return _createFileName;
        }
        set
        {
            _createFileName = value;
        }
    }

    public event PropertyChangedEventHandler PropertyChanged;

    private bool SaveBitmap(string path, BitmapSource bitmapSource)
    {
        if (bitmapSource == null)
            return false;

        // create a png bitmap encoder which knows how to save a .png file
        BitmapEncoder encoder = new PngBitmapEncoder();

        // create frame from the writable bitmap and add to encoder
        encoder.Frames.Add(BitmapFrame.Create(bitmapSource));


        // write the new file to disk
        try
        {
            // FileStream is IDisposable
            using (FileStream fs = new FileStream(path, FileMode.Create))
            {
                encoder.Save(fs);
            }

        }
        catch (IOException)
        {
            return false;
        }

        return true;            
    }
}

Happy coding

submit to reddit

Thursday, February 06, 2014

FPS and CPU Counter (Free Code)

I’ve been blogging recently about some of my experiments with the Kinect for Windows v2 developer preview. Most of my experiments have been based on the supplied samples, with a lot of copy and paste action. I’ve pulled some of the repeated code into some re-usable classes.

Here’s some code that I’ve moved out of the samples into a reusable component that measures frames-per-second and the amount of time taken per frame. The code provided here is based on the original samples, modified slightly. Free as in beer, though I doubt any of you will buy me beer so let’s just say “free”.

Example how to use:

public class MainWindow : Window, INotifyPropertyChanged
{
    private FrameCounter _frameCounter;
    private KinectSensor _sensor;
    private DepthFrameReader _reader;


    public MainWindow()
    {
        InitializeComponent();

        _frameCounter = new FrameCounter();
        _frameCounter.PropertyChanged += (o,e) =>
            StatusText = String.Format("FPS = {0:N1} / CPU = {1:N6}",
                            _frameCounter.FramesPerSecond,
                            _frameCounter.CpuTimePerFrame);

        _sensor = KinectSensor.Default;
        _sensor.Open();
        _reader = sensor.DepthFrameSource.OpenReader();
        _reader.FrameArrived += FrameArrived;

        this.DataContext = this;
    }

    private void FrameArrived(object sender, DepthFrameArrivedEventArgs e)
    {
        var reference = e.FrameReference;
        DepthFrame depthFrame = null;

        try
        {
            // increment the frame counter
            using (_frameCounter.Increment())
            {
                depthFrame = reference.AcquireFrame();
                
                // do work with depth data...


            } // disposing the frame counter will calculate CPU time for this block
        }
        catch
        {
        }
        finally
        {
            if (depthFrame != null)
                depthFrame.Dispose();
        }
    }

}

Expand the block below for code goodness.

public class FrameCounter : INotifyPropertyChanged
{
    private Stopwatch _stopWatch;
    private uint _framesSinceUpdate = 0;
    private DateTime _nextStatusUpdate = DateTime.MinValue;
    private CpuCounter _cpuCounter;
    private double _fps;
    private double _cpuTime;

    public FrameCounter()
    {
        _stopWatch = new Stopwatch();
        _cpuCounter = new CpuCounter(this);
    }

    public double FramesPerSecond
    {
        get { return _fps; }
        protected set
        {
            if (_fps != value)
            {
                _fps = value;
                NotifyPropertyChanged("FramesPerSecond");
            }
        }
    }

    public double CpuTimePerFrame
    {
        get { return _cpuTime; }
        protected set
        {
            if (_cpuTime != value)
            {
                _cpuTime = value;
                NotifyPropertyChanged("CpuTimePerFrame");
            }
        }
    }

    public event PropertyChangedEventHandler PropertyChanged;

    public void DeferFrameCount(TimeSpan timeToWait)
    {
        this._nextStatusUpdate = DateTime.Now + timeToWait;
    }

    public IDisposable Increment()
    {
        this._framesSinceUpdate++;

        if (DateTime.Now >= this._nextStatusUpdate)
        {
            double fps = 0.0;
            double cpuTime = 0.0;

            if (this._stopWatch.IsRunning)
            {
                this._stopWatch.Stop();
                fps = this._framesSinceUpdate / this._stopWatch.Elapsed.TotalSeconds;
                cpuTime = this._cpuTime / fps;
                this._stopWatch.Reset();
            }

            this._nextStatusUpdate = DateTime.Now + TimeSpan.FromSeconds(1);
            this.FramesPerSecond = fps;
            this.CpuTimePerFrame = cpuTime;
        }

        if (!this._stopWatch.IsRunning)
        {
            this._framesSinceUpdate = 0;
            this._cpuTime = 0;
            this._stopWatch.Start();
        }

        _cpuCounter.Reset();
        return _cpuCounter;
    }

    internal void IncrementCpuTime(double amount)
    {
        _cpuTime += amount;
    }

    public void Reset()
    {
        _nextStatusUpdate = DateTime.MinValue;
        _framesSinceUpdate = 0;
        FramesPerSecond = 0;
        CpuTimePerFrame = 0;
    }

    protected void NotifyPropertyChanged(string propertyName)
    {
        if (PropertyChanged != null)
        {
            PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
        }
    }

    public class CpuCounter : IDisposable
    {
        FrameCounter _parent;
        Stopwatch _stopWatch;

        internal CpuCounter(FrameCounter parent)
        {
            _parent = parent;
            _stopWatch = new Stopwatch();
        }

        public TimeSpan Elapsed
        {
            get
            {
                return _stopWatch.Elapsed;
            }
        }

        public void Reset()
        {
            _stopWatch.Reset();
            if (!_stopWatch.IsRunning)
            {
                _stopWatch.Start();
            }
            
        }

        public void Dispose()
        {
            _stopWatch.Stop();
            _parent.IncrementCpuTime(_stopWatch.Elapsed.TotalMilliseconds);
        }
    }
}

Happy coding.

submit to reddit

Tuesday, February 04, 2014

Comparing Kinect v1 and v2 Depth Data

In this post I show how the new Kinect v2 stacks up against the previous sensor. I’ve been blogging about the new Kinect as part of the Kinect for Windows Developer Preview. First and foremost, I have to get this legal disclaimer out of the way…

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

As part of this post, I thought it would be fun to build an application that targets both versions of the device and toggle back and forth between them for comparison sake. There’s a little bit of magic involved as both have the same namespaces and assembly name. Here’s you how you can do the same.

  1. Make a copy of the v1.8 Microsoft.Kinect.dll as Microsoft.Kinect.v1.dll
  2. In your project, add a reference this assembly and change the alias from global to v1.
  3. Do the same for v2.
  4. Add the following code above your using statements.
extern alias v1;
extern alias v2;

using KinectV1 = v1::Microsoft.Kinect;
using KinectV2 = v2::Microsoft.Kinect;

What’s changed between versions?

The first and perhaps most surprising difference is that the frame resolution of the depth data is 512 x 424. Don’t panic. Yes, the Kinect v1 supports a higher resolution of 640x480 resolution, but don’t get caught up on that. This apparently is the native resolution of the camera and although Microsoft may expose other resolutions in a later release, the differences between Time of Flight and Structured Light more than make up for this difference.

As I understand it, Structured Light (the depth technology in v1) reconstructs the depth data by analyzing deformations of a projected IR pattern. This produces approximations for pixels between the projected points. Time of Flight (used in Kinect v2) however blasts the scene with IR and measures how quickly the light comes back to the sensor. This produces better results as each pixel has a unique value. Time of Flight sensors also tend to have lower resolutions, so 512 x 424 is actually pretty impressive compared to other TOF sensors.

The second main change is the field of view and focal length for the camera. The Kinect can see more of the area and can detect objects much closer.

Side by Side Comparisons

The following table shows the observable differences in the depth frame data between versions.

  Kinect V1 Kinect V2
Height 480 424
Width 640 512
Aspect Ratio 4:3 6:5
Horizontal FOV 58.5 70.6
Vertical FOV 45.6 60
Diagonal FOV 70 89.5
Min Depth 0.8 m 0.5 m
Max Depth 4.0 m 4.5 m

The following snapshots are taken from two Kinect devices sitting side by side:

WP_20140203_001

Kinect V1

As part of the structured light algorithm, parts of the depth data are approximated. The subject must be at least 800mm from the camera.

KinectScreenshot-ColorMap-09-27-48

Kinect V2

The new Time of Flight sensor provides greater depth qualities per pixel and a much improved signal to noise ratio despite having an odd 6:5 aspect ratio (512 x 424 px). The increased field of view allows the camera to see more of the scene.

KinectScreenshot-ColorMap-09-28-02

submit to reddit