Hello Scene — Screen Captures for Fun and Profit

Being able to capture the display screen opens up some interesting possibilities for our demo scenes.

In this particular case, my demo app is capturing a part of my screen, and using it as a ‘texture map’ on a trapezoid, and compositing that onto a perlin noise background. The capture is live, as we’ll see shortly, but first, the code that does this little demo (sampmania.cpp).

#include "gui.h"
#include "sampledraw2d.h"
#include "screensnapshot.h"
#include "perlintexture.h"
 
ScreenSnapshot screenSamp;
 
void onFrame()
{
    // Take current snapshot of screen
    screenSamp.next();
 
    // Trapezoid
    PixelCoord verts[] = { PixelCoord({600,100}),PixelCoord({1000,100}),PixelCoord({1700,800}),PixelCoord({510,800}) };
    int nverts = 4;
    sampleConvexPolygon(*gAppSurface, 
        verts, nverts, 0, 
        screenSamp, 
        { 0,0,canvasWidth, canvasHeight });
 
}
 
void setup()
{
    setCanvasSize(1920, 1080);
    setFrameRate(15);
 
    // Draw noisy background only once
    NoiseSampler perlinSamp(4);
    sampleRectangle(*gAppSurface, gAppSurface->frame(), perlinSamp);
 
    // Setup the screen sampler
    // Capture left half of screen
    screenSamp.init(0, 0, displayWidth / 2, displayHeight);
}

Pretty standard fair for our demos. There are a couple of new concepts here though. One is a sampler, the other is the ScreenSnapshot object. Let’s first take a look at the ScreenSnapshot object. The idea here is we want to take a picture of what’s on the screen, and make it available to the program in a PixelArray, which is how we represent pixel images in general. If we can do that, we can further use the screen snapshot just like the canvas. We can draw on it, save it, whatever.

On the Windows platform, there are 2 or 3 ways to take a snapshot of the display screen. Each method comes from a different era of the evolution of the Windows APIs, and has various benefits or limitations. In this case, we use the most ancient method for taking a snapshot, relying on the good old GDI API to do the work, since it’s been reliable all the way back to Windows 3.0.

#pragma once
// ScreenSnapshot
//
// Take a snapshot of a portion of the screen and hold
// it in a PixelArray (User32PixelMap)
//
// When constructed, a single snapshot is taken.
// every time you want a new snapshot, just call 'next()'
// This is great for doing a live screen capture
//
//    ScreenSnapshot ss(x,y, width, height);
//
//    References:
//    https://www.codeproject.com/articles/5051/various-methods-for-capturing-the-screen
//    https://stackoverflow.com/questions/5069104/fastest-method-of-screen-capturing-on-windows
//  https://github.com/bmharper/WindowsDesktopDuplicationSample
//
 
#include "User32PixelMap.h"
 
class ScreenSnapshot : public User32PixelMap
{
    HDC fSourceDC;  // Device Context for the screen
 
    // which location on the screen are we capturing
    int fOriginX;   
    int fOriginY;
 
 
public:
    ScreenSnapshot()
        : fSourceDC(nullptr)
        , fOriginX(0)
        , fOriginY(0)
    {}
 
    ScreenSnapshot(int x, int y, int awidth, int aheight, HDC srcDC = NULL)
        : User32PixelMap(awidth, aheight),
        fOriginX(x),
        fOriginY(y)
    {
        init(x, y, awidth, aheight, NULL);
 
        // take at least one snapshot
        next();
    }
 
    bool init(int x, int y, int awidth, int aheight, HDC srcDC=NULL)
    {
        User32PixelMap::init(awidth, aheight);
 
        if (NULL == srcDC)
            fSourceDC = GetDC(nullptr);
        else
            fSourceDC = srcDC;
 
        fOriginX = x;
        fOriginY = y;
 
        return true;
    }
 
    // take a snapshot of current screen
    bool next()
    {
        // copy the screendc into our backing buffer
        // getDC retrieves the device context of the backing buffer
        // which in this case is the 'destination'
        // the fSourceDC is the source
        // the width and height are dictated by the width() and height() 
        // and the source origin is given by fOriginX, fOriginY
        // We use the parameters (SRCCOPY, CAPTUREBLT) because that seems 
        // to be best practice in this case
        BitBlt(getDC(), 0, 0, width(), height(), fSourceDC, fOriginX, fOriginY, SRCCOPY | CAPTUREBLT);
 
        return true;
    }
};

There’s really not much to it. The real working end of it is the ‘next()’ function. That function call to ‘BitBlt()’ is where all the magic happens. That’s a Graphics Device Interface (GDI) system call, which will copy from one “DeviceContext” to another. A DevieContext is a Windows construct that represents the interface for drawing into something. This interface exists for screens, printers, or bitmaps in memory. Very old, very basic, very functional.

So, the basics are, get a ‘DeviceContext’ for the screen, and another ‘DeviceContext’ for a bitmap in memory, and call BitBlt to copy pixes from one to the other.

Also, notice the ScreenSnapshot inherits from User32PixelMap. We first saw this early on in this series (What’s In a Window), when we were first exploring how to put pixels up on the screen. We’re just leveraging what was built there, which was essentially a Windows Bitmap.

OK, so bottom line, we can take a picture of the screen, and put it into a bitmap, that we can then use in various ways.

Here’s the movie

Well, isn’t that nifty. You might notice that if you query the internet for “screen capture”, you’ll find links to tons of products that do screen capture, and recording. Finding a library that does this for you programmatically is a bit more difficult. One method that seems to pop up a lot is to capture the screen to a file, or to the clipboard, but that’s not what you want, you just want it in a bitmap ready to go, which is what we do here.

On Windows, a more modern method is to use DirectX, because that’s the preferred interface of modern day Windows. The GDI calls under the covers probably call into DirectX. The benefit of using this simple BitBlt() method is that you don’t have to increase your dependencies, and you don’t need to learn a fairly complicated interface layer, just to capture the screen.

I’ve used a complex image here, mainly to draw attention to this subject, but really, the screen capturing and viewing can be much simpler.

Just a straight up view, without any geometric transformation, other than to fit the rectangle.

Code that looks very similar, but just using a simple mapping to a rectangle, rather than a trapezoid. This is from screenview.cpp

//
// screenview
// Simplest application to do continuous screen capture
// and display in another window.
//
#include "gui.h"
 
#include "screensnapshot.h"
 
ScreenSnapshot screenSamp;
 
void onFrame()
{
    // Get current screen snapshot
    screenSamp.next();
 
    // Draw a rectangle with snapshot as texture
    sampleRectangle(*gAppSurface,gAppSurface->frame(),screenSamp);
}
 
// Do application setup before things get
// going
void setup()
{
    // Setup application window
    setCanvasSize(displayWidth/2, displayHeight);
 
    // setup the snapshot
    screenSamp.init(0, 0, displayWidth / 2, displayHeight);
}
 
void keyReleased(const KeyboardEvent& e) {
    switch (e.keyCode)
    {
    case VK_ESCAPE:
        halt();
        break;
 
    case 'R':
    {
        recordingToggle();
    }
    break;
    }
}

Capturing the screen has additional benefit for our demo scenes. One little used feature of Windows is the fact you can use translucency, and transparency. As such, you can display rather interesting things on the display. Using the recording technique where we just capture what’s on our canvas won’t really capture what the user will see. You’ll only capture what you’re drawing in your own buffer. In order to capture the fullness of the demo, you need to capture what’s on the screen.

And just to kick it up a notch, and show off some other things you can do with transparency…

In both these cases of the chasing balls, as well as the transparent keyboard, there is a function call within the demo scene ‘layered()’. If you call this in your setup, then your window won’t have any sort of border, and if you use transparency in your colors, they’ll be composited with whatever is on the desktop.

You can go one step further (in the case of the chasing balls), and call ‘fullscreen()’, which will essentiallly do a: setCanvas(displayWidth, displayHeight); layered();

There is one additional call, which allows you to retain your window title bar (for moving around and closing), but sets a global transparency level for your window ‘windowOpacity(double)’, which takes a value between 0.0 (fully transparent), and 1.0 (fully opaque).

And of course the demo code for the disappearing rectangles trick.

#include "apphost.h"
#include "draw.h"
#include "maths.hpp"
 
using namespace maths;
 
bool outlineOnly = false;
double opacity = 1.0;
 
INLINE PixelRGBA randomColor(uint32_t alpha=255)
{
    uint32_t r = random_int(255);
    uint32_t g = random_int(255);
    uint32_t b = random_int(255);
 
    return { r,g,b,alpha };
}
 
void handleKeyboardEvent(const KeyboardEventTopic& p, const KeyboardEvent& e)
{
    if (e.keyCode == VK_ESCAPE)
        halt();
 
    if (e.keyCode == VK_SPACE)
        outlineOnly = !outlineOnly;
 
    if (e.keyCode == VK_UP)
        opacity = maths::Clamp(opacity + 0.05, 0.0, 1.0);
 
    if (e.keyCode == VK_DOWN)
        opacity = maths::Clamp(opacity - 0.05, 0.0, 1.0);
 
    windowOpacity(opacity);
}
 
void onLoop()
{
    PixelRGBA stroke;
    PixelRGBA fill;
    PixelRGBA c;
 
    gAppSurface->setAllPixels(PixelRGBA(0x0));
 
    for (int i = 1; i <= 2000; i++)
    {
        int x1 = random_int(canvasWidth - 1);
        int y1 = random_int(canvasHeight - 1);
        int lwidth = random_int(4, 60);
        int lheight = random_int(4, 60);
 
        c = randomColor(192);
 
        if (outlineOnly)
        {
            stroke = c;
            draw::rectangle_copy(*gAppSurface, x1, y1, lwidth, lheight, c);
 
        }
        else
        {
            fill = c;
            //draw::rectangle_copy(*gAppSurface, x1, y1, lwidth, lheight, c);
            draw::rectangle_blend(*gAppSurface, x1, y1, lwidth, lheight, c);
        }
    }
 
    refreshScreen();
}
 
void onLoad()
{
    subscribe(handleKeyboardEvent);
 
    setCanvasSize(800, 800);
}

Well, that’s a lot of stuff, but mostly we covered various forms of screen capture, what you can do with it, and why recording just your own drawing buffer doesn’t show the full fidelity of your work.

We also covered a little bit of Windows wizardry with transparent windows, a very little known or used feature, but we can use it to great advantage for certain kinds of apps.

From a design perspective, I chose to use an ancient, but still supported API call, because it has the least number of dependencies, is the easiest of all the screen capture methods to understand and implement, and it uses the smallest amount of code.

Another thing of note for this demo framework is the maximum usage of ‘.h’ files. In each demo sample, there’s typically only 2 or 3 ‘.cpp’ files, and NO .dll files. This is again for simplicity and portability. You could easily put things in a library, and having appmain.cpp in a .exe file would even work, but that leads down a different path. Here, we just make every demo self contained, compiling all the code needed right then and there. This works out when your file count is relatively small (fewer than 10), and you’re working on a small team (fewer than 5). This probably does not scale as well beyond that.

But, there you have it. We’ve gone all the way from putting a single pixel on the screen, to displaying complex deometries with animation in transparent windows. The only thing left in this series is to draw some text, and call it a wrap.

So, next time.