Origins and use case for ProgUI

In our school projects someone has to code the UI, there are no built in frameworks for this in the engines we use for most of the projects. I had the job of coding the UI for menus in my third project.

I poured over the requirements for the project and concluded we could fulfill them all with a single flexible UI widget, a list box with multi choice rows.

It could be a column of buttons for menus with several options, one requirement and each option could have multiple values and that would be enough to switch resolutions, change sound volume in steps, perfect!

Now preprod was coming to an end and the UI artist presented her concepts, yeah that was not exactly just a list box. So I learned that as a programmer I need to be prepared to support whatever ideas they might have. Also handling different resolutions and aspect ratios was not really simple either.

I wanted to have that squared away once and for all so I could focus on realizing the artists visions and not be bogged down with boring code.

I needed ProgUI! 

”Quirky API but does its job well enough” – total success

I made the first version version of a simple framework for UI that would then come to be used in all my coming projects in one way or another.

In each project new features were added, bugs fixed and if I may say so myself it has proved to be quite successful. However some features were also dropped as they were not needed and design changes made them obsolete.

I have not been the only one to use it and I have gotten good feedback from those that did. The introduction can be a bit daunting as it is has a little bit unconventional design, but after a bit of initial support they have been able to be productive and have admitted that it was a helpful thing that made it easier for them to code up a working UI.

A slight problem, I felt, was that the code base was branching as it was developed in several projects at the same time. It was hard to keep track of updates, bugs and features. I found myself fixing the same bugs more than once.

That is the background to why I choose to make the final project of my own choosing to wrap up all this experience and try to make a reusable modular code base that I and others, if they find it interesting, can continue to develop from a single repository and be added as a dependency to projects rather than as part of the individual projects code.

Overall design 

The refactored version has a three layered architecture.

At the base there is an integration layer that abstract the creation and rendering of sprites and text, input handling as well as providing some features needed for serialization.

ProgUI is the core library that handles the basics features of a UI framework such as scaling, positioning, input routing and the lifecycle of its elements.

DemoUI is an example on how a widget library can be created on top of the core library. This is useful to promote reuse and modularity of code.

At the top is the game code that is responsible for initializing the library and most importantly respond to the UI it not only looks pretty but actually  does something for the game.

The following sections will detail each layer from the bottom up. Fell free to skip around between them, I have tried to make each section as stand-alone as possible so you can jump to the parts that interest you the most.

Integration layer

The goal with the integration layer was to separate as much implementation specific parts of the framework as possible, leaving only the core functionality behind.

 Although most of the API for the framework is leaning heavily towards composition over inheritance I felt that it made sense here to define some interfaces to abstract the creation, modification and deletion of sprites and text.

I opted to not make an abstract interface for serialization, mostly because I didn’t really have time to do everything I wanted. Nlhomann’s json library is a small single header dependency and I was comfortable keeping that way for now.

This is how the integration API ended up. It still needs a bit more for input handling but I am not sure exactly how I want to do that bit, so for now it is only what I needed for my demo code. I would also like to add some documentation for each function to make clear how they are intended to be used, though most of them should be pretty self-explanatory.

Integration API

ProgUI – the core library

At its core ProgUI is a retained mode UI implementation.

The workhorse of the library is the UIElement class that conceptually can be thought of as an input region. In its current version it is restricted to be an axis aligned box, this is likely to change in the future to allow rotations and non rectangular borders.

UIElements has a collection of UILayers that represent the actually visible parts of the UI. Each UILayer has two UIRenderables a UISprite and a UIText that are drawn by the host application while the UILayer API is used for controlling their properties like if they shall be visible or not, color or offset position relative the UIElement they belong to.

The main UI class is responsible for creating, updating and routing input to the correct UIElements and mange their lifecycle. It also owns an instance of the abstract class IntegrationInterface that delegates the lifecycle management of UIRenderables to the host application.

Saving and restoring of the UI is also done through the UI API and is delegated to a friend class, ProgUISerializer.

Currently the UI class also contain a simple editor that is mainly there to inspect and make minor corrections of UIElements and UILayers properties that can then be saved and restored. In the future I think this should be broken out to its own class.

UI API

The entry point for the application for working with ProgUI is the UI class that handles initialization and creation of the top level UIElements.

Some things to note that might not be quite obvious:

  1. Since ProgUI needs an instance of ProgUI::IntegrationInterface it is passed in as a mandatory argument in the constructor. I am not quite happy with this, I was just not sure how to do it right. It takes a pointer which is used to initialize a std::unique_ptr internally, thus taking ownership of the integration and that is not obvious from the API. I will have to do a bit more homework on move semantics I guess. 
  2. The API contains two callback functions OnSerialize and OnDeserialize these are intended for the application level code to add any global scope data that it needs to restore a serialized UI.
  3. The application shall call Init, then keep calling Update each frame with a UIUpdateContext, this is meant as a way for the application to be able to pass on needed data to the custom code. By default it only contains a single float for the frames delta time since the last frame. But the idea is that it can be inherited from and cast to the application specific version as needed.

Size, positions and DesignSize

Every UI needs a model for managing positions and sizes when the UI is rendered in a variety of resolutions and aspect ratios.

ProgUI uses the concept of a DesignSize to which all relative positions and sizes are normalized. The idea here is that artists and programmers agree on a resolution for which the UI is optimized for example 1920 x 1080. When the UI is rendered at its DesignSize it should not need to scale any assets and look its crispest.

ProgUI does not have an explicit canvas as is traditional, however the concept is there in an implicit manner. Canvas in ProgUI is referring to the rectangular viewport where it is rendered, this might be the full area or part of the screen or window client area depending on settings. Currently the canvas is fixed to the center of the screen/window, a future item is to be able to anchor the canvas relatives to points on the screen/window. 

To handle how it should behave when rendered in any other size ProgUI relies on two parts. The first part is that all UIElements are positioned relative to one of nine anchor point of its parent, the four corners, its center and the mid points of each edge. This gives an extra level of control over where they should end up on screen.

The second part is that one can choose a scale mode for the UI that determines the form of the canvas. The scale modes are:

  • Stretch – This will stretch the canvas to the full size of the screen or window client area. This will not stretch the UIElements as well but determines where they will end up on the screen as the relative positions of the canvas’s anchor points are affected by changes in aspect ratio of the canvas.
  • FitToWidth – This will keep the aspect ratio of the DesignSize and scale the canvas so that its full width fit on the screen/window. This will cause empty space above and below the UI if rendered with a lower horizontal aspect ratio than the DesignSize and with a higher part of the top and bottom would be outside the viewport.
  • FitToHeight – this works the same as FitToWitdh but it ensures the full height is visible rather than width.
  • BestFit – this will keep the DesignSize aspect ratio for the canvas and choose to scale either to width or height of the viewport to ensure the whole UI is visible, creating empty borders on top and bottom or left and right.

UIElement API

A UIElement is the basic building blocks in ProgUI. Composition is used to shape a UIElements look and behavior.

Out of the box a UIElement doesn’t do anything but determine whether the mouse cursor is over it or not, it’s not even visible. To make it visible at least one UILayer needs to be added, these are the components that actually rendered.

To provide access to data that may be needed it has a simple std::any Data member that can be used to keep its state and other useful data.

One can think of the UIElement as something that orchestrates its layers and children to create the desired behavior and look. To that end it has a set of callbacks where the user can write its custom logic for each element.

Each callback has a reference to the relevant UIElement and those involving mouse interaction has the coordinates of the mouse as convenience arguments as well. The OnUpdate callback has an extra argument passed in that is an inheritable struct that can be cast to an application specific version as a way to pass in data from the application needed for updating the elements. Below is the full UIElement API, I won’t go into details here most things should be fairly obvious.

DemoUI – Reusable widgets

Although its quite possible to build a fully working UI using only ProgUI its flexible design comes at the cost of needing a bit verbose code.

To deal with that it can be convenient to pack up reusable ProgUI code in simpler reusable functions or classes. There is not a mature nor mandatory API for how to do this.

DemoUI is a far from fully fledged library that gives one example, a proof of concept if you will on how it can be done implementing chained mechanisms for serialization and application logic.

Let’s take a look at how it works by examine an example progress bar widget. Here is what we want it to do:

  • Show a textured bar whose length is proportional to a float value ranging from 0.0 to 1.0
  • Has a background shoving in the area the progress bar is not covering.
  • Has a frame surrounding the bar.
  • Abel to give it a color tint
  • Bindable to any float in the game play code
  • Optionally vertical instead of horizontal (rotation of UIElements is not a feature yet unfortunately).

The pattern I came up with when toying around how to do this is that widgets normally takes three arguments to create, a reference to a parent UIElement, a name as a const char* and a pointer to a descriptor struct. The descriptor is created for each type of widget and is serving a two part purpose.

  1. It collects the many parameters needed for the widget in a single a argument shortening the function signature.
  2. It serves as the runtime state for the created widget and is set as the Data member on the UIElement(s) created for the widget.

We are going to use these structures and package up our functions in a class, most of them could just be free functions but I could not find a way to do that neatly for parts of the serialization system. Suggestions are welcome!

ProgressBarDescriptor is mostly just describing the UIElement to be created it has single sate variable, normalizedFloatValue that is bound to a float in the application that will drive the progress bar.

WidgetDescriptor holds data that any widget need. The onDataSave will be explained in the Serialization section.

The implementation looks like this:

First we add a new UIElement to its parent setting its type using an enum class, this is used to recognize the widget type in other code segments. We make use of the fluent API to set the properties of the element. The descriptor is copied in to the Data member of the element where it can be referenced in any callback.

As this is a simple non interactive UIElement we only need a few of the callbacks:

  • OnInspect is used to draw a section of ImGui code for editing custom data for a element in the UI’s built in editor.
  • OnUpdate will monitor the float value and calculate the size and offset for the progress bars sprite to strtch it to its proper length.
  • OnSerialize is used to serialize the descriptor as part of the serialization process for the UIElement, in DemoUI this is a function shared for all widgets at this time. But it could be separate functions or lambdas if that is preferred.

Then we add three UILayers that will be drawn in the order we add them, so first the background sprite, then the bar sprite and finally the frame on top of them both.

Finally we return a reference to the added UIElement.

Lets take a look at those callbacks as well.

The first callback is a simple switch statement that add keys to a json object passed in for storing custom data in.

After that there is a check for if a callback is set in the WidgetDescriptor::onDataSave member. If so we call that functor and pass in a new json object that we will then add under the key ”bindingData”.

This object is meant to contain the applications specific data needed to re bind the widget to its data sources. It is optional and set by the application that can register a callback to use that data during de serialization to perform the needed binding.

The common UIElement and UILayer member data is serialized by the ProgUI’s serializer helper class.

The next two callbacks are unique for this widget.

The first one adds a slider to set the progress value which can be useful for testing that it works. Nothing more interesting there really.

The second is the update function that grabs the UILayer with the bar sprite and figure out a scale and position offset for it based on the current progress value. At the end the UIElement Refresh function is called to make sure it will be properly reformatted with the new layer setting, this is probably redundant and an artefact of a dubious design choice where I use a flag to indicate if a UIElement needs to be refreshed due to any changes like this. However the system was never extended does to the UILayer level and is not perfectly reliable. For now it will remain on the known bugs list. 

The Game Code

Finally we will take a look at the application level coding for ProgUI. This is the code used for the small demo shown shown here.

There is a bit of a chicken and the egg situation here, the UI can be completely loaded from a file but for that to work that file needs to be created somewhere. This will be discussed a bit more in the Serialization section.

First the UI needs to be created and initialized.

The member variable myUI is an instance of the ProgUI::UI class and was initialized with an integration implementation for the engine used in the containing class’s constructor.

We can now access that integration and set the callback to use for de serializing custom UIElement data, we use a common function for all widgets in DemoUI analogous to the SerializeElement callback seen previously.

We then also set a callback for reading the data binding data at de serialization time using a different technique.

Then we give it a name and initialize it with our chosen design size and scale mode.

Then we check if there exists a json serialization file and if so we load that to set up the rest. If it does not we call the function BuildUI that programmatically does the set up. Lets take a look at that code as well.

We have already taken a look at the progress bar, here is just showing that we can do both vertical and horizontal bars. It also show that the same function is used but with other sprites, in this case they are the same just rotated 90 degrees.

Artists can make them in any size and form that match the games theme.

In the top part i set up the slightly more complicated radial menu that is made from a center button and a number of child buttons that are laid out radially around the center button.

I also add a background image, this is commonly done when making menus but here I just needed a backdrop to get more of a game feel in the demo. 

Another thing to note is that I add a panel directly under the canvas and add all the widgets as children to this. Together they form a tree structure and can be moved around and scaled as a unit which is rather practical.

Serialization

In this specialization project or rather ”choose your own project” I wanted to make a pretty solid serialization system for the UI. At one point the library had a json serialization but it was made obsolete and was not updated and then disappeared altogether.

It also only serialized the layout of the UI and there was no way to handle data and function binding, nor was it ever really needed. Writing the UI in code was sufficient and the UI artist usually preferred working in other programs to create the assets, layouts and test the visual behavior.

With this in mind having a complete serialization might be a bit superfluous but there is a use case for this or two maybe that I have been pondering.

  1. It would enable the creation of a stand alone UI creation and editor tool that could be used by the artist to test out their assets against mock implementations of the game play systems it needs to access, this tool would be quick for the programmer to adapt from project to project making it possible to work on UI ahead of the game paly development that usually has quite a bit of lead time. The result can then be saved and the same mock system can be integrated in the game and successively replaced with real systems allowing UI updates be as simple as replacing a json file and possibly some updated assets.
  2. If the artist uses a tool like Figma to mockup their UI it might be possible to write an interpreter that can connect and extract assets and basic behavior from it. Then it could process the data and write a json file based on that. In this scenario there would still be a need to add the data binding to the UI. Then again it might be possible to handle many cases with some form of tag system or by using stringent naming conventions to encode the intent directly in Figma or what ever tool the artist is comfortable with.

The image shows a part of a save file focused on the progress bar widget used as an example in the DemoUI section. 

Built in editor

The UI class has a built in editor that can be turned on and off using the void ProgUI::UI::ToggleDebug(bool aDoDebug) function.

I will be the first to admit it is lacking a lot of features and that i would take an extreme makeover to make it look pretty.

Nevertheless it has proven to be useful to make adjustments of the layout and to try out and debug the custom code.

In the image we can see that extra slider named Progress where the value of the bound float can be changed to test its effect not only in the UI but also in the game.

I am not all to happy with having it as a part of the UI class it makes it a bit bloated, the advantage has simply been that as a member in the same class it has full access to protected and private data. This could probably be replaced with moving it to a helper friend class.

That said I think that if I decide to put a larger effort in editor functionality I would rather work on the stand alone editor suggested in the Serialization section as that would bring other benefits to the UI pipeline.

This is also an unnecessary external dependency that is not really needed if one doesn’t need the editor or do not already have Dear ImGui integrated in ones engine code base making that part of integration a bit bigger.