February 2024 Progress Update

Lot’s to talk about. Let’s dig in.

We’re still plugging away at our 1.20 update and development environment refactors, but instead of hearing a summarized version of February development from me, let’s hear it from the devs themselves, starting with dotModded:

The update to 1.20 is still underway, quite a bit of progress has been made however. We’ve gone through the existing interface and have decided to completely reimplement the interactions with the game and how we grab values from it. This will also lead to far fewer injection points throughout the render process making the variance from version to version far less. Currently we have our menus added, shaders compiling, resources created, and uniforms mostly registered with few remaining that change multiple times in a frame. We’re still working on locating a few injection spots to properly render a frame, and correcting the gl state for a modern render process, then restoring it after for the GUI rendering.

I’ve also looked into AMD support for Continuum 2.1 and will soon be looking at RT. I’m currently testing on the latest drivers, and it has been truly headache inducing, I believe I’ve run into a bug with bindless textures; they’re working on some driver versions, but not others. Our implementation of them appears to be valid but the driver isn’t reporting anything meaningful. This lack of detail in the logs from OpenGL makes it very hard to even begin figuring out what could possibly be going so wrong that it corrupts the gl state permanently leading to the game to crash with different exit codes every time. I’ll be going back and testing various driver versions and try to find something more modern than the current recommendation. This is an open issue and I’m looking at it.

Jake has continued working on the Tracked Value System and Lua during all this, so I’ll let him take it from here:

The last time I gave an update regarding Focal VK it, quote, detailed me bashing my head against the wall that has been the tracked value system. Here we are 3 months later and I can confidently say that I have broken through the wall and found a solution that not only does what we want, but is also workable long term. Before I dive into what’s changed with the tracked value system, I’ll give some background information on the tracked value system so that you have a better understanding of what it is, what it does and what it’s used for. If you don’t care about this and just want to know what problems I ran into and what’s changed, skip the next paragraph.

Fundamentally, the tracked value system manages and maintains a set of global variables registered to the tracked value system either by Focal Engine, Focal Engine’s Minecraft interface or the render pack itself. Examples of some global variables that may be registered include the current frame index, the total running time of the game/engine, the size of the window, the current world time in the game, etc. Each variable has an associated data type (ie integer, float, boolean, vector, matrix, etc) and array length (ie you can have an array of vectors that you index into), can be updated mid-frame either by Focal Engine directly or indirectly by an update system that lets you compute variables from other variables with automated scheduling, is stored centrally within a set of storage buffers owned by the tracked value system and may be referenced in a smart pointer-esque fashion by either Focal Engine, Focal Engine’s Minecraft interface or the render pack itself. Focal Engine Alpha currently uses a more primitive form of the tracked value system that lacks some of these features (namely the ability to register from the render pack and the ability to create update systems to compute variables from other variables), however Focal VK will use a more advanced form of the tracked value system that includes all of these features, and maybe more if we can think of more (and they’re reasonable to implement).

Whew, that was a lot. Anyways, back to explaining what’s changed. So, the core problem I kept running into is that the tracked value system, as we originally designed it prior to moving to Rust, was fundamentally at odds with Rust’s type system and how Rust’s trait objects work. The original design essentially used a three-tier class/object hierarchy for storing and working with variables within the tracked value system: at the bottom you have an “base” class which acts purely as an interface for anything that doesn’t need to know the type of a variable (Lua, internal state, etc), in the middle you have a “storage” class which extends the “base” class with a generic/template type and uses that to store a copy of the variable’s value, then at the top you have a set of three classes (“constant” for non-updating variables, “dynamic” for updated-from-C++ variables and “computed” for updated-from-Lua variables) which extends the “storage” class with additional data and functionality for their respective responsibilities.

The intent was to have the tracked value system store the “base” class and cast the “base” class up to the other two class tiers as necessary, assuming that we know the type, which lets the tracked value system store variables without needing to know the types, while still allowing us to interact with variables through typed interfaces. This does not work in Rust, however, as Rust does not allow the casting necessary to do what we want (the bottom and middle classes would be considered trait objects under Rust, and you can’t cast a trait object to another trait object, only to a concrete object), so instantly our original design for C++ does not work under Rust. One of the first things I tried was to get rid of the “storage” class, but that caused other issues as we’d need to deal with lifetimes due to how the Lua library we use worked at the time, which is less than ideal since lifetimes tend to spread across a code base really easily.

At this point I knew we had to completely redesign the tracked value system, as our original design just would not work under Rust. The first thing I reached for, and the thing I’m using in the current version of the tracked value system, was trait objects. Yes, they didn’t work for our original design since we can’t upcast to another trait object, but they’re very useful for “hiding” a generic type from Rust. The only reason why we needed to upcast under our original design was because we had update-from-Lua functionality baked into the variables themselves, so an easy way to get rid of upcasting is to simply separate out that update-from-Lua functionality into its own subsystem, which eventually became update systems. Doing so not only removed the need to upcast, but also removed the need to deal with lifetimes in variables when combined with a recent addition to the Lua library we use (they added the ability to store strongly owned Lua objects that don’t require lifetimes, whereas before you could only store weakly owned Lua objects that required lifetimes to safely maintain the reference back to the Lua instance), killing two birds with one stone.

At this point we could store variables without needing to know the type of the variable and we could store update systems without needing to deal with lifetimes, but we had to figure out a way to connect the two so that update systems could reference the necessary variables. For that we made the assumption that update systems would only contain Lua code, which allowed us to use Lua as a bridge to connect the two. We added a pair of methods to the base trait object that we use to “hide” the generic type from Rust, which would let us both convert the variable to a Lua value and update the variable with a given Lua value, where applicable. This made it so that we could no longer support any type since we could only support types that can be converted to and from Lua values (something that’ll be brought back up later), but this did let us store the base variable trait object in the update system, which in turn lets the update system read from and write to variables without needing to know the type, as it uses that pair of methods.

With that in place, all we had to do was add some more code to determine the order that update systems need to execute in (easy) and finalise the allocation of storage buffers (relatively easy), and the core of the tracked value system is functional! After over half a dozen rewrites I got to this point and set up a basic test environment, registering one set of variables from Rust and another set of variables from Lua, then registering an update system from Lua and setting it up to update each of the variables registered from Lua, then setting a breakpoint at the end of my main function to be able to see the result of half a dozen rewrites, and hit run on my debugger. It ran and hit the breakpoint, without a crash. I opened up the tracked value system in my variable inspector, navigated to where the storage buffers were located in the tracked value system, and saw that it had successfully updated the variables from Lua. Unfortunately I don’t have much code to share of this beyond this little snippet of what the Lua registration function looked like, but it felt good to actually get this thing working.

Skip ahead to the tracked value system in its current state and I’ve refined a lot of the code, streamlined update systems and storage buffer allocation, and added the ability to create typed references in both Rust and Lua. You have the ability to register variables from Lua, define update systems in Lua that can read from and write to any number of variables (with some rules governing which ones you can write to), you can create references from Lua code to pass back into Focal Engine, you can create what I’m currently calling transform systems to do some on-the-fly transformations of variables before they’re handed back to Lua, the tracked value system will seamlessly convert Lua values to “constant” variables that aren’t registered with the tracked value system but can still be used in Rust code like any other tracked values (streamlines some stuff on both ends), and you can directly read variables into Lua as Lua values, if you want to directly use them within Lua code. Currently arrays aren’t implemented as I have yet to figure out a clean way to index into them, though the tracked value system is set up to support them.

The Rust side of the tracked value system is also a bit cumbersome to use, which is something I want to try to fix with Rust’s macros. But it works, and it works enough for us to start using it. I also have some further stuff planned for it, for instance I would like to switch away from trait objects and instead use enums since we can only support a fixed subset of types anyways, and I eventually want to look at more advanced storage buffer allocation, maybe even allowing for some dynamic variable allocation. Below you can see some Rust and Lua code showing what the tracked value system currently looks like to use on both ends.

Code snippets showcasing variable registration from Rust, variable registration from Lua and variable referencing from Lua below (click to enlarge)

Quickly regarding Lua, currently I do have a functional Lua environment set up for Focal Engine, but I’m not 100% happy with it and plan to rewrite it. Currently it’s purely single threaded in design, which is problematic as we’d like to work with Lua from multiple threads, and dealing with Lua callbacks is annoying as you have to lug around the entire Lua environment any time you want to run a Lua callback. I’ll give another progress update once I’ve got the new Lua environment set up, then after that I’ll migrate the tracked value system over to it. The render graph is also somewhat in limbo at the moment as I wanted to finish both the tracked value system and Lua environment first, as both will heavily integrate into the render graph.

That concludes our fairly…lengthy update from the devs for this month. We hope you guys enjoy these, as they do take a bit more time to prepare.

Yep, you read that right. Continuum 2.0. In the previous section, dot touched on how we have been poking at Continuum 2.1 to see if we could restore AMD compatibility (on newer drivers than 22.6.1 that is). While we have not yet been successful with Continuum 2.1, due to it’s increased complexity, we were able to fix up Continuum 2.0, do some testing with a few community members with AMD GPU’s and get a hotfix released!

The latest download of Continuum 2.0.5 from our site or inside Focal Engine should function on all recent AMD graphics hardware (we’ve tested from RX 500 series onward), up to the latest 24.1.1 driver release. This may change, as it often does with AMD drivers, but for now, it’s good to go!

If you use an AMD GPU and have a copy of 2.0.5 downloaded before March 2024, just delete it, and grab a fresh download to restore functionality. Everyone else should be fine to use either version without worries, as this issue only affected AMD GPU’s. We will update our System Requirements and update you all in a future blog if a new driver release causes problems again.