The open-source game engine and graphics engine for multiplayer, cross-platform, real-time 3D action

Cafu News Cafu News

Reintroducing “layers” (aka “groups”) to the Map Editor

by Carsten Fuchs | Monday, 2014-11-17
A while ago, in September, I considered it a good idea to remove the "groups" feature from the Map Editor. This was fueled by the observation that entities too can naturally and inherently serve for grouping the individual elements in a map. Keeping a second, explicit mechanism for the same purpose (the "groups" feature) only seemed to be a redundant burden. Therefore, the intention was to turn the existing two methods for grouping into a single method, to be implemented by having entities subsume the functionality of the former groups, thus making groups fully redundant so that they could be removed without loss.

Alas, after the change was complete and a considerable effort had been spent to make entities and entity hierarchies work as well as groups did, the first tests and feedback quickly showed that it didn't work out, and that my assessment of the situation (and my enthusiasm for entities, or in fact, the entity component system) did this time not lead to the desired solution. Practically speaking, we found that the "groups", as they were implemented before, were just too good and too convenient to be removed: my attempt to achieve the same with entities was just not on par (think e.g. of the "Hide Other" button). Considering this more closely, it turned out that this was not a problem of the implementation, but of a more principal nature:

Entities and their hierarchies are an excellent way to organize (group) map elements in terms of the game or the virtual world that is being developed: You use entities to model game objects like players, opponents, cars, lifts, or anything else that is relevant for the game.

Groups, however, serve a quite different organizational purpose that, in hindsight, is orthogonal to that of entities: With groups, you express features that are important to the map designer's workflow or are of technical relevance – but not to the logical modeling of a virtual game world.

mapeditor_groups_contextmenu.png
Image Detail Image Download

Therefore, I'm currently working on undoing the commits that removed "groups" from the Map Editor, and on reintroducing them soon. At the same time, I will take the opportunity to rename the former "groups" to "layers", because "groups" is a too general word (entities group map elements, too!), whereas "layers", with their analogy to 2D drawing programs, seem to describe the purpose of the feature in a more precise and more intuitive manner.

In summary:

  • Entities group map elements by their logical meaning in the virtual game world, as observed by players, as needed by script authors, as seen by the core engine, etc.
  • Layers group map elements to facilitate technical details for the map designer, e.g. to temporarily hide objects, to lock them from being accidentally modified, etc.

Note that this is currently still a work in progress: I'll post here again as soon as the reintroduce-groups branch is completely implemented and committed. :up:

Post and read comments about this news item at the related forum topic.

Article about client prediction at Heise Developer

by Carsten Fuchs | Monday, 2014-09-01
Today there was another very nice article at the renowned German publisher Heise about compensating the lag in online games by means of the "client prediction" technique:

Verzögerungen in Online-Spielen durch Client-Side Prediction kompensieren (text is in German)

The author Christian Oeing explains the basics of online games whose simulated world is managed by a central, authoritative game server that clients can connect to. It is very well described how network latency affects basic implementations of such client-server setups, and how this causes the player to perceive the game as sluggish or lagged. As the key matter of the article, Christian explains how the "client prediction" can be used to compensate for the network lag. As "client prediction" can, quasi as a side effect, cause the client's idea of the world state and that of the server to disagree, the author also describes the techniques that can be use to overcome such divergencies: "reconciliation" (or "reprediction", in Cafu lingo) and interpolation.

What makes the article special and my definitive recommendation (for those among you who speak German), is that all this is written in a very readable manner that is easy to follow and understand – and the huge coincidence that everything that he describes is also implemented in the Cafu Engine since several years. (To be precise, Cafu implements the "client prediction" and the very important "reprediction" features, but deals with interpolation only in a slightly different manner than described in the article.) Thus, especially for those who want to familiarize themselves with the Cafu networking code but find the comments and internal documentation too terse, the article is a must read.

If you don't speak German, the article also provides links to English texts:



In the meanwhile, we're still very busy with finishing the details of our newly introduced Component Systems.
The basic work is all long done, but I would still like to fix a few more loose ties in the Map Editor before this very fundamental project can be declared done and ready for public consumption. :cheesy: :up:

Post and read comments about this news item at the related forum topic.

Article about entity component systems at Heise Developer

by Carsten Fuchs | Friday, 2014-07-18
As an interesting coincidence, the renowned German publisher Heise today published an article in their online Developers section about entity component systems:

Component-Based Entity Systems in Spielen (text is in German)

The text is a very nice description of entity component systems, and quite similar to my recent news post More about Component Systems of January 2014 (see our News and Announcements for related posts and follow-ups).

The authors present some additional details that are slightly different from ours, but the crucial elements, namely using aggregation rather than class hierarchies for feature "management", and using flexible means for managing each component's data (variables), exactly match ours.

In summary, the text is a great additional view on component systems. Comparing its details to ours is interesting and provides additional perspective and insights. :up:

Post and read comments about this news item at the related forum topic.

Progress on the Entity Component System

by Carsten Fuchs | Thursday, 2014-05-22
In my previous news post "More about Component Systems" I presented the central ideas and concepts about the two component systems that we are about to introduce to the Cafu Engine. Since then, a lot of work has been achieved, and it is time for an update:

The biggest challenge and the focus of my efforts has been the porting of the old DeathMatch game code to the new component system. This was very successful, and in fact the old DeathMatch game code could be entirely removed and be replaced with newly-written components that have just the right mix of C++ and Lua script code.

In fact, the successful removal of the old, intertwined and tricky DeathMatch C++ code was the key step that motivated me to write this news update. However, that doesn't mean that we're done yet. Some things had consciously (and temporarily) to be broken and must now be fixed again, and other things are not yet as they should be. Here is a summary of the technical aspects:


Updates to the repository

If you're following the Cafu source code repository, especially on branch master-vc2012 or entity-component-system, the following recent changes are important:

  • I just tagged revision 1debdc3 (of 2013-05-25, almost exactly one year old!) with name pre-entity-component-system. This is the revision where our master branch was at the time when the development of the entity component system was begun. The annotated tag message is this:
    Code:
    This is the last revision before the extensive work was begun to introduce a
    component system for game entities to Cafu.

    It is expected that the development will take a long time. Although it is
    intended to keep backwards-compatibility wherever possible, it is expected
    that compatibility to now existing binary files (especially `cw` world files)
    and to custom Lua map scripts (e.g. `TechDemo.lua`) must and will be given up
    in the process.

    Contrary to the already completed introduction of a component system for GUI
    windows, it is expected that the upcoming work is of a long-running nature,
    comes with extensive changes to almost all parts of the Cafu Engine, and is
    possibly "open-ended". It is therefore developed in the `master` branch rather
    than a feature branch that would run for months and years, and in the meantime
    only leave `master` back at this commit, which will sooner rather than later
    become old and unsupported.

    Also see http://www.cafu.de/forum/viewtopic.php?p=6212#p6212 for additional
    information.

  • Next, I fast-forwarded branch master to entity-component-system (integrating entity-component-system into master), in which until now the entity component system was developed, then deleted the now no longer needed branch entity-component-system (and master-vc2012 as well).

  • If you are among those who, like me, had checked out branch entity-component-system, and possibly based work on it, you can use these steps to update your working copy (the steps are shown for Windows, but are very similar under Linux):
    Code:
    # Fetch all the new branches, commits and tags:
    > git fetch --all --prune
    > git fetch --tags

    # Clean up the working copy, i.e. stash or commit local uncommitted changes
    # as necessary, and (on latest entity-component-system), delete possible
    # leftovers from previous revisions:
    > rd /s Games\DeathMatch\Code
    > rd /s Games\VSWM

    # Checkout the "old" master branch, and fast-forward to origin/master:
    > git checkout master       # old master
    > git merge origin/master --ff-only

    # In my case, it was sufficient to delete the now obsolete local branch,
    # however if you have based own work on it, you'll rather want to rebase it
    # onto or merge into master first.
    > git branch -d entity-component-system

Please note that although I felt that this is the right time to continue the development of the entity component system in the master branch rather than keeping it further artificially separated in the now gone entity-component-system, at this time master is not in a state where a release could be made from it: In fact, if you compile and run it, you will quickly find a lot of problems.

These problems are well known to me (but any feedback is welcome, don't fear duplicate reports), and the rest of this news post summarizes the problems and the plans to resolve them.


Plans regarding "Player Prototype" entities

Implementing the "carried weapons" of a human player is, in all its details, surprisingly difficult. A lot more difficult than the code for monsters and other non-player entities. In fact I managed to move its previous, old implementation into the component system -- but there it is still old and inflexible code, and not in the useful and helpful spirit of the component system as everything else.

Also, moving the code necessitated some compromises, and these compromises, while they made it possible to integrate entity-component-system into master, are also responsible for the "broken" look and feel of today's (May 2014's) master.

Pondering this gave raise to another insight: So far, we have no concept for human player entities in the component system at all! That is, giving the map designer a chance to make any settings or script code adjustments for the human player entities that are supposed to spawn in the map was just not possible.

To fix all this, the idea is to introduce a new special entity that acts as a "Player Prototype". Of this new entity, exactly one instance exists in each map that the map designer can use to customize all details to his heart's contents, including that of possibly carried weapons. The Cafu Engine then spawns an entity for each human player that joins the map by cloning the "Player Prototype".

Among others, the player prototype entity has a list of components of type "CarriedWeapon", whose properties and script callbacks define one weapon each, and each such weapon would be completely self-contained, keeping information about e.g. its availability to the player ("has it been picked up earlier?"), whether it is active or holstered (more than one weapon can be active at a time, e.g left and right hands), the current, minimum and maximum ammo, and so on.

Developing this "Player Prototype", the carried weapons, and all related functionality will amount to a total rewrite of the old code, and be the immediate goal of my next steps -- this is exactly what we have introduced the component system for, after all! Even if it is not obvious from this text alone, when done you will see that the result is very very worthwhile! :wohow:


More work to be done

The above is crucial, but there is still a lot more important work to be done, and branch master, despite my promoting it as described above, currently has known bugs and generally not the quality that we normally strive to achieve and are used to.

Most importantly, the carried weapons must be fixed as described above. However, also the particle engine needs a thorough overhaul, and we really have to re-implement the precaching feature also for our component system, whose current lack causes noticeable lag.

Finally, the Map Editor is the only part left that is not yet adapted to the entity component system. This is very important, as the Map Editor is often the program that new users look at first.

Post and read comments about this news item at the related forum topic.

Server state and client effects in online games

by Carsten Fuchs | Sunday, 2014-01-19
As explained in my previous news post, I am currently migrating the old DeathMatch game code to the new Component System architecture. Just this afternoon I finished dealing with the "hand grenade" entities in the new system, and I found that they're noteworthy in many regards:

  • Hand grenades are both dynamically created during the course of a game (when the player throws them) and dynamically removed again (after they have exploded and their traces have gone). This makes them different from many other entities that are defined in map files and often exist from the beginning to the end of the map.
  • Hand grenades employ more components than other entities in order to implement their functionality:
    • Model (the hand grenade model as thrown through the map),
    • Physics (for computing the flight path and dealing with collisions, e.g. with walls and the floor),
    • Script (for controlling the details, e.g. letting the detonation timer tick and implementing the explosion effects),
    • Light (for the flare when the hand grenade explodes),
    • Sound (for the sound at the time of the explosion),
    • ParticleSystem (for the smoke and spark particles at the explosion).
  • They very well demonstrate the distinction between Server State and Client Effects.

It is the last point that I would like to highlight in this post, because a good understanding of it is crucial for game development with Cafu:


Server State

The server state is characterized by the following properties:

  • It is synchronized via the game server to all clients,
  • every player (client) sees the same results and consequences,
  • server state is relevant for gameplay.

Typical examples are drawbridges, the position and state of elevators and doors, details about opponents, etc.
As a side note, events are often a result of a change in the server state, and transferred to the clients as such.


Client Effects

Client effects can be triggered by some event in the server state, but besides that, they:

  • run independently of each other on the clients only,
  • are often used for eye-candy,
  • it's ok if not everyone sees exactly the same (not relevant for gameplay).

Examples include particle effects such as smoke, sparks and small pieces of debris, many fade or transition effects, model animation frames, etc.


Having both

An important insight is that we sometimes need both server state and client effects combined.

For example, consider a light source that illuminates an otherwise pitch black part of a labyrinth. Whether the light source is on or off clearly makes an important difference for players who try to navigate the labyrinth, and so the state of the light source is managed as a part of the server state.

However, there are also other kinds of light sources, e.g. those that are on, but steadily flicker or pulse. This is often a means to create a specific atmosphere to a map, but the exact amplitude of the flicker or pulse is really not something that needs synchronization over the network.

In fact, as a combination we may wish that the light source's on or off state is synchronized to everyone, and if it is on, have it have a client-side flicker effect for eye-candy.

This is exactly what we can (starting with commit d11c6a9) achieve with the light source components, and what I now employ with the hand grenades: When a hand grenade explodes, the related light source is turned on in order to mimic the bright flare of the explosion. However, the exact details of how the bright light fades, changes its color from white over yellow to red, shrinks and eventually disappears (all in about half a second), needs not be synchronized over the network.

Instead, a very simple script method is used to implement this. Here is its entire code:

Code:
local clTime = 0.0
local Duration = 0.5

function Light:ClientEffect(t)
    if not self:get("On") then
        clTime = 0.0
        return false
    end

    clTime = clTime + t

    if clTime >= Duration then
        return true, 0, 0, 0, 0
    end

    local Amount = 1.0 - clTime/Duration

    return true, Amount, Amount*Amount, Amount*Amount*Amount, 400.0
end

Well, this is one of the reasons why I love the new component system! :loveit:

Post and read comments about this news item at the related forum topic.

More about Component Systems

by Carsten Fuchs | Thursday, 2014-01-02
When I first posted about Component Systems exactly one year ago, things were still at the very beginning. Since then, the efforts to introduce Component Systems to the GUI window hierarchies of 2D and 3D GUIs, and to the game entity hierarchies of game maps made very good progress. In fact, I was so busy with and dug into my work, that I didn't get down to actually talk about them. :oops:
The good news is that it is now perfectly clear that the new Component Systems are a very important step for the future of the Cafu Engine. They give joy and pleasure every day that I work with them.

So I thought that this was a very good time to briefly reflect on what has happened during the past year, and to highlight a few related aspects in greater detail.


The problems with class hierarchies

Before it sinks into oblivion for good, I would like to summarize where we came from and what the problems were, as this helps to understand what the Component Systems actually achieve.

entity-class-hierarchy.png
Image Detail Image Download
An excerpt of the old entity class hierarchy.

Until about a year ago, we used to use "classic" C++ class hierarchies both for GUI windows as well as for game entities. You can see an excerpt of the entity class hierarchy as it used to be found in the (now revised) C++ API documentation in the image to the right.

At the time that it was written, even though we were aware that inheritance is not a means for code reuse, we thought that a class hierarchy was a good idea, because it naturally seemed to group entities that have related features. And even though the implementation actually worked in the beginning, it soon turned out that there were plenty of shortcomings.

For example, at some point we wanted to augment entities to play 3D sounds, complete with Doppler effects and settings to control the volume, emission cone, and several other details. Our natural approach was to derive a new EntSoundT entity class from the BaseEntityT class, and have it implement all the features that we wanted. However, it soon turned out that we not only wanted dedicated EntSoundT entities to play sounds, but many other entities (of different entity classes) as well. For example, an item that is picked up by the player and respawns a few seconds later makes a sound at both occasions. So we re-implemented the sound functionality in the item's entity code as well. And soon in the code for monsters, for human players, and several others.

It took me a while to realize it, but when I did, the result was pretty shocking: Even though the Cafu Sound System allows for minimal code in the game entities, not only were we left with several pieces of code that all essentially did the same, scattered across many entity classes. It was also very cumbersome to properly document all this (in fact we never did), to keep it in sync with the Map Editor and tools, and to deal with any special cases that occurred.

You can find another account on the the principal problems with class hierarchies in the blog post Cowboy Programming: Evolve Your Hierarchy by Mick West.

In hindsight, I'd say that many of these shortcomings were so severe that they effectively hindered or even halted the further development of the Cafu Engine. The old code just didn't scale, as adding new features caused the combinatorial complexity to grow worse and worse. Also, the code in the Map Editor, the map compile tools and the Cafu Engine was not the same, and keeping it in sync turned more and more into a pain. Not to speak of the special-cases that were needed here and there, and which we never managed to totally avoid.

These issues also burdened game developers with unnecessary complexity, e.g. the requirement to master C++ in order to make even small changes or to add small features, a burden which many game developers understandably did not want to concern themselves with.


Two class hierarchies, two Component Systems

game-entities.jpg
Image Detail Image Download
An example of game entities.


TeleporterGUI.jpg
Image Detail Image Download
An example of GUI windows.

Before we proceed, be aware that in Cafu we have two class hierarchies that are important for game development:

  1. game entities
  2. GUI windows

Game entities are the objects that populate our game worlds, such as the human player, monsters, weapons, items, ...

GUI windows are the basic rectangles of which our GUIs are built. GUIs are used both in 2D (for the Main Menu, the in-game console, etc.), as well as in 3D, where they are a part of the game worlds and used e.g. as lift controls, computer terminals, and so on.

These two systems are designed to work together, but regarding code design, they're largely independent of each other. By coincidence however, we find that both are structurally very similar to each other and share the same ideas of hierarchical setup and organization of concrete instances.

So both systems suffer the same problems as described above, and both happen to be candidates for migration to a Component System. In fact, as the GUI windows hierarchy was so much smaller than the game entities hierarchy, I started the initial tests and the actual migration with the GUI windows. This went very well, and now all the efforts are focused on the game entities. Read on for details.

Generally, all that is said above and below holds for both game entities and GUI windows, although I may often talk about and provide examples for only one, but not the other.


The new design

After careful research and planning, I found that the new design can and should be as simple as this:

Entities → Components → Variables

Let's look at each of these elements in greater detail:

Entities

All entity instances are now of the same common class: EntityT. No inheritance, no class hierarchy. Essentially, entities are now nothing but lists (or "containers") of components.

A new aspect however is that entities can now have children, or "sub-entities". In fact, entity instances (not classes, don't get confused!) can now be hierarchically organized.

For example, consider a car that consist of a chassis and four wheels, or a complex lift system that consist of a moving platform and a set of moving doors at each floor. It is now possible to have an EntityT instance that represents the "whole thing", the entire car or the entire lift, possibly with a set of components that define features for the whole, and to have child entities that represent the parts, where each part is another EntityT instance that can contain components of its own.

The same is found (and possibly feels a bit more natural) in GUI window hierarchies: We often have a window that acts as a dialog frame, and child windows of it are used for adding details like the dialog message, text input fields and the "OK" and "Cancel" buttons.

Components

Each component implements exactly one feature, and each feature is implemented in exactly one component.

For game entities, we have (at the time of this writing) these components:

  • Basics (name, status),
  • Transform (origin, orientation),
  • Model,
  • Collision Model,
  • Point Light,
  • Radiosity Light,
  • Particle System,
  • Player Physics,
  • Script (custom scripted behavior),
  • Sound.

For GUI windows, we have these components:

  • Basics (name, visibility),
  • Transform (position, size, rotation),
  • Border (frame),
  • Text,
  • TextEdit,
  • Image, Model,
  • Choice, ListBox.

This may not look like much, but even though we're not finished yet (the lists will still get a bit longer), all previously existing functionality can be re-implemented in terms of these components. I take this as a sign that the key decision and design are powerful and correct. :up:

Generally, components are similar to a code design pattern: They naturally group things together that belong together, and encapsulate code and features in a manner that is concise and clear. Above all, components eliminate the combinatorial complexity that occurs with adding features in classic class hierarchies. As we will see in the next section, components also break up other old weaknesses and bring very worthwhile improvements.

Variables

Each component has a list of variables. This may sound like a note of trivial notability, but in fact I came up with a very nice scheme from which our Component Systems draw much of their power: Entities can iterate over their components, and components can iterate over their variables. On this basis, it was possible to implement features such as:

  • automatic dialogs in the Map and GUI Editors (see example below),
  • automatic serialization to and from disk,
  • automatic serialization and synchronization over the network,
  • automatic generation of documentation,
  • automatic binding to Lua (all variables are accessible by script),
  • automatic ability to interpolate all variables (of type float, double, or composites thereof).

In other words: Adding a new variable to a component imposes no extra work at all to get all of the above!
Isn't that absolutely rocking cool?? :cheesy:

These virtues were achieved by employing the "Visitor" code design pattern for implementing the variables (which, when you look at the source code, are actually small classes of their own).

Note that the motto "everything can be interpolated" is twofold: One aspect is "real" interpolation, where for example we move a platform from A to B, without concerning us with the intermediate positions along the track. Another aspect is the interpolation on the client, where we interpolate over client frames to fill the larger gaps between server frames, achieving a movement that is as smooth as possible, and independent of network latency that possibly affects the arrival of server updates.


The Component System for GUI Windows

WindowInspector_250px.png
The components of a window in a GUI.

As mentioned above, I started with the Component System for GUI windows. The goal was to re-implement the then existing functionality (and optionally to improve and expand it) in terms of the new Component System. This goal has long been reached, and it was a huge success (see ticket #125 for details).

When I was done in March 2013, there was only one problem left: I was eager to get started doing the same with the game entities. So eager in fact, that I didn't want to waste time with anything else, such as expanding the GUI system (adding new components was so easy then that it was very tempting to add ton of new features!), or mundane tasks like writing proper documentation. (The new reference documentation was of course a piece of cake, you can see it here. I'm talking about the introductory documentation in the Wiki, with screenshots and well-phrased prose texts...) Well, I'm sorry to say that these things are still not done, and unless someone is willing to help, are likely to stay that way at least until I'm done and happy with the Component System for game entities. ;-)


More benefits

Component Systems remove the complexity from combining features, providing both more flexibility and more capabilities to game development.

But there is more: With our implementation of Component Systems, we can finally use the same code everywhere: from the Map Editor over the map compile tools to the Cafu Engine, they all use the same code now! For example, in the Map Editor, we must support editor-specific features such as "selections" (objects clicked with the mouse for further editing). But now, such support is only a matter of adding another Component to the selected entity! That's all. All other code is the same as that in the map compile tools and in the Cafu Engine.

A similar aspect allows for unified script states. Previously, script states had a tendency to get fragmented, that is, it could be difficult to access the state of the game world from the script code of an embedded 3D GUI. My implementation of Component Systems unifies script states as well.

Important for many game developers, there is no longer a need to write C++ code (if you don't want to). There is full flexibility and full power available with scripts. (And those who love C++ get their money's worth as well – hacking Cafu in C++ is more fun than ever!)

And finally, we're able to create very fine scripting reference documentation mostly automatically. As indicated above, we can iterate over components and their variables. This allows to generate complete Doxygen documentation automatically. To improve this even further, it is of course also possible to augment the generated documentation by hand.

The strings that are used to document the components and their variables are available in CaWE (e.g. the Map Editor) as well! That means that the documentation is not only available as a nice online web resource, but is readily available in CaWE, too! You can see an example of this in the "Window Inspector" screenshot at the bottom ("hor. Align"). It is even possible to query the help string for a variable at the in-game console in Cafu, or from any other script code, if you wish.


References

For historical reference, here are some related forum posts where Component Systems were previously discussed:


Many thanks to all those who patiently kept explaining things to me (and in the beginning kept convincing me ;-) ) and who participated in discussions and provided suggestions!


Ongoing work

fireworks_2014.jpg
Image Detail Image Download
Fireworks at my home town, 2014-01-01.

In summary, the new Component Systems provide elegant solutions to old problems. They make game development much easier and much faster, a lot more flexible, and more fun than ever.

However, we're not done yet: the Component System for game entities has made excellent progress, but there is still work to do with porting old DeathMatch game code to the new system. There is also a lot of documentation to write (those introductory and prose parts that don't automatically write themselves...), and many pages of the website to update (Features, Gallery, ...). And once that's done, when the new Component Systems are in place and all obsolete code is gone, a new release is due, and the real fun begins: We can then add new features that we couldn't add before, add new scripts that we didn't have before, and so much more.

If you're interested in the details, have a look at the Cafu source code repository, where currently the active work is in the entity-component-system branch (which will be merged back to master once done).


I wish everyone a Happy New Year and all the best for 2014! :wohow:
Carsten

Post and read comments about this news item at the related forum topic.