The open-source game engine and graphics engine for multiplayer, cross-platform, real-time 3D action

I wish you all a good and happy new year 2013 and all the best wishes!




As things have been very exciting for those following our development branches in Git, but outwardly a bit calm for those who don't, let me take the opportunity for a brief review of recent developments, and an outlook on upcoming changes:

Early in the second half of 2012, we have started to focus our efforts on improving the experience for game programmers, that is, for those nice folks who are responsible for the programming part in the development of a new game that uses the Cafu Engine.
In the past, our game code has grown as the features of the Cafu Engine grew, and eventually we had to realize that customizing the DeathMatch example code for other needs, or even adding new features to it, was more complicated than it should be.

As a result, we started to fix the situation:
Improvements to and a large redesign of our game/entity code was a very important first step that brought a great relief almost immediately. The related ticket #113 is still not closed however -- we have plans for further changes.

In parallel, we found that the time was ripe for switching our version control system from Subversion to Git.
The switch turned out to consume a lot of time and efforts, but as I've pointed out in my migration protocols (Part 1 and Part 2), the benefits especially for our users are so great that it was all worth it -- and such improvements were on our agenda anyways.

And finally, in the midst of all this, our new forum member midix suggested the introduction of "entity systems" or "component systems" to the Cafu Engine. I have to admit that I was at first highly skeptical to the idea, which essentially is a different way to organize and manage all the features that game entities have:

Previously, the classic class inheritance hierarchy of game entities was the main place and instrument to organize and implement the features of the game entities. It turned out however (not only for Cafu, but also for many other game engines as well), that such class hierarchies are very inflexible when it comes to reusing or recombining features in circumstances that are even slightly different than those originally foreseen.

With "component systems", the proposal is to remove the entire game entity inheritance hierarchy, except for the root class, and instead to implement features as "components": Each component implements exactly one feature, and each feature is implemented in exactly one component. A game entity is then provided with the list of components (or features) that is should have. Of course, each component can be configured by the user: conveniently with a nice user interface in the graphical map editor, or with powerful scripting commands, or an arbitrary mix of both.

As I said, I was very skepical about it, but I could also see how a component system would improve our situation and help our mission to make game development and programming with Cafu so much easier and more flexible. And then I realized that we not only have game entity hierarchies in Cafu, but also GUI window hierarchies in our 2D and 3D GUIs, and I could see an analogy in both: The GUI windows were a much smaller hierarchy than our existing hierarchy of game entities, but they too would much benefit from the perspectives of a component system -- and as the hierarchy was so much smaller, they were the perfect test case!

So my plan was to try out and implement a component system for our GUI windows first, gather experience, then evaluate the results. If everything worked well and our expectations were met, only then implement the component system for the game entities as well.

Well, today, after only a relatively short while of working on it, and while still being in the middle of implementing the component system for GUI windows, I can already say that I'm very very pleased with the intermediate results. Things look highly promising already, and 95% of the efforts that I've spent so far will immediately be useful again when we decide to extend the component system approach to game entities as well! It's really difficult to explain that all in proper thoroughness right now, bit if you're interested in the details, check out the gui-component-system branch in our Git repository.

In summary, I'm very happy with the current perspective, and very much looking forward to the new year!

Post and read comments about this news item at the related forum topic.

Motivated by user feedback and recent developments, we've updated several pages of our website and documentation.
In this post, I'll briefly summarize the most important changes:

• Downloads: Downloading compressed archives of the Cafu source code is now possible directly via the new Git project page at Bitbucket. The source code is now always the latest, and available in zip, tar.gz and tar.bz2 formats.
  
  
• Getting Started with the Cafu Source Code has been updated to explain getting the source code with the Git version control system rather than with Subversion. (Our migration from Subversion to Git is still not entirely complete, but referring new users to the new repository is another important step.)
  
  
• The pages
  
  • Developer FAQ
  • IDEs and Text Editors (Video)
  • Starting your own Game
  • Game Development Overview
  were newly added or improved to explain better how custom games, projects or applications are created with Cafu.
  
  
• The C++ and Scripting reference documentation was upgraded to the latest version of Doxygen.

Any feedback that you may have is very much appreciated. If you would like to help, editing or contributing to the documentation Wiki is easy!

Post and read comments about this news item at the related forum topic.

In "Migrating Cafu to distributed version control – Part 1", I outlined the fundamental considerations for the migration. This post continues the subject with the more specific and technical details.


Goals and Requirements

Specifically for Cafu, what are the goals and requirements when converting to Git?

As a first step, our source code repository must technically be converted from Subversion to Git. I wanted the conversion to be done in a careful, accurate and complete manner: It should include all branches, vendor branches, tags and authors, and the proper and complete merge history. In fact, I wanted the resulting Git repository to look as if we had used Git right from the start.

Conversions like this are generally well described in the documentation and books about Git, but it turned out that our use of Vendor Branches, a normal and useful feature in Subversion, was very difficult to migrate to Git, and that only little related documentation can be found on the internet. Vendor branches are important to us because we use them to manage our external libraries.

Besides my normal work, I spent a lot of time on this, on and off for several months, slowly putting the related pieces together. I also described the problem on the git-users mailing list in thread "Importing Subversion vendor-branches to Git". The thread provides both a good technical summary as well as the remaining bits that I was still missing at that time. This post is the synopsis of the gathered results.

Secondly, as the issue tracker is closely related to the repository, it needs to be updated accordingly. Options include:

1. stay with Trac (but update it to work with Git),
   
2. migrate it to the issue tracker provided with the Cafu repository at BitBucket,
   
3. migrate it to Atlassian JIRA.

The first option would preserve the full existing flexibility of Trac, keeping us a certain degree of independence, and not require getting used to something else.
The second option would probably be the least complex and the most comfortable, but it remains to be determined if the BitBucket issue tracker is powerful enough for our needs. You can see a BitBucket issue tracker live at the BitBucket site itself.
The third option would be the most powerful, but it might as well overwhelm us.

I've not yet formed an opinion about it though, much less a decision. Fortunately, the migration of the issue tracker can be done largely independent from the migration of the repository, so that its progress does not stall the progress of the repository.


Migration Hotspots

While the bulk of the conversion is flawlessly and quickly done by the git svn ... commands, I found plenty of occasions where manual tweaking of the process, or post-processing and clean-up work was necessary in order to achieve the desired result. This is especially true whenever the Subversion source repository deviates from the classic "trunk, branches, tags" layout, or subtleties of Subversion merges prevent proper automatic conversion to Git.

In this section, I list the issues that I found the most prominent (from a Git learners perspective), along with the solutions that I eventually applied.

Branches outside branches/

If some branches are according to Subversion repository standard layout in branches/, but more branches are elsewhere, or if standard layout was never used and the branches are arbitrarily scattered across the Subversion repository, it is not immediately clear if and how these extra branches can be accounted for so that they are properly imported into Git. The solution is to split the call to git svn clone into this sequence:
The next to last line causes the subsequent fetch to load the directory vendor/ as a Git branch, as if it was another branch in branches/.

Missing Merges

The converted commit history sometimes misses merges where merges were performed in Subversion:
A was merged into master, yielding B, and the merge is properly reproduced in Git.
C was merged into master as well, yielding D, but only in Subversion. In Git, the merge is missing.

Among other reasons, this can happen if in Subversion the merge was performed not at the top directory level, but as a "partial" merge from subdirectory to subdirectory, e.g. from forum/themes/firenzie in pristine directly to the same directory in master.

The solution for such cases is to use the .git/info/grafts file, and to "fix" its results with
The --tag-name-filter cat part makes sure that attached tags are rewritten as well.
If rewriting the commits succeeded and the result is as desired, the grafts file and the references to the original commits should be deleted:
The next call to git svn fetch will automatically rebuild the rev_map that is needed for continued bidirectional communication with the source Subversion repository.

Fixing Tags

As Subversion treats tags exactly like branches, after the conversion to Git the Git branches that should be tags must be fixed manually. A good solution is described by Haenel and Plentz in their book, but it unfortunately only works with lightweight tags and thus cannot account for the tag message, which in our case is a longer text. The best solution that I have found that works with annotated tags is from this Atlassian blog post, to which I however had to make small modifications to work as desired:

Move to Subdirectory

Before I could fix missing merges in our (partially) converted Cafu repository, I had to move the contents of all commits in the "vendor" branch into a subdirectory. The documentation for git filter-branch has a related example, which I modified according to this discussion, yielding:

Vendor Branches

In our Subversion repositories, we make use of Vendor Branches, a normal and very useful feature in Subversion that is used to manage "external" software. Vendor branches are however very difficult to migrate to Git, and only very little related documentation can be found on the internet.
Possible solutions are:

1. Git submodules,
   
2. Git subtrees,
   
3. normal Git branches.

Git submodules are mentioned relatively frequently, but they really do not seem to be a good fit for vendor branches. We don't consider them any further for the reasons detailed in the "Importing Subversion vendor-branches to Git" thread.

Git subtrees are looking very interesting and well suited to the problem, and I spent a lot of time digging into them. There is a subtree extension that is likely integrated into the Git core soon, and Jakub Suder describes a solution using it that we might have adopted (without the --squash).

Using normal Git branches as vendor branches is beautifully explained in this blog post by Dominic Mitchell. In fact, our website always was structured in "live" and "pristine" branches right from the start, and mapping these 1:1 to normal Git branches was straightforward and a clear choice (but still required the "Branches outside branches/" and "Missing Merges" facilities above).

For the vendor branches in Cafu, the matter was less clear: candidates were Git subtrees or again normal Git branches. As mentioned before, I posted a detailed and complete description of the problem in thread "Importing Subversion vendor-branches to Git".

Eventually, I opted for the "normal Git branches" approach (even though it required the "Move to Subdirectory" step from above), because it is the most simple, clearest approach that requires no "extras" at all, neither for the DVCS nor for its users, and as a side effect we keep the door open to a future migration to another VCS such as Mercurial.


Migration Details

I give the exact technical steps of converting the Cafu Subversion repository to Git in a comment to this post, in order to keep this prose text readable and clear.


The next steps

Our official Git repository of the Cafu Engine is now available at:


Naturally, we will not immediately abandon the Subversion repository, but enter a gradual transition period where everyone can make the switch at a comfortable pace, and where we can deal with details like the issue tracker.

Personally, for a short while I expect to continue working mainly with Subversion, updating the Git repository in a separate step.
Thereafter, I'll probably switch to work mainly with Git, but continue to update the Subversion repository in a separate step.
Only when all users and all technical indications clearly suggest that we can do entirely without Subversion, will the Subversion repository finally switch off.

Post and read comments about this news item at the related forum topic.

For the Cafu Engine project, we have almost from the very beginning used Subversion as our version control system, and Trac as the bug and issue tracker. Both systems have served us very well in the past, and in fact until today there is nothing fundamentally wrong with either of them.

However, over the years, the requirements of developers and users have evolved, as has the Cafu Engine and available version control systems. Increased flexibility and ease of work for everyone who participates in the Cafu Engine are more important than ever.

With DVCS – distributed version control systems – being widely spread and well known today, most prominently Mercurial and Git, and with related hosting and collaboration platforms such as Bitbucket and Github readily available to everyone, it's time for change.

In this post I will summarize the current state of affairs regarding the migration of Cafu to Mercurial or Git.


Why DVCS?

Why should we migrate from Subversion to Mercurial or Git at all?
What are the goals?

• First and foremost: Our users use the Cafu source code to develop their own games and applications, but would also like to integrate the latest enhancements and bug-fixes from the official Cafu code base. Keeping up-to-date by applying the latest changes of Cafu into your own code base can be cumbersome with Subversion, but is easy to do with Mercurial or Git.
  
  
• Similary, developing with and contributing to Cafu should be easy. Anyone who wants to contribute an enhancement, a bug-fix or a new feature should be able to do so in the simplest manner possible.
  
  This will later be described in greater detail in the wiki, but in brief, the primary method would be to create a pull request at Bitbucket. Fallbacks and alternatives would be posting pull requests at the issue tracker or the mailing-list, or to submit patches by uploading them to the issue tracker or sending them to the mailing-list.
  
  
• Better branching than Subversion. Branching is a vital tool to us even with Subversion.
  
  • We use a special flavor called "vendor branches" to manage our external libraries.
    
  • In our website, we use vendor branches as well, for separating "live" from "pristine" content.
    
  • Last but not least, the use of branches in Cafu as "feature branches" was existent, but underutilized. Employing a more sophisticated branching model like the one described here will only be possible with Mercurial or Git.
    
  
  
• Implicitly contained backups. As everyone always has a full copy of everything, the worries about creating backups for the repository are virtually eliminated. This is important for our actively used Cafu source code and website repositories, but also for legacy repositories that we no longer use, but still keep around.
  
  
• Repositories that are no longer in use and have been archived require no special measures to inspect, use or even revive them at any later time. No server is necessary to do research and archeology even in long archived repositories.

Mercurial or Git?

The internet is full of related debates already. As I do not want to start another one, I'll be brief:

In my opinion, Mercurial is easy to learn, well documented, and powerful enough for 99% of all use cases and users. For new projects with no pre-existing history, Mercurial is my clear favorite.

Git is technically more sophisticated, but comes with a horrible command-line interface. These days it is easier to learn than it used to be, but developing a proper mental model and becoming a confident (admin) user still takes a very big effort.

A fair and competent opinion about Mercurial and Git was expressed by Sinbad, the founder and former lead of the Orge3D project. I much agree with his points in this discussion. (Continue reading on the subsequent pages 13 and 14, where he also links to his detailed review and comparison.)

Besides studying all the existing, prevalent sources, during the last few weeks and months I also spent huge amounts of time with evaluating both Mercurial and Git myself: The minimum that I wanted to accomplish was gaining a good understanding of each system, import the existing Cafu Subversion repository, and run some real-live tests that resemble daily routine Cafu development.

My opinion about Git:

• Regarding the negative aspects, I have to agree with Steve Bennett's blog post "Ten things I hate about Git". And he forgot to mention that Git under Windows only exists due to a small group of volunteers who are not involved in Git core development.
• BUT: Git is widely supported by web hosters. All hosters that I have a contract with (LCube and 1und1) now support Git without any further ado (none of them supports Mercurial).
• AND: Git has with git-svn a very good interface to Subversion for both parallel use or one-time conversion.

Despite my original preference for Mercurial, I must say that I spent more time with Git (maybe because it took me three printed books to get a good understanding of it), so probably in the end I was more biased than I now like to admit.

As we will see later, the initial importing of the Subversion repository turned out to be the crux. After a long battle, it turned out that I was able to complete the conversion to Git in exactly the clean, complete and careful way that I wanted, whereas my attempts at converting to Mercurial were unfortunately much less fruitful. Things had probably gone better for Mercurial if I had invested more time into it, but then I have still no idea how I could have done with Mercurial what I did with Git.

With this experience, I eventually chose Git to proceed with the next steps.

However, I'm confident that importing the now clean Git repository into Mercurial is much simpler than the original attempt of importing the Subversion repository into Mercurial, so the door to Mercurial is still open.


The next steps

After all of the above, I was ready to ponder the concrete next steps that must be undertaken to actually migrate the Cafu Engine to Git.

But as this matter in all its details is very complicated and involved, I finish this post at the overview level.

In my next post, I'll provide more details on the requirements and goals of the conversion, and why it was so difficult to implement. Things will get more technical (and practical) then, and I'm going to provide the exact steps on how the conversion was eventually implemented. As the bug and issue tracker is closely integrated with Subversion, we'll have to have a look at it, too.


Please let me know what you think, I'm looking forward to your feedback!

Post and read comments about this news item at the related forum topic.

My recent work for improving the game and entity code turned out to have a large effect on the networking layer in the Cafu Engine as well: Moving the serialization and deserialization of entity state from the management code into the entities themselves is the first key step to achieve the desired improvements in code design, as it cleanly separates entity from network implementation details.

During my work in this regard, I found it highly useful to employ Wireshark for analyzing the network packets that a Cafu client and server actually exchange. In the past, when the Cafu network protocol was initially developed, we used Wireshark only for capturing the network traffic, optionally prefiltering it, and to save it to disk. Then we used our own, hand-written C++ program "ReadDump" for analyzing the resulting dump file.

However, it turned out that both maintaining and even using the "ReadDump" program was very inconvenient and in fact rather painful - and it took not long until it became outdated: not able to read the latest version of the Cafu network messages, and thus useless.

As a result, last week I dug myself into Wireshark again, and much to my joy found that they have a (relatively) new feature that allows to write dissectors for analyzing any given protocol in our favorite scripting language Lua.

As a result, we can have the Cafu Engine network protocol dissected and analyzed directly in Wireshark. Even though my first attempt at such a dissector is probably somewhat clumsy and certainly still incomplete, it already turned out to be a lot more convenient to use than our previous tool, leads to insights much faster, and as a result, is a lot more helpful.

Here is an example screenshot from one of my sessions:



If you would like to see the related Wireshark dissector itself, here is the script file (which also has additional links if you're interesting in further technical details):
http://trac.cafu.de/browser/cafu/trunk/ ... k-cafu.lua

In summary, a Cafu-specific dissector for Wireshark helped me a lot with making the changes to the network protocol that we need to advance our game code. I've finished many of these changes already, and will commit them to our code repository in the next few days.

Post and read comments about this news item at the related forum topic.

We're happy and proud to announce immediate availability of Cafu 12.05, the May 2012 binary release of the Cafu Engine!

Cafu is an open-source 3D game and graphics engine. The new version 12.05 culminates three years of exciting work into a new stable demo release: It marks the completion and first release of the new Cafu Model Editor, many structural enhancements for game developers, and the beginning of our work on the next version (which in fact already has begun — we just couldn't wait).


New Features

For game developers, we cleaned up the source code, revised the code design, improved the directory layout, added new C++ and scripting reference documentation, fixed many bugs, added support for new compiler versions and platform variants, speeded up the release cycle for source archives and development binary releases, and much more.

Several hundred changes to the source code were made, see the complete changelog for details.


The Model Editor

The highlight of the release is the new Cafu Model Editor and the new models program code. The Model Editor answers the question:


It can import many file formats and provides a graphical user interface for making Cafu-specific settings and adjustments to the imported model.

The related new model code implements advanced features like automatic resource sharing, skeletal animation, seamless blending of several animation sequences, animation channels, skins, level-of-detail, submodels, GUI fixtures, and a lot more.

For more information about the Model Editor, please see:



We'd like to thank everyone who participated in development of Cafu 12.05: programmers, documentation writers (a work in progress), and testers. We also thank our user community for the dedication and support — we needed it more than ever.

Now that this version is finally released, we are grasping the future with both hands: A new series of improvements has already been prepared, and we're now eager to implement them as quickly as possible!

Post and read comments about this news item at the related forum topic.