The open-source game engine and graphics engine for multiplayer, cross-platform, real-time 3D action

Cafu News RSS

Server state and client effects in online games

by Carsten Fuchs | Sunday, 2014-01-19
As explained in my previous news post, I am currently migrating the old DeathMatch game code to the new Component System architecture. Just this afternoon I finished dealing with the "hand grenade" entities in the new system, and I found that they're noteworthy in many regards:

  • Hand grenades are both dynamically created during the course of a game (when the player throws them) and dynamically removed again (after they have exploded and their traces have gone). This makes them different from many other entities that are defined in map files and often exist from the beginning to the end of the map.
  • Hand grenades employ more components than other entities in order to implement their functionality:
    • Model (the hand grenade model as thrown through the map),
    • Physics (for computing the flight path and dealing with collisions, e.g. with walls and the floor),
    • Script (for controlling the details, e.g. letting the detonation timer tick and implementing the explosion effects),
    • Light (for the flare when the hand grenade explodes),
    • Sound (for the sound at the time of the explosion),
    • ParticleSystem (for the smoke and spark particles at the explosion).
  • They very well demonstrate the distinction between Server State and Client Effects.

It is the last point that I would like to highlight in this post, because a good understanding of it is crucial for game development with Cafu:


Server State

The server state is characterized by the following properties:

  • It is synchronized via the game server to all clients,
  • every player (client) sees the same results and consequences,
  • server state is relevant for gameplay.

Typical examples are drawbridges, the position and state of elevators and doors, details about opponents, etc.
As a side note, events are often a result of a change in the server state, and transferred to the clients as such.


Client Effects

Client effects can be triggered by some event in the server state, but besides that, they:

  • run independently of each other on the clients only,
  • are often used for eye-candy,
  • it's ok if not everyone sees exactly the same (not relevant for gameplay).

Examples include particle effects such as smoke, sparks and small pieces of debris, many fade or transition effects, model animation frames, etc.


Having both

An important insight is that we sometimes need both server state and client effects combined.

For example, consider a light source that illuminates an otherwise pitch black part of a labyrinth. Whether the light source is on or off clearly makes an important difference for players who try to navigate the labyrinth, and so the state of the light source is managed as a part of the server state.

However, there are also other kinds of light sources, e.g. those that are on, but steadily flicker or pulse. This is often a means to create a specific atmosphere to a map, but the exact amplitude of the flicker or pulse is really not something that needs synchronization over the network.

In fact, as a combination we may wish that the light source's on or off state is synchronized to everyone, and if it is on, have it have a client-side flicker effect for eye-candy.

This is exactly what we can (starting with commit d11c6a9) achieve with the light source components, and what I now employ with the hand grenades: When a hand grenade explodes, the related light source is turned on in order to mimic the bright flare of the explosion. However, the exact details of how the bright light fades, changes its color from white over yellow to red, shrinks and eventually disappears (all in about half a second), needs not be synchronized over the network.

Instead, a very simple script method is used to implement this. Here is its entire code:

Code:
local clTime = 0.0
local Duration = 0.5

function Light:ClientEffect(t)
    if not self:get("On") then
        clTime = 0.0
        return false
    end

    clTime = clTime + t

    if clTime >= Duration then
        return true, 0, 0, 0, 0
    end

    local Amount = 1.0 - clTime/Duration

    return true, Amount, Amount*Amount, Amount*Amount*Amount, 400.0
end

Well, this is one of the reasons why I love the new component system! :loveit:

Post and read comments about this news item at the related forum topic.

More about Component Systems

by Carsten Fuchs | Thursday, 2014-01-02
When I first posted about Component Systems exactly one year ago, things were still at the very beginning. Since then, the efforts to introduce Component Systems to the GUI window hierarchies of 2D and 3D GUIs, and to the game entity hierarchies of game maps made very good progress. In fact, I was so busy with and dug into my work, that I didn't get down to actually talk about them. :oops:
The good news is that it is now perfectly clear that the new Component Systems are a very important step for the future of the Cafu Engine. They give joy and pleasure every day that I work with them.

So I thought that this was a very good time to briefly reflect on what has happened during the past year, and to highlight a few related aspects in greater detail.


The problems with class hierarchies

Before it sinks into oblivion for good, I would like to summarize where we came from and what the problems were, as this helps to understand what the Component Systems actually achieve.

entity-class-hierarchy.png
Image Detail Image Download
An excerpt of the old entity class hierarchy.

Until about a year ago, we used to use "classic" C++ class hierarchies both for GUI windows as well as for game entities. You can see an excerpt of the entity class hierarchy as it used to be found in the (now revised) C++ API documentation in the image to the right.

At the time that it was written, even though we were aware that inheritance is not a means for code reuse, we thought that a class hierarchy was a good idea, because it naturally seemed to group entities that have related features. And even though the implementation actually worked in the beginning, it soon turned out that there were plenty of shortcomings.

For example, at some point we wanted to augment entities to play 3D sounds, complete with Doppler effects and settings to control the volume, emission cone, and several other details. Our natural approach was to derive a new EntSoundT entity class from the BaseEntityT class, and have it implement all the features that we wanted. However, it soon turned out that we not only wanted dedicated EntSoundT entities to play sounds, but many other entities (of different entity classes) as well. For example, an item that is picked up by the player and respawns a few seconds later makes a sound at both occasions. So we re-implemented the sound functionality in the item's entity code as well. And soon in the code for monsters, for human players, and several others.

It took me a while to realize it, but when I did, the result was pretty shocking: Even though the Cafu Sound System allows for minimal code in the game entities, not only were we left with several pieces of code that all essentially did the same, scattered across many entity classes. It was also very cumbersome to properly document all this (in fact we never did), to keep it in sync with the Map Editor and tools, and to deal with any special cases that occurred.

You can find another account on the the principal problems with class hierarchies in the blog post Cowboy Programming: Evolve Your Hierarchy by Mick West.

In hindsight, I'd say that many of these shortcomings were so severe that they effectively hindered or even halted the further development of the Cafu Engine. The old code just didn't scale, as adding new features caused the combinatorial complexity to grow worse and worse. Also, the code in the Map Editor, the map compile tools and the Cafu Engine was not the same, and keeping it in sync turned more and more into a pain. Not to speak of the special-cases that were needed here and there, and which we never managed to totally avoid.

These issues also burdened game developers with unnecessary complexity, e.g. the requirement to master C++ in order to make even small changes or to add small features, a burden which many game developers understandably did not want to concern themselves with.


Two class hierarchies, two Component Systems

game-entities.jpg
Image Detail Image Download
An example of game entities.


TeleporterGUI.jpg
Image Detail Image Download
An example of GUI windows.

Before we proceed, be aware that in Cafu we have two class hierarchies that are important for game development:

  1. game entities
  2. GUI windows

Game entities are the objects that populate our game worlds, such as the human player, monsters, weapons, items, ...

GUI windows are the basic rectangles of which our GUIs are built. GUIs are used both in 2D (for the Main Menu, the in-game console, etc.), as well as in 3D, where they are a part of the game worlds and used e.g. as lift controls, computer terminals, and so on.

These two systems are designed to work together, but regarding code design, they're largely independent of each other. By coincidence however, we find that both are structurally very similar to each other and share the same ideas of hierarchical setup and organization of concrete instances.

So both systems suffer the same problems as described above, and both happen to be candidates for migration to a Component System. In fact, as the GUI windows hierarchy was so much smaller than the game entities hierarchy, I started the initial tests and the actual migration with the GUI windows. This went very well, and now all the efforts are focused on the game entities. Read on for details.

Generally, all that is said above and below holds for both game entities and GUI windows, although I may often talk about and provide examples for only one, but not the other.


The new design

After careful research and planning, I found that the new design can and should be as simple as this:

Entities → Components → Variables

Let's look at each of these elements in greater detail:

Entities

All entity instances are now of the same common class: EntityT. No inheritance, no class hierarchy. Essentially, entities are now nothing but lists (or "containers") of components.

A new aspect however is that entities can now have children, or "sub-entities". In fact, entity instances (not classes, don't get confused!) can now be hierarchically organized.

For example, consider a car that consist of a chassis and four wheels, or a complex lift system that consist of a moving platform and a set of moving doors at each floor. It is now possible to have an EntityT instance that represents the "whole thing", the entire car or the entire lift, possibly with a set of components that define features for the whole, and to have child entities that represent the parts, where each part is another EntityT instance that can contain components of its own.

The same is found (and possibly feels a bit more natural) in GUI window hierarchies: We often have a window that acts as a dialog frame, and child windows of it are used for adding details like the dialog message, text input fields and the "OK" and "Cancel" buttons.

Components

Each component implements exactly one feature, and each feature is implemented in exactly one component.

For game entities, we have (at the time of this writing) these components:

  • Basics (name, status),
  • Transform (origin, orientation),
  • Model,
  • Collision Model,
  • Point Light,
  • Radiosity Light,
  • Particle System,
  • Player Physics,
  • Script (custom scripted behavior),
  • Sound.

For GUI windows, we have these components:

  • Basics (name, visibility),
  • Transform (position, size, rotation),
  • Border (frame),
  • Text,
  • TextEdit,
  • Image, Model,
  • Choice, ListBox.

This may not look like much, but even though we're not finished yet (the lists will still get a bit longer), all previously existing functionality can be re-implemented in terms of these components. I take this as a sign that the key decision and design are powerful and correct. :up:

Generally, components are similar to a code design pattern: They naturally group things together that belong together, and encapsulate code and features in a manner that is concise and clear. Above all, components eliminate the combinatorial complexity that occurs with adding features in classic class hierarchies. As we will see in the next section, components also break up other old weaknesses and bring very worthwhile improvements.

Variables

Each component has a list of variables. This may sound like a note of trivial notability, but in fact I came up with a very nice scheme from which our Component Systems draw much of their power: Entities can iterate over their components, and components can iterate over their variables. On this basis, it was possible to implement features such as:

  • automatic dialogs in the Map and GUI Editors (see example below),
  • automatic serialization to and from disk,
  • automatic serialization and synchronization over the network,
  • automatic generation of documentation,
  • automatic binding to Lua (all variables are accessible by script),
  • automatic ability to interpolate all variables (of type float, double, or composites thereof).

In other words: Adding a new variable to a component imposes no extra work at all to get all of the above!
Isn't that absolutely rocking cool?? :cheesy:

These virtues were achieved by employing the "Visitor" code design pattern for implementing the variables (which, when you look at the source code, are actually small classes of their own).

Note that the motto "everything can be interpolated" is twofold: One aspect is "real" interpolation, where for example we move a platform from A to B, without concerning us with the intermediate positions along the track. Another aspect is the interpolation on the client, where we interpolate over client frames to fill the larger gaps between server frames, achieving a movement that is as smooth as possible, and independent of network latency that possibly affects the arrival of server updates.


The Component System for GUI Windows

WindowInspector_250px.png
The components of a window in a GUI.

As mentioned above, I started with the Component System for GUI windows. The goal was to re-implement the then existing functionality (and optionally to improve and expand it) in terms of the new Component System. This goal has long been reached, and it was a huge success (see ticket #125 for details).

When I was done in March 2013, there was only one problem left: I was eager to get started doing the same with the game entities. So eager in fact, that I didn't want to waste time with anything else, such as expanding the GUI system (adding new components was so easy then that it was very tempting to add ton of new features!), or mundane tasks like writing proper documentation. (The new reference documentation was of course a piece of cake, you can see it here. I'm talking about the introductory documentation in the Wiki, with screenshots and well-phrased prose texts...) Well, I'm sorry to say that these things are still not done, and unless someone is willing to help, are likely to stay that way at least until I'm done and happy with the Component System for game entities. ;-)


More benefits

Component Systems remove the complexity from combining features, providing both more flexibility and more capabilities to game development.

But there is more: With our implementation of Component Systems, we can finally use the same code everywhere: from the Map Editor over the map compile tools to the Cafu Engine, they all use the same code now! For example, in the Map Editor, we must support editor-specific features such as "selections" (objects clicked with the mouse for further editing). But now, such support is only a matter of adding another Component to the selected entity! That's all. All other code is the same as that in the map compile tools and in the Cafu Engine.

A similar aspect allows for unified script states. Previously, script states had a tendency to get fragmented, that is, it could be difficult to access the state of the game world from the script code of an embedded 3D GUI. My implementation of Component Systems unifies script states as well.

Important for many game developers, there is no longer a need to write C++ code (if you don't want to). There is full flexibility and full power available with scripts. (And those who love C++ get their money's worth as well – hacking Cafu in C++ is more fun than ever!)

And finally, we're able to create very fine scripting reference documentation mostly automatically. As indicated above, we can iterate over components and their variables. This allows to generate complete Doxygen documentation automatically. To improve this even further, it is of course also possible to augment the generated documentation by hand.

The strings that are used to document the components and their variables are available in CaWE (e.g. the Map Editor) as well! That means that the documentation is not only available as a nice online web resource, but is readily available in CaWE, too! You can see an example of this in the "Window Inspector" screenshot at the bottom ("hor. Align"). It is even possible to query the help string for a variable at the in-game console in Cafu, or from any other script code, if you wish.


References

For historical reference, here are some related forum posts where Component Systems were previously discussed:


Many thanks to all those who patiently kept explaining things to me (and in the beginning kept convincing me ;-) ) and who participated in discussions and provided suggestions!


Ongoing work

fireworks_2014.jpg
Image Detail Image Download
Fireworks at my home town, 2014-01-01.

In summary, the new Component Systems provide elegant solutions to old problems. They make game development much easier and much faster, a lot more flexible, and more fun than ever.

However, we're not done yet: the Component System for game entities has made excellent progress, but there is still work to do with porting old DeathMatch game code to the new system. There is also a lot of documentation to write (those introductory and prose parts that don't automatically write themselves...), and many pages of the website to update (Features, Gallery, ...). And once that's done, when the new Component Systems are in place and all obsolete code is gone, a new release is due, and the real fun begins: We can then add new features that we couldn't add before, add new scripts that we didn't have before, and so much more.

If you're interested in the details, have a look at the Cafu source code repository, where currently the active work is in the entity-component-system branch (which will be merged back to master once done).


I wish everyone a Happy New Year and all the best for 2014! :wohow:
Carsten

Post and read comments about this news item at the related forum topic.

Using the Cafu Wireshark dissector

by Carsten Fuchs | Wednesday, 2013-07-24
In a recent News announcement I presented a dissector for Wireshark that one can use to study and analyze the Cafu Engine network protocol. (I currently use it to aid the development of the Cafu Component Systems, about which I'll post more information soon.)

However, what I haven't but should have done at this time, is add some information on the key steps to actually use that dissector: As I found out at the beginning at this week, the necessary setup is not particularly complicated once you know how to do it, but if you still don't, it can take a while to search the web for all required information:

  1. The first step after you've installed Wireshark is to "install" the Cafu Wireshark dissector as well. This is easily done by copying the dissector file into the user's Wireshark plugins directory. For example, from the Cafu repository's root directory on my Windows 7 system, this command does the job:
    Code:
    copy Ca3DE\wireshark-cafu.lua c:\Users\Carsten\AppData\Roaming\Wireshark\plugins\
    Make sure to manually create the plugins subdirectory beforehand, if necessary.

  2. Under Windows, Wireshark cannot capture packets from the loopback device directly, which is required in order to capture the network traffic in single player sessions on the local machine. Instead, we have to use the RawCap.exe program from http://www.netresec.com/?page=RawCap. When RawCap.exe is started, it will capture all packets from the loopback device into a dump file. While RawCap is running, we also run the Cafu Engine in order to record the network traffic. When you're done with Cafu, also quit RawCap with Ctrl+C, then start Wireshark to analyze the file that has been written by RawCap.

  3. In Wireshark, a useful filter expression for reducing the view to Cafu Engine related network packets only is
    Code:
    udp.port == 30000 or udp.port == 33000

Post and read comments about this news item at the related forum topic.

Happy New Year!

by Carsten Fuchs | Wednesday, 2013-01-02
I wish you all a good and happy new year 2013 and all the best wishes!

:groupwave1: :groupwave2:


As things have been very exciting for those following our development branches in Git, but outwardly a bit calm for those who don't, let me take the opportunity for a brief review of recent developments, and an outlook on upcoming changes:

Better game development

Early in the second half of 2012, we have started to focus our efforts on improving the experience for game programmers, that is, for those nice folks who are responsible for the programming part in the development of a new game that uses the Cafu Engine.
In the past, our game code has grown as the features of the Cafu Engine grew, and eventually we had to realize that customizing the DeathMatch example code for other needs, or even adding new features to it, was more complicated than it should be.

As a result, we started to fix the situation:
Improvements to and a large redesign of our game/entity code was a very important first step that brought a great relief almost immediately. The related ticket #113 is still not closed however -- we have plans for further changes.

In parallel, we found that the time was ripe for switching our version control system from Subversion to Git.
The switch turned out to consume a lot of time and efforts, but as I've pointed out in my migration protocols (Part 1 and Part 2), the benefits especially for our users are so great that it was all worth it -- and such improvements were on our agenda anyways.

Component Systems

And finally, in the midst of all this, our new forum member midix suggested the introduction of "entity systems" or "component systems" to the Cafu Engine. I have to admit that I was at first highly skeptical to the idea, which essentially is a different way to organize and manage all the features that game entities have:

Previously, the classic class inheritance hierarchy of game entities was the main place and instrument to organize and implement the features of the game entities. It turned out however (not only for Cafu, but also for many other game engines as well), that such class hierarchies are very inflexible when it comes to reusing or recombining features in circumstances that are even slightly different than those originally foreseen.

With "component systems", the proposal is to remove the entire game entity inheritance hierarchy, except for the root class, and instead to implement features as "components": Each component implements exactly one feature, and each feature is implemented in exactly one component. A game entity is then provided with the list of components (or features) that is should have. Of course, each component can be configured by the user: conveniently with a nice user interface in the graphical map editor, or with powerful scripting commands, or an arbitrary mix of both.

As I said, I was very skepical about it, but I could also see how a component system would improve our situation and help our mission to make game development and programming with Cafu so much easier and more flexible. And then I realized that we not only have game entity hierarchies in Cafu, but also GUI window hierarchies in our 2D and 3D GUIs, and I could see an analogy in both: The GUI windows were a much smaller hierarchy than our existing hierarchy of game entities, but they too would much benefit from the perspectives of a component system -- and as the hierarchy was so much smaller, they were the perfect test case!

So my plan was to try out and implement a component system for our GUI windows first, gather experience, then evaluate the results. If everything worked well and our expectations were met, only then implement the component system for the game entities as well.

Well, today, after only a relatively short while of working on it, and while still being in the middle of implementing the component system for GUI windows, I can already say that I'm very very pleased with the intermediate results. Things look highly promising already, and 95% of the efforts that I've spent so far will immediately be useful again when we decide to extend the component system approach to game entities as well! It's really difficult to explain that all in proper thoroughness right now, bit if you're interested in the details, check out the gui-component-system branch in our Git repository.

In summary, I'm very happy with the current perspective, and very much looking forward to the new year!
:wohow:

Post and read comments about this news item at the related forum topic.

Documentation updates

by Carsten Fuchs | Wednesday, 2012-11-14
Motivated by user feedback and recent developments, we've updated several pages of our website and documentation.
In this post, I'll briefly summarize the most important changes:


Any feedback that you may have is very much appreciated. If you would like to help, editing or contributing to the documentation Wiki is easy!

:book:

Post and read comments about this news item at the related forum topic.

Migrating Cafu to distributed version control – Part 2

by Carsten Fuchs | Thursday, 2012-10-18
In "Migrating Cafu to distributed version control – Part 1", I outlined the fundamental considerations for the migration. This post continues the subject with the more specific and technical details.


Goals and Requirements

Specifically for Cafu, what are the goals and requirements when converting to Git?

As a first step, our source code repository must technically be converted from Subversion to Git. I wanted the conversion to be done in a careful, accurate and complete manner: It should include all branches, vendor branches, tags and authors, and the proper and complete merge history. In fact, I wanted the resulting Git repository to look as if we had used Git right from the start.

Conversions like this are generally well described in the documentation and books about Git, but it turned out that our use of Vendor Branches, a normal and useful feature in Subversion, was very difficult to migrate to Git, and that only little related documentation can be found on the internet. Vendor branches are important to us because we use them to manage our external libraries.

Besides my normal work, I spent a lot of time on this, on and off for several months, slowly putting the related pieces together. I also described the problem on the git-users mailing list in thread "Importing Subversion vendor-branches to Git". The thread provides both a good technical summary as well as the remaining bits that I was still missing at that time. This post is the synopsis of the gathered results.

Secondly, as the issue tracker is closely related to the repository, it needs to be updated accordingly. Options include:

  1. stay with Trac (but update it to work with Git),
  2. migrate it to the issue tracker provided with the Cafu repository at BitBucket,
  3. migrate it to Atlassian JIRA.

The first option would preserve the full existing flexibility of Trac, keeping us a certain degree of independence, and not require getting used to something else.
The second option would probably be the least complex and the most comfortable, but it remains to be determined if the BitBucket issue tracker is powerful enough for our needs. You can see a BitBucket issue tracker live at the BitBucket site itself.
The third option would be the most powerful, but it might as well overwhelm us.

I've not yet formed an opinion about it though, much less a decision. Fortunately, the migration of the issue tracker can be done largely independent from the migration of the repository, so that its progress does not stall the progress of the repository.


Migration Hotspots

While the bulk of the conversion is flawlessly and quickly done by the git svn ... commands, I found plenty of occasions where manual tweaking of the process, or post-processing and clean-up work was necessary in order to achieve the desired result. This is especially true whenever the Subversion source repository deviates from the classic "trunk, branches, tags" layout, or subtleties of Subversion merges prevent proper automatic conversion to Git.

In this section, I list the issues that I found the most prominent (from a Git learners perspective), along with the solutions that I eventually applied.

Branches outside branches/

If some branches are according to Subversion repository standard layout in branches/, but more branches are elsewhere, or if standard layout was never used and the branches are arbitrarily scattered across the Subversion repository, it is not immediately clear if and how these extra branches can be accounted for so that they are properly imported into Git. The solution is to split the call to git svn clone into this sequence:
Code:
> git svn init https://srv7.svn-repos.de/dev123/projects/cafu -s Cafu
> cd Cafu
> git config svn.authorsfile ../authors.txt
> git config --add svn-remote.svn.fetch "vendor:refs/remotes/vendor"
> git svn fetch
The next to last line causes the subsequent fetch to load the directory vendor/ as a Git branch, as if it was another branch in branches/.

Missing Merges

The converted commit history sometimes misses merges where merges were performed in Subversion:
Code:
    ------B-----D---- master
         /
    ----A-----C------ pristine
A was merged into master, yielding B, and the merge is properly reproduced in Git.
C was merged into master as well, yielding D, but only in Subversion. In Git, the merge is missing.

Among other reasons, this can happen if in Subversion the merge was performed not at the top directory level, but as a "partial" merge from subdirectory to subdirectory, e.g. from forum/themes/firenzie in pristine directly to the same directory in master.

The solution for such cases is to use the .git/info/grafts file, and to "fix" its results with
Code:
> git filter-branch --tag-name-filter cat -- --all
The --tag-name-filter cat part makes sure that attached tags are rewritten as well.
If rewriting the commits succeeded and the result is as desired, the grafts file and the references to the original commits should be deleted:
Code:
> rm .git/info/grafts
> rm -rf .git/refs/original/
The next call to git svn fetch will automatically rebuild the rev_map that is needed for continued bidirectional communication with the source Subversion repository.

Fixing Tags

As Subversion treats tags exactly like branches, after the conversion to Git the Git branches that should be tags must be fixed manually. A good solution is described by Haenel and Plentz in their book, but it unfortunately only works with lightweight tags and thus cannot account for the tag message, which in our case is a longer text. The best solution that I have found that works with annotated tags is from this Atlassian blog post, to which I however had to make small modifications to work as desired:
Code:
> type convert_tags.sh
#!/bin/sh
# CF: from http://blogs.atlassian.com/2012/01/moving-confluence-from-subversion-to-git/ with small modifications.
# Based on https://github.com/haarg/convert-git-dbic
set -u
set -e

git for-each-ref --format='%(refname)' refs/remotes/tags/* | while read r; do
tag=${r#refs/remotes/tags/}
# CF: Note the ^ in the next line: We create the converted tag at the *parent* of the original tag.
sha1=$(git rev-parse "$r^")

commiterName="$(git show -s --pretty='format:%an' "$r")"
commiterEmail="$(git show -s --pretty='format:%ae' "$r")"
commitDate="$(git show -s --pretty='format:%ad' "$r")"
# Print the raw commit body (commit message).
git show -s --pretty='format:%B' "$r" | \
env GIT_COMMITTER_EMAIL="$commiterEmail" GIT_COMMITTER_DATE="$commitDate" GIT_COMMITTER_NAME="$commiterName" \
git tag -a -F - "$tag" "$sha1"
echo "Tag: ${tag} sha1: ${sha1} using '${commiterName}', '${commiterEmail}' on '${commitDate}'"

# Remove the svn/tags/* ref
git update-ref -d "$r"
done

Move to Subdirectory

Before I could fix missing merges in our (partially) converted Cafu repository, I had to move the contents of all commits in the "vendor" branch into a subdirectory. The documentation for git filter-branch has a related example, which I modified according to this discussion, yielding:
Code:
> git filter-branch --index-filter '
      rm -f "$GIT_INDEX_FILE"
      git read-tree --prefix=ExtLibs/ "$GIT_COMMIT"
  ' refs/heads/vendor

Vendor Branches

In our Subversion repositories, we make use of Vendor Branches, a normal and very useful feature in Subversion that is used to manage "external" software. Vendor branches are however very difficult to migrate to Git, and only very little related documentation can be found on the internet.
Possible solutions are:

  1. Git submodules,
  2. Git subtrees,
  3. normal Git branches.

Git submodules are mentioned relatively frequently, but they really do not seem to be a good fit for vendor branches. We don't consider them any further for the reasons detailed in the "Importing Subversion vendor-branches to Git" thread.

Git subtrees are looking very interesting and well suited to the problem, and I spent a lot of time digging into them. There is a subtree extension that is likely integrated into the Git core soon, and Jakub Suder describes a solution using it that we might have adopted (without the --squash).

Using normal Git branches as vendor branches is beautifully explained in this blog post by Dominic Mitchell. In fact, our website always was structured in "live" and "pristine" branches right from the start, and mapping these 1:1 to normal Git branches was straightforward and a clear choice (but still required the "Branches outside branches/" and "Missing Merges" facilities above).

For the vendor branches in Cafu, the matter was less clear: candidates were Git subtrees or again normal Git branches. As mentioned before, I posted a detailed and complete description of the problem in thread "Importing Subversion vendor-branches to Git".

Eventually, I opted for the "normal Git branches" approach (even though it required the "Move to Subdirectory" step from above), because it is the most simple, clearest approach that requires no "extras" at all, neither for the DVCS nor for its users, and as a side effect we keep the door open to a future migration to another VCS such as Mercurial.


Migration Details

I give the exact technical steps of converting the Cafu Subversion repository to Git in a comment to this post, in order to keep this prose text readable and clear.


The next steps

Our official Git repository of the Cafu Engine is now available at:


Naturally, we will not immediately abandon the Subversion repository, but enter a gradual transition period where everyone can make the switch at a comfortable pace, and where we can deal with details like the issue tracker.

Personally, for a short while I expect to continue working mainly with Subversion, updating the Git repository in a separate step.
Thereafter, I'll probably switch to work mainly with Git, but continue to update the Subversion repository in a separate step.
Only when all users and all technical indications clearly suggest that we can do entirely without Subversion, will the Subversion repository finally switch off.

:up:

Post and read comments about this news item at the related forum topic.