For any application that has real users and more than one version, sooner or later the question of data versioning and migration arises.
In this article we will analyze the problem and its solution in the form of an open source plugin and the difficulties that we had to face in the process!
TL;DR
about the author
My name is Alexey and I am a Lead developer. And for the last six months I’ve been helping Zeptolab improve the Overcrowded project.
I’m self-taught. I studied everything related to game development on my own. All my knowledge is personal experience. Worked on 5 mobile projects in different genres (mid-core, simulator, merge-3). He also did several projects in augmented reality.
After spending a lot of time answering questions in unity chats on Telegram, I realized that the topic of architecture is very poorly covered. And strange best practices from unity often have nothing in common with real projects.
That’s why I made my own blog on Telegram, where I write about the architecture of unity projects, join us!
Problem
Let’s imagine a situation where you are working on a new, ideal game. It already has clans and caravans, but this is just the beginning and you are working on adding new content.
Several sprints pass, your feature is merged into the development branch and is about to be released.
And at the testing stage it turns out that old users cannot launch the game.
You receive a task with a description:
Steps:
1. Download version 1.0.0
2. Launch the game, wait for the tutorial to start
3. Close the game, install version 2.0.0
4. Launch the gameExpected Behavior:
– The game starts, a window with training opensActual behavior:
– The game freezes, there is a JsonSerializationException error in the consoleAdd. information:
stack trace errors
After a detailed analysis of the error, you remember how, while writing the feature, you changed the type of field in the class responsible for storing the player’s profile, namely:
Was:
public class PlayerData
{
public long Money;
}
Became:
public class PlayerData
{
public Disctionary<Currency, long> Money;
}
Great change! Now we can add as many types of currencies as we want to the game and everything will work great.
And indeed, the change is good, it’s hard to argue with it. But it just broke the profile’s backward compatibility.
Or in other words, compatibility with previous versions that already exist.
Those. The deserializer simply cannot convert the schema from:
{
"Money": integer
}
To object:
{
"Money": object
}
Having understood what the problem is, you immediately formulate a solution for yourself:
-
We need to take user data in the old format and convert it to a new format that is compatible with version 2.0.0
And move the bug to the ToDo column.
Solution
Iteration 1
After delving into the details, an action plan is formed:
-
Get user data (from a database or persistent storage)
-
Deserialize old format
-
Convert to new
-
Save user data in a new format
And then you realize that there is a problem:
-
To do this you need to create a copy of the class
PlayerData
which will have a different field typeMoney
And the solution will look something like this:
public enum Currency
{
Soft,
Hard
}
public class PlayerDataV1
{
public long Money;
}
public class PlayerDataV2
{
public Dictionary<Currency, long> Money;
}
var newDataInstance = new PlayerDataV2();
var oldRawJson = File.ReadAllText(path);
var oldData = JsonConvert.DeserializeObject<PlayerDataV1>(oldRawJson);
newDataInstance.Money[Currency.Soft] = oldData.Money;
var newData = JsonConvert.SerializeObject<PlayerDataV2>(newDataInstance);
File.WriteAllText(path, newData);
Well, problem solved. You can send it for review and take on the next bug for work.
Solution Analysis
The solution above may fix the bug, but it does it extremely ineffectively:
-
We were forced to create a duplicate class. Which in real projects can have any number of fields inside it.
-
This solution is a temporary crutch, because… for version 3.0.0 you will have to repeat the scheme.
Which, at most, can result in blocking the main thread for more than 5 seconds and you will get ANR
Having received these comments from your teammates or thought of these problems yourself, you decide to do something differently.
Iteration 2
After analysis, you decide to correct all the problems described above, formulating criteria for yourself:
-
Profile classes with player data should not be duplicated in the code
-
The solution should be reusable for future versions
To satisfy these criteria you need:
-
Find a solution to save data without causing an exception
And then it comes to mind to use different names for different versions:
-
Instead of using the name in the new version
Money
useCurrencies
-
And if
Money
greater than zero, add value to new type and resetMoney
-
And the field
Money
can be marked asObsolete
so that other developers don’t use it anymore.
Repeat ad infinitum for each backward compatibility break.
And the solution itself looks like this:
public enum Currency
{
Soft,
Hard
}
public class PlayerData
{
[Obsolete("Больше не используется. Оставить для обратной совместимости с версией 1.0.0. См. так же баг: AB-1234")]
public long Money;
public Dictionary<Currency, long> Currencies;
}
var rawJson = File.ReadAllText(path);
var playerData = JsonConvert.DeserializeObject<PlayerData>(rawJson);
if (playerData.Money > 0)
{
playerData.Currencies[Currency.Soft] = playerData.Money;
playerData.Money = 0;
}
Solution Analysis
Already much better, much more compact, without class duplicators, up to ANR like the moon, but:
-
Old fields will remain in the file forever and will participate in serialization/deserialization every time
-
Maintenance overhead
Obsolete
attribute
Over time, the number of fields marked with the attributeObsolete
will grow several times and you will need to check every time that no one accidentally writes anything into these fields or does not use it somewhere in the project.
In general, we go to iteration 3
Observation from experience
On 3 projects that I did not start, this problem was solved by a mixture of the first and second solutions. What surprised me more and more from project to project.
Let me know in the comments how you did it on your project.
Iteration 3, final
No duplicates, classes, fields or Obsolete
attributes – we need a different approach that meets the criteria:
-
Profile classes with player data should not be duplicated in the code
-
The solution should be reusable for future versions
-
Data conversion (migration) from the old version to the new one must be unified.
One single, clear way to write and support migrations -
The solution should create a minimum of overhead costs
Well, to satisfy these criteria we:
-
Must isolate the feature and provide a common mechanism, optimizing all overhead
Easy, let’s go!
Open Source plugin
Search for analogues
Probably, before you fence your crutches, it’s worth looking for ready-made solutions.
Finding the plugin Migrations.Json.Net. Everything is great, but there are problems:
-
It is not clear whether the plugin works on Unity and is compatible with IL2CPP
-
The solution looks very cool, but the implementation is lame
For such a simple solution there is a lot of code with a bunch of LINQ expressions -
The solution has already been abandoned and the maintainer does not respond to open Issues at all.
Here I am, back in 2021, answeringthat this plugin works correctly in Unity.
There are other options, but they are not as popular.
Technical requirements
Product criteria, if we can call it that, we formulated above, now technical ones:
-
The solution must be compatible not only with unity, but also with other versions of dotnet
In my case I’ll stick with dotnet standard 2.0 -
The solution should be ready for use in production without modifications
Think through and cover all the logic of the migrator with tests -
The solution should not limit possibilities
Newtonsoft.Net.Json
Those. use of methodsPopulate
settings PreserveReferencesHandling, ObjectCreationHandling.Replace and attribute JsonConstructormust be saved -
The solution should be thread-safe
-
The solution should be easily accessible for downloading and integration into the project
Those. published to Nuget, openUPM and release should have unitypackage to install directly -
The solution must have an automated version compatibility check
Tests must be run both in dotnet and in all LTS versions of unity, starting from 2019.4 -
The solution must have clear, concise documentation.
In addition to xml-doc for all public classes and methods, the readme is also well-designed -
The new solution must be objectively, through measurements, better than the analogue
To do this, you need to write a benchmark and attach the resulting numbers to the readme
Implementation
👉Link to the final version of the plugin on GitHub👈
Click on ⭐️ so as not to lose!
Version 1.0.3 is production ready, you can safely implement it into your projects!
After collecting all the requirements and creating tasks in Projects (this is a simple Kanban board built directly into GitHub), I started implementation.
In total, I wrote all the logic for about 15-17 hours. Moreover, I did this in real time, streaming on YouTube.
👉Playlist link👈
ps at 2x speed looks like a breeze!
As a result, the final solution looks like this:
-
The plugin user needs to mark the migrated class/structure with the attribute
Migratable
specifying the current version in the constructor.
The version starts from 0, i.e. by default we can assume that all classes have version 0.
And from version 1, we will need to implement migration methods. -
Migration methods are methods with a specific signature that will be called automatically by the plugin through reflection
Signature:private/protected static JObject Migrate_X(JObject data)
Where X is the version number to which we are migratingJObject
-
The migrator himself is implemented as an heir JsonConverterwhich:
-
When deserializing, calls the migration methods in order, starting with the version specified in the json file
-
When serializing, it takes the version from the attribute and writes it to a file
There is no need to register an additional field in the class
-
-
The migrator needs to be set to default settings
JsonConvert.DefaultSettings
or add manually during serialization/deserialization -
If your json file version is 3, and the current version of the class is 10, then the migrator itself will call methods 4 to 10.
That is, it will update the json file format from 4 to 10.
And in code it looks like this:
public enum Currency
{
Soft,
Hard
}
[Migratable(1)]
public class PlayerData
{
public Dictionary<Currency, int> Wallet;
private static JObject Migrate_1(JObject rawJson)
{
var oldSoftToken = rawJson["soft"];
var oldHardToken = rawJson["hard"];
var oldSoftValue = oldSoftToken.ToObject<int>();
var oldHardValue = oldHardToken.ToObject<int>();
var newWallet = new Dictionary<Currency, int>
{
{Currency.Soft, oldSoftValue},
{Currency.Hard, oldHardValue}
};
rawJson.Remove("soft");
rawJson.Remove("hard");
rawJson.Add("Wallet", JToken.FromObject(newWallet));
return rawJson;
}
}
var jsonString = @"{
""soft"": 100,
""hard"": 10
}";
var migrator = new FastMigrationsConverterMock(MigratorMissingMethodHandling.ThrowException);
// Для десериализации
var deserializeResult = JsonConvert.DeserializeObject<PlayerData>(jsonString, migrator);
// Для сериализации
var serializeResult = JsonConvert.SerializeObject(deserializeResult, migrator);
// serializeResult: {"Wallet":{"Soft":100,"Hard":10},"JsonVersion":1}
Solution Analysis
-
Calling methods through reflection may not be obvious to the user
-
The plugin forces the use of methods with a specific signature and access modifiers
-
A migration error due to the absence of a method with the required signature will only be detected at runtime
-
Performance figures leave much to be desired due to usage
JObject.Load
and reflection -
Using
MigratorMissingMethodHandling.Ignore
every missing method will be searched on the object, which can lead to a large performance hit
List of possible improvements
-
Add a Roslyn analyzer that will generate a compilation error if the version has changed and a method with the required signature is not implemented. Issue
-
Add Roslyn hint to automatically generate the required method. Issue
-
Cache all methods with signature
Migrate(JObject)
on the first call. Issue
I will be glad for any contribution!
conclusions
What I personally understood about this type of work:
-
You need to have endurance and self-discipline
Honestly, if I hadn’t promised and started all this streaming activity, it would have been difficult for me to maintain a sufficient level of motivation to finish working on the plugin -
High-quality packaging of an open source plugin takes about 50% of the implementation time.
I understand that the plugin is not big, only 400 lines of code, but I clearly underestimated how much time it would take for design, CI/CD and documentation. -
Code discipline, commits, architecture, even for small projects, are important.
Otherwise, to make a normal CI/CD without pain, you will have to shovel half the project.
Subscribe to my Telegram channelthere I write about the architecture of unity projects.
And I often subscribe to movements, which then result in Open Source projects
Acknowledgement and Usage Notice
The editorial team at TechBurst Magazine acknowledges the invaluable contribution of the author of the original article that forms the foundation of our publication. We sincerely appreciate the author’s work. All images in this publication are sourced directly from the original article, where a reference to the author’s profile is provided as well. This publication respects the author’s rights and enhances the visibility of their original work. If there are any concerns or the author wishes to discuss this matter further, we welcome an open dialogue to address potential issues and find an amicable resolution. Feel free to contact us through the ‘Contact Us’ section; the link is available in the website footer.