This change aims to reduce the number of places that need to be changed
when adding or editing a Godot type to the bindings.
Since the addition of `Variant.From<T>/As<T>` and
`VariantUtils.CreateFrom<T>/ConvertTo<T>`, we can now replace a lot of
the previous code in the bindings generator and the source generators
that specify these conversions for each type manually.
The only exceptions are the generic Godot collections (`Array<T>` and
`Dictionary<TKey, TValue>`) which still use the old version, as that
one cannot be matched by our new conversion methods (limitation in the
language with generics, forcing us to use delegate pointers).
The cleanup applies to:
- Bindings generator:
- `TypeInterface.cs_variant_to_managed`
- `TypeInterface.cs_managed_to_variant`
- Source generators:
- `MarshalUtils.AppendNativeVariantToManagedExpr`
- `MarshalUtils.AppendManagedToNativeVariantExpr`
- `MarshalUtils.AppendVariantToManagedExpr`
- `MarshalUtils.AppendManagedToVariantExpr`
- Remove `VariantSpanDisposer`, no need to dispose of the Variant Spans
since we are now borrowing the Variants instead of copying them.
- Remove `VariantSpanExtensions.Cleared` that was only used so the
Span was initialized for `VariantSpanDisposer` to know what to dispose.
- Fix stackalloc Spans to use constant VarArgsSpanThreshold
and avoid bound checks.
These callbacks are used for marshaling by callables and generic Godot
collections.
C# generics don't support specialization the way C++ templates do.
I knew NativeAOT could optimize away many type checks when the types
are known at compile time, but I didn't trust the JIT would do as good
a job, so I initially went with cached function pointers.
Well, it turns out the JIT is also very good at optimizing in this
scenario, so I'm changing the methods to do the conversion directly,
rather than returning a function pointer for the conversion.
The methods were moved to `VariantUtils`, and were renamed from
`GetFromVariantCallback/GetToVariantCallback` to `ConvertTo/CreateFrom`.
The new implementation looks like it goes through many `if` checks
at runtime to find the right branch for the type, but in practice it
works pretty much like template specialization. The JIT only generates
code for the relevant branch. Together with inlining, the result is
very close or the same as doing the conversion manually:
```cs
godot_variant variant;
int foo = variant.Int;
int bar = VariantUtils.ConvertTo<int>(variant);
```
If the type is a generic Godot collection, the conversion still goes
through a function pointer call.
The new code happens to be much shorter as well, with the file going
from 1057 lines to 407.
Side note: `Variant.cs` was mistakenly created in the wrong folder,
so I moved it to the `Core` folder.
We aim to make the C# API reflection-free, mainly for concerns about
performance, and to be able to target NativeAOT in refletion-free mode,
which reduces the binary size.
One of the main usages of reflection still left was the dynamic
invokation of callable delegates, and for some time I wasn't sure
I would find an alternative solution that I'd be happy with.
The new solution uses trampoline functions to invoke the delegates:
```
static void Trampoline(object delegateObj, NativeVariantPtrArgs args, out godot_variant ret)
{
if (args.Count != 1)
throw new ArgumentException($"Callable expected 1 arguments but received {args.Count}.");
string res = ((Func<int, string>)delegateObj)(
VariantConversionCallbacks.GetToManagedCallback<int>()(args[0])
);
ret = VariantConversionCallbacks.GetToVariantCallback<string>()(res);
}
Callable.CreateWithUnsafeTrampoline((int num) => "Foo" + num, &Trampoline);
```
Of course, this is too much boilerplate for user code. To improve this,
the `Callable.From` methods were added. These are overloads that take
`Action` and `Func` delegates, which covers the most common use cases:
lambdas and method groups:
```
// Lambda
Callable.From((int num) => "Foo" + num);
// Method group
string AppendNum(int num) => "Foo" + num;
Callable.From(AppendNum);
```
Unfortunately, due to limitations in the C# language, implicit
conversions from delegates to `Callable` are not supported.
`Callable.From` does not support custom delegates. These should be
uncommon, but the Godot C# API actually uses them for event signals.
As such, the bindings generator was updated to generate trampoline
functions for event signals. It was also optimized to use `Action`
instead of a custom delegate for parameterless signals, which removes
the need for the trampoline functions for those signals.
The change to reflection-free invokation removes one of the last needs
for `ConvertVariantToManagedObjectOfType`. The only remaining usage is
from calling script constructors with parameters from the engine
(`CreateManagedForGodotObjectScriptInstance`). Once that one is made
reflection-free, `ConvertVariantToManagedObjectOfType` can be removed.
When the C# bindings generator finds a type without meta assume the type
refers to the 64-bit version of the type:
- `float` is converted to `double`
- `int` is converted to `long`
- Remove event as a valid target of `SignalAttribute`
- Stop adding the `[Signal]` attribute to events in bindings_generator
- Make bindings_generator use the `EventHandler` suffix to be consistent with the C# source generator
- Remove obsolete comment about the signal's delegate name
This new version does not support the following type arguments:
- Generic types
- Array of Godot Object (Godot.Object[]) or derived types
The new implementation uses delegate pointers to call the Variant
conversion methods. We do type checking only once in the static
constructor to get the conversion delegates.
Now, we no longer need to do type checking every time, and we no
longer have to box value types.
This is the best implementation I could come up with, as C# generics
don't support anything similar to C++ template specializations.
- Array and Dictionary now store `Variant` instead of `System.Object`.
- Removed generic Array and Dictionary.
They cause too much issues, heavily relying on reflection and
very limited by the lack of a generic specialization.
- Removed support for non-Godot collections.
Support for them also relied heavily on reflection for marshaling.
Support for them will likely be re-introduced in the future, but
it will have to rely on source generators instead of reflection.
- Reduced our use of reflection.
The remaining usages will be moved to source generators soon.
The only usage that I'm not sure yet how to replace is dynamic
invocation of delegates.
Previously, we added source generators for invoking/accessing methods,
properties and fields in scripts. This freed us from the overhead of
reflection. However, the generated code still used our dynamic
marshaling functions, which do runtime type checking and box value
types.
This commit changes the bindings and source generators to include
'static' marshaling. Based on the types known at compile time, now
we generate the appropriate marshaling call for each type.
The editor no longer needs to create temporary instances to get the
default values. The initializer values of the exported properties are
still evaluated at runtime. For example, in the following example,
`GetInitialValue()` will be called when first looks for default values:
```
[Export] int MyValue = GetInitialValue();
```
Exporting fields with a non-supported type now results in a compiler
error rather than a runtime error when the script is used.
This base implementation is still very barebones but it defines the path
for how exporting will work (at least when embedding the .NET runtime).
Many manual steps are still needed, which should be automatized in the
future. For example, in addition to the API assemblies, now you also
need to copy the GodotPlugins assembly to each game project.
Finalizers are longer guaranteed to be called on exit now that
we switched to .NET Core. This results in native instances leaking.
The only solution I can think of so far is to keep a list of all
instances alive to dispose when the AssemblyLoadContext.Unloading
event is raised.
This replaces the way we invoke methods and set/get properties.
This first iteration rids us of runtime type checking in those
cases, as it's now done at compile time.
Later it will also stop needing the use of reflection. After that,
we will only depend on reflection for generic Godot Array and
Dictionary. We're stuck with reflection in generic collections
for now as C# doesn't support generic/template specialization.
This is only the initial implementation. Further iterations are
coming, specially once we switch to the native extension system
which completely changes the way members are accessed/invoked.
For example, with the native extension system we will likely need
to create `UnmanagedCallersOnly` invoke wrapper methods and return
function pointers to the engine.
Other kind of members, like event signals will be receiving the
same treatment in the future.
We're targeting .NET 5 for now to make development easier while
.NET 6 is not yet released.
TEMPORARY REGRESSIONS
---------------------
Assembly unloading is not implemented yet. As such, many Godot
resources are leaked at exit. This will be re-implemented later
together with assembly hot-reloading.
The main focus here was to remove the majority of code that relied on
Mono's embedding APIs, specially the reflection APIs. The embedding
APIs we still use are the bare minimum we need for things to work.
A lot of code was moved to C#. We no longer deal with any managed
objects (`MonoObject*`, and such) in native code, and all marshaling
is done in C#.
The reason for restructuring the code and move away from embedding APIs
is that once we move to .NET Core, we will be limited by the much more
minimal .NET hosting.
PERFORMANCE REGRESSIONS
-----------------------
Some parts of the code were written with little to no concern about
performance. This includes code that calls into script methods and
accesses script fields, properties and events.
The reason for this is that all of that will be moved to source
generators, so any work prior to that would be a waste of time.
DISABLED FEATURES
-----------------
Some code was removed as it no longer makes sense (or won't make sense
in the future).
Other parts were commented out with `#if 0`s and TODO warnings because
it doesn't make much sense to work on them yet as those parts will
change heavily when we switch to .NET Core but also when we start
introducing source generators.
As such, the following features were disabled temporarily:
- Assembly-reloading (will be done with ALCs in .NET Core).
- Properties/fields exports and script method listing (will be
handled by source generators in the future).
- Exception logging in the editor and stack info for errors.
- Exporting games.
- Building of C# projects. We no longer copy the Godot API assemblies
to the project directory, so MSBuild won't be able to find them. The
idea is to turn them into NuGet packages in the future, which could
also be obtained from local NuGet sources during development.
We will be progressively moving most code to C#.
The plan is to only use Mono's embedding APIs to set things at launch.
This will make it much easier to later support CoreCLR too which
doesn't have rich embedding APIs.
Additionally the code in C# is more maintainable and makes it easier
to implement new features, e.g.: runtime codegen which we could use to
avoid using reflection for marshaling everytime a field, property or
method is accessed.
SOME NOTES ON INTEROP
We make the same assumptions as GDNative about the size of the Godot
structures we use. We take it a bit further by also assuming the layout
of fields in some cases, which is riskier but let's us squeeze out some
performance by avoiding unnecessary managed to native calls.
Code that deals with native structs is less safe than before as there's
no RAII and copy constructors in C#. It's like using the GDNative C API
directly. One has to take special care to free values they own.
Perhaps we could use roslyn analyzers to check this, but I don't know
any that uses attributes to determine what's owned or borrowed.
As to why we maily use pointers for native structs instead of ref/out:
- AFAIK (and confirmed with a benchmark) ref/out are pinned
during P/Invoke calls and that has a cost.
- Native struct fields can't be ref/out in the first place.
- A `using` local can't be passed as ref/out, only `in`. Calling a
method or property on an `in` value makes a silent copy, so we want
to avoid `in`.
REGARDING THE BUILD SYSTEM
There's no longer a `mono_glue=yes/no` SCons options. We no longer
need to build with `mono_glue=no`, generate the glue and then build
again with `mono_glue=yes`. We build only once and generate the glue
(which is in C# now).
However, SCons no longer builds the C# projects for us. Instead one
must run `build_assemblies.py`, e.g.:
```sh
%godot_src_root%/modules/mono/build_scripts/build_assemblies.py \
--godot-output-dir=%godot_src_root%/bin \
--godot-target=release_debug`
```
We could turn this into a custom build target, but I don't know how
to do that with SCons (it's possible with Meson).
OTHER NOTES
Most of the moved code doesn't follow the C# naming convention and
still has the word Mono in the names despite no longer dealing with
Mono's embedding APIs. This is just temporary while transitioning,
to make it easier to understand what was moved where.
Implement built-in classes Vector4, Vector4i and Projection.
* Two versions of Vector4 (float and integer).
* A Projection class, which is a 4x4 matrix specialized in projection types.
These types have been requested for a long time, but given they were very corner case they were not added before.
Because in Godot 4, reimplementing parts of the rendering engine is now possible, access to these types (heavily used by the rendering code) becomes a necessity.
**Q**: Why Projection and not Matrix4?
**A**: Godot does not use Matrix2, Matrix3, Matrix4x3, etc. naming convention because, within the engine, these types always have a *purpose*. As such, Godot names them: Transform2D, Transform3D or Basis. In this case, this 4x4 matrix is _always_ used as a _Projection_, hence the naming.
Adds support for generating C# bindings that use the generic `Array<T>`
type instead of the non-generic `Array` type when the registered ClassDB
method specifies the array element type.
* Map is unnecessary and inefficient in almost every case.
* Replaced by the new HashMap.
* Renamed Map to RBMap and Set to RBSet for cases that still make sense
(order matters) but use is discouraged.
There were very few cases where replacing by HashMap was undesired because
keeping the key order was intended.
I tried to keep those (as RBMap) as much as possible, but might have missed
some. Review appreciated!
Adds a new, cleaned up, HashMap implementation.
* Uses Robin Hood Hashing (https://en.wikipedia.org/wiki/Hash_table#Robin_Hood_hashing).
* Keeps elements in a double linked list for simpler, ordered, iteration.
* Allows keeping iterators for later use in removal (Unlike Map<>, it does not do much
for performance vs keeping the key, but helps replace old code).
* Uses a more modern C++ iterator API, deprecates the old one.
* Supports custom allocator (in case there is a wish to use a paged one).
This class aims to unify all the associative template usage and replace it by this one:
* Map<> (whereas key order does not matter, which is 99% of cases)
* HashMap<>
* OrderedHashMap<>
* OAHashMap<>
PhysicsServer3DExtension inherits from PhysicsServer3D which is a
singleton class, since singleton classes are generated as static in C#
it would generate invalid C# so for now we'll be
ignoring PhysicsServer3DExtension.
The specific `_append_xml_*` methods implement the logic that generates
the proper XML documentation for the given BBCode tag and target and
appends it to the output.
- Outputs errors for missing members or methods when generating the C#
documentation.
- Hardcodes a special case for the `_init` method, in C# we'll reference
the constructor.
- Ignores properties with slashes (since they are not declared in C# and
can't be referenced in the documentation).
Sets `AlignOperands` to `DontAlign`.
`clang-format` developers seem to mostly care about space-based indentation and
every other version of clang-format breaks the bad mismatch of tabs and spaces
that it seems to use for operand alignment. So it's better without, so that it
respects our two-tabs `ContinuationIndentWidth`.
Use `System.Array.Empty<T>` to get an empty array instead of allocating
a new one every time. Since arrays are immutable there is no need to
allocate them every time.
The order of numbers is not changed except for Transform2D. All logic is done inside of their structures (and not in Variant).
For the number of decimals printed, they now use String::num_real which works best with real_t, except for Color which is fixed at 4 decimals (this is a reliable number of float digits when converting from 16-bpc so it seems like a good choice)
The ClassDB tests will detect when the core API has dependencies on
the editor API, which is not allowed.
This should prevent or warn early about issues like #44856
Happy new year to the wonderful Godot community!
2020 has been a tough year for most of us personally, but a good year for
Godot development nonetheless with a huge amount of work done towards Godot
4.0 and great improvements backported to the long-lived 3.2 branch.
We've had close to 400 contributors to engine code this year, authoring near
7,000 commit! (And that's only for the `master` branch and for the engine code,
there's a lot more when counting docs, demos and other first-party repos.)
Here's to a great year 2021 for all Godot users 🎆
The underscore prefix was used to avoid the conflict between the `RID` class
name and the matching enum value in `Variant::Type`.
This can be fixed differently by prefixing uses of the `RID` class in `Variant`
with the scope resolution operator, as done already for `AABB`.
Godot.NET.Sdk
-------------
Godot uses its own custom MSBuild Sdk for game
projects. This new Sdk adds its own functionality
on top of 'Microsoft.NET.Sdk'.
The new Sdk is resolved from the NuGet package.
All the default boilerplate was moved from game
projects to the Sdk. The default csproj for
game project can now be as simple as:
```
<Project Sdk="Godot.NET.Sdk/4.0.0-dev2">
<PropertyGroup>
<TargetFramework>netstandard2.1</TargetFramework>
</PropertyGroup>
</Project>
```
Source files are included by automatically so
Godot no longer needs to keep the csproj in sync
when creating new source files.
Define constants
----------------
Godot defines a list of constants for conditional
compilation. When exporting games, this list also
included engine 'features' and platform 'bits'.
There were a few problems with that:
- The 'features' constants were only defined when
exporting games. Not when building the game for
running in the editor player.
- If the project was built externally by an IDE,
the constants wouldn't be defined at all.
The new Sdk assigns default values to these
constants when not built from the Godot editor,
i.e.: when built from an IDE or from the command
line. The default define constants are determined
from the system MSBuild is running on.
However, it's not possible for MSBuild to
determine the set of supported engine features.
It's also not possible to determine if a project
is being built to run on a 32-bit or 64-bit
Godot executable.
As such the 'features' and 'bits' constants had
to be removed.
The benefit of checking those at compile time
was questionable, and they can still be checked
at runtime.
The new list of define constants includes:
- GODOT
- GODOT_<PLATFORM>
Defaults to the platform MSBuild is running on.
- GODOT_<PC/MOBILE/WEB>
- TOOLS
When building with the 'Debug' configuration
(editor and editor player).
- GODOT_REAL_T_IS_DOUBLE
Not defined by default unless $(GodotRealTIsDouble)
is overriden to be 'true'.
.NET Standard
-------------
The target framework of game projects was changed
to 'netstandard2.1'.
Which means that reduz' beloved style which we all became used to
will now be changed automatically to remove the first empty line.
This makes us lean closer to 1TBS (the one true brace style) instead
of hybridating it with some Allman-inspired spacing.
There's still the case of braces around single-statement blocks that
needs to be addressed (but clang-format can't help with that, but
clang-tidy may if we agree about it).
Part of #33027.
Not sure if we should check revision too, but this is good enough for what we want.
This will be needed to load the correct Microsoft.Build when we switch to the nuget version.
Implementation for new Variant types Callable, Signal, StringName.
Added support for PackedInt64Array and PackedFloat64Array.
Add generation of signal members as events, as well as support for
user created signals as events.
NOTE: As of now, raising such events will not emit the signal. As such,
one must use `EmitSignal` instead of raising the event directly.
Removed old ThreadLocal fallback class. It's safe to use thread_local now since
it's supported on all minimum versions of compilers we support.
For us, it practically only changes the fact that `A<A<int>>` is now
used instead of the C++03 compatible `A<A<int> >`.
Note: clang-format 10+ changed the `Standard` arguments to fully
specified `c++11`, `c++14`, etc. versions, but we can't use `c++17`
now if we want to preserve compatibility with clang-format 8 and 9.
`Cpp11` is still supported as deprecated alias for `Latest`.
- Renames PackedIntArray to PackedInt32Array.
- Renames PackedFloatArray to PackedFloat32Array.
- Adds PackedInt64Array and PackedFloat64Array.
- Renames Variant::REAL to Variant::FLOAT for consistency.
Packed arrays are for storing large amount of data and creating stuff like
meshes, buffers. textures, etc. Forcing them to be 64 is a huge waste of
memory. That said, many users requested the ability to have 64 bits packed
arrays for their games, so this is just an optional added type.
For Variant, the float datatype is always 64 bits, and exposed as `float`.
We still have `real_t` which is the datatype that can change from 32 to 64
bits depending on a compile flag (not entirely working right now, but that's
the idea). It affects math related datatypes and code only.
Neither Variant nor PackedArray make use of real_t, which is only intended
for math precision, so the term is removed from there to keep only float.
