asgardius/virtualx-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
virtualx-engine/servers/rendering/renderer_rd/shader_rd.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

877 lines
28 KiB
C++
Raw Normal View History

A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
/**************************************************************************/
/* shader_rd.cpp */
/**************************************************************************/
/* This file is part of: */
/* GODOT ENGINE */
/* https://godotengine.org */
/**************************************************************************/
Vulkan: Move thirdparty code out of drivers, style fixes - `vk_enum_string_helper.h` is a generated file taken from the SDK (Vulkan-ValidationLayers). - `vk_mem_alloc.h` is a library from GPUOpen: https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator
2020-02-11 14:01:43 +01:00
/* Copyright (c) 2014-present Godot Engine contributors (see AUTHORS.md). */
/* Copyright (c) 2007-2014 Juan Linietsky, Ariel Manzur. */
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
/* */
/* Permission is hereby granted, free of charge, to any person obtaining */
/* a copy of this software and associated documentation files (the */
/* "Software"), to deal in the Software without restriction, including */
/* without limitation the rights to use, copy, modify, merge, publish, */
/* distribute, sublicense, and/or sell copies of the Software, and to */
/* permit persons to whom the Software is furnished to do so, subject to */
/* the following conditions: */
/* */
/* The above copyright notice and this permission notice shall be */
/* included in all copies or substantial portions of the Software. */
/* */
/* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, */
/* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF */
/* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. */
/* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY */
/* CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, */
/* TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE */
/* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */
/**************************************************************************/
#include "shader_rd.h"
Remove unused classes and stray headers Found by reviewing headers with 1 or less matching includes: ``` find -name thirdparty -prune -o -name "*.h" -exec basename {} \; | sort -u > headers for header in $(cat headers); do echo "$header: "; rg -l "#include \"(.*/)?$header\"" | wc -l; done > list-includes ```
2020-03-24 09:50:51 +01:00
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
#include "core/io/compression.h"
Core: Move DirAccess and FileAccess to `core/io` File handling APIs are typically considered part of I/O, and we did have most `FileAccess` implementations in `core/io` already.
2021-06-11 14:51:48 +02:00
#include "core/io/dir_access.h"
#include "core/io/file_access.h"
Move some worker_thread_pool.h includes out of header files
2023-02-13 19:45:06 +01:00
#include "core/object/worker_thread_pool.h"
Make inclusion of Godot version in shader hash universal
2023-01-09 17:44:29 +01:00
#include "core/version.h"
RenderingServer reorganization
2020-12-04 19:26:24 +01:00
#include "renderer_compositor_rd.h"
Renaming of servers for coherency. VisualServer -> RenderingServer PhysicsServer -> PhysicsServer3D Physics2DServer -> PhysicsServer2D NavigationServer -> NavigationServer3D Navigation2DServer -> NavigationServer2D Also renamed corresponding files.
2020-03-27 19:21:27 +01:00
#include "servers/rendering/rendering_device.h"
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
#include "thirdparty/misc/smolv.h"
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
#define ENABLE_SHADER_CACHE 1
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
void ShaderRD::_add_stage(const char *p_code, StageType p_stage_type) {
Vector<String> lines = String(p_code).split("\n");
String text;
for (int i = 0; i < lines.size(); i++) {
Add const references detected by clang-tidy
2023-11-18 23:40:56 +01:00
const String &l = lines[i];
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
bool push_chunk = false;
StageTemplate::Chunk chunk;
if (l.begins_with("#VERSION_DEFINES")) {
chunk.type = StageTemplate::Chunk::TYPE_VERSION_DEFINES;
push_chunk = true;
} else if (l.begins_with("#GLOBALS")) {
switch (p_stage_type) {
case STAGE_TYPE_VERTEX:
chunk.type = StageTemplate::Chunk::TYPE_VERTEX_GLOBALS;
break;
case STAGE_TYPE_FRAGMENT:
chunk.type = StageTemplate::Chunk::TYPE_FRAGMENT_GLOBALS;
break;
case STAGE_TYPE_COMPUTE:
chunk.type = StageTemplate::Chunk::TYPE_COMPUTE_GLOBALS;
break;
default: {
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
}
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
push_chunk = true;
} else if (l.begins_with("#MATERIAL_UNIFORMS")) {
chunk.type = StageTemplate::Chunk::TYPE_MATERIAL_UNIFORMS;
push_chunk = true;
} else if (l.begins_with("#CODE")) {
chunk.type = StageTemplate::Chunk::TYPE_CODE;
push_chunk = true;
chunk.code = l.replace_first("#CODE", String()).replace(":", "").strip_edges().to_upper();
} else {
text += l + "\n";
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
if (push_chunk) {
Replace String comparisons with "", String() to is_empty() Also: - Adds two stress tests to test_string.h - Changes to .empty() on std::strings
2021-12-09 10:42:46 +01:00
if (!text.is_empty()) {
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
StageTemplate::Chunk text_chunk;
text_chunk.type = StageTemplate::Chunk::TYPE_TEXT;
text_chunk.text = text.utf8();
stage_templates[p_stage_type].chunks.push_back(text_chunk);
text = String();
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
stage_templates[p_stage_type].chunks.push_back(chunk);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
}
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
Replace String comparisons with "", String() to is_empty() Also: - Adds two stress tests to test_string.h - Changes to .empty() on std::strings
2021-12-09 10:42:46 +01:00
if (!text.is_empty()) {
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
StageTemplate::Chunk text_chunk;
text_chunk.type = StageTemplate::Chunk::TYPE_TEXT;
text_chunk.text = text.utf8();
stage_templates[p_stage_type].chunks.push_back(text_chunk);
text = String();
}
}
void ShaderRD::setup(const char *p_vertex_code, const char *p_fragment_code, const char *p_compute_code, const char *p_name) {
name = p_name;
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
if (p_compute_code) {
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
_add_stage(p_compute_code, STAGE_TYPE_COMPUTE);
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
is_compute = true;
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
} else {
is_compute = false;
if (p_vertex_code) {
_add_stage(p_vertex_code, STAGE_TYPE_VERTEX);
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
}
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
if (p_fragment_code) {
_add_stage(p_fragment_code, STAGE_TYPE_FRAGMENT);
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
}
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
StringBuilder tohash;
Make inclusion of Godot version in shader hash universal
2023-01-09 17:44:29 +01:00
tohash.append("[GodotVersionNumber]");
tohash.append(VERSION_NUMBER);
tohash.append("[GodotVersionHash]");
tohash.append(VERSION_HASH);
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
tohash.append("[SpirvCacheKey]");
tohash.append(RenderingDevice::get_singleton()->shader_get_spirv_cache_key());
tohash.append("[BinaryCacheKey]");
tohash.append(RenderingDevice::get_singleton()->shader_get_binary_cache_key());
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
tohash.append("[Vertex]");
tohash.append(p_vertex_code ? p_vertex_code : "");
tohash.append("[Fragment]");
tohash.append(p_fragment_code ? p_fragment_code : "");
tohash.append("[Compute]");
tohash.append(p_compute_code ? p_compute_code : "");
base_sha256 = tohash.as_string().sha256_text();
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
RID ShaderRD::version_create() {
//initialize() was never called
Replace error checks against `size` with `is_empty`
2024-01-19 13:21:39 +01:00
ERR_FAIL_COND_V(group_to_variant_map.is_empty(), RID());
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Version version;
version.dirty = true;
version.valid = false;
version.initialize_needed = true;
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
version.variants.clear();
version.variant_data.clear();
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
return version_owner.make_rid(version);
}
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
void ShaderRD::_initialize_version(Version *p_version) {
_clear_version(p_version);
p_version->valid = false;
p_version->dirty = false;
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_version->variants.resize_zeroed(variant_defines.size());
p_version->variant_data.resize(variant_defines.size());
p_version->group_compilation_tasks.resize(group_enabled.size());
p_version->group_compilation_tasks.fill(0);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
void ShaderRD::_clear_version(Version *p_version) {
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_ensure_finished(p_version);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
// Clear versions if they exist.
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
if (!p_version->variants.is_empty()) {
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
for (int i = 0; i < variant_defines.size(); i++) {
Initialize shader placeholders up front Then use the placeholder to create the shader instead of swapping RIDs This fixes a false positive that reported leaked shaders
2023-08-03 16:13:33 +02:00
if (p_version->variants[i].is_valid()) {
Fix 'Attempted to remove invalid ID' errors
2021-07-31 08:52:50 +02:00
RD::get_singleton()->free(p_version->variants[i]);
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_version->variants.clear();
p_version->variant_data.clear();
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
}
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
void ShaderRD::_build_variant_code(StringBuilder &builder, uint32_t p_variant, const Version *p_version, const StageTemplate &p_template) {
Use range iterators in LocalVector loops
2022-12-29 01:24:45 +01:00
for (const StageTemplate::Chunk &chunk : p_template.chunks) {
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
switch (chunk.type) {
case StageTemplate::Chunk::TYPE_VERSION_DEFINES: {
builder.append("\n"); //make sure defines begin at newline
builder.append(general_defines.get_data());
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
builder.append(variant_defines[p_variant].text.get_data());
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
for (int j = 0; j < p_version->custom_defines.size(); j++) {
builder.append(p_version->custom_defines[j].get_data());
}
builder.append("\n"); //make sure defines begin at newline
if (p_version->uniforms.size()) {
builder.append("#define MATERIAL_UNIFORMS_USED\n");
}
Use range iterators for `Map`
2021-08-09 22:13:42 +02:00
for (const KeyValue<StringName, CharString> &E : p_version->code_sections) {
builder.append(String("#define ") + String(E.key) + "_CODE_USED\n");
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
}
Add Metal support for macOS (arm64) and iOS
2024-02-19 19:52:00 +01:00
#if (defined(MACOS_ENABLED) || defined(IOS_ENABLED))
if (RD::get_singleton()->get_device_capabilities().device_family == RDD::DEVICE_VULKAN) {
builder.append("#define MOLTENVK_USED\n");
}
// Image atomics are supported on Metal 3.1 but no support in MoltenVK or SPIRV-Cross yet.
builder.append("#define NO_IMAGE_ATOMICS\n");
[macOS / iOS] Use storage buffers instead of unsupported images for the volumetric fog on MoltenVK.
2021-11-24 08:14:19 +01:00
#endif
Add Metal support for macOS (arm64) and iOS
2024-02-19 19:52:00 +01:00
Polish rendering driver refactor further Mainly: - Make `max_descriptors_per_pool` project setting Vulkan-specific. - Use a common, render driver agnostic magic FourCC for shader binary data. - Downgrade spirv_reflect to Vulkan-only dependency. - Add a `RENDER_DRIVER_*` macro to GLSL shader code for per-driver customizations.
2022-10-06 20:45:56 +02:00
builder.append(String("#define RENDER_DRIVER_") + OS::get_singleton()->get_current_rendering_driver_name().to_upper() + "\n");
Add external texture support (GLES3) Co-authored-by: Fredia Huya-Kouadio <fhuyakou@gmail.com> Co-authored-by: Mauricio Narvaez <nvz@meta.com>
2023-10-14 12:58:59 +02:00
builder.append("#define samplerExternalOES sampler2D\n");
builder.append("#define textureExternalOES texture2D\n");
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
} break;
case StageTemplate::Chunk::TYPE_MATERIAL_UNIFORMS: {
builder.append(p_version->uniforms.get_data()); //uniforms (same for vertex and fragment)
} break;
case StageTemplate::Chunk::TYPE_VERTEX_GLOBALS: {
builder.append(p_version->vertex_globals.get_data()); // vertex globals
} break;
case StageTemplate::Chunk::TYPE_FRAGMENT_GLOBALS: {
builder.append(p_version->fragment_globals.get_data()); // fragment globals
} break;
case StageTemplate::Chunk::TYPE_COMPUTE_GLOBALS: {
builder.append(p_version->compute_globals.get_data()); // compute globals
} break;
case StageTemplate::Chunk::TYPE_CODE: {
if (p_version->code_sections.has(chunk.code)) {
builder.append(p_version->code_sections[chunk.code].get_data());
}
} break;
case StageTemplate::Chunk::TYPE_TEXT: {
builder.append(chunk.text.get_data());
} break;
}
}
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
void ShaderRD::_compile_variant(uint32_t p_variant, CompileData p_data) {
uint32_t variant = group_to_variant_map[p_data.group][p_variant];
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
if (!variants_enabled[variant]) {
return; // Variant is disabled, return.
Add support for low-end 3D rendering. -Reduce number of uniform sets from 6 to 4. -Remove features in low end mode, in order to reduce the number of texture units fit to 16.
2020-12-07 22:27:38 +01:00
}
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
Vector<RD::ShaderStageSPIRVData> stages;
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
String error;
String current_source;
RD::ShaderStage current_stage = RD::SHADER_STAGE_VERTEX;
bool build_ok = true;
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
if (!is_compute) {
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
//vertex stage
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
StringBuilder builder;
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_build_variant_code(builder, variant, p_data.version, stage_templates[STAGE_TYPE_VERTEX]);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
current_source = builder.as_string();
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
RD::ShaderStageSPIRVData stage;
Split RenderingDevice into API-agnostic and RenderingDeviceDriver parts Credit and thanks to @bruzvg for multiple build fixes, update of 3rd-party items and MinGW support. Co-authored-by: bruvzg <7645683+bruvzg@users.noreply.github.com>
2023-12-19 12:48:02 +01:00
stage.spirv = RD::get_singleton()->shader_compile_spirv_from_source(RD::SHADER_STAGE_VERTEX, current_source, RD::SHADER_LANGUAGE_GLSL, &error);
if (stage.spirv.size() == 0) {
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
build_ok = false;
} else {
stage.shader_stage = RD::SHADER_STAGE_VERTEX;
stages.push_back(stage);
}
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
if (!is_compute && build_ok) {
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
//fragment stage
current_stage = RD::SHADER_STAGE_FRAGMENT;
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
StringBuilder builder;
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_build_variant_code(builder, variant, p_data.version, stage_templates[STAGE_TYPE_FRAGMENT]);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
current_source = builder.as_string();
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
RD::ShaderStageSPIRVData stage;
Split RenderingDevice into API-agnostic and RenderingDeviceDriver parts Credit and thanks to @bruzvg for multiple build fixes, update of 3rd-party items and MinGW support. Co-authored-by: bruvzg <7645683+bruvzg@users.noreply.github.com>
2023-12-19 12:48:02 +01:00
stage.spirv = RD::get_singleton()->shader_compile_spirv_from_source(RD::SHADER_STAGE_FRAGMENT, current_source, RD::SHADER_LANGUAGE_GLSL, &error);
if (stage.spirv.size() == 0) {
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
build_ok = false;
} else {
stage.shader_stage = RD::SHADER_STAGE_FRAGMENT;
stages.push_back(stage);
}
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
if (is_compute) {
//compute stage
current_stage = RD::SHADER_STAGE_COMPUTE;
StringBuilder builder;
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_build_variant_code(builder, variant, p_data.version, stage_templates[STAGE_TYPE_COMPUTE]);
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
current_source = builder.as_string();
Split particle shader entry points * Particle shaders now have start() and process() * Particle collision happens between them. * The RESTART property is kept, so porting an old shader is still possible. This fixes the problem of particle collisions not functioning on the first particle frame.
2021-04-17 17:21:03 +02:00
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
RD::ShaderStageSPIRVData stage;
Split RenderingDevice into API-agnostic and RenderingDeviceDriver parts Credit and thanks to @bruzvg for multiple build fixes, update of 3rd-party items and MinGW support. Co-authored-by: bruvzg <7645683+bruvzg@users.noreply.github.com>
2023-12-19 12:48:02 +01:00
stage.spirv = RD::get_singleton()->shader_compile_spirv_from_source(RD::SHADER_STAGE_COMPUTE, current_source, RD::SHADER_LANGUAGE_GLSL, &error);
if (stage.spirv.size() == 0) {
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
build_ok = false;
} else {
stage.shader_stage = RD::SHADER_STAGE_COMPUTE;
stages.push_back(stage);
}
}
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
if (!build_ok) {
Reimplement Mutex with C++'s <mutex> Main: - It's now implemented thanks to `<mutex>`. No more platform-specific implementations. - `BinaryMutex` (non-recursive) is added, as an alternative for special cases. - Doesn't need allocation/deallocation anymore. It can live in the stack and be part of other classes. - Because of that, it's methods are now `const` and the inner mutex is `mutable` so it can be easily used in `const` contexts. - A no-op implementation is provided if `NO_THREADS` is defined. No more need to add `#ifdef NO_THREADS` just for this. - `MutexLock` now takes a reference. At this point the cases of null `Mutex`es are rare. If you ever need that, just don't use `MutexLock`. - Thread-safe utilities are therefore simpler now. Misc.: - `ScopedMutexLock` is dropped and replaced by `MutexLock`, because they were pretty much the same. - Every case of lock, do-something, unlock is replaced by `MutexLock` (complex cases where it's not straightfoward are kept as as explicit lock and unlock). - `ShaderRD` contained an `std::mutex`, which has been replaced by `Mutex`.
2020-02-26 11:28:13 +01:00
MutexLock lock(variant_set_mutex); //properly print the errors
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
ERR_PRINT("Error compiling " + String(current_stage == RD::SHADER_STAGE_COMPUTE ? "Compute " : (current_stage == RD::SHADER_STAGE_VERTEX ? "Vertex" : "Fragment")) + " shader, variant #" + itos(variant) + " (" + variant_defines[variant].text.get_data() + ").");
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
ERR_PRINT(error);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
#ifdef DEBUG_ENABLED
ERR_PRINT("code:\n" + current_source.get_with_code_lines());
#endif
return;
}
Moved the shader source compilation code outside RenderingDevice and Vulkan
2019-07-29 00:58:32 +02:00
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
Vector<uint8_t> shader_data = RD::get_singleton()->shader_compile_binary_from_spirv(stages, name + ":" + itos(variant));
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
Replace error checks against `size` with `is_empty`
2024-01-19 13:21:39 +01:00
ERR_FAIL_COND(shader_data.is_empty());
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
Reimplement Mutex with C++'s <mutex> Main: - It's now implemented thanks to `<mutex>`. No more platform-specific implementations. - `BinaryMutex` (non-recursive) is added, as an alternative for special cases. - Doesn't need allocation/deallocation anymore. It can live in the stack and be part of other classes. - Because of that, it's methods are now `const` and the inner mutex is `mutable` so it can be easily used in `const` contexts. - A no-op implementation is provided if `NO_THREADS` is defined. No more need to add `#ifdef NO_THREADS` just for this. - `MutexLock` now takes a reference. At this point the cases of null `Mutex`es are rare. If you ever need that, just don't use `MutexLock`. - Thread-safe utilities are therefore simpler now. Misc.: - `ScopedMutexLock` is dropped and replaced by `MutexLock`, because they were pretty much the same. - Every case of lock, do-something, unlock is replaced by `MutexLock` (complex cases where it's not straightfoward are kept as as explicit lock and unlock). - `ShaderRD` contained an `std::mutex`, which has been replaced by `Mutex`.
2020-02-26 11:28:13 +01:00
{
MutexLock lock(variant_set_mutex);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_data.version->variants.write[variant] = RD::get_singleton()->shader_create_from_bytecode(shader_data, p_data.version->variants[variant]);
p_data.version->variant_data.write[variant] = shader_data;
Reimplement Mutex with C++'s <mutex> Main: - It's now implemented thanks to `<mutex>`. No more platform-specific implementations. - `BinaryMutex` (non-recursive) is added, as an alternative for special cases. - Doesn't need allocation/deallocation anymore. It can live in the stack and be part of other classes. - Because of that, it's methods are now `const` and the inner mutex is `mutable` so it can be easily used in `const` contexts. - A no-op implementation is provided if `NO_THREADS` is defined. No more need to add `#ifdef NO_THREADS` just for this. - `MutexLock` now takes a reference. At this point the cases of null `Mutex`es are rare. If you ever need that, just don't use `MutexLock`. - Thread-safe utilities are therefore simpler now. Misc.: - `ScopedMutexLock` is dropped and replaced by `MutexLock`, because they were pretty much the same. - Every case of lock, do-something, unlock is replaced by `MutexLock` (complex cases where it's not straightfoward are kept as as explicit lock and unlock). - `ShaderRD` contained an `std::mutex`, which has been replaced by `Mutex`.
2020-02-26 11:28:13 +01:00
}
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Added ability to visualize native shaders
2021-01-06 00:01:50 +01:00
RS::ShaderNativeSourceCode ShaderRD::version_get_native_source_code(RID p_version) {
Rename RID's `getornull()` to `get_or_null()`
2021-09-29 19:08:41 +02:00
Version *version = version_owner.get_or_null(p_version);
Added ability to visualize native shaders
2021-01-06 00:01:50 +01:00
RS::ShaderNativeSourceCode source_code;
[Servers] Replace `ERR_FAIL_COND` with `ERR_FAIL_NULL` where applicable
2023-09-09 17:04:18 +02:00
ERR_FAIL_NULL_V(version, source_code);
Added ability to visualize native shaders
2021-01-06 00:01:50 +01:00
source_code.versions.resize(variant_defines.size());
for (int i = 0; i < source_code.versions.size(); i++) {
if (!is_compute) {
//vertex stage
StringBuilder builder;
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
_build_variant_code(builder, i, version, stage_templates[STAGE_TYPE_VERTEX]);
Added ability to visualize native shaders
2021-01-06 00:01:50 +01:00
RS::ShaderNativeSourceCode::Version::Stage stage;
stage.name = "vertex";
stage.code = builder.as_string();
source_code.versions.write[i].stages.push_back(stage);
}
if (!is_compute) {
//fragment stage
StringBuilder builder;
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
_build_variant_code(builder, i, version, stage_templates[STAGE_TYPE_FRAGMENT]);
Added ability to visualize native shaders
2021-01-06 00:01:50 +01:00
RS::ShaderNativeSourceCode::Version::Stage stage;
stage.name = "fragment";
stage.code = builder.as_string();
source_code.versions.write[i].stages.push_back(stage);
}
if (is_compute) {
//compute stage
StringBuilder builder;
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
_build_variant_code(builder, i, version, stage_templates[STAGE_TYPE_COMPUTE]);
Added ability to visualize native shaders
2021-01-06 00:01:50 +01:00
RS::ShaderNativeSourceCode::Version::Stage stage;
stage.name = "compute";
stage.code = builder.as_string();
source_code.versions.write[i].stages.push_back(stage);
}
}
return source_code;
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
String ShaderRD::_version_get_sha1(Version *p_version) const {
StringBuilder hash_build;
hash_build.append("[uniforms]");
hash_build.append(p_version->uniforms.get_data());
hash_build.append("[vertex_globals]");
hash_build.append(p_version->vertex_globals.get_data());
hash_build.append("[fragment_globals]");
hash_build.append(p_version->fragment_globals.get_data());
hash_build.append("[compute_globals]");
hash_build.append(p_version->compute_globals.get_data());
Vector<StringName> code_sections;
Use range iterators for `Map`
2021-08-09 22:13:42 +02:00
for (const KeyValue<StringName, CharString> &E : p_version->code_sections) {
code_sections.push_back(E.key);
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
}
code_sections.sort_custom<StringName::AlphCompare>();
for (int i = 0; i < code_sections.size(); i++) {
hash_build.append(String("[code:") + String(code_sections[i]) + "]");
hash_build.append(p_version->code_sections[code_sections[i]].get_data());
}
for (int i = 0; i < p_version->custom_defines.size(); i++) {
hash_build.append("[custom_defines:" + itos(i) + "]");
hash_build.append(p_version->custom_defines[i].get_data());
}
return hash_build.as_string().sha1_text();
}
static const char *shader_file_header = "GDSC";
Fix uninitialized variable ending up sent to Vulkan The first time a shader is compiled Godot performs the following: ```cpp for (uint32_t i = 0; i < SHADER_STAGE_MAX; i++) { if (spirv_data.push_constant_stages_mask.has_flag((ShaderStage)(1 << i))) { binary_data.push_constant_vk_stages_mask |= shader_stage_masks[i]; } } ``` However binary_data.push_constant_vk_stages_mask is never initialized to 0 and thus contains garbage data or'ed with the good data. This value is used by push constants (and many other things) thus it can be a big deal. Fortunately because the relevant flags are always guaranteed to be set (but not guaranteed to be unset), the damage is restricted to: 1. Performance (unnecessary flushing & over-excessive barriers) 2. Overwriting push descriptors already set (this would be serious, doesn't seem to be an issue) 3. Driver implementations going crazy when they see bits set they don't expect (unknown if this is an issue) This uninitialized value is later saved into the binary cache. Valgrind is able to detect this bug on the first run, but not on the subsequent ones because they data comes from a file. cache_file_version has been bumped to force rebuild of all cached shaders. Because the ones generated so far are compromised.
2023-07-29 23:28:33 +02:00
static const uint32_t cache_file_version = 3;
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Namespace shader cache files by graphics API
2024-01-31 20:12:48 +01:00
String ShaderRD::_get_cache_file_path(Version *p_version, int p_group) {
const String &sha1 = _version_get_sha1(p_version);
Finish splitting functionality of the Vulkan and D3D12 backends into RenderingDeviceDriver.
2023-12-19 18:57:56 +01:00
const String &api_safe_name = String(RD::get_singleton()->get_device_api_name()).validate_filename().to_lower();
Namespace shader cache files by graphics API
2024-01-31 20:12:48 +01:00
const String &path = shader_cache_dir.path_join(name).path_join(group_sha256[p_group]).path_join(sha1) + "." + api_safe_name + ".cache";
return path;
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Namespace shader cache files by graphics API
2024-01-31 20:12:48 +01:00
bool ShaderRD::_load_from_cache(Version *p_version, int p_group) {
const String &path = _get_cache_file_path(p_version, p_group);
Make FileAccess and DirAccess classes reference counted.
2022-03-23 10:08:58 +01:00
Ref<FileAccess> f = FileAccess::open(path, FileAccess::READ);
if (f.is_null()) {
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
return false;
}
char header[5] = { 0, 0, 0, 0, 0 };
f->get_buffer((uint8_t *)header, 4);
ERR_FAIL_COND_V(header != String(shader_file_header), false);
uint32_t file_version = f->get_32();
if (file_version != cache_file_version) {
return false; // wrong version
}
uint32_t variant_count = f->get_32();
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
ERR_FAIL_COND_V(variant_count != (uint32_t)group_to_variant_map[p_group].size(), false); //should not happen but check
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
for (uint32_t i = 0; i < variant_count; i++) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
int variant_id = group_to_variant_map[p_group][i];
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
uint32_t variant_size = f->get_32();
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
ERR_FAIL_COND_V(variant_size == 0 && variants_enabled[variant_id], false);
if (!variants_enabled[variant_id]) {
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
continue;
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
}
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
Vector<uint8_t> variant_bytes;
variant_bytes.resize(variant_size);
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
uint32_t br = f->get_buffer(variant_bytes.ptrw(), variant_size);
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
ERR_FAIL_COND_V(br != variant_size, false);
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_version->variant_data.write[variant_id] = variant_bytes;
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
for (uint32_t i = 0; i < variant_count; i++) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
int variant_id = group_to_variant_map[p_group][i];
if (!variants_enabled[variant_id]) {
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
MutexLock lock(variant_set_mutex);
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_version->variants.write[variant_id] = RID();
Implement Binary Shader Compilation * Added an extra stage before compiling shader, which is generating a binary blob. * On Vulkan, this allows caching the SPIRV reflection information, which is expensive to parse. * On other (future) RenderingDevices, it allows caching converted binary data, such as DXIL or MSL. This PR makes the shader cache include the reflection information, hence editor startup times are significantly improved. I tested this well and it appears to work, and I added a lot of consistency checks, but because it includes writing and reading binary information, rare bugs may pop up, so be aware. There was not much of a choice for storing the reflection information, given shaders can be a lot, take a lot of space and take time to parse.
2021-07-25 16:22:55 +02:00
continue;
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
{
MutexLock lock(variant_set_mutex);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
RID shader = RD::get_singleton()->shader_create_from_bytecode(p_version->variant_data[variant_id], p_version->variants[variant_id]);
if (shader.is_null()) {
for (uint32_t j = 0; j < i; j++) {
int variant_free_id = group_to_variant_map[p_group][j];
RD::get_singleton()->free(p_version->variants[variant_free_id]);
}
ERR_FAIL_COND_V(shader.is_null(), false);
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_version->variants.write[variant_id] = shader;
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
}
}
p_version->valid = true;
return true;
}
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
void ShaderRD::_save_to_cache(Version *p_version, int p_group) {
Add shader_cache_dir_valid check to _save_to_cache
2024-01-12 04:17:57 +01:00
ERR_FAIL_COND(!shader_cache_dir_valid);
Namespace shader cache files by graphics API
2024-01-31 20:12:48 +01:00
const String &path = _get_cache_file_path(p_version, p_group);
Make FileAccess and DirAccess classes reference counted.
2022-03-23 10:08:58 +01:00
Ref<FileAccess> f = FileAccess::open(path, FileAccess::WRITE);
ERR_FAIL_COND(f.is_null());
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
f->store_buffer((const uint8_t *)shader_file_header, 4);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
f->store_32(cache_file_version); // File version.
uint32_t variant_count = group_to_variant_map[p_group].size();
f->store_32(variant_count); // Variant count.
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
for (uint32_t i = 0; i < variant_count; i++) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
int variant_id = group_to_variant_map[p_group][i];
f->store_32(p_version->variant_data[variant_id].size()); // Stage count.
f->store_buffer(p_version->variant_data[variant_id].ptr(), p_version->variant_data[variant_id].size());
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
}
}
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
void ShaderRD::_allocate_placeholders(Version *p_version, int p_group) {
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
ERR_FAIL_COND(p_version->variants.is_empty());
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
for (uint32_t i = 0; i < group_to_variant_map[p_group].size(); i++) {
int variant_id = group_to_variant_map[p_group][i];
RID shader = RD::get_singleton()->shader_create_placeholder();
{
MutexLock lock(variant_set_mutex);
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_version->variants.write[variant_id] = shader;
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
}
}
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
// Try to compile all variants for a given group.
// Will skip variants that are disabled.
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
void ShaderRD::_compile_version_start(Version *p_version, int p_group) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
if (!group_enabled[p_group]) {
return;
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
p_version->dirty = false;
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
#if ENABLE_SHADER_CACHE
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
if (shader_cache_dir_valid) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
if (_load_from_cache(p_version, p_group)) {
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
return;
}
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
#endif
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
CompileData compile_data;
compile_data.version = p_version;
compile_data.group = p_group;
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
WorkerThreadPool::GroupID group_task = WorkerThreadPool::get_singleton()->add_template_group_task(this, &ShaderRD::_compile_variant, compile_data, group_to_variant_map[p_group].size(), -1, true, SNAME("ShaderCompilation"));
p_version->group_compilation_tasks.write[p_group] = group_task;
}
Remove ThreadWorkPool, replace by WorkerThreadPool The former needs to be allocated once per usage. The later is shared for all threads, which is more efficient. It can also be better debugged.
2022-07-23 19:12:41 +02:00
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
void ShaderRD::_compile_version_end(Version *p_version, int p_group) {
if (p_version->group_compilation_tasks.size() <= p_group || p_version->group_compilation_tasks[p_group] == 0) {
return;
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
WorkerThreadPool::GroupID group_task = p_version->group_compilation_tasks[p_group];
WorkerThreadPool::get_singleton()->wait_for_group_task_completion(group_task);
p_version->group_compilation_tasks.write[p_group] = 0;
Moved the shader source compilation code outside RenderingDevice and Vulkan
2019-07-29 00:58:32 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
bool all_valid = true;
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
for (uint32_t i = 0; i < group_to_variant_map[p_group].size(); i++) {
int variant_id = group_to_variant_map[p_group][i];
if (!variants_enabled[variant_id]) {
continue; // Disabled.
Add support for low-end 3D rendering. -Reduce number of uniform sets from 6 to 4. -Remove features in low end mode, in order to reduce the number of texture units fit to 16.
2020-12-07 22:27:38 +01:00
}
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
if (p_version->variants[variant_id].is_null()) {
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
all_valid = false;
break;
Moved the shader source compilation code outside RenderingDevice and Vulkan
2019-07-29 00:58:32 +02:00
}
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
}
Moved the shader source compilation code outside RenderingDevice and Vulkan
2019-07-29 00:58:32 +02:00
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
if (!all_valid) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
// Clear versions if they exist.
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
for (int i = 0; i < variant_defines.size(); i++) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
if (!variants_enabled[i] || !group_enabled[variant_defines[i].group]) {
continue; // Disabled.
Add support for low-end 3D rendering. -Reduce number of uniform sets from 6 to 4. -Remove features in low end mode, in order to reduce the number of texture units fit to 16.
2020-12-07 22:27:38 +01:00
}
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
if (!p_version->variants[i].is_null()) {
RD::get_singleton()->free(p_version->variants[i]);
}
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
p_version->variants.clear();
p_version->variant_data.clear();
Added a spinlock template as well as a thread work pool class. Also, optimized shader compilation to happen on threads.
2019-07-29 17:59:18 +02:00
return;
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
}
#if ENABLE_SHADER_CACHE
else if (shader_cache_dir_valid) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
_save_to_cache(p_version, p_group);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
#endif
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
p_version->valid = true;
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
void ShaderRD::_compile_ensure_finished(Version *p_version) {
// Wait for compilation of existing groups if necessary.
for (int i = 0; i < group_enabled.size(); i++) {
_compile_version_end(p_version, i);
}
}
Replace most uses of Map by HashMap * Map is unnecessary and inefficient in almost every case. * Replaced by the new HashMap. * Renamed Map to RBMap and Set to RBSet for cases that still make sense (order matters) but use is discouraged. There were very few cases where replacing by HashMap was undesired because keeping the key order was intended. I tried to keep those (as RBMap) as much as possible, but might have missed some. Review appreciated!
2022-05-13 15:04:37 +02:00
void ShaderRD::version_set_code(RID p_version, const HashMap<String, String> &p_code, const String &p_uniforms, const String &p_vertex_globals, const String &p_fragment_globals, const Vector<String> &p_custom_defines) {
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
ERR_FAIL_COND(is_compute);
Rename RID's `getornull()` to `get_or_null()`
2021-09-29 19:08:41 +02:00
Version *version = version_owner.get_or_null(p_version);
[Servers] Replace `ERR_FAIL_COND` with `ERR_FAIL_NULL` where applicable
2023-09-09 17:04:18 +02:00
ERR_FAIL_NULL(version);
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_ensure_finished(version);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
version->vertex_globals = p_vertex_globals.utf8();
version->fragment_globals = p_fragment_globals.utf8();
Completed material/2D shader support (missing SCREEN_TEXTURE)
2019-07-21 16:31:30 +02:00
version->uniforms = p_uniforms.utf8();
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
version->code_sections.clear();
Use range iterators for `Map`
2021-08-09 22:13:42 +02:00
for (const KeyValue<String, String> &E : p_code) {
version->code_sections[StringName(E.key.to_upper())] = E.value.utf8();
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
}
Completed material/2D shader support (missing SCREEN_TEXTURE)
2019-07-21 16:31:30 +02:00
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
version->custom_defines.clear();
for (int i = 0; i < p_custom_defines.size(); i++) {
version->custom_defines.push_back(p_custom_defines[i].utf8());
}
version->dirty = true;
if (version->initialize_needed) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
_initialize_version(version);
for (int i = 0; i < group_enabled.size(); i++) {
if (!group_enabled[i]) {
_allocate_placeholders(version, i);
continue;
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_version_start(version, i);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
version->initialize_needed = false;
}
}
Replace most uses of Map by HashMap * Map is unnecessary and inefficient in almost every case. * Replaced by the new HashMap. * Renamed Map to RBMap and Set to RBSet for cases that still make sense (order matters) but use is discouraged. There were very few cases where replacing by HashMap was undesired because keeping the key order was intended. I tried to keep those (as RBMap) as much as possible, but might have missed some. Review appreciated!
2022-05-13 15:04:37 +02:00
void ShaderRD::version_set_compute_code(RID p_version, const HashMap<String, String> &p_code, const String &p_uniforms, const String &p_compute_globals, const Vector<String> &p_custom_defines) {
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
ERR_FAIL_COND(!is_compute);
Rename RID's `getornull()` to `get_or_null()`
2021-09-29 19:08:41 +02:00
Version *version = version_owner.get_or_null(p_version);
[Servers] Replace `ERR_FAIL_COND` with `ERR_FAIL_NULL` where applicable
2023-09-09 17:04:18 +02:00
ERR_FAIL_NULL(version);
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_ensure_finished(version);
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
version->compute_globals = p_compute_globals.utf8();
version->uniforms = p_uniforms.utf8();
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
version->code_sections.clear();
Use range iterators for `Map`
2021-08-09 22:13:42 +02:00
for (const KeyValue<String, String> &E : p_code) {
version->code_sections[StringName(E.key.to_upper())] = E.value.utf8();
Refactor GLSL shader compilation -Used a more consistent set of keywords for the shader -Remove all harcoded entry points -Re-wrote the GLSL shader parser, new system is more flexible. Allows any entry point organization. -Entry point for sky shaders is now sky(). -Entry point for particle shaders is now process().
2021-04-13 22:01:43 +02:00
}
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
version->custom_defines.clear();
for (int i = 0; i < p_custom_defines.size(); i++) {
version->custom_defines.push_back(p_custom_defines[i].utf8());
}
version->dirty = true;
if (version->initialize_needed) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
_initialize_version(version);
for (int i = 0; i < group_enabled.size(); i++) {
if (!group_enabled[i]) {
_allocate_placeholders(version, i);
continue;
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_version_start(version, i);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
}
Untested support for compute shaders
2019-09-25 21:44:44 +02:00
version->initialize_needed = false;
}
}
Custom material support seems complete.
2019-07-27 15:23:24 +02:00
bool ShaderRD::version_is_valid(RID p_version) {
Rename RID's `getornull()` to `get_or_null()`
2021-09-29 19:08:41 +02:00
Version *version = version_owner.get_or_null(p_version);
[Servers] Replace `ERR_FAIL_COND` with `ERR_FAIL_NULL` where applicable
2023-09-09 17:04:18 +02:00
ERR_FAIL_NULL_V(version, false);
Custom material support seems complete.
2019-07-27 15:23:24 +02:00
if (version->dirty) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
_initialize_version(version);
for (int i = 0; i < group_enabled.size(); i++) {
if (!group_enabled[i]) {
_allocate_placeholders(version, i);
continue;
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_version_start(version, i);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
}
Custom material support seems complete.
2019-07-27 15:23:24 +02:00
}
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_ensure_finished(version);
Custom material support seems complete.
2019-07-27 15:23:24 +02:00
return version->valid;
}
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
bool ShaderRD::version_free(RID p_version) {
if (version_owner.owns(p_version)) {
Rename RID's `getornull()` to `get_or_null()`
2021-09-29 19:08:41 +02:00
Version *version = version_owner.get_or_null(p_version);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
_clear_version(version);
version_owner.free(p_version);
} else {
return false;
}
return true;
}
Add support for low-end 3D rendering. -Reduce number of uniform sets from 6 to 4. -Remove features in low end mode, in order to reduce the number of texture units fit to 16.
2020-12-07 22:27:38 +01:00
void ShaderRD::set_variant_enabled(int p_variant, bool p_enabled) {
ERR_FAIL_COND(version_owner.get_rid_count() > 0); //versions exist
ERR_FAIL_INDEX(p_variant, variants_enabled.size());
variants_enabled.write[p_variant] = p_enabled;
}
bool ShaderRD::is_variant_enabled(int p_variant) const {
ERR_FAIL_INDEX_V(p_variant, variants_enabled.size(), false);
return variants_enabled[p_variant];
}
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
void ShaderRD::enable_group(int p_group) {
ERR_FAIL_INDEX(p_group, group_enabled.size());
if (group_enabled[p_group]) {
// Group already enabled, do nothing.
return;
}
group_enabled.write[p_group] = true;
// Compile all versions again to include the new group.
List<RID> all_versions;
version_owner.get_owned_list(&all_versions);
Reduce and prevent unnecessary random-access to `List` Random-access access to `List` when iterating is `O(n^2)` (`O(n)` when accessing a single element) * Removed subscript operator, in favor of a more explicit `get` * Added conversion from `Iterator` to `ConstIterator` * Remade existing operations into other solutions when applicable
2024-04-15 15:18:34 +02:00
for (const RID &E : all_versions) {
Version *version = version_owner.get_or_null(E);
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
_compile_version_start(version, p_group);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
}
}
bool ShaderRD::is_group_enabled(int p_group) const {
return group_enabled[p_group];
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
bool ShaderRD::shader_cache_cleanup_on_start = false;
Rewrote how barriers work for faster rendering -Added more finegrained control in RenderingDevice API -Optimized barriers (use less ones for thee same) -General optimizations -Shadows render all together unbarriered -GI can render together with shadows. -SDFGI can render together with depth-preoass. -General fixes -Added GPU detection
2021-02-02 20:51:36 +01:00
ShaderRD::ShaderRD() {
// Do not feel forced to use this, in most cases it makes little to no difference.
bool use_32_threads = false;
if (RD::get_singleton()->get_device_vendor_name() == "NVIDIA") {
use_32_threads = true;
}
String base_compute_define_text;
if (use_32_threads) {
base_compute_define_text = "\n#define NATIVE_LOCAL_GROUP_SIZE 32\n#define NATIVE_LOCAL_SIZE_2D_X 8\n#define NATIVE_LOCAL_SIZE_2D_Y 4\n";
} else {
base_compute_define_text = "\n#define NATIVE_LOCAL_GROUP_SIZE 64\n#define NATIVE_LOCAL_SIZE_2D_X 8\n#define NATIVE_LOCAL_SIZE_2D_Y 8\n";
}
base_compute_defines = base_compute_define_text.ascii();
}
Modified light rendering to make it more compatible. Modified polygon management to make it more compatible with MoltenVK
2019-07-10 22:44:55 +02:00
void ShaderRD::initialize(const Vector<String> &p_variant_defines, const String &p_general_defines) {
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
ERR_FAIL_COND(variant_defines.size());
Replace error checks against `size` with `is_empty`
2024-01-19 13:21:39 +01:00
ERR_FAIL_COND(p_variant_defines.is_empty());
Add support for low-end 3D rendering. -Reduce number of uniform sets from 6 to 4. -Remove features in low end mode, in order to reduce the number of texture units fit to 16.
2020-12-07 22:27:38 +01:00
Modified light rendering to make it more compatible. Modified polygon management to make it more compatible with MoltenVK
2019-07-10 22:44:55 +02:00
general_defines = p_general_defines.utf8();
Add support for low-end 3D rendering. -Reduce number of uniform sets from 6 to 4. -Remove features in low end mode, in order to reduce the number of texture units fit to 16.
2020-12-07 22:27:38 +01:00
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
// When initialized this way, there is just one group and its always enabled.
group_to_variant_map.insert(0, LocalVector<int>{});
group_enabled.push_back(true);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
for (int i = 0; i < p_variant_defines.size(); i++) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
variant_defines.push_back(VariantDefine(0, p_variant_defines[i], true));
Add support for low-end 3D rendering. -Reduce number of uniform sets from 6 to 4. -Remove features in low end mode, in order to reduce the number of texture units fit to 16.
2020-12-07 22:27:38 +01:00
variants_enabled.push_back(true);
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
variant_to_group.push_back(0);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
group_to_variant_map[0].push_back(i);
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Replace String comparisons with "", String() to is_empty() Also: - Adds two stress tests to test_string.h - Changes to .empty() on std::strings
2021-12-09 10:42:46 +01:00
if (!shader_cache_dir.is_empty()) {
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
group_sha256.resize(1);
_initialize_cache();
}
}
void ShaderRD::_initialize_cache() {
for (const KeyValue<int, LocalVector<int>> &E : group_to_variant_map) {
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
StringBuilder hash_build;
hash_build.append("[base_hash]");
hash_build.append(base_sha256);
hash_build.append("[general_defines]");
hash_build.append(general_defines.get_data());
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
hash_build.append("[group_id]");
hash_build.append(itos(E.key));
for (uint32_t i = 0; i < E.value.size(); i++) {
hash_build.append("[variant_defines:" + itos(E.value[i]) + "]");
hash_build.append(variant_defines[E.value[i]].text.get_data());
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
}
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
group_sha256[E.key] = hash_build.as_string().sha256_text();
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
Make FileAccess and DirAccess classes reference counted.
2022-03-23 10:08:58 +01:00
Ref<DirAccess> d = DirAccess::open(shader_cache_dir);
ERR_FAIL_COND(d.is_null());
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
if (d->change_dir(name) != OK) {
Error err = d->make_dir(name);
ERR_FAIL_COND(err != OK);
d->change_dir(name);
}
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
// Erase other versions?
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
if (shader_cache_cleanup_on_start) {
}
//
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
if (d->change_dir(group_sha256[E.key]) != OK) {
Error err = d->make_dir(group_sha256[E.key]);
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
ERR_FAIL_COND(err != OK);
}
shader_cache_dir_valid = true;
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
print_verbose("Shader '" + name + "' (group " + itos(E.key) + ") SHA256: " + group_sha256[E.key]);
}
}
// Same as above, but allows specifying shader compilation groups.
void ShaderRD::initialize(const Vector<VariantDefine> &p_variant_defines, const String &p_general_defines) {
ERR_FAIL_COND(variant_defines.size());
Replace error checks against `size` with `is_empty`
2024-01-19 13:21:39 +01:00
ERR_FAIL_COND(p_variant_defines.is_empty());
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
general_defines = p_general_defines.utf8();
int max_group_id = 0;
for (int i = 0; i < p_variant_defines.size(); i++) {
// Fill variant array.
variant_defines.push_back(p_variant_defines[i]);
variants_enabled.push_back(true);
Implement asynchronous transfer queues, thread guards on RenderingDevice. Add ubershaders and rework pipeline caches for Forward+ and Mobile. - Implements asynchronous transfer queues from PR #87590. - Adds ubershaders that can run with specialization constants specified as push constants. - Pipelines with specialization constants can compile in the background. - Added monitoring for pipeline compilations. - Materials and shaders can now be created asynchronously on background threads. - Meshes that are loaded on background threads can also compile pipelines as part of the loading process.
2024-03-15 18:13:31 +01:00
variant_to_group.push_back(p_variant_defines[i].group);
Add Shader compile groups to RD Shader system This allows us to specify a subset of variants to compile at load time and conditionally other variants later. This works seamlessly with shader caching. Needed to ensure that users only pay the cost for variants they use
2023-07-18 11:21:27 +02:00
// Map variant array index to group id, so we can iterate over groups later.
if (!group_to_variant_map.has(p_variant_defines[i].group)) {
group_to_variant_map.insert(p_variant_defines[i].group, LocalVector<int>{});
}
group_to_variant_map[p_variant_defines[i].group].push_back(i);
// Track max size.
if (p_variant_defines[i].group > max_group_id) {
max_group_id = p_variant_defines[i].group;
}
}
// Set all to groups to false, then enable those that should be default.
group_enabled.resize_zeroed(max_group_id + 1);
bool *enabled_ptr = group_enabled.ptrw();
for (int i = 0; i < p_variant_defines.size(); i++) {
if (p_variant_defines[i].default_enabled) {
enabled_ptr[p_variant_defines[i].group] = true;
}
}
if (!shader_cache_dir.is_empty()) {
group_sha256.resize(max_group_id + 1);
_initialize_cache();
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
}
}
void ShaderRD::set_shader_cache_dir(const String &p_dir) {
shader_cache_dir = p_dir;
}
void ShaderRD::set_shader_cache_save_compressed(bool p_enable) {
shader_cache_save_compressed = p_enable;
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
}
Implement shader caching * Shader compilation is now cached. Subsequent loads take less than a millisecond. * Improved game, editor and project manager startup time. * Editor uses .godot/shader_cache to store shaders. * Game uses user://shader_cache * Project manager uses $config_dir/shader_cache * Options to tweak shader caching in project settings. * Editor path configuration moved from EditorSettings to new class, EditorPaths, so it can be available early on (before shaders are compiled). * Reworked ShaderCompilerRD to ensure deterministic shader code creation (else shader may change and cache will be invalidated). * Added shader compression with SMOLV: https://github.com/aras-p/smol-v
2021-05-25 02:25:11 +02:00
void ShaderRD::set_shader_cache_save_compressed_zstd(bool p_enable) {
shader_cache_save_compressed_zstd = p_enable;
}
void ShaderRD::set_shader_cache_save_debug(bool p_enable) {
shader_cache_save_debug = p_enable;
}
String ShaderRD::shader_cache_dir;
bool ShaderRD::shader_cache_save_compressed = true;
bool ShaderRD::shader_cache_save_compressed_zstd = true;
bool ShaderRD::shader_cache_save_debug = true;
A lot of progress with canvas rendering, still far from working.
2019-06-16 04:45:24 +02:00
ShaderRD::~ShaderRD() {
List<RID> remaining;
version_owner.get_owned_list(&remaining);
if (remaining.size()) {
ERR_PRINT(itos(remaining.size()) + " shaders of type " + name + " were never freed");
while (remaining.size()) {
version_free(remaining.front()->get());
remaining.pop_front();
}
}
}
Reference in a new issue Copy permalink