Would World Space UI Be Possible?

[Doprez]

Hello, I was curious if there were any hints as to how or if 3D world space UI could be possible with this library?

From what I understand the rendering is done on a 2D plane that is reliant on the viewport height and width with no option for quaternion rotations. Unless I'm supposed to do the 3d calculations in my case for the StrideRenderer with the Draw methods?

[Doprez]

Ah! I may have realize this was a silly question and that the StrideRenderer is using a Stride SpriteBatch which hopefully should allow me to make some relatively small changes to the projection matrix to match a 3D plane.

OK, this is still out of my depths of knowledge but I think I get/see how it should work.

edit: https://github.com/stride3d/stride/blob/master/sources/engine/Stride.Graphics/Sprite3DBatch.cs 👀

[Doprez]

So I may need to implement this into a RootRenderFeature so that I can include the UI as a set of EntityComponents within Stride that will be added by a EntityProcessor.

Then I would likely need to include a list of SpriteBatches within the EmptyKeys StrideRenderer so that I can keep track of multiple Windows rather than a single instance.

My Garbage attempt so far 😆

using System.Collections.Generic;
using EmptyKeys.UserInterface.Media;
using Stride.Core.Mathematics;
using Stride.Engine;
using Stride.Games;
using Stride.Graphics;
using Stride.Rendering;
using Texture2D = Stride.Graphics.Texture;

namespace EmptyKeys.UserInterface.Renderers
{
    public class StrideRenderer3D : Renderer
    {
        private static GraphicsDeviceManager manager;
        private static GraphicsContext graphicsContext;

        /// <summary>
        /// The graphics device
        /// </summary>
        /// <value>
        /// The graphics device.
        /// </value>
        public static GraphicsDevice GraphicsDevice
        {
            get
            {
                return manager.GraphicsDevice;
            }
        }

        /// <summary>
        /// Gets or sets the graphics context.
        /// </summary>
        /// <value>
        /// The graphics context.
        /// </value>
        public static GraphicsContext GraphicsContext
        {
            get
            {
                return graphicsContext;
            }

            set
            {
                graphicsContext = value;
            }
        }

        private Entity _entity;
        private CameraComponent _camera;

        private Matrix view => _camera.ViewMatrix; //Matrix.LookAtRH(new Vector3(0.0f, 0.0f, 1.0f), Vector3.Zero, Vector3.UnitY);
        private Matrix projection;
        private Size activeViewportSize;

        private Sprite3DBatch spriteBatch;
        private Vector2 vecPosition;
        private Vector2 vecScale;
        private Vector2 origin;
        private Color vecColor;
        private Rectangle testRectangle;
        private Rectangle sourceRect;
        private Rectangle currentScissorRectangle;
        private Rectangle[] clipArray;
        private Stack<Rectangle> clipRectanges;
        private Stack<EffectInstance> activeEffects;
        private EffectInstance currentActiveEffect;
        private EffectSystem effectSystem;
        private StrideEffect sdfFontEffect;

        private bool isSpriteRenderInProgress;
        private bool isClipped;
        private Rectangle clipRectangle;
        private MutablePipelineState geometryPipelineState;
        private RasterizerStateDescription scissorRasterizerStateDescription;
        private RasterizerStateDescription geometryRasterizerStateDescription;

        /// <summary>
        /// Gets a value indicating whether is full screen.
        /// </summary>
        /// <value>
        /// <c>true</c> if is full screen; otherwise, <c>false</c>.
        /// </value>
        public override bool IsFullScreen
        {
            get { return manager.IsFullScreen; }
        }

        /// <summary>
        /// Initializes a new instance of the <see cref="StrideRenderer3D"/> class.
        /// </summary>
        /// <param name="graphicsDeviceManager">The graphics device manager.</param>
        public StrideRenderer3D(GraphicsDeviceManager graphicsDeviceManager, EffectSystem effectSystem, Entity entity, CameraComponent camera)
            : base()
        {
            _entity = entity;
            _camera = camera;

            manager = graphicsDeviceManager;
            this.effectSystem = effectSystem;
            spriteBatch = new Sprite3DBatch(manager.GraphicsDevice);
            clipRectanges = new Stack<Rectangle>();
            activeEffects = new Stack<EffectInstance>();

            scissorRasterizerStateDescription = RasterizerStates.CullNone;
            scissorRasterizerStateDescription.ScissorTestEnable = true; // enables the scissor test

            geometryRasterizerStateDescription = RasterizerStates.CullNone;
            //geometryRasterizerStateDescription.FillMode = FillMode.Wireframe;
            geometryPipelineState = new MutablePipelineState(manager.GraphicsDevice);
            geometryPipelineState.State.DepthStencilState = DepthStencilStates.None;

            clipArray = new Rectangle[1];
        }

        /// <summary>
        /// Begins the rendering
        /// </summary>
        public override void Begin()
        {
            Begin(null);
        }

        /// <summary>
        /// Begins the rendering with custom effect
        /// </summary>
        /// <param name="effect"></param>
        public override void Begin(EffectBase effect)
        {
            isClipped = false;
            isSpriteRenderInProgress = true;
            UpdateCurrentEffect(effect);

            if (clipRectanges.Count == 0)
            {
                spriteBatch.Begin(graphicsContext, view, SpriteSortMode.Deferred, depthStencilState: DepthStencilStates.None, effect: currentActiveEffect);
            }
            else
            {
                Rectangle previousClip = clipRectanges.Pop();
                BeginClipped(previousClip);
            }
        }

        private void UpdateCurrentEffect(EffectBase effect)
        {
            EffectInstance effectInstance = effect != null ? effect.GetNativeEffect() as EffectInstance : null;
            if (effectInstance != null)
            {
                if (currentActiveEffect != null)
                {
                    activeEffects.Push(currentActiveEffect);
                }

                currentActiveEffect = effectInstance;
            }

            if (currentActiveEffect == null && activeEffects.Count > 0)
            {
                currentActiveEffect = activeEffects.Pop();
            }
        }

        /// <summary>
        /// Ends rendering
        /// </summary>
        public override void End(bool endEffect = false)
        {
            isClipped = false;
            if (endEffect)
            {
                currentActiveEffect = null;
            }
            else
            {
                activeEffects.Push(currentActiveEffect);
                currentActiveEffect = null;
            }
                        
            spriteBatch.End();
            isSpriteRenderInProgress = false;
        }

        /// <summary>
        /// Begins the clipped rendering
        /// </summary>
        /// <param name="clipRect">The clip rect.</param>
        public override void BeginClipped(Rect clipRect)
        {            
            BeginClipped(clipRect, null);
        }

        /// <summary>
        /// Begins the clipped rendering with custom effect
        /// </summary>
        /// <param name="clipRect">The clip rect.</param>
        /// <param name="effect">The effect.</param>
        public override void BeginClipped(Rect clipRect, EffectBase effect)
        {
            clipRectangle.X = (int)clipRect.X;
            clipRectangle.Y = (int)clipRect.Y;
            clipRectangle.Width = (int)clipRect.Width;
            clipRectangle.Height = (int)clipRect.Height;

            UpdateCurrentEffect(effect);

            BeginClipped(clipRectangle);
        }

        private void BeginClipped(Rectangle clipRect)
        {
            isClipped = true;
            isSpriteRenderInProgress = true;

            if (clipRectanges.Count > 0)
            {
                Rectangle previousClip = clipRectanges.Pop();
                if (previousClip.Intersects(clipRect))
                {
                    clipRect = Rectangle.Intersect(previousClip, clipRect);
                }
                else
                {
                    clipRect = previousClip;
                }

                clipRectanges.Push(previousClip);
            }

            clipArray[0] = clipRect;
            graphicsContext.CommandList.SetScissorRectangles(1, clipArray);
            
            currentScissorRectangle = clipRect;
            spriteBatch.Begin(graphicsContext, view, SpriteSortMode.Deferred, depthStencilState: DepthStencilStates.None, rasterizerState: scissorRasterizerStateDescription, effect: currentActiveEffect);
            clipRectanges.Push(clipRect);
        }

        /// <summary>
        /// Ends the clipped rendering
        /// </summary>
        public override void EndClipped(bool endEffect = false)
        {
            isClipped = false;
            isSpriteRenderInProgress = false;
            if (endEffect)
            {
                currentActiveEffect = null;
            }
            else
            {
                activeEffects.Push(currentActiveEffect);
                currentActiveEffect = null;
            }

            spriteBatch.End();
            clipRectanges.Pop();
        }

        /// <summary>
        /// Draws the text.
        /// </summary>
        /// <param name="font">The font.</param>
        /// <param name="text">The text.</param>
        /// <param name="position">The position.</param>
        /// <param name="renderSize">Size of the render.</param>
        /// <param name="color">The color.</param>
        /// <param name="scale">The scale.</param>
        /// <param name="depth">The depth.</param>
        public override void DrawText(FontBase font, string text, PointF position, Size renderSize, ColorW color, PointF scale, float depth)
        {
            if (isClipped)
            {
                testRectangle.X = (int)position.X;
                testRectangle.Y = (int)position.Y;
                testRectangle.Width = (int)renderSize.Width;
                testRectangle.Height = (int)renderSize.Height;

                if (!currentScissorRectangle.Intersects(testRectangle))
                {
                    return;
                }
            }

            vecPosition.X = position.X;
            vecPosition.Y = position.Y;
            vecScale.X = scale.X;
            vecScale.Y = scale.Y;
            vecColor.A = color.A;
            vecColor.R = color.R;
            vecColor.G = color.G;
            vecColor.B = color.B;
            SpriteFont native = font.GetNativeFont() as SpriteFont;
            //spriteBatch.DrawString(native, text, vecPosition, vecColor);
        }

        /// <summary>
        /// Draws the specified texture.
        /// </summary>
        /// <param name="texture">The texture.</param>
        /// <param name="position">The position.</param>
        /// <param name="renderSize">Size of the render.</param>
        /// <param name="color">The color.</param>
        /// <param name="centerOrigin">if set to <c>true</c> [center origin].</param>
        public override void Draw(TextureBase texture, PointF position, Size renderSize, ColorW color, bool centerOrigin)
        {
            testRectangle.X = (int)position.X;
            testRectangle.Y = (int)position.Y;
            testRectangle.Width = (int)renderSize.Width;
            testRectangle.Height = (int)renderSize.Height;
            if (centerOrigin)
            {
                testRectangle.X -= testRectangle.Width / 2;
                testRectangle.Y -= testRectangle.Height / 2;
            }

            if (isClipped && !currentScissorRectangle.Intersects(testRectangle))
            {
                return;
            }

            vecColor.A = color.A;
            vecColor.R = color.R;
            vecColor.G = color.G;
            vecColor.B = color.B;
            Texture2D native = texture.GetNativeTexture() as Texture2D;

            var elementSize = new Vector2() { X = native.Size.Width, Y = native.Size.Height };

            var rectangleF = new RectangleF()
            {
                Height = testRectangle.Height,
                Width = testRectangle.Width,
                Left = testRectangle.Left,
                Top = testRectangle.Top,
            };

            var colour = new Color4()
            {
                A = color.A,
                R = color.R,
                G = color.G,
                B = color.B,
            };

            spriteBatch.Draw(native, ref _entity.Transform.WorldMatrix, ref rectangleF, ref elementSize, ref colour);
        }

        /// <summary>
        /// Draws the specified texture.
        /// </summary>
        /// <param name="texture">The texture.</param>
        /// <param name="position">The position.</param>
        /// <param name="renderSize">Size of the render.</param>
        /// <param name="color">The color.</param>
        /// <param name="source">The source.</param>
        /// <param name="centerOrigin">if set to <c>true</c> [center origin].</param>
        public override void Draw(TextureBase texture, PointF position, Size renderSize, ColorW color, Rect source, bool centerOrigin)
        {
            testRectangle.X = (int)position.X;
            testRectangle.Y = (int)position.Y;
            testRectangle.Width = (int)renderSize.Width;
            testRectangle.Height = (int)renderSize.Height;
            if (isClipped && !currentScissorRectangle.Intersects(testRectangle))
            {
                return;
            }

            sourceRect.X = (int)source.X;
            sourceRect.Y = (int)source.Y;
            sourceRect.Width = (int)source.Width;
            sourceRect.Height = (int)source.Height;
            vecColor.A = color.A;
            vecColor.R = color.R;
            vecColor.G = color.G;
            vecColor.B = color.B;
            if (centerOrigin)
            {
                origin.X = testRectangle.Width / 2f;
                origin.Y = testRectangle.Height / 2f;
            }

            Texture2D native = texture.GetNativeTexture() as Texture2D;

            var elementSize = new Vector2() { X = native.Size.Width, Y = native.Size.Height };

            var rectangleF = new RectangleF()
            {
                Height = testRectangle.Height,
                Width = testRectangle.Width,
                Left = testRectangle.Left,
                Top = testRectangle.Top,
            };

            var colour = new Color4()
            {
                A = color.A,
                R = color.R,
                G = color.G,
                B = color.B,
            };

            spriteBatch.Draw(native, ref _entity.Transform.WorldMatrix, ref rectangleF, ref elementSize, ref colour);
        }

        /// <summary>
        /// Gets the viewport.
        /// </summary>
        /// <returns></returns>
        public override Rect GetViewport()
        {
            Viewport viewport = graphicsContext.CommandList.Viewport;
            return new Rect(viewport.X, viewport.Y, viewport.Width, viewport.Height);
        }

        /// <summary>
        /// Creates the texture.
        /// </summary>
        /// <param name="nativeTexture">The native texture.</param>
        /// <returns></returns>
        public override TextureBase CreateTexture(object nativeTexture)
        {
            if (nativeTexture == null)
            {
                return null;
            }

            return new StrideTexture3D(nativeTexture);
        }

        /// <summary>
        /// Creates the texture.
        /// </summary>
        /// <param name="width">The width.</param>
        /// <param name="height">The height.</param>
        /// <param name="mipmap">if set to <c>true</c> [mipmap].</param>
        /// <param name="dynamic">if set to <c>true</c> [dynamic].</param>
        /// <returns></returns>
        public override TextureBase CreateTexture(int width, int height, bool mipmap, bool dynamic)
        {
            if (width == 0 || height == 0)
            {
                return null;
            }

            Texture2D native = null;
            if (dynamic)
            {
                native = Texture2D.New2D(GraphicsDevice, width, height, PixelFormat.R8G8B8A8_UNorm, usage: GraphicsResourceUsage.Dynamic);
            }
            else
            {
                native = Texture2D.New2D(GraphicsDevice, width, height, PixelFormat.R8G8B8A8_UNorm);
            }

            StrideTexture3D texture = new(native);
            return texture;
        }

        /// <summary>
        /// Creates the geometry buffer.
        /// </summary>
        /// <returns></returns>
        public override GeometryBuffer CreateGeometryBuffer()
        {
            return new StrideGeometryBuffer3D();
        }

        /// <summary>
        /// Draws the color of the geometry.
        /// </summary>
        /// <param name="buffer">The buffer.</param>
        /// <param name="position">The position.</param>
        /// <param name="color">The color.</param>
        /// <param name="opacity">The opacity.</param>
        /// <param name="depth">The depth.</param>
        public override void DrawGeometryColor(GeometryBuffer buffer, PointF position, ColorW color, float opacity, float depth)
        {
            StrideGeometryBuffer3D StrideBuffer = buffer as StrideGeometryBuffer3D;

            Color4 nativeColor = new Color4(color.PackedValue) * opacity;
            StrideBuffer.EffectInstance.Parameters.Set(SpriteEffectKeys.Color, nativeColor);
            StrideBuffer.EffectInstance.Parameters.Set(TexturingKeys.Texture0, GraphicsDevice.GetSharedWhiteTexture());
            DrawGeometry(buffer, position, depth);
        }

        /// <summary>
        /// Draws the geometry texture.
        /// </summary>
        /// <param name="buffer">The buffer.</param>
        /// <param name="position">The position.</param>
        /// <param name="texture">The texture.</param>
        /// <param name="opacity">The opacity.</param>
        /// <param name="depth">The depth.</param>
        public override void DrawGeometryTexture(GeometryBuffer buffer, PointF position, TextureBase texture, float opacity, float depth)
        {
            StrideGeometryBuffer3D paradoxBuffer = buffer as StrideGeometryBuffer3D;
            Texture2D nativeTexture = texture.GetNativeTexture() as Texture2D;
            paradoxBuffer.EffectInstance.Parameters.Set(SpriteEffectKeys.Color, Color.White * opacity);
            paradoxBuffer.EffectInstance.Parameters.Set(TexturingKeys.Texture0, nativeTexture);
            DrawGeometry(buffer, position, depth);
        }

        private void UpdateProjection(CommandList commandList)
        {
            bool sameViewport = activeViewportSize.Width == commandList.Viewport.Width && activeViewportSize.Height == commandList.Viewport.Height;
            if (!sameViewport)
            {                
                activeViewportSize = new Size(commandList.Viewport.Width, commandList.Viewport.Height);
                projection = _camera.ProjectionMatrix; //Matrix.OrthoOffCenterRH(0, commandList.Viewport.Width, commandList.Viewport.Height, 0, 1.0f, 1000.0f);
            }
        }

        private void DrawGeometry(GeometryBuffer buffer, PointF position, float depth)
        {
            if (isSpriteRenderInProgress)
            {
                spriteBatch.End();
            }

            Matrix world = Matrix.Translation(position.X, position.Y, 0);

            Matrix worldView;
            Matrix.Multiply(ref world, ref _camera.ViewMatrix, out worldView);
            
            Matrix worldViewProjection;
            UpdateProjection(graphicsContext.CommandList);
            Matrix.Multiply(ref worldView, ref _camera.ProjectionMatrix, out worldViewProjection);

            StrideGeometryBuffer3D paradoxBuffer = buffer as StrideGeometryBuffer3D;
            paradoxBuffer.EffectInstance.Parameters.Set(SpriteBaseKeys.MatrixTransform, worldViewProjection);
            
            if (isClipped)
            {
                geometryPipelineState.State.RasterizerState = scissorRasterizerStateDescription;
            }
            else
            {
                geometryPipelineState.State.RasterizerState = geometryRasterizerStateDescription;
            }

            switch (buffer.PrimitiveType)
            {
                case GeometryPrimitiveType.TriangleList:
                    geometryPipelineState.State.PrimitiveType = PrimitiveType.TriangleList;
                    break;
                case GeometryPrimitiveType.TriangleStrip:
                    geometryPipelineState.State.PrimitiveType = PrimitiveType.TriangleStrip;
                    break;
                case GeometryPrimitiveType.LineList:
                    geometryPipelineState.State.PrimitiveType = PrimitiveType.LineList;
                    break;
                case GeometryPrimitiveType.LineStrip:
                    geometryPipelineState.State.PrimitiveType = PrimitiveType.LineStrip;
                    break;
                default:
                    break;
            }

            geometryPipelineState.State.RootSignature = paradoxBuffer.EffectInstance.RootSignature;
            geometryPipelineState.State.EffectBytecode = paradoxBuffer.EffectInstance.Effect.Bytecode;
            geometryPipelineState.State.InputElements = paradoxBuffer.InputElementDescriptions;
            geometryPipelineState.State.Output.CaptureState(graphicsContext.CommandList);
            geometryPipelineState.Update();
            graphicsContext.CommandList.SetPipelineState(geometryPipelineState.CurrentState);
            paradoxBuffer.EffectInstance.Apply(graphicsContext);

            graphicsContext.CommandList.SetVertexBuffer(0, paradoxBuffer.VertexBufferBinding.Buffer, 0, paradoxBuffer.VertexBufferBinding.Stride);
            graphicsContext.CommandList.Draw(paradoxBuffer.PrimitiveCount);

            if (isSpriteRenderInProgress)
            {
                if (isClipped)
                {
                    spriteBatch.Begin(graphicsContext, view, SpriteSortMode.Deferred, 
                        depthStencilState: DepthStencilStates.None, rasterizerState: scissorRasterizerStateDescription, effect: currentActiveEffect);
                }
                else
                {
                    spriteBatch.Begin(graphicsContext, view, SpriteSortMode.Deferred, depthStencilState: DepthStencilStates.None, effect: currentActiveEffect);
                }
            }
        }

        /// <summary>
        /// Creates the font.
        /// </summary>
        /// <param name="nativeFont">The native font.</param>
        /// <returns></returns>
        public override FontBase CreateFont(object nativeFont)
        {
            return new StrideFont(nativeFont);
        }

        /// <summary>
        /// Resets the size of the native. Sets NativeScreenWidth and NativeScreenHeight based on active back buffer
        /// </summary>
        public override void ResetNativeSize()
        {
        }

        /// <summary>
        /// Determines whether the specified rectangle is outside of clip bounds
        /// </summary>
        /// <param name="position">The position.</param>
        /// <param name="renderSize">Size of the render.</param>
        /// <returns></returns>
        public override bool IsClipped(PointF position, Size renderSize)
        {
            if (isClipped)
            {
                testRectangle.X = (int)position.X;
                testRectangle.Y = (int)position.Y;
                testRectangle.Width = (int)renderSize.Width;
                testRectangle.Height = (int)renderSize.Height;

                if (!currentScissorRectangle.Intersects(testRectangle))
                {
                    return true;
                }
            }

            return false;
        }

        /// <summary>
        /// Creates the effect.
        /// </summary>
        /// <param name="nativeEffect">The native effect.</param>
        /// <returns></returns>
        public override EffectBase CreateEffect(object nativeEffect)
        {
            return new StrideEffect(nativeEffect, null);
        }

        /// <summary>
        /// Gets the SDF font effect.
        /// </summary>
        /// <returns></returns>
        public override EffectBase GetSDFFontEffect()
        {
            if (sdfFontEffect == null)
            {                
                Effect effect = effectSystem.LoadEffect("SDFFontShader").WaitForResult();
                if (effect != null)
                {
                    ParameterCollection parameters = new ParameterCollection();
                    parameters.Set<Color4>(SDFFontShaderKeys.TintColor, Color4.White);
                    parameters.Set<Color4>(SDFFontShaderKeys.BorderColor, Color4.Black);
                    parameters.Set<float>(SDFFontShaderKeys.BorderThickness, 0f);

                    sdfFontEffect = new StrideEffect(effect, parameters);
                }
            }

            return sdfFontEffect;
        }
    }
}

I havent done the second part yet as I want to make sure it will work as a completely separate class first to remove any confusion on the base infrastructure/logic.

[EmptyKeys]

The biggest issue with 3D World UI would be the fact that only one screen can be active right now. It's because of the cache of control tree, which is required.

Maybe it could be done with Windows somehow, as those can exists separate from the screen itself.

[Doprez]

ah ok, I think that explains the bigger issue I left this on then. I only need the on screen UI currently either way so I will leave this alone for now as a possible fun future idea.

If I properly understand what you said though, I would need to either:

1. Create multiple rendered windows as independent UIs
   a. Get the render texture from them
   b. Render onto a Spritebatch in Stride (Im assuming threading would be a problem to handle carefully)

2. Create multiple rendered windows as independent UIs
   a. Get the handle of the Stride GameWindow
   b. Use windowing APIs to attach the separate EmptyKeys UI to the game.

Thank you, for confirming!