End user experience testing - the EUC Score Sync Player interface

● 1 Cloud and Instance type (e.g. Azure NC8asT4_V3)

● 2 Latency and network bandwidth

● 3 Click for detailed information about the VM specs, connection and endpoint

● 4 Click to maximimise the viewport

● 5 Viewport playback (examine for compression / responsiveness to mouse movements, etc.)

● 6 Task Manager showing resources used at the endpoint (not the cloud workstation)

● 7 Timeline (play back in real time or scrub up and down, as necessary) end-user experience in remote desktops and applications. By capturing the real user experience in a high-quality video on the client device of a 3D application in use, it shows what the end user is really experiencing and puts it in the context of a whole variety of telemetry data. This could be memory, GPU or CPU utilisation, remoting protocol statistics or network insights such as bandwidth, network latency or the amount of compression being applied to the video stream. The big benefit of the EUC Score Sync Player is that it brings telemetry data and the captured real user experience video together in a single environment.

When armed with this information, IT architects and directors can get a much better understanding of the impact of different VMs / network conditions on

● a Actual CPU utilisation

● b Quantisation Priority (QP) (level of compression being applied to the video stream)

● c Performance in the viewport (Frames Per Second)

● d Round trip network latency

● e Actual network bandwidth used

● f Actual GPU utilisation of Nvidia GPUs with select applications

● g GPU memory utilisation

● h GPU utilisation for encoding the video stream (H.264) end user experience, and size everything accordingly. In addition, if a user complains about their experience, it can help identify what’s wrong. After all, there’s no point in giving someone a more powerful VM, if it’s the network that’s causing the problem or the remoting protocol can’t deliver the best user experience.

For EUC testing, we selected a handful of

the Frame website (https://ux.fra.me)

EUC Score Sync Player is able to display eight different types of telemetry data at the same time, so that’s why there are different views of the telemetry data. The generic ‘Frame’ recordings are a good starting point, but you can also dig down into more detail in ‘CPU’ and ‘GPU’.

different VMs from our list of 23. We tested our 3D apps at FHD and 4K resolution using a special hardware device that simulates different network conditions.

The results are best absorbed by watching the captured videos and telemetry data, which can all be seen on

When watching the recordings, here are some things to look out for. Round trip latency is important and when this is high (anything over 100ms) it can take a while for the VM to respond to mouse and keyboard input and for the stream to come back. Any delay can make the system feel laggy, and hard to position 3D models quickly and accurately on screen. And, if you keep overshooting, it can have a massive impact on modelling productivity.

In low-bandwidth, higher latency conditions (anything below 8 Mbps) the video stream might need to be heavily compressed. As this compression is ‘lossy’ and not ‘lossless’ it can cause visual compression artefacts, which is not ideal for precise CAD work. In saying that, the Frame Remoting Protocol 8 (FRP8) Quality of Service engine does do a great job and resolves to full high-quality once you stop moving the 3D model around. Compression might be more apparent at 4K resolution than at FHD resolution, as there are four times as many pixels, meaning much more data to send.

Frame, like most graphics-optimised remoting protocols, will automatically adapt to network conditions to maintain interactivity. EUC Score not only gives you a visual reference to this compression by recording the user experience, but it also quantifies the amount of compression being applied by FRP8 to the video stream through a metric called Quantisation Priority (QP). The lower the number, the less visual compression artefacts you will see. However, the lowest you can get is 12, as to the end user this appears to be visually lossless. This highest you can get is 50 which is super blurry.

Visual compression should not be confused with Revit’s ‘simplify display during view navigation’ feature that suspends certain details and graphics effects to maintain 3D performance. In the EUC Score player you can see this in action with textures and shadows temporarily disappearing when the model is moving. In other CAD tools this is known as Level of Detail (LoD).

The recordings can also give some valuable insight into how much each application uses the GPU. Enscape and Unreal Engine, for example, utilise 100% of GPU resources so you can be certain that a more powerful GPU would boost 3D performance (in Unreal Engine, EUC Score records this with a special Nvidia GPU usage counter).

Meanwhile, GPU utilisation in Revit and Inventor is lower, so if your graphics performance is poor or you want to turn off LoD you may be better off with a CPU with a higher frequency or better IPC than a more powerful GPU.

To help find your way around the EUC Score interface, see Figure 1. In Figures 2 and 3 we show the impact of network bandwidth on visual compression

EUC Score test results

artefacts. This could be a firm that does not have sufficient bandwidth to support tens, hundreds, or thousands of cloud workstation users or when the kids come home from school, and they all start streaming Netflix.

Conclusion

If you’re a design or manufacturing firm looking into public cloud workstations for CAD or design visualisation, we hope this article has given you a good starting point for your own internal testing, something we’d always strongly recommend.

There is no one size fits all for cloud workstations and some of the instances we’ve tested make no sense for certain workflows, especially at 4K resolution. This isn’t just about applications. While it’s important to understand the demands of different tools, dataset complexity and size can also have a massive impact on performance, especially with 3D graphics at 4K resolution. What’s good for one firm, certainly might not be good for another.

Also be aware that some of the public cloud VMs are much older than others. If you consider that firms typically upgrade their desktop workstations every 3 to 5 years, a few are positively ancient. The great news about the cloud is that you can change VMs whenever you like. New machines come online, and prices change, as do your applications and workflows. But unlike desktops, you’re not stuck with a purchasing decision.

While design and engineering firms will always be under pressure to drive down costs, performance is essential. A slow workstation can have a massive negative impact on productivity and morale, even worse if it crashes. Make sure you test, test and test again, using data from your own real-world projects.

For more details, insights or advice, feel free to contact to Ruben Spruijt (ruben@fra.me) or Bernhard Tritsch (btritsch@bennytritsch.com).