Horizon 8 - 2209 - Issues with offloadig video streams to GPU.

[Lansti]

Hi. I've troubleshooting usage of GPU in Horizon VDI Environment for quite a while now.

We have Nvidia T4, A16 and M10 cards, and streaming video from Chrome or Edge are not working well.
Especially if I have 3-4 edge windows open and video in one of these, and live updates in another window.  
Video just won't play properly, and pretty frustrating for end-users.

Did i mention that we work in one of the biggests broadcasting companies in Norway?

I've been testing with 2b profile and 4q profile on our nvidia cards, and the drivers on the clients are equal the drivers on the hosts.

I have deployed 2 desktop pools for our test - Non-presistant desktops.
Pool 1: M10_2b profile. 16 CPU (4 sockets) and 36GB memory.
When i open 3 different Edge or Chrome, video streams stops, and edge telling me that there are lots of frame drops...

Pool 2: Same as pool 1, but with no GPU at all.
        Here I can start several edge windows and stream video without problem. But i see that CPU consumption is offcourse pretty high.
        Seems to work better without expensive Nvida cards and licenses.

Endpoint for our users are Dell Optiplex 3000 Thin Client:
Dell ThinOS 2402 (9.5.1079)
VMware Horizon Client: Version 2309.8.11.0.22660930.37

Broker settings: Connection Protocol - Defaut using Vmawre Blast

Our test station have 2 monitors connected - HPE272q with resolution 2560*1440. But we also have users with 3 monitors.

On this test, we are trying to run 3 Edge windows with video stream. But they are all hanging.

Hardware acceleration is turned on by policy.
I can verify that the client has the correct GPU license.

Our desktops are deployed with MS MDT, and we are running OSOT within the task sequence.
Windows OS Optimization Tool for VMware Horizon (1.1.2204.19587979)



So, why does the video stream work better without GPU?
I have also tried the following:
https://www.stephenwagner.com/2024/01/05/gpu-issues-vmware-horizon-indirect-display-driver/

Can anyone point me in the right direction for further troubleshooting?

Edited Wednesday at 09:46 AM by Lansti

[Dominik]

Hello @Lansti,

basic question but important, did you check is hardware acceleration is enable in Chrome/Edge? 

[Lansti]

hardware acceleration is enabled...
And i have also tried to disable it...
Must say, not much difference since it seems that the offloading do not work properly...

[StephenWagner7]

7 hours ago, Lansti said:

Hi. I've troubleshooting usage of GPU in Horizon VDI Environment for quite a while now.

We have Nvidia T4, A16 and M10 cards, and streaming video from Chrome or Edge are not working well.
Especially if I have 3-4 edge windows open and video in one of these, and live updates in another window.  
Video just won't play properly, and pretty frustrating for end-users.

Did i mention that we work in one of the biggests broadcasting companies in Norway?

I've been testing with 2b profile and 4q profile on our nvidia cards, and the drivers on the clients are equal the drivers on the hosts.

I have deployed 2 desktop pools for our test.
Pool 1: M10_2b profile. 16 CPU (4 sockets) and 36GB memory.
When i open 3 different Edge or Chrome, video streams stops, and edge telling me that there are lots of frame drops...

Pool 2: Same as pool 1, but with no GPU at all.
        Here I can start several edge windows and stream video without problem. But i see that CPU consumption is offcourse pretty high.
        Seems to work better without expensive Nvida cards and licenses.

So, why does the video stream work better without GPU?
I have also tried the following:
https://www.stephenwagner.com/2024/01/05/gpu-issues-vmware-horizon-indirect-display-driver/

Can anyone point me in the right direction for further troubleshooting?

HI @Lansti,

Are these on persistent or non-persistent VDI? Is there a chance that the OSOT tool has been previously ran and disabled some of the graphic capabilities?

Can you share what you're using for endpoints (client devices)? Also can you share how many displays each endpoint has and the resolution they are running at?

Can you confirm that the your VDI VMs are showing as licensed with vGPU (as non-licensed loses feature after brief interval after logon)?

Also, can you confirm that you have hardware encoding/decoding end to end? Can you paste a screenshot of your Horizon Performance tracker and what encoder is being used?

Cheers,

Stephen

[Jubish Jose]

Basic question -  are the GPUs licensed?

nvidia-smi -q

[Lansti]

Hi @Dominik, @StephenWagner7 and @Jubish Jose.
My apologize for my first incomplete post.

I have updated my original post here, hope you guys will have a look into it.

Edited Wednesday at 09:54 AM by Lansti

[Dominik]

@Lansti important question was asked by @Jubish Jose about license, because without license you don't have powerful vGPU. 

Edit:

sorry, I saw info now in your main post about license.

Edited Wednesday at 11:10 AM by Dominik

[StephenWagner7]

2 hours ago, Lansti said:

Hi @Dominik, @StephenWagner7 and @Jubish Jose.
My apologize for my first incomplete post.

I have updated my original post here, hope you guys will have a look into it.

Are you using any of the HTML or Media redirection capabilities with the browser?

When you look at your task manager, can you see what's using CPU?

Also, have you verified that the codec that the web viewers are using is supported by the GPU for decoding (I'm talking about the website, and video streams you're viewing, not the VDI codec)?

[Lansti]

3 minutes ago, StephenWagner7 said:

Are you using any of the HTML or Media redirection capabilities with the browser?

When you look at your task manager, can you see what's using CPU?

Also, have you verified that the codec that the web viewers are using is supported by the GPU for decoding (I'm talking about the website, and video streams you're viewing, not the VDI codec)?

Hi @StephenWagner7, in Horizon we allow Multimedia Redirection policy, i have alsto tried to deny it, in the testing. I really do not see the difference.
I have also tried the HTML 5 REDR poicy with chrome/edge extension. It seems to work, but you get a "funny look" in the VDI when you hover other applications over the browser. Applicatins get a sort of a frame, and video will play in front of everything. We cannot use that...

In the task manager, Edge are using some of the CPU. I can see that GPU 0 also consuming about 15-28%, but somehow it drops when the edge or chrome browser stops playing.
I have also had a look at CTRL + SHIFT + i, to record the performance and i can see that lots of frames are dropping.


I acn see that there are lots of Edge processes

These processes with GPU Motor sometime changes to GPU 0 - Video Decode.

I hae not verified that the codecs from the websites are supported by the GPU for decoding. I never thought that there could be the reason why it doesn't work.
I would think that most of the big services out there uses a codec that is supported?

 

[Dominik]

@Lansti did you check profile 2Q? I see on screenshot you use B profile. In my case I'm using Q profile. 

[Lansti]

Hi, yes, we have cluster of clients using 2B profiles, and we have separate cluster with clients using 4Q profiles.. 
And we see the same issue on both profiles.

But as i mentioned in the main post, when i created pool 2 without GPU, but added a lot of CPU, that works much better than a client with GPU, but that's not a solution...
So there are something wrong with the offloading to GPUs

[StephenWagner7]

1 hour ago, Lansti said:

Hi, yes, we have cluster of clients using 2B profiles, and we have separate cluster with clients using 4Q profiles.. 
And we see the same issue on both profiles.

But as i mentioned in the main post, when i created pool 2 without GPU, but added a lot of CPU, that works much better than a client with GPU, but that's not a solution...
So there are something wrong with the offloading to GPUs

Do your endpoints support hardware offload for decoding, or are they decoding the BLAST session using CPU?

Also, does the server(s) hosting your VDI have enough bandwidth to your network (and to your endpoints) on the networking side of things?

CPU usage on your VDI VMs may not be an issue, because you're using browsers which are CPU/MEM hogs in any event, but the visual performance should be better.

[Sean Massey-1]

These are challenging issues to troubleshoot.  You have the interaction between the Web Browser, Windows, the Horizon Agent and the NVIDIA drivers.

What version of Windows are you using?  I didn't see that listed in the original post?

Have you opened a tickets with NVIDIA and Omnissa support?

I'm also thinking that the M10 might be part of your issues here.  The M10 is basically 4 lower-powered Maxwell GPUs glued onto the same PCI card, which is a bit older.  It also has a max limit of 28 simultaneous 1080P H.264 streams. Do you see these issues when you're only running 1 or 2 videos?

 

[Lansti]

19 hours ago, Sean Massey-1 said:

These are challenging issues to troubleshoot.  You have the interaction between the Web Browser, Windows, the Horizon Agent and the NVIDIA drivers.

What version of Windows are you using?  I didn't see that listed in the original post?

Have you opened a tickets with NVIDIA and Omnissa support?

I'm also thinking that the M10 might be part of your issues here.  The M10 is basically 4 lower-powered Maxwell GPUs glued onto the same PCI card, which is a bit older.  It also has a max limit of 28 simultaneous 1080P H.264 streams. Do you see these issues when you're only running 1 or 2 videos?

 

Hi @Sean Massey-1, we are running Windows 10, 22H2.
We have discussed this issue with Nvidia for a long time ago, but we did not get any further..

I wantet to see if any of you guys have the same issues, or know where to look.
I assume i need to upgrade my nvidia driver to the latest, and upgrade Horizon to the latest before i can open a support ticket with nvidia and Omnissa...

M10 is our "backup"... As you can see i have been testing on our T4 with 2B profiles, and A16 with 4Q profiles.
Same behavior with browser that stop playing streams..
Running 1 video stream seems to be fine, and if I start a second one, I'm not able to do much other thing on my windows session before these streams stops..

I have users that vil have 4 chrome/edge windows up at the same time, video stream on one half of a screen, some live updates on the second part of the screen, and statistics on the one half of the second screen and their publishing site on the last part of the screen. So there is a lot of movement, and pages that are updated continuously

[Lansti]

20 hours ago, StephenWagner7 said:

Do your endpoints support hardware offload for decoding, or are they decoding the BLAST session using CPU?

Our endpoint have Intel Pentium Silver N6005 with Intel UHD Graphics.

20 hours ago, StephenWagner7 said:

Also, does the server(s) hosting your VDI have enough bandwidth to your network (and to your endpoints) on the networking side of things?

there are 10Gig from server to switch and then 1gig from switch to endpoint - Dell Optiplex 3000 Thin client.
No firewall between endpoint and VDI vms...

[StephenWagner7]

@Lansti, you'll need to verify, but those endpoints may or may not be enough to handle hardware decoding.

It sounds like you need to either keep working on the tickets with NVIDIA and Omnissa, or bring in a 3rd party to analyze your infrastructure and see what they can find. It's extremely difficult to troubleshoot this over forum threads.

[Sean Massey-1]

4 hours ago, Lansti said:

Hi @Sean Massey-1, we are running Windows 10, 22H2.
We have discussed this issue with Nvidia for a long time ago, but we did not get any further..

I wantet to see if any of you guys have the same issues, or know where to look.
I assume i need to upgrade my nvidia driver to the latest, and upgrade Horizon to the latest before i can open a support ticket with nvidia and Omnissa...

M10 is our "backup"... As you can see i have been testing on our T4 with 2B profiles, and A16 with 4Q profiles.
Same behavior with browser that stop playing streams..
Running 1 video stream seems to be fine, and if I start a second one, I'm not able to do much other thing on my windows session before these streams stops..

I have users that vil have 4 chrome/edge windows up at the same time, video stream on one half of a screen, some live updates on the second part of the screen, and statistics on the one half of the second screen and their publishing site on the last part of the screen. So there is a lot of movement, and pages that are updated continuously

So if I understand the issue correctly, your users are running multiple web browser sessions playing multiple video streams across multiple monitors?  And the session becomes unusable after starting multiple video streams?

In one of the earlier posts, you said that you get a warning that there are frames being dropped.  Where are you seeing this warning?  Can you share a screenshot of it?

To the point that @StephenWagner7 brought up - this could be an endpoint issue.  Have you tested this using other endpoints to eliminate that as the possible root cause?  I'm not sure this is the case, but it is worth testing just to eliminate this as an issue.

There are a couple of other things you can do to gather more information.  One of these items will need to be monitored from the host level over SSH, and the other can be done with vCenter.

1. Log into your ESXi Server Shell and run nvidia-smi while your desktop is trying to play multiple video streams.  This may give you some additional information on per-VM GPU utilization. You can also run nvidia-smi inside your desktop session to see which processes are using the GPU.

2. You mentioned that your VM has 16 vCPUs.  This is a lot of vCPUs. You might be hitting CPU Ready or Co-Stop issues with your workload.  You can look at CPU Ready % and Co-Stop on the Performance Tab in vCenter.  This Broadcom KB can help you with interpreting the CPU Ready results: https://knowledge.broadcom.com/external/article/306576/converting-between-cpu-summation-and-cpu.html

This is kind of a long shot...but have you tried reducing the number of vCPUs on your desktop?  Maybe cut it down to 8 vCPUs. 

What version of Horizon and the NVIDIA drivers do you have deployed today?  I would still open a ticket even if you're not on the latest version to get the support process started, and having your vendors say you need to upgrade can help move things through any change control processes.