Lansti Posted September 17 Share Posted September 17 (edited) Hi. I've troubleshooting usage of GPU in Horizon VDI Environment for quite a while now. We have Nvidia T4, A16 and M10 cards, and streaming video from Chrome or Edge are not working well. Especially if I have 3-4 edge windows open and video in one of these, and live updates in another window. Video just won't play properly, and pretty frustrating for end-users. Did i mention that we work in one of the biggests broadcasting companies in Norway? I've been testing with 2b profile and 4q profile on our nvidia cards, and the drivers on the clients are equal the drivers on the hosts. I have deployed 2 desktop pools for our test - Non-presistant desktops. Pool 1: M10_2b profile. 4 CPU 2 cores pr socket. 20GB RAM When i open 3 different Edge or Chrome, video streams stops, and edge telling me that there are lots of frame drops... Pool 2: Same Master with another HW config - added CPU and memory: 16 CPU (4 sockets) and 36GB memory, and removed GPU card. Here I can start several edge windows and stream video without problem. But i see that CPU consumption is offcourse pretty high. Seems to work better without expensive Nvida cards and licenses. Endpoint for our users are Dell Optiplex 3000 Thin Client: Dell ThinOS 2402 (9.5.1079) VMware Horizon Client: Version 2309.8.11.0.22660930.37 Broker settings: Connection Protocol - Defaut using Vmawre Blast Our test station have 2 monitors connected - HPE272q with resolution 2560*1440. But we also have users with 3 monitors. On this test, we are trying to run 3 Edge windows with video stream. But they are all hanging. Hardware acceleration is turned on by policy. I can verify that the client has the correct GPU license. Our desktops are deployed with MS MDT, and we are running OSOT within the task sequence. Windows OS Optimization Tool for VMware Horizon (1.1.2204.19587979) So, why does the video stream work better without GPU? I have also tried the following: https://www.stephenwagner.com/2024/01/05/gpu-issues-vmware-horizon-indirect-display-driver/ Can anyone point me in the right direction for further troubleshooting? Edited September 23 by Lansti Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Dominik Posted September 17 Share Posted September 17 Hello @Lansti, basic question but important, did you check is hardware acceleration is enable in Chrome/Edge? Quote Dominik Jakubowski EUC Expert | vExpert ⭐️⭐️⭐️ VDI Ninja https://vdesktop.ninja Link to comment Share on other sites More sharing options...
Lansti Posted September 17 Author Share Posted September 17 hardware acceleration is enabled... And i have also tried to disable it... Must say, not much difference since it seems that the offloading do not work properly... Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
StephenWagner7 Posted September 17 Share Posted September 17 7 hours ago, Lansti said: Hi. I've troubleshooting usage of GPU in Horizon VDI Environment for quite a while now. We have Nvidia T4, A16 and M10 cards, and streaming video from Chrome or Edge are not working well. Especially if I have 3-4 edge windows open and video in one of these, and live updates in another window. Video just won't play properly, and pretty frustrating for end-users. Did i mention that we work in one of the biggests broadcasting companies in Norway? I've been testing with 2b profile and 4q profile on our nvidia cards, and the drivers on the clients are equal the drivers on the hosts. I have deployed 2 desktop pools for our test. Pool 1: M10_2b profile. 16 CPU (4 sockets) and 36GB memory. When i open 3 different Edge or Chrome, video streams stops, and edge telling me that there are lots of frame drops... Pool 2: Same as pool 1, but with no GPU at all. Here I can start several edge windows and stream video without problem. But i see that CPU consumption is offcourse pretty high. Seems to work better without expensive Nvida cards and licenses. So, why does the video stream work better without GPU? I have also tried the following: https://www.stephenwagner.com/2024/01/05/gpu-issues-vmware-horizon-indirect-display-driver/ Can anyone point me in the right direction for further troubleshooting? HI @Lansti, Are these on persistent or non-persistent VDI? Is there a chance that the OSOT tool has been previously ran and disabled some of the graphic capabilities? Can you share what you're using for endpoints (client devices)? Also can you share how many displays each endpoint has and the resolution they are running at? Can you confirm that the your VDI VMs are showing as licensed with vGPU (as non-licensed loses feature after brief interval after logon)? Also, can you confirm that you have hardware encoding/decoding end to end? Can you paste a screenshot of your Horizon Performance tracker and what encoder is being used? Cheers, Stephen Quote Stephen Wagner (President, Digitally Accurate Inc.) VMware vExpert (vExpert Pro, vSphere, vSAN Awards), Omnissa Tech Insider, NVIDIA NGCA Advisor, VMUG Leader, and Director (Board of Directors) at World of EUC Check out my Tech Blog: https://www.StephenWagner.com Link to comment Share on other sites More sharing options...
Jubish Jose Posted September 17 Share Posted September 17 Basic question - are the GPUs licensed? nvidia-smi -q Quote Link to comment Share on other sites More sharing options...
Lansti Posted September 18 Author Share Posted September 18 (edited) Hi @Dominik, @StephenWagner7 and @Jubish Jose. My apologize for my first incomplete post. I have updated my original post here, hope you guys will have a look into it. Edited September 18 by Lansti Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Dominik Posted September 18 Share Posted September 18 (edited) @Lansti important question was asked by @Jubish Jose about license, because without license you don't have powerful vGPU. Edit: sorry, I saw info now in your main post about license. Edited September 18 by Dominik Quote Dominik Jakubowski EUC Expert | vExpert ⭐️⭐️⭐️ VDI Ninja https://vdesktop.ninja Link to comment Share on other sites More sharing options...
StephenWagner7 Posted September 18 Share Posted September 18 2 hours ago, Lansti said: Hi @Dominik, @StephenWagner7 and @Jubish Jose. My apologize for my first incomplete post. I have updated my original post here, hope you guys will have a look into it. Are you using any of the HTML or Media redirection capabilities with the browser? When you look at your task manager, can you see what's using CPU? Also, have you verified that the codec that the web viewers are using is supported by the GPU for decoding (I'm talking about the website, and video streams you're viewing, not the VDI codec)? Quote Stephen Wagner (President, Digitally Accurate Inc.) VMware vExpert (vExpert Pro, vSphere, vSAN Awards), Omnissa Tech Insider, NVIDIA NGCA Advisor, VMUG Leader, and Director (Board of Directors) at World of EUC Check out my Tech Blog: https://www.StephenWagner.com Link to comment Share on other sites More sharing options...
Lansti Posted September 18 Author Share Posted September 18 3 minutes ago, StephenWagner7 said: Are you using any of the HTML or Media redirection capabilities with the browser? When you look at your task manager, can you see what's using CPU? Also, have you verified that the codec that the web viewers are using is supported by the GPU for decoding (I'm talking about the website, and video streams you're viewing, not the VDI codec)? Hi @StephenWagner7, in Horizon we allow Multimedia Redirection policy, i have alsto tried to deny it, in the testing. I really do not see the difference. I have also tried the HTML 5 REDR poicy with chrome/edge extension. It seems to work, but you get a "funny look" in the VDI when you hover other applications over the browser. Applicatins get a sort of a frame, and video will play in front of everything. We cannot use that... In the task manager, Edge are using some of the CPU. I can see that GPU 0 also consuming about 15-28%, but somehow it drops when the edge or chrome browser stops playing. I have also had a look at CTRL + SHIFT + i, to record the performance and i can see that lots of frames are dropping. I acn see that there are lots of Edge processes These processes with GPU Motor sometime changes to GPU 0 - Video Decode. I hae not verified that the codecs from the websites are supported by the GPU for decoding. I never thought that there could be the reason why it doesn't work. I would think that most of the big services out there uses a codec that is supported? Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Dominik Posted September 19 Share Posted September 19 @Lansti did you check profile 2Q? I see on screenshot you use B profile. In my case I'm using Q profile. Quote Dominik Jakubowski EUC Expert | vExpert ⭐️⭐️⭐️ VDI Ninja https://vdesktop.ninja Link to comment Share on other sites More sharing options...
Lansti Posted September 19 Author Share Posted September 19 Hi, yes, we have cluster of clients using 2B profiles, and we have separate cluster with clients using 4Q profiles.. And we see the same issue on both profiles. But as i mentioned in the main post, when i created pool 2 without GPU, but added a lot of CPU, that works much better than a client with GPU, but that's not a solution... So there are something wrong with the offloading to GPUs Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
StephenWagner7 Posted September 19 Share Posted September 19 1 hour ago, Lansti said: Hi, yes, we have cluster of clients using 2B profiles, and we have separate cluster with clients using 4Q profiles.. And we see the same issue on both profiles. But as i mentioned in the main post, when i created pool 2 without GPU, but added a lot of CPU, that works much better than a client with GPU, but that's not a solution... So there are something wrong with the offloading to GPUs Do your endpoints support hardware offload for decoding, or are they decoding the BLAST session using CPU? Also, does the server(s) hosting your VDI have enough bandwidth to your network (and to your endpoints) on the networking side of things? CPU usage on your VDI VMs may not be an issue, because you're using browsers which are CPU/MEM hogs in any event, but the visual performance should be better. Quote Stephen Wagner (President, Digitally Accurate Inc.) VMware vExpert (vExpert Pro, vSphere, vSAN Awards), Omnissa Tech Insider, NVIDIA NGCA Advisor, VMUG Leader, and Director (Board of Directors) at World of EUC Check out my Tech Blog: https://www.StephenWagner.com Link to comment Share on other sites More sharing options...
Sean Massey-1 Posted September 19 Share Posted September 19 These are challenging issues to troubleshoot. You have the interaction between the Web Browser, Windows, the Horizon Agent and the NVIDIA drivers. What version of Windows are you using? I didn't see that listed in the original post? Have you opened a tickets with NVIDIA and Omnissa support? I'm also thinking that the M10 might be part of your issues here. The M10 is basically 4 lower-powered Maxwell GPUs glued onto the same PCI card, which is a bit older. It also has a max limit of 28 simultaneous 1080P H.264 streams. Do you see these issues when you're only running 1 or 2 videos? Quote Sean Massey Independent Consultant/Analyst/Blogger | VCDX-EUC 247 Vice Chairman of the Board - World of EUC Blog: thevirtualhorizon.com Mastodon: @seanpmassey@vmst.io Instagram/Thread: @seanpmassey LI: https://www.linkedin.com/in/seanpmassey/ Link to comment Share on other sites More sharing options...
Lansti Posted September 20 Author Share Posted September 20 19 hours ago, Sean Massey-1 said: These are challenging issues to troubleshoot. You have the interaction between the Web Browser, Windows, the Horizon Agent and the NVIDIA drivers. What version of Windows are you using? I didn't see that listed in the original post? Have you opened a tickets with NVIDIA and Omnissa support? I'm also thinking that the M10 might be part of your issues here. The M10 is basically 4 lower-powered Maxwell GPUs glued onto the same PCI card, which is a bit older. It also has a max limit of 28 simultaneous 1080P H.264 streams. Do you see these issues when you're only running 1 or 2 videos? Hi @Sean Massey-1, we are running Windows 10, 22H2. We have discussed this issue with Nvidia for a long time ago, but we did not get any further.. I wantet to see if any of you guys have the same issues, or know where to look. I assume i need to upgrade my nvidia driver to the latest, and upgrade Horizon to the latest before i can open a support ticket with nvidia and Omnissa... M10 is our "backup"... As you can see i have been testing on our T4 with 2B profiles, and A16 with 4Q profiles. Same behavior with browser that stop playing streams.. Running 1 video stream seems to be fine, and if I start a second one, I'm not able to do much other thing on my windows session before these streams stops.. I have users that vil have 4 chrome/edge windows up at the same time, video stream on one half of a screen, some live updates on the second part of the screen, and statistics on the one half of the second screen and their publishing site on the last part of the screen. So there is a lot of movement, and pages that are updated continuously Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Lansti Posted September 20 Author Share Posted September 20 20 hours ago, StephenWagner7 said: Do your endpoints support hardware offload for decoding, or are they decoding the BLAST session using CPU? Our endpoint have Intel Pentium Silver N6005 with Intel UHD Graphics. 20 hours ago, StephenWagner7 said: Also, does the server(s) hosting your VDI have enough bandwidth to your network (and to your endpoints) on the networking side of things? there are 10Gig from server to switch and then 1gig from switch to endpoint - Dell Optiplex 3000 Thin client. No firewall between endpoint and VDI vms... Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
StephenWagner7 Posted September 20 Share Posted September 20 @Lansti, you'll need to verify, but those endpoints may or may not be enough to handle hardware decoding. It sounds like you need to either keep working on the tickets with NVIDIA and Omnissa, or bring in a 3rd party to analyze your infrastructure and see what they can find. It's extremely difficult to troubleshoot this over forum threads. Quote Stephen Wagner (President, Digitally Accurate Inc.) VMware vExpert (vExpert Pro, vSphere, vSAN Awards), Omnissa Tech Insider, NVIDIA NGCA Advisor, VMUG Leader, and Director (Board of Directors) at World of EUC Check out my Tech Blog: https://www.StephenWagner.com Link to comment Share on other sites More sharing options...
Sean Massey-1 Posted September 20 Share Posted September 20 4 hours ago, Lansti said: Hi @Sean Massey-1, we are running Windows 10, 22H2. We have discussed this issue with Nvidia for a long time ago, but we did not get any further.. I wantet to see if any of you guys have the same issues, or know where to look. I assume i need to upgrade my nvidia driver to the latest, and upgrade Horizon to the latest before i can open a support ticket with nvidia and Omnissa... M10 is our "backup"... As you can see i have been testing on our T4 with 2B profiles, and A16 with 4Q profiles. Same behavior with browser that stop playing streams.. Running 1 video stream seems to be fine, and if I start a second one, I'm not able to do much other thing on my windows session before these streams stops.. I have users that vil have 4 chrome/edge windows up at the same time, video stream on one half of a screen, some live updates on the second part of the screen, and statistics on the one half of the second screen and their publishing site on the last part of the screen. So there is a lot of movement, and pages that are updated continuously So if I understand the issue correctly, your users are running multiple web browser sessions playing multiple video streams across multiple monitors? And the session becomes unusable after starting multiple video streams? In one of the earlier posts, you said that you get a warning that there are frames being dropped. Where are you seeing this warning? Can you share a screenshot of it? To the point that @StephenWagner7 brought up - this could be an endpoint issue. Have you tested this using other endpoints to eliminate that as the possible root cause? I'm not sure this is the case, but it is worth testing just to eliminate this as an issue. There are a couple of other things you can do to gather more information. One of these items will need to be monitored from the host level over SSH, and the other can be done with vCenter. 1. Log into your ESXi Server Shell and run nvidia-smi while your desktop is trying to play multiple video streams. This may give you some additional information on per-VM GPU utilization. You can also run nvidia-smi inside your desktop session to see which processes are using the GPU. 2. You mentioned that your VM has 16 vCPUs. This is a lot of vCPUs. You might be hitting CPU Ready or Co-Stop issues with your workload. You can look at CPU Ready % and Co-Stop on the Performance Tab in vCenter. This Broadcom KB can help you with interpreting the CPU Ready results: https://knowledge.broadcom.com/external/article/306576/converting-between-cpu-summation-and-cpu.html This is kind of a long shot...but have you tried reducing the number of vCPUs on your desktop? Maybe cut it down to 8 vCPUs. What version of Horizon and the NVIDIA drivers do you have deployed today? I would still open a ticket even if you're not on the latest version to get the support process started, and having your vendors say you need to upgrade can help move things through any change control processes. 1 Quote Sean Massey Independent Consultant/Analyst/Blogger | VCDX-EUC 247 Vice Chairman of the Board - World of EUC Blog: thevirtualhorizon.com Mastodon: @seanpmassey@vmst.io Instagram/Thread: @seanpmassey LI: https://www.linkedin.com/in/seanpmassey/ Link to comment Share on other sites More sharing options...
Lansti Posted September 23 Author Share Posted September 23 Hi, On 9/20/2024 at 3:40 PM, Sean Massey-1 said: So if I understand the issue correctly, your users are running multiple web browser sessions playing multiple video streams across multiple monitors? And the session becomes unusable after starting multiple video streams? Correct. Here you can see a typical setup: with liveupdates, one news stream and other sites where video are playing "teasers"... In this session, my video stream to the left, has stopped... On 9/20/2024 at 3:40 PM, Sean Massey-1 said: In one of the earlier posts, you said that you get a warning that there are frames being dropped. Where are you seeing this warning? Can you share a screenshot of it? I already posted this in an earlier post, but all the red squares are Dropped Frames: On 9/20/2024 at 3:40 PM, Sean Massey-1 said: There are a couple of other things you can do to gather more information. One of these items will need to be monitored from the host level over SSH, and the other can be done with vCenter. I'll try your suggestions today... On 9/20/2024 at 3:40 PM, Sean Massey-1 said: What version of Horizon and the NVIDIA drivers do you have deployed today? Horizon 2209 with nvidia driver 16.2 - 537.7 (Long term) Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Lansti Posted September 23 Author Share Posted September 23 On 9/20/2024 at 3:40 PM, Sean Massey-1 said: 2. You mentioned that your VM has 16 vCPUs. This is a lot of vCPUs. You might be hitting CPU Ready or Co-Stop issues with your workload. You can look at CPU Ready % and Co-Stop on the Performance Tab in vCenter. This Broadcom KB can help you with interpreting the CPU Ready results: https://knowledge.broadcom.com/external/article/306576/converting-between-cpu-summation-and-cpu.html 16 CPUs was only on a test pool with a client without GPU... On this testpool i can stream several streams at the same time, but offcorse the CPU are running high. Just a test without GPU. On my regular clients i have 4 CPUs (2 sockets) Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Lansti Posted September 23 Author Share Posted September 23 @Sean Massey-1 Not sure what to look for here... This is from my client: Cant see that GPU is running at all... And on my host where my client are running (client RiOGf-004) - slot 6: Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Sean Massey-1 Posted September 23 Share Posted September 23 2 hours ago, Lansti said: 16 CPUs was only on a test pool with a client without GPU... On this testpool i can stream several streams at the same time, but offcorse the CPU are running high. Just a test without GPU. On my regular clients i have 4 CPUs (2 sockets) That's not what you said originally. Here is what you said in the original post: On 9/17/2024 at 9:00 AM, Lansti said: I have deployed 2 desktop pools for our test - Non-presistant desktops. Pool 1: M10_2b profile. 16 CPU (4 sockets) and 36GB memory. When i open 3 different Edge or Chrome, video streams stops, and edge telling me that there are lots of frame drops... It's hard to help you troubleshoot or provide recommendations if we're not getting accurate details about your environment. Quote Sean Massey Independent Consultant/Analyst/Blogger | VCDX-EUC 247 Vice Chairman of the Board - World of EUC Blog: thevirtualhorizon.com Mastodon: @seanpmassey@vmst.io Instagram/Thread: @seanpmassey LI: https://www.linkedin.com/in/seanpmassey/ Link to comment Share on other sites More sharing options...
Lansti Posted September 23 Author Share Posted September 23 10 minutes ago, Sean Massey-1 said: It's hard to help you troubleshoot or provide recommendations if we're not getting accurate details about your environment. I understand that, and it seems that I had a Copy/paste error during my editing of my post. My apologize Quote Best regards Lansti Horizon 2209, DEM 2209,AppVols 4, version 2212 Windows 10 22H2, FSLogix 2.9.7979.62170 Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.