Launching Remotly in Hyper-V VM cause the ENTIRE PC to crash

[mirillis]

Hi Clement,

I'm still setting up Hyper-V on a dual Nvidia RTX system.

In the meantime please find how I have made it work on the Intel GPU system:

1. Install git if you do not have it

2. Clone the Easy-GPU-PV repository - run PowerShell with admin rights, go to the directory where you want to checkout Easy-GPU-PV and type:
git clone https://github.com/jamesstringerparsec/Easy-GPU-PV.git

3. Still in power shell change the directory to Easy-GPU-PV to make it current

4. Type dir command to make sure you have everything like I do:

05/05/2023 09:19 PM <DIR> .
05/05/2023 09:19 PM <DIR> ..
05/05/2023 09:19 PM 3,642 Add-VMGpuPartitionAdapterFiles.psm1
05/05/2023 09:19 PM 9,957 autounattend.xml
05/05/2023 09:19 PM 178,404 CopyFilesToVM.ps1
05/05/2023 09:19 PM 232 gpt.ini
05/05/2023 09:19 PM <DIR> Machine
05/05/2023 09:19 PM 2,296 PreChecks.ps1
05/05/2023 09:19 PM 8,645 README.md
05/05/2023 09:19 PM 1,372 Update-VMGpuPartitionDriver.ps1
05/05/2023 09:19 PM <DIR> User
05/05/2023 09:19 PM <DIR> VMScripts

6. No while still in this directory type:
.\Update-VMGpuPartitionDriver.ps1 -VMName "Name of your Hyper-V VM" -GPUName "AUTO"
Of course replace the "Name of your Hyper-V VM" with your VM name

You should see something like this:
Mounting Drive...
Copying GPU Files - this could take a while...
INFO : Finding and copying driver files for Intel(R) UHD Graphics 630 to VM. This could take a while...
Dismounting Drive...
Done...

Now the drivers for the shared GPU are installed

7. Now configure the GPU memory setup by running:
$vm = "Name of your Hyper-V VM"
if (Get-VMGpuPartitionAdapter -VMName $vm -ErrorAction SilentlyContinue) {
Remove-VMGpuPartitionAdapter -VMName $vm
}
Set-VM -GuestControlledCacheTypes $true -VMName $vm
Set-VM -LowMemoryMappedIoSpace 1Gb -VMName $vm
Set-VM -HighMemoryMappedIoSpace 32Gb -VMName $vm
Add-VMGpuPartitionAdapter -VMName $vm

Just copy/past everything in green text again replacing "Name of your Hyper-V VM" with your VM name and then press enter.

8. Make sure to disable Checkpoints for your VM

9. Run your VM and try to run Remotly

 

[ClémentRED]

As I already have a GPU VM, I tried only 7,8 and 9, no error or anything in Powershell, disabled checkpoint and still same bug.

 

[mirillis]

Ok have the same problem on RTX 2080. The entire PC crashes (SYSTEM_THREAD_EXCEPTION_NOT_HANDLED).

 

[mirillis]

I will start by disabling GPU initialization of various modules. I hope I can find what is causing this in less than 2 hours ;/
Anyway, I think this should be reported to Nvidia. Imagine a server with shared GPUs in a datacenter and someone runs this...
This is surely a bug in Nvidia drivers.

BTW. Have you tried an AMD GPU to check if the same problem exists?

 

[mirillis]

I know more now. As expected the crash happens during querying GPU encoder capabilities. It might be something with HDR.

 

[mirillis]

I give up The crash is in nvencodeapi.dll. The same dll is used in non-VM (physical) system and it works fine there. If AMD will work then this is Nvidia's bug.

BTW All Nvidia HW video decoders work fine from the Hyper-V VM.

 

[mirillis]

I have managed to start Remotly without crash on the Hyper-V VM running with Nvidia.
To do so you need to rename this file (on the VMs file system):
C:\Windows\SysWOW64\nvencodeapi.dll
to another name for example:
C:\Windows\SysWOW64\nvencodeapi.dll_

This way Remotly won't find it and will not try to initialize HW encoders on Nvidia GPU.
Funny thing is that all HW decoders work and when I connect to another PC from the Hyper-V I can even use HEVC and I see in the task manager of the physical PC that it decoded by the shared GPU.

Of course after connecting to this Hyper-V VM you will only get SW encoding and since we limit SW encoding to 30fps you will only get connection in 30fps.

If you have an Intel GPU in this system you could try to enable it in the VM and have both Intel and Nvidia and use HW encoding from Intel. But this might be difficult to setup.

 

[ClémentRED]

Should I try to contact NVIDIA Support myself ? And I will try AMD and see if there is the same bug.

 

[mirillis]

Hi Clement,

Yes, please write/contact Nvidia. We will also contact them but from our experience they respond faster when they get feedback from multiple sources. They'll also see this crash in their logs from Microsoft when BSOD crash occurs but God knows how long it will take for them to respond.

I'd be very grateful if you could test AMD on your side. We will also do so but it's always better to have more test cases.

We will add a message box to the 1.11.2 version that will inform about running Nvidia on a VM during Remotly startup and potential problems. This message box will ask whether to skip encoding acceleration on Nvidia to prevent the crash (as changing file names of driver files is not elegant).

 

[mirillis]

The new v1.11.2 version is almost ready. You will now see this message during startup:

 

[mirillis]

Version 1.11.2 is now available in the download section.

 

[mirillis]

It looks like it also crashes on AMD RX560. This time only the VM not the entire PC.

 

[mirillis]

UPDATE: AMD RX560 works after disabling "Enhanced Session" and checkpoints. But AMD AMF video encoders fail. Only SW encoding works.

 

[mirillis]

RX560/580 does not fully support GPU virtualization... We will check with RX6xxx tomorrow.

 

[mirillis]

Hi Clement

We have successfully run an AMD RX5700 on the Hyper-V. It works and HEVC is streamed using GPU acceleration in Remotly on the VM.

It looks like recent Intel GPUs work and AMD RX5xxx (and newer) too.

 

[ClémentRED]

I just tested my RX 5600 XT and came to the same conclusion, it seems to work. So it's NVIDIA ...
By the way I tested RX 5600 XT with HW encoding with 100 Mbps output on VM (CLIENT) and NVIDIA Decoder (HOST) and it doesn't run really well, the game itself is at way more than 60 FPS but the "Remotly connection" seems to lag a little but more importantly the game is pretty bad visually, mainly when moving the camera, and seem to have a pretty high latency, I think I heard somewhere that AMD encoder/decoder is worse than the NVIDIA one but still ... Is it that bad or is there something I'm missing here ? And I can't even test NVIDIA encoder with this bug

 

[mirillis]

Hi Clement,

At least a small progress is done. I need some more info about your setup and would ask for another test:

1. Can you run the same test but not between VM but the physical PC (with the same AMD) and connect Nvidia HOST to it? We need to make sure the problem is related with the VM and not the connection itself.

2. When running the VM do you have the Hyper-V display adapter enabled in the device manager? And what is the refresh rate of the virtual display? We see that it's running at 64Hz (no idea why it's forced to this number) but we see there is at most 32fps on the VM's desktop (runs poorly no matter what GPU is used)

3. Are you testing 100 Mbps @ 4K or 1080p?

4. Please try a less intensive test for example 1080p @ 25Mbps to check if the quality improves.

5. Is the connection/test done in a local network via P2P? When doing such tests between RTX 2080 (CLIENT) and another Nvidia Decoder (HOST run on some GTX) and streaming in 4K we get great quality @100Mbps and a near zero (~1 frame) lag.

 

[ClémentRED]

1. Like connecting to a physical AMD PC with a physical NVIDIA PC ? I will edit the results a bit later ...
2. Yes, the Hyper-V display adapter is enabled but display "unkown" where the refresh rate should be (in the windows graphics settings at least).
3. The test was done 100Mps @ 1080p (BTW Why can't I type "@" for my email when I try to connect to my account in the app ?)
4. I will edit the results a bit later ...
5. Yes P2P, local network.

 

[mirillis]

1. Like connecting to a physical AMD PC with a physical NVIDIA PC ? I will edit the results a bit later ...
> Yes, exactly. To have the same test but eliminate the VM from the equation

2. Yes, the Hyper-V display adapter is enabled but display "unknown" where the refresh rate should be (in the windows graphics settings at least).
> Please check the screen how we see the virtual "Monitor" it in the Device Manager of the VM:



As you can see there is 64 Hertz. No way to change this :/ This is the only option on the list.

3. The test was done 100Mps @ 1080p (BTW Why can't I type "@" for my email when I try to connect to my account in the app ?)
> Please make sure to use HEVC for 100Mbps @ 1080p as it is off specification for AVC/H.264 (the encoder does not need to conform/run properly with values exceeding specification).
"BTW Why can't I type "@" for my email when I try to connect to my account in the app ?" - I don't get it? When I use the app to login I'm entering my account email with "@"
Let's use 25-40Mbps for the initial testing.

4. I will edit the results a bit later ...
>

5. Yes P2P, local network.
>

 

[ClémentRED]

Wait, I just saw that the streaming uses Connect Anywhere, how am I supposed to disable it ? It seems I need to do something with the ports of my router but I can't find anything ... It's not P2P !
EDIT : I think I need to port forward my router but what ports would it be ?