Server 2012 Hyper-V cluster
VMs stored on CSVs
FEP 2010 installed on the host servers
Performing VM backups using any backup product causes the RAM on the CSV owner to be 100% used.
The amount of RAM used is equal to the regular RAM used on the host, plus the size of the VHD/VHDX currently being backed up.
You can see this issue occurring in real time during a backup, watching the used RAM will show a steadily increasing RAM usage until it hits 100%
Memory preasure causes the CSV owning node to become so slow that is becomes unresponsive, and eventually VMs will start to shut down and the cluster will eventually fail.
Tested and confirmed as an issue with DPM and Netbackup.
Microsoft is totally lost on this issue and after working on it for months is yet to find a solution. However there does seem to be 2 work arounds for the issue.
1/ Uninstall FEP
2/ On top of the recommended antivirus exclusions for Hyper-V, exclude the processes corresponding to your backup product.
dpmra.exe (DPM process)
bpbkar32.exe (Netbackup process)
bpfis.exe (Netbackup process)
If you use another backup product you will need to research which processes to exclude.
Let me know in the comments if this helps you.