Browse to %WINDIR%\RemotePackages\CPubFarms\CPubRemoteApps on each server holding the RD Connection Broker role and you will see all the .ico files.

In the following example, I have a RemoteApp called ‘Notepad’.  If i want to change the icon I’d remove “notepad.ico” and copy another .ico file of my choice and rename it to “notepad.ico”.
This will replace the icon for the RemoteApp in your Remote Apps and Desktop Connections application on the next refresh. For RD Web Access you’ll have to stop and start the Web application on the RD Web Access server.

This will only work for .ico files. If your icon is in a .exe or .dll you’ll still have to use powershell.