I recently encountered a problem with custom installers and .NET 4. If you are using an Installer-derived class in an assembly that allows partially trusted callers then you may see this when you attempt to install with InstallUtil:
Inheritance security rules violated by type: 'Class'. Derived types must either match the security accessibility of the base type or be less accessible.
After a little googling, I found this post that helped me solve part of the issue. The other thing you need to do is go to your installer class and add the following attributes to it. These will demand full trust for the installer class.
[PermissionSetAttribute(SecurityAction.InheritanceDemand, Name = "FullTrust")]
[PermissionSetAttribute(SecurityAction.LinkDemand, Name = "FullTrust")]
That’s it! Your installer class should work with .NET 2 and .NET 4 now.