We ship a DDEX provider with Connector/Net. This provider plugs into Visual Studio and integrates into server explorer allowing a user to create data connections to MySQL from within the IDE. This integration is handled, in large part, by a lengthy series of registry entries. Until 5.2.2, these entries were made by my installer which is written in WiX. Having the registry changes made in the installer has two problems. First, I often need to register the provider during debugging and I don't want to do a full install of the product so I end up hand editing a registry file and manually merging that file. This a awkward at best. Second I plan to ship a stand-alone configuration utility in 5.3 that will allow the user to configure which installed version of Connector/Net should be used for VS integration. Currently you can only have one instance installed at a time.
My approach to solving this was to write an installer class that made the registry changes for me and just use installutil to register the assembly. I was already doing this with my core and web assemblies so this is nothing new. The problem is that installutil will scan the assembly and all referenced assemblies for types looking for installer classes. This fails when some of the Microsoft assemblies are scanned. After much effort I gave up and decided to write my own installutil that I would ship with my installer.
I had no trouble creating this application and then set about executing it from within my installer. I attempted to use the CAQuietExec custom action available with WiX v3 but just couldn't make it work right. So I gave up and decided to write my own execute custom action.
So I cracked open Visual Studio 2008 and read a couple of blogs about custom action writing. One of them mentioned mixing managed and unmanaged code, the unmanaged code being necessary for the proper DLL exports. I decided I had to see this work.
Within minutes I had a DLL project setup with the /clr option and had hacked out the following code:
Yes, that is managed code and unmanaged code *in the same function*! Now this code might not compile as it is not what I wound up using and I just grabbed it out of an old folder but you get the idea (and yes I did test a version of this so I know the concept works). I didn't use this approach because the /clr switch requires the dynamic CRT and I didn't feel like bundling that up.
And, in case you were wondering about the "boo" messagebox on line 3, that is my debugging trap. You run the installer until that pops up, attach to the proper msiexec using Visual Studio, set a break point, and go. Yup, that's cool.