wix - Windows Installer custom actions: commit action won't get executed

Question

I am designing a windows installer project using WIX. Now it fails in the installation, because the Commit custom action won't get executed, which is caused, weirdly, because a scheduled following custom action has one route to return ActionResult.Success. Let me explain in details:

Basically I have two custom actions regarding to installation: one called CommitCA and one called InstallCA. The execution schedule is listed as below:

<CustomAction Id="CommitCA" BinaryKey="CustomActionDll" DllEntry="CommitCA" Execute="commit" Return="check" Impersonate="no"/>
<CustomAction Id="InstallCA" BinaryKey="CustomActionDll" DllEntry="InstallCA" Execute="deferred" Return="check" Impersonate="no"/>

I have verified from ORCA that in the compiled installer, the CommitCA is scheduled before InstallCA. The type of CommitCA is 3585 and InstallCA is 3703 (I can't find the reference to them but I assume one is for commit action and the other is defered action like I declared). The idea is in the CommitCA it will generate some file which will be used by InstallCA.

The structure of my InstallCA looks like this:

Try
    ' Do some work
    ...

Catch ex As Exception
    ' Log the failure into installation log
    Return ActionResult.Failure
End Try
Return ActionResult.Success

Here comes the confusing part: originally I have a working version, where I didn't have the line of returning ActionResult.Failure in the Catch block. In another word, the function still logs the failure, but eventually returns Success even if an exception was thrown. I find it a bit misleading, because sometimes when an exception happens and I check the log, it says the custom action succeeded. That's why I added the line

        Return ActionResult.Failure

However, now it doesn't work anymore! Whenever I install, I find that my CommitCA never gets executed. NOTE: not the InstallCA, which has the only code modification I made, but the CommitCA, which is actually scheduled before the InstallCA. And when I say "never gets executed", I mean I can't find any log entry which I put inside the function calls, and MMSIBREAK environment variable doesn't even work at all (the window informing me to attach to the rundll32 process never pops up). But the following InstallCA still gets executed.

If I comment out the line Return ActionResult.Failure, everything works again: I can attach to get into CommitCA and InstallCA gets called as well.

There is probably something that I misunderstood about the return result of custom action. Am I not supposed to return ActionResult.Failure in this situation? But actually I return ActionResult.Failure in some other places too. In fact in the CommitCA function itself, I used the same structure: returning ActionResult.Failure in the Catch block. So could anybody tell me what I did wrong?

Answer 1

I would start by reading:

Installation Phases and In-Script Execution Options for Custom Actions in Windows Installer

A few things to understand about commit CA's:

Commit custom action are generally not meant to change the state of the machine. They are meant to cleanup rollback temp data.

Commit custom actions do not execute if rollback is disabled (by property or system policy) because that would mean there is no rollback data to clean up.

Occasionally commit custom actions are used to configure the machine when it's not possible to do a rollback and you want to defer the change until as late as possible. (Changing a user password, installing a DLL to the GAC ) In these scenarios costing should instruct the deferred custom action to do the work since the rollback/commit will never execute.

Rollback custom actions should be scheduled before the deferred custom action and commit actions should get scheduled after the deferred custom action. This is so that the script generation phase can create the script correctly.

If the deferred phase fails, the script is walked backwards so that the rollbacks are executed.

If all of the deferred phase succeeds, the commit script is executed walked forwards. This means that all deferred CA's must finish before any of the commits execute.