Last week I came across this error when setting up a connect (and also using the application). ITSM (or GMEE) will still allow you to create the connection and connect to the application server but will through this error and prevent saving of the object if you have the BPML engine running.
The issue is with the security setup of the Microsoft DTC service on both the ITSM and SQL Servers. There is a very helpful article from technet about 'New functionality in the Distributed Transaction Coordinator service in Windows Server 2003 Service Pack 1 and in Windows XP Service Pack 2'
This outlines the settings that need to be enabled to allow the ITSM and SQL servers to communicate.