Ok I finally figured it out. IsUserGroupMember returns a boolean not a string so you need to use that instead. I also rearranged things around a bit. This is the customsettings.ini that got this working:
Priority=INT, SetTaskSequenceAdmin, SetTaskSequenceDefault, Default
Properties=MyCustomProperty, IsGroupMemberAdmin, UsersGroupadmins, Username, Groupname
If a user is part of the IT-Admins group then they will see the task sequences associated with ShowAdmintaskSequences selection profile (Set up in MDT workbench)
If not then it will show task sequences associated with ShowDefaultTaskSequences.
Note: Once WizardSelectionProfile is set it can not be overwritten. I am using
MaikKoster.Deployment.Webservice 7.3 beta. The command ad.asmx/IsUserGroupMember will only work if a person is a direct member of the group (does not do group inside of groups) I emailed Maik about this and he said he will look into this. Thanks