This is purely speculative, but consider this: given that certain mental imagery (for example: a sad movie) can trigger physical bodily reactions in humans (for example: tears being excreted by the tear ducts), it's not far-fetched to assume that telepathy/mental invasion with certain "harmful", "broken", or "badly compatible" elements (however small) may induce certain psychological reactions/rearrangements that, in specific cases, might deregulate the physical working of the typical stoic Vulcan mind in the long run.