Confusion, blindness, stunning, poison, or paralysis - that's five. Am I the only one that thinks that giving each of the five orb guardians one of these as a weakness would be an interesting approach? Perhaps they could each be weak to one, and partially resistant to two, while being immune to another two.
Looking through the orb guardians, I'd suggest the following:
Snake from Beyond - immune to poison (duh) and blindness, weak to confusion
ACW - immune to confusion and stunning, weak to blindness
ASB - immune to paralysis and blindness, weak to stunning
Yulgash - immune to stunning and poison, weak to paralysis
Archmage - immune to confusion and paralysis, weak to poison
Why this set? Picture a pentagon. On each vertex, in clockwise order, put confusion, paralysis, blindness, poison, and stunning. Now, each one is weak to one point, and immune to the two that are "opposite" it, while resisting the two adjacent to it. I specifically chose the archmage to be weak to poison because magic users tend to have low toughness and high willpower. I also specifically made the snake immune to poison and the stone beast immune to paralysis.
I'd then make Andy strongly resist, but not be immune to, all five.
Beyond that, I'd make it so that potions are only 100% accurate when used on non-hostile or tame creatures - if the creature is hostile, perhaps it's controlled by "thrown rocks and clubs" in combination with dexterity and luck? This would ensure that you can still use potions on pets and non-hostile NPCs without the accuracy loss, but reduce the accuracy against foes.