Huh? Of course it uses the connection. All spells require arcane or mystic/divine energy to fuel. If you don't have enough of it, some of your life force is pulled in instead, but since it's not the kind of energy that the spell expects, it damages your connection to the magic energy inherent in the world (your Mana stat), sort of like running an engine using poorly suited fuel. (For instance diesel engines can use petrol - but not nearly as well, and it will do quite some damage to the engine over time.)but it seems to me that casting from HP is completely ignoring the astral component
Everything about that makes sense. Having your Toughness damaged from having your life force pulled out would make sense too if it were that way, but I don't see a necessity for it to be that way. Having your life energy sucked out by vampires or the like is still something quite distinct from most everything else; that's a distinctly undead feature, and the fact the drain is permanent can be explained by the vile undead magics at work in these creatures.
As for Teleportation, it's a spell listed in a spellbook, presumably equipped with all kinds of safeguards to make sure that the travel through the astral plane goes smoothly. Also, teleportation happens instantly - even though beings on the material plane can travel through astral space to reach different places, they don't really spend any definable amount of time on the astral plane itself, and I think that's because the teleportation effects in the game don't have the means to make that happen. Otherwise, why have characters exit the plane at all in the case of teleportation traps? Get sucked into astral space -> die seems much more satisfying for whoever placed that trap. If you spent barely any time in the astral plane - in fact, an infinitely small amount of time (an instant) - that could well be simply not enough time for the body to sustain any damage.