I've thought of this as intentional behaviour. They are hills / mountains / swamp creatures, so why wouldn't they be much more capable in moving their habitat than some random trespasser?
Sure, companions and summons are also unaffected, and that might be an RFE item. Not sure if it's a necessary one. :)