You're able to merge the antediluvian map from unpaid fragments
The exorbitantly priced map fragment can be acquired with zero cost by just picking it up and having all other fragments with you. Read one of your own ones, and the map is merged, debts are forgiven and the shopkeeper doesn't mind at all.
In fact, you can even sell 4 other fragments to the shopkeeper, then pick them up and read the only one you own. As long as you have one paid fragment, you are able to do this. :)