Unidentified spellbooks are sorted alphabetically based on their identified names
For instance, a 'small tome' I just picked up was in between 'Divine wrath' and 'farsight', which lead me to conclude it was a spellbook of Earthquake (it was. Couldn't be ethereal bridge because I already identified that).