From some extremely limited testing, it seems that the difference in actual potential toughness is determined solely/almost entirely by race, and not class. For example, I rolled a Gnome Wizard with 10 To and 17 P.To, while a Mist Elf Barbarian started with 11 To and 12 P.To. Having race as a primary factor is a good idea, but it should also be modified by your class, otherwise you get absurd situations like my example, where any frail Wizard on a moderately tough race can get much higher toughness than the toughest of the toughest classes on a frail race. Of course this is a two way street: not only would the potential for, say, a Beastfighter be upped, but the potential for a Wizard would go down.

I am not sure how stat potentials work in general, as I only looked at toughness, but it is quite plausible that some other stat potentials need to be looked at, as well.