For what it's worth, I think the in-game auto-target works the exact same way tab targeting works. It goes from left to right, front to back (importantly, in that order) and picks the first mob in the "tab" order.
The problem with this is when there's a mob to your left and 10 yalms away, and also one right in front of you and 2 yalms away. It will go with the mob really far away and mess you up.
Would be nice if it worked like F8 instead of like tab targeting, but IMHO that's what's happening here.
Of course most people are also probably aware, but this only applies to mobs who are aggressive to you and your party, not including trusts.