‘I want you to sort these in order of size - put the biggest first, the second biggest next and so on’. Very often ‘in’ after ‘sort’ can mean as in that example, arranging things following a pattern or sequence.
‘I want you to sort these books by their category - novels on this shelf, autobigraphies on that shelf and handbooks here’. In this sense ‘by’ means according to.
This could mean (I say ‘could’ because I don’t know the context) that you have to find your luggage at an airport or wherever, which is all stacked with a lot of other luggage belonging to other passengers.