He had <some description of features of a particular face. Lips, eyes,…>. Not altogether a pleasant face.
It sounds a bit unnatural to me.
More natural would be to say something like Altogether not a pleasant face or Not a pleasant face, altogether.
(i.e. to relate not directly to the ‘result’ (not a pleasant) of the 'union action’ expressed by the word ‘altogether’)
Or is ‘not altogether a…’ a fixed expression?