The first sentence is not incorrect, but I would more naturally write “Increases in the amounts…”.
The second sentence should be “An increase in the amount of all three compounds was observed”, but the first sentence seems preferable because (presumably) there are three increases and three amounts.
‘the all three Xs’ is incorrect. You can say “all the three Xs”, but usually this is unnecessary. “all three Xs” suffices.
“the both” is always incorrect in Standard English. It is used in certain colloquial or dialect expressions (e.g. “the both of us”).