a question about one of the listening exercises

Dear to whom may it concern,
english-test.net/toeic/liste … rrive.html
In part 3/why is this answer wrong? I’m sorry that I did’. (Means I’m sorry that I did not ask you?)
With the best regard

‘I’m sorry that I did’ would indicate that the person did ask before ordering.

“I’m sorry that I didn’t” would be the phrase he’d need to use to indicate that he was sorry that he did not ask.