1. Nice to meet you all. is it grammatically correct to say “you all”
  2. Nice to meet you guys. ( over here guys refers to boys and girls? )
  3. It was nice talking to you. (Is it grammatically correct and can you also suggest me some other phrases)

“Nice to meet you all” seems very correct. “Nice to meet you” would also be correct because you is used for the second person singular and plural, but the meaning could be misunderstood.
Generally, in a group that contains boys, you address the group by saying guys. This is something available in many languages. Note, however, that this sentence is very informal, so you can use it only with friends of your age.
Regarding the last sentence, I would rather use “It’s been nice talking to you” (It has been nice…). Why? Because the talk extends until you say goodbye so, theoretically speaking, you are still talking to them while saying this. The sentence you wrote would fit best if you had previously talked to the people of the group and you now say that the chat you had back then was very nice.

In addition to Jimmy’s comments, I know that it is also acceptable in some circles (particularly among younger people) to use ‘guys’ when speaking to a group including females.