Can anyone please explain me that the use of “to continue studying” is correct or I should use “to continue study” or something other.
(a) Inspite of my attempts to encourage him to continue studying he decided to take up a job.
Your usage is correct - to continue with the act of studying. Otherwise you would need ‘to continue to study’ which would be grammatically correct, but having the two 'to’s so close together wouldn’t sound great.
Incidentally. ‘In spite’ is a two word phrase. ‘Inspite’ (one word) is incorrect.