What you need to say is - refuse to speak. ‘Reject’ isn’t followed by an infinitive or gerund and means - turn aside/not include/refuse. It means the opposite of ‘accept’.
I don’t know how you rate the site myenglishpages.com/site_php_ … nitive.php , where you come across, “He rejected working with them.” (see 2. After verbs such as:…)