I think most vegetarians who follow the diet for ethical reasons, as opposed to just not liking the taste of meat, are usually quite serious about avoiding anything with a meat product in, like gelatine and rennet. Your experience may vary, of course.

Milk chocolate is certainly not vegan, but that's a separate issue.