I googled a little and found this.. Ingredient wise it looks safe. I would be interested to see if it helps.
PupTeeth Granules? - Soothe Kittens & Puppies During Normal Teething
It does seem a little expensive, you might try and google it or try amazon to see if you could find it cheaper..