If your dog has ear mites, his ears will be full of brown, icky stuff. This is ear mite "poop". It sounds to me like he probably has ear mites but like a previous poster said, it's no big deal at all. Maggie had ear mites when we got her and it just took some drops once a week for about a month and they all went away!
Even if it's not mites, I would have your dog checked out. If the ears smell bad, something is wrong. He could have an infection.
Good luck!