I would drop it.. there is nothing worse than struggling bc of the teacher... especially when you pay sooo much for college. However, you could also make him explain everything twice and ask tons of questions ... there are probably many people in the class who feel the same way you do but are too afraid to speak up or ask questions too. (Ive taken this class I know how hard it is)
Good Luck with whatever you decide... but if you think you are going to struggle then do something rather than nothing... ya know what I mean?