Yeah I just spoke with the breeder (who absolutely LOVES his babies) and he said that he thinks he'll be fine. (we're only going Saturday afternoon - Sunday morning/ afternoon). Nothing too long.
He said that as long as he has plenty of shade and has his crate and water, he'll be just fine. He also said I can drop him off at his house and he'll babysit for me, but I want him with me!
Still thinking about it though. but I do feel a bit better seeing that other people had taken their babies at such a young age!