Yes, coffee can be toxic in the same way as chocolate. Even caffeine free coffee still contains caffeine...It's just a lesser amount.
Avoid Giving Your Dog Toxic Foods "The symptoms of coffee toxicity are similar to the symptoms of chocolate toxicity, and just as serious, if not more so."
Aside from the high sodium content...generally speaking, fruit pits/seeds are toxic to dogs and olives are fruits, so it's possible that the pit could be toxic to dogs, but I don't know for sure.