I say you need to draw for the following reasons (most of which have been pointed out already).
1. There's no way to be sure that he is unarmed.
2. Even if he is, he can still do great bodily harm. If you don't think someone can kill you with their hands and feet, you need to wake up and smell the coffee.
3. Size and strength, while relevant, have less to do with who wins the fight than most people think. Knowledge, experience, speed, and tolerance for pain most often are the keys to who comes out on top. My training has shown me that you cannot judge how good of a fighter a person is by their looks.
4. Even if you win, you are still going to get hurt. Unless you are a badass extraordinaire, you are going to get punched, kicked, stomped, scratched, bitten, and/ or gouged. You can also get hurt falling and defending yourself. Plus, unless you have some training, you may not even know how to punch correctly so that you don't break your hand.
5. Almost all fights end up on the ground. Again, unless you a trained in ground fighting, the chances of a favorable outcome are not good. At this point it is doubtful that you would even be able to draw you weapon.
6. If your attacker gets a hold of your weapon while you are struggling with him, he's probably going to use it on you, even if he only uses it for a club.
That's just my $.02 worth.
David