I don't know if ebay still sells reloading presses or not but that is where I got mine. It cost me about $25. Other places to watch would be the classifieds on here and in the newspapers. I'd try to find either an RCBS or a Lyman press to start with. With many people needing christmas money and with taxes coming up soon, this might be a good time to put a "wanted" ad out too.
For dies I would also go with Lyman or RCBS for handgun calibers. For rifle calibers I would choose Lyman. RCBS decapping stems seem to have the expander ball screwed on too tight from the factory and that strips the threads out. I haven't had this problem with Lyman dies. For most rifles regular full length dies will work fine but some lever actions, pumps, and semi-automatics require small base dies. I have never needed them but some guys on here have. You'll also need case lube. I prefer RCBS case slick.
You'll need a scale. Again, RCBS or Lyman should do fine.
For reloading components, I would go to a trusted local shop and start asking questions. It would also help you out with other questions you have. Most importantly is that it might put you in contact with a real live handloader who can help you out. Places like Midway are great for components and harder to find specialty tools but IMO you should wait until you really know what you need to order from them - maybe on your second reloading purchase. IMO there just isn't any substitute for one on one help from an experienced reloader.
You'll want a powder measure eventually but you could get by with a clean bowl and a teaspoon for awhile. Use it to dispense powder to your scale. But if you can afford the measure it would be way better to just start with it.
Most important, get a manual and read it before you do anything. Then get another manual and read it too.
Then ask us more questions.