The most important thing to keep in mind when going this route is to work to the strength of the drill press as much as possible.
Basically, remove as much material as you can with the drill heads before taking out material with the end mill bits.
This will reduce the overall stress on the drill press, and hopefully reduce some of the time you spend milling.
The bottom line is that if you want to use a drill press to do this, you need to be tremendously meticulous, and have a great deal of patience.