I've not ever cast lead, but I do reload about 3K rounds a year (so I'd say I am a casual reloader). I've only shot at indoor ranges a few dozen times and most were modern well ventilated. I was tested for lead last year as a result of a previous discussion on THR. I tested very low at 1 MCG/DL (<5 considered good) and will continue to be tested each year going forward.
I am kind of a lead Nazi, wash my hands with lead removing soap after every shooting, cleaning of guns, or reloading session. I never eat during any of those activities. I am very strict with the kids (now in their 30's) as well.
Ten years ago loaded about 24,000 rounds a years now down to about 12,000 rounds a years. I think at about 3,000 your going to safe.