Awesome thread, minus the ATV/bike argument ! I'm 47. Not ready to race (yet ?) but absolutely amazed at how many older men are in this sport and how great it is. I should have been doing this 10 years ago.
I will just say that you may not ever think you're ready to start racing, but the sooner you start, the better rider you'll become, sooner. I honestly had no plans to start racing, but did this year finally at the encouragement of a couple of my buddies. I'm only in my first season, obviously, but my riding has come a long way even just since my first race in April. And similar to you, I hadn't ridden a dirtbike in several years, really since about 1996 until I bought the WR in May '13 (except for one day in '08 visiting family in SoCal, spent an entire afternoon trail ride on a CRF450X). I rode a decent bit last summer, was out of the country for a few months last fall/winter, and since early January, I've been riding as much as possible every week. But offroad racing just makes you focus more and naturally become faster and deal with obstacles and sketchy areas much better. I'm a mid-pack C rider, not fast by really anyone's standards, but it's really cool to recognize progress and improvements on a regular basis (as well as continually pursue a better bike set-up).
Anyway, that's my take on it - you may have family stuff or other constraints - but if you're not sure if you're ready because of skills or where you think you'll stack up against everybody else in your class, I say go for it ASAP! And you're right, there are a lot of older dudes out there - old, gray-haired codgers who are faster than I'll probably ever be! But everyone, in our series at least, is extremely friendly and always willing to help out (whether that's with advice, letting me keep my bike with them over night while we stay at a hotel in town, unloading or lifting stuff, working on the bike...whatever, people are just helpful and cool).
Oh, and also - if you haven't seen it already, look on Netflix (or buy the DVD on eBay for $6) and watch "Dust to Glory." It is awesome, and that's both part of my inspiration for desert racing and something I think of when I think I have it rough or get tired during a measly 50 mile race.
Here are a couple more pictures I found on the series website from my last race. If you can't be fast, at least look fast! That's what I always say.