Go to the Brown Hotel for drinks Thursday night. Go to the upstairs mezzanine bar/lounge area...great place to drink bourbon, watch womyns and handicap. Then when you're good and drunk and have the Oaks winner figured out, go downstairs and order a "Hot Brown" sandwich.
I don't have any advice for Friday and Saturday nights, as I am always at a friend's place, but I'd imagine repeating the same itinerary should work well.

Also, CD is dark the day after the Derby. So if you fly out late enough on Sunday, try to rent a car and go out to Bluegrass country, about 70 miles east.
Alternatively, you can head that way on Thursday and get a good distillery tour (I recommend Buffalo Trace, where Blantons, Rip Van Winkle etc are made) and check out the steeds over at Coolmore, but then you'll miss a day at the track.
Tough call.