1) Start with an overspray of weathered or grimy black. These cars got very grimy from smoke, dirt, dust and road grime and were rarely washed or repainted. This also dulls the markings.
2) Add additional grime from spillage in the dome area. I like oily black for this.
3) Add bearing oil grunge to the bottom and ends of the tank where leaky bearings on the car, or neighboring car would have spun oil out onto the tank and frame.
4) Add a rust wash to the edges of the tank bands to indicate where expansion and contraction have rubbed the paint off allowing rust. Also rust the truck springs.
5) Add fresh spills with gloss paint.
6) Add appropriate placards and chalk marks.
And Tony Thompson also has some illustrated recommendations:
Here are some prototype tank cars weathered by the hand of man and nature:
Finally, some models: