Free mail delivery began nationally in 1863. It began in 44 northern cities. (The South was not the place to begin a new federal program at that time, to say the least!) Following the Civil War, the service grew to other urban residents. Providing mail delivery to rural areas was debated, and in 1896, Congress allocated funds to test rural delivery in West Virginia. People loved it and the service spread.
The Smithsonian's National Postal Museum has put up a very good on-line exhibit showing (and telling) the history of rural delivery. It can be found here.
Great pictures and good history. Rural delivery allowed rural residents to receive newspapers (and weather reports.). Interestinly, one of the early proponents of rural delivery was John Wanamaker, a Philadelphian who created the modern department store. Doing without rural delivery, he noted, "obliges people to go or send for mail, and that means, in the winter or stormy seasons, and for families of aged people, the depredation of going w/out letters & periodicals (hardly less valuable) that lie in post offices for long periods not called for. We shall look back with astonishment before many years that the present system had to be suffered so long."