In 1853 Gerrit Smith, abolitionist, prohibitionist, temperance advocate, and businessman, donated funds to establish the Oswego City Library. On April 15, 1854 by act of the New York State Legislature, the library was incorporated. In 1855, the library was constructed.

The library building, erected by the architectural firm of Hewes and Rose of Syracuse, is built in the "Norman" style, with towers, corbels, and crenellated battlements reflecting those of a medieval castle. Now on the National Registry of Historical Buildings, the building is one of the oldest libraries built in New YorkState to serve continuously as a library.