A childhood home of artist James A. M. Whistler, the Captain Amos Palmer House garden takes its name from a former owner, whose original marble tombstone stands in the garden among English ivy (his present grave is in the local cemetery).

The entrance to the garden off Main Street has granite stairs that rise past a head-high fence topped by Chippendale fretwork. There is an outbuilding that acted as a summer kitchen and cool winter buttery and is now a potting shed. A wide stone path inset in gravel passes the house and enters the full rectangular garden, a winter garden built atop the hummock edged by Wall Street. The long side bed with perennial plants has rounded 'Newport Blue' boxwood and a small fishpond with a circulating waterfall over flat granite stones. The corner linden tree, planted after the hurricane of 1938, rises from a bed of ivy.

The most striking feature is a red Chinese gazebo. Boxwood hedges (Buxus sempervirens 'Suffruticosa')--thick, vigorous, and trimmed knee high--frame the four central, brick-edged squares that are each of a slightly different size. Four other species of boxwood grow in large round shapes to counterbalance the square hedge framing; gravel paths are laid between. The smallest square has a hedge of Ilex crenata 'Compacta' with a raised central festoon of mixed flowers in summer replaced with a shell birdbath in winter. The present layout plan and plantings were begun in the 1980s by the current owners.