Born in the Fort Edward area of New York to Justus and Keziah (Moss) Ashmun. He grew up in Blandford, MA, where his father operated a tavern and was a leading citizen of that town. Phineas studied law and practiced in Stockbridge, MA. In 1791 he married Alithea Wells in Hartford, CT. Their only known child, Maria Antoinette Ashmun, was baptized in West Hartford in 1793 as "Mariah". There may have been more children who did not survive.

He was nominated and served as Captain in the Provisional Army set up by President John Adams in 1799 and 1800. He wrote to Major General Alexander Hamilton on several occasions and Hamilton responded, though indirectly.

Around 1808/9, Phineas moved his family to Belfast, Maine, where he served as a land agent for Sears, Thorndike and Prescott. Phineas was also a trustee of Belfast Academy, which is no longer in operation. They moved to Washington Plantation, just outside of Belfast, where he helped to incorporate the town as Brooks, Maine. He was the first postmaster and a lawyer there. He entertained Daniel Webster in his home.

In 1814, their daughter Maria married William White, a lawyer and Dartmouth graduate. Rev. Silas Warren performed the ceremony. The couple lived in Belfast where William practiced law. Maria died on April 5, 1815, at age 22, four months after giving birth to a stillborn daughter (granddaughter to Phineas and Alithea). No grave has been found for Maria. She may have been buried in an early Belfast cemetery that closed.

Phineas' first wife Alithea died in 1820. He remarried in 1821 to Hannah Clary, daughter of Daniel Clary and Jane Patterson of Brooks. Hannah (Clary) Ashmun died in 1831 at age 30, leaving Phineas with five children under the age of nine. It is assumed that the grandparents and other local family members helped to raise the children. Most of them went on to settle in Massachusetts and beyond. Only son Henry Clay Ashmun remained in Brooks and died there.