Answer by taking a quote from the following article:

Sir Walter Scott, 1st Baronet, FRSE (15 August 1771 - 21 September 1832) was a Scottish historical novelist, playwright, poet and historian. Many of his works remain classics of both English-language literature and of Scottish literature. Famous titles include Ivanhoe, Rob Roy, Old Mortality, The Lady of the Lake, Waverley, The Heart of Midlothian and The Bride of Lammermoor. Although primarily remembered for his extensive literary works and his political engagement, Scott was an advocate, judge and legal administrator by profession, and throughout his career combined his writing and editing work with his daily occupation as Clerk of Session and Sheriff-Depute of Selkirkshire.

When Scott was a boy, he sometimes travelled with his father from Selkirk to Melrose, where some of his novels are set. At a certain spot the old gentleman would stop the carriage and take his son to a stone on the site of the Battle of Melrose (1526).  During the summers from 1804, Scott made his home at the large house of Ashestiel, on the south bank of the River Tweed 6 miles (9.7 km) north of Selkirk. When his lease on this property expired in 1811, Scott bought Cartley Hole Farm, downstream on the Tweed nearer Melrose. The farm had the nickname of "Clarty Hole", and when Scott built a family cottage there in 1812 he named it "Abbotsford". He continued to expand the estate, and built Abbotsford House in a series of extensions. The farmhouse developed into a wonderful home that has been likened to a fairy palace. Scott was a pioneer of the Scottish Baronial style of architecture, therefore Abbotsford is festooned with turrets and stepped gabling. Through windows enriched with the insignia of heraldry the sun shone on suits of armour, trophies of the chase, a library of more than 9,000 volumes, fine furniture, and still finer pictures. Panelling of oak and cedar and carved ceilings relieved by coats of arms in their correct colours added to the beauty of the house.  It is estimated that the building cost Scott more than PS25,000 (equivalent to PS1,900,000 in 2016). More land was purchased until Scott owned nearly 1,000 acres (4.0 km2). A Roman road with a ford near Melrose used in olden days by the abbots of Melrose suggested the name of Abbotsford. Scott was buried in Dryburgh Abbey, where his wife had earlier been interred. Nearby is a large statue of William Wallace, one of Scotland's many romanticised historical figures. Abbotsford later gave its name to the Abbotsford Club, founded in 1834 in memory of Sir Walter Scott.

Is that where he grew up?
Scott built a family cottage there in 1812 he named it "Abbotsford".