Yes, in the first stanza of the poem, it seems that the narrator of the poem was feeling comfortable knowing that the owner of the woods lived in the village. So he would not see him stopping there and watching the woods. The narrator indeed feared the owner or anyone for that matter seeing him there. Probably he was a man of lower status in society. Another reason was that he didn’t want to be disturbed.

“Whose woods these are I think I know.
His house is in the village, though;
He will not see me stopping here
To watch his woods fill up with snow.”