Seen similar shapes around the old mill buildings just north of Pendle, so I dont think you're losing it at all. Shadow figures specifically tend to cluster around sites with a long history of emotional distress - old houses, asylums, places where people suffered. The Brennan house has a reputation going back decades if the local accounts are anything to go by.
What I'd want to know is whether the figure had any discernible shape to it. Hat man profile, or more of a flat amorphous mass? That distinction actually matters quite a bit when you're trying to categorise the phenomenon. Hat man reports tend to have a very specific behavioural pattern - standing still, watching, then vanishing when directly observed. The amorphous type moves differently.
Were you with anyone else when you saw it, or on your own? Solo witnesses get dismissed too easily but honestly solo sightings are often more credible because theres no group hysteria effect muddying the water. Would be good to hear more detail from you if you're comfortable sharing.