Troy Maxon (Denzel Washington) had an affair and had a child outside of his marriage to Rose (Viola Davis).
Rose agree to raise the child (after the mother died in childbirth) but Troy eventually become increasingly estranged from both Rose and his son Cory (Jovan Adepo).
Cory become increasingly frustrated with his fathers action. Cory's believes that his better athlete and will go father than his father ever will. (We are led to believe that Troy's age not color made him unable to play in the majors) To the point where Cory and Troy have a fight to which Troy kicks him out the house.
Several years later Troy has died and Cory now a Marine visits but doesn't want to go the funeral. Rose basically tells him that he is going and rose loved his father even though she may not full forgive his actions.
The Fence were are led to believe is symbolic in Troy's mind is to keep the grim reaper at bay. For the viewer and for Troy, Rose and Lyons (Russell Hornsby) its the emotional barrier that Troy put up to keep his distance from them.