The painful secret Tom (Nick Nolte) was so resilient to reveal, and which also explained Savannah's (Melinda Dillon's) recurrent suicidal tendencies was that, one night, back when both he and Savannah were 13, three escaped convicts broke into their house and raped them both, as well as their mother Lila (Kate Nelligan). Older brother Luke caught sight of this and came in, shotgun in hand, killing two of them. The third one was about to shoot Luke when Lila reacted and stabbed him with a kitchen knife, killing him as well. They all vowed to never talk about that again and hide it from their father Henry (Brad Sullivan), as if it never happened.
Eventually, Tom decides to break the silence and tell Dr. Susan Lowenstein (Barbra Streisand) about it, thus finally making amends with the demons of his past. At the same time, he can't hide his growing attraction for her, which surfaces at a party in which her husband, violinist Herbert Woodruff (Jeroen Krabbe), tries to ridiculize him for "turning his son Bernard (Jason Gould) -who had left for college to play football- into a caveman". In reaction, Susan lets Herbert know she is aware that he is cheating on her with the pianist, and Tom forces him to apologize to both himself and Susan, or he will destroy his million-dollar violin.
Susan corresponds to him, and they begin a romance which doesn't last too long, because Susan fears Tom's wife Sally (Blythe Danner) will call him and try to win him back, which eventually happens, even though he had thought of staying in New York to be with Susan and close to Savannah.
He accepts his reality, embraces his newfound peace of mind and leaves Susan and New York to go back to his wife and daughters in Carolina.