Super Short Version:
The key never actually belonged to Oskar's father, but instead a man who accidently sold it to Oskar's father at an estate sale. Instead, Oskar finds one last note from his father under his father's favorite swing, giving Oskar the closure he needed.
After his father (Tom Hanks) dies in the 9/11 attacks, Oskar Schell (Thomas Horn) eventually finds an envelope with the name "Black" written on it inside a vase in his father's closet. Inside the envelope is a key and newspaper cutout that has words circled, telling him to never stop looking. Oskar believes this is one last adventure mystery his father has left for him, which was a common thing when he was still alive. Oskar decides to visit every person with the last name of Black in New York. Eventually, he meets his grandmother's "tenant", an old man who doesn't speak (Max von Sydow). The old man agrees to help him with his search. After meeting numerous Blacks to no avail over a more than 3 month period, the search seems like it will amount to nothing.
Oskar then comes to two realizations. One is that the old man is actually his grandfather, who stopped speaking after abandoning his wife and child for a period. The other revelation is that on the backside of the newspaper, a phone number for an estate sale is also circled. Oskar calls this number and reaches Abby Black (Viola Davis), the very first person he visited on his search. She takes him to her now ex-husband, William (Jefferey Wright). William himself had lost a father shortly before Oskar. His father left him a letter, that he didn't open up for some time. In the letter, it explained that William's father had left him the key to his safety deposit inside a vase. However, William didn't open the letter until after he had sold the vase to Oskar's father. So, the key was not a clue Oskar's father left him after all. Oskar hands over the key to Williams, who offers to let Oskar come with him to open the box. Oskar declines, and leaves in a fit of tears, running home.
Once home, his mom (Sandra Bullock), comforts him. She has mostly been emotionally absent for Oskar since his father died, to the point where Oskar calls her on her absenteeism. However, his mother has been keeping tabs on him all along. She figured out his plan to meet all the Black's very quickly and even figured out in which order he would visit them (Oskar had a grid system and a certain filing method, he was a very intelligent kid). She visited most of the Blacks before Oskar did, so they would be kinder and not confused when he talked to them. Some day after this, Oskar visits the park where his father tried to get him to play on the swing set, but Oskar thought is was too dangerous. Oskar decides to give it a try and approaches his father's favorite swing. Underneath the swing is taped a message from his father, giving Oskar the closure he was desperately searching for with the key.