(AP Photo/Tony Gutierrez)
Kennedy and his sister, Rory, spoke about their family Friday night while being interviewed in front of an audience by Charlie Rose at the Winspear Opera House in Dallas.
The event comes as a year of observances begins for the 50th anniversary of the president's death.
Their uncle was killed on Nov. 22, 1963, while riding in a motorcade through Dallas. Five years later, their father was assassinated in a Los Angeles hotel while celebrating his win in the California Democratic presidential primary.
Kennedy said his father spent a year trying to come to grips with his brother's death, reading the work of Greek philosophers, Catholic scholars, Henry David Thoreau, poets and others, "trying to figure out kind of the existential implications of why a just God would allow injustice to happen of the magnitude he was seeing."
He said his father thought the Warren Commission, which concluded Lee Harvey Oswald acted alone in killing the president, was a "shoddy piece of craftsmanship."
"The evidence at this point I think is very, very convincing that it was not a lone gunman," Kennedy said, but he didn't say what he believed may have happened.
Kennedy said his father, U.S. attorney general at the time, and later elected U.S. senator to New York, was "fairly convinced" others were involved.
Rose asked if he believed his father felt "some sense of guilt because he thought there might have been a link between his very aggressive efforts against organized crime."
Kennedy replied: "I think that's true. He talked about that. He publicly supported the Warren Commission report but privately he was dismissive of it."
He said his father had investigators do research into the assassination and found that phone records of Oswald and nightclub owner Jack Ruby, who killed Oswald two days after the president's assassination, "were like an inventory" of mafia leaders the government had been investigating.
The attorney and well-known environmentalist also told the audience lighthearted stories Friday about memories of his uncle. As a young child with an interest in the environment, he said, he made an appointment with his uncle to speak with him in the Oval Office about pollution.
He'd even caught a salamander to present to the president, which unfortunately died before the meeting.
"He kept saying to me, `It doesn't look well,"' he recalled.
Rory Kennedy, a documentary filmmaker whose recent film "Ethel" looks at the life of her mother, also focused on the happier memories. She said she and her siblings grew up in a culture where it was important to give back.
"In all of the tragedy and challenge, when you try to make sense of it and understand it, it's very difficult to fully make sense of it," she said. "But I do feel that in everything that I've experienced that has been difficult and that has been hard and that has been loss, that I've gained something in it."
"We were kind of lucky because we lost our members of our family when they were involved in a great endeavor," her brother added. "And that endeavor is to make this country live up to her ideals."