The National Security Agency for almost three years searched a massive database of Americans’ phone call records attempting to identify potential terrorists in violation of court-approved privacy rules, and the problem went unfixed because no one at the agency had a full technical understanding of how its system worked, according to new documents and senior government officials.