But that was a personal failing, says Sanford, who insists he never let down taxpayers or voters.
Yet some voters -- and a growing chorus of his challengers in the 1st Congressional District race -- remember 2009 and 2010 differently. They say the affair was only part of a body of evidence that demonstrated Sanford is unfit for public office, including another term in Congress.
As Sanford works to highlight the parts of his record that, he says, demonstrate a consistent 20-year record of fighting for spending cuts and debt reduction, opponents are highlighting other parts that they say tarnished his public service record including his disappearance from the state for six days and misuse of state aircraft.
Never miss a local story.
"He left his state; he left his country without telling anyone where he was going," said S.C. Rep. Andy Patrick, R-Hilton Head Island, a former Secret Service agent also running for the GOP nomination. Patrick has called Sanford's campaign an "apology tour."
"When I was in the military, they called that going AWOL," Patrick said. "He definitely deserves forgiveness. Every Christian believes that is true. But that doesn't mean he should be given another chance in Congress. We should expect more out of our elected representatives."
Sanford's disappearance and other transgressions resulted in a majority of Republicans in the state Senate calling for him to step down in 2009.
A state House of Representatives committee considered whether to impeach him, ultimately voting to censure him for bringing "ridicule, dishonor, disgrace and shame" on the state, its citizens and the governor's office. He also was censured by the S.C. Republican Party, the first time in the party's history it had ever reprimanded a sitting governor.
That history isn't getting enough attention, contend his fellow GOP rivals, who realize Sanford is the top dog in a crowded race.
And the jabs are getting sharper as the March 19 primary nears.
"He's getting a pass on so much. He's not addressing so much that happened," said Teddy Turner, one of the top three contenders in the race, along with Sanford and Rep. Chip Limehouse, R-Charleston, according to internal polling from various candidates. "It's a pattern of politics as usual."
But Sanford and his allies say he never broke the law, was never completely out of touch during his absence from the state and saved the state money by traveling less than the three previous governors. He also sold the state's jet, a one-time savings of more the $1 million and a recurring $120,000 a year on maintenance, insurance and operation costs.
"That's what's been lost," said state Sen. Tom Davis, R-Beaufort, a friend of Sanford's since college and his former chief of staff, "and what's not being told -- all he did over those eight years to save the state money."
Sanford's traveling troubles took off in June 2009 when he disappeared from the state.
Staff members said they didn't know where he was.
Neither did Lt. Gov. Andre Bauer, who would act in the governor's absence.
Or SLED, which provides governors with security detail.
Or his wife and children, even though it was Father's Day weekend.
After refusing to comment for several days, Sanford's spokesman finally said he was hiking the Appalachian Trail.
But when Sanford was intercepted by a reporter at the Atlanta airport on June 24, he said a last-minute change of plans -- unbeknown to his staff -- took him to Argentina for a few days out of the public spotlight. Later that day, Sanford confessed the trip's real purpose was to visit a woman with whom he was having an extramarital affair.
In the weeks that followed, Sanford's allies claimed he was never totally out of touch with his office during the secret trip.
"He had a back channel way of being reached the entire time," said Sanford's campaign spokesman Joel Sawyer, who was the governor's communications director then. It was Sawyer who told the media Sanford was hiking. "When he was notified on what was going on, he came back."
Sawyer would not identify the person who knew where Sanford was. Sanford never has either.
Sanford himself was not available last week to discuss the disappearance, the ethics fine or use of the state plane. Instead, Sawyer and Davis, who has endorsed Sanford, took questions for him.
Six months after his return, the State Ethics Commission wrapped up an investigation and charged Sanford with breaking state ethics laws 37 times.
In the years since, Sanford has called the charges "minor and technical" and maintained that he never broke the law. But he still paid a $74,000 fine, the largest in state history, and repaid the state for investigative costs.
"We looked at this and said, 'I only have a year and a half left in office, I could spend it contesting this stuff or I could just pay it and move on,'" Sanford told the National Review, a conservative political journal, in January.
Sawyer added Thursday, "We still believe very strongly that if we had argued before the Ethics Commission, we would have been cleared on all fronts."
Cathy Hazelwood, an attorney with the commission, sees it differently.
"We won't speculate what the commission would have done when faced with a mountain of evidence (it had to prove the claims)," Hazelwood said Friday.
MISUSE OF STATE PLANE?
The commission said Sanford broke the law when he flew business or first class 18 times between 2005 and 2009 while on state business to Europe, Asia and South America.
State law requires officials to choose the most economical fare unless there is an urgent reason to do otherwise.
Four of the business-class flights involved a 2008 state Commerce Department trip to Brazil. Sanford asked that the economic-development trip be extended to include Argentina, where he saw his now fiancee, Maria Belen Chapur. He has since reimbursed the state $3,300 for the Argentina leg of the trip.
Sanford's defenders pointed out that previous governors and state leaders routinely flew first class on the state's dime when traveling internationally. The state Commerce Department confirmed the claim, saying the practice meant top state officials arrived rested.
The commission fired back that, in many of the instances, Sanford had a day to recover before any meetings. He also flew business class to return to Columbia.
The ethics matter ended with the commission's ruling.
Both SLED, the state's top law enforcement division, and the state Attorney General's Office looked into the travel for evidence that warranted criminal charges. None was found.
"That tells you something. They looked and found nothing," said Sawyer, adding the Senate never considered an impeachment resolution.
PERSONAL TRIPS ON STATE TIME?
The commission also alleged Sanford broke the law when he used state aircraft for personal travel nine times between 2005 and 2008, including a 2006 flight from North Myrtle Beach to Columbia after which Sanford got a haircut and a 2007 flight from North Myrtle Beach to Columbia for Sanford to attend a son's sporting event.
"With every (trip) there was an official component," Sawyer said.
Sanford has said his office was thorough, perhaps too thorough, in including every activity on his calendar, which gave the false impression that the private events were the reasons for the flight instead of add-ons to official business.
"We just wrote down every little detail on the schedule as things evolved," Sanford told the National Journal.
While some lawmakers on both sides of the aisle were riled and wanted Sanford to step down, they decided his offenses did not merit impeachment.
The sole vote for impeachment was Rep. Greg Delleney, R-Chester, one of seven members of a special House committee that considered impeachment.
"What matters to me," Delleney said at the time, "is his absence from the state without giving anyone notice and being AWOL for five days, leaving no established chain of command or protocol for the exercise of the executive authority of the state and his preconceived deceit and cover-up of his whereabouts using his staff, who are state employees, to mislead public officials of South Carolina and the public of South Carolina. That, coupled with the shame and disgrace he brought to the reputation of South Carolina."
A majority of panel members were mindful that the only two governors recently removed from office were indicted for felonies.
"I was disappointed in the affair and the other issues, including the flights," said Rep. Garry Smith, R-Greenville, who was a member of the special panel. "But we didn't feel it reached the high bar that has been set for impeachment in this state. I still feel we made the right decision."
Smith wouldn't say whether Sanford deserves another turn in Congress.
"That's something only the people in the congressional district can decide," he said.
Follow reporter Gina Smith at twitter.com/GinaNSmith.