President Trump defended his initial response to the Charlottesville violence on Monday, insisting that he couldn't have made his condemnation of white supremacists and other hate groups earlier "because I didn't know all the facts."

"I wanted to make sure, unlike most politicians, that what I said was correct," Trump said.

On Saturday, Trump chided "many sides" for the violence in Charlottesville. A full two days later, on Monday, he directly condemned the KKK, neo-Nazis and white supremacists.

At Trump Tower in New York City on Tuesday, Trump even read back his initial statement, calling it a "fine statement" and blamed the press for being dishonest in its coverage.

"There was no way of making a correct statement that early. I had to see the facts. Unlike a lot of reporters – I didn't know David Duke was there. I wanted to see the facts."

Trump returned to his residence in the gold-leaf comfort of Trump Tower for the first time since he took office in January – but it hasn't been a particularly joyful homecoming.

After arriving late Monday night, when his motorcade passed throngs of protesters gathered on Fifth Avenue chanting "New York hates you" and "Black Lives Matter," Trump spent part of Tuesday dealing with the continued fallout from his belated response to the violence over the weekend Charlottesville, Va.

Trump huddled with staff and signed an executive order on infrastructure – at a podium affixed with the presidential seal in front of the elevator bank – that did little to change the conversation.

In the gray marble and gold-mirrored lobby of his famed tower, shoppers and tourists, having passed through metal detectors at the Fifth Avenue entrance, milled with Secret Secret agents and other law enforcement officials keeping ever-watchful eyes on the proceedings.

Reporters, meanwhile, remained in a small pen bordered by velvet ropes. Television cameras stood like an artillery line.

The infrastructure announcement – intended to streamline the permitting process for infrastructure projects – is part of Trump's ongoing effort to try and toll back federal regulations that he says undermine economic development. Many of the targeted regulations involve environmental restrictions.

Yet even as Trump heralded the order as a way to promote jobs, business leaders within his circle appeared to be more focused on how the president handled clashes between white supremacists and counter-protesters.

So far, four senior leaders from president's business council stepped down amid criticism that Trump was too slow to directly condemn violence involving white supremacists and neo-Nazis. Activists are continuing to pressure remaining members to follow suit.

"No adviser committed to the bipartisan American traditions of government can possibly believe he or she is being effective at this point," tweeted Lawrence Summers, a high-level economic adviser to Democratic presidents Bill Clinton and Barack Obama.

Intel CEO Brian Krzanich, joining leaders of Under Armour and Merck in his resignation from the council, said in a company blog post Monday that he resigned "to call attention to the serious harm our divided political climate is causing to critical issues, including the serious need to address the decline of American manufacturing."

Trump tweeted his displeasure, saying that "for every CEO that drops out of the Manufacturing Council, I have many to take their place. Grandstanders should not have gone on. JOBS!"

Right after, a fourth member of the manufacturing council – Scott Paul, president of the Alliance for American Manufacturing – announced he, too, would step down.

Trump and aides said he has been as critical as possible of the Charlottesville violence, including of a man who rammed his car into a crowd, killing a woman who had counter-protested the rally by white supremacists who objected to the city's planned removal of a statue of Confederate Gen. Robert E. Lee.

Yet Trump did little to stem the controversy. Hours after making a formal statement Monday denouncing those who perpetuate racially-motivated attacks, Trump returned Monday night to a usual line of criticism: the press. "Made additional remarks on Charlottesville and realize once again that the #Fake News Media will never be satisfied...truly bad people!"

Trump stirred up more Twitter trouble after that. He retweeted a prominent figure from the alt-right who pushed the "Pizzagate" and Seth Rich conspiracy theories, who questioned why there was no similar outrage over violence in Chicago.

He tweeted than deleted a post that included the cartoon of a train – the "Trump train" running over a CNN reporter, an image that drew criticism in the wake of the deadly car ramming incident in Virginia.

The president also re-tweeted than deleted a comment from a critic calling him "a fascist."

Steady rain in midtown Manhattan kept away many protesters Tuesday -- as did a heavy security perimeter that included a line of white sanitation trucks parked along Fifth Avenue, flanking the entrance to Trump Tower.

Trump is expected to return to his 17-day working vacation at his golf club in Beminster, N.J., later on Wednesday.