Because a fact is by definition actual, the phrase actual fact is redundant. Fact alone conveys the same meaning in fewer words. If you want something that sounds more substantial, consider in actuality, in reality, or actually in place of in actual fact.
Examples
In these examples, actual could be removed with no loss of meaning:
This has in actual fact back-fired, because now it gives me no choice but to comment . . . [North Country Gazette]
We would not expect Mount Sinai to have a marketing department but in actual fact does. [Financial Post]
Fisichella was in actual fact reputed to be in pole position to seal the deal . . . [Crash.net]
