It depends. If you're still working and under 59 1/2, the answer is no.
If you're retired (or still working and age 59 1/2 or older), the answer is yes.
In both the above situations, there would be no penalty. However, there might be taxes due. If you were able to invest in precious metals through an IRA, or other tax advantaged account, you could rollover funds from the TSP to the new plan and not have to pay taxes. If, on the other hand, your precious metal investment was not in a tax advantaged account, you would pay tax (at your rate for ordinary income) on your withdrawal.