An infected belly button piercing causes swelling, pus, fever, chills, and even an upset stomach. If the belly button piercing is infected keep it in and clean the area with antibacterial soap.