If the formula has a syntax error it is reported while adding/editing the formula fields. However, other errors cannot be determined until runtime and hence you will see ERR! while viewing the fields in reports and other places. You can move your mouse over the icon to see the exact error message.
You can then edit the formula and fix it.
From the error message in the screenshot you can see that the formula returns a value which cannot be converted to a date. Hence, the application shows a message accordingly. You now, need to edit the formula to select the appropriate Formula Data Type for fixing this.