First, let's look at a box plot of the difference between the world record and the winning Olympic time (a value less than zero denotes that the world record was broken). Note: 2 men's world records were set in 2011, so I am comparing to these current records rather than pre-2010 records.
While the times are generally above zero (slower than WR time), the boxplot whiskers do extend below zero. There is one clear outlier for the men, and this ocurred when the 1500m free WR was broken by 3 seconds. Most of the variation is due to events being different distances (50, 100, 200, 400, 800 and 1500m). To account for this difference, I have normalized all times to 100m (multiply 50m time differences by 2, divide 200m time differences by 2, etc.). The normalized times are reported in the following box plots. Now the times are much less variable and there are no clear outliers.
To officially answer our question of whether times were significantly slower without fast suits, I performed a t-test for mean difference. Our null hypothesis is:
Ho: no difference between average world record time and winning Olympic time.
Leaving out the details, we obtain p-values of 0.18 for the men and 0.26 for the women. Thus, since these p-values are large (> typical cutoff of 0.05), we fail to reject the null hypothesis. We can conclude that there is no significant evidence that the winning swimming times in the 2012 Olympics were significantly slower than the world records. I also repeated the calculations after removing the 3 relays from the analysis and arrived at the same conclusion.
We cannot tell from this analysis if the fast suits has a smaller influence on time decreases as originally thought, or if swimmers are just training harder and getting stronger (I tend to believe the latter). It's also too early to tell if any of these records will be thought of as unbreakable (example: Phelp's 2008 Olympic performance + fast suit = some really fast world records). But, I think we can safely conclude that, unlike the steroid era in baseball, world records set in the fast suit era will not require an asterisk.