I really think it is perhaps just your imagination. Personally I don't really like the taste of cooked fish much, so the likes of sushi, lightly smoked salmon etc are far more appealing to me and I am yet to have any bad experiences so I really don't think it you should worry too much.