Duffy spoke with reporters one day after Two On Your Side reported Howard Milstein is interested in buying the Buffalo Bills