Duplicates removal

[turchmo]

How does the InvitationModel manage possible duplicated pairs both inside the in- and out-domain data?

[amirkamran]

The algorithm does not really care if there are duplicate sentences in both in- or out- domain data. As it is based on IBM Model 1, so if there will be duplicates, it will have some effect on the counts/probabilities. Thats up to the users, if they want to pass data with unique sentences or not. In the end you will get a ranking of the sentences, the duplicate sentences will going to have the same ranking score.