In order to ensure a signatory isn't an adversary implement a form of signatory verification. Make the amount of data a signatory can upload be a function of how recently they were verified. Make the number of peers it requires to become verified be around 50%, preventing cliques of adversaries from supporting each other. Signatories can periodically seek verification.
This would allow for the eventual removal of adversarial peers without significantly impacting the main use cases.