This type of augmentation might be risky to apply under some conditions so in the README it would be good to add "What are the limitations of this transformation?" section and explicitly add domain constraints, language constraints and so on.

Potentially generation model could output top n predictions instead of one

I don't see a Robustness Evaluation in the README.md. Per the organizers e-mail, can you run evaluate.py?

Concerning the robustness evaluation, do you you mean running code from https://github.com/GEM-benchmark/NL-Augmenter/tree/main/evaluation?

Potentially generation model could output top n predictions instead of one

that's correct, this is currently supported by our code by setting num_return_sequences parameter of generate method to n.
Adding it explicitly to the README

I don't see a Robustness Evaluation in the README.md. Per the organizers e-mail, can you run evaluate.py?

Concerning the robustness evaluation, do you you mean running code from https://github.com/GEM-benchmark/NL-Augmenter/tree/main/evaluation?

Kaustubh sent an e-mail about the PR process. In it, it says "To evaluate your transformation, run the evaluate.py command and add the results of the models in your transformation's readme under the "Robustness Evaluation" section." That's all I am asking for - to add that section the README.

Please add your transformation name to the test/mapper.py in the right dictionary for the pytest to pick up your test.json. By default, we're testing only light transformations and filters.

Done!

Hi
do you need any additional change before approval?
Thanks