Regarding selecting the least important word for removing

[ashim95]

Hi @ihsgnef
Thanks for releasing the code, which I must say is very well documented.

For your input reduction method, you want to remove the least important words from the sequence. In line 97 here, you use np.argsort(- * () ) which should sort it by most important to least important words (as you are doing argsort over negative, so it sorts by descending to ascending). You are removing these words, which are most important instead of least important words. Shouldn't you be removing the words with least delta?

Am I missing something?

Also can you describe what you are trying to do from line 98 onwards?

Thanks,
Ashim

[ihsgnef]

@ashim95 I think you are probably right. I need to look into this further and see how this affects the results. People have independently replicated the result so I suspect this might be an issue with this code release. I'll keep you posted. Thanks!

[ihsgnef]

Sorry it took a while for me to work on this and update code for latest pytorch version. Extraction of input gradient under evaluation mode (mainly for dropout to run in eval mode) is now a bit trickier than older pytorch version but I think it's implemented correctly.

Back to your question, I think the implementation is correct: largest input gradient corresponds to the least important word. Let's go through this step by step:

• we are computing the gradient of CrossEntropyLoss w.r.t. embedding dot product with embedding
• large gradient means large increase in loss when epsilon is added to embedding
• large gradient means large decrease in loss when epsilon is subtracted from embedding
• large gradient approx large decrease in loss when word is removed
• large gradient approx small decrease in confidence when word is removed
• large gradient means word is unimportant
• we sort by descending order as you pointed out, so first word is the least important one

Here is a notebook comparing leave-one-out and input-gradient. About 80% of the time, the most important word selected by input gradient is among the top 3 most important word selected by leave-one-out, about 50% it's exact match. Note that this is the computation done in one step of input reduction on the original input. 37% exact match of most unimportant word is lower than I remembered, but last time I checked this I run the full reduction process, and with fewer words left in the input, gradient is usually more accurate.

https://github.com/ihsgnef/pathologies/blob/a87f9222f3f26fd7de01064942dfe69a9d32a492/snli_bimpm/grad_vs_l1o.ipynb

[ashim95]

Thank you @ihsgnef for the jupyter notebook. I will go through the notebook and ask if something is not clear.

Once again, thanks a lot!