bug in small binary- “Sampler's range is too small“

[0xling]

I use deepbindiff as following:

➜  DeepBinDiff git:(master) ✗ cat test.c 
#include<stdio.h>
void main()
{
    printf("hello world\n");
}
➜  DeepBinDiff git:(master) ✗ gcc test.c -o test1
➜  DeepBinDiff git:(master) ✗ gcc test.c -o test2
➜  DeepBinDiff git:(master) ✗ python3 src/deepbindiff.py --input1 ./test1 --input2 ./test2 --outputDir ./out

The error full output is as following. It seems the problem is "Sampler's range is too small".

Traceback (most recent call last):
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Sampler's range is too small.
     [[{{node nce_loss/LogUniformCandidateSampler}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/deepbindiff.py", line 233, in <module>
    main()
  File "src/deepbindiff.py", line 220, in main
    tokenEmbeddings = featureGen.tokenEmbeddingGeneration(article, blockBoundaryIndex, insnStartingIndices, indexToCurrentInsnsStart, dictionary, reversed_dictionary, opcode_idx_list)
  File "/mnt/hgfs/deepbindiff/DeepBinDiff/src/featureGen.py", line 392, in tokenEmbeddingGeneration
    embeddings = buildAndTraining(article, blockBoundaryIndex, insnStartingIndices, indexToCurrentInsnsStart, dictionary, opcode_idx_list)
  File "/mnt/hgfs/deepbindiff/DeepBinDiff/src/featureGen.py", line 348, in buildAndTraining
    _, loss_val = session.run([optimizer, loss], feed_dict=feed_dict)
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 957, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1180, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1358, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Sampler's range is too small.
     [[node nce_loss/LogUniformCandidateSampler (defined at /mnt/hgfs/deepbindiff/DeepBinDiff/src/featureGen.py:310) ]]

Original stack trace for 'nce_loss/LogUniformCandidateSampler':
  File "src/deepbindiff.py", line 233, in <module>
    main()
  File "src/deepbindiff.py", line 220, in main
    tokenEmbeddings = featureGen.tokenEmbeddingGeneration(article, blockBoundaryIndex, insnStartingIndices, indexToCurrentInsnsStart, dictionary, reversed_dictionary, opcode_idx_list)
  File "/mnt/hgfs/deepbindiff/DeepBinDiff/src/featureGen.py", line 392, in tokenEmbeddingGeneration
    embeddings = buildAndTraining(article, blockBoundaryIndex, insnStartingIndices, indexToCurrentInsnsStart, dictionary, opcode_idx_list)
  File "/mnt/hgfs/deepbindiff/DeepBinDiff/src/featureGen.py", line 310, in buildAndTraining
    loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weights,
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/ops/nn_impl.py", line 2046, in nce_loss
    logits, labels = _compute_sampled_logits(
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/ops/nn_impl.py", line 1742, in _compute_sampled_logits
    sampled_values = candidate_sampling_ops.log_uniform_candidate_sampler(
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/ops/candidate_sampling_ops.py", line 149, in log_uniform_candidate_sampler
    return gen_candidate_sampling_ops.log_uniform_candidate_sampler(
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/ops/gen_candidate_sampling_ops.py", line 656, in log_uniform_candidate_sampler
    _, _, _op, _outputs = _op_def_library._apply_op_helper(
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
    op = g._create_op_internal(op_type_name, inputs, dtypes=None,
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3319, in _create_op_internal
    ret = Operation(
  File "/home/ling/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1791, in __init__
    self._traceback = tf_stack.extract_stack()

[kgautam01]

Hi, @0xling @yueduan @deepbindiff,

As mentioned above, I am getting the same error but for the stripped versions of two binaries (compiled with clang O0 and O1).
Please let me know how to resolve the issue.

Thanks

[kgautam01]

Hi, @0xling @yueduan @deepbindiff,

As mentioned above, I am getting the same error but for the stripped versions of two binaries (compiled with clang O0 and O1). Please let me know how to resolve the issue.

Thanks

I found the solution to the above problem. This happens when the hyperparameters batch size and num_samples have higher values than what is required for the given binary. Reduce them in the same ratio and it should work.