Support std::string_view & boost::string_view

[poyenc]

Description

Support both std::string_view and boost::string_view as tokenizer's string type argument for following separators:

1. offset_separator
2. char_separator
3. char_delimiters_separator

Due to the escaped_list_separator's transformation semantics, we cannot simply use std::string_view as return token type. But in the future, I think we can let users to decide the return token type and input type separately instead of bind them together, to support escaped_list_separator in some conditions, e.g. tokenize std::string_view to std::string(s).

Changes

1. Remove useless helper type assign_or_plus_equal which does nothing special.
2. Add named function templates assign()/append()/clear() for token object to replace overload operator usages, and provide customization points to users (find candidates via ADL).
3. Remove repeated string reading logic, improve time complexity from O(2n) down to O(n).

Note

This pull request is based on #17, please merge it first.

[codecov]

Codecov Report

Merging #18 into develop will increase coverage by 17.72%.
The diff coverage is 87.80%.

@@             Coverage Diff              @@
##           develop      #18       +/-   ##
============================================
+ Coverage    70.95%   88.67%   +17.72%     
============================================
  Files            3        3               
  Lines          210      212        +2     
  Branches        74       64       -10     
============================================
+ Hits           149      188       +39     
+ Misses          18        0       -18     
+ Partials        43       24       -19     

Impacted Files	Coverage Δ
include/boost/token_functions.hpp	88.88% <87.50%> (+17.70%)	⬆️
include/boost/token_iterator.hpp	88.46% <100.00%> (+18.09%)	⬆️
include/boost/tokenizer.hpp	83.33% <0.00%> (+16.66%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c8a8bf...6a3c01b. Read the comment docs.

[poyenc]

need to figure out why we have coverage drop

[jeking3]

Please do not check in a .gitignore file, it is not needed here (you can use user profile based ignores if necessary).

Please first get #17 committed then rebase this on top of that.

Thanks!