Smaller crates.io package

[kornelski]

I've noticed that encoding_rs is in the top 10 crates using most bandwidth of all crates.io.

I've managed to make the package 3× smaller:

• I've separated tests that need test_data and excluded them and the data from the tarball. All other tests still work even when run from the crates.io package.
• I've replaced [u8; 2] tables with include_bytes!. These tables were using 18 bytes to represent two. Now the bytes aren't even loaded when their feature flags are off. I haven't changed the u16 tables, because they're trickier due to alignment.

The generate-encoding-data.py is for Python 2, so I was unable to run it. I've dumped the data this way:

    for (name, data) in [
        ("big5_level1_hanzi_bytes", &BIG5_LEVEL1_HANZI_BYTES[..]),
        ("big5_unified_ideograph_bytes", &BIG5_UNIFIED_IDEOGRAPH_BYTES),
        ("jis0208_level1_kanji_shift_jis_bytes", &JIS0208_LEVEL1_KANJI_SHIFT_JIS_BYTES),
        ("jis0208_kanji_bytes", &JIS0208_KANJI_BYTES),
        ("cp949_hangul_bytes", &CP949_HANGUL_BYTES),
        ("ksx1001_unified_hanja_bytes", &KSX1001_UNIFIED_HANJA_BYTES),
        ("ksx1001_compatibility_hanja_bytes", &KSX1001_COMPATIBILITY_HANJA_BYTES),
        ("gb2312_level1_hanzi_bytes", &GB2312_LEVEL1_HANZI_BYTES),
        ("gbk_hanzi_bytes", &GBK_HANZI_BYTES),
    ] {
        let flattened: Vec<u8> = data.iter().flat_map(|x| x).copied().collect();
        std::fs::write(format!("data/{name}.bin"), flattened);
    }

The diff for deletion is huge, but commits should be viable individually.

[hsivonen]

I acknowledge that I have seen this, but this will take more time for me to merge, because I want to port this to the generator script. (Which I need to do in a VM due to Python 2. Rewriting the generator in Rust would be nice in theory but never important enough. I tried porting the script to Python 3 in September but gave up, because Python 3 had done disruptive-enough changes to how comparison methods work to make porting non-trivial.)

People have complained about test data being included before, so I guess it's time to remove it. It remains to be seen if Linux distros will then complain about the crates.io package not being complete.

[kornelski]

I've moved test data to a separate PR to unblock that change.

[kornelski]

As for the generation script, maybe you can make it write the arrays to different Rust files of a helper crate, and use the Rust code I've provided as a post-processing step? It's not elegant, but that doesn't seem that important given how rarely if ever this needs to run.

[hsivonen]

Acknowledgement that I haven't forgotten about this. Happily, thanks to #124, the generator script now works with Python 3.