Skip to content

Extend Key Length #2242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
echeipesh merged 25 commits into from
Dec 7, 2017
Merged

Extend Key Length #2242

echeipesh merged 25 commits into from
Dec 7, 2017

Conversation

@jamesmcclain
Copy link
Member

@jamesmcclain jamesmcclain commented Jun 21, 2017 •
edited
Loading

  • File
    • Read/write new layers
    • Backward compatibility with old layers
  • Hadoop
    • Read/write new layers
    • Backward compatibility with old layers
    • Is there a Hadoop Collections API bug? Not seen for a while
  • S3
    • Read/write new layers
    • Backward compatibility with old lyaers
  • Accumulo
    • Read/write new layers
    • Backward compatibility with old layers: The original machinery is fixed at 8 bytes. The machinery in this PR always uses at least 8 bytes. For a particular index, if the statement "The number of bytes required by some key in the index is more than 8 implies all keys require the same number of bytes" is a true statement, then this is guaranteed to work. In fact, different-length keys should work too if Accumulo's lexicographic ordering of row IDs behaves as I think/expect/hope.
  • Cassandra
    • Read/write layers
    • Needs a "BigInt iff bigint" codec. That codec will need to be able to read keys serialized by the original "Long iff bigint" codec in order to be able to read old layers.
  • HBase
    • Read/write new layers
    • Backward compatibility with old layers. Old machinery used org.apache.hadoop.hbase.util.Bytes for the Long -> Array[Byte] conversion. The machinery in this PR uses BigInt.toByteArray then pads to at least 8 bytes. See the note about Accumulo backward compatibility above.

See also locationtech/sfcurve#18, feature/geowave-index

Fixes #2238

  • Rebase

@jamesmcclain jamesmcclain force-pushed the feature/key-extension branch 2 times, most recently from 186170f to 702f828 Compare June 22, 2017 02:52
@jamesmcclain jamesmcclain changed the title [WIP] Extend Key Length Extend Key Length Jun 22, 2017
@jamesmcclain jamesmcclain force-pushed the feature/key-extension branch from cb3010a to fa6549f Compare June 23, 2017 00:07
@echeipesh echeipesh added this to the 2.0 milestone Jun 29, 2017
@jamesmcclain jamesmcclain force-pushed the feature/key-extension branch from fa6549f to fd9ddc8 Compare July 3, 2017 22:21
James McClain added 22 commits November 17, 2017 08:38
Remove Excess Whitespace
62283e0
Extend Key Length in KeyIndex.scala
c8db087 
The range length is still expressed as a `Double`, the maximum bin size
is still a `Long`.

Signed-off-by: James McClain <jmcclain@azavea.com>
Second-Level Fallout From Key Extension
7f9fd55 
The range lengths are still expressed as a `Long` values.
Extend Key Length in Partitioning Machinery
bacdaf7 
Note: `Long.MaxValue` has been changed to `BigInt(-1)`.
Extend Key Length in ZCurve Machinery
688f5a7
Extend Key Length in Hilbert Machinery
63ea154
Extend Key Length in "Row Major" Machinery
0428946
Allow Tests To Compile
baf8832
Extend Key Length in File Layer Machinery
c896b64
Extend Key Length in Hadoop Layer Machinery
af2d0e9 
Note: `Long.MaxValue` has been changed to `BigInt(-1)`.
Account For Long.MaxValue -> BigInt(-1) Change
0e9abd3
Account For Behavioral Differences
fbc066e 
Account for differences between `LongWritable` and `BytesWritable`.
BigIntWritable Instead of BytesWritable
fc6f1b5
Extend Range Lengths and Bin Sizes
565222b
Augement Accumulo
71b62bc 
An key length of at least eight bytes is ensured in
`AccumuloKeyEncoder`.  That was because keys of one byte (in the unit
tests) were not working -- extra zero bytes were being mysteriously
added.

Behavior with keys of more than 8 bytes is unknown.
Augment Cassandra
6d3d7b4 
Key length still capped at 64-bits, needs custom coded.
Augment S3
1024d02
Augment HBase
1ac80f4
Update Documentation Examples
5e88964
HBase: Retain Backward Compatibility
48eb6ad
Cassandra: Extend, Retain Backward Compatibility
dff5dbf
Rehabilitate Hadoop After Rebase
e6ecd36
@echeipesh echeipesh merged commit 1b41891 into locationtech:master Dec 7, 2017
@jamesmcclain jamesmcclain deleted the feature/key-extension branch December 9, 2017 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@echeipesh echeipesh echeipesh approved these changes

Assignees

No one assigned

Labels

api change

Projects

None yet

Milestone

2.0

Development

Successfully merging this pull request may close these issues.

Concerning The Length Of Index Keys

2 participants

@jamesmcclain @echeipesh