hint optimize

[laohanlinux]

bitcask-rs only has one hint file, and flush hint into disk when yields a new data file, the whole process spent too much time and block.

how to optimize?

1. Async write hint file
2. mapping multil hint file to multil data file
3. Use state tree, just write the key need to update only time.

[laohanlinux]

Maybe Use Boltdb store hint file 😹 :

[laohanlinux]

graph LR;
   Hint-->DataFile1;
   Hint-->DataFile2;
   Hint-->DataFile3;
   Hint-->DataFile4;

Loading

• DataFile4 is the current WF (WriteFile)
• HintFile is store with boltdb
  • cache the WF all index

Crash Consistent

1. Only DataFile, Hint Write succeed, Key/Value is writted succeed
2. If crash that the WF is DataFile4, Just recovery from DataFile4
3. If crash that it is Hint，recovery Hint
4. because can recover from WF，don't need WAL file （WAL file also need flush disk），Just flush DataFile

[laohanlinux]

graph LR;
   MemIndex --Sink--> ImmBoltDB;
    MemIndex-->DataFile1;
   MemIndex-->DataFile2;
   MemIndex-->DataFile3;
   ImmBoltDB-->DataFile1;
   ImmBoltDB-->DataFile2;
   ImmBoltDB-->DataFile3;

Loading

How to guard ACID