Implementation of sc-memory based on disk space

[bogdansemchenko]

• Read PR documentation
• Update changelog
• Update documentation

---

Important

Implements disk-based sc-memory with L1/L2 caching, asynchronous I/O, and buffered operations for enhanced efficiency and scalability.

• Memory Management:
  • Introduces L1 and L2 caches in sc_storage.c for efficient memory management using DRAM and SSD/NVMe.
  • Implements _sc_l1_cache_init(), _sc_l1_cache_destroy(), _sc_l1_cache_add(), _sc_l1_cache_get(), _sc_l2_cache_init(), _sc_l2_cache_destroy(), _sc_l2_cache_add(), _sc_l2_cache_get() for cache operations.
  • Adds sc_segment_buffer for buffering operations in sc_storage.c.
• Asynchronous and Parallel I/O:
  • Implements write_segment_async() and write_segments_parallel() in sc_fs_memory.c for asynchronous and parallel segment writing.
  • Utilizes libaio and pthread for I/O operations.
• Buffering and Operations:
  • Adds _sc_storage_add_operation() and _sc_storage_flush_buffer() in sc_storage.c to handle buffered operations.
  • Supports operations like SC_OP_NODE_NEW, SC_OP_ARC_NEW, SC_OP_LINK_NEW, SC_OP_SET_LINK_CONTENT, SC_OP_ERASE_ELEMENT.
• File System Integration:
  • Modifies sc_fs_memory.c to handle file-based storage, including reading and writing segments to disk.
  • Updates sc_file_system.c and sc_io.h for file operations.
• Miscellaneous:
  • Updates sc_storage_initialize() and sc_storage_shutdown() in sc_storage.c to integrate new memory management features.
  • Adds new data structures and functions in sc_storage_private.h for internal storage management.

^{This description was created by for fc6f7f5. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

Caution

Changes requested ❌

Reviewed everything up to fc6f7f5 in 3 minutes and 18 seconds. Click for details.

• Reviewed 4739 lines of code in 12 files
• Skipped 0 files when reviewing.
• Skipped posting 4 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. sc-memory/sc-core/src/sc-store/sc-fs-memory/sc_file_system.c:92

• Draft comment:
  BUG: The allocated 'command' buffer is not initialized before using strcat. This may lead to undefined behavior. Use strcpy or zero‐initialize the buffer.
• Reason this comment was not posted:
  Comment was on unchanged code.

2. sc-memory/sc-core/src/sc-store/sc-fs-memory/sc_file_system.c:205

• Draft comment:
  BUG: 'char_result' is used without allocation. sc_str_cpy is called on an uninitialized pointer.
• Reason this comment was not posted:
  Comment was on unchanged code.

3. sc-memory/sc-core/src/sc-store/sc_segment.h:35

• Draft comment:
  Typo in the parameter comment: "Number of created instance in sc-memory" should be "Number of created instances in sc-memory".
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

4. sc-memory/sc-core/src/sc-store/sc_storage.c:660

• Draft comment:
  Typographical note: The identifier 'null_ptr' in this new line may be a typo. If the intention is to use a null pointer literal, consider using 'nullptr' (in C++) or 'NULL' (in C) depending on your code's conventions.
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_Q1jJLymj506I50iT

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

BUG: In _sc_l1_cache_add, using &seg_num as key for hash table insertion is unsafe. Use a stable key (e.g. GINT_TO_POINTER) instead.

Suggested change

	sc_hash_table_insert(cache->segments, &seg_num, node);
	sc_hash_table_insert(cache->segments, GUINT_TO_POINTER(seg_num), node);

[ellipsis-dev]

BUG: In _sc_l2_cache_add, the key is inserted using &seg_num, a pointer to a local variable. Use a stable representation (e.g. GINT_TO_POINTER(seg_num)).