Finalize full reindex procedure #895
Description
NOTE: Before starting this check with Zhou to see we want to change the full reindex to take advantage of the new reindex queue functionality, and if so how that affects this work.
- Update the step-by-step guide in both README.md and the script
helpinstructions - Remove misleading print outputs at the end of each command run
- Add a new command to delete those
flushYYYYMMDD_fresh_index_*indices AFTERcatch-up
Edited on 11/09/2024, catch-up threw this error on PROD
[2024-11-09 12:55:42] INFO in hubmap_translator:1362: Start executing _call_indexer() on uuid: 12dcb9de1b1aa098fabaf7991fd3cf78, entity_type: Upload
[2024-11-09 12:55:42] INFO in hubmap_translator:632: Start executing direct 'entities' updates for entity['uuid']=12dcb9de1b1aa098fabaf7991fd3cf78, entity['entity_type']=Upload
[2024-11-09 12:55:42] INFO in hubmap_translator:1567: Start executing _generate_doc() for Upload of uuid: 12dcb9de1b1aa098fabaf7991fd3cf78 for the entities index group.
[2024-11-09 12:55:42] ERROR in hubmap_translator:1754: Exceptions during executing hubmap_translator._generate_doc()
Traceback (most recent call last):
File "/usr/src/app/src/hubmap_translator.py", line 1691, in _generate_doc
entity['ancestors'] = self._relatives_for_index_group( relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment
[2024-11-09 12:55:42] ERROR in hubmap_translator:654: Exception document generation for uuid: 12dcb9de1b1aa098fabaf7991fd3cf78, entity_type: Upload for 'entities' reindex caused 'local variable 'ancestor_ids' referenced before assignment'
Traceback (most recent call last):
File "/usr/src/app/src/hubmap_translator.py", line 1691, in _generate_doc
entity['ancestors'] = self._relatives_for_index_group( relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/app/src/hubmap_translator.py", line 641, in _transform_and_write_entity_to_index_group
private_doc = self._generate_doc( entity=doc_entity
File "/usr/src/app/src/hubmap_translator.py", line 1757, in _generate_doc
raise Exception(e)
Exception: local variable 'ancestor_ids' referenced before assignment
[2024-11-09 12:55:42] ERROR in hubmap_translator:657: For Upload 12dcb9de1b1aa098fabaf7991fd3cf78, failed to generate document for consortium indices.
[2024-11-09 12:55:42] INFO in hubmap_translator:667: Unable to find 'entities' transformer, indexing documents untransformed.
[2024-11-09 12:55:42] ERROR in hubmap_translator:1375: Encountered exception e=local variable 'private_doc' referenced before assignment executing _call_indexer() with uuid: 12dcb9de1b1aa098fabaf7991fd3cf78, entity_type: Upload
Traceback (most recent call last):
File "/usr/src/app/src/hubmap_translator.py", line 1367, in _call_indexer
self._transform_and_write_entity_to_index_group(entity=entity
File "/usr/src/app/src/hubmap_translator.py", line 668, in _transform_and_write_entity_to_index_group
docs_to_write_dict[self.index_group_es_indices[index_group]['private']] = private_doc
UnboundLocalError: local variable 'private_doc' referenced before assignment
[2024-11-09 12:55:42] INFO in hubmap_translator:1195: Finished executing reindex_entity() on uuid: 12dcb9de1b1aa098fabaf7991fd3cf78