diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..dfe7056 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,31 @@ +--- +name: Bug report +about: Create a report to help us improve +title: '' +labels: Bug +assignees: '' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**Full copied text from the MS-DOS view** +Please, include the command used to run regionfixer. + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Screenshots** +If applicable, add screenshots to help explain your problem. + +**Files that would help solving the issue** +If possible, the world/files that triggers the error. + +**Desktop (please complete the following information):** + - OS: [e.g. iOS] + - Python version: [e.g. 2.7] + - Region Fixer Version [e.g. 2.0.1] + +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..ae32cb5 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,23 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: '' +labels: Feature request +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to happen. + +**Describe how the solution would be implemented** +If possible, describe how the solution would be implemented. + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. + +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt index e4fc248..9cb2cc9 100644 --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -1,6 +1,17 @@ -In no particular order: +Original author: Fenixin (Alejandro Aguilera) - Main developer + +Contributors (in no particular order): + aheadley (Alex Headley) - First multiprocessing version of Region Fixer. -carlallen (Carl Allen) - Fix problem in MacOS +734F96 (Lavander) - Update RegionFixer for Minecraft 1.18 +sleiss (Simon Leiß) - Fix typos kbn (Kristian Berge) - Small fixes +KasperFranz (Kasper Sanguesa-Franz) - Fix typo in readme +macfreek (Freek Dijkstra) - Fixes and lots of help +Pisich (carloser) - Changes to the readme +carlallen (Carl Allen) - Fix problem in MacOS +charlyhue (Charly Hue) - Fix logging with onliners +andm (andm) - Fix typos +sandtechnology (sandtechnology) - Fix problem scanning old worlds diff --git a/DONORS.txt b/DONORS.txt index 699dca4..0e35a92 100644 --- a/DONORS.txt +++ b/DONORS.txt @@ -4,6 +4,11 @@ Travis Wicks Nico van Duuren (Knights and Merchants) Diana Rotter Biocraft +Andrew Van Hise +Eugene Sterner +Udell Ross Burton +Powercraft Network +David Wilczewski Sponsors: -Intial development was sponsored by: NITRADO Servers (http://nitrado.net) +Initial development was sponsored by: NITRADO Servers (http://nitrado.net) diff --git a/README.rst b/README.rst index 5130756..98c99a4 100644 --- a/README.rst +++ b/README.rst @@ -3,13 +3,12 @@ Minecraft Region Fixer ====================== By Alejandro Aguilera (Fenixin) -Sponsored by NITRADO servers (http://nitrado.net) Locates problems and tries to fix Minecraft worlds (or region files). -Tries to fix corrupted chunks in region files using old backup copies +Minecraft Region Fixer tries to fix corrupted chunks in region files using old backup copies of the Minecraft world. If you don't have a copy, you can eliminate the -corrupted chunks making Minecraft recreate them. +corrupted chunks making Minecraft regenerate them. It also scans the 'level.dat' file and the player '\*.dat' and tries to read them. If there are any problems it prints warnings. At the moment @@ -18,301 +17,35 @@ it doesn't fix any problem in these files. Web page: https://github.com/Fenixin/Minecraft-Region-Fixer +Mincraft forums posts: +https://www.minecraftforum.net/forums/support/server-support-and/1903200-minecraft-region-fixer +https://www.minecraftforum.net/forums/mapping-and-modding-java-edition/minecraft-tools/1261480-minecraft-region-fixer Supported platforms =================== -This program seems to work with Python 2.7.x, and DOESN'T work with -python 3.x. There is also a windows executable for ease of use, if you -use the windows executable you don't need to install Python. - - -Windows .exe downloads -====================== -The window executable is generated using py2exe and is the choice if -you don't want to install python in your system. - -These downloads were usually in the downloads section of github, but -github has deprecated this feature. So, from Region Fixer v0.1.0 -downloads are stored in mediafire: - -http://www.mediafire.com/?1exub0d8ys83y -or -http://adf.ly/HVHGu (if you want to contribute a little) - +This program only works with Python 3.x, and DOESN'T work with +python 2.x. There was a Windows .exe for older versions, but right +now you need to install the python interpreter to run this +program. Notes ===== -Older versions of Minecraft had big problems when loading corrupted -chunks. But in the latest versions of Minecraft (tested in 1.4.7) the -server itself removes corrupted chunks (when loading them) and -regenerate those chunks. Region-Fixer still is useful for replacing -those chunks with a backup, removing entities, or trying to see what's -going wrong with your world. +Older versions of Minecraft had big problems when loading broken +worlds. Newer versions of Minecraft are improving the way +they deal with corruption and other things. + +Region-Fixer still is useful for replacing chunks/regions with a +backup, removing entities, or trying to see what's going wrong +with your world. Usage ===== -You can read the program help running: “python region-fixer.py --help” - -(NOTE: if you downloaded the .exe version for windows, use - "region-fixer.exe" instead of "python region-fixer.py") - -Here are some examples: - -From v0.1.0 Region-Fixer can scan single region files and arbitrary -region sets. For example, if you know where the problem is you could -scan a single region file instead of scanning the whole world. You -can also scan a few region files from different locations. Example:: - - $ python region-fixer.py ~/.minecraft/saves/World1/region/r.0.0.mca - - Welcome to Region Fixer! - - ############################################################ - ############## Scanning separate region files ############## - ############################################################ - Scanning: 1 / 1 100% [########################################] Time: 00:00:01 - - Found 0 corrupted, 0 wrong located chunks and 0 chunks with too many entities of a total of 976 - -The next example will scan your world and report any problems:: - - $ python region-fixer.py ~/.minecraft/saves/corrupted-world - - Welcome to Region Fixer! - - ############################################################ - ############ Scanning world: Testing corruption ############ - ############################################################ - Scanning directory... - Info: No nether dimension in the world directory. - Info: No end dimension in the world directory. - There are 1 region files and 1 player files in the world directory. - - -------------------- Checking level.dat -------------------- - 'level.dat' is redable - - ------------------ Checking player files ------------------- - All player files are readable. - - ------------------ Scanning the overworld ------------------ - Scanning: 1 / 1 100% [########################################] Time: 00:00:20 - - Found 19 corrupted, 0 wrong located chunks and 0 chunks with too many entities of a total of 625 - -You can use --verbose or -v option if you want more info. This option -will print a line per region file showing problems found in that region -file. - -To delete corrupted chunks you can use "--delete-corrupted" or "--dc":: - - $ python region-fixer.py --delete-corrupted ~/.minecraft/saves/corrupted-world - - Welcome to Region Fixer! - - ############################################################ - ############ Scanning world: Testing corruption ############ - ############################################################ - Scanning directory... - Info: No nether dimension in the world directory. - Info: No end dimension in the world directory. - There are 1 region files and 1 player files in the world directory. - - -------------------- Checking level.dat -------------------- - 'level.dat' is redable - - ------------------ Checking player files ------------------- - All player files are readable. - - ------------------ Scanning the overworld ------------------ - Scanning: 1 / 1 100% [########################################] Time: 00:00:19 - - Found 19 corrupted, 0 wrong located chunks and 0 chunks with too many entities of a total of 625 - - ################ Deleting corrupted chunks ################ - Deleting chunks in region set "/home/alejandro/.minecraft/saves/corrupted-world/region/": Done! Removed 19 chunks - Done! - Deleted 19 corrupted chunks - -If we have a backup of our world we can use them to fix the problems -found chunks, this method can spam a lot of output text, because writes -a log for every chunk that is trying to fix:: - - $ python region-fixer.py --backups ~/backup/2013.01.05/ --replace-corrupted ~/.minecraft/saves/corrupted-world - - Welcome to Region Fixer! - - ############################################################ - ############ Scanning world: Testing corruption ############ - ############################################################ - Scanning directory... - Info: No nether dimension in the world directory. - Info: No end dimension in the world directory. - There are 1 region files and 1 player files in the world directory. - - -------------------- Checking level.dat -------------------- - 'level.dat' is redable - - ------------------ Checking player files ------------------- - All player files are readable. - - ------------------ Scanning the overworld ------------------ - Scanning: 1 / 1 100% [########################################] Time: 00:00:19 +You can read the program help running: "python regionfixer.py --help" - Found 19 corrupted, 0 wrong located chunks and 0 chunks with too many entities of a total of 625 +For usage examples and more info visit the wiki: - ############ Trying to replace corrupted chunks ############ - - ---------- New chunk to replace! Coords (-16, 9) ----------- - Backup region file found in: - ~/backup/2013.01.05/region/r.-1.0.mca - Replacing... - Chunk replaced using backup dir: ~/backup/2013.01.05/ - - ---------- New chunk to replace! Coords (-10, 19) ---------- - Backup region file found in: - ~/backup/2013.01.05/region/r.-1.0.mca - Replacing... - Chunk replaced using backup dir: ~/backup/2013.01.05/ - - ... long log of replaced chunks ... - - ---------- New chunk to replace! Coords (-13, 16) ---------- - Backup region file found in: - ~/backup/2013.01.05/region/r.-1.0.mca - Replacing... - Chunk replaced using backup dir: ~/backup/2013.01.05/ - - ---------- New chunk to replace! Coords (-13, 25) ---------- - Backup region file found in: - ~/backup/2013.01.05/region/r.-1.0.mca - Replacing... - Chunk replaced using backup dir: ~/backup/2013.01.05/ - - 19 replaced chunks of a total of 19 corrupted chunks - -These options have an equivalent for wrong located chunks. - -Another problem that Region Fixer can fix is an entity problem. -Sometimes worlds store thousands of entities in one chunk, hanging the -server when loaded. This can happen with squids, spiders, or even items. -A very common way to make this happen in your server is to ignite a few -thousands of TNTs at the same time. All those TNTs are entities and -the server will hang trying to move them all. - -This problem can be fixed with this method. Using the option -"--delete-entities" Region Fixer will delete all the entities in that -chunk if it does have more entities than entity-limit (see the help). -It doesn't touch TileEntities (chests, singposts, noteblocks, etc...). -At the moment of writing this Entities stored in chunks are: - -- mobs -- projectiles (arrows, snowballs...) -- primed TNT -- ender crystal -- paintings -- items on the ground (don't worry chests are safe) -- vehicles (boats and minecarts) -- dynamic tiles (falling sand and activated TNT) - -Note that you still need to load the chunk in Region Fixer to fix it, -and it may need GIGs of RAM and lot of time. You can use this in -combination with "--entity-limit" to set your limit (default 300 -entities, note that a chunk has 256 square meters of surface and if you -put a mob in every sun lighted block of a chunk that will make 256 -mobs, so it's a big limit!):: - - python region-fixer.py --entity-limit 50 --delete-entities ~/.minecraft/saves/corrupted-world - - Welcome to Region Fixer! - - ############################################################ - ############ Scanning world: Testing corruption ############ - ############################################################ - Scanning directory... - Info: No nether dimension in the world directory. - Info: No end dimension in the world directory. - There are 1 region files and 1 player files in the world directory. - - -------------------- Checking level.dat -------------------- - 'level.dat' is redable - - ------------------ Checking player files ------------------- - All player files are readable. - - ------------------ Scanning the overworld ------------------ - Deleted 102 entities in chunk (14,8) of the region file: r.-1.0.mca - Deleted 111 entities in chunk (14,10) of the region file: r.-1.0.mca - Deleted 84 entities in chunk (15,4) of the region file: r.-1.0.mca - Deleted 75 entities in chunk (21,4) of the region file: r.-1.0.mca - Scanning: 1 / 1 100% [########################################] Time: 00:00:20 - - Found 0 corrupted, 0 wrong located chunks and 0 chunks with too many entities of a total of 625 - - -From version v0.1.0 there is also an interactive mode for Region-Fixer. -If you don't know what's wrong with your world this mode can be very -useful. To start using the mode use the '--interactive' option:: - - $ python region-fixer.py --interactive ~/.minecraft/saves/corrutped-world - -In this mode the scan results are saved in memory, so one scanned you -can delete chunks, delete entities, replace chunks, replace chunks with -too many entities and read a summary of what's wrong without needing to -scan the world again. Example of usage:: - - $ python region-fixer.py --interactive ~/.minecraft/saves/corrupted-world - Welcome to Region Fixer! - Minecraft Region-Fixer interactive mode. - (Use tab to autocomplete. Type help for a list of commands.) - - #-> scan - Scanning directory... - Info: No nether dimension in the world directory. - Info: No end dimension in the world directory. - There are 1 region files and 1 player files in the world directory. - - -------------------- Checking level.dat -------------------- - 'level.dat' is redable - - ------------------ Checking player files ------------------- - All player files are readable. - - ------------------ Scanning the overworld ------------------ - Scanning: 1 / 1 100% [########################################] Time: 00:00:21 - - #-> summary - - ############################################################ - ############## World name: Testing corruption ############## - ############################################################ - - level.dat: - 'level.dat' is readable - - Player files: - All player files are readable. - - Overworld: - Region file: r.-1.0.mca - |-+-Chunk coords: header (16, 9), global (-16, 9). - | +-Status: Corrupted - - ... big summary... - - |-+-Chunk coords: header (19, 25), global (-13, 25). - | +-Status: Corrupted - | - + - - - #-> remove_chunks corrupted - Deleting chunks in region set "/home/alejandro/.minecraft/saves/corrupted-world/region/": Done! Removed 19 chunks - Done! Removed 19 chunks - #-> - - -For more info: “python region-fixer.py --help” +https://github.com/Fenixin/Minecraft-Region-Fixer/wiki/Usage Bugs, suggestions, feedback, questions @@ -321,13 +54,19 @@ Suggestions and bugs should go to the github page: https://github.com/Fenixin/Minecraft-Region-Fixer -Feedback and questions should go preferably to the forums posts: +Feedback and questions should preferably go to these forums posts: (server administration) -http://www.minecraftforum.net/topic/275730-tool-minecraft-region-fixer/ +https://www.minecraftforum.net/forums/support/server-support-and/1903200-minecraft-region-fixer (mapping and modding) -http://www.minecraftforum.net/topic/302380-tool-minecraft-region-fixer/ +https://www.minecraftforum.net/forums/mapping-and-modding-java-edition/minecraft-tools/1261480-minecraft-region-fixer + + +Donations and sponsors +====================== +Region-Fixer was created thanks to sponsors and donations. You can find +information about that in DONORS.txt Contributors @@ -337,11 +76,11 @@ See CONTRIBUTORS.txt Warning ======= -This program has been tested with a lot of worlds, but there may be +This program has been tested with a lot of worlds, but there may exist bugs, so please, MAKE A BACKUP OF YOUR WORLD BEFORE RUNNING it, I'M NOT RESPONSIBLE OF WHAT HAPPENS TO YOUR WORLD. Other way to say it is USE THIS TOOL AT YOUR OWN RISK. -Think that you are playing with you precious saved games :P . +Think that you are playing with your precious saved games :P . Good luck! :) diff --git a/gui/__init__.py b/gui/__init__.py new file mode 100644 index 0000000..54ac455 --- /dev/null +++ b/gui/__init__.py @@ -0,0 +1,6 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +from .main import MainWindow +from .backups import BackupsWindow +from .starter import Starter diff --git a/gui/about.py b/gui/about.py new file mode 100644 index 0000000..d635510 --- /dev/null +++ b/gui/about.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +import wx + +from regionfixer_core.version import version_string as rf_ver +from gui.version import version_string as gui_ver + + +class AboutWindow(wx.Frame): + def __init__(self, parent, title="About"): + wx.Frame.__init__(self, parent, title=title, + style=wx.CLOSE_BOX | wx.RESIZE_BORDER | wx.CAPTION) + # Every windows should use panel as parent. Not doing so will + # make the windows look non-native (very ugly) + panel = wx.Panel(self) + + self.about1 = wx.StaticText(panel, style=wx.ALIGN_CENTER, + label="Minecraft Region-Fixer (GUI) (ver. {0})\n(using Region-Fixer ver. {1})".format(gui_ver,rf_ver)) + self.about2 = wx.StaticText(panel, style=wx.ALIGN_CENTER, + label="Fix problems in Minecraft worlds.") + self.about3 = wx.StaticText(panel, style=wx.ALIGN_CENTER, + label="Official-web:") + self.link_github = wx.HyperlinkCtrl(panel, wx.ID_ABOUT, + "https://github.com/Fenixin/Minecraft-Region-Fixer", + "https://github.com/Fenixin/Minecraft-Region-Fixer", + style=wx.ALIGN_CENTER) + self.about4 = wx.StaticText(panel, + style=wx.TE_MULTILINE | wx.ALIGN_CENTER, + label="Minecraft forums post:") + self.link_minecraft_forums = wx.HyperlinkCtrl(panel, wx.ID_ABOUT, + "http://www.minecraftforum.net/topic/302380-minecraft-region-fixer/", + "http://www.minecraftforum.net/topic/302380-minecraft-region-fixer/", + style=wx.ALIGN_CENTER) + + self.close_button = wx.Button(panel, wx.ID_CLOSE) + + self.sizer = wx.BoxSizer(wx.VERTICAL) + self.sizer.Add(self.about1, 0, wx.ALIGN_CENTER | wx.TOP, 10) + self.sizer.Add(self.about2, 0, wx.ALIGN_CENTER| wx.TOP, 20) + self.sizer.Add(self.about3, 0, wx.ALIGN_CENTER | wx.TOP, 20) + self.sizer.Add(self.link_github, 0, wx.ALIGN_CENTER | wx.ALL, 5) + self.sizer.Add(self.about4, 0, wx.ALIGN_CENTER | wx.TOP, 20) + self.sizer.Add(self.link_minecraft_forums, 0,wx.ALIGN_CENTER | wx.ALL, 5) + self.sizer.Add(self.close_button, 0, wx.ALIGN_CENTER | wx.ALL, 20) + + # Fit sizers and make the windows not resizable + panel.SetSizerAndFit(self.sizer) + self.sizer.Fit(self) + size = self.GetSize() + self.SetMinSize(size) + self.SetMaxSize(size) + + self.Bind(wx.EVT_BUTTON, self.OnClose, self.close_button) + + def OnClose(self, e): + self.Show(False) diff --git a/gui/backups.py b/gui/backups.py new file mode 100644 index 0000000..968a67b --- /dev/null +++ b/gui/backups.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +import wx +import os + +# TODO: just copied this file to this module, is a cutre solution +# improve it! See Importing python modules from relative paths, or +# order this in a better way +from regionfixer_core.world import World + + +class BackupsWindow(wx.Frame): + def __init__(self, parent, title): + wx.Frame.__init__(self, parent, title=title) + # Every windows should use panel as parent. Not doing so will + # make the windows look non-native (very ugly) + panel = wx.Panel(self) + + # Sizer with all the elements in the window + self.all_sizer = wx.BoxSizer(wx.VERTICAL) + + # Text with help in the top + self.help_text = wx.StaticText(panel, style=wx.TE_MULTILINE, + label=("Region-Fixer will use the worlds in\n" + "this list in top-down order.")) + + # List of worlds to use as backups + self.world_list_box = wx.ListBox(panel, size=(180, 100)) + test_list = [] + self.world_list_box.Set(test_list) + # Here will be the worlds to use as backup + self.world_list = test_list[:] + self.world_list_text = test_list[:] + # Last path we used in the file dialog + self.last_path = "" + + # Buttons + self.buttons_sizer = wx.BoxSizer(wx.HORIZONTAL) + self.add = wx.Button(panel, label="Add") + self.move_up = wx.Button(panel, label="Move up") + self.move_down = wx.Button(panel, label="Move down") + self.buttons_sizer.Add(self.add, 0, 0) + self.buttons_sizer.Add(self.move_up, 0, 0) + self.buttons_sizer.Add(self.move_down, 0, 0) + + # Add things to the general sizer + self.all_sizer.Add(self.help_text, proportion=0, + flag=wx.GROW | wx.ALL, border=10) + self.all_sizer.Add(self.world_list_box, proportion=1, + flag=wx.EXPAND | wx.ALL, border=10) + self.all_sizer.Add(self.buttons_sizer, proportion=0, + flag=wx.ALIGN_CENTER | wx.ALL, border=10) + + # Layout sizers + panel.SetSizerAndFit(self.all_sizer) + + # Bindings + self.Bind(wx.EVT_CLOSE, self.OnClose) + self.Bind(wx.EVT_BUTTON, self.OnAddWorld, self.add) + self.Bind(wx.EVT_BUTTON, self.OnMoveUp, self.move_up) + self.Bind(wx.EVT_BUTTON, self.OnMoveDown, self.move_down) + + # Show the window, usually False, True for fast testing + self.Show(False) + + def get_dirs(self, list_dirs): + """ From a list of paths return only the directories. """ + + tmp = [] + for p in self.dirnames: + if os.path.isdir(p): + tmp.append(p) + return tmp + + def are_there_files(self, list_dirs): + """ Given a list of paths return True if there are any files. """ + + for d in list_dirs: + if not os.path.isdir(d): + return True + return False + + def OnAddWorld(self, e): + """ Called when the buttom Add is clicked. """ + + dlg = wx.DirDialog(self, "Choose a Minecraft world folder") + # Set the last path used + dlg.SetPath(self.last_path) + if dlg.ShowModal() == wx.ID_OK: + self.dirname = dlg.GetPath() + # Check if it's a minecraft world + w = World(self.dirname) + if not w.isworld: + error = wx.MessageDialog(self, "This directory doesn't look like a Minecraft world", "Error", wx.ICON_EXCLAMATION) + error.ShowModal() + error.Destroy() + else: + # Insert it in the ListBox + self.world_list.append(w) + index = self.world_list.index(w) + # TODO check if it's a minecraft world + self.world_list_box.InsertItems([w.name], pos = index) + + # Properly recover the last path used + self.last_path = os.path.split(dlg.GetPath())[0] + dlg.Destroy() + + def get_selected_index(self, list_box): + """ Returns the index of the selected item in a list_box. """ + + index = None + for i in range(len(self.world_list)): + if list_box.IsSelected(i): + index = i + return index + + def move_left_inlist(self, l, index): + """ Move the element in the list with index to the left. + + Return the index where the moved element is. + + """ + + tmp = l.pop(index) + index = index - 1 if index != 0 else 0 + l.insert(index, tmp) + + return index + + def move_right_inlist(self, l, index): + """ Move the element in the list with index to the right. + + Return the index where the moved element is. + + """ + + len_l = len(l) + tmp = l.pop(index) + index = index + 1 + if index == len_l: + l.append(tmp) + index = len_l - 1 + else: + l.insert(index, tmp) + + return index + + def get_names_from_worlds(self, world_list): + """ Return a list of names from a list of worlds in order. """ + + t = [] + for i in world_list: + t.append(i.name) + return t + + def OnMoveUp(self, e): + """ Move up in the world list the selected item. """ + + index = self.get_selected_index(self.world_list_box) + + if index is not None: + index = self.move_left_inlist(self.world_list, index) + #~ self.world_list_box.Set(self.world_list) + self.world_list_box.Set(self.get_names_from_worlds(self.world_list)) + self.world_list_box.Select(index) + + def OnMoveDown(self, e): + """ Move down in the world list the selected item. """ + + index = self.get_selected_index(self.world_list_box) + len_world_list = len(self.world_list) + + if index is not None: + index = self.move_right_inlist(self.world_list, index) + self.world_list_box.Set(self.get_names_from_worlds(self.world_list)) + #~ self.world_list_box.Set(self.world_list) + self.world_list_box.Select(index) + + def OnClose(self, e): + """ Ran when the user closes this window. """ + self.Show(False) diff --git a/gui/help.py b/gui/help.py new file mode 100644 index 0000000..e75db57 --- /dev/null +++ b/gui/help.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +import wx + +class HelpWindow(wx.Frame): + def __init__(self, parent, title="Help"): + wx.Frame.__init__(self, parent, title=title, + style=wx.CLOSE_BOX | wx.RESIZE_BORDER | wx.CAPTION) + # Every windows should use panel as parent. Not doing so will + # make the windows look non-native (very ugly) + panel = wx.Panel(self) + + self.help1 = wx.StaticText(panel, style=wx.ALIGN_CENTER, + label="If you need help you can give a look to the wiki:") + self.link_github = wx.HyperlinkCtrl(panel, wx.ID_ABOUT, + "https://github.com/Fenixin/Minecraft-Region-Fixer/wiki", + style=wx.ALIGN_CENTER, + url="https://github.com/Fenixin/Minecraft-Region-Fixer/wiki") + self.help2 = wx.StaticText(panel, + style=wx.TE_MULTILINE | wx.ALIGN_CENTER, + label="Or ask in the minecraft forums:") + self.link_minecraft_forums = wx.HyperlinkCtrl(panel, wx.ID_ABOUT, + "http://www.minecraftforum.net/topic/302380-minecraft-region-fixer/", + "http://www.minecraftforum.net/topic/302380-minecraft-region-fixer/", + style=wx.ALIGN_CENTER) + + self.close_button = wx.Button(panel, wx.ID_CLOSE) + + self.sizer = wx.BoxSizer(wx.VERTICAL) + self.sizer.Add(self.help1, 0, wx.ALIGN_CENTER | wx.TOP, 10) + self.sizer.Add(self.link_github, 0, wx.ALIGN_CENTER | wx.ALL, 5) + self.sizer.Add(self.help2, 0, wx.ALIGN_CENTER | wx.TOP, 20) + self.sizer.Add(self.link_minecraft_forums, 0, wx.ALIGN_CENTER | wx.ALL, 5) + self.sizer.Add(self.close_button, 0, wx.ALIGN_CENTER | wx.ALL, 20) + + # Fit sizers and make the windows not resizable + panel.SetSizerAndFit(self.sizer) + self.sizer.Fit(self) + size = self.GetSize() + self.SetMinSize(size) + self.SetMaxSize(size) + + self.Bind(wx.EVT_BUTTON, self.OnClose, self.close_button) + + def OnClose(self, e): + self.Show(False) diff --git a/gui/main.py b/gui/main.py new file mode 100644 index 0000000..1a1bcb3 --- /dev/null +++ b/gui/main.py @@ -0,0 +1,383 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +import wx +from time import sleep +from os.path import split, abspath +from os import name as os_name + +from .backups import BackupsWindow +from regionfixer_core.scan import AsyncWorldRegionScanner, AsyncDataScanner,\ + ChildProcessException +from regionfixer_core import world +from regionfixer_core.world import World + +if os_name == 'nt': + # Proper way to set an icon in windows 7 and above + # Thanks to http://stackoverflow.com/a/15923439 + import ctypes + myappid = 'Fenixin.region-fixer.gui.100' # arbitrary string + ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(myappid) + + +class MainWindow(wx.Frame): + def __init__(self, parent, title, backups=None): + wx.Frame.__init__(self, parent, title=title, size=(300, 400)) + # Every windows should use panel as parent. Not doing so will + # make the windows look non-native (very ugly) + panel = wx.Panel(self) + + self.backups = backups + + # Icon + ico = wx.Icon('icon.ico', wx.BITMAP_TYPE_ICO) + self.SetIcon(ico) + + # Open world stuff + self.last_path = "" # Last path opened + self.world = None # World to scan + + # Status bar + self.CreateStatusBar() + + # Create menu + filemenu = wx.Menu() + windowsmenu = wx.Menu() + helpmenu = wx.Menu() + + # Add elements to filemenu + menuOpen = filemenu.Append(wx.ID_OPEN, "&Open", "Open a Minecraft world") + filemenu.AppendSeparator() + menuExit = filemenu.Append(wx.ID_EXIT, "E&xit","Terminate program") + + # Add elements to helpmenu + menuHelp = helpmenu.Append(wx.ID_HELP, "&Help", "Where to find help") + helpmenu.AppendSeparator() + menuAbout = helpmenu.Append(wx.ID_ABOUT, "&About", "Information about this program") + + # Add elements to windowsmenu + menuBackups = windowsmenu.Append(-1, "&Backups", "Manage list of backups") +# menuAdvanced = windowsmenu.Append(-1, "A&dvanced actions", "Manage list of backups") + + # Create a menu bar + menuBar = wx.MenuBar() + menuBar.Append(filemenu,"&File") + menuBar.Append(windowsmenu,"&View") + menuBar.Append(helpmenu,"&Help") + self.SetMenuBar(menuBar) + + # Create elements in the window + # First row: + self.status_text = wx.StaticText(panel, style=wx.TE_MULTILINE, label="No world loaded") + self.open_button = wx.Button(panel, label="Open") + self.scan_button = wx.Button(panel, label="Scan") + self.scan_button.Disable() + self.firstrow_sizer = wx.BoxSizer(wx.HORIZONTAL) + self.firstrow_sizer.Add(self.status_text, 1, wx.ALIGN_CENTER) + self.firstrow_sizer.Add(self.open_button, 0, wx.EXPAND) + self.firstrow_sizer.Add(self.scan_button, 0, wx.EXPAND) + self.firstrow_static_box = wx.StaticBox(panel, label="World loaded") + self.firstrow_static_box_sizer = wx.StaticBoxSizer(self.firstrow_static_box) + self.firstrow_static_box_sizer.Add(self.firstrow_sizer, 1, wx.EXPAND) + + # Second row: + self.proc_info_text = wx.StaticText(panel, label="Processes to use: ") + self.proc_text = wx.TextCtrl(panel, value="1", size=(30, 24), style=wx.TE_CENTER) + self.el_info_text = wx.StaticText(panel, label="Entity limit: " ) + self.el_text = wx.TextCtrl(panel, value="150", size=(50, 24), style=wx.TE_CENTER) + self.secondrow_sizer = wx.BoxSizer(wx.HORIZONTAL) + self.secondrow_sizer.Add(self.proc_info_text, flag=wx.ALIGN_CENTER) + self.secondrow_sizer.Add(self.proc_text, 0, flag=wx.RIGHT | wx.ALIGN_LEFT, border=15) + self.secondrow_sizer.Add(self.el_info_text, 0, wx.ALIGN_CENTER) + self.secondrow_sizer.Add(self.el_text, 0, wx.ALIGN_RIGHT) + self.secondrow_static_box_sizer = wx.StaticBoxSizer(wx.StaticBox(panel, label="Scan options")) + self.secondrow_static_box_sizer.Add(self.secondrow_sizer, 1, flag=wx.EXPAND) + + # Third row: + # Note: In order to use a static box add it directly to a + # static box sizer and add to the same sizer it's contents + self.results_text = wx.TextCtrl(panel, style=wx.TE_READONLY | wx.TE_MULTILINE, value="Scan the world to get results", size = (500,200)) + # Lets try to create a monospaced font: + ffont = wx.Font(9, wx.FONTFAMILY_MODERN, wx.FONTSTYLE_NORMAL, wx.FONTWEIGHT_NORMAL) +# print ffont.IsFixedWidth() + textattr = wx.TextAttr(font = ffont) + self.results_text.SetFont(ffont) + self.results_text_box = wx.StaticBox(panel, label="Results", size = (100,100)) + self.results_text_box_sizer = wx.StaticBoxSizer(self.results_text_box) + self.results_text_box_sizer.Add(self.results_text, 1, wx.EXPAND) + + self.delete_all_chunks_button = wx.Button(panel, label = "Delete all bad chunks") + self.replace_all_chunks_button = wx.Button(panel, label = "Replace all bad chunks (using backups)") + self.delete_all_regions_button = wx.Button(panel, label = "Delete all bad regions") + self.replace_all_regions_button = wx.Button(panel, label = "Replace all bad regions (using backups)") + self.update_delete_buttons_status(False) + self.update_replace_buttons_status(False) + + self.thirdrow_sizer = wx.BoxSizer(wx.HORIZONTAL) + self.thirdrow_actions_box = wx.StaticBox(panel, label="Actions", size = (-1,-1)) + self.thirdrow_buttons_box_sizer = wx.StaticBoxSizer(self.thirdrow_actions_box) + self.thirdrow_buttons_sizer = wx.BoxSizer(wx.VERTICAL) + self.thirdrow_buttons_sizer.Add(self.delete_all_chunks_button, 1, wx.EXPAND) + self.thirdrow_buttons_sizer.Add(self.replace_all_chunks_button, 1, wx.EXPAND) + self.thirdrow_buttons_sizer.Add(self.delete_all_regions_button, 1, wx.EXPAND) + self.thirdrow_buttons_sizer.Add(self.replace_all_regions_button, 1, wx.EXPAND) + self.thirdrow_buttons_box_sizer.Add(self.thirdrow_buttons_sizer, 1, wx.EXPAND) + self.thirdrow_sizer.Add(self.results_text_box_sizer, 1, wx.EXPAND) + self.thirdrow_sizer.Add(self.thirdrow_buttons_box_sizer, 0, wx.EXPAND) + + # All together now + self.frame_sizer = wx.BoxSizer(wx.VERTICAL) + self.frame_sizer.Add(self.firstrow_static_box_sizer, 0, wx.EXPAND) + self.frame_sizer.Add(self.secondrow_static_box_sizer, 0, wx.EXPAND) + self.frame_sizer.Add(self.thirdrow_sizer, 1, wx.EXPAND) + + # Layout sizers + panel.SetSizerAndFit(self.frame_sizer) + + self.frame_sizer.Fit(self) + + # Bindings + self.Bind(wx.EVT_MENU, self.OnAbout, menuAbout) + self.Bind(wx.EVT_MENU, self.OnHelp, menuHelp) + self.Bind(wx.EVT_MENU, self.OnOpen, menuOpen) + self.Bind(wx.EVT_MENU, self.OnBackups, menuBackups) + self.Bind(wx.EVT_MENU, self.OnExit, menuExit) + self.Bind(wx.EVT_BUTTON, self.OnScan, self.scan_button) + self.Bind(wx.EVT_BUTTON, self.OnOpen, self.open_button) + self.Bind(wx.EVT_BUTTON, self.OnDeleteChunks, self.delete_all_chunks_button) + self.Bind(wx.EVT_BUTTON, self.OnReplaceChunks, self.replace_all_chunks_button) + self.Bind(wx.EVT_BUTTON, self.OnDeleteRegions, self.delete_all_regions_button) + self.Bind(wx.EVT_BUTTON, self.OnReplaceRegions, self.replace_all_regions_button) + + self.Show(True) + + def OnExit(self, e): + self.Close(True) + + def OnBackups(self, e): + self.backups.Show(True) + + def OnAbout(self, e): + self.about.Show(True) + + def OnHelp(self, e): + self.help.Show(True) + + def OnOpen(self, e): + """ Called when the open world button is pressed. """ + dlg = wx.DirDialog(self, "Choose a Minecraft world folder") + # Set the last path used + dlg.SetPath(self.last_path) + if dlg.ShowModal() == wx.ID_OK: + self.dirname = dlg.GetPath() + # Check if it's a minecraft world + w = World(self.dirname) + if not w.isworld: + error = wx.MessageDialog(self, "This directory doesn't look like a Minecraft world", "Error", wx.ICON_EXCLAMATION) + error.ShowModal() + error.Destroy() + else: + # Insert it in the ListBox + self.world = w + self.update_world_status(self.world) + + # Properly recover the last path used + self.last_path = split(dlg.GetPath())[0] + dlg.Destroy() + + # Rest the results textctrl + self.results_text.SetValue("") + + + def OnScan(self, e): + """ Called when the scan button is pressed. """ + processes = int(self.proc_text.GetValue()) + entity_limit = int(self.el_text.GetValue()) + delete_entities = False + + ps = AsyncDataScanner(self.world.players, processes) + ops = AsyncDataScanner(self.world.old_players, processes) + ds = AsyncDataScanner(self.world.data_files, processes) + ws = AsyncWorldRegionScanner(self.world, processes, entity_limit, + delete_entities) + + things_to_scan = [ws, ops, ps, ds] + dialog_texts = ["Scanning region files", + "Scanning old format player files", + "Scanning players", + "Scanning data files"] + try: + for scanner, dialog_title in zip(things_to_scan, dialog_texts): + progressdlg = wx.ProgressDialog( + dialog_title, + "Last scanned:\n starting...", + len(scanner), self, + style=wx.PD_ELAPSED_TIME | wx.PD_ESTIMATED_TIME | + wx.PD_REMAINING_TIME | wx.PD_CAN_ABORT | + wx.PD_AUTO_HIDE | wx.PD_SMOOTH) + scanner.scan() + counter = 0 + # NOTE TO SELF: ShowModal behaves different in windows and Linux! + # Use it with care. + progressdlg.Show() + while not scanner.finished: + sleep(0.001) + result = scanner.get_last_result() + + if result: + counter += 1 + not_cancelled, not_skipped = progressdlg.Update(counter, + "Last scanned:\n" + scanner.str_last_scanned) + if not not_cancelled: + # User pressed cancel + scanner.terminate() + break + progressdlg.Destroy() + if not not_cancelled: + break + else: + # The scan finished successfully + self.world.scanned = True + self.results_text.SetValue(self.world.generate_report(True)) + self.update_delete_buttons_status(True) + self.update_replace_buttons_status(True) + except ChildProcessException as e: + # Will be handled in starter.py by _excepthook() + scanner.terminate() + progressdlg.Destroy() + raise e + #=================================================================== + # error_log_path = e.save_error_log() + # filename = e.scanned_file.filename + # scanner.terminate() + # progressdlg.Destroy() + # error = wx.MessageDialog(self, + # ("Something went really wrong scanning {0}\n\n" + # "This is probably an error in the code. Please, " + # "if you have the time report it. " + # "I have saved all the error information in:\n\n" + # "{1}").format(filename, error_log_path), + # "Error", + # wx.ICON_ERROR) + # error.ShowModal() + #=================================================================== + + def OnDeleteChunks(self, e): + progressdlg = wx.ProgressDialog("Removing chunks", "This may take a while", + self.world.count_regions(), self, + style=wx.PD_ELAPSED_TIME | + wx.PD_ESTIMATED_TIME | + wx.PD_REMAINING_TIME | + wx.PD_CAN_SKIP | + wx.PD_CAN_ABORT | + wx.PD_AUTO_HIDE | + wx.PD_SMOOTH + ) + progressdlg = progressdlg + progressdlg.Pulse() + remove_chunks = self.world.remove_problematic_chunks + for problem in world.CHUNK_PROBLEMS: + progressdlg.Pulse("Removing chunks with problem: {}".format(world.CHUNK_STATUS_TEXT[problem])) + remove_chunks(problem) + progressdlg.Destroy() + progressdlg.Destroy() + + self.results_text.SetValue("Scan again the world for results.") + self.update_delete_buttons_status(False) + self.update_delete_buttons_status(False) + + def OnDeleteRegions(self, e): + progressdlg = wx.ProgressDialog("Removing regions", "This may take a while...", + self.world.count_regions(), self, + style=wx.PD_ELAPSED_TIME | + wx.PD_ESTIMATED_TIME | + wx.PD_REMAINING_TIME | + wx.PD_AUTO_HIDE | + wx.PD_SMOOTH + ) + progressdlg = progressdlg + progressdlg.Pulse() + remove_regions = self.world.remove_problematic_regions + for problem in world.REGION_PROBLEMS: + progressdlg.Pulse("Removing regions with problem: {}".format(world.REGION_STATUS_TEXT[problem])) + remove_regions(problem) + progressdlg.Destroy() + + self.results_text.SetValue("Scan again the world for results.") + self.update_delete_buttons_status(False) + self.update_replace_buttons_status(False) + + def OnReplaceChunks(self, e): + # Get options + entity_limit = int(self.el_text.GetValue()) + delete_entities = False + + progressdlg = wx.ProgressDialog("Removing chunks", "Removing...", + self.world.count_regions(), self, + style=wx.PD_ELAPSED_TIME | + wx.PD_ESTIMATED_TIME | + wx.PD_REMAINING_TIME | + wx.PD_AUTO_HIDE | + wx.PD_SMOOTH + ) + progressdlg = progressdlg + backups = self.backups.world_list + progressdlg.Pulse() + replace_chunks = self.world.replace_problematic_chunks + for problem in world.CHUNK_PROBLEMS: + progressdlg.Pulse("Replacing chunks with problem: {}".format(world.CHUNK_STATUS_TEXT[problem])) + replace_chunks(backups, problem, entity_limit, delete_entities) + progressdlg.Destroy() + + self.results_text.SetValue("Scan again the world for results.") + self.update_delete_buttons_status(False) + self.update_replace_buttons_status(False) + + def OnReplaceRegions(self, e): + # Get options + entity_limit = int(self.el_text.GetValue()) + delete_entities = False + progressdlg = wx.ProgressDialog("Removing regions", "Removing...", + self.world.count_regions(), self, + style = wx.PD_ELAPSED_TIME | + wx.PD_ESTIMATED_TIME | + wx.PD_REMAINING_TIME | + #~ wx.PD_CAN_SKIP | + #~ wx.PD_CAN_ABORT | + wx.PD_AUTO_HIDE | + wx.PD_SMOOTH + ) + progressdlg = progressdlg + backups = self.backups.world_list + progressdlg.Pulse() + replace_regions = self.world.replace_problematic_regions + for problem in world.REGION_PROBLEMS: + progressdlg.Pulse("Replacing regions with problem: {}".format(world.REGION_STATUS_TEXT[problem])) + replace_regions(backups, problem, entity_limit, delete_entities) + progressdlg.Destroy() + + self.results_text.SetValue("Scan again the world for results.") + self.update_delete_buttons_status(False) + self.update_replace_buttons_status(False) + + def update_delete_buttons_status(self, status): + + if status: + self.delete_all_chunks_button.Enable() + self.delete_all_regions_button.Enable() + else: + self.delete_all_chunks_button.Disable() + self.delete_all_regions_button.Disable() + + def update_replace_buttons_status(self, status): + + if status: + self.replace_all_chunks_button.Enable() + self.replace_all_regions_button.Enable() + else: + self.replace_all_chunks_button.Disable() + self.replace_all_regions_button.Disable() + + def update_world_status(self, world): + self.status_text.SetLabel(world.path) + self.scan_button.Enable() diff --git a/gui/starter.py b/gui/starter.py new file mode 100644 index 0000000..b2d764e --- /dev/null +++ b/gui/starter.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +import wx +import sys +import traceback +from io import StringIO + +from .main import MainWindow +from .backups import BackupsWindow +from .about import AboutWindow +from .help import HelpWindow + +from regionfixer_core.scan import ChildProcessException +from regionfixer_core.bug_reporter import BugReporter +from regionfixer_core.util import get_str_from_traceback + +ERROR_MSG = "\n\nOps! Something went really wrong and regionfixer crashed.\n\nI can try to send an automatic bug rerpot if you wish.\n" +QUESTION_TEXT = ('Do you want to send an anonymous bug report to the region fixer ftp?\n' + '(Answering no will print the bug report)') + +# Thanks to: +# http://wxpython-users.1045709.n5.nabble.com/Exception-handling-strategies-td2369185.html +# For a way to handle exceptions +class MyApp(wx.App): + def OnInit(self): + sys.excepthook = self._excepthook + return True + + def _excepthook(self, etype, value, tb): + if isinstance(etype, ChildProcessException): + s = "Using GUI:\n\n" + value.printable_traceback + else: + s = "Using GUI:\n\n" + get_str_from_traceback(etype, value, tb) + # bug - display a dialog with the entire exception and traceback printed out + traceback.print_tb(tb) + dlg = wx.MessageDialog(self.main_window, + ERROR_MSG + "\n" + QUESTION_TEXT, + style=wx.ICON_ERROR | wx.YES_NO) + # Get a string with the traceback and send it + + answer = dlg.ShowModal() + if answer == wx.ID_YES: + print("Sending bug report!") + bugsender = BugReporter(error_str=s) + success = bugsender.send() + # Dialog with success or not of the ftp uploading + if success: + msg = "The bug report was successfully uploaded." + style = 0 + else: + msg = "Couldn't upload the bug report!\n\nPlease, try again later." + style = wx.ICON_ERROR + dlg = wx.MessageDialog(self.main_window, msg, style=style) + dlg.ShowModal() + else: + dlg = wx.MessageDialog(self.main_window, "Error msg:\n\n" + s, + style=wx.ICON_ERROR) + dlg.ShowModal() + + +class Starter(object): + def __init__(self): + """ Create the windows and set some variables. """ + + self.app = MyApp(False) + + self.frame = MainWindow(None, "Region-Fixer-GUI") + # NOTE: It's very important that the MainWindow is parent of all others windows + self.backups = BackupsWindow(self.frame, "Backups") + self.about = AboutWindow(self.frame, "About") + self.frame.backups = self.backups + self.frame.about = self.about + self.frame.help = HelpWindow(self.frame, "Help") +# self.frame.error = ErrorWindow(self.frame, "Error") + + self.app.main_window = self.frame + + def run(self): + """ Run the app main loop. """ + + self.app.MainLoop() diff --git a/gui/version.py b/gui/version.py new file mode 100644 index 0000000..9364a6b --- /dev/null +++ b/gui/version.py @@ -0,0 +1,8 @@ +''' +Created on 24/06/2014 + +@author: Alejandro +''' + +version_string = "0.0.1" +version_numbers = version_string.split(".") diff --git a/icon.ico b/icon.ico new file mode 100644 index 0000000..bdbc714 Binary files /dev/null and b/icon.ico differ diff --git a/interactive.py b/interactive.py deleted file mode 100644 index 80f88d8..0000000 --- a/interactive.py +++ /dev/null @@ -1,501 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -# -# Region Fixer. -# Fix your region files with a backup copy of your Minecraft world. -# Copyright (C) 2011 Alejandro Aguilera (Fenixin) -# https://github.com/Fenixin/Minecraft-Region-Fixer -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . -# - - -# TODO needs big update! -import world - -from cmd import Cmd -from scan import scan_world, scan_regionset - -class interactive_loop(Cmd): - def __init__(self, world_list, regionset, options, backup_worlds): - Cmd.__init__(self) - self.world_list = world_list - self.regionset = regionset - self.world_names = [str(i.name) for i in self.world_list] - # if there's only one world use it - if len(self.world_list) == 1 and len(self.regionset) == 0: - self.current = world_list[0] - elif len(self.world_list) == 0 and len(self.regionset) > 0: - self.current = self.regionset - else: - self.current = None - self.options = options - self.backup_worlds = backup_worlds - self.prompt = "#-> " - self.intro = "Minecraft Region-Fixer interactive mode.\n(Use tab to autocomplete. Autocomplete doens't work on Windows. Type help for a list of commands.)\n" - - # other region-fixer stuff - - # possible args for chunks stuff - possible_args = "" - first = True - for i in world.CHUNK_PROBLEMS_ARGS.values() + ['all']: - if not first: - possible_args += ", " - possible_args += i - first = False - self.possible_chunk_args_text = possible_args - - # possible args for region stuff - possible_args = "" - first = True - for i in world.REGION_PROBLEMS_ARGS.values() + ['all']: - if not first: - possible_args += ", " - possible_args += i - first = False - self.possible_region_args_text = possible_args - - - # do - def do_set(self,arg): - """ Command to change some options and variables in interactive - mode """ - args = arg.split() - if len(args) > 2: - print "Error: too many parameters." - elif len(args) == 0: - print "Write \'help set\' to see a list of all possible variables" - else: - if args[0] == "entity-limit": - if len(args) == 1: - print "entity-limit = {0}".format(self.options.entity_limit) - else: - try: - if int(args[1]) >= 0: - self.options.entity_limit = int(args[1]) - print "entity-limit = {0}".format(args[1]) - print "Updating chunk status..." - self.current.rescan_entities(self.options) - else: - print "Invalid value. Valid values are positive integers and zero" - except ValueError: - print "Invalid value. Valid values are positive integers and zero" - - elif args[0] == "workload": - - if len(args) == 1: - if self.current: - print "Current workload:\n{0}\n".format(self.current.__str__()) - print "List of possible worlds and region-sets (determined by the command used to run region-fixer):" - number = 1 - for w in self.world_list: - print " ### world{0} ###".format(number) - number += 1 - # add a tab and print - for i in w.__str__().split("\n"): print "\t" + i - print - print " ### regionset ###" - for i in self.regionset.__str__().split("\n"): print "\t" + i - print "\n(Use \"set workload world1\" or name_of_the_world or regionset to choose one)" - - else: - a = args[1] - if len(a) == 6 and a[:5] == "world" and int(a[-1]) >= 1 : - # get the number and choos the correct world from the list - number = int(args[1][-1]) - 1 - try: - self.current = self.world_list[number] - print "workload = {0}".format(self.current.world_path) - except IndexError: - print "This world is not in the list!" - elif a in self.world_names: - for w in self.world_list: - if w.name == args[1]: - self.current = w - print "workload = {0}".format(self.current.world_path) - break - else: - print "This world name is not on the list!" - elif args[1] == "regionset": - if len(self.regionset): - self.current = self.regionset - print "workload = set of region files" - else: - print "The region set is empty!" - else: - print "Invalid world number, world name or regionset." - - elif args[0] == "processes": - if len(args) == 1: - print "processes = {0}".format(self.options.processes) - else: - try: - if int(args[1]) > 0: - self.options.processes = int(args[1]) - print "processes = {0}".format(args[1]) - else: - print "Invalid value. Valid values are positive integers." - except ValueError: - print "Invalid value. Valid values are positive integers." - - elif args[0] == "verbose": - if len(args) == 1: - print "verbose = {0}".format(str(self.options.verbose)) - else: - if args[1] == "True": - self.options.verbose = True - print "verbose = {0}".format(args[1]) - elif args[1] == "False": - self.options.verbose = False - print "verbose = {0}".format(args[1]) - else: - print "Invalid value. Valid values are True and False." - else: - print "Invalid argument! Write \'help set\' to see a list of valid variables." - - def do_summary(self, arg): - """ Prints a summary of all the problems found in the region - files. """ - if len(arg) == 0: - if self.current: - if self.current.scanned: - text = self.current.summary() - if text: print text - else: print "No problems found!" - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - else: - print "No world/region-set is set! Use \'set workload\' to set a world/regionset to work with." - else: - print "This command doesn't use any arguments." - - def do_current_workload(self, arg): - """ Prints the info of the current workload """ - if len(arg) == 0: - if self.current: print self.current - else: print "No world/region-set is set! Use \'set workload\' to set a world/regionset to work with." - else: - print "This command doesn't use any arguments." - - def do_scan(self, arg): - # TODO: what about scanning while deleting entities as done in non-interactive mode? - # this would need an option to choose which of the two methods use - """ Scans the current workload. """ - if len(arg.split()) > 0: - print "Error: too many parameters." - else: - if self.current: - if isinstance(self.current, world.World): - self.current = world.World(self.current.path) - scan_world(self.current, self.options) - elif isinstance(self.current, world.RegionSet): - print "\n{0:-^60}".format(' Scanning region files ') - scan_regionset(self.current, self.options) - else: - print "No world set! Use \'set workload\'" - - def do_count_chunks(self, arg): - """ Counts the number of chunks with the given problem and - prints the result """ - if self.current and self.current.scanned: - if len(arg.split()) == 0: - print "Possible counters are: {0}".format(self.possible_chunk_args_text) - elif len(arg.split()) > 1: - print "Error: too many parameters." - else: - if arg in world.CHUNK_PROBLEMS_ARGS.values() or arg == 'all': - total = self.current.count_chunks(None) - for problem, status_text, a in world.CHUNK_PROBLEMS_ITERATOR: - if arg == 'all' or arg == a: - n = self.current.count_chunks(problem) - print "Chunks with status \'{0}\': {1}".format(status_text, n) - print "Total chunks: {0}".format(total) - else: - print "Unknown counter." - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - - def do_count_regions(self, arg): - """ Counts the number of regions with the given problem and - prints the result """ - if self.current and self.current.scanned: - if len(arg.split()) == 0: - print "Possible counters are: {0}".format(self.possible_region_args_text) - elif len(arg.split()) > 1: - print "Error: too many parameters." - else: - if arg in world.REGION_PROBLEMS_ARGS.values() or arg == 'all': - total = self.current.count_regions(None) - for problem, status_text, a in world.REGION_PROBLEMS_ITERATOR: - if arg == 'all' or arg == a: - n = self.current.count_regions(problem) - print "Regions with status \'{0}\': {1}".format(status_text, n) - print "Total regions: {0}".format(total) - else: - print "Unknown counter." - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - - def do_count_all(self, arg): - """ Print all the counters for chunks and regions. """ - if self.current and self.current.scanned: - if len(arg.split()) > 0: - print "This command doesn't requiere any arguments" - else: - print "{0:#^60}".format("Chunk problems:") - self.do_count_chunks('all') - print "\n" - print "{0:#^60}".format("Region problems:") - self.do_count_regions('all') - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - - def do_remove_entities(self, arg): - if self.current and self.current.scanned: - if len(arg.split()) > 0: - print "Error: too many parameters." - else: - print "WARNING: This will delete all the entities in the chunks that have more entities than entity-limit, make sure you know what entities are!.\nAre you sure you want to continue? (yes/no):" - answer = raw_input() - if answer == 'yes': - counter = self.current.remove_entities() - print "Deleted {0} entities.".format(counter) - if counter: - self.current.scanned = False - self.current.rescan_entities(self.options) - elif answer == 'no': - print "Ok!" - else: print "Invalid answer, use \'yes\' or \'no\' the next time!." - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - - def do_remove_chunks(self, arg): - if self.current and self.current.scanned: - if len(arg.split()) == 0: - print "Possible arguments are: {0}".format(self.possible_chunk_args_text) - elif len(arg.split()) > 1: - print "Error: too many parameters." - else: - if arg in world.CHUNK_PROBLEMS_ARGS.values() or arg == 'all': - for problem, status_text, a in world.CHUNK_PROBLEMS_ITERATOR: - if arg == 'all' or arg == a: - n = self.current.remove_problematic_chunks(problem) - if n: - self.current.scanned = False - print "Removed {0} chunks with status \'{1}\'.\n".format(n, status_text) - else: - print "Unknown argument." - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - - def do_replace_chunks(self, arg): - if self.current and self.current.scanned: - if len(arg.split()) == 0: - print "Possible arguments are: {0}".format(self.possible_chunk_args_text) - elif len(arg.split()) > 1: - print "Error: too many parameters." - else: - if arg in world.CHUNK_PROBLEMS_ARGS.values() or arg == 'all': - for problem, status_text, a in world.CHUNK_PROBLEMS_ITERATOR: - if arg == 'all' or arg == a: - n = self.current.replace_problematic_chunks(self.backup_worlds, problem, self.options) - if n: - self.current.scanned = False - print "\nReplaced {0} chunks with status \'{1}\'.".format(n, status_text) - else: - print "Unknown argument." - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - - def do_replace_regions(self, arg): - if self.current and self.current.scanned: - if len(arg.split()) == 0: - print "Possible arguments are: {0}".format(self.possible_region_args_text) - elif len(arg.split()) > 1: - print "Error: too many parameters." - else: - if arg in world.REGION_PROBLEMS_ARGS.values() or arg == 'all': - for problem, status_text, a in world.REGION_PROBLEMS_ITERATOR: - if arg == 'all' or arg == a: - n = self.current.replace_problematic_regions(self.backup_worlds, problem, self.options) - if n: - self.current.scanned = False - print "\nReplaced {0} regions with status \'{1}\'.".format(n, status_text) - else: - print "Unknown argument." - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - - def do_remove_regions(self, arg): - if self.current and self.current.scanned: - if len(arg.split()) == 0: - print "Possible arguments are: {0}".format(self.possible_region_args_text) - elif len(arg.split()) > 1: - print "Error: too many parameters." - else: - if arg in world.REGION_PROBLEMS_ARGS.values() or arg == 'all': - for problem, status_text, a in world.REGION_PROBLEMS_ITERATOR: - if arg == 'all' or arg == a: - n = self.current.remove_problematic_regions(problem) - if n: - self.current.scanned = False - print "\nRemoved {0} regions with status \'{1}\'.".format(n, status_text) - else: - print "Unknown argument." - else: - print "The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it." - pass - - def do_quit(self, arg): - print "Quitting." - return True - - def do_exit(self, arg): - print "Exiting." - return True - - def do_EOF(self, arg): - print "Quitting." - return True - - # complete - def complete_arg(self, text, possible_args): - l = [] - for arg in possible_args: - if text in arg and arg.find(text) == 0: - l.append(arg + " ") - return l - - def complete_set(self, text, line, begidx, endidx): - if "workload " in line: - # return the list of world names plus 'regionset' plus a list of world1, world2... - possible_args = tuple(self.world_names) + ('regionset',) + tuple([ 'world' + str(i+1) for i in range(len(self.world_names))]) - elif 'verbose ' in line: - possible_args = ('True','False') - else: - possible_args = ('entity-limit','verbose','processes','workload') - return self.complete_arg(text, possible_args) - - def complete_count_chunks(self, text, line, begidx, endidx): - possible_args = world.CHUNK_PROBLEMS_ARGS.values() + ['all'] - return self.complete_arg(text, possible_args) - - def complete_remove_chunks(self, text, line, begidx, endidx): - possible_args = world.CHUNK_PROBLEMS_ARGS.values() + ['all'] - return self.complete_arg(text, possible_args) - - def complete_replace_chunks(self, text, line, begidx, endidx): - possible_args = world.CHUNK_PROBLEMS_ARGS.values() + ['all'] - return self.complete_arg(text, possible_args) - - def complete_count_regions(self, text, line, begidx, endidx): - possible_args = world.REGION_PROBLEMS_ARGS.values() + ['all'] - return self.complete_arg(text, possible_args) - - def complete_remove_regions(self, text, line, begidx, endidx): - possible_args = world.REGION_PROBLEMS_ARGS.values() + ['all'] - return self.complete_arg(text, possible_args) - - def complete_replace_regions(self, text, line, begidx, endidx): - possible_args = world.REGION_PROBLEMS_ARGS.values() + ['all'] - return self.complete_arg(text, possible_args) - - # help - # TODO sería una buena idea poner un artículo de ayuda de como usar el programa en un caso típico. - # TODO: the help texts need a normalize - def help_set(self): - print "\nSets some variables used for the scan in interactive mode. If you run this command without an argument for a variable you can see the current state of the variable. You can set:" - print " verbose" - print "If True prints a line per scanned region file instead of showing a progress bar." - print "\n entity-limit" - print "If a chunk has more than this number of entities it will be added to the list of chunks with too many entities problem." - print "\n processes" - print "Number of cores used while scanning the world." - print "\n workload" - print "If you input a few worlds you can choose wich one will be scanned using this command.\n" - def help_current_workload(self): - print "\nPrints information of the current region-set/world. This will be the region-set/world to scan and fix.\n" - def help_scan(self): - print "\nScans the current world set or the region set.\n" - - def help_count_chunks(self): - print "\n Prints out the number of chunks with the given status. For example" - print "\'count corrupted\' prints the number of corrupted chunks in the world." - print - print "Possible status are: {0}\n".format(self.possible_chunk_args_text) - def help_remove_entities(self): - print "\nRemove all the entities in chunks that have more than entity-limit entities." - print - print "This chunks are the ones with status \'too many entities\'.\n" - def help_remove_chunks(self): - print "\nRemoves bad chunks with the given problem." - print - print "Please, be careful, when used with the status too-many-entities this will" - print "REMOVE THE CHUNKS with too many entities problems, not the entities." - print "To remove only the entities see the command remove_entities." - print - print "For example \'remove_chunks corrupted\' this will remove corrupted chunks." - print - print "Possible status are: {0}\n".format(self.possible_chunk_args_text) - print - def help_replace_chunks(self): - print "\nReplaces bad chunks with the given status using the backups directories." - print - print "Exampe: \"replace_chunks corrupted\"" - print - print "this will replace the corrupted chunks with the given backups." - print - print "Possible status are: {0}\n".format(self.possible_chunk_args_text) - print - print "Note: after replacing any chunks you have to rescan the world.\n" - - def help_count_regions(self): - print "\n Prints out the number of regions with the given status. For example " - print "\'count_regions too-small\' prints the number of region with \'too-small\' status." - print - print "Possible status are: {0}\n".format(self.possible_region_args_text) - def help_remove_regions(self): - print "\nRemoves regions with the given status." - print - print "Example: \'remove_regions too-small\'" - print - print "this will remove the region files with status \'too-small\'." - print - print "Possible status are: {0}".format(self.possible_region_args_text) - print - print "Note: after removing any regions you have to rescan the world.\n" - def help_replace_regions(self): - print "\nReplaces regions with the given status." - print - print "Example: \"replace_regions too-small\"" - print - print "this will try to replace the region files with status \'too-small\'" - print "with the given backups." - print - print "Possible status are: {0}".format(self.possible_region_args_text) - print - print "Note: after replacing any regions you have to rescan the world.\n" - - def help_summary(self): - print "\nPrints a summary of all the problems found in the current workload.\n" - def help_quit(self): - print "\nQuits interactive mode, exits region-fixer. Same as \'EOF\' and \'exit\' commands.\n" - def help_EOF(self): - print "\nQuits interactive mode, exits region-fixer. Same as \'quit\' and \'exit\' commands\n" - def help_exit(self): - print "\nQuits interactive mode, exits region-fixer. Same as \'quit\' and \'EOF\' commands\n" - def help_help(self): - print "Prints help help." diff --git a/mutf8/LICENSE b/mutf8/LICENSE new file mode 100644 index 0000000..49e36a9 --- /dev/null +++ b/mutf8/LICENSE @@ -0,0 +1,19 @@ +Copyright (c) 2012-2015 Tyler Kennedy . All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/mutf8/README.md b/mutf8/README.md new file mode 100644 index 0000000..cf6a2ca --- /dev/null +++ b/mutf8/README.md @@ -0,0 +1,82 @@ +![Tests](https://github.com/TkTech/mutf8/workflows/Tests/badge.svg?branch=master) + +# mutf-8 + +This package contains simple pure-python as well as C encoders and decoders for +the MUTF-8 character encoding. In most cases, you can also parse the even-rarer +CESU-8. + +These days, you'll most likely encounter MUTF-8 when working on files or +protocols related to the JVM. Strings in a Java `.class` file are encoded using +MUTF-8, strings passed by the JNI, as well as strings exported by the object +serializer. + +This library was extracted from [Lawu][], a Python library for working with JVM +class files. + +## 🎉 Installation + +Install the package from PyPi: + +``` +pip install mutf8 +``` + +Binary wheels are available for the following: + +| | py3.6 | py3.7 | py3.8 | py3.9 | +| ---------------- | ----- | ----- | ----- | ----- | +| OS X (x86_64) | y | y | y | y | +| Windows (x86_64) | y | y | y | y | +| Linux (x86_64) | y | y | y | y | + +If binary wheels are not available, it will attempt to build the C extension +from source with any C99 compiler. If it could not build, it will fall back +to a pure-python version. + +## Usage + +Encoding and decoding is simple: + +```python +from mutf8 import encode_modified_utf8, decode_modified_utf8 + +unicode = decode_modified_utf8(byte_like_object) +bytes = encode_modified_utf8(unicode) +``` + +This module *does not* register itself globally as a codec, since importing +should be side-effect-free. + +## 📈 Benchmarks + +The C extension is significantly faster - often 20x to 40x faster. + + + +### MUTF-8 Decoding +| Name | Min (μs) | Max (μs) | StdDev | Ops | +|------------------------------|------------|------------|----------|---------------| +| cmutf8-decode_modified_utf8 | 0.00009 | 0.00080 | 0.00000 | 9957678.56358 | +| pymutf8-decode_modified_utf8 | 0.00190 | 0.06040 | 0.00000 | 450455.96019 | + +### MUTF-8 Encoding +| Name | Min (μs) | Max (μs) | StdDev | Ops | +|------------------------------|------------|------------|----------|----------------| +| cmutf8-encode_modified_utf8 | 0.00008 | 0.00151 | 0.00000 | 11897361.05101 | +| pymutf8-encode_modified_utf8 | 0.00180 | 0.16650 | 0.00000 | 474390.98091 | + + +## C Extension + +The C extension is optional. If a binary package is not available, or a C +compiler is not present, the pure-python version will be used instead. If you +want to ensure you're using the C version, import it directly: + +```python +from mutf8.cmutf8 import decode_modified_utf8 + +decode_modified_utf(b'\xED\xA1\x80\xED\xB0\x80') +``` + +[Lawu]: https://github.com/tktech/lawu diff --git a/mutf8/__init__.py b/mutf8/__init__.py new file mode 100644 index 0000000..943dc4d --- /dev/null +++ b/mutf8/__init__.py @@ -0,0 +1,21 @@ +""" +Utility methods for handling oddities in character encoding encountered +when parsing and writing JVM ClassFiles or object serialization archives. + +MUTF-8 is the same as CESU-8, but with different encoding for 0x00 bytes. + +.. note:: + + http://bugs.python.org/issue2857 was an attempt in 2008 to get support + for MUTF-8/CESU-8 into the python core. +""" + + +try: + from mutf8.cmutf8 import decode_modified_utf8, encode_modified_utf8 +except ImportError: + from mutf8.mutf8 import decode_modified_utf8, encode_modified_utf8 + + +# Shut up linters. +ALL_IMPORTS = [decode_modified_utf8, encode_modified_utf8] diff --git a/mutf8/cmutf8.c b/mutf8/cmutf8.c new file mode 100644 index 0000000..e05ddf3 --- /dev/null +++ b/mutf8/cmutf8.c @@ -0,0 +1,256 @@ +#define PY_SSIZE_T_CLEAN +#include +#include + +PyDoc_STRVAR(decode_doc, + "Decodes a bytestring containing MUTF-8 as defined in section\n" + "4.4.7 of the JVM specification.\n\n" + ":param s: A byte/buffer-like to be converted.\n" + ":returns: A unicode representation of the original string."); +static PyObject * +decode_modified_utf8(PyObject *self, PyObject *args) +{ +#define return_err(_msg) \ + do { \ + PyObject *exc = PyObject_CallFunction(PyExc_UnicodeDecodeError, \ + "sy#nns", "mutf-8", view.buf, \ + view.len, ix, ix + 1, _msg); \ + if (exc != NULL) { \ + PyCodec_StrictErrors(exc); \ + Py_DECREF(exc); \ + } \ + PyMem_Free(cp_out); \ + PyBuffer_Release(&view); \ + return NULL; \ + } while (0) + + Py_buffer view; + + if (!PyArg_ParseTuple(args, "y*", &view)) { + return NULL; + } + + // MUTF-8 input. + uint8_t *buf = (uint8_t *)view.buf; + // Array of temporary UCS-4 codepoints. + // There's no point using PyUnicode_new and _WriteChar, because + // it requires us to have iterated the string to get the maximum unicode + // codepoint and count anyways. + Py_UCS4 *cp_out = PyMem_Calloc(view.len, sizeof(Py_UCS4)); + if (!cp_out) { + return PyErr_NoMemory(); + } + + // # of codepoints we found & current index into cp_out. + Py_ssize_t cp_count = 0; + + for (Py_ssize_t ix = 0; ix < view.len; ix++) { + Py_UCS4 x = buf[ix]; + + if (x == 0) { + return_err("Embedded NULL byte in input."); + } + else if (x < 0x80) { + // ASCII/one-byte codepoint. + x &= 0x7F; + } + else if ((x & 0xE0) == 0xC0) { + // Two-byte codepoint. + if (ix + 1 >= view.len) { + return_err( + "2-byte codepoint started, but input too short" + " to finish."); + } + x = ((x & 0x1F) << 0x06 | (buf[ix + 1] & 0x3F)); + ix++; + } + else if ((x & 0xF0) == 0xE0) { + // Three-byte codepoint. + if (ix + 2 >= view.len) { + return_err( + "3-byte or 6-byte codepoint started, but input too short" + " to finish."); + } + uint8_t b2 = buf[ix + 1]; + uint8_t b3 = buf[ix + 2]; + + if (x == 0xED && (b2 & 0xF0) == 0xA0) { + if (ix + 5 >= view.len) { + return_err( + "6-byte codepoint started, but input too short" + " to finish."); + } + + // Possible six-byte codepoint. + uint8_t b4 = buf[ix + 3]; + uint8_t b5 = buf[ix + 4]; + uint8_t b6 = buf[ix + 5]; + + if (b4 == 0xED && (b5 & 0xF0) == 0xB0) { + // Definite six-byte codepoint. + x = ( + 0x10000 | + (b2 & 0x0F) << 0x10 | + (b3 & 0x3F) << 0x0A | + (b5 & 0x0F) << 0x06 | + (b6 & 0x3F) + ); + ix += 5; + cp_out[cp_count++] = x; + continue; + } + } + + x = ( + (x & 0x0F) << 0x0C | + (b2 & 0x3F) << 0x06 | + (b3 & 0x3F) + ); + + ix += 2; + } + cp_out[cp_count++] = x; + } + + PyObject *out = + PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, cp_out, cp_count); + + PyMem_Free(cp_out); + PyBuffer_Release(&view); + return out; +#undef return_err +} + +inline Py_ssize_t _encoded_size(void *data, Py_ssize_t length, int kind) { + Py_ssize_t byte_count = 0; + + for (Py_ssize_t i = 0; i < length; i++) { + Py_UCS4 cp = PyUnicode_READ(kind, data, i); + if (cp == 0x00) { + // NULLs will get encoded as C0 80. + byte_count += 2; + } else if (cp <= 0x7F) { + byte_count++; + } else if (cp <= 0x7FF) { + byte_count += 2; + } else if (cp <= 0xFFFF) { + byte_count += 3; + } else { + byte_count += 6; + } + } + + return byte_count; +} + +PyDoc_STRVAR(encoded_size_doc, + "Returns the number of bytes required to store the given\n" + "unicode string when encoded as MUTF-8.\n\n" + ":param u: Unicode string to be converted.\n" + ":returns: The number of bytes required."); +static PyObject * +encoded_size(PyObject *self, PyObject *args) +{ + PyObject *src = NULL; + + if (!PyArg_ParseTuple(args, "U", &src)) { + return NULL; + } + + return PyLong_FromSsize_t( + _encoded_size( + PyUnicode_DATA(src), + PyUnicode_GET_LENGTH(src), + PyUnicode_KIND(src) + ) + ); +} + +PyDoc_STRVAR(encode_doc, + "Encodes a unicode string as MUTF-8 as defined in section\n" + "4.4.7 of the JVM specification.\n\n" + ":param u: Unicode string to be converted.\n" + ":returns: The encoded string as a `bytes` object."); +static PyObject * +encode_modified_utf8(PyObject *self, PyObject *args) +{ + PyObject *src = NULL; + + if (!PyArg_ParseTuple(args, "U", &src)) { + return NULL; + } + + void *data = PyUnicode_DATA(src); + Py_ssize_t length = PyUnicode_GET_LENGTH(src); + int kind = PyUnicode_KIND(src); + char *byte_out = PyMem_Calloc(_encoded_size(data, length, kind), 1); + + if (!byte_out) { + return PyErr_NoMemory(); + } + + Py_ssize_t byte_count = 0; + + for (Py_ssize_t i = 0; i < length; i++) { + Py_UCS4 cp = PyUnicode_READ(kind, data, i); + if (cp == 0x00) { + // NULL byte encoding shortcircuit. + byte_out[byte_count++] = 0xC0; + byte_out[byte_count++] = 0x80; + } + else if (cp <= 0x7F) { + // ASCII + byte_out[byte_count++] = cp; + } + else if (cp <= 0x7FF) { + // Two-byte codepoint. + byte_out[byte_count++] = (0xC0 | (0x1F & (cp >> 0x06))); + byte_out[byte_count++] = (0x80 | (0x3F & cp)); + } + else if (cp <= 0xFFFF) { + // Three-byte codepoint + byte_out[byte_count++] = (0xE0 | (0x0F & (cp >> 0x0C))); + byte_out[byte_count++] = (0x80 | (0x3F & (cp >> 0x06))); + byte_out[byte_count++] = (0x80 | (0x3F & cp)); + } + else { + // "Two-times-three" byte codepoint. + byte_out[byte_count++] = 0xED; + byte_out[byte_count++] = 0xA0 | ((cp >> 0x10) & 0x0F); + byte_out[byte_count++] = 0x80 | ((cp >> 0x0A) & 0x3F); + byte_out[byte_count++] = 0xED; + byte_out[byte_count++] = 0xB0 | ((cp >> 0x06) & 0x0F); + byte_out[byte_count++] = 0x80 | (cp & 0x3F); + } + } + + PyObject *out = PyBytes_FromStringAndSize(byte_out, byte_count); + PyMem_Free(byte_out); + return out; +} + +static PyMethodDef module_methods[] = { + {"decode_modified_utf8", decode_modified_utf8, METH_VARARGS, decode_doc}, + {"encode_modified_utf8", encode_modified_utf8, METH_VARARGS, encode_doc}, + {"encoded_size", encoded_size, METH_VARARGS, encoded_size_doc}, + {NULL, NULL, 0, NULL}}; + +static struct PyModuleDef cmutf8_module = { + PyModuleDef_HEAD_INIT, + "mutf8.cmutf8", + PyDoc_STR("Encoders and decoders for the MUTF-8 encoding."), + -1, + module_methods, +}; + +PyMODINIT_FUNC +PyInit_cmutf8(void) +{ + PyObject *m; + + m = PyModule_Create(&cmutf8_module); + if (m == NULL) + return NULL; + + return m; +} diff --git a/mutf8/mutf8.py b/mutf8/mutf8.py new file mode 100644 index 0000000..ceec8f5 --- /dev/null +++ b/mutf8/mutf8.py @@ -0,0 +1,147 @@ +def decode_modified_utf8(s: bytes) -> str: + """ + Decodes a bytestring containing modified UTF-8 as defined in section + 4.4.7 of the JVM specification. + + :param s: bytestring to be converted. + :returns: A unicode representation of the original string. + """ + s_out = [] + s_len = len(s) + s_ix = 0 + + while s_ix < s_len: + b1 = s[s_ix] + s_ix += 1 + + if b1 == 0: + raise UnicodeDecodeError( + 'mutf-8', + s, + s_ix - 1, + s_ix, + 'Embedded NULL byte in input.' + ) + if b1 < 0x80: + # ASCII/one-byte codepoint. + s_out.append(chr(b1)) + elif (b1 & 0xE0) == 0xC0: + # Two-byte codepoint. + if s_ix >= s_len: + raise UnicodeDecodeError( + 'mutf-8', + s, + s_ix - 1, + s_ix, + '2-byte codepoint started, but input too short to' + ' finish.' + ) + + s_out.append( + chr( + (b1 & 0x1F) << 0x06 | + (s[s_ix] & 0x3F) + ) + ) + s_ix += 1 + elif (b1 & 0xF0) == 0xE0: + # Three-byte codepoint. + if s_ix + 1 >= s_len: + raise UnicodeDecodeError( + 'mutf-8', + s, + s_ix - 1, + s_ix, + '3-byte or 6-byte codepoint started, but input too' + ' short to finish.' + ) + + b2 = s[s_ix] + b3 = s[s_ix + 1] + + if b1 == 0xED and (b2 & 0xF0) == 0xA0: + # Possible six-byte codepoint. + if s_ix + 4 >= s_len: + raise UnicodeDecodeError( + 'mutf-8', + s, + s_ix - 1, + s_ix, + '3-byte or 6-byte codepoint started, but input too' + ' short to finish.' + ) + + b4 = s[s_ix + 2] + b5 = s[s_ix + 3] + b6 = s[s_ix + 4] + + if b4 == 0xED and (b5 & 0xF0) == 0xB0: + # Definite six-byte codepoint. + s_out.append( + chr( + 0x10000 | + (b2 & 0x0F) << 0x10 | + (b3 & 0x3F) << 0x0A | + (b5 & 0x0F) << 0x06 | + (b6 & 0x3F) + ) + ) + s_ix += 5 + continue + + s_out.append( + chr( + (b1 & 0x0F) << 0x0C | + (b2 & 0x3F) << 0x06 | + (b3 & 0x3F) + ) + ) + s_ix += 2 + else: + raise RuntimeError + + return u''.join(s_out) + + +def encode_modified_utf8(u: str) -> bytes: + """ + Encodes a unicode string as modified UTF-8 as defined in section 4.4.7 + of the JVM specification. + + :param u: unicode string to be converted. + :returns: A decoded bytearray. + """ + final_string = bytearray() + + for c in (ord(char) for char in u): + if c == 0x00: + # NULL byte encoding shortcircuit. + final_string.extend([0xC0, 0x80]) + elif c <= 0x7F: + # ASCII + final_string.append(c) + elif c <= 0x7FF: + # Two-byte codepoint. + final_string.extend([ + (0xC0 | (0x1F & (c >> 0x06))), + (0x80 | (0x3F & c)) + ]) + elif c <= 0xFFFF: + # Three-byte codepoint. + final_string.extend([ + (0xE0 | (0x0F & (c >> 0x0C))), + (0x80 | (0x3F & (c >> 0x06))), + (0x80 | (0x3F & c)) + ]) + else: + # Six-byte codepoint. + final_string.extend([ + 0xED, + 0xA0 | ((c >> 0x10) & 0x0F), + 0x80 | ((c >> 0x0A) & 0x3f), + 0xED, + 0xb0 | ((c >> 0x06) & 0x0f), + 0x80 | (c & 0x3f) + ]) + + return bytes(final_string) diff --git a/nbt/CONTRIBUTORS.txt b/nbt/CONTRIBUTORS.txt index 9857b67..d1f6a19 100644 --- a/nbt/CONTRIBUTORS.txt +++ b/nbt/CONTRIBUTORS.txt @@ -1,11 +1,21 @@ d0sboots (David Walker) dtrauma (Thomas Roesner) Fenixin (Alejandro Aguilera) +fwaggle (Jamie Fraser) +jlsajfj (Joseph) +k1988 (Terry Zhao) kamyu2 MacFreek (Freek Dijkstra) +MFLD.fr MidnightLightning (Brooks Boyd) MostAwesomeDude (Corbin Simpson) +psolyca (Damien) +s-leroux (Sylvain Leroux) SBliven (Spencer Bliven) +steffen-kiess (Steffen Kieß) Stumpylog (Trenton Holmes) +suresttexas00 (Surest Texas) tWoolie (Thomas Woolford) -Xgkkp \ No newline at end of file +underscoren (Marius Steffens) +Xgkkp +Zachy (Zachary Howard) diff --git a/nbt/README.md b/nbt/README.md index 694b468..0083f5c 100644 --- a/nbt/README.md +++ b/nbt/README.md @@ -8,6 +8,7 @@ From The spec: read the full spec at http://www.minecraft.net/docs/NBT.txt [![Build Status](https://secure.travis-ci.org/twoolie/NBT.png?branch=master)](http://travis-ci.org/#!/twoolie/NBT) +[![Test Coverage Status](https://coveralls.io/repos/twoolie/NBT/badge.svg)](https://coveralls.io/r/twoolie/NBT) Usage: 1) Reading files. diff --git a/nbt/README.txt b/nbt/README.txt index 0b09590..71c064e 100644 --- a/nbt/README.txt +++ b/nbt/README.txt @@ -1,99 +1,148 @@ -This is a Named Binary Tag parser based upon the specification by Markus Persson. - -From The spec: - "NBT (Named Binary Tag) is a tag based binary format designed to carry large - amounts of binary data with smaller amounts of additional data. - An NBT file consists of a single GZIPped Named Tag of type TAG_Compound." - -read the full spec at http://www.minecraft.net/docs/NBT.txt - -Usage: - 1) Reading files. - - The easiest way to read an nbt file is to instantiate an NBTFile object e.g. - - >>> import nbt - >>> nbtfile = nbt.NBTFile("bigtest.nbt",'rb') - >>> nbtfile.name - u'Level' - >>> nbtfile["nested compound test"].tag_info() - TAG_Compound("nested compound test"): 2 Entries - >>> for tag in nbtfile["nested compound test"]["ham"].tags: - ... print(tag.tag_info()) - ... - TAG_String("name"): Hampus - TAG_Float("value"): 0.75 - >>> [tag.value for tag in nbtfile["listTest (long)"].value] - [11, 12, 13, 14, 15] - - Files can also be read from a fileobj (file-like object that contains a compressed - stream) or a buffer (file-like object that contains an uncompressed stream of NBT - Tags) which can be accomplished thusly: - - >>> import nbt - >>> nbtfile = NBTFile(fileobj=previously_opened_file) - # or.... - >>> nbtfile = NBTFile(buffer=net_socket.makefile()) - - 2) Writing files. - - Writing files is easy too! if you have a NBTFile object, simply call it's - write_file() method. If the NBTFile was instantiated with a filename, then - write_file needs no extra arguments. It just works. If however you created a new - file object from scratch (or even if you just want to save it somewhere else) - call write_file('path\to\new\file.nbt') - - >>> import nbt - >>> nbtfile = nbt.NBTFile("bigtest.nbt",'rb') - >>> nbtfile["listTest (compound)"].tags[0]["name"].value = "Different name" - >>> nbtfile.write_file("newnbtfile.nbt") - - It is also possible to write to a buffer or fileobj using the same keyword args. - - >>> nbtfile.write_file(fileobj = my_file) #compressed - >>> nbtfile.write_file(buffer = sock.makefile()) #uncompressed - - 3) Creating files - - Creating files is trickier but ultimately should give you no issue, as long as - you have read the NBT spec (hint.. it's very short). Also be sure to note that - the NBTFile object is actually a TAG_Compound with some wrapper features, so - you can use all the standard tag features - - >>> from nbt import * - >>> nbtfile = NBTFile() - - first, don't forget to name the top level tag - - >>> nbtfile.name = "My Top Level Tag" - >>> nbtfile.tags.append(TAG_Float(name="My Float Name", value=3.152987593947)) - >>> mylist = TAG_List(name="TestList", type=TAG_Long) #type needs to be pre-declared! - >>> mylist.tags.append(TAG_Long(100)) - >>> mylist.tags.extend([TAG_Long(120),TAG_Long(320),TAG_Long(19)]) - >>> nbtfile.tags.append(mylist) - >>> print(nbtfile.pretty_tree()) - TAG_Compound("My Top Level Tag"): 2 Entries - { - TAG_Float("My Float Name"): 3.15298759395 - TAG_List("TestList"): 4 entries of type TAG_Long - { - TAG_Long: 100 - TAG_Long: 120 - TAG_Long: 320 - TAG_Long: 19 - } - } - >>> nbtfile["TestList"].tags.sort(key = lambda tag: tag.value) - >>> print(nbtfile.pretty_tree()) - TAG_Compound("My Top Level Tag"): 2 Entries - { - TAG_Float("My FloatName"): 3.15298759395 - TAG_List("TestList"): 4 entries of type TAG_Long - { - TAG_Long: 19 - TAG_Long: 100 - TAG_Long: 120 - TAG_Long: 320 - } - } - >>> nbtfile.write_file("mynbt.dat") +========================== +The NBT library for Python +========================== + +Forewords +========= + +This is mainly a `Named Binary Tag` parser & writer library. + +From the initial specification by Markus Persson:: + + NBT (Named Binary Tag) is a tag based binary format designed to carry large + amounts of binary data with smaller amounts of additional data. + An NBT file consists of a single GZIPped Named Tag of type TAG_Compound. + +Current specification is on the official [Minecraft Wiki](https://minecraft.wiki/w/NBT_format). + +This library is very suited to inspect & edit the Minecraft data files. Provided +examples demonstrate how to: +- get player and world statistics, +- list mobs, chest contents, biomes, +- draw a simple world map, +- etc. + +.. image:: world.png + +*Note: Examples are just here to help using and testing the library. +Developing Minecraft tools is out of the scope of this project.* + + +Status +====== + +The library supports all the currently known tag types (including the arrays +of 'Integer' and 'Long'), and the examples work with the McRegion, +pre-"flattened" and "flattened" Anvil formats. + +Last update was tested on Minecraft version **1.13.2**. + + +Dependencies +============ + +The library, the tests and the examples are only using the Python core library, +except `curl` for downloading some test reference data and `PIL` (Python +Imaging Library) for the `map` example. + +Supported Python releases: 2.7, 3.4 to 3.7 + + +Usage +===== + +Reading files +------------- + +The easiest way to read an nbt file is to instantiate an NBTFile object e.g.:: + + >>> from nbt import nbt + >>> nbtfile = nbt.NBTFile("bigtest.nbt",'rb') + >>> nbtfile.name + u'Level' + >>> nbtfile["nested compound test"].tag_info() + TAG_Compound("nested compound test"): 2 Entries + >>> for tag in nbtfile["nested compound test"]["ham"].tags: + ... print(tag.tag_info()) + ... + TAG_String("name"): Hampus + TAG_Float("value"): 0.75 + >>> [tag.value for tag in nbtfile["listTest (long)"].value] + [11, 12, 13, 14, 15] + +Files can also be read from a fileobj (file-like object that contains a compressed +stream) or a buffer (file-like object that contains an uncompressed stream of NBT +Tags) which can be accomplished thusly:: + + >>> from nbt.nbt import * + >>> nbtfile = NBTFile(fileobj=previously_opened_file) + # or.... + >>> nbtfile = NBTFile(buffer=net_socket.makefile()) + + +Writing files +------------- + +Writing files is easy too! if you have a NBTFile object, simply call it's +write_file() method. If the NBTFile was instantiated with a filename, then +write_file needs no extra arguments. It just works. If however you created a new +file object from scratch (or even if you just want to save it somewhere else) +call write_file('path\to\new\file.nbt'):: + + >>> from nbt import nbt + >>> nbtfile = nbt.NBTFile("bigtest.nbt",'rb') + >>> nbtfile["listTest (compound)"].tags[0]["name"].value = "Different name" + >>> nbtfile.write_file("newnbtfile.nbt") + +It is also possible to write to a buffer or fileobj using the same keyword args:: + + >>> nbtfile.write_file(fileobj = my_file) #compressed + >>> nbtfile.write_file(buffer = sock.makefile()) #uncompressed + + +Creating files +-------------- + +Creating files is trickier but ultimately should give you no issue, as long as +you have read the NBT spec (hint.. it's very short). Also be sure to note that +the NBTFile object is actually a TAG_Compound with some wrapper features, so +you can use all the standard tag features:: + + >>> from nbt.nbt import * + >>> nbtfile = NBTFile() + + +First, don't forget to name the top level tag:: + + >>> nbtfile.name = "My Top Level Tag" + >>> nbtfile.tags.append(TAG_Float(name="My Float Name", value=3.152987593947)) + >>> mylist = TAG_List(name="TestList", type=TAG_Long) #type needs to be pre-declared! + >>> mylist.tags.append(TAG_Long(100)) + >>> mylist.tags.extend([TAG_Long(120),TAG_Long(320),TAG_Long(19)]) + >>> nbtfile.tags.append(mylist) + >>> print(nbtfile.pretty_tree()) + TAG_Compound("My Top Level Tag"): 2 Entries + { + TAG_Float("My Float Name"): 3.15298759395 + TAG_List("TestList"): 4 entries of type TAG_Long + { + TAG_Long: 100 + TAG_Long: 120 + TAG_Long: 320 + TAG_Long: 19 + } + } + >>> nbtfile["TestList"].tags.sort(key = lambda tag: tag.value) + >>> print(nbtfile.pretty_tree()) + TAG_Compound("My Top Level Tag"): 2 Entries + { + TAG_Float("My FloatName"): 3.15298759395 + TAG_List("TestList"): 4 entries of type TAG_Long + { + TAG_Long: 19 + TAG_Long: 100 + TAG_Long: 120 + TAG_Long: 320 + } + } + >>> nbtfile.write_file("mynbt.dat") diff --git a/nbt/__init__.py b/nbt/__init__.py index 6fc5768..dd5211d 100644 --- a/nbt/__init__.py +++ b/nbt/__init__.py @@ -4,7 +4,7 @@ # Documentation only automatically includes functions specified in __all__. # If you add more functions, please manually include them in doc/index.rst. -VERSION = (1, 4, 1) +VERSION = (1, 5, 1) """NBT version as tuple. Note that the major and minor revision number are always present, but the patch identifier (the 3rd number) is only used in 1.4.""" diff --git a/nbt/chunk.py b/nbt/chunk.py index 1897d14..e0dd661 100644 --- a/nbt/chunk.py +++ b/nbt/chunk.py @@ -1,17 +1,107 @@ """ Handles a single chunk of data (16x16x128 blocks) from a Minecraft save. -Chunk is currently McRegion only. + +For more information about the chunck format: +https://minecraft.wiki/w/Chunk_format """ + from io import BytesIO -from struct import pack, unpack -import array, math +from struct import pack +from math import ceil +import array + + +# Legacy numeric block identifiers +# mapped to alpha identifiers in best effort +# See https://minecraft.wiki/w/Java_Edition_data_values/Pre-flattening +# TODO: move this map into a separate file + +block_ids = { + 0: 'air', + 1: 'stone', + 2: 'grass_block', + 3: 'dirt', + 4: 'cobblestone', + 5: 'oak_planks', + 6: 'sapling', + 7: 'bedrock', + 8: 'flowing_water', + 9: 'water', + 10: 'flowing_lava', + 11: 'lava', + 12: 'sand', + 13: 'gravel', + 14: 'gold_ore', + 15: 'iron_ore', + 16: 'coal_ore', + 17: 'oak_log', + 18: 'oak_leaves', + 19: 'sponge', + 20: 'glass', + 21: 'lapis_ore', + 24: 'sandstone', + 30: 'cobweb', + 31: 'grass', + 32: 'dead_bush', + 35: 'white_wool', + 37: 'dandelion', + 38: 'poppy', + 39: 'brown_mushroom', + 40: 'red_mushroom', + 43: 'stone_slab', + 44: 'stone_slab', + 47: 'bookshelf', + 48: 'mossy_cobblestone', + 49: 'obsidian', + 50: 'torch', + 51: 'fire', + 52: 'spawner', + 53: 'oak_stairs', + 54: 'chest', + 56: 'diamond_ore', + 58: 'crafting_table', + 59: 'wheat', + 60: 'farmland', + 61: 'furnace', + 62: 'furnace', + 63: 'sign', # will change to oak_sign in 1.14 + 64: 'oak_door', + 65: 'ladder', + 66: 'rail', + 67: 'cobblestone_stairs', + 72: 'oak_pressure_plate', + 73: 'redstone_ore', + 74: 'redstone_ore', + 78: 'snow', + 79: 'ice', + 81: 'cactus', + 82: 'clay', + 83: 'sugar_cane', + 85: 'oak_fence', + 86: 'pumpkin', + 91: 'lit_pumpkin', + 101: 'iron_bars', + 102: 'glass_pane', + } + + +def block_id_to_name(bid): + try: + name = block_ids[bid] + except KeyError: + name = 'unknown_%d' % (bid,) + print("warning: unknown block id %i" % bid) + print("hint: add that block to the 'block_ids' map") + return name + + +# Generic Chunk class Chunk(object): """Class for representing a single chunk.""" def __init__(self, nbt): - chunk_data = nbt['Level'] - self.coords = chunk_data['xPos'],chunk_data['zPos'] - self.blocks = BlockArray(chunk_data['Blocks'].value, chunk_data['Data'].value) + self.chunk_data = nbt['Level'] + self.coords = self.chunk_data['xPos'],self.chunk_data['zPos'] def get_coords(self): """Return the coordinates of this chunk.""" @@ -22,6 +112,225 @@ def __repr__(self): return "Chunk("+str(self.coords[0])+","+str(self.coords[1])+")" +# Chunk in Region old format + +class McRegionChunk(Chunk): + + def __init__(self, nbt): + Chunk.__init__(self, nbt) + self.blocks = BlockArray(self.chunk_data['Blocks'].value, self.chunk_data['Data'].value) + + def get_max_height(self): + return 127 + + def get_block(self, x, y, z): + name = block_id_to_name(self.blocks.get_block(x, y, z)) + return name + + def iter_block(self): + for y in range(0, 128): + for z in range(0, 16): + for x in range(0, 16): + yield self.get_block(x, y, z) + + +# Section in Anvil new format + +class AnvilSection(object): + + def __init__(self, nbt, version): + self.names = [] + self.indexes = [] + + # Is the section flattened ? + # See https://minecraft.wiki/w/1.13/Flattening + + if version == 0 or version == 1343: # 1343 = MC 1.12.2 + self._init_array(nbt) + elif version >= 1631 and version <= 2230: # MC 1.13 to MC 1.15.2 + self._init_index_unpadded(nbt) + elif version >= 2566 and version <= 2730: # MC 1.16.0 to MC 1.17.2 (latest tested version) + self._init_index_padded(nbt) + else: + raise NotImplementedError() + + # Section contains 4096 blocks whatever data version + + assert len(self.indexes) == 4096 + + + # Decode legacy section + # Contains an array of block numeric identifiers + + def _init_array(self, nbt): + bids = [] + for bid in nbt['Blocks'].value: + try: + i = bids.index(bid) + except ValueError: + bids.append(bid) + i = len(bids) - 1 + self.indexes.append(i) + + for bid in bids: + bname = block_id_to_name(bid) + self.names.append(bname) + + + # Decode modern section + # Contains palette of block names and indexes packed with run-on between elements (pre 1.16 format) + + def _init_index_unpadded(self, nbt): + + for p in nbt['Palette']: + name = p['Name'].value + self.names.append(name) + + states = nbt['BlockStates'].value + + # Block states are packed into an array of longs + # with variable number of bits per block (min: 4) + + num_bits = (len(self.names) - 1).bit_length() + if num_bits < 4: num_bits = 4 + assert num_bits == len(states) * 64 / 4096 + mask = pow(2, num_bits) - 1 + + i = 0 + bits_left = 64 + curr_long = states[0] + + for _ in range(0,4096): + if bits_left == 0: + i = i + 1 + curr_long = states[i] + bits_left = 64 + + if num_bits <= bits_left: + self.indexes.append(curr_long & mask) + curr_long = curr_long >> num_bits + bits_left = bits_left - num_bits + else: + i = i + 1 + next_long = states[i] + remaining_bits = num_bits - bits_left + + next_long = (next_long & (pow(2, remaining_bits) - 1)) << bits_left + curr_long = (curr_long & (pow(2, bits_left) - 1)) + self.indexes.append(next_long | curr_long) + + curr_long = states[i] + curr_long = curr_long >> remaining_bits + bits_left = 64 - remaining_bits + + + # Decode modern section + # Contains palette of block names and indexes packed with padding if elements don't fit (post 1.16 format) + + def _init_index_padded(self, nbt): + + for p in nbt['Palette']: + name = p['Name'].value + self.names.append(name) + + states = nbt['BlockStates'].value + num_bits = (len(self.names) - 1).bit_length() + if num_bits < 4: num_bits = 4 + mask = 2**num_bits - 1 + + indexes_per_element = 64 // num_bits + last_state_elements = 4096 % indexes_per_element + if last_state_elements == 0: last_state_elements = indexes_per_element + + assert len(states) == ceil(4096 / indexes_per_element) + + for i in range(len(states)-1): + long = states[i] + + for _ in range(indexes_per_element): + self.indexes.append(long & mask) + long = long >> num_bits + + + long = states[-1] + for _ in range(last_state_elements): + self.indexes.append(long & mask) + long = long >> num_bits + + + + def get_block(self, x, y, z): + # Blocks are stored in YZX order + i = y * 256 + z * 16 + x + p = self.indexes[i] + return self.names[p] + + + def iter_block(self): + for i in range(0, 4096): + p = self.indexes[i] + yield self.names[p] + + +# Chunck in Anvil new format + +class AnvilChunk(Chunk): + + def __init__(self, nbt): + Chunk.__init__(self, nbt) + + # Started to work on this class with MC version 1.13.2 + # so with the chunk data version 1631 + # Backported to first Anvil version (= 0) from examples + # Could work with other versions, but has to be tested first + + try: + version = nbt['DataVersion'].value + if version != 1343 and not (version >= 1631 or version <= 2730): + raise NotImplementedError('DataVersion %d not implemented' % (version,)) + except KeyError: + version = 0 + + # Load all sections + + self.sections = {} + if 'Sections' in self.chunk_data: + for s in self.chunk_data['Sections']: + if "BlockStates" in s.keys(): # sections may only contain lighting information + self.sections[s['Y'].value] = AnvilSection(s, version) + + + def get_section(self, y): + """Get a section from Y index.""" + if y in self.sections: + return self.sections[y] + + return None + + + def get_max_height(self): + ymax = 0 + for y in self.sections.keys(): + if y > ymax: ymax = y + return ymax * 16 + 15 + + + def get_block(self, x, y, z): + """Get a block from relative x,y,z.""" + sy,by = divmod(y, 16) + section = self.get_section(sy) + if section == None: + return None + + return section.get_block(x, by, z) + + + def iter_block(self): + for s in self.sections.values(): + for b in s.iter_block(): + yield b + + class BlockArray(object): """Convenience class for dealing with a Block/data byte array.""" def __init__(self, blocksBytes=None, dataBytes=None): @@ -36,28 +345,6 @@ def __init__(self, blocksBytes=None, dataBytes=None): else: self.dataList = [0]*16384 # Create an empty data list (32768 4-bit entries of zero make 16384 byte entries) - # Get all block entries - def get_all_blocks(self): - """Return the blocks that are in this BlockArray.""" - return self.blocksList - - # Get all data entries - def get_all_data(self): - """Return the data of all the blocks in this BlockArray.""" - bits = [] - for b in self.dataList: - # The first byte of the Blocks arrays correspond - # to the LEAST significant bits of the first byte of the Data. - # NOT to the MOST significant bits, as you might expected. - bits.append(b & 15) # Little end of the byte - bits.append((b >> 4) & 15) # Big end of the byte - return bits - - # Get all block entries and data entries as tuples - def get_all_blocks_and_data(self): - """Return both blocks and data, packed together as tuples.""" - return list(zip(self.get_all_blocks(), self.get_all_data())) - def get_blocks_struct(self): """Return a dictionary with block ids keyed to (x, y, z).""" cur_x = 0 @@ -173,26 +460,3 @@ def get_block(self, x,y,z, coord=False): offset = y + z*128 + x*128*16 if (coord == False) else coord[1] + coord[2]*128 + coord[0]*128*16 return self.blocksList[offset] - - # Get a given X,Y,Z or a tuple of three coordinates - def get_data(self, x,y,z, coord=False): - """Return the data of the block at x, y, z.""" - offset = y + z*128 + x*128*16 if (coord == False) else coord[1] + coord[2]*128 + coord[0]*128*16 - # The first byte of the Blocks arrays correspond - # to the LEAST significant bits of the first byte of the Data. - # NOT to the MOST significant bits, as you might expected. - if (offset % 2 == 1): - # offset is odd - index = (offset-1)//2 - b = self.dataList[index] - return b & 15 # Get little (last 4 bits) end of byte - else: - # offset is even - index = offset//2 - b = self.dataList[index] - return (b >> 4) & 15 # Get big end (first 4 bits) of byte - - def get_block_and_data(self, x,y,z, coord=False): - """Return the tuple of (id, data) for the block at x, y, z""" - return (self.get_block(x,y,z,coord),self.get_data(x,y,z,coord)) - diff --git a/nbt/nbt.py b/nbt/nbt.py index e98cacb..c5864ba 100644 --- a/nbt/nbt.py +++ b/nbt/nbt.py @@ -1,20 +1,27 @@ """ Handle the NBT (Named Binary Tag) data format + +For more information about the NBT format: +https://minecraft.wiki/w/NBT_format """ from struct import Struct, error as StructError from gzip import GzipFile -import zlib -from collections import MutableMapping, MutableSequence, Sequence -import os, io -try: - unicode - basestring -except NameError: - unicode = str # compatibility for Python 3 - basestring = str # compatibility for Python 3 +from mutf8 import encode_modified_utf8, decode_modified_utf8 +try: + from collections.abc import MutableMapping, MutableSequence, Sequence +except ImportError: # for Python 2.7 + from collections import MutableMapping, MutableSequence, Sequence +import sys + +_PY3 = sys.version_info >= (3,) +if _PY3: + unicode = str + basestring = str +else: + range = xrange TAG_END = 0 TAG_BYTE = 1 @@ -28,11 +35,14 @@ TAG_LIST = 9 TAG_COMPOUND = 10 TAG_INT_ARRAY = 11 +TAG_LONG_ARRAY = 12 + class MalformedFileError(Exception): """Exception raised on parse error.""" pass + class TAG(object): """TAG, a variable with an intrinsic name.""" id = None @@ -41,114 +51,145 @@ def __init__(self, value=None, name=None): self.name = name self.value = value - #Parsers and Generators + # Parsers and Generators def _parse_buffer(self, buffer): raise NotImplementedError(self.__class__.__name__) def _render_buffer(self, buffer): raise NotImplementedError(self.__class__.__name__) - #Printing and Formatting of tree + # Printing and Formatting of tree def tag_info(self): """Return Unicode string with class, name and unnested value.""" - return self.__class__.__name__ + \ - ('(%r)' % self.name if self.name else "") + \ - ": " + self.valuestr() + return self.__class__.__name__ + ( + '(%r)' % self.name if self.name + else "") + ": " + self.valuestr() + def valuestr(self): - """Return Unicode string of unnested value. For iterators, this returns a summary.""" + """Return Unicode string of unnested value. For iterators, this + returns a summary.""" return unicode(self.value) + def namestr(self): + """Return Unicode string of tag name.""" + return unicode(self.name) + def pretty_tree(self, indent=0): - """Return formated Unicode string of self, where iterable items are recursively listed in detail.""" - return ("\t"*indent) + self.tag_info() + """Return formated Unicode string of self, where iterable items are + recursively listed in detail.""" + return ("\t" * indent) + self.tag_info() # Python 2 compatibility; Python 3 uses __str__ instead. def __unicode__(self): - """Return a unicode string with the result in human readable format. Unlike valuestr(), the result is recursive for iterators till at least one level deep.""" + """Return a unicode string with the result in human readable format. + Unlike valuestr(), the result is recursive for iterators till at least + one level deep.""" return unicode(self.value) def __str__(self): - """Return a string (ascii formated for Python 2, unicode for Python 3) with the result in human readable format. Unlike valuestr(), the result is recursive for iterators till at least one level deep.""" + """Return a string (ascii formated for Python 2, unicode for Python 3) + with the result in human readable format. Unlike valuestr(), the result + is recursive for iterators till at least one level deep.""" return str(self.value) + # Unlike regular iterators, __repr__() is not recursive. # Use pretty_tree for recursive results. - # iterators should use __repr__ or tag_info for each item, like regular iterators + # iterators should use __repr__ or tag_info for each item, like + # regular iterators def __repr__(self): - """Return a string (ascii formated for Python 2, unicode for Python 3) describing the class, name and id for debugging purposes.""" - return "<%s(%r) at 0x%x>" % (self.__class__.__name__,self.name,id(self)) + """Return a string (ascii formated for Python 2, unicode for Python 3) + describing the class, name and id for debugging purposes.""" + return "<%s(%r) at 0x%x>" % ( + self.__class__.__name__, self.name, id(self)) + class _TAG_Numeric(TAG): """_TAG_Numeric, comparable to int with an intrinsic name""" + def __init__(self, value=None, name=None, buffer=None): super(_TAG_Numeric, self).__init__(value, name) if buffer: self._parse_buffer(buffer) - #Parsers and Generators + # Parsers and Generators def _parse_buffer(self, buffer): - # Note: buffer.read() may raise an IOError, for example if buffer is a corrupt gzip.GzipFile + # Note: buffer.read() may raise an IOError, for example if buffer is a + # corrupt gzip.GzipFile self.value = self.fmt.unpack(buffer.read(self.fmt.size))[0] def _render_buffer(self, buffer): buffer.write(self.fmt.pack(self.value)) + class _TAG_End(TAG): id = TAG_END fmt = Struct(">b") def _parse_buffer(self, buffer): - # Note: buffer.read() may raise an IOError, for example if buffer is a corrupt gzip.GzipFile + # Note: buffer.read() may raise an IOError, for example if buffer is a + # corrupt gzip.GzipFile value = self.fmt.unpack(buffer.read(1))[0] if value != 0: - raise ValueError("A Tag End must be rendered as '0', not as '%d'." % (value)) + raise ValueError( + "A Tag End must be rendered as '0', not as '%d'." % value) def _render_buffer(self, buffer): buffer.write(b'\x00') -#== Value Tags ==# + +# == Value Tags ==# class TAG_Byte(_TAG_Numeric): """Represent a single tag storing 1 byte.""" id = TAG_BYTE fmt = Struct(">b") + class TAG_Short(_TAG_Numeric): """Represent a single tag storing 1 short.""" id = TAG_SHORT fmt = Struct(">h") + class TAG_Int(_TAG_Numeric): """Represent a single tag storing 1 int.""" id = TAG_INT fmt = Struct(">i") """Struct(">i"), 32-bits integer, big-endian""" + class TAG_Long(_TAG_Numeric): """Represent a single tag storing 1 long.""" id = TAG_LONG fmt = Struct(">q") + class TAG_Float(_TAG_Numeric): """Represent a single tag storing 1 IEEE-754 floating point number.""" id = TAG_FLOAT fmt = Struct(">f") + class TAG_Double(_TAG_Numeric): - """Represent a single tag storing 1 IEEE-754 double precision floating point number.""" + """Represent a single tag storing 1 IEEE-754 double precision floating + point number.""" id = TAG_DOUBLE fmt = Struct(">d") + class TAG_Byte_Array(TAG, MutableSequence): """ TAG_Byte_Array, comparable to a collections.UserList with an intrinsic name whose values must be bytes """ id = TAG_BYTE_ARRAY + def __init__(self, name=None, buffer=None): + # TODO: add a value parameter as well super(TAG_Byte_Array, self).__init__(name=name) if buffer: self._parse_buffer(buffer) - #Parsers and Generators + # Parsers and Generators def _parse_buffer(self, buffer): length = TAG_Int(buffer=buffer) self.value = bytearray(buffer.read(length.value)) @@ -176,20 +217,22 @@ def __setitem__(self, key, value): self.value[key] = value def __delitem__(self, key): - del(self.value[key]) + del (self.value[key]) def insert(self, key, value): # TODO: check type of value, or is this done by self.value already? self.value.insert(key, value) - #Printing and Formatting of tree + # Printing and Formatting of tree def valuestr(self): return "[%i byte(s)]" % len(self.value) def __unicode__(self): - return '['+",".join([unicode(x) for x in self.value])+']' + return '[' + ",".join([unicode(x) for x in self.value]) + ']' + def __str__(self): - return '['+",".join([str(x) for x in self.value])+']' + return '[' + ",".join([str(x) for x in self.value]) + ']' + class TAG_Int_Array(TAG, MutableSequence): """ @@ -197,7 +240,9 @@ class TAG_Int_Array(TAG, MutableSequence): an intrinsic name whose values must be integers """ id = TAG_INT_ARRAY + def __init__(self, name=None, buffer=None): + # TODO: add a value parameter as well super(TAG_Int_Array, self).__init__(name=name) if buffer: self._parse_buffer(buffer) @@ -206,7 +251,7 @@ def update_fmt(self, length): """ Adjust struct format description to length given """ self.fmt = Struct(">" + str(length) + "i") - #Parsers and Generators + # Parsers and Generators def _parse_buffer(self, buffer): length = TAG_Int(buffer=buffer).value self.update_fmt(length) @@ -235,37 +280,95 @@ def __setitem__(self, key, value): self.value[key] = value def __delitem__(self, key): - del(self.value[key]) + del (self.value[key]) def insert(self, key, value): self.value.insert(key, value) - #Printing and Formatting of tree + # Printing and Formatting of tree def valuestr(self): return "[%i int(s)]" % len(self.value) +class TAG_Long_Array(TAG, MutableSequence): + """ + TAG_Long_Array, comparable to a collections.UserList with + an intrinsic name whose values must be integers + """ + id = TAG_LONG_ARRAY + + def __init__(self, name=None, buffer=None): + super(TAG_Long_Array, self).__init__(name=name) + if buffer: + self._parse_buffer(buffer) + + def update_fmt(self, length): + """ Adjust struct format description to length given """ + self.fmt = Struct(">" + str(length) + "q") + + # Parsers and Generators + def _parse_buffer(self, buffer): + length = TAG_Int(buffer=buffer).value + self.update_fmt(length) + self.value = list(self.fmt.unpack(buffer.read(self.fmt.size))) + + def _render_buffer(self, buffer): + length = len(self.value) + self.update_fmt(length) + TAG_Int(length)._render_buffer(buffer) + buffer.write(self.fmt.pack(*self.value)) + + # Mixin methods + def __len__(self): + return len(self.value) + + def __iter__(self): + return iter(self.value) + + def __contains__(self, item): + return item in self.value + + def __getitem__(self, key): + return self.value[key] + + def __setitem__(self, key, value): + self.value[key] = value + + def __delitem__(self, key): + del (self.value[key]) + + def insert(self, key, value): + self.value.insert(key, value) + + # Printing and Formatting of tree + def valuestr(self): + return "[%i long(s)]" % len(self.value) + + class TAG_String(TAG, Sequence): """ TAG_String, comparable to a collections.UserString with an intrinsic name """ id = TAG_STRING + def __init__(self, value=None, name=None, buffer=None): super(TAG_String, self).__init__(value, name) if buffer: self._parse_buffer(buffer) - #Parsers and Generators + # Parsers and Generators def _parse_buffer(self, buffer): length = TAG_Short(buffer=buffer) read = buffer.read(length.value) if len(read) != length.value: raise StructError() - self.value = read.decode("utf-8") + #self.value = read.decode("utf-8") + self.value = decode_modified_utf8(read) def _render_buffer(self, buffer): - save_val = self.value.encode("utf-8") + #save_val = self.value.encode("utf-8") + save_val = encode_modified_utf8(self.value) length = TAG_Short(len(save_val)) length._render_buffer(buffer) buffer.write(save_val) @@ -283,16 +386,18 @@ def __contains__(self, item): def __getitem__(self, key): return self.value[key] - #Printing and Formatting of tree + # Printing and Formatting of tree def __repr__(self): return self.value -#== Collection Tags ==# + +# == Collection Tags ==# class TAG_List(TAG, MutableSequence): """ TAG_List, comparable to a collections.UserList with an intrinsic name """ id = TAG_LIST + def __init__(self, type=None, value=None, name=None, buffer=None): super(TAG_List, self).__init__(value, name) if type: @@ -302,10 +407,10 @@ def __init__(self, type=None, value=None, name=None, buffer=None): self.tags = [] if buffer: self._parse_buffer(buffer) - if self.tagID == None: - raise ValueError("No type specified for list: %s" % (name)) + # if self.tagID == None: + # raise ValueError("No type specified for list: %s" % (name)) - #Parsers and Generators + # Parsers and Generators def _parse_buffer(self, buffer): self.tagID = TAG_Byte(buffer=buffer).value self.tags = [] @@ -319,8 +424,9 @@ def _render_buffer(self, buffer): length._render_buffer(buffer) for i, tag in enumerate(self.tags): if tag.id != self.tagID: - raise ValueError("List element %d(%s) has type %d != container type %d" % - (i, tag, tag.id, self.tagID)) + raise ValueError( + "List element %d(%s) has type %d != container type %d" % + (i, tag, tag.id, self.tagID)) tag._render_buffer(buffer) # Mixin methods @@ -340,66 +446,76 @@ def __setitem__(self, key, value): self.tags[key] = value def __delitem__(self, key): - del(self.tags[key]) + del (self.tags[key]) def insert(self, key, value): self.tags.insert(key, value) - #Printing and Formatting of tree + # Printing and Formatting of tree def __repr__(self): - return "%i entries of type %s" % (len(self.tags), TAGLIST[self.tagID].__name__) + return "%i entries of type %s" % ( + len(self.tags), TAGLIST[self.tagID].__name__) - #Printing and Formatting of tree + # Printing and Formatting of tree def valuestr(self): return "[%i %s(s)]" % (len(self.tags), TAGLIST[self.tagID].__name__) + def __unicode__(self): - return "["+", ".join([tag.tag_info() for tag in self.tags])+"]" + return "[" + ", ".join([tag.tag_info() for tag in self.tags]) + "]" + def __str__(self): - return "["+", ".join([tag.tag_info() for tag in self.tags])+"]" + return "[" + ", ".join([tag.tag_info() for tag in self.tags]) + "]" def pretty_tree(self, indent=0): output = [super(TAG_List, self).pretty_tree(indent)] if len(self.tags): - output.append(("\t"*indent) + "{") + output.append(("\t" * indent) + "{") output.extend([tag.pretty_tree(indent + 1) for tag in self.tags]) - output.append(("\t"*indent) + "}") + output.append(("\t" * indent) + "}") return '\n'.join(output) + class TAG_Compound(TAG, MutableMapping): """ TAG_Compound, comparable to a collections.OrderedDict with an intrinsic name """ id = TAG_COMPOUND - def __init__(self, buffer=None): + + def __init__(self, buffer=None, name=None): + # TODO: add a value parameter as well super(TAG_Compound, self).__init__() self.tags = [] - self.name = "" + if name: + self.name = name + else: + self.name = "" if buffer: self._parse_buffer(buffer) - #Parsers and Generators + # Parsers and Generators def _parse_buffer(self, buffer): while True: type = TAG_Byte(buffer=buffer) if type.value == TAG_END: - #print("found tag_end") + # print("found tag_end") break else: name = TAG_String(buffer=buffer).value try: - tag = TAGLIST[type.value](buffer=buffer) - tag.name = name - self.tags.append(tag) + tag = TAGLIST[type.value]() except KeyError: - raise ValueError("Unrecognised tag type") + raise ValueError("Unrecognised tag type %d" % type.value) + tag.name = name + self.tags.append(tag) + tag._parse_buffer(buffer) def _render_buffer(self, buffer): for tag in self.tags: TAG_Byte(tag.id)._render_buffer(buffer) TAG_String(tag.name)._render_buffer(buffer) tag._render_buffer(buffer) - buffer.write(b'\x00') #write TAG_END + buffer.write(b'\x00') # write TAG_END # Mixin methods def __len__(self): @@ -431,7 +547,9 @@ def __getitem__(self, key): else: raise KeyError("Tag %s does not exist" % key) else: - raise TypeError("key needs to be either name of tag, or index of tag, not a %s" % type(key).__name__) + raise TypeError( + "key needs to be either name of tag, or index of tag, " + "not a %s" % type(key).__name__) def __setitem__(self, key, value): assert isinstance(value, TAG), "value must be an nbt.TAG" @@ -448,11 +566,12 @@ def __setitem__(self, key, value): def __delitem__(self, key): if isinstance(key, int): - del(self.tags[key]) + del (self.tags[key]) elif isinstance(key, basestring): self.tags.remove(self.__getitem__(key)) else: - raise ValueError("key needs to be either name of tag, or index of tag") + raise ValueError( + "key needs to be either name of tag, or index of tag") def keys(self): return [tag.name for tag in self.tags] @@ -461,11 +580,12 @@ def iteritems(self): for tag in self.tags: yield (tag.name, tag) - #Printing and Formatting of tree + # Printing and Formatting of tree def __unicode__(self): - return "{"+", ".join([tag.tag_info() for tag in self.tags])+"}" + return "{" + ", ".join([tag.tag_info() for tag in self.tags]) + "}" + def __str__(self): - return "{"+", ".join([tag.tag_info() for tag in self.tags])+"}" + return "{" + ", ".join([tag.tag_info() for tag in self.tags]) + "}" def valuestr(self): return '{%i Entries}' % len(self.tags) @@ -473,23 +593,41 @@ def valuestr(self): def pretty_tree(self, indent=0): output = [super(TAG_Compound, self).pretty_tree(indent)] if len(self.tags): - output.append(("\t"*indent) + "{") + output.append(("\t" * indent) + "{") output.extend([tag.pretty_tree(indent + 1) for tag in self.tags]) - output.append(("\t"*indent) + "}") + output.append(("\t" * indent) + "}") return '\n'.join(output) -TAGLIST = {TAG_END: _TAG_End, TAG_BYTE:TAG_Byte, TAG_SHORT:TAG_Short, TAG_INT:TAG_Int, TAG_LONG:TAG_Long, TAG_FLOAT:TAG_Float, TAG_DOUBLE:TAG_Double, TAG_BYTE_ARRAY:TAG_Byte_Array, TAG_STRING:TAG_String, TAG_LIST:TAG_List, TAG_COMPOUND:TAG_Compound, TAG_INT_ARRAY:TAG_Int_Array} +TAGLIST = {TAG_END: _TAG_End, TAG_BYTE: TAG_Byte, TAG_SHORT: TAG_Short, + TAG_INT: TAG_Int, TAG_LONG: TAG_Long, TAG_FLOAT: TAG_Float, + TAG_DOUBLE: TAG_Double, TAG_BYTE_ARRAY: TAG_Byte_Array, + TAG_STRING: TAG_String, TAG_LIST: TAG_List, + TAG_COMPOUND: TAG_Compound, TAG_INT_ARRAY: TAG_Int_Array, + TAG_LONG_ARRAY: TAG_Long_Array} + class NBTFile(TAG_Compound): """Represent an NBT file object.""" + def __init__(self, filename=None, buffer=None, fileobj=None): + """ + Create a new NBTFile object. + Specify either a filename, file object or data buffer. + If filename of file object is specified, data should be GZip-compressed. + If a data buffer is specified, it is assumed to be uncompressed. + + If filename is specified, the file is closed after reading and writing. + If file object is specified, the caller is responsible for closing the + file. + """ super(NBTFile, self).__init__() self.filename = filename self.type = TAG_Byte(self.id) closefile = True - #make a file object + # make a file object if filename: + self.filename = filename self.file = GzipFile(filename, 'rb') elif buffer: if hasattr(buffer, 'name'): @@ -503,12 +641,12 @@ def __init__(self, filename=None, buffer=None, fileobj=None): else: self.file = None closefile = False - #parse the file given initially + # parse the file given initially if self.file: self.parse_file() if closefile: - # Note: GzipFile().close() does NOT close the fileobj, - # So the caller is still responsible for closing that. + # Note: GzipFile().close() does NOT close the fileobj, + # So we are still responsible for closing that. try: self.file.close() except (AttributeError, IOError): @@ -517,12 +655,14 @@ def __init__(self, filename=None, buffer=None, fileobj=None): def parse_file(self, filename=None, buffer=None, fileobj=None): """Completely parse a file, extracting all tags.""" + closefile = True if filename: self.file = GzipFile(filename, 'rb') elif buffer: if hasattr(buffer, 'name'): self.filename = buffer.name self.file = buffer + closefile = False elif fileobj: if hasattr(fileobj, 'name'): self.filename = fileobj.name @@ -534,13 +674,19 @@ def parse_file(self, filename=None, buffer=None, fileobj=None): name = TAG_String(buffer=self.file).value self._parse_buffer(self.file) self.name = name - self.file.close() + if closefile: + self.file.close() else: - raise MalformedFileError("First record is not a Compound Tag") + raise MalformedFileError( + "First record is not a Compound Tag") except StructError as e: - raise MalformedFileError("Partial File Parse: file possibly truncated.") + raise MalformedFileError( + "Partial File Parse: file possibly truncated.") else: - raise ValueError("NBTFile.parse_file(): Need to specify either a filename or a file object") + raise ValueError( + "NBTFile.parse_file(): Need to specify either a " + "filename or a file object" + ) def write_file(self, filename=None, buffer=None, fileobj=None): """Write this NBT file to a file.""" @@ -558,12 +704,15 @@ def write_file(self, filename=None, buffer=None, fileobj=None): elif self.filename: self.file = GzipFile(self.filename, "wb") elif not self.file: - raise ValueError("NBTFile.write_file(): Need to specify either a filename or a file object") - #Render tree to file + raise ValueError( + "NBTFile.write_file(): Need to specify either a " + "filename or a file object" + ) + # Render tree to file TAG_Byte(self.id)._render_buffer(self.file) TAG_String(self.name)._render_buffer(self.file) self._render_buffer(self.file) - #make sure the file is complete + # make sure the file is complete try: self.file.flush() except (AttributeError, IOError): @@ -581,8 +730,12 @@ def __repr__(self): debugging purposes. """ if self.filename: - return "<%s(%r) with %s(%r) at 0x%x>" % (self.__class__.__name__, self.filename, \ - TAG_Compound.__name__, self.name, id(self)) + return "<%s(%r) with %s(%r) at 0x%x>" % ( + self.__class__.__name__, self.filename, + TAG_Compound.__name__, self.name, id(self) + ) else: - return "<%s with %s(%r) at 0x%x>" % (self.__class__.__name__, \ - TAG_Compound.__name__, self.name, id(self)) + return "<%s with %s(%r) at 0x%x>" % ( + self.__class__.__name__, TAG_Compound.__name__, + self.name, id(self) + ) diff --git a/nbt/region.py b/nbt/region.py index 803aa2d..d1c8f46 100644 --- a/nbt/region.py +++ b/nbt/region.py @@ -1,18 +1,20 @@ """ Handle a region file, containing 32x32 chunks. -For more info of the region file format look: -http://www.minecraftwiki.net/wiki/Region_file_format + +For more information about the region file format: +https://minecraft.wiki/w/Region_file_format """ from .nbt import NBTFile, MalformedFileError from struct import pack, unpack -from gzip import GzipFile -from collections import Mapping +try: + from collections.abc import Mapping +except ImportError: # for Python 2.7 + from collections import Mapping import zlib import gzip from io import BytesIO -import math, time -from os.path import getsize +import time from os import SEEK_END # constants @@ -20,6 +22,8 @@ SECTOR_LENGTH = 4096 """Constant indicating the length of a sector. A Region file is divided in sectors of 4096 bytes each.""" +# TODO: move status codes to an (Enum) object + # Status is a number representing: # -5 = Error, the chunk is overlapping with another chunk # -4 = Error, the chunk length is too large to fit in the sector length in the region header @@ -29,7 +33,7 @@ # 0 = Ok # 1 = Chunk non-existant yet STATUS_CHUNK_OVERLAPPING = -5 -"""Constant indicating an error status: the chunk is allocated a sector already occupied by another chunk""" +"""Constant indicating an error status: the chunk is allocated to a sector already occupied by another chunk""" STATUS_CHUNK_MISMATCHED_LENGTHS = -4 """Constant indicating an error status: the region header length and the chunk length are incompatible""" STATUS_CHUNK_ZERO_LENGTH = -3 @@ -44,11 +48,11 @@ """Constant indicating an normal status: the chunk does not exist""" COMPRESSION_NONE = 0 -"""Constant indicating tha tthe chunk is not compressed.""" +"""Constant indicating that the chunk is not compressed.""" COMPRESSION_GZIP = 1 -"""Constant indicating tha tthe chunk is GZip compressed.""" +"""Constant indicating that the chunk is GZip compressed.""" COMPRESSION_ZLIB = 2 -"""Constant indicating tha tthe chunk is zlib compressed.""" +"""Constant indicating that the chunk is zlib compressed.""" # TODO: reconsider these errors. where are they catched? Where would an implementation make a difference in handling the different exceptions. @@ -118,7 +122,7 @@ def __init__(self, x, z): - STATUS_CHUNK_OK - STATUS_CHUNK_NOT_CREATED""" def __str__(self): - return "%s(%d, %d, sector=%s, length=%s, timestamp=%s, lenght=%s, compression=%s, status=%s)" % \ + return "%s(%d, %d, sector=%s, blocklength=%s, timestamp=%s, bytelength=%s, compression=%s, status=%s)" % \ (self.__class__.__name__, self.x, self.z, self.blockstart, self.blocklength, self.timestamp, \ self.length, self.compression, self.status) def __repr__(self): @@ -139,7 +143,7 @@ def __getitem__(self, xz): m = self.metadata[xz] return (m.blockstart, m.blocklength, m.timestamp, m.status) def __iter__(self): - return iter(self.metadata) # iterates of the keys + return iter(self.metadata) # iterates over the keys def __len__(self): return len(self.metadata) class _ChunkHeaderWrapper(Mapping): @@ -150,16 +154,24 @@ def __getitem__(self, xz): m = self.metadata[xz] return (m.length if m.length > 0 else None, m.compression, m.status) def __iter__(self): - return iter(self.metadata) # iterates of the keys + return iter(self.metadata) # iterates over the keys def __len__(self): return len(self.metadata) +class Location(object): + def __init__(self, x=None, y=None, z=None): + self.x = x + self.y = y + self.z = z + def __str__(self): + return "%s(x=%s, y=%s, z=%s)" % (self.__class__.__name__, self.x, self.y, self.z) + class RegionFile(object): """A convenience class for extracting NBT files from the Minecraft Beta Region Format.""" # Redefine constants for backward compatibility. STATUS_CHUNK_OVERLAPPING = STATUS_CHUNK_OVERLAPPING - """Constant indicating an error status: the chunk is allocated a sector + """Constant indicating an error status: the chunk is allocated to a sector already occupied by another chunk. Deprecated. Use :const:`nbt.region.STATUS_CHUNK_OVERLAPPING` instead.""" STATUS_CHUNK_MISMATCHED_LENGTHS = STATUS_CHUNK_MISMATCHED_LENGTHS @@ -181,14 +193,17 @@ class RegionFile(object): """Constant indicating an normal status: the chunk does not exist. Deprecated. Use :const:`nbt.region.STATUS_CHUNK_NOT_CREATED` instead.""" - def __init__(self, filename=None, fileobj=None): + def __init__(self, filename=None, fileobj=None, chunkclass = None): """ - Read a region file by filename of file object. - If a fileobj is specified, it is not closed after use; it is the callers responibility to close that. + Read a region file by filename or file object. + If a fileobj is specified, it is not closed after use; it is the callers responibility to close it. """ self.file = None self.filename = None self._closefile = False + self.closed = False + """Set to true if `close()` was successfully called on that region""" + self.chunkclass = chunkclass if filename: self.filename = filename self.file = open(filename, 'r+b') # open for read and write in binary mode @@ -244,6 +259,9 @@ def __init__(self, filename=None, fileobj=None): Deprecated. Use :attr:`metadata` instead. """ + self.loc = Location() + """Optional: x,z location of a region within a world.""" + self._init_header() self._parse_header() self._parse_chunk_headers() @@ -263,9 +281,23 @@ def _bytes_to_sector(bsize, sectorlength=SECTOR_LENGTH): sectors, remainder = divmod(bsize, sectorlength) return sectors if remainder == 0 else sectors + 1 - def __del__(self): + def close(self): + """ + Clean up resources after use. + + Note that the instance is no longer readable nor writable after calling close(). + The method is automatically called by garbage collectors, but made public to + allow explicit cleanup. + """ if self._closefile: - self.file.close() + try: + self.file.close() + self.closed = True + except IOError: + pass + + def __del__(self): + self.close() # Parent object() has no __del__ method, otherwise it should be called here. def _init_file(self): @@ -302,7 +334,7 @@ def _parse_header(self): m = self.metadata[x, z] self.file.seek(index) - offset, length = unpack(">IB", b"\0"+self.file.read(4)) + offset, length = unpack(">IB", b"\0" + self.file.read(4)) m.blockstart, m.blocklength = offset, length self.file.seek(index + SECTOR_LENGTH) m.timestamp = unpack(">I", self.file.read(4))[0] @@ -335,6 +367,8 @@ def _parse_chunk_headers(self): m = self.metadata[x, z] if m.status not in (STATUS_CHUNK_OK, STATUS_CHUNK_OVERLAPPING, \ STATUS_CHUNK_MISMATCHED_LENGTHS): + # skip to next if status is NOT_CREATED, OUT_OF_FILE, IN_HEADER, + # ZERO_LENGTH or anything else. continue try: self.file.seek(m.blockstart*SECTOR_LENGTH) # offset comes in sectors of 4096 bytes @@ -345,7 +379,9 @@ def _parse_chunk_headers(self): except IOError: m.status = STATUS_CHUNK_OUT_OF_FILE continue - if m.length <= 1: # chunk can't be zero length + if m.blockstart*SECTOR_LENGTH + m.length + 4 > self.size: + m.status = STATUS_CHUNK_OUT_OF_FILE + elif m.length <= 1: # chunk can't be zero length m.status = STATUS_CHUNK_ZERO_LENGTH elif m.length + 4 > m.blocklength * SECTOR_LENGTH: # There are not enough sectors allocated for the whole block @@ -365,9 +401,10 @@ def _sectors(self, ignore_chunk=None): if ignore_chunk == m: continue if m.blocklength and m.blockstart: - for b in range(m.blockstart, m.blockstart + max(m.blocklength, m.requiredblocks())): - if 2 <= b < sectorsize: - sectors[b].append(m) + blockend = m.blockstart + max(m.blocklength, m.requiredblocks()) + # Ensure 2 <= b < sectorsize, as well as m.blockstart <= b < blockend + for b in range(max(m.blockstart, 2), min(blockend, sectorsize)): + sectors[b].append(m) return sectors def _locate_free_sectors(self, ignore_chunk=None): @@ -447,14 +484,36 @@ def iter_chunks(self): yield self.get_chunk(m.x, m.z) except RegionFileFormatError: pass - + + # The following method will replace 'iter_chunks' + # but the previous is kept for the moment + # until the users update their code + + def iter_chunks_class(self): + """ + Yield each readable chunk present in the region. + Chunks that can not be read for whatever reason are silently skipped. + This function returns a :class:`nbt.chunk.Chunk` instance. + """ + for m in self.get_metadata(): + try: + yield self.chunkclass(self.get_chunk(m.x, m.z)) + except RegionFileFormatError: + pass + def __iter__(self): return self.iter_chunks() def get_timestamp(self, x, z): - """Return the timestamp of when this region file was last modified.""" - # TODO: raise an exception if chunk does not exist? - # TODO: return a datetime.datetime object using datetime.fromtimestamp() + """ + Return the timestamp of when this region file was last modified. + + Note that this returns the timestamp as-is. A timestamp may exist, + while the chunk does not, or it may return a timestamp of 0 even + while the chunk exists. + + To convert to an actual date, use `datetime.fromtimestamp()`. + """ return self.metadata[x,z].timestamp def chunk_count(self): @@ -462,33 +521,45 @@ def chunk_count(self): return len(self.get_metadata()) def get_blockdata(self, x, z): - """Return the decompressed binary data representing a chunk.""" + """ + Return the decompressed binary data representing a chunk. + + May raise a RegionFileFormatError(). + If decompression of the data succeeds, all available data is returned, + even if it is shorter than what is specified in the header (e.g. in case + of a truncated while and non-compressed data). + """ # read metadata block m = self.metadata[x, z] if m.status == STATUS_CHUNK_NOT_CREATED: - raise InconceivedChunk("Chunk is not created") + raise InconceivedChunk("Chunk %d,%d is not present in region" % (x,z)) elif m.status == STATUS_CHUNK_IN_HEADER: raise RegionHeaderError('Chunk %d,%d is in the region header' % (x,z)) - elif m.status == STATUS_CHUNK_OUT_OF_FILE: + elif m.status == STATUS_CHUNK_OUT_OF_FILE and (m.length <= 1 or m.compression == None): + # Chunk header is outside of the file. raise RegionHeaderError('Chunk %d,%d is partially/completely outside the file' % (x,z)) elif m.status == STATUS_CHUNK_ZERO_LENGTH: if m.blocklength == 0: raise RegionHeaderError('Chunk %d,%d has zero length' % (x,z)) else: raise ChunkHeaderError('Chunk %d,%d has zero length' % (x,z)) + elif m.blockstart * SECTOR_LENGTH + 5 >= self.size: + raise RegionHeaderError('Chunk %d,%d is partially/completely outside the file' % (x,z)) - # status is STATUS_CHUNK_OK, STATUS_CHUNK_MISMATCHED_LENGTHS or STATUS_CHUNK_OVERLAPPING. + # status is STATUS_CHUNK_OK, STATUS_CHUNK_MISMATCHED_LENGTHS, STATUS_CHUNK_OVERLAPPING + # or STATUS_CHUNK_OUT_OF_FILE. # The chunk is always read, but in case of an error, the exception may be different # based on the status. - # offset comes in sectors of 4096 bytes + length bytes + compression byte - self.file.seek(m.blockstart * SECTOR_LENGTH + 5) - chunk = self.file.read(m.length-1) # the length in the file includes the compression byte - err = None - if m.compression > 2: - raise ChunkDataError('Unknown chunk compression/format (%d)' % m.compression) try: + # offset comes in sectors of 4096 bytes + length bytes + compression byte + self.file.seek(m.blockstart * SECTOR_LENGTH + 5) + # Do not read past the length of the file. + # The length in the file includes the compression byte, hence the -1. + length = min(m.length - 1, self.size - (m.blockstart * SECTOR_LENGTH + 5)) + chunk = self.file.read(length) + if (m.compression == COMPRESSION_GZIP): # Python 3.1 and earlier do not yet support gzip.decompress(chunk) f = gzip.GzipFile(fileobj=BytesIO(chunk)) @@ -496,11 +567,16 @@ def get_blockdata(self, x, z): f.close() elif (m.compression == COMPRESSION_ZLIB): chunk = zlib.decompress(chunk) + elif m.compression != COMPRESSION_NONE: + raise ChunkDataError('Unknown chunk compression/format (%s)' % m.compression) + return chunk + except RegionFileFormatError: + raise except Exception as e: # Deliberately catch the Exception and re-raise. # The details in gzip/zlib/nbt are irrelevant, just that the data is garbled. - err = str(e) + err = '%s' % e # avoid str(e) due to Unicode issues in Python 2. if err: # don't raise during exception handling to avoid the warning # "During handling of the above exception, another exception occurred". @@ -517,14 +593,21 @@ def get_nbt(self, x, z): Return a NBTFile of the specified chunk. Raise InconceivedChunk if the chunk is not included in the file. """ + # TODO: cache results? data = self.get_blockdata(x, z) # This may raise a RegionFileFormatError. data = BytesIO(data) err = None try: - return NBTFile(buffer=data) + nbt = NBTFile(buffer=data) + if self.loc.x != None: + x += self.loc.x*32 + if self.loc.z != None: + z += self.loc.z*32 + nbt.loc = Location(x=x, z=z) + return nbt # this may raise a MalformedFileError. Convert to ChunkDataError. except MalformedFileError as e: - err = str(e) + err = '%s' % e # avoid str(e) due to Unicode issues in Python 2. if err: raise ChunkDataError(err) @@ -538,12 +621,24 @@ def get_chunk(self, x, z): """ return self.get_nbt(x, z) - def write_blockdata(self, x, z, data): + def write_blockdata(self, x, z, data, compression=COMPRESSION_ZLIB): """ Compress the data, write it to file, and add pointers in the header so it can be found as chunk(x,z). """ - data = zlib.compress(data) # use zlib compression, rather than Gzip + if compression == COMPRESSION_GZIP: + # Python 3.1 and earlier do not yet support `data = gzip.compress(data)`. + compressed_file = BytesIO() + f = gzip.GzipFile(fileobj=compressed_file) + f.write(data) + f.close() + compressed_file.seek(0) + data = compressed_file.read() + del compressed_file + elif compression == COMPRESSION_ZLIB: + data = zlib.compress(data) # use zlib compression, rather than Gzip + elif compression != COMPRESSION_NONE: + raise ValueError("Unknown compression type %d" % compression) length = len(data) # 5 extra bytes are required for the chunk block header @@ -561,10 +656,17 @@ def write_blockdata(self, x, z, data): free_sectors = self._locate_free_sectors(ignore_chunk=current) sector = self._find_free_location(free_sectors, nsectors, preferred=current.blockstart) + # If file is smaller than sector*SECTOR_LENGTH (it was truncated), pad it with zeroes. + if self.size < sector*SECTOR_LENGTH: + # jump to end of file + self.file.seek(0, SEEK_END) + self.file.write((sector*SECTOR_LENGTH - self.size) * b"\x00") + assert self.file.tell() == sector*SECTOR_LENGTH + # write out chunk to region self.file.seek(sector*SECTOR_LENGTH) self.file.write(pack(">I", length + 1)) #length field - self.file.write(pack(">B", COMPRESSION_ZLIB)) #compression field + self.file.write(pack(">B", compression)) #compression field self.file.write(data) #compressed data # Write zeros up to the end of the chunk @@ -601,7 +703,8 @@ def write_blockdata(self, x, z, data): self.file.write(SECTOR_LENGTH*b'\x00') # update file size and header information - self.size = self.get_size() + self.size = max((sector + nsectors)*SECTOR_LENGTH, self.size) + assert self.get_size() == self.size current.blockstart = sector current.blocklength = nsectors current.status = STATUS_CHUNK_OK diff --git a/nbt/world.py b/nbt/world.py index 0555fbc..d185f01 100644 --- a/nbt/world.py +++ b/nbt/world.py @@ -1,11 +1,14 @@ """ Handles a Minecraft world save using either the Anvil or McRegion format. + +For more information about the world format: +https://minecraft.wiki/w/Level_format """ import os, glob, re from . import region from . import chunk -from .region import InconceivedChunk +from .region import InconceivedChunk, Location class UnknownWorldFormat(Exception): """Unknown or invalid world folder.""" @@ -13,7 +16,6 @@ def __init__(self, msg=""): self.msg = msg - class _BaseWorldFolder(object): """ Abstract class, representing either a McRegion or Anvil world folder. @@ -21,6 +23,8 @@ class _BaseWorldFolder(object): Simply calling WorldFolder() will do this automatically. """ type = "Generic" + extension = '' + chunkclass = chunk.Chunk def __init__(self, world_folder): """Initialize a WorldFolder.""" @@ -34,6 +38,9 @@ def __init__(self, world_folder): self.set_regionfiles(self.get_filenames()) def get_filenames(self): + """Find all matching file names in the world folder. + + This method is private, and it's use it deprecated. Use get_regionfiles() instead.""" # Warning: glob returns a empty list if the directory is unreadable, without raising an Exception return list(glob.glob(os.path.join(self.worldfolder,'region','r.*.*.'+self.extension))) @@ -59,63 +66,78 @@ def set_regionfiles(self, filenames): pass self.regionfiles[(x,z)] = filename - def nonempty(self): - """Return True is the world is non-empty.""" - return len(self.regionfiles) > 0 - def get_regionfiles(self): """Return a list of full path of all region files.""" return list(self.regionfiles.values()) + def nonempty(self): + """Return True is the world is non-empty.""" + return len(self.regionfiles) > 0 + def get_region(self, x,z): """Get a region using x,z coordinates of a region. Cache results.""" - if (x,z) not in self.regions: + if (x,z) not in self.regions or self.regions[x,z].closed: if (x,z) in self.regionfiles: self.regions[(x,z)] = region.RegionFile(self.regionfiles[(x,z)]) else: # Return an empty RegionFile object # TODO: this does not yet allow for saving of the region file + # TODO: this currently fails with a ValueError! + # TODO: generate the correct name, and create the file + # and add the fie to self.regionfiles self.regions[(x,z)] = region.RegionFile() + self.regions[(x,z)].loc = Location(x=x,z=z) return self.regions[(x,z)] def iter_regions(self): - for x,z in self.regionfiles.keys(): - yield self.get_region(x,z) - - def iter_nbt(self): """ - Return an iterable list of all NBT. Use this function if you only - want to loop through the chunks once, and don't need the block or data arrays. + Return an iterable list of all region files. Use this function if you only + want to loop through each region files once, and do not want to cache the results. """ # TODO: Implement BoundingBox # TODO: Implement sort order - for region in self.iter_regions(): - for c in region.iter_chunks(): - yield c + for x,z in self.regionfiles.keys(): + close_after_use = False + if (x,z) in self.regions: + regionfile = self.regions[(x,z)] + else: + # It is not yet cached. + # Get file, but do not cache later. + regionfile = region.RegionFile(self.regionfiles[(x,z)], chunkclass = self.chunkclass) + regionfile.loc = Location(x=x,z=z) + close_after_use = True + try: + yield regionfile + finally: + if close_after_use: + regionfile.close() - def iter_chunks(self): + def call_for_each_region(self, callback_function, boundingbox=None): """ - Return an iterable list of all chunks. Use this function if you only - want to loop through the chunks once or have a very large world. - Use get_chunks() if you access the chunk list frequently and want to cache - the results. Use iter_nbt() if you are concerned about speed and don't want - to parse the block data. + Return an iterable that calls callback_function for each region file + in the world. This is equivalent to: + ``` + for the_region in iter_regions(): + yield callback_function(the_region) + ```` + + This function is threaded. It uses pickle to pass values between threads. + See [What can be pickled and unpickled?](https://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled) in the Python documentation + for limitation on the output of `callback_function()`. """ - # TODO: Implement BoundingBox - # TODO: Implement sort order - for c in self.iter_nbt(): - yield self.chunkclass(c) + raise NotImplementedError() def get_nbt(self,x,z): """ Return a NBT specified by the chunk coordinates x,z. Raise InconceivedChunk if the NBT file is not yet generated. To get a Chunk object, use get_chunk. """ - rx,x = divmod(x,32) - rz,z = divmod(z,32) - nbt = self.get_region(rx,rz).get_chunk(x,z) - if nbt == None: - raise InconceivedChunk("Chunk %s,%s not present in world" % (32*rx+x,32*rz+z)) + rx,cx = divmod(x,32) + rz,cz = divmod(z,32) + if (rx,rz) not in self.regions and (rx,rz) not in self.regionfiles: + raise InconceivedChunk("Chunk %s,%s is not present in world" % (x,z)) + nbt = self.get_region(rx,rz).get_nbt(cx,cz) + assert nbt != None return nbt def set_nbt(self,x,z,nbt): @@ -124,9 +146,35 @@ def set_nbt(self,x,z,nbt): adds it to the Regionfile. May create a new Regionfile if that did not exist yet. nbt must be a nbt.NBTFile instance, not a Chunk or regular TAG_Compound object. """ - raise NotImplemented() + raise NotImplementedError() # TODO: implement + def iter_nbt(self): + """ + Return an iterable list of all NBT. Use this function if you only + want to loop through the chunks once, and don't need the block or data arrays. + """ + # TODO: Implement BoundingBox + # TODO: Implement sort order + for region in self.iter_regions(): + for c in region.iter_chunks(): + yield c + + def call_for_each_nbt(self, callback_function, boundingbox=None): + """ + Return an iterable that calls callback_function for each NBT structure + in the world. This is equivalent to: + ``` + for the_nbt in iter_nbt(): + yield callback_function(the_nbt) + ```` + + This function is threaded. It uses pickle to pass values between threads. + See [What can be pickled and unpickled?](https://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled) in the Python documentation + for limitation on the output of `callback_function()`. + """ + raise NotImplementedError() + def get_chunk(self,x,z): """ Return a chunk specified by the chunk coordinates x,z. Raise InconceivedChunk @@ -145,6 +193,19 @@ def get_chunks(self, boundingbox=None): self.chunks = list(self.iter_chunks()) return self.chunks + def iter_chunks(self): + """ + Return an iterable list of all chunks. Use this function if you only + want to loop through the chunks once or have a very large world. + Use get_chunks() if you access the chunk list frequently and want to cache + the results. Use iter_nbt() if you are concerned about speed and don't want + to parse the block data. + """ + # TODO: Implement BoundingBox + # TODO: Implement sort order + for c in self.iter_nbt(): + yield self.chunkclass(c) + def chunk_count(self): """Return a count of the chunks in this world folder.""" c = 0 @@ -166,26 +227,6 @@ def get_boundingbox(self): b.expand(x,None,z) return b - def cache_test(self): - """ - Debug routine: loop through all chunks, fetch them again by coordinates, - and check if the same object is returned. - """ - # TODO: make sure this test succeeds (at least True,True,False, preferable True,True,True) - # TODO: Move this function to test class. - for rx,rz in self.regionfiles.keys(): - region = self.get_region(rx,rz) - rx,rz = 32*rx,32*rz - for cc in region.get_chunk_coords(): - x,z = (rx+cc['x'],rz+cc['z']) - c1 = self.chunkclass(region.get_chunk(cc['x'],cc['z'])) - c2 = self.get_chunk(x,z) - correct_coords = (c2.get_coords() == (x,z)) - is_comparable = (c1 == c2) # test __eq__ function - is_equal = (id(c1) == id(c2)) # test if they point to the same memory location - # DEBUG (prints a tuple) - print((x,z,c1,c2,correct_coords,is_comparable,is_equal)) - def __repr__(self): return "%s(%r)" % (self.__class__.__name__,self.worldfolder) @@ -194,18 +235,17 @@ class McRegionWorldFolder(_BaseWorldFolder): """Represents a world save using the old McRegion format.""" type = "McRegion" extension = 'mcr' - chunkclass = chunk.Chunk - # chunkclass = chunk.McRegionChunk # TODO: change to McRegionChunk when done + chunkclass = chunk.McRegionChunk + class AnvilWorldFolder(_BaseWorldFolder): """Represents a world save using the new Anvil format.""" type = "Anvil" extension = 'mca' - chunkclass = chunk.Chunk - # chunkclass = chunk.AnvilChunk # TODO: change to AnvilChunk when done + chunkclass = chunk.AnvilChunk -class _WorldFolderFactory(): +class _WorldFolderFactory(object): """Factory class: instantiate the subclassses in order, and the first instance whose nonempty() method returns True is returned. If no nonempty() returns True, a UnknownWorldFormat exception is raised.""" @@ -216,7 +256,7 @@ def __call__(self, *args, **kwargs): wf = cls(*args, **kwargs) if wf.nonempty(): # Check if the world is non-empty return wf - raise UnknownWorldFormat("Empty world or unknown format: %r" % world_folder) + raise UnknownWorldFormat("Empty world or unknown format") WorldFolder = _WorldFolderFactory([AnvilWorldFolder, McRegionWorldFolder]) """ @@ -252,10 +292,16 @@ def expand(self,x,y,z): if self.maxz is None or z > self.maxz: self.maxz = z def lenx(self): + if self.maxx is None or self.minx is None: + return 0 return self.maxx-self.minx+1 def leny(self): + if self.maxy is None or self.miny is None: + return 0 return self.maxy-self.miny+1 def lenz(self): + if self.maxz is None or self.minz is None: + return 0 return self.maxz-self.minz+1 def __repr__(self): return "%s(%s,%s,%s,%s,%s,%s)" % (self.__class__.__name__,self.minx,self.maxx, diff --git a/progressbar/__init__.py b/progressbar/__init__.py new file mode 100644 index 0000000..89daf46 --- /dev/null +++ b/progressbar/__init__.py @@ -0,0 +1,49 @@ +#!/usr/bin/python +# -*- coding: utf-8 -*- +# +# progressbar - Text progress bar library for Python. +# Copyright (c) 2005 Nilton Volpato +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +"""Text progress bar library for Python. + +A text progress bar is typically used to display the progress of a long +running operation, providing a visual cue that processing is underway. + +The ProgressBar class manages the current progress, and the format of the line +is given by a number of widgets. A widget is an object that may display +differently depending on the state of the progress bar. There are three types +of widgets: + - a string, which always shows itself + + - a ProgressBarWidget, which may return a different value every time its + update method is called + + - a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it + expands to fill the remaining width of the line. + +The progressbar module is very easy to use, yet very powerful. It will also +automatically enable features like auto-resizing when the system supports it. +""" + +__author__ = 'Nilton Volpato' +__author_email__ = 'nilton.volpato@gmail.com' +__date__ = '2011-05-14' +__version__ = '2.5' + +from .compat import * +from .widgets import * +from .progressbar import * diff --git a/progressbar/compat.py b/progressbar/compat.py new file mode 100644 index 0000000..a39f4a1 --- /dev/null +++ b/progressbar/compat.py @@ -0,0 +1,44 @@ +# -*- coding: utf-8 -*- +# +# progressbar - Text progress bar library for Python. +# Copyright (c) 2005 Nilton Volpato +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +"""Compatibility methods and classes for the progressbar module.""" + + +# Python 3.x (and backports) use a modified iterator syntax +# This will allow 2.x to behave with 3.x iterators +try: + next +except NameError: + def next(iter): + try: + # Try new style iterators + return iter.__next__() + except AttributeError: + # Fallback in case of a "native" iterator + return iter.next() + + +# Python < 2.5 does not have "any" +try: + any +except NameError: + def any(iterator): + for item in iterator: + if item: return True + return False diff --git a/progressbar/progressbar.py b/progressbar/progressbar.py new file mode 100644 index 0000000..3baf530 --- /dev/null +++ b/progressbar/progressbar.py @@ -0,0 +1,305 @@ +# -*- coding: utf-8 -*- +# +# progressbar - Text progress bar library for Python. +# Copyright (c) 2005 Nilton Volpato +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +"""Main ProgressBar class.""" + +from __future__ import division + +import math +import os +import signal +import sys +import time + +try: + from fcntl import ioctl + from array import array + import termios +except ImportError: + pass + +from .compat import * # for: any, next +from . import widgets + + +class ProgressBar(object): + """The ProgressBar class which updates and prints the bar. + + A common way of using it is like: + >>> pbar = ProgressBar().start() + >>> for i in range(100): + ... # do something + ... pbar.update(i+1) + ... + >>> pbar.finish() + + You can also use a ProgressBar as an iterator: + >>> progress = ProgressBar() + >>> for i in progress(some_iterable): + ... # do something + ... + + Since the progress bar is incredibly customizable you can specify + different widgets of any type in any order. You can even write your own + widgets! However, since there are already a good number of widgets you + should probably play around with them before moving on to create your own + widgets. + + The term_width parameter represents the current terminal width. If the + parameter is set to an integer then the progress bar will use that, + otherwise it will attempt to determine the terminal width falling back to + 80 columns if the width cannot be determined. + + When implementing a widget's update method you are passed a reference to + the current progress bar. As a result, you have access to the + ProgressBar's methods and attributes. Although there is nothing preventing + you from changing the ProgressBar you should treat it as read only. + + Useful methods and attributes include (Public API): + - currval: current progress (0 <= currval <= maxval) + - maxval: maximum (and final) value + - finished: True if the bar has finished (reached 100%) + - start_time: the time when start() method of ProgressBar was called + - seconds_elapsed: seconds elapsed since start_time and last call to + update + - percentage(): progress in percent [0..100] + """ + + __slots__ = ('currval', 'fd', 'finished', 'last_update_time', + 'left_justify', 'maxval', 'next_update', 'num_intervals', + 'poll', 'seconds_elapsed', 'signal_set', 'start_time', + 'term_width', 'update_interval', 'widgets', '_time_sensitive', + '__iterable') + + _DEFAULT_MAXVAL = 100 + _DEFAULT_TERMSIZE = 80 + _DEFAULT_WIDGETS = [widgets.Percentage(), ' ', widgets.Bar()] + + def __init__(self, maxval=None, widgets=None, term_width=None, poll=1, + left_justify=True, fd=None): + """Initializes a progress bar with sane defaults.""" + + # Don't share a reference with any other progress bars + if widgets is None: + widgets = list(self._DEFAULT_WIDGETS) + + self.maxval = maxval + self.widgets = widgets + self.fd = fd if fd is not None else sys.stderr + self.left_justify = left_justify + + self.signal_set = False + if term_width is not None: + self.term_width = term_width + else: + try: + self._handle_resize() + signal.signal(signal.SIGWINCH, self._handle_resize) + self.signal_set = True + except (SystemExit, KeyboardInterrupt): raise + except: + self.term_width = self._env_size() + + self.__iterable = None + self._update_widgets() + self.currval = 0 + self.finished = False + self.last_update_time = None + self.poll = poll + self.seconds_elapsed = 0 + self.start_time = None + self.update_interval = 1 + self.next_update = 0 + + + def __call__(self, iterable): + """Use a ProgressBar to iterate through an iterable.""" + + try: + self.maxval = len(iterable) + except: + if self.maxval is None: + self.maxval = widgets.UnknownLength + + self.__iterable = iter(iterable) + return self + + + def __iter__(self): + return self + + + def __next__(self): + try: + value = next(self.__iterable) + if self.start_time is None: + self.start() + else: + self.update(self.currval + 1) + return value + except StopIteration: + if self.start_time is None: + self.start() + self.finish() + raise + + + # Create an alias so that Python 2.x won't complain about not being + # an iterator. + next = __next__ + + + def _env_size(self): + """Tries to find the term_width from the environment.""" + + return int(os.environ.get('COLUMNS', self._DEFAULT_TERMSIZE)) - 1 + + + def _handle_resize(self, signum=None, frame=None): + """Tries to catch resize signals sent from the terminal.""" + + h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2] + self.term_width = w + + + def percentage(self): + """Returns the progress as a percentage.""" + if self.maxval is widgets.UnknownLength: + return float("NaN") + if self.currval >= self.maxval: + return 100.0 + return (self.currval * 100.0 / self.maxval) if self.maxval else 100.00 + + percent = property(percentage) + + + def _format_widgets(self): + result = [] + expanding = [] + width = self.term_width + + for index, widget in enumerate(self.widgets): + if isinstance(widget, widgets.WidgetHFill): + result.append(widget) + expanding.insert(0, index) + else: + widget = widgets.format_updatable(widget, self) + result.append(widget) + width -= len(widget) + + count = len(expanding) + while count: + portion = max(int(math.ceil(width * 1. / count)), 0) + index = expanding.pop() + count -= 1 + + widget = result[index].update(self, portion) + width -= len(widget) + result[index] = widget + + return result + + + def _format_line(self): + """Joins the widgets and justifies the line.""" + + widgets = ''.join(self._format_widgets()) + + if self.left_justify: return widgets.ljust(self.term_width) + else: return widgets.rjust(self.term_width) + + + def _need_update(self): + """Returns whether the ProgressBar should redraw the line.""" + if self.currval >= self.next_update or self.finished: return True + + delta = time.time() - self.last_update_time + return self._time_sensitive and delta > self.poll + + + def _update_widgets(self): + """Checks all widgets for the time sensitive bit.""" + + self._time_sensitive = any(getattr(w, 'TIME_SENSITIVE', False) + for w in self.widgets) + + + def update(self, value=None): + """Updates the ProgressBar to a new value.""" + + if value is not None and value is not widgets.UnknownLength: + if (self.maxval is not widgets.UnknownLength + and not 0 <= value <= self.maxval): + + raise ValueError('Value out of range') + + self.currval = value + + + if not self._need_update(): return + if self.start_time is None: + raise RuntimeError('You must call "start" before calling "update"') + + now = time.time() + self.seconds_elapsed = now - self.start_time + self.next_update = self.currval + self.update_interval + self.fd.write(self._format_line() + '\r') + self.fd.flush() + self.last_update_time = now + + + def start(self): + """Starts measuring time, and prints the bar at 0%. + + It returns self so you can use it like this: + >>> pbar = ProgressBar().start() + >>> for i in range(100): + ... # do something + ... pbar.update(i+1) + ... + >>> pbar.finish() + """ + + if self.maxval is None: + self.maxval = self._DEFAULT_MAXVAL + + self.num_intervals = max(100, self.term_width) + self.next_update = 0 + + if self.maxval is not widgets.UnknownLength: + if self.maxval < 0: raise ValueError('Value out of range') + self.update_interval = self.maxval / self.num_intervals + + + self.start_time = self.last_update_time = time.time() + self.update(0) + + return self + + + def finish(self): + """Puts the ProgressBar bar in the finished state.""" + + if self.finished: + return + self.finished = True + self.update(self.maxval) + self.fd.write('\n') + if self.signal_set: + signal.signal(signal.SIGWINCH, signal.SIG_DFL) diff --git a/progressbar/widgets.py b/progressbar/widgets.py new file mode 100644 index 0000000..dd3c6ef --- /dev/null +++ b/progressbar/widgets.py @@ -0,0 +1,359 @@ +# -*- coding: utf-8 -*- +# +# progressbar - Text progress bar library for Python. +# Copyright (c) 2005 Nilton Volpato +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +"""Default ProgressBar widgets.""" + +from __future__ import division + +import datetime +import math + +try: + from abc import ABCMeta, abstractmethod +except ImportError: + AbstractWidget = object + abstractmethod = lambda fn: fn +else: + AbstractWidget = ABCMeta('AbstractWidget', (object,), {}) + +class UnknownLength: + pass + +def format_updatable(updatable, pbar): + if hasattr(updatable, 'update'): return updatable.update(pbar) + else: return updatable + + +class Widget(AbstractWidget): + """The base class for all widgets. + + The ProgressBar will call the widget's update value when the widget should + be updated. The widget's size may change between calls, but the widget may + display incorrectly if the size changes drastically and repeatedly. + + The boolean TIME_SENSITIVE informs the ProgressBar that it should be + updated more often because it is time sensitive. + """ + + TIME_SENSITIVE = False + __slots__ = () + + @abstractmethod + def update(self, pbar): + """Updates the widget. + + pbar - a reference to the calling ProgressBar + """ + + +class WidgetHFill(Widget): + """The base class for all variable width widgets. + + This widget is much like the \\hfill command in TeX, it will expand to + fill the line. You can use more than one in the same line, and they will + all have the same width, and together will fill the line. + """ + + @abstractmethod + def update(self, pbar, width): + """Updates the widget providing the total width the widget must fill. + + pbar - a reference to the calling ProgressBar + width - The total width the widget must fill + """ + + +class Timer(Widget): + """Widget which displays the elapsed seconds.""" + + __slots__ = ('format_string',) + TIME_SENSITIVE = True + + def __init__(self, format='Elapsed Time: %s'): + self.format_string = format + + @staticmethod + def format_time(seconds): + """Formats time as the string "HH:MM:SS".""" + + return str(datetime.timedelta(seconds=int(seconds))) + + + def update(self, pbar): + """Updates the widget to show the elapsed time.""" + + return self.format_string % self.format_time(pbar.seconds_elapsed) + + +class ETA(Timer): + """Widget which attempts to estimate the time of arrival.""" + + TIME_SENSITIVE = True + + def update(self, pbar): + """Updates the widget to show the ETA or total time when finished.""" + + if pbar.maxval is UnknownLength or pbar.currval == 0: + return 'ETA: --:--:--' + elif pbar.finished: + return 'Time: %s' % self.format_time(pbar.seconds_elapsed) + else: + elapsed = pbar.seconds_elapsed + eta = elapsed * pbar.maxval / pbar.currval - elapsed + return 'ETA: %s' % self.format_time(eta) + + +class AdaptiveETA(Timer): + """Widget which attempts to estimate the time of arrival. + + Uses a weighted average of two estimates: + 1) ETA based on the total progress and time elapsed so far + 2) ETA based on the progress as per the last 10 update reports + + The weight depends on the current progress so that to begin with the + total progress is used and at the end only the most recent progress is + used. + """ + + TIME_SENSITIVE = True + NUM_SAMPLES = 10 + + def _update_samples(self, currval, elapsed): + sample = (currval, elapsed) + if not hasattr(self, 'samples'): + self.samples = [sample] * (self.NUM_SAMPLES + 1) + else: + self.samples.append(sample) + return self.samples.pop(0) + + def _eta(self, maxval, currval, elapsed): + return elapsed * maxval / float(currval) - elapsed + + def update(self, pbar): + """Updates the widget to show the ETA or total time when finished.""" + if pbar.maxval is UnknownLength or pbar.currval == 0: + return 'ETA: --:--:--' + elif pbar.finished: + return 'Time: %s' % self.format_time(pbar.seconds_elapsed) + else: + elapsed = pbar.seconds_elapsed + currval1, elapsed1 = self._update_samples(pbar.currval, elapsed) + eta = self._eta(pbar.maxval, pbar.currval, elapsed) + if pbar.currval > currval1: + etasamp = self._eta(pbar.maxval - currval1, + pbar.currval - currval1, + elapsed - elapsed1) + weight = (pbar.currval / float(pbar.maxval)) ** 0.5 + eta = (1 - weight) * eta + weight * etasamp + return 'ETA: %s' % self.format_time(eta) + + +class FileTransferSpeed(Widget): + """Widget for showing the transfer speed (useful for file transfers).""" + + FMT = '%6.2f %s%s/s' + PREFIXES = ' kMGTPEZY' + __slots__ = ('unit',) + + def __init__(self, unit='B'): + self.unit = unit + + def update(self, pbar): + """Updates the widget with the current SI prefixed speed.""" + + if pbar.seconds_elapsed < 2e-6 or pbar.currval < 2e-6: # =~ 0 + scaled = power = 0 + else: + speed = pbar.currval / pbar.seconds_elapsed + power = int(math.log(speed, 1000)) + scaled = speed / 1000.**power + + return self.FMT % (scaled, self.PREFIXES[power], self.unit) + + +class AnimatedMarker(Widget): + """An animated marker for the progress bar which defaults to appear as if + it were rotating. + """ + + __slots__ = ('markers', 'curmark') + + def __init__(self, markers='|/-\\'): + self.markers = markers + self.curmark = -1 + + def update(self, pbar): + """Updates the widget to show the next marker or the first marker when + finished""" + + if pbar.finished: return self.markers[0] + + self.curmark = (self.curmark + 1) % len(self.markers) + return self.markers[self.curmark] + +# Alias for backwards compatibility +RotatingMarker = AnimatedMarker + + +class Counter(Widget): + """Displays the current count.""" + + __slots__ = ('format_string',) + + def __init__(self, format='%d'): + self.format_string = format + + def update(self, pbar): + return self.format_string % pbar.currval + + +class Percentage(Widget): + """Displays the current percentage as a number with a percent sign.""" + + def update(self, pbar): + return '%3.0f%%' % pbar.percentage() + + +class FormatLabel(Timer): + """Displays a formatted label.""" + + mapping = { + 'elapsed': ('seconds_elapsed', Timer.format_time), + 'finished': ('finished', None), + 'last_update': ('last_update_time', None), + 'max': ('maxval', None), + 'seconds': ('seconds_elapsed', None), + 'start': ('start_time', None), + 'value': ('currval', None) + } + + __slots__ = ('format_string',) + def __init__(self, format): + self.format_string = format + + def update(self, pbar): + context = {} + for name, (key, transform) in self.mapping.items(): + try: + value = getattr(pbar, key) + + if transform is None: + context[name] = value + else: + context[name] = transform(value) + except: pass + + return self.format_string % context + + +class SimpleProgress(Widget): + """Returns progress as a count of the total (e.g.: "5 of 47").""" + + __slots__ = ('sep',) + + def __init__(self, sep=' of '): + self.sep = sep + + def update(self, pbar): + if pbar.maxval is UnknownLength: + return '%d%s?' % (pbar.currval, self.sep) + return '%d%s%s' % (pbar.currval, self.sep, pbar.maxval) + + +class Bar(WidgetHFill): + """A progress bar which stretches to fill the line.""" + + __slots__ = ('marker', 'left', 'right', 'fill', 'fill_left') + + def __init__(self, marker='#', left='|', right='|', fill=' ', + fill_left=True): + """Creates a customizable progress bar. + + marker - string or updatable object to use as a marker + left - string or updatable object to use as a left border + right - string or updatable object to use as a right border + fill - character to use for the empty part of the progress bar + fill_left - whether to fill from the left or the right + """ + self.marker = marker + self.left = left + self.right = right + self.fill = fill + self.fill_left = fill_left + + + def update(self, pbar, width): + """Updates the progress bar and its subcomponents.""" + + left, marked, right = (format_updatable(i, pbar) for i in + (self.left, self.marker, self.right)) + + width -= len(left) + len(right) + # Marked must *always* have length of 1 + if pbar.maxval is not UnknownLength and pbar.maxval: + marked *= int(pbar.currval / pbar.maxval * width) + else: + marked = '' + + if self.fill_left: + return '%s%s%s' % (left, marked.ljust(width, self.fill), right) + else: + return '%s%s%s' % (left, marked.rjust(width, self.fill), right) + + +class ReverseBar(Bar): + """A bar which has a marker which bounces from side to side.""" + + def __init__(self, marker='#', left='|', right='|', fill=' ', + fill_left=False): + """Creates a customizable progress bar. + + marker - string or updatable object to use as a marker + left - string or updatable object to use as a left border + right - string or updatable object to use as a right border + fill - character to use for the empty part of the progress bar + fill_left - whether to fill from the left or the right + """ + self.marker = marker + self.left = left + self.right = right + self.fill = fill + self.fill_left = fill_left + + +class BouncingBar(Bar): + def update(self, pbar, width): + """Updates the progress bar and its subcomponents.""" + + left, marker, right = (format_updatable(i, pbar) for i in + (self.left, self.marker, self.right)) + + width -= len(left) + len(right) + + if pbar.finished: return '%s%s%s' % (left, width * marker, right) + + position = int(pbar.currval % (width * 2 - 1)) + if position > width: position = width * 2 - position + lpad = self.fill * (position - 1) + rpad = self.fill * (width - len(marker) - len(lpad)) + + # Swap if we want to bounce the other way + if not self.fill_left: rpad, lpad = lpad, rpad + + return '%s%s%s%s%s' % (left, lpad, marker, rpad, right) diff --git a/region-fixer.py b/region-fixer.py deleted file mode 100644 index 911f87d..0000000 --- a/region-fixer.py +++ /dev/null @@ -1,305 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -# -# Region Fixer. -# Fix your region files with a backup copy of your Minecraft world. -# Copyright (C) 2011 Alejandro Aguilera (Fenixin) -# https://github.com/Fenixin/Minecraft-Region-Fixer -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . -# - -from multiprocessing import freeze_support -from optparse import OptionParser, OptionGroup -from getpass import getpass -import sys - -import world -from scan import scan_regionset, scan_world -from interactive import interactive_loop -from util import entitle, is_bare_console, parse_world_list, parse_paths, parse_backup_list - -def delete_bad_chunks(options, scanned_obj): - """ Takes a scanned object (world object or regionset object) and - the options given to region-fixer, it deletes all the chunks with - problems iterating through all the possible problems. """ - print # a blank line - options_delete = [options.delete_corrupted, options.delete_wrong_located, options.delete_entities, options.delete_shared_offset] - deleting = zip(options_delete, world.CHUNK_PROBLEMS) - for delete, problem in deleting: - status = world.CHUNK_STATUS_TEXT[problem] - total = scanned_obj.count_chunks(problem) - if delete: - if total: - text = ' Deleting chunks with status: {0} '.format(status) - print "{0:#^60}".format(text) - counter = scanned_obj.remove_problematic_chunks(problem) - - print "\nDeleted {0} chunks with status: {1}".format(counter,status) - else: - print "No chunks to delete with status: {0}".format(status) - -def delete_bad_regions(options, scanned_obj): - """ Takes an scanned object (world object or regionset object) and - the options give to region-fixer, it deletes all the region files - with problems iterating through all the possible problems. """ - print # a blank line - options_delete = [options.delete_too_small] - deleting = zip(options_delete, world.REGION_PROBLEMS) - for delete, problem in deleting: - status = world.REGION_STATUS_TEXT[problem] - total = scanned_obj.count_regions(problem) - if delete: - if total: - text = ' Deleting regions with status: {0} '.format(status) - print "{0:#^60}".format(text) - counter = scanned_obj.remove_problematic_regions(problem) - - print "Deleted {0} regions with status: {1}".format(counter,status) - else: - print "No regions to delete with status: {0}".format(status) - -def main(): - - usage = 'usage: %prog [options] ... ...' - epilog = 'Copyright (C) 2011 Alejandro Aguilera (Fenixin) \ - https://github.com/Fenixin/Minecraft-Region-Fixer \ - This program comes with ABSOLUTELY NO WARRANTY; for details see COPYING.txt. This is free software, and you are welcome to redistribute it under certain conditions; see COPYING.txt for details.' - - parser = OptionParser(description='Script to check the integrity of Minecraft worlds and fix them when possible. It uses NBT by twoolie. Author: Alejandro Aguilera (Fenixin)',\ - prog = 'region-fixer', version='0.1.3', usage=usage, epilog=epilog) - - parser.add_option('--backups', '-b', help = 'List of backup directories of the Minecraft world to use to fix corrupted chunks and/or wrong located chunks. Warning! Region-Fixer is not going to check if it\'s the same world, be careful! This argument can be a comma separated list (but never with spaces between elements!). This option can be only used scanning one world.',\ - metavar = '', type = str, dest = 'backups', default = None) - - parser.add_option('--replace-corrupted','--rc', help = 'Tries to replace the corrupted chunks using the backup directories. This option can be only used scanning one world.',\ - default = False, dest = 'replace_corrupted', action='store_true') - - parser.add_option('--replace-wrong-located','--rw', help = 'Tries to replace the wrong located chunks using the backup directories. This option can be only used scanning one world.',\ - default = False, dest = 'replace_wrong_located', action='store_true') - - parser.add_option('--replace-entities','--re', help = 'Tries to replace the chunks with too many entities using the backup directories. This option can be only used scanning one world.',\ - default = False, dest = 'replace_entities', action='store_true') - - parser.add_option('--replace-shared-offset','--rs', help = 'Tries to replace the chunks with a shared offset using the backup directories. This option can be only used scanning one world.',\ - default = False, dest = 'replace_shared_offset', action='store_true') - - parser.add_option('--replace-too-small','--rt', help = 'Tries to replace the region files that are too small to be actually be a region file using the backup directories. This option can be only used scanning one world.',\ - default = False, dest = 'replace_too_small', action='store_true') - - parser.add_option('--delete-corrupted', '--dc', help = '[WARNING!] This option deletes! This option will delete all the corrupted chunks. Used with --replace-corrupted or --replace-wrong-located it will delete all the non-replaced chunks.',\ - action = 'store_true', default = False) - - parser.add_option('--delete-wrong-located', '--dw', help = '[WARNING!] This option deletes! The same as --delete-corrupted but for wrong located chunks',\ - action = 'store_true', default = False, dest='delete_wrong_located') - - parser.add_option('--delete-entities', '--de', help = '[WARNING!] This option deletes! This option deletes ALL the entities in chunks with more entities than --entity-limit (300 by default). In a Minecraft entities are mostly mobs and items dropped in the grond, items in chests and other stuff won\'t be touched. Read the README for more info. Region-Fixer will delete the entities while scanning so you can stop and resume the process',\ - action = 'store_true', default = False, dest = 'delete_entities') - - parser.add_option('--delete-shared-offset', '--ds', help = '[WARNING!] This option deletes! This option will delete all the chunk with status shared offset. It will remove the region header for the false chunk, note that you don\'t loos any chunk doing this.',\ - action = 'store_true', default = False, dest = 'delete_shared_offset') - - parser.add_option('--delete-too-small', '--dt', help = '[WARNING!] This option deletes! Removes any region files found to be too small to actually be a region file.',\ - dest ='delete_too_small', default = False, action = 'store_true') - - parser.add_option('--entity-limit', '--el', help = 'Specify the limit for the --delete-entities option (default = 300).',\ - dest = 'entity_limit', default = 300, action = 'store', type = int) - - parser.add_option('--processes', '-p', help = 'Set the number of workers to use for scanning. (defaulta = 1, not use multiprocessing at all)',\ - action = 'store', type = int, default = 1) - - parser.add_option('--verbose', '-v', help='Don\'t use a progress bar, instead print a line per scanned region file with results information. The letters mean c: corrupted; w: wrong located; t: total of chunksm; tme: too many entities problem',\ - action='store_true', default = False) - - parser.add_option('--interactive', '-i',help='Enter in interactive mode, where you can scan, see the problems, and fix them in a terminal like mode',\ - dest = 'interactive',default = False, action='store_true',) - - parser.add_option('--log', '-l',help='Saves a log of all the problems found in the spicifyed file. The log file contains all the problems found with this information: region file, chunk coordinates and problem. Use \'-\' as name to show the log at the end of the scan.',\ - type = str, default = None, dest = 'summary') - - (options, args) = parser.parse_args() - - if is_bare_console(): - print - print "Minecraft Region Fixer is a command line aplication, if you want to run it" - print "you need to open a command line (cmd.exe in the start menu in windows 7)." - print - getpass("Press enter to continue:") - return 1 - - # Args are world_paths and region files - if not args: - parser.error("No world paths or region files specified! Use --help for a complete list of options.") - - world_list, region_list = parse_paths(args) - - if not (world_list or region_list): - print ("Error: No worlds or region files to scan!") - return 1 - - # Check basic options compatibilities - any_chunk_replace_option = options.replace_corrupted or options.replace_wrong_located or options.replace_entities or options.replace_shared_offset - any_chunk_delete_option = options.delete_corrupted or options.delete_wrong_located or options.delete_entities or options.delete_shared_offset - any_region_replace_option = options.replace_too_small - any_region_delete_option = options.delete_too_small - - if options.interactive or options.summary: - if any_chunk_replace_option or any_region_replace_option: - parser.error("Can't use the options --replace-* , --delete-* and --log with --interactive. You can choose all this while in the interactive mode.") - - else: # not options.interactive - if options.backups: - if not any_chunk_replace_option and not any_region_replace_option: - parser.error("The option --backups needs at least one of the --replace-* options") - else: - if (len(region_list.regions) > 0): - parser.error("You can't use the replace options while scanning sparate region files. The input should be only one world and you intruduced {0} individual region files.".format(len(region_list.regions))) - elif (len(world_list) > 1): - parser.error("You can't use the replace options while scanning multiple worlds. The input should be only one world and you intruduced {0} worlds.".format(len(world_list))) - - if not options.backups and any_chunk_replace_option: - parser.error("The options --replace-* need the --backups option") - - if options.entity_limit < 0: - parser.error("The entity limit must be at least 0!") - - print "\nWelcome to Region Fixer!" - print "(version: {0})".format(parser.version) - - # do things with the option options args - if options.backups: # create a list of worlds containing the backups of the region files - backup_worlds = parse_backup_list(options.backups) - if not backup_worlds: - print "[WARNING] No valid backup directories found, won't fix any chunk." - else: - backup_worlds = [] - - - # The program starts - if options.interactive: - # TODO: WARNING, NEEDS CHANGES FOR WINDOWS. check while making the windows exe - c = interactive_loop(world_list, region_list, options, backup_worlds) - c.cmdloop() - - else: - summary_text = "" - # scan the separate region files - if len(region_list.regions) > 0: - print entitle("Scanning separate region files", 0) - scan_regionset(region_list, options) - - print region_list.generate_report(True) - - # delete chunks - delete_bad_chunks(options, region_list) - - # delete region files - delete_bad_regions(options, region_list) - - # verbose log - if options.summary: - summary_text += "\n" - summary_text += entitle("Separate region files") - summary_text += "\n" - t = region_list.summary() - if t: - summary_text += t - else: - summary_text += "No problems found.\n\n" - - # scan all the world folders - for world_obj in world_list: - print entitle(' Scanning world: {0} '.format(world_obj.get_name()),0) - - scan_world(world_obj, options) - - print world_obj.generate_report(standalone = True) - corrupted, wrong_located, entities_prob, shared_prob, total_chunks, too_small_region, unreadable_region, total_regions = world_obj.generate_report(standalone = False) - print - - # replace chunks - if backup_worlds and not len(world_list) > 1: - options_replace = [options.replace_corrupted, options.replace_wrong_located, options.replace_entities, options.replace_shared_offset] - replacing = zip(options_replace, world.CHUNK_PROBLEMS_ITERATOR) - for replace, (problem, status, arg) in replacing: - if replace: - total = world_obj.count_chunks(problem) - if total: - text = " Replacing chunks with status: {0} ".format(status) - print "{0:#^60}".format(text) - fixed = world_obj.replace_problematic_chunks(backup_worlds, problem, options) - print "\n{0} replaced of a total of {1} chunks with status: {2}".format(fixed, total, status) - else: print "No chunks to replace with status: {0}".format(status) - - elif any_chunk_replace_option and not backup_worlds: - print "Info: Won't replace any chunk." - print "No backup worlds found, won't replace any chunks/region files!" - elif any_chunk_replace_option and backup_worlds and len(world_list) > 1: - print "Info: Won't replace any chunk." - print "Can't use the replace options while scanning more than one world!" - - # replace region files - if backup_worlds and not len(world_list) > 1: - options_replace = [options.replace_too_small] - replacing = zip(options_replace, world.REGION_PROBLEMS_ITERATOR) - for replace, (problem, status, arg) in replacing: - if replace: - total = world_obj.count_regions(problem) - if total: - text = " Replacing regions with status: {0} ".format(status) - print "{0:#^60}".format(text) - fixed = world_obj.replace_problematic_regions(backup_worlds, problem, options) - print "\n{0} replaced of a total of {1} regions with status: {2}".format(fixed, total, status) - else: print "No region to replace with status: {0}".format(status) - - elif any_region_replace_option and not backup_worlds: - print "Info: Won't replace any regions." - print "No valid backup worlds found, won't replace any chunks/region files!" - print "Note: You probably inserted some backup worlds with the backup option but they are probably no valid worlds, the most common issue is wrong path." - elif any_region_replace_option and backup_worlds and len(world_list) > 1: - print "Info: Won't replace any regions." - print "Can't use the replace options while scanning more than one world!" - - # delete chunks - delete_bad_chunks(options, world_obj) - - # delete region files - delete_bad_regions(options, world_obj) - - # print a summary for this world - if options.summary: - summary_text += world_obj.summary() - - # verbose log text - if options.summary == '-': - print "\nPrinting log:\n" - print summary_text - elif options.summary != None: - try: - f = open(options.summary, 'w') - f.write(summary_text) - f.write('\n') - f.close() - print "Log file saved in \'{0}\'.".format(options.summary) - except: - print "Something went wrong while saving the log file!" - - return 0 - - -if __name__ == '__main__': - freeze_support() - value = main() - sys.exit(value) diff --git a/regionfixer.py b/regionfixer.py new file mode 100644 index 0000000..ba1baf0 --- /dev/null +++ b/regionfixer.py @@ -0,0 +1,606 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# +# Region Fixer. +# Fix your region files with a backup copy of your Minecraft world. +# Copyright (C) 2020 Alejandro Aguilera (Fenixin) +# https://github.com/Fenixin/Minecraft-Region-Fixer +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import argparse +from getpass import getpass +from multiprocessing import freeze_support +import sys + + +from regionfixer_core.bug_reporter import BugReporter +import regionfixer_core.constants as c +from regionfixer_core.interactive import InteractiveLoop +from regionfixer_core.scan import (console_scan_world, + console_scan_regionset, + ChildProcessException) +from regionfixer_core.util import entitle, is_bare_console +from regionfixer_core.version import version_string +from regionfixer_core import world + + + +def fix_bad_chunks(options, scanned_obj): + """ Fixes chunks that can be repaired. + + Inputs: + options -- argparse arguments, the whole argparse.ArgumentParser() object + scanned_obj -- this can be a RegionSet or World objects from world.py + + Returns nothing. + + It will fix the chunks as requested by options and modify the RegionSet and World objects + with the new fixed chunks. + """ + + print("") + total = scanned_obj.count_chunks(c.CHUNK_MISSING_ENTITIES_TAG) + problem = c.CHUNK_MISSING_ENTITIES_TAG + status = c.CHUNK_STATUS_TEXT[c.CHUNK_MISSING_ENTITIES_TAG] + # In the same order as in FIXABLE_CHUNK_PROBLEMS + options_fix = [options.fix_corrupted, + options.fix_missing_tag, + options.fix_wrong_located] + fixing = list(zip(options_fix, c.FIXABLE_CHUNK_PROBLEMS)) + for fix, problem in fixing: + status = c.CHUNK_STATUS_TEXT[problem] + total = scanned_obj.count_chunks(problem) + if fix: + if total: + text = ' Repairing chunks with status: {0} '.format(status) + print(("\n{0:#^60}".format(text))) + counter = scanned_obj.fix_problematic_chunks(problem) + print(("\nRepaired {0} chunks with status: {1}".format(counter, + status))) + else: + print(("No chunks to fix with status: {0}".format(status))) + + +def delete_bad_chunks(options, scanned_obj): + """ Takes a scanned object and deletes all the bad chunks. + + Inputs: + options -- argparse arguments, the whole argparse.ArgumentParser() object + scanned_obj -- this can be a RegionSet or World objects from world.py + + Returns nothing. + + This function will deletes all the chunks with problems + iterating through all the possible problems and using the + options given. + """ + + print("") + # In the same order as in CHUNK_PROBLEMS + options_delete = [options.delete_corrupted, + options.delete_wrong_located, + options.delete_entities, + options.delete_shared_offset, + options.delete_missing_tag] + deleting = list(zip(options_delete, c.CHUNK_PROBLEMS)) + for delete, problem in deleting: + status = c.CHUNK_STATUS_TEXT[problem] + total = scanned_obj.count_chunks(problem) + if delete: + if total: + text = ' Deleting chunks with status: {0} '.format(status) + print(("\n{0:#^60}".format(text))) + counter = scanned_obj.remove_problematic_chunks(problem) + print(("\nDeleted {0} chunks with status: {1}".format(counter, + status))) + else: + print(("No chunks to delete with status: {0}".format(status))) + + +def delete_bad_regions(options, scanned_obj): + """ Takes a scanned object and deletes all bad region files. + + Inputs: + options -- argparse arguments, the whole argparse.ArgumentParser() object + scanned_obj -- this can be a RegionSet or World objects from world.py + + Returns nothing. + + Takes an scanned object (World object or RegionSet object) and + the options given to region-fixer and it deletes all the region files + with problems iterating through all the possible problems. + """ + + print("") + options_delete = [options.delete_too_small] + deleting = list(zip(options_delete, c.REGION_PROBLEMS)) + for delete, problem in deleting: + status = c.REGION_STATUS_TEXT[problem] + total = scanned_obj.count_regions(problem) + if delete: + if total: + text = ' Deleting regions with status: {0} '.format(status) + print(("{0:#^60}".format(text))) + counter = scanned_obj.remove_problematic_regions(problem) + print(("Deleted {0} regions with status: {1}".format(counter, + status))) + else: + print(("No regions to delete with status: {0}".format(status))) + + +def main(): + usage = ('%(prog)s [options] ' + ' ... ...') + epilog = ('Copyright (C) 2020 Alejandro Aguilera (Fenixin)\n' + 'https://github.com/Fenixin/Minecraft-Region-Fixer\n' + 'This program comes with ABSOLUTELY NO WARRANTY; for ' + 'details see COPYING.txt. This is free software, and you ' + 'are welcome to redistribute it under certain conditions; ' + 'see COPYING.txt for details.') + + parser = argparse.ArgumentParser(description=('Program to check the integrity of ' + 'Minecraft worlds and fix them when ' + 'possible. It uses NBT by twoolie. ' + 'Author: Alejandro Aguilera (Fenixin)'), + prog='region_fixer', + usage=usage, + epilog=epilog) + + parser.add_argument('--text-file-input', + '--tf', + help=('Path to a text file with a list of world folders and region ' + 'files. One line per element, wildcards can be used, empty lines' + 'will be ignored and # can be used at the start of a line as comment' + '. These will be treated the same as adding paths to command input.'), + metavar='', + type=str, + dest='text_file_input', + default=None) + + parser.add_argument('--backups', + '-b', + help=('List of backup directories of the Minecraft world ' + 'to use to fix corrupted chunks and/or wrong located ' + 'chunks. Warning! Region-Fixer is not going to check if' + 'it\'s the same world, be careful! This argument can be a' + ' comma separated list (but never with spaces between ' + 'elements!). This option can be only used scanning one ' + 'world.'), + metavar='', + type=str, + dest='backups', + default=None) + + for solvable_status in c.CHUNK_PROBLEMS_SOLUTIONS: + if c.CHUNK_SOLUTION_REMOVE in c.CHUNK_PROBLEMS_SOLUTIONS[solvable_status]: + parser.add_argument('--delete-' + c.CHUNK_PROBLEMS_ARGS[solvable_status], + '--d' + c.CHUNK_PROBLEMS_ABBR[solvable_status], + help='[WARNING!] This option deletes! Delete all chunks with ' + 'status: ' + c.CHUNK_STATUS_TEXT[solvable_status], + action='store_true', + default=False) + if c.CHUNK_SOLUTION_REPLACE in c.CHUNK_PROBLEMS_SOLUTIONS[solvable_status]: + parser.add_argument('--replace-' + c.CHUNK_PROBLEMS_ARGS[solvable_status], + '--r' + c.CHUNK_PROBLEMS_ABBR[solvable_status], + help='This option can be only used while scanning one world. ' + 'Try to replace the problematic chunks with the status "{0}" ' + 'using backup directories.'.format(c.CHUNK_STATUS_TEXT[solvable_status]), + action='store_true', + default=False) + if c.CHUNK_SOLUTION_RELOCATE_USING_DATA in c.CHUNK_PROBLEMS_SOLUTIONS[solvable_status]: + parser.add_argument('--relocate-' + c.CHUNK_PROBLEMS_ARGS[solvable_status], + '--rl' + c.CHUNK_PROBLEMS_ABBR[solvable_status], + help='This option can be only used while scanning one world. ' + 'Try to replace the problematic chunks with the status "{0}" ' + 'using backup directories.'.format(c.CHUNK_STATUS_TEXT[solvable_status]), + action='store_true', + default=False) + + for solvable_status in c.REGION_PROBLEMS_SOLUTIONS: + if c.REGION_SOLUTION_REMOVE in c.REGION_PROBLEMS_SOLUTIONS[solvable_status]: + parser.add_argument('--delete-' + c.REGION_PROBLEMS_ARGS[solvable_status], + '--d' + c.REGION_PROBLEMS_ABBR[solvable_status], + help='[WARNING!] This option deletes! Delete all chunks with ' + 'status: ' + c.REGION_STATUS_TEXT[solvable_status], + action='store_true', + default=False) + if c.REGION_SOLUTION_REPLACE in c.REGION_PROBLEMS_SOLUTIONS[solvable_status]: + parser.add_argument('--replace-' + c.REGION_PROBLEMS_ARGS[solvable_status], + '--r' + c.REGION_PROBLEMS_ABBR[solvable_status], + help='This option can be only used while scanning one world. ' + 'Try to replace the problematic chunks with the status "{0}" ' + 'using backup directories.'.format(c.REGION_STATUS_TEXT[solvable_status]), + action='store_true', + default=False) + + parser.add_argument('--delete-entities', + '--de', + help='[WARNING!] This option deletes! Delete ALL ' + 'the entities in chunks with more entities than ' + '--entity-limit (300 by default). In a Minecraft ' + 'entities are mostly mobs and items dropped in the ' + 'ground, items in chests and other stuff won\'t be ' + 'touched. Read the README for more info. Region-Fixer ' + 'will delete the entities while scanning so you can ' + 'stop and resume the process', + action='store_true', + default=False, + dest='delete_entities') + + parser.add_argument('--fix-corrupted', + '--fc', + help='Try to fix chunks that are corrupted by extracting as much ' + 'information as possible', + dest='fix_corrupted', + default=False, + action='store_true') + + parser.add_argument('--fix-missing-tag', + '--fm', + help='Fix chunks that have the Entities tag missing. This will add ' + 'the missing tag.', + dest='fix_missing_tag', + default=False, + action='store_true') + + parser.add_argument('--fix-wrong-located', + '--fw', + help='Fix chunks that are wrong located. This will save them in the coordinates ' + 'stored in their data.', + dest='fix_wrong_located', + default=False, + action='store_true') + + parser.add_argument('--entity-limit', + '--el', + help='Specify the limit for the --delete-entities option ' + '(default = 300).', + dest='entity_limit', + default=300, + action='store', + type=int) + + parser.add_argument('--processes', + '-p', + help='Set the number of workers to use for scanning. (default ' + '= 1, not use multiprocessing at all)', + action='store', + type=int, + default=1) + + status_abbr = "" + for status in c.CHUNK_PROBLEMS: + status_abbr += "{0}: {1}; ".format(c.CHUNK_PROBLEMS_ABBR[status], c.CHUNK_STATUS_TEXT[status]) + parser.add_argument('--verbose', + '-v', + help=('Don\'t use a progress bar, instead print a line per ' + 'scanned file with results information. The ' + 'letters mean:\n') + status_abbr, + action='store_true', + default=False) + + #=========================================================================== + # parser.add_argument('--interactive', + # '-i', + # help='Enter in interactive mode, where you can scan, see the ' + # 'problems, and fix them in a terminal like mode', + # dest='interactive', + # default=False, + # action='store_true', ) + #=========================================================================== + + parser.add_argument('--log', + '-l', + help='Save a log of all the problems found in the specified ' + 'file. The log file contains all the problems found with ' + 'this information: region file, chunk coordinates and ' + 'problem. Use \'-\' as name to show the log at the end ' + 'of the scan.', + type=str, + default=None, + dest='summary') + + parser.add_argument('paths', + help='List with world or region paths', + nargs='*') + + args = parser.parse_args() + + if sys.version_info[0] != 3: + print("") + print("Minecraft Region Fixer only works with python 3.x") + print(("(And you just tried to run it in python {0})".format(sys.version))) + print("") + return c.RV_CRASH + + if is_bare_console(): + print("") + print("Minecraft Region Fixer is a command line application and \n" + "you have just double clicked it. If you really want to run \n" + "the command line interface you have to use a command prompt.\n" + "Run cmd.exe in the run window.\n\n") + print("") + getpass("Press enter to continue:") + return c.RV_CRASH + + # First, read paths from file + path_lines = [] + if args.text_file_input: + try: + tf = open(args.text_file_input, 'r') + path_lines = tf.readlines() + tmp = [] + # Process it + for i in range(len(path_lines)): + # Remove end of lines characters + line = path_lines[i].replace('\n', '') + # Remove comment lines and empty lines + if line and "#" not in line: + tmp.append(line) + + path_lines = tmp + tf.close() + + except: + print("Something went wrong while reading the text file input!") + + # Parse all the paths, from text file and command input + world_list, regionset = world.parse_paths(args.paths + path_lines) + + # print greetings an version number + print("\nWelcome to Region Fixer!") + print(("(v {0})".format(version_string))) + + # Check if there are valid worlds to scan + if not (world_list or regionset): + print('Error: No worlds or region files to scan! Use ' + '--help for a complete list of options.') + return c.RV_NOTHING_TO_SCAN + + # Check basic options compatibilities + any_chunk_replace_option = args.replace_corrupted or \ + args.replace_wrong_located or \ + args.replace_entities or \ + args.replace_shared_offset + any_region_replace_option = args.replace_too_small + + if False or args.summary: # removed interactive mode args.interactive + if any_chunk_replace_option or any_region_replace_option: + parser.error('Error: Can\'t use the options --replace-* , --delete-* with ' + '--log') + + else: + # Not options.interactive + if args.backups: + if not any_chunk_replace_option and not any_region_replace_option: + parser.error('Error: The option --backups needs at least one of the ' + '--replace-* options') + else: + if len(regionset) > 0: + parser.error('Error: You can\'t use the replace options while scanning ' + 'separate region files. The input should be only one ' + 'world and you introduced {0} individual region ' + 'files.'.format(len(regionset))) + elif len(world_list) > 1: + parser.error('Error: You can\'t use the replace options while scanning ' + 'multiple worlds. The input should be only one ' + 'world and you introduced {0} ' + 'worlds.'.format(len(world_list))) + + if not args.backups and any_chunk_replace_option: + parser.error("Error: The options --replace-* need the --backups option") + + if args.entity_limit < 0: + parser.error("Error: The entity limit must be at least 0!") + + # Do things with the option options args + # Create a list of worlds containing the backups of the region files + if args.backups: + backup_worlds = world.parse_backup_list(args.backups) + if not backup_worlds: + print('[WARNING] No valid backup directories found, won\'t fix ' + 'any chunk.') + else: + backup_worlds = [] + + # The scanning process starts + found_problems_in_regionsets = False + found_problems_in_worlds = False + if False: # removed args.interactive + ci = InteractiveLoop(world_list, regionset, args, backup_worlds) + ci.cmdloop() + return c.RV_OK + else: + summary_text = "" + # Scan the separate region files + + if len(regionset) > 0: + + console_scan_regionset(regionset, args.processes, args.entity_limit, + args.delete_entities, args.verbose) + print((regionset.generate_report(True))) + + # Delete chunks + delete_bad_chunks(args, regionset) + + # Delete region files + delete_bad_regions(args, regionset) + + # fix chunks + fix_bad_chunks(args, regionset) + + # Verbose log + if args.summary: + summary_text += "\n" + summary_text += entitle("Separate region files") + summary_text += "\n" + t = regionset.summary() + if t: + summary_text += t + else: + summary_text += "No problems found.\n\n" + + # Check if problems have been found + if regionset.has_problems: + found_problems_in_regionsets = True + + # scan all the world folders + + for w in world_list: + w_name = w.get_name() + print((entitle(' Scanning world: {0} '.format(w_name), 0))) + + console_scan_world(w, args.processes, args.entity_limit, + args.delete_entities, args.verbose) + + print("") + print((entitle('Scan results for: {0}'.format(w_name), 0))) + print((w.generate_report(True))) + print("") + + # Replace chunks + if backup_worlds and len(world_list) <= 1: + del_ent = args.delete_entities + ent_lim = args.entity_limit + options_replace = [args.replace_corrupted, + args.replace_wrong_located, + args.replace_entities, + args.replace_shared_offset] + replacing = list(zip(options_replace, c.CHUNK_PROBLEMS_ITERATOR)) + for replace, (problem, status, arg) in replacing: + if replace: + total = w.count_chunks(problem) + if total: + text = " Replacing chunks with status: {0} ".format(status) + print(("{0:#^60}".format(text))) + fixed = w.replace_problematic_chunks(backup_worlds, problem, ent_lim, del_ent) + print(("\n{0} replaced of a total of {1} chunks with status: {2}".format(fixed, total, status))) + else: + print(("No chunks to replace with status: {0}".format(status))) + + elif any_chunk_replace_option and not backup_worlds: + print("Info: Won't replace any chunk.") + print("No backup worlds found, won't replace any chunks/region files!") + elif any_chunk_replace_option and backup_worlds and len(world_list) > 1: + print("Info: Won't replace any chunk.") + print("Can't use the replace options while scanning more than one world!") + + # replace region files + if backup_worlds and len(world_list) <= 1: + del_ent = args.delete_entities + ent_lim = args.entity_limit + options_replace = [args.replace_too_small] + replacing = list(zip(options_replace, c.REGION_PROBLEMS_ITERATOR)) + for replace, (problem, status, arg) in replacing: + if replace: + total = w.count_regions(problem) + if total: + text = " Replacing regions with status: {0} ".format(status) + print(("{0:#^60}".format(text))) + fixed = w.replace_problematic_regions(backup_worlds, problem, ent_lim, del_ent) + print(("\n{0} replaced of a total of {1} regions with status: {2}".format(fixed, total, status))) + else: + print(("No region to replace with status: {0}".format(status))) + + elif any_region_replace_option and not backup_worlds: + print("Info: Won't replace any regions.") + print("No valid backup worlds found, won't replace any chunks/region files!") + print("Note: You probably inserted some backup worlds with the backup option but they are probably no valid worlds, the most common issue is wrong path.") + elif any_region_replace_option and backup_worlds and len(world_list) > 1: + print("Info: Won't replace any regions.") + print("Can't use the replace options while scanning more than one world!") + + # delete chunks + delete_bad_chunks(args, w) + + # delete region files + delete_bad_regions(args, w) + + # fix chunks + fix_bad_chunks(args, w) + + # print a summary for this world + if args.summary: + summary_text += w.summary() + + # check if problems have been found + if w.has_problems: + found_problems_in_worlds = True + + # verbose log text + if args.summary == '-': + print("\nPrinting log:\n") + print(summary_text) + elif args.summary is not None: + try: + f = open(args.summary, 'w') + f.write(summary_text) + f.write('\n') + f.close() + print(("Log file saved in \'{0}\'.".format(args.summary))) + except: + print("Something went wrong while saving the log file!") + + if found_problems_in_regionsets or found_problems_in_worlds: + return c.RV_BAD_WORLD + + return c.RV_OK + + +if __name__ == '__main__': + ERROR_MSG = "\n\nOps! Something went really wrong and regionfixer crashed.\n" + QUESTION_TEXT = ('Do you want to send an anonymous bug report to the region fixer ftp?\n' + '(Answering no will print the bug report)') + had_exception = False + auto_reported = False + value = 0 + + try: + freeze_support() + value = main() + + except SystemExit as e: + # sys.exit() was called within the program + had_exception = False + value = e.code + + except ChildProcessException as e: + had_exception = True + print(ERROR_MSG) + bug_sender = BugReporter(e.printable_traceback) + # auto_reported = bug_sender.ask_and_send(QUESTION_TEXT) + bug_report = bug_sender.error_str + value = c.RV_CRASH + + except Exception as e: + had_exception = True + print(ERROR_MSG) + # Traceback will be taken in init + bug_sender = BugReporter() + # auto_reported = bug_sender.ask_and_send(QUESTION_TEXT) + bug_report = bug_sender.error_str + value = c.RV_CRASH + + finally: + if had_exception and not auto_reported: + print("") + print("Bug report:") + print("") + print(bug_report) + elif had_exception and auto_reported: + print("Bug report uploaded successfully") + sys.exit(value) diff --git a/regionfixer_core/bug_reporter.py b/regionfixer_core/bug_reporter.py new file mode 100644 index 0000000..6fd2332 --- /dev/null +++ b/regionfixer_core/bug_reporter.py @@ -0,0 +1,99 @@ +''' +Created on 16/09/2014 + +@author: Alejandro +''' + +import sys +import ftplib +import datetime +from io import StringIO +from .util import query_yes_no, get_str_from_traceback + + +SERVER = 'regionfixer.no-ip.org' +USER = 'regionfixer_bugreporter' +PASSWORD = 'supersecretpassword' +BUGREPORTS_DIR = 'bugreports' + + +class BugReporter(object): + ''' + Class to report bugs to region fixer ftp. + + You can init it without arguments and it will extract the traceback + directly from sys.exc_info(). The traceback will be formated and + uploaded as a text file. + Or you can init it using an error string (error_str). The string + will be uploaded as a text file. + ''' + + def __init__(self, error_str=None, server=SERVER, + user=USER, password=PASSWORD): + ''' + Constructor + ''' + if error_str: + self.error_file_obj = self._get_fileobj_from_str(error_str) + else: + (ty, value, tb) = sys.exc_info() + self.error_file_obj = self._get_fileobj_from_tb(ty, value, tb) + self.server = server + self.user = user + self.password = password + + self._exception = None + + def _get_fileobj_from_tb(self, ty, value, tb): + ''' Return a file obj from a traceback object. ''' + f = StringIO(get_str_from_traceback(ty, value, tb)) + f.seek(0) + return f + + def _get_fileobj_from_str(self, error_str): + ''' Return a file object from a string. ''' + f = StringIO(error_str) + f.seek(0) + return f + + @property + def error_str(self): + ''' Return the string that is currently ready for upload. ''' + self.error_file_obj.seek(0) + s = self.error_file_obj.read() + self.error_file_obj.seek(0) + return s + + @property + def exception_str(self): + ''' Return the exception caused by uploading the file. ''' + return self._exception.message + + def ask_and_send(self, question_text): + ''' Query the user yes/no to send the file and send it. ''' + if query_yes_no(question_text): + return self.send() + + def send(self): + ''' Send the file to the ftp. + + If an exception is thrown, you can retrieve it at + exception_str. + ''' + try: + s = ftplib.FTP(self.server, self.user, + self.password) + + s.cwd(BUGREPORTS_DIR) + + error_name = str(datetime.datetime.now()) + + s.storlines("STOR " + error_name, self.error_file_obj) + s.quit() + return True + except Exception as e: + # TODO: prints shouldn't be here! + print("Couldn't send the bug report!") + self._exception = e + print(e) + return False diff --git a/regionfixer_core/constants.py b/regionfixer_core/constants.py new file mode 100644 index 0000000..46951a7 --- /dev/null +++ b/regionfixer_core/constants.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# +# Region Fixer. +# Fix your region files with a backup copy of your Minecraft world. +# Copyright (C) 2020 Alejandro Aguilera (Fenixin) +# https://github.com/Fenixin/Minecraft-Region-Fixer +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + + + +################ +# Return values +################ + +RV_OK = 0 # world scanned and no problems found +RV_CRASH = 1 # crash or end unexpectedly +RV_NOTHING_TO_SCAN = 20 # no files/worlds to scan +# RV_WRONG_COMMAND = 2 # the command line used is wrong and region fixer didn't execute. argparse uses this value by default +RV_BAD_WORLD = 3 # scan completed successfully but problems have been found in the scan + + + + +# -------------- +# Chunk related: +# -------------- +# Used to mark the status of chunks: +CHUNK_NOT_CREATED = -1 +CHUNK_OK = 0 +CHUNK_CORRUPTED = 1 +CHUNK_WRONG_LOCATED = 2 +CHUNK_TOO_MANY_ENTITIES = 3 +CHUNK_SHARED_OFFSET = 4 +CHUNK_MISSING_ENTITIES_TAG = 5 + +# Chunk statuses +CHUNK_STATUSES = [CHUNK_NOT_CREATED, + CHUNK_OK, + CHUNK_CORRUPTED, + CHUNK_WRONG_LOCATED, + CHUNK_TOO_MANY_ENTITIES, + CHUNK_SHARED_OFFSET, + CHUNK_MISSING_ENTITIES_TAG] + +# Status that are considered problems +CHUNK_PROBLEMS = [CHUNK_CORRUPTED, + CHUNK_WRONG_LOCATED, + CHUNK_TOO_MANY_ENTITIES, + CHUNK_SHARED_OFFSET, + CHUNK_MISSING_ENTITIES_TAG] + +# Text describing each chunk status +CHUNK_STATUS_TEXT = {CHUNK_NOT_CREATED: "Not created", + CHUNK_OK: "OK", + CHUNK_CORRUPTED: "Corrupted", + CHUNK_WRONG_LOCATED: "Wrong located", + CHUNK_TOO_MANY_ENTITIES: "Too many entities", + CHUNK_SHARED_OFFSET: "Sharing offset", + CHUNK_MISSING_ENTITIES_TAG: "Missing Entities tag" + } + +# arguments used in the options +CHUNK_PROBLEMS_ARGS = {CHUNK_CORRUPTED: 'corrupted', + CHUNK_WRONG_LOCATED: 'wrong-located', + CHUNK_TOO_MANY_ENTITIES: 'entities', + CHUNK_SHARED_OFFSET: 'shared-offset', + CHUNK_MISSING_ENTITIES_TAG: 'missing_tag' + } + +# used in some places where there is less space +CHUNK_PROBLEMS_ABBR = {CHUNK_CORRUPTED: 'c', + CHUNK_WRONG_LOCATED: 'w', + CHUNK_TOO_MANY_ENTITIES: 'tme', + CHUNK_SHARED_OFFSET: 'so', + CHUNK_MISSING_ENTITIES_TAG: 'mt' + } + +# Dictionary with possible solutions for the chunks problems, +# used to create options dynamically +# The possible solutions right now are: +CHUNK_SOLUTION_REMOVE = 51 +CHUNK_SOLUTION_REPLACE = 52 +CHUNK_SOLUTION_REMOVE_ENTITIES = 53 +CHUNK_SOLUTION_RELOCATE_USING_DATA = 54 + +CHUNK_PROBLEMS_SOLUTIONS = {CHUNK_CORRUPTED: [CHUNK_SOLUTION_REMOVE, CHUNK_SOLUTION_REPLACE], + CHUNK_WRONG_LOCATED: [CHUNK_SOLUTION_REMOVE, CHUNK_SOLUTION_REPLACE, CHUNK_SOLUTION_RELOCATE_USING_DATA], + CHUNK_TOO_MANY_ENTITIES: [CHUNK_SOLUTION_REMOVE_ENTITIES, CHUNK_SOLUTION_REPLACE], + CHUNK_SHARED_OFFSET: [CHUNK_SOLUTION_REMOVE, CHUNK_SOLUTION_REPLACE], + CHUNK_MISSING_ENTITIES_TAG: [CHUNK_SOLUTION_REMOVE, CHUNK_SOLUTION_REPLACE]} + +# chunk problems that can be fixed (so they don't need to be removed or replaced) +FIXABLE_CHUNK_PROBLEMS = [CHUNK_CORRUPTED, CHUNK_MISSING_ENTITIES_TAG, CHUNK_WRONG_LOCATED] + +# list with problem, status-text, problem arg tuples +CHUNK_PROBLEMS_ITERATOR = [] +for problem in CHUNK_PROBLEMS: + CHUNK_PROBLEMS_ITERATOR.append((problem, + CHUNK_STATUS_TEXT[problem], + CHUNK_PROBLEMS_ARGS[problem])) + +# Used to know where to look in a chunk status tuple +TUPLE_NUM_ENTITIES = 0 +TUPLE_STATUS = 1 + + + + +# --------------- +# Region related: +# --------------- +# Used to mark the status of region files: +REGION_OK = 100 +REGION_TOO_SMALL = 101 +REGION_UNREADABLE = 102 +REGION_UNREADABLE_PERMISSION_ERROR = 103 + +# Region statuses +REGION_STATUSES = [REGION_OK, + REGION_TOO_SMALL, + REGION_UNREADABLE, + REGION_UNREADABLE_PERMISSION_ERROR] + +# Text describing each region status used to list all the problem at the end of the scan +REGION_STATUS_TEXT = {REGION_OK: "OK", + REGION_TOO_SMALL: "Too small", + REGION_UNREADABLE: "Unreadable IOError", + # This status differentiates IOError from a file that you don't have permission to access + # TODO: It would be better to open region files only in write mode when needed + REGION_UNREADABLE_PERMISSION_ERROR: "Permission error" + } + +# Status that are considered problems +REGION_PROBLEMS = [REGION_TOO_SMALL, + REGION_UNREADABLE, + REGION_UNREADABLE_PERMISSION_ERROR] + +# arguments used in the options +REGION_PROBLEMS_ARGS = {REGION_TOO_SMALL: 'too_small', + REGION_UNREADABLE: 'unreadable', + REGION_UNREADABLE_PERMISSION_ERROR: 'permission_error' + } + +# used in some places where there is less space +REGION_PROBLEMS_ABBR = {REGION_TOO_SMALL: 'ts', + REGION_UNREADABLE: 'ur', + REGION_UNREADABLE_PERMISSION_ERROR: 'pe' + } + +# Dictionary with possible solutions for the region problems, +# used to create options dynamically +# The possible solutions right now are: +REGION_SOLUTION_REMOVE = 151 +REGION_SOLUTION_REPLACE = 152 + +REGION_PROBLEMS_SOLUTIONS = {REGION_TOO_SMALL: [REGION_SOLUTION_REMOVE, REGION_SOLUTION_REPLACE]} + +# list with problem, status-text, problem arg tuples +REGION_PROBLEMS_ITERATOR = [] +for problem in REGION_PROBLEMS: + try: + REGION_PROBLEMS_ITERATOR.append((problem, + REGION_STATUS_TEXT[problem], + REGION_PROBLEMS_ARGS[problem])) + except KeyError: + pass + + + +# ------------------ +# Data file related: +# ------------------ +# Used to mark the status of data files: +DATAFILE_OK = 200 +DATAFILE_UNREADABLE = 201 + +# Data files statuses +DATAFILE_STATUSES = [DATAFILE_OK, + DATAFILE_UNREADABLE] + +# Status that are considered problems +DATAFILE_PROBLEMS = [DATAFILE_UNREADABLE] + +# Text describing each chunk status +DATAFILE_STATUS_TEXT = {DATAFILE_OK: "OK", + DATAFILE_UNREADABLE: "The data file cannot be read" + } + +# arguments used in the options +DATAFILE_PROBLEMS_ARGS = {DATAFILE_OK: 'OK', + DATAFILE_UNREADABLE: 'unreadable' + } + +# used in some places where there is less space +DATAFILE_PROBLEM_ABBR = {DATAFILE_OK: 'ok', + DATAFILE_UNREADABLE: 'ur' + } + +# Dictionary with possible solutions for the chunks problems, +# used to create options dynamically +# The possible solutions right now are: +DATAFILE_SOLUTION_REMOVE = 251 + +DATAFILE_PROBLEMS_SOLUTIONS = {DATAFILE_UNREADABLE: [DATAFILE_SOLUTION_REMOVE]} + +# list with problem, status-text, problem arg tuples +DATAFILE_PROBLEMS_ITERATOR = [] +for problem in DATAFILE_PROBLEMS: + DATAFILE_PROBLEMS_ITERATOR.append((problem, + DATAFILE_STATUS_TEXT[problem], + DATAFILE_PROBLEMS_ARGS[problem])) + +CHUNK_PROBLEMS_ITERATOR = [] +for problem in CHUNK_PROBLEMS: + CHUNK_PROBLEMS_ITERATOR.append((problem, + CHUNK_STATUS_TEXT[problem], + CHUNK_PROBLEMS_ARGS[problem])) + +# Dimension names: +DIMENSION_NAMES = {"": "Overworld", + "DIM1": "The End", + "DIM-1": "Nether" + } + +# Region files types +LEVEL_DIR = "region" +POI_DIR = "poi" +ENTITIES_DIR = "entities" +REGION_TYPES_NAMES = {LEVEL_DIR: ("level/region", "Level/Region"), + POI_DIR: ("POIs", "POIs"), + ENTITIES_DIR: ("entities", "Entities" ) + } diff --git a/regionfixer_core/interactive.py b/regionfixer_core/interactive.py new file mode 100644 index 0000000..e64789f --- /dev/null +++ b/regionfixer_core/interactive.py @@ -0,0 +1,537 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# +# Region Fixer. +# Fix your region files with a backup copy of your Minecraft world. +# Copyright (C) 2020 Alejandro Aguilera (Fenixin) +# https://github.com/Fenixin/Minecraft-Region-Fixer +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + + +from cmd import Cmd + +import regionfixer_core.constants as c +from regionfixer_core import world +from regionfixer_core.scan import console_scan_world, console_scan_regionset + + +class InteractiveLoop(Cmd): + def __init__(self, world_list, regionset, options, backup_worlds): + Cmd.__init__(self) + self.world_list = world_list + self.regionset = regionset + self.world_names = [str(i.name) for i in self.world_list] + # if there's only one world use it + if len(self.world_list) == 1 and len(self.regionset) == 0: + self.current = world_list[0] + elif len(self.world_list) == 0 and len(self.regionset) > 0: + self.current = self.regionset + else: + self.current = None + self.options = options + self.backup_worlds = backup_worlds + self.prompt = "#-> " + self.intro = ("Minecraft Region-Fixer interactive mode.\n(Use tab to " + "autocomplete. Type help for a list of commands.)\n") + + # Possible args for chunks stuff + possible_args = "" + first = True + for i in list(c.CHUNK_PROBLEMS_ARGS.values()) + ['all']: + if not first: + possible_args += ", " + possible_args += i + first = False + self.possible_chunk_args_text = possible_args + + # Possible args for region stuff + possible_args = "" + first = True + for i in list(c.REGION_PROBLEMS_ARGS.values()) + ['all']: + if not first: + possible_args += ", " + possible_args += i + first = False + self.possible_region_args_text = possible_args + + ################################################# + # Do methods + ################################################# + def do_set(self, arg): + """ Command to change some options and variables in interactive + mode """ + args = arg.split() + if len(args) > 2: + print("Error: too many parameters.") + elif len(args) == 0: + print("Write \'help set\' to see a list of all possible variables") + else: + if args[0] == "entity-limit": + if len(args) == 1: + print("entity-limit = {0}".format(self.options.entity_limit)) + else: + try: + if int(args[1]) >= 0: + self.options.entity_limit = int(args[1]) + print("entity-limit = {0}".format(args[1])) + print("Updating chunk status...") + self.current.rescan_entities(self.options) + else: + print("Invalid value. Valid values are positive integers and zero") + except ValueError: + print("Invalid value. Valid values are positive integers and zero") + + elif args[0] == "workload": + + if len(args) == 1: + if self.current: + print("Current workload:\n{0}\n".format(self.current.__str__())) + print("List of possible worlds and region-sets (determined by the command used to run region-fixer):") + number = 1 + for w in self.world_list: + print(" ### world{0} ###".format(number)) + number += 1 + # add a tab and print + for i in w.__str__().split("\n"): + print("\t" + i) + print() + print(" ### regionset ###") + for i in self.regionset.__str__().split("\n"): + print("\t" + i) + print("\n(Use \"set workload world1\" or name_of_the_world or regionset to choose one)") + + else: + a = args[1] + if len(a) == 6 and a[:5] == "world" and int(a[-1]) >= 1: + # get the number and choos the correct world from the list + number = int(args[1][-1]) - 1 + try: + self.current = self.world_list[number] + print("workload = {0}".format(self.current.world_path)) + except IndexError: + print("This world is not in the list!") + elif a in self.world_names: + for w in self.world_list: + if w.name == args[1]: + self.current = w + print("workload = {0}".format(self.current.world_path)) + break + else: + print("This world name is not on the list!") + elif args[1] == "regionset": + if len(self.regionset): + self.current = self.regionset + print("workload = set of region files") + else: + print("The region set is empty!") + else: + print("Invalid world number, world name or regionset.") + + elif args[0] == "processes": + if len(args) == 1: + print("processes = {0}".format(self.options.processes)) + else: + try: + if int(args[1]) > 0: + self.options.processes = int(args[1]) + print("processes = {0}".format(args[1])) + else: + print("Invalid value. Valid values are positive integers.") + except ValueError: + print("Invalid value. Valid values are positive integers.") + + elif args[0] == "verbose": + if len(args) == 1: + print("verbose = {0}".format(str(self.options.verbose))) + else: + if args[1] == "True": + self.options.verbose = True + print("verbose = {0}".format(args[1])) + elif args[1] == "False": + self.options.verbose = False + print("verbose = {0}".format(args[1])) + else: + print("Invalid value. Valid values are True and False.") + else: + print("Invalid argument! Write \'help set\' to see a list of valid variables.") + + def do_summary(self, arg): + """ Prints a summary of all the problems found in the region + files. """ + if len(arg) == 0: + if self.current: + if self.current.scanned: + text = self.current.generate_report(True) + if text: + print(text) + else: + print("No problems found!") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + else: + print("No world/region-set is set! Use \'set workload\' to set a world/regionset to work with.") + else: + print("This command doesn't use any arguments.") + + def do_current_workload(self, arg): + """ Prints the info of the current workload """ + if len(arg) == 0: + if self.current: + print(self.current) + else: + print("No world/region-set is set! Use \'set workload\' to set a world/regionset to work with.") + else: + print("This command doesn't use any arguments.") + + def do_scan(self, arg): + """ Scans the current workload. """ + # TODO: what about scanning while deleting entities as done in non-interactive mode? + # this would need an option to choose which of the two methods use + o = self.options + if len(arg.split()) > 0: + print("Error: too many parameters.") + else: + if self.current: + if isinstance(self.current, world.World): + self.current = world.World(self.current.path) + console_scan_world(self.current, o.processes, + o.entity_limit, o.delete_entities, + o.verbose) + elif isinstance(self.current, world.RegionSet): + print("\n{0:-^60}".format(' Scanning region files ')) + console_scan_regionset(self.current, o.processes, + o.entity_limit, o.delete_entities, + o.verbose) + else: + print("No world set! Use \'set workload\'") + + def do_count_chunks(self, arg): + """ Counts the number of chunks with the given problem and + prints the result """ + if self.current and self.current.scanned: + if len(arg.split()) == 0: + print("Possible counters are: {0}".format(self.possible_chunk_args_text)) + elif len(arg.split()) > 1: + print("Error: too many parameters.") + else: + if arg in list(c.CHUNK_PROBLEMS_ARGS.values()) or arg == 'all': + total = self.current.count_chunks(None) + for problem, status_text, a in c.CHUNK_PROBLEMS_ITERATOR: + if arg == 'all' or arg == a: + n = self.current.count_chunks(problem) + print("Chunks with status \'{0}\': {1}".format(status_text, n)) + print("Total chunks: {0}".format(total)) + else: + print("Unknown counter.") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_count_regions(self, arg): + """ Counts the number of regions with the given problem and + prints the result """ + if self.current and self.current.scanned: + if len(arg.split()) == 0: + print("Possible counters are: {0}".format(self.possible_region_args_text)) + elif len(arg.split()) > 1: + print("Error: too many parameters.") + else: + if arg in list(c.REGION_PROBLEMS_ARGS.values()) or arg == 'all': + total = self.current.count_regions(None) + for problem, status_text, a in c.REGION_PROBLEMS_ITERATOR: + if arg == 'all' or arg == a: + n = self.current.count_regions(problem) + print("Regions with status \'{0}\': {1}".format(status_text, n)) + print("Total regions: {0}".format(total)) + else: + print("Unknown counter.") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_count_all(self, arg): + """ Print all the counters for chunks and regions. """ + if self.current and self.current.scanned: + if len(arg.split()) > 0: + print("This command doesn't requiere any arguments") + else: + print("{0:#^60}".format("Chunk problems:")) + self.do_count_chunks('all') + print("\n") + print("{0:#^60}".format("Region problems:")) + self.do_count_regions('all') + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_remove_entities(self, arg): + if self.current and self.current.scanned: + if len(arg.split()) > 0: + print("Error: too many parameters.") + else: + print("WARNING: This will delete all the entities in the chunks that have more entities than entity-limit, make sure you know what entities are!.\nAre you sure you want to continue? (yes/no):") + answer = input() + if answer == 'yes': + counter = self.current.remove_entities() + print("Deleted {0} entities.".format(counter)) + if counter: + self.current.scanned = False + self.current.rescan_entities(self.options) + elif answer == 'no': + print("Ok!") + else: + print("Invalid answer, use \'yes\' or \'no\' the next time!.") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_remove_chunks(self, arg): + if self.current and self.current.scanned: + if len(arg.split()) == 0: + print("Possible arguments are: {0}".format(self.possible_chunk_args_text)) + elif len(arg.split()) > 1: + print("Error: too many parameters.") + else: + if arg in list(c.CHUNK_PROBLEMS_ARGS.values()) or arg == 'all': + for problem, status_text, a in c.CHUNK_PROBLEMS_ITERATOR: + if arg == 'all' or arg == a: + n = self.current.remove_problematic_chunks(problem) + if n: + self.current.scanned = False + print("Removed {0} chunks with status \'{1}\'.\n".format(n, status_text)) + else: + print("Unknown argument.") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_replace_chunks(self, arg): + el = self.options.entity_limit + de = self.options.delete_entities + if self.current and self.current.scanned: + if len(arg.split()) == 0: + print("Possible arguments are: {0}".format(self.possible_chunk_args_text)) + elif len(arg.split()) > 1: + print("Error: too many parameters.") + else: + if arg in list(c.CHUNK_PROBLEMS_ARGS.values()) or arg == 'all': + for problem, status_text, a in c.CHUNK_PROBLEMS_ITERATOR: + if arg == 'all' or arg == a: + n = self.current.replace_problematic_chunks(self.backup_worlds, problem, el, de) + if n: + self.current.scanned = False + print("\nReplaced {0} chunks with status \'{1}\'.".format(n, status_text)) + else: + print("Unknown argument.") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_replace_regions(self, arg): + el = self.options.entity_limit + de = self.options.delete_entities + if self.current and self.current.scanned: + if len(arg.split()) == 0: + print("Possible arguments are: {0}".format(self.possible_region_args_text)) + elif len(arg.split()) > 1: + print("Error: too many parameters.") + else: + if arg in list(c.REGION_PROBLEMS_ARGS.values()) or arg == 'all': + for problem, status_text, a in c.REGION_PROBLEMS_ITERATOR: + if arg == 'all' or arg == a: + n = self.current.replace_problematic_regions(self.backup_worlds, problem, el, de) + if n: + self.current.scanned = False + print("\nReplaced {0} regions with status \'{1}\'.".format(n, status_text)) + else: + print("Unknown argument.") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_remove_regions(self, arg): + if self.current and self.current.scanned: + if len(arg.split()) == 0: + print("Possible arguments are: {0}".format(self.possible_region_args_text)) + elif len(arg.split()) > 1: + print("Error: too many parameters.") + else: + if arg in list(c.REGION_PROBLEMS_ARGS.values()) or arg == 'all': + for problem, status_text, a in c.REGION_PROBLEMS_ITERATOR: + if arg == 'all' or arg == a: + n = self.current.remove_problematic_regions(problem) + if n: + self.current.scanned = False + print("\nRemoved {0} regions with status \'{1}\'.".format(n, status_text)) + else: + print("Unknown argument.") + else: + print("The world hasn't be scanned (or it needs a rescan). Use \'scan\' to scan it.") + + def do_quit(self, arg): + print("Quitting.") + return True + + def do_exit(self, arg): + print("Exiting.") + return True + + def do_EOF(self, arg): + print("Quitting.") + return True + + ################################################# + # Complete methods + ################################################# + def complete_arg(self, text, possible_args): + l = [] + for arg in possible_args: + if text in arg and arg.find(text) == 0: + l.append(arg + " ") + return l + + def complete_set(self, text, line, begidx, endidx): + if "workload " in line: + # return the list of world names plus 'regionset' plus a list of world1, world2... + possible_args = tuple(self.world_names) + ('regionset',) + tuple(['world' + str(i + 1) for i in range(len(self.world_names))]) + elif 'verbose ' in line: + possible_args = ('True', 'False') + else: + possible_args = ('entity-limit', 'verbose', 'processes', 'workload') + return self.complete_arg(text, possible_args) + + def complete_count_chunks(self, text, line, begidx, endidx): + possible_args = list(c.CHUNK_PROBLEMS_ARGS.values()) + ['all'] + return self.complete_arg(text, possible_args) + + def complete_remove_chunks(self, text, line, begidx, endidx): + possible_args = list(c.CHUNK_PROBLEMS_ARGS.values()) + ['all'] + return self.complete_arg(text, possible_args) + + def complete_replace_chunks(self, text, line, begidx, endidx): + possible_args = list(c.CHUNK_PROBLEMS_ARGS.values()) + ['all'] + return self.complete_arg(text, possible_args) + + def complete_count_regions(self, text, line, begidx, endidx): + possible_args = list(c.REGION_PROBLEMS_ARGS.values()) + ['all'] + return self.complete_arg(text, possible_args) + + def complete_remove_regions(self, text, line, begidx, endidx): + possible_args = list(c.REGION_PROBLEMS_ARGS.values()) + ['all'] + return self.complete_arg(text, possible_args) + + def complete_replace_regions(self, text, line, begidx, endidx): + possible_args = list(c.REGION_PROBLEMS_ARGS.values()) + ['all'] + return self.complete_arg(text, possible_args) + + ################################################# + # Help methods + ################################################# + # TODO sería una buena idea poner un artículo de ayuda de como usar el programa en un caso típico. + # TODO: the help texts need a normalize + def help_set(self): + print("\nSets some variables used for the scan in interactive mode. " + "If you run this command without an argument for a variable " + "you can see the current state of the variable. You can set:\n" + " verbose\n" + "If True prints a line per scanned region file instead of " + "showing a progress bar.\n" + " entity-limit\n" + "If a chunk has more than this number of entities it will be " + "added to the list of chunks with too many entities problem.\n" + " processes" + "Number of cores used while scanning the world.\n" + " workload\n" + "If you input a few worlds you can choose wich one will be " + "scanned using this command.\n") + + def help_current_workload(self): + print("\nPrints information of the current region-set/world. This will be the region-set/world to scan and fix.\n") + + def help_scan(self): + print("\nScans the current world set or the region set.\n") + + def help_count_chunks(self): + print("\n Prints out the number of chunks with the given status. For example") + print("\'count corrupted\' prints the number of corrupted chunks in the world.") + print() + print("Possible status are: {0}\n".format(self.possible_chunk_args_text)) + + def help_remove_entities(self): + print("\nRemove all the entities in chunks that have more than entity-limit entities.") + print() + print("This chunks are the ones with status \'too many entities\'.\n") + + def help_remove_chunks(self): + print("\nRemoves bad chunks with the given problem.") + print() + print("Please, be careful, when used with the status too-many-entities this will") + print("REMOVE THE CHUNKS with too many entities problems, not the entities.") + print("To remove only the entities see the command remove_entities.") + print() + print("For example \'remove_chunks corrupted\' this will remove corrupted chunks.") + print() + print("Possible status are: {0}\n".format(self.possible_chunk_args_text)) + print() + + def help_replace_chunks(self): + print("\nReplaces bad chunks with the given status using the backups directories.") + print() + print("Exampe: \"replace_chunks corrupted\"") + print() + print("this will replace the corrupted chunks with the given backups.") + print() + print("Possible status are: {0}\n".format(self.possible_chunk_args_text)) + print() + print("Note: after replacing any chunks you have to rescan the world.\n") + + def help_count_regions(self): + print("\n Prints out the number of regions with the given status. For example ") + print("\'count_regions too-small\' prints the number of region with \'too-small\' status.") + print() + print("Possible status are: {0}\n".format(self.possible_region_args_text)) + + def help_remove_regions(self): + print("\nRemoves regions with the given status.") + print() + print("Example: \'remove_regions too-small\'") + print() + print("this will remove the region files with status \'too-small\'.") + print() + print("Possible status are: {0}".format(self.possible_region_args_text)) + print() + print("Note: after removing any regions you have to rescan the world.\n") + + def help_replace_regions(self): + print("\nReplaces regions with the given status.") + print() + print("Example: \"replace_regions too-small\"") + print() + print("this will try to replace the region files with status \'too-small\'") + print("with the given backups.") + print() + print("Possible status are: {0}".format(self.possible_region_args_text)) + print() + print("Note: after replacing any regions you have to rescan the world.\n") + + def help_summary(self): + print("\nPrints a summary of all the problems found in the current workload.\n") + + def help_quit(self): + print("\nQuits interactive mode, exits region-fixer. Same as \'EOF\' and \'exit\' commands.\n") + + def help_EOF(self): + print("\nQuits interactive mode, exits region-fixer. Same as \'quit\' and \'exit\' commands\n") + + def help_exit(self): + print("\nQuits interactive mode, exits region-fixer. Same as \'quit\' and \'EOF\' commands\n") + + def help_help(self): + print("Prints help help.") diff --git a/progressbar.py b/regionfixer_core/progressbar.py similarity index 100% rename from progressbar.py rename to regionfixer_core/progressbar.py diff --git a/regionfixer_core/scan.py b/regionfixer_core/scan.py new file mode 100644 index 0000000..53d62d3 --- /dev/null +++ b/regionfixer_core/scan.py @@ -0,0 +1,1057 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# +# Region Fixer. +# Fix your region files with a backup copy of your Minecraft world. +# Copyright (C) 2020 Alejandro Aguilera (Fenixin) +# https://github.com/Fenixin/Minecraft-Region-Fixer +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + + +import sys +import logging +import multiprocessing +from os.path import split, abspath, join +from time import sleep, time +from copy import copy +from traceback import extract_tb + +import nbt.region as region +import nbt.nbt as nbt +from nbt.nbt import MalformedFileError +from nbt.region import (ChunkDataError, + ChunkHeaderError, + RegionHeaderError, + InconceivedChunk) + +from progressbar import ProgressBar, Bar, AdaptiveETA, SimpleProgress + +import regionfixer_core.constants as c +from regionfixer_core.util import entitle +from regionfixer_core import world + + + +logging.basicConfig(filename=None, level=logging.CRITICAL) + + +class ChildProcessException(Exception): + """ Raised when a child process has problems. + + Inputs: + - partial_scanned_file -- ScannedObject from world.py partially filled with + the results of the scan + - exc_type -- Type of the exception being handled, extracted from sys.exc_info() + - exc_class -- The exception instance, extracted from sys.exc_info() + - tb_text -- The traceback text, extracted from traceback object from sys.exc_info() + + Stores all the info given by sys.exc_info() and the scanned file object which is + probably partially filled. + + """ + #TODO: not sure about the tb_text argument is that. + def __init__(self, partial_scanned_file, exc_type, exc_class, tb_text): + self.scanned_file = partial_scanned_file + self.exc_type = exc_type + self.exc_class = exc_class + self.tb_text = tb_text + + @property + def printable_traceback(self): + """ Returns a nice printable traceback. + + This traceback reports: + - The file that was being scanned + - The type and class of exception + - The text of the traceback + + It uses a lot of asteriks as indentation to ensure it doesn't mix with + the main process traceback. + + """ + + text = "" + scanned_file = self.scanned_file + text += "*" * 10 + "\n" + text += "*** Exception while scanning:" + "\n" + text += "*** " + str(scanned_file.filename) + "\n" + text += "*" * 10 + "\n" + text += "*** Printing the child's traceback:" + "\n" + text += "*** Exception:" + str(self.exc_type) + str(self.exc_class) + "\n" + for tb in self.tb_text: + text += "*" * 10 + "\n" + text += "*** File {0}, line {1}, in {2} \n*** {3}".format(*tb) + text += "\n" + "*" * 10 + "\n" + + return text + + def save_error_log(self, filename='error.log'): + """ Save the error in filename, return the path. + + Keyword argument: + - filename -- Name of the file to write the error log. + + Return: + - error_log_path -- Path where the error log was saved. + + """ + + f = open(filename, 'w') + error_log_path = abspath(f.name) + filename = self.scanned_file.filename + f.write("Error while scanning: {0}\n".format(filename)) + f.write(self.printable_traceback) + f.write('\n') + f.close() + + return error_log_path + + +def multiprocess_scan_data(data): + """ Does the multithread stuff for scan_data """ + # Protect everything so an exception will be returned from the worker + try: + result = scan_data(data) + multiprocess_scan_data.q.put(result) + except KeyboardInterrupt as e: + raise e + except: + except_type, except_class, tb = sys.exc_info() + s = (data, (except_type, except_class, extract_tb(tb))) + multiprocess_scan_data.q.put(s) + + +def multiprocess_scan_regionfile(region_file): + """ Does the multithread stuff for scan_region_file """ + # Protect everything so an exception will be returned from the worker + try: + r = region_file + entity_limit = multiprocess_scan_regionfile.entity_limit + remove_entities = multiprocess_scan_regionfile.remove_entities + # call the normal scan_region_file with this parameters + r = scan_region_file(r, entity_limit, remove_entities) + multiprocess_scan_regionfile.q.put(r) + except KeyboardInterrupt as e: + raise e + except: + except_type, except_class, tb = sys.exc_info() + s = (region_file, (except_type, except_class, extract_tb(tb))) + multiprocess_scan_regionfile.q.put(s) + + +def _mp_data_pool_init(d): + """ Function to initialize the multiprocessing in scan_dataset. + + Inputs: + - d -- Dictionary containing the information to copy to the function of the child process. + + This function adds the queue to each of the child processes objects. This queue + is used to get the results from the child process. + + """ + + assert isinstance(d, dict) + assert 'queue' in d + multiprocess_scan_data.q = d['queue'] + + +def _mp_regionset_pool_init(d): + """ Function to initialize the multiprocessing in scan_regionset. + + Inputs: + - d -- Dictionary containing the information to copy to the function of the child process. + + This function adds the queue to each of the child processes objects. This queue + is used to get the results from the child process. + + """ + + assert isinstance(d, dict) + assert 'regionset' in d + assert 'queue' in d + assert 'entity_limit' in d + assert 'remove_entities' in d + multiprocess_scan_regionfile.regionset = d['regionset'] + multiprocess_scan_regionfile.q = d['queue'] + multiprocess_scan_regionfile.entity_limit = d['entity_limit'] + multiprocess_scan_regionfile.remove_entities = d['remove_entities'] + + +class AsyncScanner: + """ Class to derive all the scanner classes from. + + Inputs: + - data_structure -- Is one of the objects in world: DataSet, RegionSet + - processes -- Integer with the number of child processes to use for the scan + - scan_function -- Function used to scan the data + - init_args -- These are the initialization arguments passed to __init__ + - _mp_init_function -- Function used to initialize the child processes + + To implement a scanner you have to override: + update_str_last_scanned() + + It's imperative to use try-finally to call terminate at the end of the run, + if not processes will be hanging in the background for all eternity. + + """ + + def __init__(self, data_structure, processes, scan_function, init_args, + _mp_init_function): + """ Init the scanner """ + assert isinstance(data_structure, world.DataSet) + self.data_structure = data_structure + self.list_files_to_scan = data_structure._get_list() + self.processes = processes + self.scan_function = scan_function + + # Queue used by processes to pass results + self.queue = multiprocessing.SimpleQueue() + init_args.update({'queue': self.queue}) + # NOTE TO SELF: initargs doesn't handle kwargs, only args! + # Pass a dict with all the args + self.pool = multiprocessing.Pool(processes=processes, + initializer=_mp_init_function, + initargs=(init_args,)) + + # Recommended time to sleep between polls for results + self.SCAN_START_SLEEP_TIME = 0.001 + self.SCAN_MIN_SLEEP_TIME = 1e-6 + self.SCAN_MAX_SLEEP_TIME = 0.1 + self.scan_sleep_time = self.SCAN_START_SLEEP_TIME + self.queries_without_results = 0 + self.last_time = time() + self.MIN_QUERY_NUM = 1 + self.MAX_QUERY_NUM = 5 + + # Holds a friendly string with the name of the last file scanned + self._str_last_scanned = None + + def scan(self): + """ Launch the child processes and scan all the files. """ + + logging.debug("########################################################") + logging.debug("########################################################") + logging.debug("Starting scan in: %s", str(self)) + logging.debug("########################################################") + logging.debug("########################################################") + # Tests indicate that smaller amount of jobs per worker make all type + # of scans faster + jobs_per_worker = 5 + # jobs_per_worker = max(1, total_files // self.processes + self._results = self.pool.map_async(self.scan_function, + self.list_files_to_scan, + jobs_per_worker) + + # No more tasks to the pool, exit the processes once the tasks are done + self.pool.close() + + # See method + self._str_last_scanned = "" + + def get_last_result(self): + """ Return results of last file scanned. """ + + q = self.queue + ds = self.data_structure + if not q.empty(): + d = q.get() + if isinstance(d, tuple): + self.raise_child_exception(d) + # Copy it to the father process + ds._replace_in_data_structure(d) + ds._update_counts(d) + self.update_str_last_scanned(d) + # Got result! Reset it! + self.queries_without_results = 0 + return d + else: + # Count amount of queries without result + self.queries_without_results += 1 + return None + + def terminate(self): + """ Terminate the pool, this will exit no matter what. + """ + self.pool.terminate() + + def raise_child_exception(self, exception_tuple): + """ Raises a ChildProcessException. + + Inputs: + - exception_tuple -- Tuple containing all the information about the exception + of the child process. + + """ + + e = exception_tuple + raise ChildProcessException(e[0], e[1][0], e[1][1], e[1][2]) + + def update_str_last_scanned(self): + """ Updates the string that represents the last file scanned. """ + raise NotImplementedError + + def sleep(self): + """ Sleep waiting for results. + + This method will adjust automatically the sleep time. It will sleep less + when results arrive faster and more when they arrive slower. + + """ + + # If the query number is outside of our range... + if not ((self.queries_without_results < self.MAX_QUERY_NUM) & + (self.queries_without_results > self.MIN_QUERY_NUM)): + # ... increase or decrease it to optimize queries + if self.queries_without_results < self.MIN_QUERY_NUM: + self.scan_sleep_time *= 0.5 + elif self.queries_without_results > self.MAX_QUERY_NUM: + self.scan_sleep_time *= 2.0 + # and don't go farther than max/min + if self.scan_sleep_time > self.SCAN_MAX_SLEEP_TIME: + logging.debug("Setting sleep time to MAX") + self.scan_sleep_time = self.SCAN_MAX_SLEEP_TIME + elif self.scan_sleep_time < self.SCAN_MIN_SLEEP_TIME: + logging.debug("Setting sleep time to MIN") + self.scan_sleep_time = self.SCAN_MIN_SLEEP_TIME + + # Log how it's going + logging.debug("") + logging.debug("Nº of queries without result: %s", str(self.queries_without_results)) + logging.debug("Current sleep time: %s", str(self.scan_sleep_time)) + logging.debug("Time between calls to sleep(): %s", str(time() - self.last_time)) + self.last_time = time() + + # Sleep, let the other processes do their job + sleep(self.scan_sleep_time) + + @property + def str_last_scanned(self): + """ A friendly string with last scanned result. """ + return self._str_last_scanned if self._str_last_scanned \ + else "Scanning..." + + @property + def finished(self): + """ Return True if the scan has finished. + + It checks if the queue is empty and if the results are ready. + + """ + + return self._results.ready() and self.queue.empty() + + @property + def results(self): + """ Yield all the results from the scan. + + This is the simpler method to control the scanning process, + but also the most sloppy. If you want to closely control the + scan process (for example cancel the process in the middle, + whatever is happening) use get_last_result(). + + Usage: + for result in scanner.results: + # do things + + """ + + q = self.queue + T = self.SCAN_WAIT_TIME + while not q.empty() or not self.finished: + sleep(T) + if not q.empty(): + d = q.get() + if isinstance(d, tuple): + self.raise_child_exception(d) + # Overwrite it in the data dict + self.replace_in_data_structure(d) + yield d + + def __len__(self): + return len(self.data_structure) + + +class AsyncDataScanner(AsyncScanner): + """ Scan a DataFileSet and fill the data structure. + + Inputs: + - data_structure -- A DataFileSet from world.py containing the files to scan + - processes -- An integer with the number of child processes to use + + """ + + def __init__(self, data_structure, processes): + scan_function = multiprocess_scan_data + init_args = {} + _mp_init_function = _mp_data_pool_init + + AsyncScanner.__init__(self, data_structure, processes, scan_function, + init_args, _mp_init_function) + + # Recommended time to sleep between polls for results + self.scan_wait_time = 0.0001 + + def update_str_last_scanned(self, data): + self._str_last_scanned = data.filename + + +class AsyncRegionsetScanner(AsyncScanner): + """ Scan a RegionSet and fill the data structure. + + Inputs: + - data_structure -- A RegionSet from world.py containing the files to scan + - processes -- An integer with the number of child processes to use + - entity_limit -- An integer, threshold of entities for a chunk to be considered + with too many entities + - remove_entities -- A boolean, defaults to False, to remove the entities whilel + scanning. This is really handy because opening chunks with + too many entities for scanning can take minutes. + + """ + + def __init__(self, regionset, processes, entity_limit, + remove_entities=False): + assert isinstance(regionset, world.DataSet) + + scan_function = multiprocess_scan_regionfile + _mp_init_function = _mp_regionset_pool_init + + init_args = {} + init_args['regionset'] = regionset + init_args['processes'] = processes + init_args['entity_limit'] = entity_limit + init_args['remove_entities'] = remove_entities + + AsyncScanner.__init__(self, regionset, processes, scan_function, + init_args, _mp_init_function) + + # Recommended time to sleep between polls for results + self.scan_wait_time = 0.001 + + def update_str_last_scanned(self, r): + self._str_last_scanned = self.data_structure.get_name() + ": " + r.filename + + +class AsyncWorldRegionScanner: + """ Wrapper around the calls of AsyncScanner the whole world. + + Inputs: + - world_obj -- A World object from world.py + - processes -- An integer with the number of child processes to use + - entity_limit -- An integer, threshold of entities for a chunk to be considered + with too many entities + - remove_entities -- A boolean, defaults to False, to remove the entities while + scanning. This is really handy because opening chunks with + too many entities for scanning can take minutes. + + This class is just a wrapper around AsyncRegionsetScanner to scan all the region sets + of the world. + + + """ + + def __init__(self, world_obj, processes, entity_limit, + remove_entities=False): + + self._world_obj = world_obj + self.processes = processes + self.entity_limit = entity_limit + self.remove_entities = remove_entities + + self.regionsets = copy(world_obj.regionsets) + + self._current_regionset = None + self._str_last_scanned = None + + # Holds a friendly string with the name of the last file scanned + self.scan_wait_time = 0.001 + + def sleep(self): + """ Sleep waiting for results. + + This method will sleep less when results arrive faster and + more when they arrive slower. See AsyncScanner.sleep(). + + """ + + self._current_regionset.sleep() + + def scan(self): + """ Scan and fill the given regionset. """ + + cr = AsyncRegionsetScanner(self.regionsets.pop(0), + self.processes, + self.entity_limit, + self.remove_entities) + self._current_regionset = cr + cr.scan() + + # See method + self._str_last_scanned = "" + + def get_last_result(self): + """ Return results of last region file scanned. + + If there are left no scanned region files return None. The + ScannedRegionFile returned is the same instance in the regionset, + don't modify it or you will modify the regionset results. + + This method is better if you want to closely control the scan + process. + + """ + + cr = self._current_regionset + + if cr is not None: + if not cr.finished: + r = cr.get_last_result() + self._str_last_scanned = cr.str_last_scanned + return r + elif self.regionsets: + self.scan() + return None + else: + return None + else: + return None + + def terminate(self): + """ Terminates scan of the current RegionSet. """ + + self._current_regionset.terminate() + + @property + def str_last_scanned(self): + """ A friendly string with last scanned thing. """ + return self._str_last_scanned + + @property + def current_regionset(self): + """ Returns the current RegionSet being scanned. """ + + return self._current_regionset.regionset + + @property + def finished(self): + """ Return True if the scan has finished. + + It checks if the queue is empty and if the results are ready. + + """ + + return not self.regionsets and self._current_regionset.finished + + @property + def world_obj(self): + return self._world_obj + + @property + def results(self): + """ Yield all the results from the scan. + + This is the simpler method to control the scanning process, + but also the most sloppy. If you want to closely control the + scan process (for example cancel the process in the middle, + whatever is happening) use get_last_result(). + + Usage: + for result in scanner.results: + # do things + """ + + while not self.finished: + cr = self._current_regionset + if cr and not cr.finished: + for r in cr.results: + yield r + elif self.regionsets: + self.scan() + + def __len__(self): + l = 0 + for rs in self.regionsets: + l += len(rs) + return l + + +def console_scan_loop(scanners, scan_titles, verbose): + """ Scan all the AsyncScanner object printing status to console. + + Inputs: + - scanners -- List of AsyncScanner objects to scan. + - scan_titles -- List of string with the names of the world/regionsets in the same + order as in scanners. + - verbose -- Boolean, if true it will print a line per scanned region file. + + """ + + try: + for scanner, title in zip(scanners, scan_titles): + print("\n{0:-^60}".format(title)) + if not len(scanner): + print("Info: No files to scan.") + else: + total = len(scanner) + if not verbose: + pbar = ProgressBar(widgets=[SimpleProgress(), Bar(), AdaptiveETA()], maxval=total).start() + try: + scanner.scan() + counter = 0 + while not scanner.finished: + scanner.sleep() + result = scanner.get_last_result() + if result: + logging.debug("\nNew result: {0}\n\nOneliner: {1}\n".format(result, result.oneliner_status)) + counter += 1 + if not verbose: + pbar.update(counter) + else: + status = "(" + result.oneliner_status + ")" + fn = result.filename + fol = result.folder + print("Scanned {0: <12} {1:.<43} {2}/{3}".format(join(fol, fn), status, counter, total)) + if not verbose: + pbar.finish() + except KeyboardInterrupt as e: + # If not, dead processes will accumulate in windows + scanner.terminate() + raise e + except ChildProcessException as e: + # print "\n\nSomething went really wrong scanning a file." + # print ("This is probably a bug! If you have the time, please report " + # "it to the region-fixer github or in the region fixer post " + # "in minecraft forums") + # print e.printable_traceback + raise e + + +def console_scan_world(world_obj, processes, entity_limit, remove_entities, + verbose): + """ Scans a world folder prints status to console. + + Inputs: + - world_obj -- World object from world.py that will be scanned + - processes -- An integer with the number of child processes to use + - entity_limit -- An integer, threshold of entities for a chunk to be considered + with too many entities + - remove_entities -- A boolean, defaults to False, to remove the entities whilel + scanning. This is really handy because opening chunks with + too many entities for scanning can take minutes. + - verbose -- Boolean, if true it will print a line per scanned region file. + + """ + + # Time to wait between asking for results. Note that if the time is too big + # results will be waiting in the queue and the scan will take longer just + # because of this. + w = world_obj + # Scan the world directory + print("World info:") + + counters = w.get_number_regions() + if c.LEVEL_DIR in counters: + print(" - {0} region/level files,".format(counters[c.LEVEL_DIR])) + if c.POI_DIR in counters: + print(" - {0} POI files,".format(counters[c.POI_DIR])) + if c.ENTITIES_DIR in counters: + print(" - {0} entities files,".format(counters[c.ENTITIES_DIR])) + print(" - {0} player files,".format(len(w.players) + len(w.old_players))) + print(" - and {0} data files.".format(len(w.data_files))) + + # check the level.dat + print("\n{0:-^60}".format(' Checking level.dat ')) + + if not w.scanned_level.path: + print("[WARNING!] \'level.dat\' doesn't exist!") + else: + if w.scanned_level.status not in c.DATAFILE_PROBLEMS: + print("\'level.dat\' is readable") + else: + print("[WARNING!]: \'level.dat\' is corrupted with the following error/s:") + print("\t {0}".format(c.DATAFILE_STATUS_TEXT[w.scanned_level.status])) + + ps = AsyncDataScanner(w.players, processes) + ops = AsyncDataScanner(w.old_players, processes) + ds = AsyncDataScanner(w.data_files, processes) + ws = AsyncWorldRegionScanner(w, processes, entity_limit, remove_entities) + + scanners = [ps, ops, ds, ws] + + scan_titles = [' Scanning UUID player files ', + ' Scanning old format player files ', + ' Scanning structures and map data files ', + ' Scanning region, POI and entities files '] + console_scan_loop(scanners, scan_titles, verbose) + w.scanned = True + + +def console_scan_regionset(regionset, processes, entity_limit, remove_entities, verbose): + """ Scan a regionset printing status to console. + + Inputs: + - regionset -- RegionSet object from world.py that will be scanned + - processes -- An integer with the number of child processes to use + - entity_limit -- An integer, threshold of entities for a chunk to be considered + with too many entities + - remove_entities -- A boolean, defaults to False, to remove the entities whilel + scanning. This is really handy because opening chunks with + too many entities for scanning can take minutes. + - verbose -- Boolean, if true it will print a line per scanned region file. + + """ + + rs = AsyncRegionsetScanner(regionset, processes, entity_limit, + remove_entities) + scanners = [rs] + titles = [entitle("Scanning separate region files", 0)] + console_scan_loop(scanners, titles, verbose) + regionset.scanned = True + + +def scan_data(scanned_dat_file): + """ Try to parse the nbt data file, and fill the scanned object. + + Inputs: + - scanned_dat_file -- ScannedDataFile object from world.py. + + If something is wrong it will return a tuple with useful info + to debug the problem. + + NOTE: idcounts.dat (number of map files) is a nbt file and + is not compressed, we handle the special case here. + + """ + + s = scanned_dat_file + try: + if s.filename == 'idcounts.dat': + # TODO: This is ugly + # Open the file and create a buffer, this way + # NBT won't try to de-gzip the file + f = open(s.path) + + _ = nbt.NBTFile(buffer=f) + else: + _ = nbt.NBTFile(filename=s.path) + s.status = c.DATAFILE_OK + except MalformedFileError: + s.status = c.DATAFILE_UNREADABLE + except IOError: + s.status = c.DATAFILE_UNREADABLE + except UnicodeDecodeError: + s.status = c.DATAFILE_UNREADABLE + except TypeError: + s.status = c.DATAFILE_UNREADABLE + except EOFError: + # There is a compressed stream in the file but ends abruptly + s.status = c.DATAFILE_UNREADABLE + + except: + s.status = c.DATAFILE_UNREADABLE + except_type, except_class, tb = sys.exc_info() + s = (s, (except_type, except_class, extract_tb(tb))) + + return s + + +def scan_region_file(scanned_regionfile_obj, entity_limit, remove_entities): + """ Scan a region file filling the ScannedRegionFile object + + Inputs: + - scanned_regionfile_obj -- ScannedRegionfile object from world.py that will be scanned + - entity_limit -- An integer, threshold of entities for a chunk to be considered + with too many entities + - remove_entities -- A boolean, defaults to False, to remove the entities while + scanning. This is really handy because opening chunks with + too many entities for scanning can take minutes. + + """ + + try: + r = scanned_regionfile_obj + + # try to open the file and see if we can parse the header + try: + region_file = region.RegionFile(r.path) + except region.NoRegionHeader: # The region has no header + r.status = c.REGION_TOO_SMALL + r.scan_time = time() + r.scanned = True + return r + + except PermissionError: + r.status = c.REGION_UNREADABLE_PERMISSION_ERROR + r.scan_time = time() + r.scanned = True + return r + + except IOError: + r.status = c.REGION_UNREADABLE + r.scan_time = time() + r.scanned = True + return r + + for x in range(32): + for z in range(32): + # start the actual chunk scanning + g_coords = r.get_global_chunk_coords(x, z) + chunk, tup = scan_chunk(region_file, + (x, z), + g_coords, + entity_limit) + if tup: + r[(x, z)] = tup + else: + # chunk not created + continue + + if tup[c.TUPLE_STATUS] == c.CHUNK_OK: + continue + elif tup[c.TUPLE_STATUS] == c.CHUNK_TOO_MANY_ENTITIES: + # Deleting entities is in here because parsing a chunk + # with thousands of wrong entities takes a long time, + # and sometimes GiB of RAM, and once detected is better + # to fix it at once. + if remove_entities: + world.delete_entities(region_file, x, z) + print(("Deleted {0} entities in chunk" + " ({1},{2}) of the region file: {3}").format(tup[c.TUPLE_NUM_ENTITIES], x, z, r.filename)) + # entities removed, change chunk status to OK + r[(x, z)] = (0, c.CHUNK_OK) + + else: + # This stores all the entities in a file, + # comes handy sometimes. + # ~ pretty_tree = chunk['Level']['Entities'].pretty_tree() + # ~ name = "{2}.chunk.{0}.{1}.txt".format(x,z,split(region_file.filename)[1]) + # ~ archivo = open(name,'w') + # ~ archivo.write(pretty_tree) + pass + elif tup[c.TUPLE_STATUS] == c.CHUNK_CORRUPTED: + pass + elif tup[c.TUPLE_STATUS] == c.CHUNK_WRONG_LOCATED: + pass + + # Now check for chunks sharing offsets: + # Please note! region.py will mark both overlapping chunks + # as bad (the one stepping outside his territory and the + # good one). Only wrong located chunk with a overlapping + # flag are really BAD chunks! Use this criterion to + # discriminate + # + # TODO: Why? I don't remember why + # TODO: Leave this to nbt, which code is much better than this + + metadata = region_file.metadata + sharing = [k for k in metadata if (metadata[k].status == region.STATUS_CHUNK_OVERLAPPING and + r[k][c.TUPLE_STATUS] == c.CHUNK_WRONG_LOCATED)] + shared_counter = 0 + for k in sharing: + r[k] = (r[k][c.TUPLE_NUM_ENTITIES], c.CHUNK_SHARED_OFFSET) + shared_counter += 1 + + r.scan_time = time() + r.status = c.REGION_OK + r.scanned = True + return r + + except KeyboardInterrupt: + print("\nInterrupted by user\n") + # TODO this should't exit. It should return to interactive + # mode if we are in it. + sys.exit(1) + + # Fatal exceptions: + except: + # Anything else is a ChildProcessException + # NOTE TO SELF: do not try to return the traceback object directly! + # A multiprocess pythonic hell comes to earth if you do so. + except_type, except_class, tb = sys.exc_info() + r = (scanned_regionfile_obj, + (except_type, except_class, extract_tb(tb))) + + return r + + +def scan_chunk(region_file, coords, global_coords, entity_limit): + """ Scans a chunk returning its status and number of entities. + + Keywords arguments: + region_file -- nbt.RegionFile object + coords -- tuple containing the local (region) coordinates of the chunk + global_coords -- tuple containing the global (world) coordinates of the chunk + entity_limit -- the number of entities that is considered to be too many + + Return: + chunk -- as a nbt file + (num_entities, status) -- tuple with the number of entities of the chunk and + the status described by the CHUNK_* variables in + world.py + + If the chunk does not exist (is not yet created it returns None) + + This function also scan the chunks contained in the POI region files. + + """ + + el = entity_limit + + try: + chunk = region_file.get_chunk(*coords) + chunk_type = world.get_chunk_type(chunk) + + if chunk_type == c.LEVEL_DIR: + # to know if is a poi chunk or a level chunk check the contents + # if 'Level' is at root is a level chunk + + # Level chunk + try: + data_coords = world.get_chunk_data_coords(chunk) + + # Since snapshot 20w45a (1.17), entities MAY BE separated + if "DataVersion" in chunk and chunk["DataVersion"].value >= 2681 : + num_entities = None + + # Since snapshot 21w43a (1.18), "Level" tag doesn't exist anymore + # According to the wiki, an "entities" tag can still be there (But I've never seen it) + if chunk["DataVersion"].value >= 2844 : + if "entities" in chunk : + num_entities = len(chunk["entities"]) + + # >= 20w45a and < 21w43a + # Don't check if "Level" tag exist, at this point, it should exist + elif "Entities" in chunk["Level"] : + num_entities = len(chunk["Level"]["Entities"]) + else : + num_entities = len(chunk["Level"]["Entities"]) + + if data_coords != global_coords: + # wrong located chunk + status = c.CHUNK_WRONG_LOCATED + elif num_entities != None and num_entities > el: + # too many entities in the chunk + status = c.CHUNK_TOO_MANY_ENTITIES + else: + # chunk ok + status = c.CHUNK_OK + + ############################ + # Chunk error detection + ############################ + except KeyError: + # chunk with the mandatory tag Entities missing + status = c.CHUNK_MISSING_ENTITIES_TAG + chunk = None + data_coords = None + global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) + num_entities = None + + except TypeError: + # TODO: This should another kind of error, it's now being handled as corrupted chunk + status = c.CHUNK_CORRUPTED + chunk = None + data_coords = None + global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) + num_entities = None + + elif chunk_type == c.POI_DIR: + # To check if it's a POI chunk check for the tag "Sections" + # If we give a look to the wiki: + # https://minecraft.wiki/w/Java_Edition_level_format#poi_format + # We can see that there are two TAGs at root of a POI, "Data" and "DataVersion", but + # in my tests the TAGs at root are "Sections and "DataVersion", no trace of "Data". + # + # So, let's use "Sections" as a differentiating factor + + # POI chunk + data_coords = None + num_entities = None + status = c.CHUNK_OK + + elif chunk_type == c.ENTITIES_DIR: + # To check if it's a entities chunk check for the tag "Entities" + # If entities are in the region files, the tag "Entities" is in "Level" + # https://minecraft.wiki/w/Entity_format + # We use "Entities" as a differentiating factor + + # Entities chunk + data_coords = world.get_chunk_data_coords(chunk) + num_entities = len(chunk["Entities"]) + + if data_coords != global_coords: + # wrong located chunk + status = c.CHUNK_WRONG_LOCATED + elif num_entities > el: + # too many entities in the chunk + status = c.CHUNK_TOO_MANY_ENTITIES + else: + # chunk ok + status = c.CHUNK_OK + + else: + # what is this? we shouldn't reach this part of the code, as far as + # we know there is only POI chunks, Entities chunks, and Level chunks + raise AssertionError("Unsupported chunk type in scan_chunk().") + + ############################################### + # POI chunk and Level chunk common errors + ############################################### + except InconceivedChunk: + # chunk not created + chunk = None + data_coords = None + num_entities = None + status = c.CHUNK_NOT_CREATED + + except RegionHeaderError: + # corrupted chunk, because of region header + status = c.CHUNK_CORRUPTED + chunk = None + data_coords = None + global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) + num_entities = None + + except ChunkDataError: + # corrupted chunk, usually because of bad CRC in compression + status = c.CHUNK_CORRUPTED + chunk = None + data_coords = None + global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) + num_entities = None + + except ChunkHeaderError: + # corrupted chunk, error in the header of the chunk + status = c.CHUNK_CORRUPTED + chunk = None + data_coords = None + global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) + num_entities = None + + except UnicodeDecodeError: + # TODO: This should another kind of error, it's now being handled as corrupted chunk + status = c.CHUNK_CORRUPTED + chunk = None + data_coords = None + global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) + num_entities = None + + return chunk, (num_entities, status) if status != c.CHUNK_NOT_CREATED else None + + +if __name__ == '__main__': + pass diff --git a/regionfixer_core/util.py b/regionfixer_core/util.py new file mode 100644 index 0000000..4859a6d --- /dev/null +++ b/regionfixer_core/util.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# +# Region Fixer. +# Fix your region files with a backup copy of your Minecraft world. +# Copyright (C) 2020 Alejandro Aguilera (Fenixin) +# https://github.com/Fenixin/Minecraft-Region-Fixer +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import platform +import sys +import traceback + + +def get_str_from_traceback(ty, value, tb): + """ Return a string from a traceback plus exception. + + Inputs: + - ty -- Exception type + - value -- value of the traceback + - tb -- Traceback + + """ + + t = traceback.format_exception(ty, value, tb) + s = str(ty) + "\n" + for i in t: + s += i + return s + + +# Stolen from: +# http://stackoverflow.com/questions/3041986/python-command-line-yes-no-input +def query_yes_no(question, default="yes"): + """Ask a yes/no question via raw_input() and return their answer. + + "question" is a string that is presented to the user. + "default" is the presumed answer if the user just hits . + It must be "yes" (the default), "no" or None (meaning + an answer is required of the user). + + The "answer" return value is one of "yes" or "no". + """ + valid = {"yes": True, "y": True, "ye": True, + "no": False, "n": False + } + if default is None: + prompt = " [y/n] " + elif default == "yes": + prompt = " [Y/n] " + elif default == "no": + prompt = " [y/N] " + else: + raise ValueError("invalid default answer: '%s'" % default) + + while True: + sys.stdout.write(question + prompt) + choice = input().lower() + if default is not None and choice == '': + return valid[default] + elif choice in valid: + return valid[choice] + else: + sys.stdout.write("Please respond with 'yes' or 'no' " + "(or 'y' or 'n').\n") + + +# stolen from minecraft overviewer +# https://github.com/overviewer/Minecraft-Overviewer/ +def is_bare_console(): + """Returns true if the python script is running in a bare console + + In Windows, that is, if the script wasn't started in a cmd.exe + session. + + """ + + if platform.system() == 'Windows': + try: + import ctypes + GetConsoleProcessList = ctypes.windll.kernel32.GetConsoleProcessList + num = GetConsoleProcessList(ctypes.byref(ctypes.c_int(0)), ctypes.c_int(1)) + if num == 1: + return True + + except Exception: + pass + return False + + +def entitle(text, level=0): + """ Put the text in a title with lot's of hashes around it. """ + + t = '' + if level == 0: + t += "\n" + t += "{0:#^60}\n".format('') + t += "{0:#^60}\n".format(' ' + text + ' ') + t += "{0:#^60}\n".format('') + return t + + +def table(columns): + """ Generates a text containing a pretty table. + + Input: + - columns -- A list containing lists in which each one of the is a column + of the table. + + """ + + def get_max_len(l): + """ Takes a list of strings and returns the length of the biggest string """ + m = 0 + for e in l: + if len(str(e)) > m: + m = len(str(e)) + return m + + text = "" + # stores the size of the biggest element in that column + ml = [] + # fill up ml + for c in columns: + m = 0 + t = get_max_len(c) + if t > m: + m = t + ml.append(m) + # get the total width of the table: + ml_total = 0 + for i in range(len(ml)): + ml_total += ml[i] + 2 # size of each word + 2 spaces + ml_total += 1 + 2 # +1 for the separator | and +2 for the borders + text += "-" * ml_total + "\n" + # all the columns have the same number of rows + row = len(columns[0]) + for r in range(row): + line = "|" + # put all the elements in this row together with spaces + for i in range(len(columns)): + line += "{0: ^{width}}".format(columns[i][r], width=ml[i] + 2) + # add a separator for the first column + if i == 0: + line += "|" + + text += line + "|" + "\n" + if r == 0: + text += "-" * ml_total + "\n" + text += "-" * ml_total + return text + diff --git a/regionfixer_core/version.py b/regionfixer_core/version.py new file mode 100644 index 0000000..75458ac --- /dev/null +++ b/regionfixer_core/version.py @@ -0,0 +1,25 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# +# Region Fixer. +# Fix your region files with a backup copy of your Minecraft world. +# Copyright (C) 2020 Alejandro Aguilera (Fenixin) +# https://github.com/Fenixin/Minecraft-Region-Fixer +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +version_string = "0.3.6" +version_numbers = version_string.split('.') diff --git a/regionfixer_core/world.py b/regionfixer_core/world.py new file mode 100644 index 0000000..c0c3d6c --- /dev/null +++ b/regionfixer_core/world.py @@ -0,0 +1,1934 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# +# Region Fixer. +# Fix your region files with a backup copy of your Minecraft world. +# Copyright (C) 2020 Alejandro Aguilera (Fenixin) +# https://github.com/Fenixin/Minecraft-Region-Fixer +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +from glob import glob +from os.path import join, split, exists, isfile +from os import remove +from shutil import copy +import zlib + +import nbt.region as region +import nbt.nbt as nbt +from .util import table +from nbt.nbt import TAG_List + +import regionfixer_core.constants as c + + + +class InvalidFileName(IOError): + """ Exception raised when a filename is wrong. """ + pass + + +class ScannedDataFile: + """ Stores all the information of a scanned data file. + + Inputs: + - path -- String with the path of the data file. Defaults to None. + """ + + def __init__(self, path=None): + super().__init__() + self.path = path + if self.path and exists(self.path): + self.filename = split(path)[1] + self.folder = split(split(path)[0])[1] + else: + self.filename = None + # The status of the region file. + self.status = None + + def __str__(self): + text = "NBT file:" + str(self.filename) + "\n" + text += "\tStatus:" + c.DATAFILE_STATUS_TEXT[self.status] + "\n" + return text + + @property + def oneliner_status(self): + """ One line describing the status of the file. """ + return "File: \"" + self.filename + "\"; status: " + c.DATAFILE_STATUS_TEXT[self.status] + + +class ScannedChunk: + """ Stores all the information of a scanned chunk. + + Not used at the moment, it's nice but takes an huge amount of memory when + is not strange for chunks to be in the order of millions.""" + # WARNING: This is here so I remember to not use objects as ScannedChunk + # They take too much memory. + + +class ScannedRegionFile: + """ Stores all the scan information for a region file. + + Keywords arguments: + - path -- A string with the path of the region file + - scanned_time -- Float, time as returned by bult-in time module. The time + at which the region file has been scanned. None by default. + - folder -- Used to enhance print() + + """ + + def __init__(self, path, scanned_time=None, folder=""): + # general region file info + self.path = path + self.filename = split(path)[1] + self.folder = folder + self.x = self.z = None + self.x, self.z = self.get_coords() + self.coords = (self.x, self.z) + + # dictionary storing all the state tuples of all the chunks + # in the region file, keys are the local coords of the chunk + # sometimes called header coords + self._chunks = {} + + # Dictionary containing counters to for all the chunks + self._counts = {} + for s in c.CHUNK_STATUSES: + self._counts[s] = 0 + + # time when the scan for this file finished + self.scan_time = scanned_time + + # The status of the region file. + self.status = None + + # has the file been scanned yet? + self.scanned = False + + @property + def oneliner_status(self): + """ On line description of the status of the region file. """ + if self.scanned: + status = self.status + if status == c.REGION_OK: # summary with all found in scan + stats = "" + for s in c.CHUNK_PROBLEMS: + stats += "{0}:{1}, ".format(c.CHUNK_PROBLEMS_ABBR[s], self.count_chunks(s)) + stats += "t:{0}".format(self.count_chunks()) + else: + stats = c.REGION_STATUS_TEXT[status] + else: + stats = "Not scanned" + + return stats + + def __str__(self): + text = "Path: {0}".format(self.path) + scanned = False + if self.scan_time: + scanned = True + text += "\nScanned: {0}".format(scanned) + + return text + + def __getitem__(self, key): + return self._chunks[key] + + def __setitem__(self, key, value): + self._chunks[key] = value + self._counts[value[c.TUPLE_STATUS]] += 1 + + def get_coords(self): + """ Returns the region file coordinates as two integers. + + Return: + - coordX, coordZ -- Integers with the x and z coordinates of the + region file. + + Either parse the region file name or uses the stored in the object. + + """ + + if self.x != None and self.z != None: + return self.x, self.z + else: + splited = split(self.filename) + filename = splited[1] + l = filename.split('.') + try: + coordX = int(l[1]) + coordZ = int(l[2]) + except ValueError: + raise InvalidFileName() + + return coordX, coordZ + + def keys(self): + """Returns a list with all the local coordinates (header coordinates). + + Return: + - list -- A list with all the local chunk coordinates extracted form the + region file header as integer tuples + """ + + return list(self._chunks.keys()) + + @property + def has_problems(self): + """ Return True if the region file has problem in itself or in its chunks. + + Return: + - boolean -- True f the region has problems or False otherwise. + + """ + + if self.status in c.REGION_PROBLEMS: + return True + for s in c.CHUNK_PROBLEMS: + if self.count_chunks(s): + return True + return False + + def get_path(self): + """ Returns the path of the region file. + + Return: + - path -- A string with the path of the region file. + + """ + + return self.path + + def count_chunks(self, status=None): + """ Counts chunks in the region file with the given problem. + + Inputs: + - status -- Integer with the status of the chunk to count for. See + CHUNK_PROBLEMS in constants.py. + + Return: + - counter -- Integer with the number of chunks with that status + + If problem is omitted or None, counts all the chunks. Returns + an integer with the counter. + + """ + + if status == None: + counter = 0 + for s in c.CHUNK_STATUSES: + counter += self._counts[s] + else: + counter = self._counts[status] + + return counter + + def get_global_chunk_coords(self, chunkX, chunkZ): + """ Takes the chunk local coordinates and returns its global coordinates. + + Inputs: + - chunkX -- Integer, local X chunk coordinate. + - chunkZ -- Integer, local Z chunk coordinate. + + Return: + - chunkX, chunkZ -- Integers with the x and z global chunk coordinates + + """ + + regionX, regionZ = self.get_coords() + chunkX += regionX * 32 + chunkZ += regionZ * 32 + + return chunkX, chunkZ + + def list_chunks(self, status=None): + """ Returns a list of tuples of chunks for all the chunks with 'status'. + + Inputs: + - status -- Defaults to None. Integer with the status of the chunk to list, + see CHUNK_STATUSES in constants.py + + Return: + - list - List with tuples like (global_coordinates, status_tuple) where status + tuple is (number_of_entities, status) + + If status is omitted or None, returns all the chunks in the region file + + """ + + l = [] + for ck in list(self.keys()): + t = self[ck] + if status == t[c.TUPLE_STATUS]: + l.append((self.get_global_chunk_coords(*ck), t)) + elif status == None: + l.append((self.get_global_chunk_coords(*ck), t)) + + return l + + def summary(self): + """ Returns a summary of all the problematic chunks. + + Return: + - text -- Human readable string with the summary of the scan. + + The summary is a human readable string with region file, global + coordinates, local coordinates, and status of every problematic + chunk, in a subtree like format. + + """ + + text = "" + if self.status in c.REGION_PROBLEMS: + text += " |- This region has status: {0}.\n".format(c.REGION_STATUS_TEXT[self.status]) + else: + for ck in list(self.keys()): + if self[ck][c.TUPLE_STATUS] not in c.CHUNK_PROBLEMS: + continue + status = self[ck][c.TUPLE_STATUS] + h_coords = ck + g_coords = self.get_global_chunk_coords(*h_coords) + text += " |-+-Chunk coords: header {0}, global {1}.\n".format(h_coords, g_coords) + text += " | +-Status: {0}\n".format(c.CHUNK_STATUS_TEXT[status]) + if self[ck][c.TUPLE_STATUS] == c.CHUNK_TOO_MANY_ENTITIES: + text += " | +-No. entities: {0}\n".format(self[ck][c.TUPLE_NUM_ENTITIES]) + text += " |\n" + + return text + + def remove_problematic_chunks(self, status): + """ Removes all the chunks with the given status + + Inputs: + - status -- Integer with the status of the chunks to remove. + See CHUNK_STATUSES in constants.py + + Return: + - counter -- An integer with the amount of removed chunks. + + """ + + counter = 0 + bad_chunks = self.list_chunks(status) + for ck in bad_chunks: + global_coords = ck[0] + local_coords = _get_local_chunk_coords(*global_coords) + region_file = region.RegionFile(self.path) + region_file.unlink_chunk(*local_coords) + counter += 1 + # create the new status tuple + # (num_entities, chunk status) + self[local_coords] = (0, c.CHUNK_NOT_CREATED) + + return counter + + def fix_problematic_chunks(self, status): + """ This fixes problems in chunks that can be somehow fixed. + + Inputs: + - status -- Integer with the status of the chunks to fix. See + FIXABLE_CHUNK_PROBLEMS in constants.py + + Return: + - counter -- An integer with the amount of fixed chunks. + + Right now it only fixes chunks missing the TAG_List Entities, wrong located chunks and + in some cases corrupted chunks. + + -TAG_List is fixed by adding said tag. + + -Wrong located chunks are relocated to the data coordinates stored in the zip stream. + We suppose these coordinates are right because the data has checksum. + + -Corrupted chunks: tries to read the the compressed stream byte by byte until it raises + exception. After that compares the size of the compressed chunk stored in the region file + with the compressed chunk extracted from the strem, if they are the same it's good to go! + + """ + + # TODO: it seems having the Entities TAG missing is just a little part. Some of the + # chunks have like 3 or 4 tag missing from the NBT structure. I don't really know which + # of them are mandatory. + + assert(status in c.FIXABLE_CHUNK_PROBLEMS) + counter = 0 + bad_chunks = self.list_chunks(status) + for ck in bad_chunks: + global_coords = ck[0] + local_coords = _get_local_chunk_coords(*global_coords) + region_file = region.RegionFile(self.path) + # catch the exception of corrupted chunks + try: + chunk = region_file.get_chunk(*local_coords) + except region.ChunkDataError: + # if we are here the chunk is corrupted, but still + if status == c.CHUNK_CORRUPTED: + # read the data raw + m = region_file.metadata[local_coords[0], local_coords[1]] + region_file.file.seek(m.blockstart * region.SECTOR_LENGTH + 5) + # these status doesn't provide a good enough data, we could end up reading garbage + if m.status not in (region.STATUS_CHUNK_IN_HEADER, region.STATUS_CHUNK_MISMATCHED_LENGTHS, + region.STATUS_CHUNK_OUT_OF_FILE, region.STATUS_CHUNK_OVERLAPPING, + region.STATUS_CHUNK_ZERO_LENGTH): + # get the raw data of the chunk + raw_chunk = region_file.file.read(m.length - 1) + # decompress byte by byte so we can get as much as we can before the error happens + dc = zlib.decompressobj() + out = "" + for i in raw_chunk: + out += dc.decompress(i) + # compare the sizes of the new compressed strem and the old one to see if we've got something good + cdata = zlib.compress(out.encode()) + if len(cdata) == len(raw_chunk): + # the chunk is probably good, write it in the region file + region_file.write_blockdata(local_coords[0], local_coords[1], out) + print("The chunk {0},{1} in region file {2} was fixed successfully.".format(local_coords[0], local_coords[1], join(self.folder,self.filename))) + else: + print("The chunk {0},{1} in region file {2} couldn't be fixed.".format(local_coords[0], local_coords[1], join(self.folder,self.filename))) + #======================================================= + # print("Extracted: " + str(len(out))) + # print("Size of the compressed stream: " + str(len(raw_chunk))) + #======================================================= + except (region.ChunkHeaderError, region.RegionHeaderError, UnicodeDecodeError): + # usually a chunk with zero length in the first two cases, or veeery broken chunk in the third + print("The chunk {0},{1} in region file {2} couldn't be fixed.".format(local_coords[0], local_coords[1], join(self.folder,self.filename))) + + if status == c.CHUNK_MISSING_ENTITIES_TAG: + # The arguments to create the empty TAG_List have been somehow extracted by comparing + # the tag list from a healthy chunk with the one created by nbt + chunk_type = get_chunk_type(chunk) + if chunk_type == c.LEVEL_DIR : + if "DataVersion" in chunk and chunk["DataVersion"].value >= 2844 : # Snapshot 21w43a (1.18) + chunk['entities'] = TAG_List(name='entities', type=nbt._TAG_End) + else : + chunk['Level']['Entities'] = TAG_List(name='Entities', type=nbt._TAG_End) + elif chunk_type == c.ENTITIES_DIR : + chunk['Entities'] = TAG_List(name='Entities', type=nbt._TAG_End) + else : + raise AssertionError("Unsupported chunk type.") + region_file.write_chunk(local_coords[0],local_coords[1], chunk) + + # create the new status tuple + # (num_entities, chunk status) + self[local_coords] = (0 , c.CHUNK_NOT_CREATED) + counter += 1 + + elif status == c.CHUNK_WRONG_LOCATED: + data_coords = get_chunk_data_coords(chunk) + data_l_coords = _get_local_chunk_coords(*data_coords) + region_file.write_chunk(data_l_coords[0], data_l_coords[1], chunk) + region_file.unlink_chunk(*local_coords) + # what to do with the old chunk in the wrong position? + # remove it or keep it? It's probably the best to remove it. + # create the new status tuple + + # remove the wrong position of the chunk and update the status + # (num_entities, chunk status) + self[local_coords] = (0 , c.CHUNK_NOT_CREATED) + self[data_l_coords]= (0 , c.CHUNK_OK) + counter += 1 + + return counter + + def remove_entities(self): + """ Removes all the entities in chunks with status c.CHUNK_TOO_MANY_ENTITIES. + + Return: + - counter -- Integer with the number of removed entities. + + """ + + status = c.CHUNK_TOO_MANY_ENTITIES + counter = 0 + bad_chunks = self.list_chunks(status) + for ck in bad_chunks: + global_coords = ck[0] + local_coords = _get_local_chunk_coords(*global_coords) + counter += self.remove_chunk_entities(*local_coords) + # create new status tuple: + # (num_entities, chunk status) + self[local_coords] = (0, c.CHUNK_OK) + return counter + + def remove_chunk_entities(self, x, z): + """ Takes a chunk local coordinates and remove its entities. + + Inputs: + - x -- Integer with the X local (header) coordinate of the chunk + - z -- Integer with the Z local (header) coordinate of the chunk + + Return: + - counter -- An integer with the number of entities removed. + + This will remove all the entities in the chunk, it will not perform any + kind of check. + + """ + + return delete_entities( region.RegionFile(self.path), x, z ) + + def rescan_entities(self, options): + """ Updates the status of all the chunks after changing entity_limit. + + Inputs: + - options -- argparse arguments, the whole argparse.ArgumentParser() object as used + by regionfixer.py + + """ + + for ck in list(self.keys()): + # for safety reasons use a temporary list to generate the + # new tuple + t = [0, 0] + if self[ck][c.TUPLE_STATUS] in (c.CHUNK_TOO_MANY_ENTITIES, c.CHUNK_OK): + # only touch the ok chunks and the too many entities chunk + if self[ck][c.TUPLE_NUM_ENTITIES] > options.entity_limit: + # now it's a too many entities problem + t[c.TUPLE_NUM_ENTITIES] = self[ck][c.TUPLE_NUM_ENTITIES] + t[c.TUPLE_STATUS] = c.CHUNK_TOO_MANY_ENTITIES + + elif self[c][c.TUPLE_NUM_ENTITIES] <= options.entity_limit: + # the new limit says it's a normal chunk + t[c.TUPLE_NUM_ENTITIES] = self[ck][c.TUPLE_NUM_ENTITIES] + t[c.TUPLE_STATUS] = c.CHUNK_OK + + self[ck] = tuple(t) + + +class DataSet: + """ Stores data items to be scanned by AsyncScanner in scan.py. + + Inputs: + - typevalue -- The type of the class to store in the set. In initialization it will be + asserted if it is of that type + + The data will be stored in the self._set dictionary. + + Implemented private methods are: __getitem__, __setitem__, _get_list, __len__. + + Three methods should be overridden to work with a DataSet, two of the mandatory: + - _replace_in_data_structure -- (mandatory) Should be created because during the scan the + different processes create copies of the original data, so replacing it in + the original data set is mandatory in order to keep everything working. + + - _update_counts -- (mandatory) Makes sure that the DataSet stores all the counts and + that it is not needed to loop through all of them to know the real count. + + - has_problems -- (optional but used) Should return True only if any element + of the set has problems + + """ + + def __init__(self, typevalue, *args, **kwargs): + self._set = {} + self._typevalue = typevalue + + def _get_list(self): + """ Returns a list with all the values in the set. """ + + return list(self._set.values()) + + def __getitem__(self, key): + return self._set[key] + + def __delitem__(self, key): + del self._set[key] + + def __setitem__(self, key, value): + assert self._typevalue == type(value) + self._set[key] = value + self._update_counts(value) + + def __len__(self): + return len(self._set) + + # mandatory implementation methods + def summary(self): + """ Return a summary of problems found in this set. """ + + raise NotImplementedError + + @property + def has_problems(self): + """ Returns True if the scanned set has problems. """ + + raise NotImplementedError + + def _replace_in_data_structure(self, data, key): + """ For multiprocessing. Replaces the data in the set with the new data. + + Inputs: + - data -- Value of the data to be stored + - key -- Key in which to store the data + + Child scanning processes make copies of the ScannedRegion/DataFile when they scan them. + The AsyncScanner will call this function so the ScannedRegion/DataFile is stored + in the set properly. + """ + + raise NotImplementedError + + def _update_counts(self, s): + """ This functions is used by __set__ to update the counters. """ + + raise NotImplementedError + + +class DataFileSet(DataSet): + """ DataSet for Minecraft data files (.dat). + + Inputs: + - path -- Path to the folder containing data files + - title -- Some user readable string to represent the DataSet + """ + + def __init__(self, path, title, *args, **kwargs): + DataSet.__init__(self, ScannedDataFile, *args, **kwargs) + d = self._set + + self.title = title + self.path = path + data_files_path = glob(join(path, "*.dat")) + + for path in data_files_path: + d[path] = ScannedDataFile(path) + + # stores the counts of files + self._counts = {} + for s in c.DATAFILE_STATUSES: + self._counts[s] = 0 + + @property + def has_problems(self): + """ Returns True if the dataset has problems and false otherwise. """ + + for d in self._set.values(): + if d.status in c.DATAFILE_PROBLEMS: + return True + return False + + def _replace_in_data_structure(self, data): + self._set[data.path] = data + + def _update_counts(self, s): + assert isinstance(s, self._typevalue) + self._counts[s.status] += 1 + + def count_datafiles(self, status): + pass + + def summary(self): + """ Return a summary of problems found in this set. """ + + text = "" + bad_data_files = [i for i in list(self._set.values()) if i.status in c.DATAFILE_PROBLEMS] + for f in bad_data_files: + text += "\t" + f.oneliner_status + text += "\n" + return text + + +class RegionSet(DataSet): + """Stores an arbitrary number of region files and their scan results. + + Inputs: + - regionset_path -- Path to the folder containing region files + IT MUST NOT END WITH A SLASH ("/") + - region_list -- List of paths to all the region files + - overworld -- Tweak to tell it's a dimension and not the overworld + """ + + def __init__(self, regionset_path=None, region_list=[], overworld=True): + DataSet.__init__(self, ScannedRegionFile) + # Otherwise, problems in _get_dimension_directory() and _get_region_type_directory() + if regionset_path != None : + assert regionset_path[-1] != "/" + self.overworld = overworld + + if regionset_path: + self.path = regionset_path + self.region_list = glob(join(self.path, "r.*.*.mca")) + else: + self.path = None + self.region_list = region_list + self._set = {} + for path in self.region_list: + try: + r = ScannedRegionFile(path, folder=self._get_dim_type_string()) + self._set[r.get_coords()] = r + + except InvalidFileName: + try : + region_type = c.REGION_TYPES_NAMES[self._get_region_type_directory()][0] + except: + region_type = "region (?)" + print("Warning: The file {0} is not a valid name for a {1} file. I'll skip it.".format(path, region_type)) + + # region and chunk counters with all the data from the scan + self._region_counters = {} + for status in c.REGION_STATUSES: + self._region_counters[status] = 0 + + self._chunk_counters = {} + for status in c.CHUNK_STATUSES: + self._chunk_counters[status] = 0 + + # has this regionset been scanned? + self.scanned = False + + def get_name(self): + """ Return a string with a representative name for the regionset + + The order for getting the name is: + 1 - The name derived by the dimension path + 2 - The name of the last directory in the path as returned by _get_dimension_directory + 3 - Empty string "" + + """ + + dim_directory = self._get_dimension_directory() + region_type_directory = self._get_region_type_directory() + if (dim_directory or self.overworld) and region_type_directory: + try: dim_directory = c.DIMENSION_NAMES[dim_directory] + except: dim_directory = "\"" + dim_directory + "\"" + try: region_type_directory = c.REGION_TYPES_NAMES[region_type_directory][1] + except: region_type_directory = "\"" + region_type_directory + "\"" + return "{0} files for {1}".format(region_type_directory, dim_directory) + else: + return "" + + def _get_dimension_directory(self): + """ Returns a string with the parent directory containing the RegionSet. + + If there is no such a directory returns None. If it's composed + of sparse region files returns 'regionset'. + + """ + + if self.path: + if self.overworld : + return "" + rest, type_dir = split(self.path) + rest, dim_path = split(rest) + return dim_path + else: + return None + + def _get_region_type_directory(self): + """ Returns a string with the directory containing the RegionSet. + + If there is no such a directory returns None. If it's composed + of sparse region files returns 'regionset'. + """ + + if self.path: + rest, type_dir = split(self.path) + return type_dir + else: + return None + + def _get_dim_type_string(self) : + dim = self._get_dimension_directory() + rg_type = self._get_region_type_directory() + string = "" + if rg_type != None : string = rg_type + if dim != None and dim != "" : string = dim + "/" + rg_type + return string + + def _update_counts(self, scanned_regionfile): + """ Updates the counters of the regionset with the new regionfile. """ + + assert isinstance(scanned_regionfile, ScannedRegionFile) + + self._region_counters[scanned_regionfile.status] += 1 + + for status in c.CHUNK_STATUSES: + self._chunk_counters[status] += scanned_regionfile.count_chunks(status) + + def _replace_in_data_structure(self, data): + self._set[data.get_coords()] = data + + def __str__(self): + text = "RegionSet: {0}\n".format(self.get_name()) + if self.path: + text += " Regionset path: {0}\n".format(self.path) + text += " Region files: {0}\n".format(len(self._set)) + text += " Scanned: {0}".format(str(self.scanned)) + return text + + @property + def has_problems(self): + """ Returns True if the regionset has chunk or region problems and false otherwise. """ + + for s in c.REGION_PROBLEMS: + if self.count_regions(s): + return True + + for s in c.CHUNK_PROBLEMS: + if self.count_chunks(s): + return True + + return False + + def keys(self): + return list(self._set.keys()) + + def list_regions(self, status=None): + """ Returns a list of all the ScannedRegionFile objects with 'status'. + + Inputs: + - status -- The region file status. See c.REGION_STATUSES + + Return: + - t -- List with all the ScannedRegionFile objects with that status + + If status = None it returns all the objects. + + """ + + if status is None: + return list(self._set.values()) + t = [] + for coords in list(self._set.keys()): + r = self._set[coords] + if r.status == status: + t.append(r) + return t + + def count_regions(self, status=None): + """ Return the number of region files with status. + + Inputs: + - status -- The region file status. See c.REGION_STATUSES + + Return: + - counter -- Integer with the number of regions with that status + + If none returns the total number of region files in this regionset. + + """ + + counter = 0 + if status is None: + for s in c.REGION_STATUSES: + counter += self._region_counters[s] + else: + counter = self._region_counters[status] + + return counter + + def count_chunks(self, status=None): + """ Returns the number of chunks with the given status. + + Inputs: + - status -- Integer with the chunk status to count. See + c.CHUNK_STATUSES in constants.py + + Return: + - counter -- Integer with the number of chunks removed + + If status is None returns the number of chunks in this region file. + + """ + + counter = 0 + if status is None: + for s in c.CHUNK_STATUSES: + counter += self._chunk_counters[s] + else: + counter = self._chunk_counters[status] + + return counter + + def list_chunks(self, status=None): + """ Returns a list of all the chunk tuples with 'status'. + + Inputs: + - status -- The chunk status to list. See c.CHUNK_STATUSES + + Return: + - l -- List with tuples like (global_coordinates, status_tuple) where status + tuple is (number_of_entities, status). For more details see + ScannedRegionFile.list_chunks() + + If status = None it returns all the chunk tuples. + + """ + + l = [] + for r in list(self.keys()): + l.extend(self[r].list_chunks(status)) + return l + + def summary(self): + """ Returns a string with a summary of the problematic chunks. + + Return: + - text -- String, human readable text with information about the scan. + + The summary contains global coordinates, local coordinates, + data coordinates and status. + + """ + + text = "" + for r in list(self.keys()): + if not self[r].has_problems: + continue + text += "Region file: {0}\n".format(join(self._get_dim_type_string(),self[r].filename)) + + text += self[r].summary() + text += " +\n\n" + return text + + def locate_chunk(self, global_coords): + """ Takes the global coordinates of a chunk and returns where is it. + + Inputs: + - global_coords -- Tuple of two integers with the global chunk coordinates to locate. + + Return: + - path -- String, with the path of the region file where + the chunk is stored + - local_coords -- Tuple of two integers with local coordinates of the + chunk in the region file + + """ + + path = join(self.path, get_chunk_region(*global_coords)) + local_coords = _get_local_chunk_coords(*global_coords) + + return path, local_coords + + def locate_region(self, coords): + """ Returns a string with the path of the region file. + + Inputs: + - coords -- Tuple of two integers with the global region coordinates of the region + file to locate in this RegionSet. + + Return: + - region_name -- String containing the path of the region file or None if it + doesn't exist + + """ + + x, z = coords + region_name = 'r.' + str(x) + '.' + str(z) + '.mca' + + return region_name + + def remove_problematic_chunks(self, status): + """ Removes all the chunks with the given status. + + Inputs: + - status -- Integer with the chunk status to remove. See c.CHUNK_STATUSES + in constants.py for a list of possible statuses. + + Return: + - counter -- Integer with the number of chunks removed + """ + + counter = 0 + if self.count_chunks(): + dim_name = self.get_name() + print(' Deleting chunks in regionset \"{0}\":'.format(dim_name if dim_name else "selected region files")) + for r in list(self._set.keys()): + counter += self._set[r].remove_problematic_chunks(status) + print("Removed {0} chunks in this regionset.\n".format(counter)) + + return counter + + def fix_problematic_chunks(self, status): + """ Try to fix all the chunks with the given problem. + + Inputs: + - status -- Integer with the chunk status to fix. See c.CHUNK_STATUSES in constants.py + for a list of possible statuses. + + Return: + - counter -- Integer with the number of chunks fixed. + """ + + counter = 0 + if self.count_chunks(): + dim_name = self.get_name() + print('Repairing chunks in regionset \"{0}\":'.format(dim_name if dim_name else "selected region files")) + for r in list(self._set.keys()): + counter += self._set[r].fix_problematic_chunks(status) + print(" Repaired {0} chunks in this regionset.\n".format(counter)) + + return counter + + def remove_entities(self): + """ Removes entities in chunks with the status TOO_MANY_ENTITIES. + + Return: + - counter -- Integer with the number of removed entities. + """ + + counter = 0 + for r in list(self._set.keys()): + counter += self._set[r].remove_entities() + return counter + + def rescan_entities(self, options): + """ Updates the c.CHUNK_TOO_MANY_ENTITIES status of all the chunks in the RegionSet. + + This should be ran when the option entity limit is changed. + """ + + for r in list(self.keys()): + self[r].rescan_entities(options) + + def generate_report(self, standalone): + """ Generates a report with the results of the scan. + + Inputs: + - standalone -- If true the report will be a human readable String. If false the + report will be a dictionary with all the counts of chunks and regions. + + Return if standalone = True: + - text -- A human readable string of text with the results of the scan. + + Return if standlone = False: + - chunk_counts -- Dictionary with all the counts of chunks for all the statuses. To read + it use the CHUNK_* constants. + - region_counts -- Dictionary with all the counts of region files for all the statuses. To read + it use the REGION_* constants. + + """ + + # collect chunk data + chunk_counts = {} + has_chunk_problems = False + for p in c.CHUNK_PROBLEMS: + chunk_counts[p] = self.count_chunks(p) + if chunk_counts[p] != 0: + has_chunk_problems = True + chunk_counts['TOTAL'] = self.count_chunks() + + # collect region data + region_counts = {} + has_region_problems = False + for p in c.REGION_PROBLEMS: + region_counts[p] = self.count_regions(p) + if region_counts[p] != 0: + has_region_problems = True + region_counts['TOTAL'] = self.count_regions() + + # create a text string with a report of all found + if standalone: + text = "" + + # add all chunk info in a table format + text += "\nChunk problems:\n" + if has_chunk_problems: + table_data = [] + table_data.append(['Problem', 'Count']) + for p in c.CHUNK_PROBLEMS: + if chunk_counts[p] != 0: + table_data.append([c.CHUNK_STATUS_TEXT[p], chunk_counts[p]]) + table_data.append(['Total', chunk_counts['TOTAL']]) + text += table(table_data) + else: + text += "No problems found.\n" + + # add all region information + text += "\n\nRegion problems:\n" + if has_region_problems: + table_data = [] + table_data.append(['Problem', 'Count']) + for p in c.REGION_PROBLEMS: + if region_counts[p] != 0: + table_data.append([c.REGION_STATUS_TEXT[p], region_counts[p]]) + table_data.append(['Total', region_counts['TOTAL']]) + text += table(table_data) + + else: + text += "No problems found." + + return text + else: + return chunk_counts, region_counts + + def remove_problematic_regions(self, status): + """ Removes all the regions files with the given status. See the warning! + + Inputs: + - status -- Integer with the status of the region files to remove. + See c.REGION_STATUSES in constants.py for a list. + + Return: + - counter -- An integer with the amount of removed region files. + + Warning! This is NOT the same as removing chunks, this WILL DELETE the region files + from the hard drive. + """ + + counter = 0 + for r in self.list_regions(status): + remove(r.get_path()) + counter += 1 + return counter + +class World: + """ This class stores information and scan results for a Minecraft world. + + Inputs: + - world_path -- String with the path of the world. + + Once scanned, stores all the problems found in it. It also has all the tools + needed to modify the world. + + """ + + def __init__(self, world_path): + self.path = world_path + + # list with RegionSets + self.regionsets = [] + + self.regionsets.append(RegionSet(join(self.path, "region"))) + for directory in glob(join(self.path, "DIM*/region")): + self.regionsets.append(RegionSet(directory, overworld=False)) + + self.regionsets.append(RegionSet(join(self.path, "poi"))) + for directory in glob(join(self.path, "DIM*/poi")): + self.regionsets.append(RegionSet(directory, overworld=False)) + + self.regionsets.append(RegionSet(join(self.path, "entities"))) + for directory in glob(join(self.path, "DIM*/entities")): + self.regionsets.append(RegionSet(directory, overworld=False)) + + # level.dat + # Let's scan level.dat here so we can extract the world name + level_dat_path = join(self.path, "level.dat") + if exists(level_dat_path): + try: + self.level_data = nbt.NBTFile(level_dat_path)["Data"] + self.name = self.level_data["LevelName"].value + self.scanned_level = ScannedDataFile(level_dat_path) + self.scanned_level.status = c.DATAFILE_OK + except Exception: + self.name = None + self.scanned_level = ScannedDataFile(level_dat_path) + self.scanned_level.status = c.DATAFILE_UNREADABLE + else: + self.level_file = None + self.level_data = None + self.name = None + self.scanned_level = ScannedDataFile(level_dat_path) + self.scanned_level.status = c.DATAFILE_UNREADABLE + + # Player files + self.datafilesets = [] + PLAYERS_DIRECTORY = 'playerdata' + OLD_PLAYERS_DIRECTORY = ' players' + STRUCTURES_DIRECTORY = 'data' + + self.players = DataFileSet(join(self.path, PLAYERS_DIRECTORY), + "\nPlayer UUID files:\n") + self.datafilesets.append(self.players) + self.old_players = DataFileSet(join(self.path, OLD_PLAYERS_DIRECTORY), + "\nOld format player files:\n") + self.datafilesets.append(self.old_players) + self.data_files = DataFileSet(join(self.path, STRUCTURES_DIRECTORY), + "\nStructures and map data files:\n") + self.datafilesets.append(self.data_files) + + # Does it look like a world folder? + region_files = False + for region_directory in self.regionsets: + if region_directory: + region_files = True + if region_files: + self.isworld = True + else: + self.isworld = False + # TODO: Make a Exception for this! so we can use try/except + + # Set in scan.py, used in interactive.py + self.scanned = False + + def __str__(self): + counters = self.get_number_regions() + text = "World information:\n" + text += " World path: {0}\n".format(self.path) + text += " World name: {0}\n".format(self.name) + if c.LEVEL_DIR in counters : + text += " Region/Level files: {0}\n".format(counters[c.LEVEL_DIR]) + if c.POI_DIR in counters : + text += " POI files: {0}\n".format(counters[c.POI_DIR]) + if c.ENTITIES_DIR in counters : + text += " Entities files: {0}\n".format(counters[c.ENTITIES_DIR]) + text += " Scanned: {0}".format(str(self.scanned)) + return text + + @property + def has_problems(self): + """ Returns True if the regionset has chunk or region problems and false otherwise. + + Return: + - boolean -- A boolean, True if the world has any problems, false otherwise + + """ + + if self.scanned_level.status in c.DATAFILE_PROBLEMS: + return True + + for d in self.datafilesets: + if d.has_problems: + return True + + for r in self.regionsets: + if r.has_problems: + return True + + return False + + def get_number_regions(self): + """ Returns a dictionnary with the number of regions files in this world + + Return: + - counters -- An dictionnary with the amount of region files. + + """ + + counters = {} + for dim in self.regionsets: + region_type = dim._get_region_type_directory() + if not region_type in counters : + counters[region_type] = 0 + counters[region_type] += len(dim) + + return counters + + def summary(self): + """ Returns a string with a summary of the problems in this world. + + Return: + - text -- A String with a human readable summary of all the problems in this world. + + This method calls the other summary() methods in RegionSet and DataSet. See these + methods for more details. + + """ + + final = "" + + # intro with the world name + final += "{0:#^60}\n".format('') + final += "{0:#^60}\n".format(" World name: {0} ".format(self.name)) + final += "{0:#^60}\n".format('') + + # leve.dat and data files + final += "\nlevel.dat:\n" + if self.scanned_level.status not in c.DATAFILE_PROBLEMS: + final += "\t\'level.dat\' is readable\n" + else: + final += "\t[WARNING]: \'level.dat\' isn't readable, error: {0}\n".format(c.DATAFILE_STATUS_TEXT[self.scanned_level.status]) + + sets = [self.players, + self.old_players, + self.data_files] + + for s in sets: + final += s.title + text = s.summary() + final += text if text else "All files ok.\n" + + final += "\n" + + # chunk info + chunk_info = "" + for regionset in self.regionsets: + title = regionset.get_name() + final += "\n" + title + ":\n" + + # don't add text if there aren't broken chunks + text = regionset.summary() + chunk_info += text if text else "" + final += chunk_info if chunk_info else "All the chunks are ok." + + final += "\n\n" + + return final + + def get_name(self): + """ Returns a string with the name of the world. + + Return: + - name -- String with either the world name as found in level.dat or the last + directory in the world path. + + """ + + if self.name: + return self.name + else: + n = split(self.path) + if n[1] == '': + n = split(n[0])[1] + return n + + def count_regions(self, status=None): + """ Returns an integer with the count of region files with status. + + Inputs: + - status -- An integer from c.REGION_STATUSES to region files with that status. + For a list of status see REGION_STATUSES in constants.py + + Return: + - counter -- An integer with the number of region files with the given status. + + """ + + counter = 0 + for r in self.regionsets: + counter += r.count_regions(status) + return counter + + def count_chunks(self, status=None): + """ Returns an integer with the count of chunks with 'status'. + + Inputs: + - status -- An integer from c.CHUNK_STATUSES to count chunks with that status. + For a list of status see c.CHUNK_STATUSES. + + Return: + - counter -- An integer with the number of chunks with the given status. + + """ + + counter = 0 + for r in self.regionsets: + count = r.count_chunks(status) + counter += count + return counter + + def replace_problematic_chunks(self, backup_worlds, status, entity_limit, delete_entities): + """ Replaces problematic chunks using backups. + + Inputs: + - backup_worlds -- A list of World objects to use as backups. Backup worlds will be used + in a ordered way. + - status -- An integer indicating the status of chunks to be replaced. + See CHUNK_STATUSES in constants.py for a complete list. + - entity_limit -- The threshold to consider a chunk with the status TOO_MANY_ENTITIES. + - delete_entities -- Boolean indicating if the chunks with too_many_entities should have + their entities removed. + + Return: + - counter -- An integer with the number of chunks replaced. + + """ + + counter = 0 + scanned_regions = {} + for regionset in self.regionsets: + for backup in backup_worlds: + # choose the correct regionset based on the dimension + # folder name and the type name (region, POI and entities) + for temp_regionset in backup.regionsets: + if ( temp_regionset._get_dimension_directory() == regionset._get_dimension_directory() and + temp_regionset._get_region_type_directory() == regionset._get_region_type_directory()): + b_regionset = temp_regionset + break + + # this don't need to be aware of region status, it just + # iterates the list returned by list_chunks() + bad_chunks = regionset.list_chunks(status) + + if ( bad_chunks and + b_regionset._get_dimension_directory() != regionset._get_dimension_directory() and + b_regionset._get_region_type_directory() != regionset._get_region_type_directory() ): + print("The regionset \'{0}\' doesn't exist in the backup directory. Skipping this backup directory.".format(regionset._get_dim_type_string())) + else: + for ck in bad_chunks: + global_coords = ck[0] + status_tuple = ck[1] + local_coords = _get_local_chunk_coords(*global_coords) + print("\n{0:-^60}".format(' New chunk to replace. Coords: x = {0}; z = {1} '.format(*global_coords))) + + # search for the region file + backup_region_path, local_coords = b_regionset.locate_chunk(global_coords) + tofix_region_path, _ = regionset.locate_chunk(global_coords) + if exists(backup_region_path): + print("Backup region file found in:\n {0}".format(backup_region_path)) + # Scan the whole region file, pretty slow, but + # absolutely needed to detect sharing offset chunks + # The backups world doesn't change, check if the + # region_file is already scanned: + try: + coords = get_region_coords(split(backup_region_path)[1]) + r = scanned_regions[coords] + except KeyError: + from .scan import scan_region_file + r = scan_region_file(ScannedRegionFile(backup_region_path), entity_limit, delete_entities) + scanned_regions[r.coords] = r + try: + status_tuple = r[local_coords] + except KeyError: + status_tuple = None + + # Retrive the status from status_tuple + if status_tuple == None: + status = c.CHUNK_NOT_CREATED + else: + status = status_tuple[c.TUPLE_STATUS] + + if status == c.CHUNK_OK: + backup_region_file = region.RegionFile(backup_region_path) + working_chunk = backup_region_file.get_chunk(local_coords[0], local_coords[1]) + + print("Replacing...") + # the chunk exists and is healthy, fix it! + tofix_region_file = region.RegionFile(tofix_region_path) + # first unlink the chunk, second write the chunk. + # unlinking the chunk is more secure and the only way to replace chunks with + # a shared offset without overwriting the good chunk + tofix_region_file.unlink_chunk(*local_coords) + tofix_region_file.write_chunk(local_coords[0], local_coords[1], working_chunk) + counter += 1 + print("Chunk replaced using backup dir: {0}".format(backup.path)) + + else: + print("Can't use this backup directory, the chunk has the status: {0}".format(c.CHUNK_STATUS_TEXT[status])) + continue + + else: + print("The region file doesn't exist in the backup directory: {0}".format(backup_region_path)) + + return counter + + def remove_problematic_chunks(self, status): + """ Removes all the chunks with the given status. + + Inputs: + - status -- Integer with the chunk status to remove. See CHUNK_STATUSES in constants.py + for a list of possible statuses. + + Return: + - counter -- Integer with the number of chunks removed + + This method calls remove_problematic_chunks() in the RegionSets. + + """ + + counter = 0 + for regionset in self.regionsets: + counter += regionset.remove_problematic_chunks(status) + return counter + + def fix_problematic_chunks(self, status): + """ Try to fix all the chunks with the given status. + + Inputs: + - status -- Integer with the chunk status to remove. See CHUNK_STATUSES in constants.py + for a list of possible statuses. + + Return: + - counter -- Integer with the number of chunks fixed. + + This method calls remove_problematic_chunks() in the RegionSets. + + """ + + counter = 0 + for regionset in self.regionsets: + counter += regionset.fix_problematic_chunks(status) + return counter + + def replace_problematic_regions(self, backup_worlds, status, entity_limit, delete_entities): + """ Replaces problematic region files using backups. + + Inputs: + - backup_worlds -- A list of World objects to use as backups. Backup worlds will be used + in a ordered way. + - status -- An integer indicating the status of region files to be replaced. + See c.REGION_STATUSES for a complete list. + - entity_limit -- The threshold to consider a chunk with the status TOO_MANY_ENTITIES. + (variable not used, just for inputs to be homogeneous) + - delete_entities -- Boolean indicating if the chunks with too_many_entities should have + their entities removed. (variable not used, just for inputs to be homogeneous) + Return: + - counter -- An integer with the number of chunks replaced. + + Note: entity_limit and delete_entities are not really used here. They are just there to make all + the methods homogeneous. + + """ + + counter = 0 + for regionset in self.regionsets: + for backup in backup_worlds: + # choose the correct regionset based on the dimension + # folder name and the type name (region, POI and entities) + for temp_regionset in backup.regionsets: + if ( temp_regionset._get_dimension_directory() == regionset._get_dimension_directory() and + temp_regionset._get_region_type_directory() == regionset._get_region_type_directory()): + b_regionset = temp_regionset + break + + bad_regions = regionset.list_regions(status) + if ( bad_regions and + b_regionset._get_dimension_directory() != regionset._get_dimension_directory() and + b_regionset._get_region_type_directory() != regionset._get_region_type_directory() ): + print("The regionset \'{0}\' doesn't exist in the backup directory. Skipping this backup directory.".format(regionset._get_dim_type_string())) + else: + for r in bad_regions: + print("\n{0:-^60}".format(' New region file to replace! Coords {0} '.format(r.get_coords()))) + + # search for the region file + + try: + backup_region_path = b_regionset[r.get_coords()].get_path() + except: + backup_region_path = None + tofix_region_path = r.get_path() + + if backup_region_path != None and exists(backup_region_path): + print("Backup region file found in:\n {0}".format(backup_region_path)) + # check the region file, just open it. + try: + backup_region_file = region.RegionFile(backup_region_path) + except region.NoRegionHeader as e: + print("Can't use this backup directory, the error while opening the region file: {0}".format(e)) + continue + except Exception as e: + print("Can't use this backup directory, unknown error: {0}".format(e)) + continue + copy(backup_region_path, tofix_region_path) + print("Region file replaced!") + counter += 1 + else: + print("The region file doesn't exist in the backup directory: {0}".format(backup_region_path)) + + return counter + + def remove_problematic_regions(self, status): + """ Removes all the regions files with the given status. See the warning! + + Inputs: + - status -- Integer with the status of the region files to remove. + See REGION_STATUSES in constants. py for a list. + + Return: + - counter -- An integer with the amount of removed region files. + + Warning! This is NOT the same as removing chunks, this WILL DELETE the region files + from the hard drive. + + """ + + counter = 0 + for regionset in self.regionsets: + counter += regionset.remove_problematic_regions(status) + return counter + + def remove_entities(self): + """ Removes entities in chunks with the status TOO_MANY_ENTITIES. + + Return: + - counter -- Integer with the number of removed entities. + + """ + + counter = 0 + for regionset in self.regionsets: + counter += regionset.remove_entities() + return counter + + def rescan_entities(self, options): + """ Updates the CHUNK_TOO_MANY_ENTITIES status of all the chunks in the RegionSet. + + This should be ran when the option entity limit is changed. + + """ + + for regionset in self.regionsets: + regionset.rescan_entities(options) + + def generate_report(self, standalone): + """ Generates a report with the results of the scan. + + Inputs: + - standalone -- Boolean, if true the report will be a human readable String. If false the + report will be a dictionary with all the counts of chunks and regions. + + Return if standalone = True: + - text -- A human readable string of text with the results of the scan. + + Return if standlone = False: + - chunk_counts -- Dictionary with all the counts of chunks for all the statuses. To read + it use the CHUNK_* constants. + - region_counts -- Dictionary with all the counts of region files for all the statuses. To read + it use the REGION_* constants. + + """ + + # collect chunk data + chunk_counts = {} + has_chunk_problems = False + for p in c.CHUNK_PROBLEMS: + chunk_counts[p] = self.count_chunks(p) + if chunk_counts[p] != 0: + has_chunk_problems = True + chunk_counts['TOTAL'] = self.count_chunks() + + # collect region data + region_counts = {} + has_region_problems = False + for p in c.REGION_PROBLEMS: + region_counts[p] = self.count_regions(p) + if region_counts[p] != 0: + has_region_problems = True + region_counts['TOTAL'] = self.count_regions() + + # create a text string with a report of all found + if standalone: + text = "" + + # add all the player files with problems + text += "\nUnreadable player files:\n" + broken_players = [p for p in self.players._get_list() if p.status in c.DATAFILE_PROBLEMS] + broken_players.extend([p for p in self.old_players._get_list() if p.status in c.DATAFILE_PROBLEMS]) + if broken_players: + broken_player_files = [p.filename for p in broken_players] + text += "\n".join(broken_player_files) + text += "\n" + else: + text += "No problems found.\n" + + # Now all the data files + text += "\nUnreadable data files:\n" + broken_data_files = [d for d in self.data_files._get_list() if d.status in c.DATAFILE_PROBLEMS] + if broken_data_files: + broken_data_filenames = [p.filename for p in broken_data_files] + text += "\n".join(broken_data_filenames) + text += "\n" + else: + text += "No problems found.\n" + + # add all chunk info in a table format + text += "\nChunk problems:\n" + if has_chunk_problems: + table_data = [] + table_data.append(['Problem', 'Count']) + for p in c.CHUNK_PROBLEMS: + if chunk_counts[p] != 0: + table_data.append([c.CHUNK_STATUS_TEXT[p], chunk_counts[p]]) + table_data.append(['Total', chunk_counts['TOTAL']]) + text += table(table_data) + else: + text += "No problems found.\n" + + # add all region information + text += "\n\nRegion problems:\n" + if has_region_problems: + table_data = [] + table_data.append(['Problem', 'Count']) + for p in c.REGION_PROBLEMS: + if region_counts[p] != 0: + table_data.append([c.REGION_STATUS_TEXT[p], region_counts[p]]) + table_data.append(['Total', region_counts['TOTAL']]) + text += table(table_data) + + else: + text += "No problems found." + + return text + else: + return chunk_counts, region_counts + + + +def parse_paths(args): + """ Parse a list of paths to and returns World and a RegionSet objects. + + Keywords arguments: + args -- arguments as argparse got them + + Return: + world_list -- A list of World objects + RegionSet -- A RegionSet object with all the regionfiles found in args + """ + + # windows shell doesn't parse wildcards, parse them here using glob + expanded_args = [] + for arg in args: + earg = glob(arg) + # glob eats away any argument that doesn't match a file, keep those, they will be world folders + if earg: expanded_args.extend(earg) + else: expanded_args.append(arg) + args = expanded_args + + # parese the list of region files and worlds paths + world_list = [] + region_list = [] + warning = False + for arg in args: + if arg[-4:] == ".mca": + region_list.append(arg) + elif arg[-4:] == ".mcr": # ignore pre-anvil region files + if not warning: + print("Warning: Region-Fixer only works with anvil format region files. Ignoring *.mcr files") + warning = True + else: + world_list.append(arg) + + # check if they exist + region_list_tmp = [] + for f in region_list: + if exists(f): + if isfile(f): + region_list_tmp.append(f) + else: + print("Warning: \"{0}\" is not a file. Skipping it and scanning the rest.".format(f)) + else: + print("Warning: The region file {0} doesn't exists. Skipping it and scanning the rest.".format(f)) + region_list = region_list_tmp + + # init the world objects + world_list = parse_world_list(world_list) + + return world_list, RegionSet(region_list = region_list) + + +def parse_world_list(world_path_list): + """ Parses a world path list. Returns a list of World objects. + + Keywords arguments: + world_path_list -- A list of string with paths where minecraft worlds are supposed to be + + Return: + world_list -- A list of World objects using the paths from the input + + Parses a world path list checking if they exists and are a minecraft + world folders. Returns a list of World objects. Prints errors for the + paths that are not minecraft worlds. + """ + + world_list = [] + for d in world_path_list: + if exists(d): + w = World(d) + if w.isworld: + world_list.append(w) + else: + print("Warning: The folder {0} doesn't look like a minecraft world. I'll skip it.".format(d)) + else: + print("Warning: The folder {0} doesn't exist. I'll skip it.".format(d)) + return world_list + + +def parse_backup_list(world_backup_dirs): + """ Generates a list with the input of backup dirs containing the + world objects of valid world directories.""" + + directories = world_backup_dirs.split(',') + backup_worlds = parse_world_list(directories) + return backup_worlds + + +def delete_entities(region_file, x, z): + """ Removes entities in chunks with the status TOO_MANY_ENTITIES. + + Keyword entities: + - x -- Integer, X local coordinate of the chunk in the region files + - z -- Integer, Z local coordinate of the chunk in the region files + - region_file -- RegionFile object where the chunk is stored + + Return: + - counter -- Integer with the number of removed entities. + + This function is used in scan.py. + + """ + + chunk = region_file.get_chunk(x, z) + chunk_type = get_chunk_type(chunk) + empty_tag_list = nbt.TAG_List(nbt.TAG_Byte, '', 'Entities') + + if chunk_type == c.LEVEL_DIR : # Region file + if "DataVersion" in chunk and chunk["DataVersion"].value >= 2844 : # Snapshot 21w43a (1.18) + counter = len(chunk['entities']) + chunk['entities'] = empty_tag_list + else : + counter = len(chunk['Level']['Entities']) + chunk['Level']['Entities'] = empty_tag_list + + elif chunk_type == c.ENTITIES_DIR : # Entities file (>=1.17) + counter = len(chunk['Entities']) + chunk['Entities'] = empty_tag_list + + else : + raise AssertionError("Unsupported chunk type in delete_entities().") + + region_file.write_chunk(x, z, chunk) + + return counter + + +def _get_local_chunk_coords(chunkx, chunkz): + """ Gives the chunk local coordinates from the global coordinates. + + Inputs: + - chunkx -- Integer, X chunk global coordinate in the world. + - chunkz -- Integer, Z chunk global coordinate in the world. + + Return: + - x, z -- X and Z local coordinates of the chunk in the region file. + + """ + + return chunkx % 32, chunkz % 32 + + +def get_chunk_region(chunkX, chunkZ): + """ Returns the name of the region file given global chunk coordinates. + + Inputs: + - chunkx -- Integer, X chunk global coordinate in the world. + - chunkz -- Integer, Z chunk global coordinate in the world. + + Return: + - region_name -- A string with the name of the region file where the chunk + should be. + + """ + + regionX = chunkX // 32 + regionZ = chunkZ // 32 + + region_name = 'r.' + str(regionX) + '.' + str(regionZ) + '.mca' + + return region_name + + +def get_chunk_type(chunk): + """Get the type of the chunk (Region/level, POIs or entities) + + Input: + - chunk -- A chunk, from the NBT module + + Return: + - type -- The chunk type (LEVEL_DIR, POI_DIR or ENTITIES_DIR) + """ + + # DataVersion was introduced in snapshot 15w32a (1.9) + # https://minecraft.wiki/w/Data_version + data_version = 0 + if "DataVersion" in chunk: + data_version = chunk["DataVersion"].value + + # Region/level < 21w43a (1.17) + if data_version < 2844 and "Level" in chunk: + return c.LEVEL_DIR + + # Region/level >= 21w43a (1.18) + # The "or" is important, because some tags doesn't seem to be mandatory + if data_version >= 2844 and ("structures" in chunk or "sections" in chunk): + return c.LEVEL_DIR + + # POIs >= 1.14 (Which snapshot ?) + # I couldn't find when POI files were added + # But it's certainly a snapshot after 18w43a (DataVersion = 1901) + if data_version >= 1901 and "Sections" in chunk: + return c.POI_DIR + + # Entities >= 20w45a (1.17) + if data_version >= 2681 and "Entities" in chunk: + return c.ENTITIES_DIR + + raise AssertionError("Unrecognized chunk type in get_chunk_type().") + + +def get_chunk_data_coords(nbt_file): + """ Gets and returns the coordinates stored in the NBT structure of the chunk. + + Inputs: + - nbt_file -- An NBT file. From the nbt module. + + Return: + - coordX, coordZ -- Integers with the X and Z global coordinates of the chunk. + + Do not confuse with the coordinates returned by get_global_coords, which could be different, + marking this chunk as wrong_located. + + """ + + chunk_type = get_chunk_type(nbt_file) + + # Region file + if chunk_type == c.LEVEL_DIR : + # Since snapshot 21w43a (1.18), "Level" tag doesn't exist anymore + if "DataVersion" in nbt_file and nbt_file["DataVersion"].value >= 2844 : + level = nbt_file + else : + level = nbt_file.__getitem__('Level') + + coordX = level.__getitem__('xPos').value + coordZ = level.__getitem__('zPos').value + + # Entities file : + elif chunk_type == c.ENTITIES_DIR : + coordX, coordZ = nbt_file.__getitem__('Position').value + + else : + raise AssertionError("Unrecognized chunk in get_chunk_data_coords().") + + return coordX, coordZ + + +def get_region_coords(filename): + """ Get and return a region file coordinates from path. + + Inputs: + - filename -- Filename or path of the region file. + + Return: + - coordX, coordZ -- X and z coordinates of the region file. + + """ + + l = filename.split('.') + coordX = int(l[1]) + coordZ = int(l[2]) + + return coordX, coordZ + + +def get_global_chunk_coords(region_name, chunkX, chunkZ): + """ Get and return a region file coordinates from path. + + Inputs: + - region_name -- String with filename or path of the region file. + - chunkX -- Integer, X local coordinate of the chunk + - chunkZ -- Integer, Z local coordinate of the chunk + + Return: + - coordX, coordZ -- X and z global coordinates of the + chunk in that region file. + + """ + + regionX, regionZ = get_region_coords(region_name) + chunkX += regionX * 32 + chunkZ += regionZ * 32 + + return chunkX, chunkZ diff --git a/regionfixer_gui.py b/regionfixer_gui.py new file mode 100644 index 0000000..9ab54f8 --- /dev/null +++ b/regionfixer_gui.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +from multiprocessing import freeze_support +import sys + +# Needed for the gui +import regionfixer_core +import nbt + +from gui import Starter +if __name__ == '__main__': + freeze_support() + s = Starter() + value = s.run() + sys.exit(value) diff --git a/scan.py b/scan.py deleted file mode 100644 index 324126f..0000000 --- a/scan.py +++ /dev/null @@ -1,430 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -# -# Region Fixer. -# Fix your region files with a backup copy of your Minecraft world. -# Copyright (C) 2011 Alejandro Aguilera (Fenixin) -# https://github.com/Fenixin/Minecraft-Region-Fixer -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . -# - -import nbt.region as region -import nbt.nbt as nbt -#~ from nbt.region import STATUS_CHUNK_OVERLAPPING, STATUS_CHUNK_MISMATCHED_LENGTHS - #~ - STATUS_CHUNK_ZERO_LENGTH - #~ - STATUS_CHUNK_IN_HEADER - #~ - STATUS_CHUNK_OUT_OF_FILE - #~ - STATUS_CHUNK_OK - #~ - STATUS_CHUNK_NOT_CREATED -from os.path import split, join -import progressbar -import multiprocessing -from multiprocessing import queues -import world -import time - -import sys -import traceback - -class ChildProcessException(Exception): - """Takes the child process traceback text and prints it as a - real traceback with asterisks everywhere.""" - def __init__(self, error): - # Helps to see wich one is the child process traceback - traceback = error[2] - print "*"*10 - print "*** Error while scanning:" - print "*** ", error[0] - print "*"*10 - print "*** Printing the child's Traceback:" - print "*** Exception:", traceback[0], traceback[1] - for tb in traceback[2]: - print "*"*10 - print "*** File {0}, line {1}, in {2} \n*** {3}".format(*tb) - print "*"*10 - -class FractionWidget(progressbar.ProgressBarWidget): - """ Convenience class to use the progressbar.py """ - def __init__(self, sep=' / '): - self.sep = sep - - def update(self, pbar): - return '%2d%s%2d' % (pbar.currval, self.sep, pbar.maxval) - -def scan_world(world_obj, options): - """ Scans a world folder including players, region folders and - level.dat. While scanning prints status messages. """ - w = world_obj - # scan the world dir - print "Scanning directory..." - - if not w.scanned_level.path: - print "Warning: No \'level.dat\' file found!" - - if w.players: - print "There are {0} region files and {1} player files in the world directory.".format(\ - w.get_number_regions(), len(w.players)) - else: - print "There are {0} region files in the world directory.".format(\ - w.get_number_regions()) - - # check the level.dat file and the *.dat files in players directory - print "\n{0:-^60}".format(' Checking level.dat ') - - if not w.scanned_level.path: - print "[WARNING!] \'level.dat\' doesn't exist!" - else: - if w.scanned_level.readable == True: - print "\'level.dat\' is readable" - else: - print "[WARNING!]: \'level.dat\' is corrupted with the following error/s:" - print "\t {0}".format(w.scanned_level.status_text) - - print "\n{0:-^60}".format(' Checking player files ') - # TODO multiprocessing! - # Probably, create a scanner object with a nice buffer of logger for text and logs and debugs - if not w.players: - print "Info: No player files to scan." - else: - scan_all_players(w) - all_ok = True - for name in w.players: - if w.players[name].readable == False: - print "[WARNING]: Player file {0} has problems.\n\tError: {1}".format(w.players[name].filename, w.players[name].status_text) - all_ok = False - if all_ok: - print "All player files are readable." - - # SCAN ALL THE CHUNKS! - if w.get_number_regions == 0: - print "No region files to scan!" - else: - for r in w.regionsets: - if r.regions: - print "\n{0:-^60}".format(' Scanning the {0} '.format(r.get_name())) - scan_regionset(r, options) - w.scanned = True - - -def scan_player(scanned_dat_file): - """ At the moment only tries to read a .dat player file. It returns - 0 if it's ok and 1 if has some problem """ - - s = scanned_dat_file - try: - player_dat = nbt.NBTFile(filename = s.path) - s.readable = True - except Exception, e: - s.readable = False - s.status_text = e - - -def scan_all_players(world_obj): - """ Scans all the players using the scan_player function. """ - - for name in world_obj.players: - scan_player(world_obj.players[name]) - - -def scan_region_file(scanned_regionfile_obj, options): - """ Given a scanned region file object with the information of a - region files scans it and returns the same obj filled with the - results. - - If delete_entities is True it will delete entities while - scanning - - entiti_limit is the threshold tof entities to conisder a chunk - with too much entities problems. - """ - o = options - delete_entities = o.delete_entities - entity_limit = o.entity_limit - try: - r = scanned_regionfile_obj - # counters of problems - chunk_count = 0 - corrupted = 0 - wrong = 0 - entities_prob = 0 - shared = 0 - # used to detect chunks sharing headers - offsets = {} - filename = r.filename - # try to open the file and see if we can parse the header - try: - region_file = region.RegionFile(r.path) - except region.NoRegionHeader: # the region has no header - r.status = world.REGION_TOO_SMALL - return r - except IOError, e: - print "\nWARNING: I can't open the file {0} !\nThe error is \"{1}\".\nTypical causes are file blocked or problems in the file system.\n".format(filename,e) - r.status = world.REGION_UNREADABLE - r.scan_time = time.time() - print "Note: this region file won't be scanned and won't be taken into acount in the summaries" - # TODO count also this region files - return r - except: # whatever else print an error and ignore for the scan - # not really sure if this is a good solution... - print "\nWARNING: The region file \'{0}\' had an error and couldn't be parsed as region file!\nError:{1}\n".format(join(split(split(r.path)[0])[1], split(r.path)[1]),sys.exc_info()[0]) - print "Note: this region file won't be scanned and won't be taken into acount." - print "Also, this may be a bug. Please, report it if you have the time.\n" - return None - - try:# start the scanning of chunks - - for x in range(32): - for z in range(32): - - # start the actual chunk scanning - g_coords = r.get_global_chunk_coords(x, z) - chunk, c = scan_chunk(region_file, (x,z), g_coords, o) - if c != None: # chunk not created - r.chunks[(x,z)] = c - chunk_count += 1 - else: continue - if c[TUPLE_STATUS] == world.CHUNK_OK: - continue - elif c[TUPLE_STATUS] == world.CHUNK_TOO_MANY_ENTITIES: - # deleting entities is in here because parsing a chunk with thousands of wrong entities - # takes a long time, and once detected is better to fix it at once. - if delete_entities: - world.delete_entities(region_file, x, z) - print "Deleted {0} entities in chunk ({1},{2}) of the region file: {3}".format(c[TUPLE_NUM_ENTITIES], x, z, r.filename) - # entities removed, change chunk status to OK - r.chunks[(x,z)] = (0, world.CHUNK_OK) - - else: - entities_prob += 1 - # This stores all the entities in a file, - # comes handy sometimes. - #~ pretty_tree = chunk['Level']['Entities'].pretty_tree() - #~ name = "{2}.chunk.{0}.{1}.txt".format(x,z,split(region_file.filename)[1]) - #~ archivo = open(name,'w') - #~ archivo.write(pretty_tree) - - elif c[TUPLE_STATUS] == world.CHUNK_CORRUPTED: - corrupted += 1 - elif c[TUPLE_STATUS] == world.CHUNK_WRONG_LOCATED: - wrong += 1 - - # Now check for chunks sharing offsets: - # Please note! region.py will mark both overlapping chunks - # as bad (the one stepping outside his territory and the - # good one). Only wrong located chunk with a overlapping - # flag are really BAD chunks! Use this criterion to - # discriminate - metadata = region_file.metadata - sharing = [k for k in metadata if ( - metadata[k].status == region.STATUS_CHUNK_OVERLAPPING and - r[k][TUPLE_STATUS] == world.CHUNK_WRONG_LOCATED)] - shared_counter = 0 - for k in sharing: - r[k] = (r[k][TUPLE_NUM_ENTITIES], world.CHUNK_SHARED_OFFSET) - shared_counter += 1 - - except KeyboardInterrupt: - print "\nInterrupted by user\n" - # TODO this should't exit - sys.exit(1) - - r.chunk_count = chunk_count - r.corrupted_chunks = corrupted - r.wrong_located_chunks = wrong - r.entities_prob = entities_prob - r.shared_offset = shared_counter - r.scan_time = time.time() - r.status = world.REGION_OK - return r - - # Fatal exceptions: - except: - # anything else is a ChildProcessException - except_type, except_class, tb = sys.exc_info() - r = (r.path, r.coords, (except_type, except_class, traceback.extract_tb(tb))) - return r - -def multithread_scan_regionfile(region_file): - """ Does the multithread stuff for scan_region_file """ - r = region_file - o = multithread_scan_regionfile.options - - # call the normal scan_region_file with this parameters - r = scan_region_file(r,o) - - # exceptions will be handled in scan_region_file which is in the - # single thread land - multithread_scan_regionfile.q.put(r) - - - -def scan_chunk(region_file, coords, global_coords, options): - """ Takes a RegionFile obj and the local coordinatesof the chunk as - inputs, then scans the chunk and returns all the data.""" - try: - chunk = region_file.get_chunk(*coords) - data_coords = world.get_chunk_data_coords(chunk) - num_entities = len(chunk["Level"]["Entities"]) - if data_coords != global_coords: - status = world.CHUNK_WRONG_LOCATED - status_text = "Mismatched coordinates (wrong located chunk)." - scan_time = time.time() - elif num_entities > options.entity_limit: - status = world.CHUNK_TOO_MANY_ENTITIES - status_text = "The chunks has too many entities (it has {0}, and it's more than the limit {1})".format(num_entities, options.entity_limit) - scan_time = time.time() - else: - status = world.CHUNK_OK - status_text = "OK" - scan_time = time.time() - - except region.InconceivedChunk as e: - chunk = None - data_coords = None - num_entities = None - status = world.CHUNK_NOT_CREATED - status_text = "The chunk doesn't exist" - scan_time = time.time() - - except region.RegionHeaderError as e: - error = "Region header error: " + e.msg - status = world.CHUNK_CORRUPTED - status_text = error - scan_time = time.time() - chunk = None - data_coords = None - global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) - num_entities = None - - except region.ChunkDataError as e: - error = "Chunk data error: " + e.msg - status = world.CHUNK_CORRUPTED - status_text = error - scan_time = time.time() - chunk = None - data_coords = None - global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) - num_entities = None - - except region.ChunkHeaderError as e: - error = "Chunk herader error: " + e.msg - status = world.CHUNK_CORRUPTED - status_text = error - scan_time = time.time() - chunk = None - data_coords = None - global_coords = world.get_global_chunk_coords(split(region_file.filename)[1], coords[0], coords[1]) - num_entities = None - - return chunk, (num_entities, status) if status != world.CHUNK_NOT_CREATED else None - -#~ TUPLE_COORDS = 0 -#~ TUPLE_DATA_COORDS = 0 -#~ TUPLE_GLOBAL_COORDS = 2 -TUPLE_NUM_ENTITIES = 0 -TUPLE_STATUS = 1 - -#~ def scan_and_fill_chunk(region_file, scanned_chunk_obj, options): - #~ """ Takes a RegionFile obj and a ScannedChunk obj as inputs, - #~ scans the chunk, fills the ScannedChunk obj and returns the chunk - #~ as a NBT object.""" -#~ - #~ c = scanned_chunk_obj - #~ chunk, region_file, c.h_coords, c.d_coords, c.g_coords, c.num_entities, c.status, c.status_text, c.scan_time, c.region_path = scan_chunk(region_file, c.h_coords, options) - #~ return chunk - -def _mp_pool_init(regionset,options,q): - """ Function to initialize the multiprocessing in scan_regionset. - Is used to pass values to the child process. """ - multithread_scan_regionfile.regionset = regionset - multithread_scan_regionfile.q = q - multithread_scan_regionfile.options = options - - -def scan_regionset(regionset, options): - """ This function scans all te region files in a regionset object - and fills the ScannedRegionFile obj with the results - """ - - total_regions = len(regionset.regions) - total_chunks = 0 - corrupted_total = 0 - wrong_total = 0 - entities_total = 0 - too_small_total = 0 - unreadable = 0 - - # init progress bar - if not options.verbose: - pbar = progressbar.ProgressBar( - widgets=['Scanning: ', FractionWidget(), ' ', progressbar.Percentage(), ' ', progressbar.Bar(left='[',right=']'), ' ', progressbar.ETA()], - maxval=total_regions) - - # queue used by processes to pass finished stuff - q = queues.SimpleQueue() - pool = multiprocessing.Pool(processes=options.processes, - initializer=_mp_pool_init,initargs=(regionset,options,q)) - - if not options.verbose: - pbar.start() - - # start the pool - # Note to self: every child process has his own memory space, - # that means every obj recived by them will be a copy of the - # main obj - result = pool.map_async(multithread_scan_regionfile, regionset.list_regions(None), max(1,total_regions//options.processes)) - - # printing status - region_counter = 0 - - while not result.ready() or not q.empty(): - time.sleep(0.01) - if not q.empty(): - r = q.get() - if r == None: # something went wrong scanning this region file - # probably a bug... don't know if it's a good - # idea to skip it - continue - if not isinstance(r,world.ScannedRegionFile): - raise ChildProcessException(r) - else: - corrupted, wrong, entities_prob, shared_offset, num_chunks = r.get_counters() - filename = r.filename - # the obj returned is a copy, overwrite it in regionset - regionset[r.get_coords()] = r - corrupted_total += corrupted - wrong_total += wrong - total_chunks += num_chunks - entities_total += entities_prob - if r.status == world.REGION_TOO_SMALL: - too_small_total += 1 - elif r.status == world.REGION_UNREADABLE: - unreadable += 1 - region_counter += 1 - if options.verbose: - if r.status == world.REGION_OK: - stats = "(c: {0}, w: {1}, tme: {2}, so: {3}, t: {4})".format( corrupted, wrong, entities_prob, shared_offset, num_chunks) - elif r.status == world.REGION_TOO_SMALL: - stats = "(Error: not a region file)" - elif r.status == world.REGION_UNREADABLE: - stats = "(Error: unreadable region file)" - print "Scanned {0: <12} {1:.<43} {2}/{3}".format(filename, stats, region_counter, total_regions) - else: - pbar.update(region_counter) - - if not options.verbose: pbar.finish() - - regionset.scanned = True diff --git a/setup.py b/setup.py index ff3a335..684b3bc 100644 --- a/setup.py +++ b/setup.py @@ -1,9 +1,196 @@ +# taken from: http://www.wiki.wxpython.org/py2exe-python26 + +# ======================================================# +# File automagically generated by GUI2Exe version 0.3 +# Andrea Gavana, 01 April 2007 +# ======================================================# + +# Let's start with some default (for me) imports... + from distutils.core import setup -import nbt import py2exe -import sys +import glob +import os +import zlib +import shutil + +from regionfixer_core import version as cli_version +#=============================================================================== +# from gui import version as gui_version +#=============================================================================== + + +# Remove the build folder +shutil.rmtree("build", ignore_errors=True) + +# do the same for dist folder +shutil.rmtree("dist", ignore_errors=True) + +MANIFEST_TEMPLATE = """ + + + + %(prog)s + + + + + + + + + + + + + + + + + + + + +""" + +class Target(object): + """ A simple class that holds information on our executable file. """ + def __init__(self, **kw): + """ Default class constructor. Update as you need. """ + self.__dict__.update(kw) + + +# Ok, let's explain why I am doing that. +# Often, data_files, excludes and dll_excludes (but also resources) +# can be very long list of things, and this will clutter too much +# the setup call at the end of this file. So, I put all the big lists +# here and I wrap them using the textwrap module. + +data_files = ['COPYING.txt', 'README.rst', 'CONTRIBUTORS.txt', 'DONORS.txt', 'icon.ico'] + +includes = [] +excludes = ['_gtkagg', '_tkagg', 'bsddb', 'curses', 'email', 'pywin.debugger', + 'pywin.debugger.dbgcon', 'pywin.dialogs', 'tcl', + 'Tkconstants', 'Tkinter'] +packages = [] +dll_excludes = ['libgdk-win32-2.0-0.dll', 'libgobject-2.0-0.dll', 'tcl84.dll', + 'tk84.dll', + 'MSVCP90.dll', 'mswsock.dll', 'powrprof.dll'] +icon_resources = [(1, 'icon.ico')] +bitmap_resources = [] +other_resources = [] +other_resources = [(24, 1, MANIFEST_TEMPLATE % dict(prog="MyAppName"))] + + +# This is a place where the user custom code may go. You can do almost +# whatever you want, even modify the data_files, includes and friends +# here as long as they have the same variable name that the setup call +# below is expecting. + + +# +# The following will copy the MSVC run time dll's +# (msvcm90.dll, msvcp90.dll and msvcr90.dll) and +# the Microsoft.VC90.CRT.manifest which I keep in the +# "Py26MSdlls" folder to the dist folder +# +# depending on wx widgets you use, you might need to add +# gdiplus.dll to the above collection + +py26MSdll = glob.glob(r"c:\Dev\Py26MSdlls-9.0.21022.8\msvc\*.*") + +# install the MSVC 9 runtime dll's into the application folder +data_files += [("", py26MSdll),] + +# I found on some systems one has to put them into sub-folders. +##data_files += [("Microsoft.VC90.CRT", py26MSdll), +## ("lib\Microsoft.VC90.CRT", py26MSdll)] + + + +# Ok, now we are going to build our target class. +# I chose this building strategy as it works perfectly for me :-D + +#=============================================================================== +# GUI_Target = Target( +# # what to build +# script = "regionfixer_gui.py", +# icon_resources = icon_resources, +# bitmap_resources = bitmap_resources, +# other_resources = other_resources, +# dest_base = "regionfixer_gui", +# version = gui_version.version_string, +# company_name = "No Company", +# copyright = "Copyright (C) 2020 Alejandro Aguilera", +# name = "Region Fixer GUI" +# ) +#=============================================================================== + +CLI_Target = Target( + # what to build + script = "regionfixer.py", + icon_resources = icon_resources, + bitmap_resources = bitmap_resources, + other_resources = other_resources, + dest_base = "regionfixer", + version = cli_version.version_string, + company_name = "No Company", + copyright = "Copyright (C) 2019 Alejandro Aguilera", + name = "Region Fixer" + ) + + +# That's serious now: we have all (or almost all) the options py2exe +# supports. I put them all even if some of them are usually defaulted +# and not used. Some of them I didn't even know about. + +setup( + + data_files = data_files, + + options = {"py2exe": {"compressed": 2, + "optimize": 2, + "includes": includes, + "excludes": excludes, + "packages": packages, + "dll_excludes": dll_excludes, + "bundle_files": 2, + "dist_dir": "dist", + "xref": False, + "skip_archive": False, + "ascii": False, + "custom_boot_script": '', + } + }, + + zipfile = "lib\library.zip", + console = [CLI_Target] + #windows = [GUI_Target] + ) + +# This is a place where any post-compile code may go. +# You can add as much code as you want, which can be used, for example, +# to clean up your folders or to do some particular post-compilation +# actions. + +# And we are done. That's a setup script :-D -if sys.argv[1] == "py2exe": - setup(console=['region-fixer.py'], data_files=['COPYING.txt','README.rst','CONTRIBUTORS.txt','DONORS.txt']) -else: - print "Use \'python setup.py py2exe\' to build a windows executable." diff --git a/util.py b/util.py deleted file mode 100644 index 2715bd0..0000000 --- a/util.py +++ /dev/null @@ -1,190 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -# -# Region Fixer. -# Fix your region files with a backup copy of your Minecraft world. -# Copyright (C) 2011 Alejandro Aguilera (Fenixin) -# https://github.com/Fenixin/Minecraft-Region-Fixer -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . -# - -import platform -from os.path import join, split, exists, isfile -import world - -# stolen from minecraft overviewer -# https://github.com/overviewer/Minecraft-Overviewer/ -def is_bare_console(): - """Returns true if Overviewer is running in a bare console in - Windows, that is, if overviewer wasn't started in a cmd.exe - session. - """ - if platform.system() == 'Windows': - try: - import ctypes - GetConsoleProcessList = ctypes.windll.kernel32.GetConsoleProcessList - num = GetConsoleProcessList(ctypes.byref(ctypes.c_int(0)), ctypes.c_int(1)) - if (num == 1): - return True - - except Exception: - pass - return False - -def entitle(text, level = 0): - """ Put the text in a title with lot's of hashes everywhere. """ - t = '' - if level == 0: - t += "\n" - t += "{0:#^60}\n".format('') - t += "{0:#^60}\n".format(' ' + text + ' ') - t += "{0:#^60}\n".format('') - return t - - -def table(columns): - """ Gets a list with lists in which each list is a column, - returns a text string with a table. """ - - def get_max_len(l): - """ Takes a list and returns the length of the biggest - element """ - m = 0 - for e in l: - if len(str(e)) > m: - m = len(e) - return m - - text = "" - # stores the size of the biggest element in that column - ml = [] - # fill up ml - for c in columns: - m = 0 - t = get_max_len(c) - if t > m: - m = t - ml.append(m) - # get the total width of the table: - ml_total = 0 - for i in range(len(ml)): - ml_total += ml[i] + 2 # size of each word + 2 spaces - ml_total += 1 + 2# +1 for the separator | and +2 for the borders - text += "-"*ml_total + "\n" - # all the columns have the same number of rows - row = get_max_len(columns) - for r in range(row): - line = "|" - # put all the elements in this row together with spaces - for i in range(len(columns)): - line += "{0: ^{width}}".format(columns[i][r],width = ml[i] + 2) - # add a separator for the first column - if i == 0: - line += "|" - - text += line + "|" + "\n" - if r == 0: - text += "-"*ml_total + "\n" - text += "-"*ml_total - return text - - -def parse_chunk_list(chunk_list, world_obj): - """ Generate a list of chunks to use with world.delete_chunk_list. - - It takes a list of global chunk coordinates and generates a list of - tuples containing: - - (region fullpath, chunk X, chunk Z) - - """ - # this is not used right now - parsed_list = [] - for line in chunk_list: - try: - chunk = eval(line) - except: - print "The chunk {0} is not valid.".format(line) - continue - region_name = world.get_chunk_region(chunk[0], chunk[1]) - fullpath = join(world_obj.world_path, "region", region_name) - if fullpath in world_obj.all_mca_files: - parsed_list.append((fullpath, chunk[0], chunk[1])) - else: - print "The chunk {0} should be in the region file {1} and this region files doesn't extist!".format(chunk, fullpath) - - return parsed_list - -def parse_paths(args): - """ Parse the list of args passed to region-fixer.py and returns a - RegionSet object with the list of regions and a list of World - objects. """ - # parese the list of region files and worlds paths - world_list = [] - region_list = [] - warning = False - for arg in args: - if arg[-4:] == ".mca": - region_list.append(arg) - elif arg[-4:] == ".mcr": # ignore pre-anvil region files - if not warning: - print "Warning: Region-Fixer only works with anvil format region files. Ignoring *.mcr files" - warning = True - else: - world_list.append(arg) - - # check if they exist - region_list_tmp = [] - for f in region_list: - if exists(f): - if isfile(f): - region_list_tmp.append(f) - else: - print "Warning: \"{0}\" is not a file. Skipping it and scanning the rest.".format(f) - else: - print "Warning: The region file {0} doesn't exists. Skipping it and scanning the rest.".format(f) - region_list = region_list_tmp - - # init the world objects - world_list = parse_world_list(world_list) - - return world_list, world.RegionSet(region_list = region_list) - -def parse_world_list(world_path_list): - """ Parses a world list checking if they exists and are a minecraft - world folders. Returns a list of World objects. """ - - tmp = [] - for d in world_path_list: - if exists(d): - w = world.World(d) - if w.isworld: - tmp.append(w) - else: - print "Warning: The folder {0} doesn't look like a minecraft world. I'll skip it.".format(d) - else: - print "Warning: The folder {0} doesn't exist. I'll skip it.".format(d) - return tmp - - - -def parse_backup_list(world_backup_dirs): - """ Generates a list with the input of backup dirs containing the - world objects of valid world directories.""" - - directories = world_backup_dirs.split(',') - backup_worlds = parse_world_list(directories) - return backup_worlds diff --git a/world.py b/world.py deleted file mode 100644 index e169c1d..0000000 --- a/world.py +++ /dev/null @@ -1,974 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -# -# Region Fixer. -# Fix your region files with a backup copy of your Minecraft world. -# Copyright (C) 2011 Alejandro Aguilera (Fenixin) -# https://github.com/Fenixin/Minecraft-Region-Fixer -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . -# - -import nbt.region as region -import nbt.nbt as nbt -from util import table - -from glob import glob -from os.path import join, split, exists -from os import remove -from shutil import copy - -import time - -# Constants: -# Used to mark the status of a chunks: -CHUNK_NOT_CREATED = -1 -CHUNK_OK = 0 -CHUNK_CORRUPTED = 1 -CHUNK_WRONG_LOCATED = 2 -CHUNK_TOO_MANY_ENTITIES = 3 -CHUNK_SHARED_OFFSET = 4 -CHUNK_STATUS_TEXT = {CHUNK_NOT_CREATED:"Not created", - CHUNK_OK:"OK", - CHUNK_CORRUPTED:"Corrupted", - CHUNK_WRONG_LOCATED:"Wrong located", - CHUNK_TOO_MANY_ENTITIES:"Too many entities", - CHUNK_SHARED_OFFSET:"Sharing offset"} - -CHUNK_PROBLEMS = [CHUNK_CORRUPTED, CHUNK_WRONG_LOCATED, CHUNK_TOO_MANY_ENTITIES, CHUNK_SHARED_OFFSET] - -CHUNK_PROBLEMS_ARGS = {CHUNK_CORRUPTED:'corrupted',CHUNK_WRONG_LOCATED:'wrong',CHUNK_TOO_MANY_ENTITIES:'entities',CHUNK_SHARED_OFFSET:'sharing'} -# list with problem status-text tuples -CHUNK_PROBLEMS_ITERATOR = [] -for problem in CHUNK_PROBLEMS: - CHUNK_PROBLEMS_ITERATOR.append((problem, CHUNK_STATUS_TEXT[problem], CHUNK_PROBLEMS_ARGS[problem])) - - - -# Used to mark the status of region files: -REGION_OK = 10 -REGION_TOO_SMALL = 11 -REGION_UNREADABLE = 12 -REGION_STATUS_TEXT = {REGION_OK: "Ok", REGION_TOO_SMALL: "Too small", REGION_UNREADABLE: "Unreadable"} - -REGION_PROBLEMS = [REGION_TOO_SMALL] -REGION_PROBLEMS_ARGS = {REGION_TOO_SMALL: 'too-small'} - -# list with problem status-text tuples -REGION_PROBLEMS_ITERATOR = [] -for problem in REGION_PROBLEMS: - try: - REGION_PROBLEMS_ITERATOR.append((problem, REGION_STATUS_TEXT[problem], REGION_PROBLEMS_ARGS[problem])) - except KeyError: - pass - -REGION_PROBLEMS_ARGS = {REGION_TOO_SMALL:'too-small'} - -# Used to know where to look in a chunk status tuple -#~ TUPLE_COORDS = 0 -#~ TUPLE_DATA_COORDS = 0 -#~ TUPLE_GLOBAL_COORDS = 2 -TUPLE_NUM_ENTITIES = 0 -TUPLE_STATUS = 1 - -# Dimension names: -DIMENSION_NAMES = { "region":"Overworld", "DIM1":"The End", "DIM-1":"Nether" } - -class ScannedDatFile(object): - def __init__(self, path = None, readable = None, status_text = None): - self.path = path - if self.path and exists(self.path): - self.filename = split(path)[1] - else: - self.filename = None - self.readable = readable - self.status_text = status_text - - def __str__(self): - text = "NBT file:" + str(self.path) + "\n" - text += "\tReadable:" + str(self.readable) + "\n" - return text - -class ScannedChunk(object): - """ Stores all the results of the scan. Not used at the moment, it - prette nice but takes an huge amount of memory. """ - # WARNING: not used at the moment, it probably has bugs ans is - # outdated - # The problem with it was it took too much memory. It has been - # remplaced with a tuple - def __init__(self, header_coords, global_coords = None, data_coords = None, status = None, num_entities = None, scan_time = None, region_path = None): - """ Inits the object with all the scan information. """ - self.h_coords = header_coords - self.g_coords = global_coords - self.d_coords = data_coords - self.status = status - self.status_text = None - self.num_entities = num_entities - self.scan_time = scan_time - self.region_path = region_path - - def __str__(self): - text = "Chunk with header coordinates:" + str(self.h_coords) + "\n" - text += "\tData coordinates:" + str(self.d_coords) + "\n" - text +="\tGlobal coordinates:" + str(self.g_coords) + "\n" - text += "\tStatus:" + str(self.status_text) + "\n" - text += "\tNumber of entities:" + str(self.num_entities) + "\n" - text += "\tScan time:" + time.ctime(self.scan_time) + "\n" - return text - - def get_path(): - """ Returns the path of the region file. """ - return self.region_path - - def rescan_entities(self, options): - """ Updates the status of the chunk when the the option - entity limit is changed. """ - if self.num_entities >= options.entity_limit: - self.status = CHUNK_TOO_MANY_ENTITIES - self.status_text = CHUNK_STATUS_TEXT[CHUNK_TOO_MANY_ENTITIES] - else: - self.status = CHUNK_OK - self.status_text = CHUNK_STATUS_TEXT[CHUNK_OK] - -class ScannedRegionFile(object): - """ Stores all the scan information for a region file """ - def __init__(self, filename, corrupted = 0, wrong = 0, entities_prob = 0, shared_offset = 0, chunks = 0, status = 0, time = None): - # general region file info - self.path = filename - self.filename = split(filename)[1] - self.folder = split(filename)[0] - self.x = self.z = None - self.x, self.z = self.get_coords() - self.coords = (self.x, self.z) - - # dictionary storing all the state tuples of all the chunks - # in the region file - self.chunks = {} - - # TODO: these values aren't really used. - # count_chunks() is used instead. - # counters with the number of chunks - self.corrupted_chunks = corrupted - self.wrong_located_chunks = wrong - self.entities_prob = entities_prob - self.shared_offset = shared_offset - self.chunk_count = chunks - - # time when the scan for this file finished - self.scan_time = time - - # The status of the region file. At the moment can be OK, - # TOO SMALL or UNREADABLE see the constants at the start - # of the file. - self.status = status - - def __str__(self): - text = "Path: {0}".format(self.path) - scanned = False - if time: - scanned = True - text += "\nScanned: {0}".format(scanned) - - return text - - def __getitem__(self, key): - return self.chunks[key] - - def __setitem__(self, key, value): - self.chunks[key] = value - - def keys(self): - return self.chunks.keys() - - def get_counters(self): - """ Returns integers with all the problem counters in this - region file. The order is corrupted, wrong located, entities - shared header, total chunks """ - return self.corrupted_chunks, self.wrong_located_chunks, self.entities_prob, self.shared_offset, self.count_chunks() - - def get_path(self): - """ Returns the path of the region file. """ - return self.path - - def count_chunks(self, problem = None): - """ Counts chunks in the region file with the given problem. - If problem is omited or None, counts all the chunks. Returns - an integer with the counter. """ - counter = 0 - for coords in self.keys(): - if self[coords] and (self[coords][TUPLE_STATUS] == problem or problem == None): - counter += 1 - - return counter - - def get_global_chunk_coords(self, chunkX, chunkZ): - """ Takes the region filename and the chunk local - coords and returns the global chunkcoords as integerss """ - - regionX, regionZ = self.get_coords() - chunkX += regionX*32 - chunkZ += regionZ*32 - - return chunkX, chunkZ - - def get_coords(self): - """ Splits the region filename (full pathname or just filename) - and returns his region X and Z coordinates as integers. """ - if self.x != None and self.z != None: - return self.x, self.z - else: - splited = split(self.filename) - filename = splited[1] - l = filename.split('.') - coordX = int(l[1]) - coordZ = int(l[2]) - - return coordX, coordZ - - def list_chunks(self, status = None): - """ Returns a list of all the ScannedChunk objects of the chunks - with the given status, if no status is omited or None, - returns all the existent chunks in the region file """ - - l = [] - for c in self.keys(): - t = self[c] - if status == t[TUPLE_STATUS]: - l.append((self.get_global_chunk_coords(*c),t)) - elif status == None: - l.append((self.get_global_chunk_coords(*c),t)) - return l - - def summary(self): - """ Returns a summary of the problematic chunks. The summary - is a string with region file, global coords, local coords, - and status of every problematic chunk. """ - text = "" - if self.status == REGION_TOO_SMALL: - text += " |- This region file is too small in size to actually be a region file.\n" - else: - for c in self.keys(): - if self[c][TUPLE_STATUS] == CHUNK_OK or self[c][TUPLE_STATUS] == CHUNK_NOT_CREATED: continue - status = self[c][TUPLE_STATUS] - h_coords = c - g_coords = self.get_global_chunk_coords(*h_coords) - text += " |-+-Chunk coords: header {0}, global {1}.\n".format(h_coords, g_coords) - text += " | +-Status: {0}\n".format(CHUNK_STATUS_TEXT[status]) - if self[c][TUPLE_STATUS] == CHUNK_TOO_MANY_ENTITIES: - text += " | +-Nº entities: {0}\n".format(self[c][TUPLE_NUM_ENTITIES]) - text += " |\n" - - return text - - def remove_problematic_chunks(self, problem): - """ Removes all the chunks with the given problem, returns a - counter with the number of deleted chunks. """ - - counter = 0 - bad_chunks = self.list_chunks(problem) - for c in bad_chunks: - global_coords = c[0] - status_tuple = c[1] - local_coords = _get_local_chunk_coords(*global_coords) - region_file = region.RegionFile(self.path) - region_file.unlink_chunk(*local_coords) - counter += 1 - # create the new status tuple - # (num_entities, chunk status) - self[local_coords] = (0 , CHUNK_NOT_CREATED) - - return counter - - def remove_entities(self): - """ Removes all the entities in chunks with the problematic - status CHUNK_TOO_MANY_ENTITIES that are in this region file. - Returns a counter of all the removed entities. """ - problem = CHUNK_TOO_MANY_ENTITIES - counter = 0 - bad_chunks = self.list_chunks(problem) - for c in bad_chunks: - global_coords = c[0] - status_tuple = c[1] - local_coords = _get_local_chunk_coords(*global_coords) - counter += self.remove_chunk_entities(*local_coords) - # create new status tuple: - # (num_entities, chunk status) - self[local_coords] = (0 , CHUNK_OK) - return counter - - def remove_chunk_entities(self, x, z): - """ Takes a chunk coordinates, opens the chunk and removes all - the entities in it. Return an integer with the number of - entities removed""" - region_file = region.RegionFile(self.path) - chunk = region_file.get_chunk(x,z) - counter = len(chunk['Level']['Entities']) - empty_tag_list = nbt.TAG_List(nbt.TAG_Byte,'','Entities') - chunk['Level']['Entities'] = empty_tag_list - region_file.write_chunk(x, z, chunk) - - return counter - - def rescan_entities(self, options): - """ Updates the status of all the chunks in the region file when - the the option entity limit is changed. """ - for c in self.keys(): - # for safety reasons use a temporary list to generate the - # new tuple - t = [0,0] - if self[c][TUPLE_STATUS] in (CHUNK_TOO_MANY_ENTITIES, CHUNK_OK): - # only touch the ok chunks and the too many entities chunk - if self[c][TUPLE_NUM_ENTITIES] > options.entity_limit: - # now it's a too many entities problem - t[TUPLE_NUM_ENTITIES] = self[c][TUPLE_NUM_ENTITIES] - t[TUPLE_STATUS] = CHUNK_TOO_MANY_ENTITIES - - elif self[c][TUPLE_NUM_ENTITIES] <= options.entity_limit: - # the new limit says it's a normal chunk - t[TUPLE_NUM_ENTITIES] = self[c][TUPLE_NUM_ENTITIES] - t[TUPLE_STATUS] = CHUNK_OK - - self[c] = tuple(t) - - -class RegionSet(object): - """Stores an arbitrary number of region files and the scan results. - Inits with a list of region files. The regions dict is filled - while scanning with ScannedRegionFiles and ScannedChunks.""" - def __init__(self, regionset_path = None, region_list = []): - if regionset_path: - self.path = regionset_path - self.region_list = glob(join(self.path, "r.*.*.mca")) - else: - self.path = None - self.region_list = region_list - self.regions = {} - for path in self.region_list: - r = ScannedRegionFile(path) - self.regions[r.get_coords()] = r - self.corrupted_chunks = 0 - self.wrong_located_chunks = 0 - self.entities_problems = 0 - self.shared_header = 0 - self.bad_list = [] - self.scanned = False - - def get_name(self): - """ Return a string with the name of the dimension, the - directory if there is no name or "" if there's nothing """ - - dim_directory = self._get_dimension_directory() - if dim_directory: - try: return DIMENSION_NAMES[dim_directory] - except: return dim_directory - else: - return "" - - def _get_dimension_directory(self): - """ Returns a string with the directory of the dimension, None - if there is no such a directory and the regionset is composed - of sparse region files. """ - if self.path: - rest, region = split(self.path) - rest, dim_path = split(rest) - if dim_path == "": dim_path = split(rest)[1] - return dim_path - - else: return None - - def __str__(self): - text = "Region-set information:\n" - if self.path: - text += " Regionset path: {0}\n".format(self.path) - text += " Region files: {0}\n".format(len(self.regions)) - text += " Scanned: {0}".format(str(self.scanned)) - return text - - def __getitem__(self, key): - return self.regions[key] - - def __setitem__(self, key, value): - self.regions[key] = value - - def __delitem__(self, key): - del self.regions[key] - - def __len__(self): - return len(self.regions) - - def keys(self): - return self.regions.keys() - - def list_regions(self, status = None): - """ Returns a list of all the ScannedRegionFile objects stored - in the RegionSet with status. If status = None it returns - all the objects.""" - - if status == None: - #~ print "Estamos tras pasar el if para status None" - #~ print "Los valores de el dict son:" - #~ print self.regions.values() - #~ print "El diccionario es si es:" - #~ print self.regions - return self.regions.values() - t = [] - for coords in self.regions.keys(): - r = self.regions[coords] - if r.status == status: - t.append(r) - return t - - def count_regions(self, status = None): - """ Return the number of region files with status. If none - returns the number of region files in this regionset. - Possible status are: empty, too_small """ - - counter = 0 - for r in self.keys(): - if status == self[r].status: counter += 1 - elif status == None: counter += 1 - return counter - - def count_chunks(self, problem = None): - """ Returns the number of chunks with the given problem. If - problem is None returns the number of chunks. """ - counter = 0 - for r in self.keys(): - counter += self[r].count_chunks(problem) - return counter - - def list_chunks(self, status = None): - """ Returns a list of the ScannedChunk objects of the chunks - with the given status. If status = None returns all the - chunks. """ - l = [] - for r in self.keys(): - l.extend(self[r].list_chunks(status)) - return l - - def summary(self): - """ Returns a summary of the problematic chunks in this - regionset. The summary is a string with global coords, - local coords, data coords and status. """ - text = "" - for r in self.keys(): - if not (self[r].count_chunks(CHUNK_CORRUPTED) or self[r].count_chunks(CHUNK_TOO_MANY_ENTITIES) or self[r].count_chunks(CHUNK_WRONG_LOCATED) or self[r].count_chunks(CHUNK_SHARED_OFFSET) or self[r].status == REGION_TOO_SMALL): - continue - text += "Region file: {0}\n".format(self[r].filename) - text += self[r].summary() - text += " +\n\n" - return text - - def locate_chunk(self, global_coords): - """ Takes the global coordinates of a chunk and returns the - region filename and the local coordinates of the chunk or - None if it doesn't exits in this RegionSet """ - - filename = self.path + get_chunk_region(*global_coords) - local_coords = _get_local_chunk_coords(*global_coords) - - return filename, local_coords - - def locate_region(self, coords): - """ Returns a string with the path of the region file with - the given coords in this regionset or None if not found. """ - - x, z = coords - region_name = 'r.' + str(x) + '.' + str(z) + '.mca' - - return region_name - - - def remove_problematic_chunks(self, problem): - """ Removes all the chunks with the given problem, returns a - counter with the number of deleted chunks. """ - - counter = 0 - if self.count_chunks(): - print ' Deleting chunks in region set \"{0}\":'.format(self._get_dimension_directory()) - for r in self.regions.keys(): - counter += self.regions[r].remove_problematic_chunks(problem) - print "Removed {0} chunks in this regionset.\n".format(counter) - - return counter - - def remove_entities(self): - """ Removes entities in chunks with the status - TOO_MANY_ENTITIES. """ - counter = 0 - for r in self.regions.keys(): - counter += self.regions[r].remove_entities() - return counter - - def rescan_entities(self, options): - """ Updates the status of all the chunks in the regionset when - the option entity limit is changed. """ - for r in self.keys(): - self[r].rescan_entities(options) - - def generate_report(self, standalone): - """ Generates a report of the last scan. If standalone is True - it will generate a report to print in a terminal. If it's False - it will returns the counters of every problem. """ - - # collect data - corrupted = self.count_chunks(CHUNK_CORRUPTED) - wrong_located = self.count_chunks(CHUNK_WRONG_LOCATED) - entities_prob = self.count_chunks(CHUNK_TOO_MANY_ENTITIES) - shared_prob = self.count_chunks(CHUNK_SHARED_OFFSET) - total_chunks = self.count_chunks() - - too_small_region = self.count_regions(REGION_TOO_SMALL) - unreadable_region = self.count_regions(REGION_UNREADABLE) - total_regions = self.count_regions() - - if standalone: - text = "" - - # Print all this info in a table format - # chunks - chunk_errors = ("Problem","Corrupted","Wrong l.","Etities","Shared o.", "Total chunks") - chunk_counters = ("Counts",corrupted, wrong_located, entities_prob, shared_prob, total_chunks) - table_data = [] - for i, j in zip(chunk_errors, chunk_counters): - table_data.append([i,j]) - text += "\nChunk problems:" - if corrupted or wrong_located or entities_prob or shared_prob: - text += table(table_data) - else: - text += "\nNo problems found.\n" - - # regions - text += "\n\nRegion problems:\n" - region_errors = ("Problem","Too small","Unreadable","Total regions") - region_counters = ("Counts", too_small_region,unreadable_region, total_regions) - table_data = [] - # compose the columns for the table - for i, j in zip(region_errors, region_counters): - table_data.append([i,j]) - if too_small_region: - text += table(table_data) - else: - text += "No problems found." - - return text - else: - return corrupted, wrong_located, entities_prob, shared_prob, total_chunks, too_small_region, unreadable_region, total_regions - - def remove_problematic_regions(self, problem): - """ Removes all the regions files with the given problem. - This is NOT the same as removing chunks, this WILL DELETE - the region files from the hard drive. """ - counter = 0 - for r in self.list_regions(problem): - remove(r.get_path()) - counter += 1 - return counter - -class World(object): - """ This class stores all the info needed of a world, and once - scanned, stores all the problems found. It also has all the tools - needed to modify the world.""" - - def __init__(self, world_path): - self.path = world_path - - # list with RegionSets - self.regionsets = [] - - self.regionsets.append(RegionSet(join(self.path, "region/"))) - for directory in glob(join(self.path, "DIM*/region")): - self.regionsets.append(RegionSet(join(self.path, directory))) - - # level.dat - # let's scan level.dat here so we can extract the world name - # right now - level_dat_path = join(self.path, "level.dat") - if exists(level_dat_path): - try: - self.level_data = nbt.NBTFile(level_dat_path)["Data"] - self.name = self.level_data["LevelName"].value - self.scanned_level = ScannedDatFile(level_dat_path, readable = True, status_text = "OK") - except Exception, e: - self.name = None - self.scanned_level = ScannedDatFile(level_dat_path, readable = False, status_text = e) - else: - self.level_file = None - self.level_data = None - self.name = None - self.scanned_level = ScannedDatFile(None, False, "The file doesn't exist") - - # player files - player_paths = glob(join(join(self.path, "players"), "*.dat")) - self.players = {} - for path in player_paths: - name = split(path)[1].split(".")[0] - self.players[name] = ScannedDatFile(path) - - # does it look like a world folder? - region_files = False - for region_directory in self.regionsets: - if region_directory: - region_files = True - if region_files: - self.isworld = True - else: - self.isworld = False - - # set in scan.py, used in interactive.py - self.scanned = False - - def __str__(self): - text = "World information:\n" - text += " World path: {0}\n".format(self.path) - text += " World name: {0}\n".format(self.name) - text += " Region files: {0}\n".format(self.get_number_regions()) - text += " Scanned: {0}".format(str(self.scanned)) - return text - - def get_number_regions(self): - """ Returns a integer with the number of regions in this world""" - counter = 0 - for dim in self.regionsets: - counter += len(dim) - - return counter - - def summary(self): - """ Returns a text string with a summary of all the problems - found in the world object.""" - final = "" - - # intro with the world name - final += "{0:#^60}\n".format('') - final += "{0:#^60}\n".format(" World name: {0} ".format(self.name)) - final += "{0:#^60}\n".format('') - - # dat files info - final += "\nlevel.dat:\n" - if self.scanned_level.readable: - final += "\t\'level.dat\' is readable\n" - else: - final += "\t[WARNING]: \'level.dat\' isn't readable, error: {0}\n".format(self.scanned_level.status_text) - - all_ok = True - final += "\nPlayer files:\n" - for name in self.players: - if not self.players[name].readable: - all_ok = False - final += "\t-[WARNING]: Player file {0} has problems.\n\t\tError: {1}\n\n".format(self.players[name].filename, self.players[name].status_text) - if all_ok: - final += "\tAll player files are readable.\n\n" - - # chunk info - chunk_info = "" - for regionset in self.regionsets: - - title = regionset.get_name() - - # don't add text if there aren't broken chunks - text = regionset.summary() - chunk_info += (title + text) if text else "" - final += chunk_info if chunk_info else "All the chunks are ok." - - return final - - def get_name(self): - """ Returns a string with the name as found in level.dat or - with the world folder's name. """ - if self.name: - return self.name - else: - n = split(self.path) - if n[1] == '': - n = split(n[0])[1] - return n - - def count_regions(self, status = None): - """ Returns a number with the count of region files with - status. """ - counter = 0 - for r in self.regionsets: - counter += r.count_regions(status) - return counter - - def count_chunks(self, status = None): - """ Counts problems """ - counter = 0 - for r in self.regionsets: - count = r.count_chunks(status) - counter += count - return counter - - def replace_problematic_chunks(self, backup_worlds, problem, options): - """ Takes a list of world objects and a problem value and try - to replace every chunk with that problem using a working - chunk from the list of world objects. It uses the world - objects in left to riht order. """ - - counter = 0 - for regionset in self.regionsets: - for backup in backup_worlds: - # choose the correct regionset based on the dimension - # folder name - for temp_regionset in backup.regionsets: - if temp_regionset._get_dimension_directory() == regionset._get_dimension_directory(): - b_regionset = temp_regionset - break - - # this don't need to be aware of region status, it just - # iterates the list returned by list_chunks() - bad_chunks = regionset.list_chunks(problem) - - if bad_chunks and b_regionset._get_dimension_directory() != regionset._get_dimension_directory(): - print "The regionset \'{0}\' doesn't exist in the backup directory. Skipping this backup directory.".format(regionset._get_dimension_directory()) - else: - for c in bad_chunks: - global_coords = c[0] - status_tuple = c[1] - local_coords = _get_local_chunk_coords(*global_coords) - print "\n{0:-^60}".format(' New chunk to replace. Coords: x = {0}; z = {1} '.format(*global_coords)) - - # search for the region file - backup_region_path, local_coords = b_regionset.locate_chunk(global_coords) - tofix_region_path, _ = regionset.locate_chunk(global_coords) - if exists(backup_region_path): - print "Backup region file found in:\n {0}".format(backup_region_path) - - # scan the whole region file, pretty slow, but completely needed to detec sharing offset chunks - from scan import scan_region_file - r = scan_region_file(ScannedRegionFile(backup_region_path),options) - try: - status_tuple = r[local_coords] - except KeyError: - status_tuple = None - - # retrive the status from status_tuple - if status_tuple == None: - status = CHUNK_NOT_CREATED - else: - status = status_tuple[TUPLE_STATUS] - - if status == CHUNK_OK: - backup_region_file = region.RegionFile(backup_region_path) - working_chunk = backup_region_file.get_chunk(local_coords[0],local_coords[1]) - - print "Replacing..." - # the chunk exists and is healthy, fix it! - tofix_region_file = region.RegionFile(tofix_region_path) - # first unlink the chunk, second write the chunk. - # unlinking the chunk is more secure and the only way to replace chunks with - # a shared offset withou overwriting the good chunk - tofix_region_file.unlink_chunk(*local_coords) - tofix_region_file.write_chunk(local_coords[0], local_coords[1],working_chunk) - counter += 1 - print "Chunk replaced using backup dir: {0}".format(backup.path) - - else: - print "Can't use this backup directory, the chunk has the status: {0}".format(CHUNK_STATUS_TEXT[status]) - continue - - else: - print "The region file doesn't exist in the backup directory: {0}".format(backup_region_path) - - return counter - - - def remove_problematic_chunks(self, problem): - """ Removes all the chunks with the given problem. """ - counter = 0 - for regionset in self.regionsets: - counter += regionset.remove_problematic_chunks(problem) - return counter - - def replace_problematic_regions(self, backup_worlds, problem, options): - """ Replaces region files with the given problem using a backup - directory. """ - counter = 0 - for regionset in self.regionsets: - for backup in backup_worlds: - # choose the correct regionset based on the dimension - # folder name - for temp_regionset in backup.regionsets: - if temp_regionset._get_dimension_directory() == regionset._get_dimension_directory(): - b_regionset = temp_regionset - break - - bad_regions = regionset.list_regions(problem) - if bad_regions and b_regionset._get_dimension_directory() != regionset._get_dimension_directory(): - print "The regionset \'{0}\' doesn't exist in the backup directory. Skipping this backup directory.".format(regionset._get_dimension_directory()) - else: - for r in bad_regions: - print "\n{0:-^60}".format(' New region file to replace! Coords {0} '.format(r.get_coords())) - - # search for the region file - - try: - backup_region_path = b_regionset[r.get_coords()].get_path() - except: - backup_region_path = None - tofix_region_path = r.get_path() - - if backup_region_path != None and exists(backup_region_path): - print "Backup region file found in:\n {0}".format(backup_region_path) - # check the region file, just open it. - try: - backup_region_file = region.RegionFile(backup_region_path) - except region.NoRegionHeader as e: - print "Can't use this backup directory, the error while opening the region file: {0}".format(e) - continue - except Exception as e: - print "Can't use this backup directory, unknown error: {0}".format(e) - continue - copy(backup_region_path, tofix_region_path) - print "Region file replaced!" - counter += 1 - else: - print "The region file doesn't exist in the backup directory: {0}".format(backup_region_path) - - return counter - - - def remove_problematic_regions(self, problem): - """ Removes all the regions files with the given problem. - This is NOT the same as removing chunks, this WILL DELETE - the region files from the hard drive. """ - counter = 0 - for regionset in self.regionsets: - counter += regionset.remove_problematic_regions(problem) - return counter - - def remove_entities(self): - """ Delete all the entities in the chunks that have more than - entity-limit entities. """ - counter = 0 - for regionset in self.regionsets: - counter += regionset.remove_entities() - return counter - - def rescan_entities(self, options): - """ Updates the status of all the chunks in the world when the - option entity limit is changed. """ - for regionset in self.regionsets: - regionset.rescan_entities(options) - - def generate_report(self, standalone): - - # collect data - corrupted = self.count_chunks(CHUNK_CORRUPTED) - wrong_located = self.count_chunks(CHUNK_WRONG_LOCATED) - entities_prob = self.count_chunks(CHUNK_TOO_MANY_ENTITIES) - shared_prob = self.count_chunks(CHUNK_SHARED_OFFSET) - total_chunks = self.count_chunks() - - too_small_region = self.count_regions(REGION_TOO_SMALL) - unreadable_region = self.count_regions(REGION_UNREADABLE) - total_regions = self.count_regions() - - if standalone: - text = "" - - # Print all this info in a table format - chunk_errors = ("Problem","Corrupted","Wrong l.","Etities","Shared o.", "Total chunks") - chunk_counters = ("Counts",corrupted, wrong_located, entities_prob, shared_prob, total_chunks) - table_data = [] - for i, j in zip(chunk_errors, chunk_counters): - table_data.append([i,j]) - text += "\nChunk problems:\n" - if corrupted or wrong_located or entities_prob or shared_prob: - text += table(table_data) - else: - text += "No problems found.\n" - - text += "\n\nRegion problems:\n" - region_errors = ("Problem","Too small","Unreadable","Total regions") - region_counters = ("Counts", too_small_region,unreadable_region, total_regions) - table_data = [] - # compose the columns for the table - for i, j in zip(region_errors, region_counters): - table_data.append([i,j]) - if too_small_region: - text += table(table_data) - else: - text += "No problems found." - - return text - else: - return corrupted, wrong_located, entities_prob, shared_prob, total_chunks, too_small_region, unreadable_region, total_regions - - - -def delete_entities(region_file, x, z): - """ This function is used while scanning the world in scan.py! Takes - a region file obj and a local chunks coords and deletes all the - entities in that chunk. """ - chunk = region_file.get_chunk(x,z) - counter = len(chunk['Level']['Entities']) - empty_tag_list = nbt.TAG_List(nbt.TAG_Byte,'','Entities') - chunk['Level']['Entities'] = empty_tag_list - region_file.write_chunk(x, z, chunk) - - return counter - - -def _get_local_chunk_coords(chunkx, chunkz): - """ Takes the chunk global coords and returns the local coords """ - return chunkx % 32, chunkz % 32 - -def get_chunk_region(chunkX, chunkZ): - """ Returns the name of the region file given global chunk - coords """ - - regionX = chunkX / 32 - regionZ = chunkZ / 32 - - region_name = 'r.' + str(regionX) + '.' + str(regionZ) + '.mca' - - return region_name - -def get_chunk_data_coords(nbt_file): - """ Gets the coords stored in the NBT structure of the chunk. - - Takes an nbt obj and returns the coords as integers. - Don't confuse with get_global_chunk_coords! """ - - level = nbt_file.__getitem__('Level') - - coordX = level.__getitem__('xPos').value - coordZ = level.__getitem__('zPos').value - - return coordX, coordZ - -def get_region_coords(filename): - """ Splits the region filename (full pathname or just filename) - and returns his region X and Z coordinates as integers. """ - - l = filename.split('.') - coordX = int(l[1]) - coordZ = int(l[2]) - - return coordX, coordZ - -def get_global_chunk_coords(region_name, chunkX, chunkZ): - """ Takes the region filename and the chunk local - coords and returns the global chunkcoords as integerss. This - version does exactly the same as the method in - ScannedRegionFile. """ - - regionX, regionZ = get_region_coords(region_name) - chunkX += regionX*32 - chunkZ += regionZ*32 - - return chunkX, chunkZ