LogFetcher3000 is a Python based tool to retrieve access and error logs from WP Engine hosted websites that utilize the WP Engine Must Use plugins.
This tool uses a JSON file to hold the current access and current error log for the site in question. Links should be added as lists within the dictionary:
{"access": ["https://mysite.com/link/to/access.log"], "error": ["https://mysite.com/link/to/error.log", "https://myothersite.com/link/to/error.log"]}
The links can be obtained from within the WP Admin Dashboard within the WPE MU Plugin and inserted in a file called links.json.
This script can be called by a cron at a specific time to download the day's logs. For example to grab the logs on the hour every hour:
59 17 * * * bash -c 'cd /path/to/logfetcher3000 && source env/bin/activate && python3 logfetcher3000.py'
All notable changes to this project will be documented in this section.
- Bug with unittest not running. Moved test_logfetcher3000.py to main project directory and updated import in file
- Moved the main loops in to their own functions from main
- Moved logging functionality to within main
- Improved formatting for log messages
- Additional check for Apache error logs to ensure correct link is used
- Reverted logging capability from a function due to a bug being introduced causing repeated error msg output
- Added default filename to file argparse
- Argparse to handle verbose debug error output
- Logging module to take over logging errors within the application as opposed to using print statements
- Ensuring requests are returning 200 response codes when downloading files
- Moved D_STAMP from a global variable to within main and using a single variable
- Turning dict items to lowercase within get_links_from_file()
- Moved filename in main to a variable for re-use and ease of change
- Check if the set file exists in the cwd
- Checks for the presence of the file in the cwd and exits if not present
- Created new function for opening the file as opposed to having this done in main
- Regex HTTP protocol validation
- Set the D_STAMP global variables to uppercase
- Moved opening the JSON file under main
- Handling the file under get_log() to be with a content manager
- Generated filenames are now in the format of
environmentname_[access|error]_datestamp.log
- Support for dynamically reading each dictionary value by looping through the list provided in
links.json - Regex library to get the environment name from error log link
- Created requirements.txt
- Ability to add links to a JSON file to be read by the script
- Automaed filename creation based on the domain used for the access logs