Implement rescale #1
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Upgrade pythia to latest
Update master to latest fixed version
commit 4b652ea8eab2f24cbae67c5da56a6f8e7cd9349e
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Wed Jun 2 13:04:22 2021 -0400
Fix DSSAT#29
- Analytics no longer depends on ha_area to execute.
- With this structure, should be able to extend to other fixed variable reporting
commit 5fda15f
Merge: 923d034 0933b35
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Tue Jun 1 13:50:29 2021 -0400
Merge bulk fixes
commit eabc95f
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Tue Jun 1 12:38:07 2021 -0400
Overall cleanup of the codebase
* Fix pre-release versioning
* Modify poetry configuration to use local venvs
* Remove unneeded packages
* Update existing packages
* Remove v1 codebase
commit 923d034
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Wed Jan 27 12:12:11 2021 -0500
Check for missing profile_id in lookup_ghr()
* Potential KeyError crash when looking for the profile ID in the cache.
commit 52da6a2
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Tue Jan 19 16:56:18 2021 -0500
Remove unneeded packages
Fix DSSAT#18
commit 57611eb
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Tue Jan 19 16:29:43 2021 -0500
Use context vs run for variable search
* Fix DSSAT#17
* When doing a `$var` search, check the current `context`. This SHOULD
include `run` as well, but with variables from rasters already in
place.
commit 88c829b
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Tue Jan 19 16:29:23 2021 -0500
Add logging to date conversion utils
* Fixes DSSAT#16
commit cf7c2b8
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Tue Jan 19 08:54:02 2021 -0500
Implement GHR caching
* Build the cache on first request to GHR.
* Utilize the cache on each subsequent call.
* VERY rudimentary cache implementation.
commit d6c9ea7
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Wed Jan 6 16:28:21 2021 -0500
Migrate to script executable
* Implement __main__.py to run as module
* Implement tool.poetry.scripts to install through poetry into the
environment.
* Move pythia.py to pythia/cli.py for proper package layout
commit 149375e
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Wed Dec 23 22:46:25 2020 -0500
Add more details to logging for DSSAT#14
commit 9bc5b45
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Wed Dec 23 22:37:09 2020 -0500
Log missing assignment for values
Fix DSSAT#14
commit a54b396
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Thu Dec 17 16:08:43 2020 -0500
Add datetime stamp to logfiles
commit e6c8988
Merge: 3637042 4913c65
Author: Christopher Villalobos <cvillalobos@ufl.edu>
Date: Wed Nov 25 16:36:26 2020 -0500
Fix path issues for analytics
- date_from_doy_raster::raster::<file> - Raster should have Day of Year values (1 to 366) - Uses startYear parameter for complete year - If startYear is not a leap year and day 366 is found, it rolls over to Jan 1 of the next year. - Support for 4 digit Julian dates in _from_julian_date() utility function.
- date_offset::$<variable>::<offset int> - Currently only supports using existing variables in the context.
* Pass the entire plugin config block to the plugin now. * Need to work on the way this works in the future commit 1209c70 Author: Christopher Villalobos <cvillalobos@ufl.edu> Date: Mon Jul 26 16:59:25 2021 -0400 Fix issue with no_rename being embedded in the config * Will need to work on better ways to pass config commit 2a1454b Author: Christopher Villalobos <cvillalobos@ufl.edu> Date: Mon Jul 26 16:10:19 2021 -0400 Modify how plugin configuration is read * Pass the entire plugin config block for each plugin to the initializer. * Let the plugin handle any filtering or modifications and save what is required by the plugin during registration.
* Simple project management for things that need to be done.
* Implement FODATE and 4-digit year formatter * Add default parameter to template formatting (default default is -99) * Add support for 4-digit Julian format * Implement run_mode configuration parameter - Specify the `run_mode` as a string which matches the DSSAT run modes. - Defaults the `run_mode` to `A`
* Add batch_mode support
* Allow for current batch run modes to be marked as batch
* Check for a default file (DssBatch.*) to mark as the batch file
* Allow user to specify custom file name for batch as `batch_file`
* Allow user to specify custom file name for regular as `filex`
* Add a population hack
* Just hardcode the population for the moment.
* The goal is to extend the output process to enable this to be
easier. Unfortunately, we are under a time pressure and cannot
implement this properly yet.
* Update dependencies
* Format code
* Update project goals
Signed-off-by: Christopher Villalobos <cvillalobos@ufl.edu>
* Remove run from `generate_funs()`
* Harvest Date (hdate) * Planting Date (pdate) * Planting Window (pfrst, plast)
* Remove run from `combine_outputs()` * Remove run from `generate_funs()` * Fix population only writing on no value * Removed tuple from `collate_outputs` function
* sh2o is now ":>6.3f" * snh4 and sno3 is now ":>6.2f"
* Support for DSSAT 4.8+ * Support for 4-digit years (fodate) * Support DSSAT Run modes * Support DSSAT Batch files (simple) * Support DSSAT Forecast Mode (replaces simple_forecast plugin) The simple_forecast plugin is still in-place for DSSAT <4.8 * Fix DSSAT#36 * Fix DSSAT#37 * Fix DSSAT#38 * Add HDATE, PDATE, PFRST, and PLAST as complete date variables * Add hardwired population values
* Fix DSSAT#39
Updated from pythia-2.2.1-py3-none-any.whl to pythia-2.3.0-py3-none-any.whl in 8. Install the Pythia wheel file Added exemplary path under Windows where the poetry script is found for "7. Install DSSAT-Pythia:"
…e errors. This should allow for more consistent plugin operation.
…pythia into implement-rescale
…e errors. This should allow for more consistent plugin operation. (cherry picked from commit f1c040a)
Error count bug fix
Added soil analysis variables
Update README.md to reflect 2.3.0. version
…sest_vector_coords function. The following report originates from running this function twice per pixel on Tanzania 10x10 km grid: BEFORE (041cd34) -> Performance counter stats for 'python -X perf -m pythia --clean-work-dir --setup test2/pythia-config.json' (5 runs): 17,438,093.49 msec task-clock # 27.858 CPUs utilized ( +- 0.08% ) 568,468 context-switches # 32.599 /sec ( +- 2.15% ) 16,541 cpu-migrations # 0.949 /sec ( +- 1.30% ) 341,541 page-faults # 19.586 /sec ( +- 0.52% ) 85,137,469,787,762 instructions # 1.22 insn per cycle # 0.29 stalled cycles per insn ( +- 0.08% ) 69,737,313,303,103 cycles # 3.999 GHz ( +- 0.08% ) 24,862,264,765,548 stalled-cycles-frontend # 35.65% frontend cycles idle ( +- 0.25% ) 18,281,479,410,162 branches # 1.048 G/sec ( +- 0.07% ) 172,136,315,036 branch-misses # 0.94% of all branches ( +- 0.75% ) 625.953 +- 0.429 seconds time elapsed ( +- 0.07% ) AFTER -> Performance counter stats for 'python -X perf -m pythia --clean-work-dir --setup test2/pythia-config.json' (5 runs): 4,436,558.37 msec task-clock # 27.514 CPUs utilized ( +- 0.22% ) 255,284 context-switches # 57.541 /sec ( +- 1.80% ) 14,824 cpu-migrations # 3.341 /sec ( +- 1.76% ) 223,269 page-faults # 50.325 /sec ( +- 0.11% ) 22,793,963,559,968 instructions # 1.28 insn per cycle # 0.27 stalled cycles per insn ( +- 0.15% ) 17,739,586,160,894 cycles # 3.999 GHz ( +- 0.22% ) 6,203,511,000,548 stalled-cycles-frontend # 34.97% frontend cycles idle ( +- 0.34% ) 4,890,648,814,890 branches # 1.102 G/sec ( +- 0.13% ) 42,420,497,493 branch-misses # 0.87% of all branches ( +- 0.64% ) 161.248 +- 0.403 seconds time elapsed ( +- 0.25% )
…vector_coords``. The following report originates from running this function twice per pixel on Tanzania 10x10 km grid: BEFORE (4ad13a6) -> Performance counter stats for 'python -X perf -m pythia --clean-work-dir --setup test2/pythia-config.json' (3 runs): 4,435,650.67 msec task-clock # 27.544 CPUs utilized ( +- 0.39% ) 220,607 context-switches # 49.735 /sec ( +- 3.27% ) 13,995 cpu-migrations # 3.155 /sec ( +- 0.83% ) 222,164 page-faults # 50.086 /sec ( +- 0.21% ) 22,727,605,668,167 instructions # 1.28 insn per cycle # 0.28 stalled cycles per insn ( +- 0.06% ) 17,736,864,542,617 cycles # 3.999 GHz ( +- 0.39% ) 6,306,883,751,606 stalled-cycles-frontend # 35.56% frontend cycles idle ( +- 0.86% ) 4,876,195,822,590 branches # 1.099 G/sec ( +- 0.06% ) 41,536,653,902 branch-misses # 0.85% of all branches ( +- 1.52% ) 161.038 +- 0.870 seconds time elapsed ( +- 0.54% ) AFTER -> Performance counter stats for 'python -X perf -m pythia --clean-work-dir --setup test2/pythia-config.json' (3 runs): 919,616.65 msec task-clock # 26.100 CPUs utilized ( +- 0.14% ) 143,583 context-switches # 156.134 /sec ( +- 4.34% ) 8,313 cpu-migrations # 9.040 /sec ( +- 5.51% ) 259,900 page-faults # 282.618 /sec ( +- 0.11% ) 6,434,303,689,973 instructions # 1.75 insn per cycle # 0.12 stalled cycles per insn ( +- 0.01% ) 3,674,947,238,272 cycles # 3.996 GHz ( +- 0.14% ) 750,853,165,462 stalled-cycles-frontend # 20.43% frontend cycles idle ( +- 0.51% ) 1,430,992,318,806 branches # 1.556 G/sec ( +- 0.01% ) 7,936,273,176 branch-misses # 0.55% of all branches ( +- 1.64% ) 35.234 +- 0.108 seconds time elapsed ( +- 0.31% )
Fix get_vector_resolution inferring the y resolution to zero
Merge backlogged commits.
…pythia into implement-rescale # Conflicts: # pythia/rescale.py
…ords About 18x speedup in the build phase.
Implement rescale
Update code to reflect shapely removal.
…mplement-rescale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here is the code for you to test/work with before our meeting with Dr. Hoogenboom. Once I am done with any revisions that are suggested, I can create the pull request for the DSSAT repository (or whatever is appropriate).
It looks like the DSSAT repo was updated more recently than this branch you were on. If you need me to redirect the request, I can do that.