Does a thing happen? #2
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…actions_2.11;2.0.0
…actions-concrete_2.11;2.0.0
…loader-concrete_2.11;2.0.0
These shas landed in Fsq.io and all referenced commits are in that tree. (sapling split of 6e33eb4726cabf530a9d495c64c45e2805aef2ed)
Sonatype requires these fields for all published poms. (sapling split of f32699c6e3f7bea7c789d16e35c876985956b122)
…ervice_2.11;2.0.1
…tryinfo_2.11;2.0.1
…loader-service_2.11;2.0.1
…callback_2.11;2.0.1
…ervice_2.11;2.0.1
…oncrete_2.11;2.0.1
…used-symbols_2.11;2.0.1
## Description
* Uses set literals, i.e. `{foo, bar}` and set comprehensions, i.e. `{foo(x) for x in bar}` wherever possible, instead of the constructor `set([foo, bar])`.
* Updated [style guide](http://wiki.prod.foursquare.com/code/python-style-guide#the-long-version_use-set-literals) to document set literals > set constructor when possible.
## Motivation and Context
The set literal syntax is preferred for both readability and performance.
According to https://renzo.lucioni.xyz/pythons-set-literals/, set literals average an execution twice as fast as the set constructor, in part because `set([foo, bar])` generates the intermediate list before converting it to a set.
## How change was generated
Used [this script](https://github.com/foursquare/python3-port-utilities/blob/master/foursquare/sets.py) to generate proposed fixes, which then prompts a review of each proposal and requires actively confirming by pressing `y`.
Regex used for normal set:
```python
def _generate_set_fix(line: str) -> str:
list_regex = r'set\(\[(?P<literal>.*)\]\)'
tuple_regex = r'set\(\((?P<literal>.*)\)\)'
list_fix = re.sub(list_regex, '{\g<literal>}', line)
tuple_fix = re.sub(tuple_regex, '{\g<literal>}', list_fix)
return tuple_fix
```
Regex used for frozenset:
```python
def _generate_frozenset_fix(line: str) -> str:
list_regex = r'\(\[(?P<literal>.*)\]\)'
tuple_regex = r'\(\((?P<literal>.*)\)\)'
list_fix = re.sub(list_regex, '({\g<literal>})', line)
tuple_fix = re.sub(tuple_regex, '({\g<literal>})', list_fix)
return tuple_fix
```
Regex used for comprehensions:
```python
def _generate_comprehension_fix(line: str) -> str:
regex = r'set\((?P<comp>.*)\)'
return re.sub(regex, '{\g<comp>}', line)
```
(sapling split of 328efd5dea3c9ad67d5c1b5fff1fa56a011200e8)
Follow up to https://github.com/foursquare/foursquare.web/pull/7226. * Removes MyPy's check for local install * MyPy will check interpreter's version to determine what to run * Cleans up `env.sh` * Reorder sections to be grouped logically. * Remove duplicate variable declarations. * Removes REFERENCE_PYTHON_ROOT, REFERENCE_PYTHON, and FSQ_PYTHON_ROOT in favor of Python 2 names. (sapling split of 431fa93c61c82439ecd96a8437ba3210ce21744d)
v1: Benjy Weinberger v2: Aaron Mitchell (w/ Patrick Lawson) v3: Todd Walker and Mateo Rodriguez The s3 code in our Pants fork, really should be upstreamed. foursquare/pants@0e1cd99 (sapling split of cc0d579d894561b76417528b4688e29ab88b6c84)
(sapling split of 9d950a93608c56e957ddbecae57d44e93f435a57) (sapling split of 17b0721cb0a356b3a560e9c6aca6a127ac26555c) (sapling split of 74ce9d256ac7c579c8ebc4adb8f5c5a858ed04e3) (sapling split of 68b2751295f64423850a345562875a63ae5ae0e2) (sapling split of 5c5fc943978370be50a91b9c8115ba8f59ed2d16) (sapling split of 5c5fc943978370be50a91b9c8115ba8f59ed2d16) (sapling split of aa98986d46cfb3f65a8057e64b0cdb522b42149e)
(sapling split of 86359f05fe9a0635430bdeba9162039c833ec3de)
(sapling split of ed6f34a660675f6bc6e2546b2a8a397fbf411293)
(sapling split of ed2000552ece52bfd90be260a8bec7e3e57ea72b)
## Description Replaces all uses of `foo.iteritems()`, `foo.itervalues()`, and `foo.iterkeys()` with `viewitems(foo)`, `viewvalues(foo)`, and `viewkeys(foo)`. ## Motivation and Context Python 2.7 offers 3 versions to iterate through dictionaries: `items()`, `iteritems()`, and `viewitems()`. Python 3 simplifies this to only offer `items()`, which has the same semantics as Python 2.7's `viewitems()`. So, `foo.iteritems()` will raise an `AttributeError` in Python 3. ### `iteritems()` vs `viewitems()` `iteritems()` and `viewitems()` have very similar semantics; `iteritems()` returns an actual iterator object, whereas `viewitems()` returns a `dict_items` view object. Both allow for lazy iteration. I went with `viewitems` because Future recommends it, and it copies Py3 semantics exactly, while still behaving as we intend in all the use cases below. See [PEP 469](https://legacy.python.org/dev/peps/pep-0469/) for more information if interested. ## How change was generated Wrote script at https://github.com/foursquare/python3-port-utilities/blob/master/foursquare/iteritems.py. Regex used: ```python def _generate_iteritems_fix(line: str) -> str: regex = r'(?P<iterable>[^\s]*)\.iteritems\(\)' return re.sub(regex, 'viewitems(\g<iterable>)', line) def _generate_itervalues_fix(line: str) -> str: regex = r'(?P<iterable>[^\s]*)\.itervalues\(\)' return re.sub(regex, 'viewvalues(\g<iterable>)', line) def _generate_iterkeys_fix(line: str) -> str: regex = r'(?P<iterable>[^\s]*)\.iterkeys\(\)' return re.sub(regex, 'viewkeys(\g<iterable>)', line) ``` (sapling split of d2f3b6cd9fbf738ed2ca6cb031d13cd8ecedd8cd)
* Install specific python3.6 * Remove clean from upkeep --all and add nuke * preface BUILD_ROOT (sapling split of 81c08fffc228bb90e5fd2cb63dffa7cfc016f45f)
(sapling split of 025c122ee9c55c935d3d6220e3caea60c2838aac)
* ExceptionatorRequest is now a case class instead of a RequestProxy (finagle removed this in httpx). This causes some noise where you need to do for example exceptionatorRequest.request.path instead of exceptionatorRequest.path. See https://github.com/foursquare/foursquare.web/pull/8599/files#diff-2a8589588160fc2113610e5ebc7d5ba4L12 * Replace some netty references with the new finagle versions (HttpMethod => Method, HttpResponseStatus => Status, ChannelBuffer => Buf) (sapling split of 5173ddd7efe0d7f3849674b0e5339ae93b80409c) (sapling split of 82b40b6fa119f1414046bf63bc4b4beeeb9e68d5)
* Migrate Dash to new Finagle httpx * Delete TryThriftOverHttpFilter and TestManagementServer which appear to be unused (they depend on some of the same code as drag which means I would have to migrate them as well) (sapling split of 328942cea40e20ef2bbd98165fc1f14fed293c48)
* Upgrade Pants from 1.4.0 to 1.7.0 This includes deprecating our internal resources_bundle and enabling the v2 engine. There will be learning curves on the pain points. * Disable test caching by default and add an FS_TEST_CACHING toggle. The immediate followup is planned to revert this, possibly in two stages as caution dictates. But this slow roll hopes to provide us with less edges to debug if we see problems in prod. (sapling split of 4b52fb2b13f37a48444cac0290f74ff5c377519e) (sapling split of d0fca104c65d021e45941aab501000b54b4f14e7) (sapling split of c86b10f3c7c96c800922a7086dc33fe2e2834616)
* Update type hints cheat sheet to favor unicode * Mathieu feedback * remove trailing whitespace * Make linter happy, less dead kittens. (sapling split of 1a1badedb47b372cec254f83ccbe1fcd39cb97bb)
The scalacheck jar has been proven to be released multiple times for the same version number. Only one jar of the 1.10.1 jars actually compiles in our repo (the newer publish, fwiw). This upgrades scalacheck past that instability to a modern version. Doing this required republishing Specs as well. Specs consumes scalacheck and fails claiming a dirty classpath. specs upgrade was: - scala2.11 -> scala2.11.8 - scalacheck 1.10.1 -> 1.14.0 Specs is long ago deprecated and by all rights should be ripped out. But the effort of rewriting the legacy tests made this the fastest way forward. (sapling split of 934dec438fe41e5fa913fe1534470c420c79060e)
(sapling split of 6c796a2c79ab15c5a3b630500a07b717ed40af11)
(sapling split of 8e04d66d1c52c892b5d0427802f5095b5341750a) (sapling split of d4be66597a30e405722f104dd75c2de4b25a1241)
(sapling split of 622faaa047ef2b2c2bdfe97b541d374557195caa)
(sapling split of cd7284b174e52e9d8c9f532e71c3ac1b53109a35)
(sapling split of 7127d341e6a20298684b877cb180162a55446b87) (sapling split of 919c87ea4f44b8d948a86b7dd38533e7f136ff86) (sapling split of 7d11ab4a525988dd462d2313795e3fc0ca3bf3a0)
* just change the target sources * remove generate_target_map as it is the same as parent now * buildgen * fix typo in helpful message (sapling split of f9651cd903cb687ee2cc32417f8ff0cf164c8302)
…ees (#9075) These just got lost in the noise of the pants upgrade due to the api change. I believe these are all the locations where the new behavior differed from what we were doing previously -- there is one other use in the recompile_on_changed script which I did not change because I suspect nobody knows about it or uses it, and it seemed like not including dependees for something like that is the correct move anyway. (sapling split of 5f35e6b764e7d1a31f9360c948e7fe6a174ff593)
This adds a new flag to the wheelwright pipeline, --python3.
./scripts/wheelhouse/build.sh --python3
This is off by default.
Passing this flag will run a second phase in the wheelwright that builds our python wheels.
This second phase passes the pinned python2 requirements.txt.freeze file, one at a time.
Pip then tries to build python3 whl for each individual line.
This new pipeline does not hard fail but instead aggregates failures. We have ~7-10 total
misses for python3 whls, the intention is for the followups to update/deprecate/remove
those libs.
This --python3 flag is not protected in CI today - we are punting that until the
requirements.txt.freeze has a translation layer for pins and extras.
(sapling split of a6e7d73b3a7f40ee11e611767f72a90ea6072fdf)
* just change the target sources * remove generate_target_map as it is the same as parent now * more aggressive filtering of stubs (sapling split of a5939391db688d77636f49672a29e0607a5dcfb9)
* Delete unused plugins and scripts We never fully productionized the tracing backends, although they were promising. Pom-resolve was a success for a long time, I published that library to PyPi as https://pypi.org/project/fsqio.pants.pom-resolve/ (sapling split of cac503b199d9cf669fd3965f89a0f8e84a7b9894)
* Support platform_dependent remote_sources. Adding a `platform_dependent="True"` to a remote_sources BUILD target will remove the linux/mac namespacing and instead use a universal path. Almost all of our remote_sources packages are platform-agnostic, they are almost all source or data packages. This can make a hell of a lot of difference, because we still passively support 3 OSX versions + linux. And some of our larger data packages (libpostal and AndroidSDK) are well over a gb. This patch cuts the storage by a quarter, so up to 4gb of cloud storage/transfer back for every version of the larger libraries. Using cc-shapefiles (our current example case), we migrated: ~/.cache/foursquare.web/bin/cc-shapefiles/mac/10.11/0.1b/cc-shapefiles.tar.gz ~/.cache/foursquare.web/bin/cc-shapefiles/mac/10.12/0.1b/cc-shapefiles.tar.gz ~/.cache/foursquare.web/bin/cc-shapefiles/mac/10.13/0.1b/cc-shapefiles.tar.gz ~/.cache/foursquare.web/bin/cc-shapefiles/linux/x86_64/0.1b/cc-shapefiles.tar.gz for ~/.cache/foursquare.web/bin/cc-shapefiles/0.1b/cc-shapefiles.tar.gz A future effort should deprecate the other source packages native-namespacing and earmark that binary hosting for future deletion. (sapling split of 13457fc07af24051f4a03e8a028f22ab4452dc8e)
(sapling split of f493bb3942427f1bbe09d9add10798f212423af3)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.