Skip to content

Migrate to Python 3.10#171

Draft
Copilot wants to merge 26 commits intodevelopmentfrom
copilot/update-to-python-3-10
Draft

Migrate to Python 3.10#171
Copilot wants to merge 26 commits intodevelopmentfrom
copilot/update-to-python-3-10

Conversation

Copy link

Copilot AI commented Dec 8, 2025 •
edited
Loading

Updates the repository to use Python 3.10 across all components (Pipfile, Docker, everywhere).

Changes

Pipfile

  • python_version: 3.8 → 3.10

ckan/Dockerfile

  • Base image: python:3.8 → python:3.10
  • Removed python3-distutils (included in Python 3.10+)
  • Added ca-certificates for SSL support
  • Fixed Node.js/npm installation for Debian Trixie (Python 3.10 base OS)

docker-compose.yml

  • Added build context for ckan and supervisor services to enable local image builds

Notes

Pipfile.lock regeneration will occur during next CI/CD build with initialized submodules.

Verified: Python 3.10.19, pip 25.3, pipenv 2025.1.1 installed and functional in CKAN image.

Original prompt

This section details on the original issue you should resolve

<issue_title>Move to Python 3.10</issue_title>
<issue_description>Update the repo to use Python 3.10 everywhere - this means pipenv, docker, everywhere.</issue_description>

<agent_instructions>To test your change run docker compose up</agent_instructions>

Comments on the Issue (you are @copilot in this section)

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits December 8, 2025 12:03
@ChasNelson1990
Update Python version to 3.10 in Pipfile and Dockerfile
37a5f41 
Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>
@ChasNelson1990
Update Dockerfile for Python 3.10 compatibility and add local build c…
4173c3a 
…onfig

Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>
@ChasNelson1990
Add documentation comments for SSL workarounds in Dockerfile
93be52a 
Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>
Copilot AI changed the title [WIP] Update repository to use Python 3.10 Migrate to Python 3.10 Dec 8, 2025
Copilot AI requested a review from ChasNelson1990 December 8, 2025 12:24
ChasNelson1990
ChasNelson1990 requested changes Dec 8, 2025
View reviewed changes
Copy link
Member

@ChasNelson1990 ChasNelson1990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove strict-ssl and trusted-host flags - they will not be needed in our systems

@ChasNelson1990
Remove SSL workaround flags as they are not needed in production systems
9654350 
Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>
Copy link
Author

Copilot AI commented Dec 8, 2025

Remove strict-ssl and trusted-host flags - they will not be needed in our systems

Done. Removed --trusted-host flags from pip commands and strict-ssl false from npm command, along with the related documentation comments. (9654350)

Copilot AI requested a review from ChasNelson1990 December 8, 2025 12:49
@ChasNelson1990
Copy link
Member

ChasNelson1990 commented Dec 9, 2025

@A-Souhei - I have some additional work on this PR on my other laptop. Focus on those ckanexts for now I think.

@A-Souhei
Upgrade to Python 3.10 and CKAN 2.11.4
c31442f 
Major changes:
- Updated all CKAN core dependencies to 2.11.4 versions
- Fixed Flask 3.x compatibility issues
- Fixed Python 3.10 collections.abc imports
- Updated all CKAN 2.11.4 API changes across extensions
- Replaced deprecated ReclineView with DataTablesView
- Added missing dependencies: frictionless, markupsafe
- Re-added ckanext-pages that was accidentally removed
- Updated plugin configuration to remove deprecated view plugins

See PYTHON310_CKAN2114_MIGRATION.md for detailed change log.
@A-Souhei
Update submodule commits for CKAN 2.11.4 compatibility
6a6785c
@A-Souhei
Update Pipfile.lock after adding frictionless dependency
5bb15d6
@A-Souhei
Fix ckanext-pages installation
4aa888d 
- Remove editable flag from git dependency (caused pipenv errors)
- Keep as regular git dependency pointing to v0.5.2
- Update migration notes
@A-Souhei
Remove sentry plugin from configuration
a63c79b 
- ckanext-sentry is not installed
- Remove from plugin list to avoid PluginNotFoundException
- Update migration notes
@A-Souhei
Fix ckanext-versions SQLAlchemy 1.4 compatibility
5f23078 
- Update model.py to pass bind parameter to table operations
- Fixes UnboundExecutionError when checking if tables exist
- Update migration notes
@A-Souhei
Update ckanext-versions: use Inspector for SQLAlchemy 1.4
5a1cb50 
- Use inspector.has_table() instead of deprecated table.exists()
- Properly handle engine binding for table operations
@A-Souhei
Fix SQLAlchemy 1.4 compatibility across all extensions
5a75ce7 
- ckanext-versions: Use inspector for table checks
- ckanext-validation: Use inspector for table checks
- ckanext-unaids: Use inspector for dataset_transfer table checks
- All use None check for engine during startup
- Update migration documentation
@A-Souhei
Add DataPusher API token configuration for CKAN 2.11.4 compatibility
264d2d2
@A-Souhei
Update ckanext-blob-storage submodule: fix setup.py version import
55d7f65
@A-Souhei
Add ckanext-pages as submodule and fix SQLAlchemy 1.4 compatibility i…
9b0f2bb 
…n ckanext-pages and ckanext-harvest
@A-Souhei
Fix pipenv dependency installation and add missing packages
6c5dbef 
- Remove --skip-lock flag from bootstrap.sh to ensure proper dependency resolution
- Add async-timeout==4.0.3 (required by redis 5.0.7 on Python 3.10)
- Add numpy>=1.21.0 (required by pandas>=1.4.2)
- Update migration documentation with fixes
@A-Souhei
Add verbose output and error checking to pipenv install
a49d38b 
- Add --verbose flag to see detailed package installation progress
- Add error checking to fail fast if pipenv install fails
- Add success message after dependencies are installed
- Pipfile.lock updated with numpy 2.2.6 and other dependency updates
@A-Souhei
Fix infinite recursion in ckanext-unaids user identification
a9edc90 
- Update ckanext-unaids to use direct g.user/g.userobj assignment
- Avoid calling identify_user() which causes infinite recursion
- Update migration documentation with detailed explanation
@A-Souhei
Add activity plugin to fix package_activity_list chained action
236eabe 
- In CKAN 2.11.4, activity functionality moved to ckanext/activity extension
- The activity plugin must be enabled for package_activity_list action to exist
- ckanext-unaids chains to this action, causing NotFound error when missing
- This also fixes the KeyError: 'model' issue which was caused by action
  wrapping failing during plugin initialization
@A-Souhei
Fix ckanext-restricted AnonymousUser handling for CKAN 2.11.4
e164761 
- Update restricted_get_username_from_context to check is_authenticated
- Handle Flask-Login's AnonymousUser which lacks as_dict() method
- Update migration documentation
@A-Souhei
Fix ckanext-emailasusername for CKAN 2.11.4 User.by_email() change
5943813 
- Handle None return from User.by_email() instead of empty list
- Fixes TypeError during OAuth user creation
- Update migration documentation
@A-Souhei
Add Flask 3.x compatibility patch for ckanext-saml2auth
291acb2 
- Create patch_saml2auth.py to fix session serialization issues
- Convert SAML ava dict values to strings for JSON compatibility
- Add patch execution to bootstrap.sh
- Update migration documentation

This fixes TypeError: Object of type NameID is not JSON serializable
during SAML OAuth login in Flask 3.x
@A-Souhei
Enhance SAML2 auth patching for Flask 3.x session serialization
3d15a74 
- Improve patch_saml2auth.py to handle both ava and session_info objects
- Add dynamic venv path detection for Docker and local environments
- Ensure all SAML session data is JSON serializable for Flask 3.x
- Update Pipfile.lock platform markers

The activity plugin was already correctly configured in CKAN 2.11.4.
The plugin loading issue was resolved by allowing bootstrap process to complete.
@A-Souhei
Update migration docs with activity plugin troubleshooting notes
c50dee6 
- Add clarification about activity plugin entry point registration
- Document bootstrap process completion requirement
- Add SAML2 session serialization fixes to issues list
- Document activity plugin verification steps
@A-Souhei
Update submodule URLs to use adx-tmp forks
f867795 
- Change ckanext-authz-service: datopian -> adx-tmp
- Change ckanext-pages: ckan -> adx-tmp

This allows us to maintain our Python 3.10/CKAN 2.11.4 migration
patches without pushing to official upstream repositories.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChasNelson1990 ChasNelson1990 Awaiting requested review from ChasNelson1990

Requested changes must be addressed to merge this pull request.

Assignees

@ChasNelson1990 ChasNelson1990

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Move to Python 3.10

3 participants

@ChasNelson1990 @A-Souhei

Comments