Feature/keycloak oauth2 integration POC nova only

.gitignore

@@ -1,9 +1,14 @@
 gh-md-toc
 config/config.yaml
 *.swp
+*.patch
 order-ipfo/*.conf
 order-ipfo/order-venv/
 done-*
+.kube/
 **/*.terraform
 **/*.tfstate*
 **/*.lock.hcl
+# Image files (downloaded for tests)
+*.img
+*qcow2

README.md

@@ -278,6 +278,7 @@ frep k8s/neutron.yaml.in:- --load config/config.yaml | kubectl apply -f -
 frep k8s/nova.yaml.in:- --load config/config.yaml | kubectl apply -f -
 frep k8s/skyline.yaml.in:- --load config/config.yaml | kubectl apply -f -
 frep k8s/mistral.yaml.in:- --load config/config.yaml | kubectl apply -f -
+frep k8s/keycloak.yaml.in:- --load config/config.yaml | kubectl apply -f -
 ```
 
 # compute-x and network-x
@@ -360,6 +361,10 @@ openstack server list
 ```
 If this is answering an empty line, you're good! (you don't have any instance yet)
 
+### Keycloak OAuth2 (optional)
+
+Nova supports dual authentication: Keystone (X-Auth-Token) and Keycloak OAuth2 (Bearer token). See [Keycloak OAuth2 feature](docs/KEYCLOAK_OAUTH2_FEATURE.md) for details.
+
 ## Skyline
 
 Skyline is a web interface, so you should browse the page.

config/config.yaml.sample

@@ -37,3 +37,9 @@ amp:
   flavor_id: "amp_flavor_id"
   boot_network_list: "amp_boot_network_list"
   secgroup_list: "amp_secgroup_list"
+
+# Keycloak OAuth2 configuration
+# Client secret for the nova-middleware client in Keycloak
+# This client is used by Nova middleware to authenticate with Keycloak for token introspection
+# The secret is automatically generated by keycloak-bootstrap and written to config.yaml
+keycloak_client_secret: changeme

docs/KEYCLOAK_OAUTH2_FEATURE.md

@@ -0,0 +1,74 @@
+# Keycloak OAuth2 integration for OpenStack Nova
+
+This feature adds **dual authentication** to Nova: requests can use either **Keystone** (X-Auth-Token) or **Keycloak OAuth2** (Bearer token). The same Nova API accepts both; no separate pipeline or endpoint.
+
+## What we did
+
+1. **Install Keycloak**
+   Keycloak is deployed on the control plane (k8s) with a dedicated MySQL database. The deploy userdata installs the Keycloak CLI (kcadm.sh) and creates a keycloakrc file (similar to openrc).
+
+2. **Configure Keycloak with Keystone**
+   After Keystone is bootstrapped, the deploy script configures Keycloak:
+   - Realm `openstack`
+   - Clients: `nova-middleware` (token introspection by Nova), `openstack-client` (user tokens)
+   - Protocol mappers so tokens carry OpenStack attributes: `project_id`, `project_name`, `project_domain`, `user_domain`, `project_domain_id`, `user_domain_id`, `openstack_roles` (string)
+   - User profile updated for Keycloak 26.x so custom attributes persist
+   - Realm role `nova:reboot` and users: `demo`, `demo-reboot-only`, `demo-keycloak`, `nova-restart-user` with correct attributes and passwords (demo password from openrc_demo)
+
+3. **Configure Nova**
+   - Nova uses a composite middleware that routes:
+     - `Authorization: Bearer <token>` -> keystonemiddleware `external_oauth2_token` (Keycloak)
+     - `X-Auth-Token: <token>` -> Keystone auth_token
+   - Nova conf: `[ext_oauth2_auth]` with Keycloak introspection URL, client credentials (nova-middleware), and claim mappings so the middleware gets project/domain/roles from the token.
+   - Nova policy: `os_compute_api:servers:reboot` (and `:create`) allow `role:nova:reboot` so Keycloak users with that role can reboot servers.
+   - Policy file is in a ConfigMap and mounted under `/etc/nova/policy.d`.
+
+4. **Tests**
+   - **Keycloak**: CLI installation, connection, realm openstack (clients, users, roles).
+   - **Nova**: create instance with Keystone (demo), create instance with Keycloak (demo-keycloak), reboot instance with Keycloak (demo-reboot-only), and a test that a user without create permission cannot create (keycloak-cannot-create). All tests run as the demo user (openrc_demo).
+
+## Flow
+
+- User gets an OAuth2 token from Keycloak (realm openstack, client openstack-client, e.g. username demo-reboot-only / demo password).
+- User calls Nova with `Authorization: Bearer <token>`.
+- Nova composite middleware forwards to external_oauth2_token, which introspects the token with Keycloak (using nova-middleware client).
+- Introspection response must contain the mapped claims (project_id, project_domain_id, user_domain_id, openstack_roles string, etc.); Keycloak is configured with protocol mappers and user attributes so that is true.
+- Middleware fills request env (project, user, roles); Nova policy allows reboot for role `nova:reboot`.
+
+## Files touched
+
+| Purpose | Files |
+|--------|--------|
+| Keycloak install | k8s/keycloak.yaml.in, k8s/mysql.yaml.in (keycloak db) |
+| Keycloak config at deploy | keycloak-bootstrap/, files/keycloakrc.in |
+| Nova dual auth | k8s/config.yaml.in (ext_oauth2_auth, api-paste.ini, nova policy), k8s/nova.yaml.in (policy.d mount) |
+| Tests | tests/keycloak/*, tests/nova/* |
+
+## Troubleshooting
+
+### Keycloak connectivity
+
+```bash
+curl http://keycloak.<domain>/health/ready
+```
+
+### Nova logs
+
+```bash
+kubectl logs deployment/nova-api -f
+```
+
+### Token introspection
+
+```bash
+curl -X POST "http://keycloak.<domain>/realms/openstack/protocol/openid-connect/token/introspect" \
+  -u "nova-middleware:<client_secret>" \
+  -d "token=$TOKEN" \
+  -d "token_type_hint=access_token"
+```
+
+### Common issues
+
+- **401 Unauthorized on introspection**: Check `keycloak_client_secret` in `config.yaml` matches the nova-middleware client secret in Keycloak.
+- **403 Forbidden**: Verify protocol mappers include required claims (`project_id`, `project_domain_id`, `user_domain_id`, `openstack_roles` string) and user attributes are set.
+- **Invalid token**: Check token expiration and realm (`openstack`).

files/keycloakrc.in

@@ -0,0 +1,22 @@
+# Keycloak CLI Configuration
+# This file is sourced to configure kcreg.sh and kcadm.sh credentials
+# Usage: source /root/keycloakrc
+
+export KEYCLOAK_URL=http://keycloak.{{.domain}}
+export KEYCLOAK_REALM=openstack
+export KEYCLOAK_USER=admin
+export KEYCLOAK_PASSWORD={{.password}}
+
+# Configure kcreg.sh credentials (Client Registration CLI)
+/root/keycloak-26.0.0/bin/kcreg.sh config credentials \
+  --server ${KEYCLOAK_URL} \
+  --realm master \
+  --user ${KEYCLOAK_USER} \
+  --password ${KEYCLOAK_PASSWORD} > /dev/null 2>&1 || true
+
+# Configure kcadm.sh credentials (Admin CLI)
+/root/keycloak-26.0.0/bin/kcadm.sh config credentials \
+  --server ${KEYCLOAK_URL} \
+  --realm master \
+  --user ${KEYCLOAK_USER} \
+  --password ${KEYCLOAK_PASSWORD} > /dev/null 2>&1 || true

k8s/config.yaml.in

@@ -219,6 +219,12 @@ data:
     os_region_name = RegionOne
     [database]
     connection = mysql+pymysql://root:{{.password}}@mysql-nova/nova
+    [api]
+    # Authentication strategy: keystone (default) or keystone+oauth2
+    # keystone: use Keystone only (default)
+    # keystone+oauth2: use OAuth2 (Keycloak) only
+    # Note: To allow both, you need to create a custom composite middleware
+    auth_strategy = keystone+oauth2
     [glance]
     api_servers = http://glance.{{.domain}}
     [keystone_authtoken]
@@ -232,6 +238,33 @@ data:
     region_name = RegionOne
     #memcached_servers = keystone:11211
     auth_type = password
+    # OAuth2.0 External Authorization Server (Keycloak) configuration
+    # This section allows Nova to authenticate with Keycloak OAuth2 tokens
+    # Section name must be [ext_oauth2_auth] for keystonemiddleware.external_oauth2_token
+    [ext_oauth2_auth]
+    # Keycloak introspection endpoint
+    introspect_endpoint = http://keycloak.{{.domain}}/realms/openstack/protocol/openid-connect/token/introspect
+    # Audience - matches the 'aud' field in Keycloak token (typically 'account' for user tokens)
+    audience = account
+    # Client credentials for Keycloak (used by middleware to authenticate for token introspection)
+    client_id = nova-middleware
+    client_secret = {{.keycloak_client_secret}}
+    # Authentication method: client_secret_basic, client_secret_post, tls_client_auth, client_secret_jwt, private_key_jwt
+    auth_method = client_secret_basic
+    # Mapping rules to extract OpenStack attributes from Keycloak token introspection response
+    # These map Keycloak token claims to OpenStack environment variables
+    mapping_user_id = sub
+    mapping_user_name = preferred_username
+    mapping_user_domain_id = user_domain_id
+    mapping_user_domain_name = user_domain
+    mapping_project_id = project_id
+    mapping_project_name = project_name
+    mapping_project_domain_id = project_domain_id
+    mapping_project_domain_name = project_domain
+    # Use openstack_roles (string "admin,member,reader") instead of realm_access.roles (array)
+    mapping_roles = openstack_roles
+    # Cache configuration
+    memcached_servers = memcached:11211
     [neutron]
     default_floating_pool = ext-net
     service_metadata_proxy = true
@@ -274,13 +307,207 @@ data:
     [cache]
     backend = oslo_cache.memcache_pool
     memcache_servers = memcached:11211
+    [oslo_policy]
+    policy_dirs = /etc/nova/policy.d
     [oslo_messaging_rabbit]
     rabbit_quorum_queue = True
     rabbit_transient_quorum_queue = True
     rabbit_qos_prefetch_count = 1
     use_queue_manager = True
     rabbit_stream_fanout = True
+  api-paste.ini: |
+    [composite:osapi_compute]
+    use = call:nova.api.openstack.urlmap:urlmap_factory
+    /: oscomputeversions
+    /v2: oscomputeversion_legacy_v2
+    /v2.1: oscomputeversion_v2
+    /v2/+: openstack_compute_api_v21_legacy_v2_compatible
+    /v2.1/+: openstack_compute_api_v21
+
+    [composite:main]
+    use = egg:Paste#urlmap
+    /v2 = openstack_compute_api_v2
+    /v2.1 = openstack_compute_api_v21
+    / = openstack_compute_api_legacy_v2
+
+    [composite:openstack_compute_api_v2]
+    use = call:nova.api.auth:pipeline_factory
+    noauth = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit noauth osapi_compute_app_v2
+    keystone = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v2
+    keystone_nolimit = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v2
+    keystone+oauth2 = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit external_oauth2_token keystonecontext osapi_compute_app_v2
+
+    [composite:openstack_compute_api_v21]
+    use = call:nova.api.auth:pipeline_factory_v21
+    noauth = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit noauth osapi_compute_app_v21
+    keystone = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit composite_auth keystonecontext osapi_compute_app_v21
+    keystone_nolimit = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit composite_auth keystonecontext osapi_compute_app_v21
+    oauth2 = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit external_oauth2_token keystonecontext osapi_compute_app_v21
+
+    [composite:openstack_compute_api_legacy_v2]
+    use = call:nova.api.auth:pipeline_factory
+    noauth = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit noauth osapi_compute_app_v21
+    keystone = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v21
+
+    [composite:openstack_compute_api_v21_legacy_v2_compatible]
+    use = call:nova.api.auth:pipeline_factory_v21
+    keystone = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v21
+
+    [pipeline:oscomputeversions]
+    pipeline = cors faultwrap request_log http_proxy_to_wsgi oscomputeversionapp
+
+    [pipeline:oscomputeversion_v2]
+    pipeline = cors compute_req_id faultwrap request_log http_proxy_to_wsgi oscomputeversionapp_v2
+
+    [pipeline:oscomputeversion_legacy_v2]
+    pipeline = cors compute_req_id faultwrap request_log http_proxy_to_wsgi oscomputeversionapp_v2
+
+    [app:oscomputeversionapp]
+    paste.app_factory = nova.api.openstack.compute.versions:Versions.factory
+
+    [app:oscomputeversionapp_v2]
+    paste.app_factory = nova.api.openstack.compute.versions:VersionsV2.factory
+
+    [filter:external_oauth2_token]
+    paste.filter_factory = keystonemiddleware.external_oauth2_token:filter_factory
+
+    [filter:composite_auth]
+    paste.filter_factory = nova_oauth2_middleware:filter_factory
+
+    [filter:authtoken]
+    paste.filter_factory = keystonemiddleware.auth_token:filter_factory
+
+    [filter:keystonecontext]
+    paste.filter_factory = nova.api.auth:NovaKeystoneContext.factory
+
+    [filter:compute_req_id]
+    paste.filter_factory = nova.api.compute_req_id:ComputeReqIdMiddleware.factory
+
+    [filter:request_log]
+    paste.filter_factory = nova.api.openstack.requestlog:RequestLog.factory
+
+    [filter:faultwrap]
+    paste.filter_factory = nova.api.openstack:FaultWrapper.factory
+
+    [filter:osprofiler]
+    paste.filter_factory = nova.profiler:WsgiMiddleware.factory
+
+    [filter:noauth]
+    paste.filter_factory = nova.api.openstack.auth:NoAuthMiddleware.factory
+
+    [filter:sizelimit]
+    paste.filter_factory = oslo_middleware:RequestBodySizeLimiter.factory
+
+    [filter:http_proxy_to_wsgi]
+    paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory
+
+    [filter:cors]
+    paste.filter_factory = oslo_middleware.cors:filter_factory
+    oslo_config_project = nova
 
+    [app:osapi_compute_app_v2]
+    paste.app_factory = nova.api.openstack.compute:APIRouter.factory
+
+    [app:osapi_compute_app_v21]
+    paste.app_factory = nova.api.openstack.compute:APIRouterV21.factory
+
+  nova_oauth2_middleware.py: |
+    """
+    Composite middleware for Nova allowing both OAuth2 (Keycloak) and Keystone authentication.
+
+    Strategy:
+    - If Authorization: Bearer header is present -> use OAuth2 (Keycloak)
+    - If X-Auth-Token header is present -> use Keystone
+    - Otherwise -> return 401
+    """
+    import webob.dec
+    import webob.exc
+    from oslo_log import log as logging
+
+    LOG = logging.getLogger(__name__)
+
+
+    class CompositeAuthMiddleware(object):
+        """Middleware that routes to OAuth2 or Keystone based on the auth header."""
+
+        def __init__(self, app, conf):
+            self.app = app
+            self.conf = conf
+            self._oauth2_middleware = None
+            self._keystone_middleware = None
+
+            # Lazy load middlewares
+            self._init_oauth2()
+            self._init_keystone()
+
+        def _init_oauth2(self):
+            try:
+                from keystonemiddleware.external_oauth2_token import ExternalAuth2Protocol
+                self._oauth2_middleware = ExternalAuth2Protocol(self.app, self.conf)
+                LOG.info("OAuth2 middleware initialized successfully")
+            except Exception as e:
+                LOG.warning("Could not initialize OAuth2 middleware: %s", e)
+                self._oauth2_middleware = None
+
+        def _init_keystone(self):
+            try:
+                from keystonemiddleware import auth_token
+                self._keystone_middleware = auth_token.AuthProtocol(self.app, self.conf)
+                LOG.info("Keystone middleware initialized successfully")
+            except Exception as e:
+                LOG.warning("Could not initialize Keystone middleware: %s", e)
+                self._keystone_middleware = None
+
+        @webob.dec.wsgify
+        def __call__(self, req):
+            # Check for Bearer token (OAuth2)
+            auth_header = req.headers.get('Authorization', '')
+            x_auth_token = req.headers.get('X-Auth-Token', '')
+
+            if auth_header.lower().startswith('bearer '):
+                # Use OAuth2 middleware
+                if self._oauth2_middleware:
+                    LOG.debug("Using OAuth2 authentication (Bearer token present)")
+                    return req.get_response(self._oauth2_middleware)
+                else:
+                    LOG.warning("Bearer token provided but OAuth2 middleware not available")
+                    return webob.exc.HTTPUnauthorized("OAuth2 authentication not configured")
+
+            elif x_auth_token:
+                # Use Keystone middleware
+                if self._keystone_middleware:
+                    LOG.debug("Using Keystone authentication (X-Auth-Token present)")
+                    return req.get_response(self._keystone_middleware)
+                else:
+                    LOG.warning("X-Auth-Token provided but Keystone middleware not available")
+                    return webob.exc.HTTPUnauthorized("Keystone authentication not configured")
+
+            else:
+                # No auth header - pass to app (will likely fail with 401)
+                LOG.debug("No authentication header found, passing to application")
+                return req.get_response(self.app)
+
+
+    def filter_factory(global_conf, **local_conf):
+        """Factory to create the composite middleware."""
+        conf = global_conf.copy()
+        conf.update(local_conf)
+
+        def filter_(app):
+            return CompositeAuthMiddleware(app, conf)
+
+        return filter_
+
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nova-policy
+data:
+  keycloak-policy.yaml: |
+    # Allow Keycloak role nova:reboot to perform server reboot
+    "os_compute_api:servers:reboot": "role:nova:reboot or role:admin"
+    "os_compute_api:servers:reboot:create": "role:nova:reboot or role:admin"
 ---
 apiVersion: v1
 kind: ConfigMap