2019-02-07: FG; Reverted back to functional version, very close to charm version 31.

README.md

 Overview
 ========
 
-This charm provides Keystone, the Openstack identity service. Its target
+This charm provides Keystone, the Openstack identity service. It's target
 platform is (ideally) Ubuntu LTS + Openstack.
 
 This is a modified version,  which adds support for Identity
@@ -14,7 +14,7 @@ The following interfaces are provided:
 
     - nrpe-external-master: Used to generate Nagios checks.
 
-    - identity-service: Openstack API endpoints request an entry in the
+    - identity-service: Openstack API endpoints request an entry in the 
       Keystone service catalog + endpoint template catalog. When a relation
       is established, Keystone receives: service name, region, public_url,
       admin_url and internal_url. It first checks that the requested service
@@ -97,28 +97,33 @@ If 'dns-ha' is set and none of the os-{admin,internal,public}-hostname(s) are se
 SSL/HTTPS
 ---------
 
-Support for SSL and https endpoint is provided via various charm configuration
-options.
+Support for SSL and https endpoint is provided via a set of configuration
+options on the charm. There are two types supported;
 
-To enable SSL and https endpoint with a charm-generated CA, set the following
-configuration options:
+use-https - if enabled this option tells Keystone to configure the identity
+endpoint as https. Under this model the keystone charm will either use the CA
+as provided by the user (see ssl_* options below) or will generate its own and
+sync across peers. The cert will be distributed to all service endpoints which
+will be configured to use https.
 
-- use-https - if enabled this option tells Keystone to configure the identity
-  endpoint as https, and the keystone charm will generate its own CA and sync
-  across peers. The cert will be distributed to all service endpoints which
-  will be configured to use https.
+https-service-endpoints - if enabled this option tells Keystone to configure
+ALL endpoints as https. Under this model the keystone charm will either use the
+CA as provided by the user (see ssl_* options below) or will generate its own
+and sync across peers. The cert will be distributed to all service endpoints
+which will be configured to use https as well as configuring themselves to be
+used as https.
 
-- https-service-endpoints - if enabled this option tells Keystone to configure
-  ALL endpoints as https. Under this model the keystone charm will generate its
-  own CA and sync across peers. The cert will be distributed to all service
-  endpoints which will be configured to use https as well as configuring
-  themselves to be used as https.
+When configuring the charms to use SSL there are three charm config options as
+ssl_ca, ssl_cert and ssl_key.
 
-To enable SSL and https endpoint with your own CA, SSL cert, and key set the
-following configuration options: ssl_ca, ssl_cert, and ssl_key. The user can
-provide SSL cert and key using ssl_cert and ssl_key only when the cert is
-signed by a trusted CA. These options should not be used with use-https and
-https-service-endpoints.
+- The user can provide their own CA, SSL cert and key using the options ssl_ca,
+  ssl_cert, ssl_key.
+
+- The user can provide SSL cert and key using ssl_cert and ssl_key when the cert
+  is signed by a trusted CA.
+
+- If not provided, the keystone charm will automatically generate a CA and certs
+  to distribute to endpoints.
 
 When the charm configures itself as a CA (generally only recommended for test
 purposes) it will elect an "ssl-cert-master" whose duty is to generate the CA
@@ -164,7 +169,7 @@ To use this feature, use the --bind option when deploying the charm:
 
     juju deploy keystone --bind "public=public-space internal=internal-space admin=admin-space shared-db=internal-space"
 
-Alternatively these can also be provided as part of a juju native bundle configuration:
+alternatively these can also be provided as part of a juju native bundle configuration:
 
     keystone:
       charm: cs:xenial/keystone
@@ -177,7 +182,7 @@ Alternatively these can also be provided as part of a juju native bundle configu
 
 NOTE: Spaces must be configured in the underlying provider prior to attempting to use them.
 
-NOTE: Existing deployments using os\-\*-network configuration options will continue to function; these options are preferred over any network space binding provided if set.
+NOTE: Existing deployments using os-*-network configuration options will continue to function; these options are preferred over any network space binding provided if set.
 
 Federated Authentication
 ------------------------

actions.yaml

+git-reinstall:
+  description: Reinstall keystone from the openstack-origin-git repositories.
 pause:
   description: |
     Pause keystone services.

actions/charmhelpers

-../charmhelpers
\ No newline at end of file

actions/git-reinstall

+#!/usr/bin/python
+#
+# Copyright 2016 Canonical Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import traceback
+
+from charmhelpers.contrib.openstack.utils import (
+    git_install_requested,
+)
+
+from charmhelpers.core.hookenv import (
+    action_set,
+    action_fail,
+    config,
+)
+
+from hooks.keystone_utils import (
+    git_install,
+)
+
+from hooks.keystone_hooks import (
+    config_changed,
+)
+
+
+def git_reinstall():
+    """Reinstall from source and restart services.
+
+    If the openstack-origin-git config option was used to install openstack
+    from source git repositories, then this action can be used to reinstall
+    from updated git repositories, followed by a restart of services."""
+    if not git_install_requested():
+        action_fail('openstack-origin-git is not configured')
+        return
+
+    try:
+        git_install(config('openstack-origin-git'))
+        config_changed()
+    except:
+        action_set({'traceback': traceback.format_exc()})
+        action_fail('git-reinstall resulted in an unexpected error')
+
+
+if __name__ == '__main__':
+    git_reinstall()

actions/git_reinstall.py

+#!/usr/bin/python
+#
+# Copyright 2016 Canonical Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import traceback
+
+from charmhelpers.contrib.openstack.utils import (
+    git_install_requested,
+)
+
+from charmhelpers.core.hookenv import (
+    action_set,
+    action_fail,
+    config,
+)
+
+from hooks.keystone_utils import (
+    git_install,
+)
+
+from hooks.keystone_hooks import (
+    config_changed,
+)
+
+
+def git_reinstall():
+    """Reinstall from source and restart services.
+
+    If the openstack-origin-git config option was used to install openstack
+    from source git repositories, then this action can be used to reinstall
+    from updated git repositories, followed by a restart of services."""
+    if not git_install_requested():
+        action_fail('openstack-origin-git is not configured')
+        return
+
+    try:
+        git_install(config('openstack-origin-git'))
+        config_changed()
+    except:
+        action_set({'traceback': traceback.format_exc()})
+        action_fail('git-reinstall resulted in an unexpected error')
+
+
+if __name__ == '__main__':
+    git_reinstall()

actions/hooks

-../hooks
\ No newline at end of file

charmhelpers/charmhelpers

-../charmhelpers
\ No newline at end of file

charmhelpers/contrib/hahelpers/apache.py

@@ -65,8 +65,7 @@ def get_ca_cert():
     if ca_cert is None:
         log("Inspecting identity-service relations for CA SSL certificate.",
             level=INFO)
-        for r_id in (relation_ids('identity-service') +
-                     relation_ids('identity-credentials')):
+        for r_id in relation_ids('identity-service'):
             for unit in relation_list(r_id):
                 if ca_cert is None:
                     ca_cert = relation_get('ca_cert',
@@ -77,7 +76,7 @@ def get_ca_cert():
 def retrieve_ca_cert(cert_file):
     cert = None
     if os.path.isfile(cert_file):
-        with open(cert_file, 'rb') as crt:
+        with open(cert_file, 'r') as crt:
             cert = crt.read()
     return cert
 

charmhelpers/contrib/hahelpers/cluster.py

@@ -223,11 +223,6 @@ def https():
         return True
     if config_get('ssl_cert') and config_get('ssl_key'):
         return True
-    for r_id in relation_ids('certificates'):
-        for unit in relation_list(r_id):
-            ca = relation_get('ca', rid=r_id, unit=unit)
-            if ca:
-                return True
     for r_id in relation_ids('identity-service'):
         for unit in relation_list(r_id):
             # TODO - needs fixing for new helper as ssl_cert/key suffixes with CN
@@ -376,7 +371,6 @@ def distributed_wait(modulo=None, wait=None, operation_name='operation'):
     ''' Distribute operations by waiting based on modulo_distribution
 
     If modulo and or wait are not set, check config_get for those values.
-    If config values are not set, default to modulo=3 and wait=30.
 
     :param modulo: int The modulo number creates the group distribution
     :param wait: int The constant time wait value
@@ -388,17 +382,10 @@ def distributed_wait(modulo=None, wait=None, operation_name='operation'):
     :side effect: Calls time.sleep()
     '''
     if modulo is None:
-        modulo = config_get('modulo-nodes') or 3
+        modulo = config_get('modulo-nodes')
     if wait is None:
-        wait = config_get('known-wait') or 30
-    if juju_is_leader():
-        # The leader should never wait
-        calculated_wait = 0
-    else:
-        # non_zero_wait=True guarantees the non-leader who gets modulo 0
-        # will still wait
-        calculated_wait = modulo_distribution(modulo=modulo, wait=wait,
-                                              non_zero_wait=True)
+        wait = config_get('known-wait')
+    calculated_wait = modulo_distribution(modulo=modulo, wait=wait)
     msg = "Waiting {} seconds for {} ...".format(calculated_wait,
                                                  operation_name)
     log(msg, DEBUG)

charmhelpers/contrib/network/ip.py

@@ -27,7 +27,6 @@ from charmhelpers.core.hookenv import (
     network_get_primary_address,
     unit_get,
     WARNING,
-    NoNetworkBinding,
 )
 
 from charmhelpers.core.host import (
@@ -110,12 +109,7 @@ def get_address_in_network(network, fallback=None, fatal=False):
         _validate_cidr(network)
         network = netaddr.IPNetwork(network)
         for iface in netifaces.interfaces():
-            try:
-                addresses = netifaces.ifaddresses(iface)
-            except ValueError:
-                # If an instance was deleted between
-                # netifaces.interfaces() run and now, its interfaces are gone
-                continue
+            addresses = netifaces.ifaddresses(iface)
             if network.version == 4 and netifaces.AF_INET in addresses:
                 for addr in addresses[netifaces.AF_INET]:
                     cidr = netaddr.IPNetwork("%s/%s" % (addr['addr'],
@@ -584,9 +578,6 @@ def get_relation_ip(interface, cidr_network=None):
     except NotImplementedError:
         # If network-get is not available
         address = get_host_ip(unit_get('private-address'))
-    except NoNetworkBinding:
-        log("No network binding for {}".format(interface), WARNING)
-        address = get_host_ip(unit_get('private-address'))
 
     if config('prefer-ipv6'):
         # Currently IPv6 has priority, eventually we want IPv6 to just be

charmhelpers/contrib/openstack/amulet/deployment.py

@@ -21,9 +21,6 @@ from collections import OrderedDict
 from charmhelpers.contrib.amulet.deployment import (
     AmuletDeployment
 )
-from charmhelpers.contrib.openstack.amulet.utils import (
-    OPENSTACK_RELEASES_PAIRS
-)
 
 DEBUG = logging.DEBUG
 ERROR = logging.ERROR
@@ -274,8 +271,11 @@ class OpenStackAmuletDeployment(AmuletDeployment):
            release.
            """
         # Must be ordered by OpenStack release (not by Ubuntu release):
-        for i, os_pair in enumerate(OPENSTACK_RELEASES_PAIRS):
-            setattr(self, os_pair, i)
+        (self.trusty_icehouse, self.trusty_kilo, self.trusty_liberty,
+         self.trusty_mitaka, self.xenial_mitaka, self.xenial_newton,
+         self.yakkety_newton, self.xenial_ocata, self.zesty_ocata,
+         self.xenial_pike, self.artful_pike, self.xenial_queens,
+         self.bionic_queens,) = range(13)
 
         releases = {
             ('trusty', None): self.trusty_icehouse,
@@ -291,8 +291,6 @@ class OpenStackAmuletDeployment(AmuletDeployment):
             ('zesty', None): self.zesty_ocata,
             ('artful', None): self.artful_pike,
             ('bionic', None): self.bionic_queens,
-            ('bionic', 'cloud:bionic-rocky'): self.bionic_rocky,
-            ('cosmic', None): self.cosmic_rocky,
         }
         return releases[(self.series, self.openstack)]
 
@@ -308,7 +306,6 @@ class OpenStackAmuletDeployment(AmuletDeployment):
             ('zesty', 'ocata'),
             ('artful', 'pike'),
             ('bionic', 'queens'),
-            ('cosmic', 'rocky'),
         ])
         if self.openstack:
             os_origin = self.openstack.split(':')[1]

charmhelpers/contrib/openstack/amulet/utils.py

@@ -40,7 +40,6 @@ import novaclient
 import pika
 import swiftclient
 
-from charmhelpers.core.decorators import retry_on_exception
 from charmhelpers.contrib.amulet.utils import (
     AmuletUtils
 )
@@ -51,13 +50,6 @@ ERROR = logging.ERROR
 
 NOVA_CLIENT_VERSION = "2"
 
-OPENSTACK_RELEASES_PAIRS = [
-    'trusty_icehouse', 'trusty_kilo', 'trusty_liberty',
-    'trusty_mitaka', 'xenial_mitaka', 'xenial_newton',
-    'yakkety_newton', 'xenial_ocata', 'zesty_ocata',
-    'xenial_pike', 'artful_pike', 'xenial_queens',
-    'bionic_queens', 'bionic_rocky', 'cosmic_rocky']
-
 
 class OpenStackAmuletUtils(AmuletUtils):
     """OpenStack amulet utilities.
@@ -71,34 +63,7 @@ class OpenStackAmuletUtils(AmuletUtils):
         super(OpenStackAmuletUtils, self).__init__(log_level)
 
     def validate_endpoint_data(self, endpoints, admin_port, internal_port,
-                               public_port, expected, openstack_release=None):
-        """Validate endpoint data. Pick the correct validator based on
-           OpenStack release. Expected data should be in the v2 format:
-           {
-               'id': id,
-               'region': region,
-               'adminurl': adminurl,
-               'internalurl': internalurl,
-               'publicurl': publicurl,
-               'service_id': service_id}
-
-           """
-        validation_function = self.validate_v2_endpoint_data
-        xenial_queens = OPENSTACK_RELEASES_PAIRS.index('xenial_queens')
-        if openstack_release and openstack_release >= xenial_queens:
-                validation_function = self.validate_v3_endpoint_data
-                expected = {
-                    'id': expected['id'],
-                    'region': expected['region'],
-                    'region_id': 'RegionOne',
-                    'url': self.valid_url,
-                    'interface': self.not_null,
-                    'service_id': expected['service_id']}
-        return validation_function(endpoints, admin_port, internal_port,
-                                   public_port, expected)
-
-    def validate_v2_endpoint_data(self, endpoints, admin_port, internal_port,
-                                  public_port, expected):
+                               public_port, expected):
         """Validate endpoint data.
 
            Validate actual endpoint data vs expected endpoint data. The ports
@@ -127,7 +92,7 @@ class OpenStackAmuletUtils(AmuletUtils):
             return 'endpoint not found'
 
     def validate_v3_endpoint_data(self, endpoints, admin_port, internal_port,
-                                  public_port, expected, expected_num_eps=3):
+                                  public_port, expected):
         """Validate keystone v3 endpoint data.
 
         Validate the v3 endpoint data which has changed from v2.  The
@@ -173,89 +138,10 @@ class OpenStackAmuletUtils(AmuletUtils):
                 if ret:
                     return 'unexpected endpoint data - {}'.format(ret)
 
-        if len(found) != expected_num_eps:
+        if len(found) != 3:
             return 'Unexpected number of endpoints found'
 
-    def convert_svc_catalog_endpoint_data_to_v3(self, ep_data):
-        """Convert v2 endpoint data into v3.
-
-           {
-               'service_name1': [
-                   {
-                       'adminURL': adminURL,
-                       'id': id,
-                       'region': region.
-                       'publicURL': publicURL,
-                       'internalURL': internalURL
-                   }],
-               'service_name2': [
-                   {
-                       'adminURL': adminURL,
-                       'id': id,
-                       'region': region.
-                       'publicURL': publicURL,
-                       'internalURL': internalURL
-                   }],
-           }
-          """
-        self.log.warn("Endpoint ID and Region ID validation is limited to not "
-                      "null checks after v2 to v3 conversion")
-        for svc in ep_data.keys():
-            assert len(ep_data[svc]) == 1, "Unknown data format"
-            svc_ep_data = ep_data[svc][0]
-            ep_data[svc] = [
-                {
-                    'url': svc_ep_data['adminURL'],
-                    'interface': 'admin',
-                    'region': svc_ep_data['region'],
-                    'region_id': self.not_null,
-                    'id': self.not_null},
-                {
-                    'url': svc_ep_data['publicURL'],
-                    'interface': 'public',
-                    'region': svc_ep_data['region'],
-                    'region_id': self.not_null,
-                    'id': self.not_null},
-                {
-                    'url': svc_ep_data['internalURL'],
-                    'interface': 'internal',
-                    'region': svc_ep_data['region'],
-                    'region_id': self.not_null,
-                    'id': self.not_null}]
-        return ep_data
-
-    def validate_svc_catalog_endpoint_data(self, expected, actual,
-                                           openstack_release=None):
-        """Validate service catalog endpoint data. Pick the correct validator
-           for the OpenStack version. Expected data should be in the v2 format:
-           {
-               'service_name1': [
-                   {
-                       'adminURL': adminURL,
-                       'id': id,
-                       'region': region.
-                       'publicURL': publicURL,
-                       'internalURL': internalURL
-                   }],
-               'service_name2': [
-                   {
-                       'adminURL': adminURL,
-                       'id': id,
-                       'region': region.
-                       'publicURL': publicURL,
-                       'internalURL': internalURL
-                   }],
-           }
-
-           """
-        validation_function = self.validate_v2_svc_catalog_endpoint_data
-        xenial_queens = OPENSTACK_RELEASES_PAIRS.index('xenial_queens')
-        if openstack_release and openstack_release >= xenial_queens:
-            validation_function = self.validate_v3_svc_catalog_endpoint_data
-            expected = self.convert_svc_catalog_endpoint_data_to_v3(expected)
-        return validation_function(expected, actual)
-
-    def validate_v2_svc_catalog_endpoint_data(self, expected, actual):
+    def validate_svc_catalog_endpoint_data(self, expected, actual):
         """Validate service catalog endpoint data.
 
            Validate a list of actual service catalog endpoints vs a list of
@@ -424,7 +310,6 @@ class OpenStackAmuletUtils(AmuletUtils):
         self.log.debug('Checking if tenant exists ({})...'.format(tenant))
         return tenant in [t.name for t in keystone.tenants.list()]
 
-    @retry_on_exception(num_retries=5, base_delay=1)
     def keystone_wait_for_propagation(self, sentry_relation_pairs,
                                       api_version):
         """Iterate over list of sentry and relation tuples and verify that
@@ -443,7 +328,7 @@ class OpenStackAmuletUtils(AmuletUtils):
             if rel.get('api_version') != str(api_version):
                 raise Exception("api_version not propagated through relation"
                                 " data yet ('{}' != '{}')."
-                                "".format(rel.get('api_version'), api_version))
+                                "".format(rel['api_version'], api_version))
 
     def keystone_configure_api_version(self, sentry_relation_pairs, deployment,
                                        api_version):
@@ -465,13 +350,16 @@ class OpenStackAmuletUtils(AmuletUtils):
         deployment._auto_wait_for_status()
         self.keystone_wait_for_propagation(sentry_relation_pairs, api_version)
 
-    def authenticate_cinder_admin(self, keystone, api_version=2):
+    def authenticate_cinder_admin(self, keystone_sentry, username,
+                                  password, tenant, api_version=2):
         """Authenticates admin user with cinder."""
-        self.log.debug('Authenticating cinder admin...')
+        # NOTE(beisner): cinder python client doesn't accept tokens.
+        keystone_ip = keystone_sentry.info['public-address']
+        ept = "http://{}:5000/v2.0".format(keystone_ip.strip().decode('utf-8'))
         _clients = {
             1: cinder_client.Client,
             2: cinder_clientv2.Client}
-        return _clients[api_version](session=keystone.session)
+        return _clients[api_version](username, password, tenant, ept)
 
     def authenticate_keystone(self, keystone_ip, username, password,
                               api_version=False, admin_port=False,
@@ -479,36 +367,13 @@ class OpenStackAmuletUtils(AmuletUtils):
                               project_domain_name=None, project_name=None):
         """Authenticate with Keystone"""
         self.log.debug('Authenticating with keystone...')
-        if not api_version:
-            api_version = 2
-        sess, auth = self.get_keystone_session(
-            keystone_ip=keystone_ip,
-            username=username,
-            password=password,
-            api_version=api_version,
-            admin_port=admin_port,
-            user_domain_name=user_domain_name,
-            domain_name=domain_name,
-            project_domain_name=project_domain_name,
-            project_name=project_name
-        )
-        if api_version == 2:
-            client = keystone_client.Client(session=sess)
-        else:
-            client = keystone_client_v3.Client(session=sess)
-        # This populates the client.service_catalog
-        client.auth_ref = auth.get_access(sess)
-        return client
-
-    def get_keystone_session(self, keystone_ip, username, password,
-                             api_version=False, admin_port=False,
-                             user_domain_name=None, domain_name=None,
-                             project_domain_name=None, project_name=None):
-        """Return a keystone session object"""
-        ep = self.get_keystone_endpoint(keystone_ip,
-                                        api_version=api_version,
-                                        admin_port=admin_port)
-        if api_version == 2:
+        port = 5000
+        if admin_port:
+            port = 35357
+        base_ep = "http://{}:{}".format(keystone_ip.strip().decode('utf-8'),
+                                        port)
+        if not api_version or api_version == 2:
+            ep = base_ep + "/v2.0"
             auth = v2.Password(
                 username=username,
                 password=password,
@@ -516,7 +381,12 @@ class OpenStackAmuletUtils(AmuletUtils):
                 auth_url=ep
             )
             sess = keystone_session.Session(auth=auth)
+            client = keystone_client.Client(session=sess)
+            # This populates the client.service_catalog
+            client.auth_ref = auth.get_access(sess)
+            return client
         else:
+            ep = base_ep + "/v3"
             auth = v3.Password(
                 user_domain_name=user_domain_name,
                 username=username,
@@ -527,57 +397,10 @@ class OpenStackAmuletUtils(AmuletUtils):
                 auth_url=ep
             )
             sess = keystone_session.Session(auth=auth)
-        return (sess, auth)
-
-    def get_keystone_endpoint(self, keystone_ip, api_version=None,
-                              admin_port=False):
-        """Return keystone endpoint"""
-        port = 5000
-        if admin_port:
-            port = 35357
-        base_ep = "http://{}:{}".format(keystone_ip.strip().decode('utf-8'),
-