GitLab Authentication

This section explains the authentication and user-management integration of Kubeflow with GitLab.

OIDC Authentication

OIDC

OIDC, is a standardized OAuth2 flow that allows a Client (Kubeflow) to get a user’s identity from a trusted Provider (GitLab).

Kubeflow

The Istio Gateway is an OIDC Client. Kubeflow’s applications do not contain any authentication code. Authentication is performed once at the Istio Gateway, when requests enter the Kubeflow cluster.

However, the Istio Gateway cannot act as an OIDC Client on its own. This is why we enhanced the Istio Gateway’s capabilities by creating an AuthService. which enables the Istio Gateway to act as an OIDC client. After the user is authenticated, the identity is placed in HTTP headers and the request is forwarded to Kubeflow’s applications. More details about this architecture can be found in this blog post.

GitLab

GitLab is an identity provider that exposes an OIDC interface.

Single Sign-On (SSO)

Single sign-on (SSO) enables users to authenticate with multiple applications by logging in only once. GitLab, as an Identity Provider, offers the following experience.

  1. User logs in to GitLab for the first time, typing their credentials.
  2. User navigates to Kubeflow, which is an OIDC Client to GitLab.
  3. Kubeflow initiates an OIDC flow to authenticate the user, redirecting them to GitLab to sign in.
  4. GitLab recognises the user is signed in, because it has stored a session cookie in the user’s browser.
  5. User is not asked to type their credentials. They may be asked to authorize the application to access their identity (configurable).
  6. After granting access to Kubeflow, the user is logged in and redirected to Kubeflow.

Single Logout (SLO)

Single logout (SLO) enables users to logout from all applications at once, without having to sign out from each one separately.

OIDC defines the following specs related to logout:

Because Kubeflow is a confidential client that doesn’t live in the user’s browser, the Back-Channel logout is what applies here. The gist of the spec is:

  • When a user logs out from the Provider (GitLab), the Provider sends requests to a logout endpoint at every Client (Kubeflow).
  • When a user logs out from a Client, they are redirected to a special page at the Provider where they are prompted to logout. Then, the Provider initiated procedure described above logs them out of all Clients.

However, GitLab doesn’t seem to either:

  • Implement the OpenID Connect Back-Channel Logout spec.
  • Provide an administrative endpoint to log out users from all Clients.

To work around those issues, we found the following solutions:

  • To logout a user from GitLab when they logout from Kubeflow, we use an HTTP POST request to GitLab’s /users/sign_out endpoint. This is effectively using CSRF so it is not ideal, however a recent GitLab issue removed CSRF protection of the logout endpoint for exactly this use.

  • To logout a user from Kubeflow (or other applications) when they logout from GitLab, we will force the AuthService to effectively use GitLab for session management and check the access/refresh token’s validity for every request.

    Important

    Currently, when a user logs out from GitLab, GitLab doesn’t revoke all the tokens issued to applications. This means that logging out of GitLab does not log the user out from Kubeflow. Since OIDC tokens are used for authentication and we want users to logout from all applications when they logout from GitLab, GitLab has to be extended to allow revoking all of the user’s OIDC tokens on logout.

    To log out globally, the user needs to navigate to their Profile’s Settings -> Applications, and in the Authorized Applications section revoke every application’s access token.

GitLab as OIDC provider for Kubeflow

In this section we will describe how to use GitLab as OIDC provider for Kubeflow.

Important

In case you have a self-managed GitLab installation you need to ensure that Kubeflow can access GitLab without problems. More specifically, Kubeflow’s Authservice (authentication proxy) will try to connect to GitLab using its user facing URL, e.g., https://gitlab.example.com (matching the external_url of GitLab’s configuration).

In the setup below we have GitLab and Kubeflow running in the same EKS cluster behind the same ALB. This means that there will be traffic coming out of the cluster and coming back in. The incoming traffic will be blocked since we have enabled a firewall via the alb.ingress.kubernetes.io/inbound-cidrs annotation on the ingress-nginx Ingress.

Pod traffic will come out either:

  • from the node’s public address, in case of public subnets

or

  • from the NAT gateway, in case of private subnets

To allow incoming traffic and also preserve the existing firewall configuration we will extend list of allowed CIDRs specified in the alb.ingress.kubernetes.io/inbound-cidrs annotation of the ingress-nginx Ingress with:

  • the addresses of all K8s nodes

    Important

    This is an ephemeral workaround since the public addresses of the nodes might change if the cluster scales up/down.

  • the public IP of the corresponding NAT Gateway

To update the firewall configuration you need to follow these steps below:

  1. First obtain the public addresses (if any) of the EKS cluster nodes as CIDR:

    $ export ADDRESSES=$(aws ec2 describe-instances --filters Name=tag-key,Values=kubernetes.io/cluster/${CLUSTERNAME?} | jq -r '.Reservations[].Instances[].PublicIpAddress' | sed -e 's|$|/32|' | xargs)
    
  2. Find the ID of the VPC that your EKS cluster lives in:

    $ export VPCID=$(aws eks describe-cluster --name ${CLUSTERNAME?} | jq -r '.cluster.resourcesVpcConfig.vpcId')
    
  3. Obtain the public IP of the NAT gateway (if any) inside the VPC as CIDR:

    $ export GWIP=$(aws ec2 describe-nat-gateways --filter Name=vpc-id,Values=$VPCID | jq -r '.NatGateways[].NatGatewayAddresses[].PublicIp' | sed -e 's|$|/32|' | xargs)
    
  4. Create a comma separated list with the CIDRs found above along with your trusted CIDR (see also Create Security Group and Create EKS Cluster sections):

    $ export CIDRS=$(echo ${CIDR?} ${ADDRESSES} ${GWIP} | sed 's/ /,/g')
    
  5. Finally annotate the ingress-nginx Ingress with the comma separated list of allowed CIDRs:

    $ kubectl annotate ingress -n ingress-nginx ingress-nginx \
    >     --overwrite alb.ingress.kubernetes.io/inbound-cidrs=${CIDRS?}
    

GitLab + Authservice

Authservice is a OIDC client. By default it is configured to talk to dex. Here we will bypass Dex completely and setup authservice to talk to GitLab directly.

First create an application, i.e., OIDC client credentials, in your GitLab. Use the admin account, go to Admin area -> Applications -> New Application, with the following settings:

Application
Callback URL https://demo.example.com/authservice/oidc/callback
Trusted N
Confidential Y
Scopes
  • read_user (Read the authenticated user’s personal information)
  • openid (Authenticate using OpenID Connect)
  • profile (Allows read-only access to the user’s personal information using OpenID Connect)
  • email (Allows read-only access to the user’s primary email address using OpenID Connect)

Update kubeflow/manifests/istio/oidc-authservice/overlays/deploy/secret_params.env to include the generated credentials:

CLIENT_ID=<Application ID>
CLIENT_SECRET=<Secret>

Edit kubeflow/manifests/istio/oidc-authservice/overlays/deploy/params.env:

USERID_HEADER=kubeflow-userid
USERID_CLAIM=email
OIDC_PROVIDER=https://gitlab.example.com
OIDC_SCOPES=read_user,profile,email,openid
# Use the default one instead of the one inherited from the ekf overlay.
OIDC_AUTH_URL=
AUTHSERVICE_URL_PREFIX=https://demo.example.com/authservice/
CLIENT_NAME=Kubeflow
TEMPLATE_PATH=web/templates/gitlab/auto_logout
AFTER_LOGIN_URL=/reception
STRICT_SESSION_VALIDATION=true

Commit and apply changes:

$ git commit -am "Use GitLab OIDC provider instead of Dex"
$ rok-deploy --apply install/kubeflow

For changes to take effect we have to restart the pods manually:

$ kubectl delete pods -n istio-system -l app=authservice

Advanced configuration

For more advanced configuration of the AuthService, please see the relevant section. For example, how to: