Update Auth GEP with Implementable details

[youngnick]

What type of PR is this?
/kind gep

What this PR does / why we need it:
This adds the design rationale and API design for phase 1 of the Auth GEP, adding a Filter to HTTPRoute.

Which issue(s) this PR fixes:

Updates #1494

Does this PR introduce a user-facing change?:

NONE

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: youngnick

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• geps/OWNERS [youngnick]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[howardjohn]

👍

[howardjohn]

Legitimate question, not a suggestion: do we need both protocols? I would think there would be two reasons to support both:

1. Implementations can only support one, in which case I would expect there to maybe be a feature per type or something?
2. Ext_auth servers only support one or the other

If neither of these I would think we would just support gRPC which is more powerful(?).

Even if one of these is true, I still wonder if we want both. Essentially we already have a fairly tricky GEP, introducing an external authorization API, and are putting two completely distinct protocols into it. If 2 protocols, why not 3 or 4 (this is rhetorical, lets please not do that!! 🙂 )

[youngnick]

The main reason for me is that the gRPC mode is only supported on Envoy, while at least a couple of other dataplanes (Traefik and HAProxy) can support the HTTP semantics. Doing this in this way means that dataplanes that don't support gRPC can make do with HTTP. It also allows for great use of extension servers, since they don't only need to support ext_auth. These two specific protocols are also included in Envoy's ext_auth filter anyway, so Envoy-based implementations can do either.

Probably does make sense to include a feature for each protocol, that's a good idea, thanks.

[howardjohn]

After some discussion with folks I have heard that authz-server implementations also often use HTTP since its simpler. So seems both (1) and (2) are true.

I did also notice there is seemingly no spec at all for HTTP authz server... maybe we need to step up to fill in that gap (unless I just missed it, in which case we should link to it)

[kflynn]

It absolutely needs a spec. Working on it. 😐

[htuch]

I think we should look to moving ext_authz specifications to a proxy neutral location. One option that has been suggested is https://github.com/cncf/xds (yes, ext_authz isn't xDS but it is at proto-level in a related family and CNCF is as good a home as any). This has also come up for ext_proc.

I had the same reaction as @howardjohn at first. I think I'm leaning towards both HTTP / gRPC as discussed here, but I'd point out that one of the goals of standardization is to create a portable ecosystem. E.g. I can swap out Envoy for Traefik and things should continue to work. This won't happen if the ext_authz server is gRPC-based and Traefik only supports HTTP. I wonder if HTTP should be core and gRPC extended, since HTTP seems the lowest common denominator. This would unfortunately have the side effect of encouraging growth in the less optimal HTTP ext_authz protocol.

[kflynn]

@htuch Completely agreed both that the spec needs to move to a neutral home, and that HTTP should be core with gRPC extended. I'm not completely sold on cncf/xds, but I'm not completely opposed either.

[youngnick]

As it stands, both HTTP and gRPC are extended, with features to mark their support. We can revisit if we make either of them Core at a later date, as part of graduation discussion. I'd rather not try and make this decision now (we're making enough already).

[cccs-abwolfe]

Speaking as a Gateway API end user (hopefully it's OK to provide feedback here), HTTP based external authorization is important to us because it's what the software we use already supports.

For example, it's possible to share an oauth2-proxy instance and App Registration between a bunch of applications/serverless functions that have a common user base using ingress-nginx external auth annotations. The alternative is each one getting its own oauth2-proxy instance running in proxy mode. Not an absolute dealbreaker but we would need to go from a few oauth2-proxy pods currently to several dozen + a way to spin them up automatically.

Of the major external SSO providers / authenticating proxies I know of (oauth2-proxy, authentik, authelia. keycloak), I don't think any support GRPC external authentication? On the ingress side ingress-nginx and Traefik Forward Auth only support HTTP. For custom auth use cases I think developers are generally a lot more familiar with HTTP than GRPC and find it easier to work with.

[youngnick]

Thanks for that feedback, @cccs-abwolfe, that's really useful. I think I'll leave both as options in here for now.

[howardjohn]

Not suggesting any changes here, just a comment: this has interesting implications as we consider filter vs policy. Any backend policies, not just TLS, could clear apply here. But anything that is a filter could not, as there is no way to insert a filter.

So 1 point in favor of policies I suppose.

[LiorLieberman]

focusing on TLS - I think thats already becoming slightly complex/messy that you are unable to define that your ext auth server is requiring TLS in the same config where you define it.

I'd expect an API that lets me defines the extension server, whether I use TLS to connect, faliureMode(open/close) in the same config. BackendTLSPolicy can potentially override that wrt to TLS?

[howardjohn]

Istio takes a slightly different approach to the comment and the PR:

• Users define extension servers at a top level. This includes the address, common config, etc
• Policies reference the extension, and that is all they need to know
• Traffic level controls for how to reach it (like TLS) are handled by standard policies modifying Service/backends/etc.

Its good and bad. Its nice for a user to just be able to say "I want keycloak" and not need to configure the details of keycloak over and over. But if you just have 1 usage, then that abstract is overhead.

That being said, I do think that re-creating each policy on each extension is not likely a great pattern long term as we will eventually have many more backend policies (first and third party)

[youngnick]

Yes, that was my thought as well. For example, Envoy Gateway has a standard stanza that is used to control connections to any backend, setting connection timeouts, pooling behaviors, load balancing behaviors, etc, that is included in each SecurityPolicy. I don't want this Filter to end up including lots of backend settings that will also end up in Backend controls in other ways.

[howardjohn]

Likely the validation would come in the real Go change but I would assume we need to validate you don't set grpc+http and that it matches the protocol?

[youngnick]

Sure. It's only for UX, not functionality, because using a union discriminator on the enum does the same thing - if you put HTTP config in the struct when you choose GRPC, then it has no effect.

[howardjohn]

While I agree ordering is important for auth, I am not certain making it a filter meaningfully solves this. It seems like you pretty much always want auth before any of the current filters, and its really only the ordering amongst other 3rd party extensions that is important, which may likely not even be filters. The one thing that makes sense among the core filters, IMO, is ordering wrt CORS.

[youngnick]

I thought this too, until people insisted I build it otherwise on Contour. People sometimes want to transform headers in some way before auth, for example, usually to fix broken things.

[youngnick]

Also, a filter gives us a way to define this semantic, where Policy does not - without adding even more complexity into Policy Attachment like ordering or something.

[LiorLieberman]

focusing on TLS - I think thats already becoming slightly complex/messy that you are unable to define that your ext auth server is requiring TLS in the same config where you define it.

I'd expect an API that lets me defines the extension server, whether I use TLS to connect, faliureMode(open/close) in the same config. BackendTLSPolicy can potentially override that wrt to TLS?

[htuch]

There are a lot of configuration options in ext_authz.proto. In the proposal, a few have been selected, e.g. alowed request headers (but not disallowed?), body settings, etc. but not other features (e.g. clearing route cache to influence service selection - very Envoy specific but also a very useful capability). What rubric is used to determine which capabilities to include and exclude?

[youngnick]

I've tried to keep it to "things I've definitely seen people ask for that also exist across multiple implementations". Part of the graduation requirement is for multiple implementations, preferably with multiple dataplanes, to implement this, so it's important that we try to keep it both small to start with and as portable as possible.

We can definitely add more stuff when people need it, this is me trying to pick a minimum viable feature set.

Edit: Also important to remember this is an experimental API, so we can (and should) add more to it for common use cases, as long as they fit the above requirements.

[kflynn]

After @howardjohn's entirely appropriate gentle mockery for the lack of proper specification for this protocol, I've abused the GEP machinery a bit to get the doc off my laptop into a state and place where at least other folks can read it: #3892 (or https://deploy-preview-3892--kubernetes-sigs-gateway-api.netlify.app/geps/gep-9999/ if you just want the formatted version).

(As stated in that document, I don't think that Gateway API is actually the correct place for this -- it was just a low-friction way to let others see it.)

[htuch]

How do we get consistent interop with features like frontend mTLS? Ideally we can present the frontend mTLS extracted identity to the ext_authz server. The ext_authz protocol has a notion of attribute context, will this be required as part of compliance across proxy implementations? I thought I had opened a thread on this but don't see it.

[youngnick]

I agree this is a useful use case, and we should look at adding the ability to say "populate X-Forwarded-Client-Cert if a client cert was used" - but I think we can come back and add that later. I agree that users will definitely want it, but I would like to get super basic functionality in first, and add things as users ask.

[htuch]

It will be inevitably needed for a solution combining these features but yeah, it's fine if you want to downscope for now, SG.

[youngnick]

Okay, reviewing this, I think I've answered all the outstanding things, it seems that this is ready to move to Implementable, so I'll remove the hold and wait for an LGTM from someone. Next steps will be for me to actually add the APIs to the Go types and then move to Experimental. But I can do that after 22 July.

/unhold

[shaneutt]

My thoughts are mostly: what are the pros/cons of ONLY doing phase 1 for now, and allowing the policy approach be its own separate enhancement later?

[shaneutt]

@youngnick I'm curious whether you considered doing just the filter for the first iteration (and not including the policy components)? Would that provide enough value to move forward, while also keeping the scope really tight so we can hopefully get some velocity?

[youngnick]

That's the plan, yes. I don't think I would have been able to get this moving without a plan for the Policy though.

[shaneutt]

I suppose what I meant was, should we pull OUT the policy for now. However I didn't mean for this to be a blocking comment and it's going to Experimental so I'm OK with seeing where this goes.

[youngnick]

To be clear, the first round of API changes to be included in this release will NOT include the Policy. That will happen in release 1.5 or later.

[robscott]

Thanks @youngnick! Will leave hold to give you a chance to reply to @shaneutt's comment above.

/lgtm
/hold

[youngnick]

/unhold