feat: treat AWS permission errors as hard errors to fail readiness probes

[u-kai]

What would you like to be added:

AWS provider should treat permission/authorization errors as hard errors instead of soft errors, causing external-dns to terminate and readiness probes to fail.
Currently, AWS permission errors (AccessDenied, UnauthorizedOperation) are wrapped as provider.SoftError, so the process continues running and /healthz stays healthy.

Why is this needed:
Permission errors are non-recoverable and require manual intervention.
In real-world setups (IRSA/Pod Identity, EC2 instance profiles), identity association changes often require a fresh pod/instance to pick up the new mapping/credentials. Failing fast accelerates that refresh path and surfaces the error to operators and monitoring.

Current behavior creates silent failures - DNS records aren't updated but the service appears healthy.
By failing fast on permission errors:

• Kubernetes restarts the pod automatically
• Operators get immediate feedback about misconfigurations
• Monitoring systems properly detect issues This aligns with "fail fast" principles for non-recoverable errors and improves operational visibility.

If the proposed direction makes sense, I’d be happy to open a PR.

[mloiseleur]

It makes sense to me 👍 . This is clearly an error case where a fail fast error is a better behavior.

[ivankatliarchuk]

Not aware of any single kubetnetes or kubernetes-sigs that behave in proposed way

[ivankatliarchuk]

Main reason why hard error could be considered a correct solution

• Delayed troubleshooting → since pods don’t restart, operators might miss errors unless they check logs or metrics.
• Retry storms → missing permissions cause infinite retries, which can flood logs and API rate limits.

First of all, I do not think there is a correct resolution. There’s no universal answer between “fail hard” vs “retry and log”. It’s a tradeoff between visibility (fail fast) and resilience (keep going where possible). And configuration related issues are pretty much on the edge.

Pod restarts is not the only way to identify that something is not right. Operators should rely on observability, monitoring and alerting but not on pod restarts as is. More important, with pod restart you may miss log or metric that's explaining WHY exactly the restart happens.

Let's have a look at aws-vpc-cni or autoscaler.

1. Pods don’t restart on logs/errors
   Kubernetes restarts a pod only if the container process exits with a non-zero exit code (CrashLoopBackOff).
   If the process is still running (even if it logs IAM/RBAC errors), Kubernetes assumes the container is "healthy" from a runtime perspective.
   Both AWS VPC CNI and Cluster Autoscaler catch permission errors internally and log them instead of crashing.

2. Long-running controllers
   These components are controllers/daemons. Their design is to keep running even when something fails, and retry in a loop. Autoscaler and VPC CNI are meant to run forever. They handle transient errors (like API throttling, temporary network failures, partial IAM) by retrying, not by crashing. This aligns with Kubernetes’ controller pattern.

Example: Cluster Autoscaler logs AccessDenied but retries after backoff.
AWS VPC CNI might log failed to assign IP but continues monitoring.
Restarting the pod wouldn’t help if IAM permissions are still missing — it would just crash repeatedly.

3. Error handling philosophy

Fail fast vs retry:

• Apps like controllers (autoscaler, CNI) are expected to operate continuously, so they retry instead of terminating.
• Restarting would create noise (CrashLoopBackOff) without solving the issue.
• The operator’s responsibility is to detect these restarts,logs,metrics or alerts and fix the IAM policy, not rely on pod restarts.It’s the operator’s job to monitor for AccessDenied, UnauthorizedOperation, or retry storms via alerts/metrics.
• As well consider cases when some permissions missing

What happens with partial IAM permissions?

The controller uses the permissions it has, and fails only on the operations that require missing permissions.

For example:

Cluster Autoscaler might scale up nodes successfully (if it has EC2 permissions) but fail to tag instances (if ec2:CreateTags is missing).
AWS VPC CNI might attach secondary ENIs but fail to set certain attributes (e.g., ec2:AssignPrivateIpAddresses).

In both cases:

• The pod does not crash.
• It logs AccessDenied or UnauthorizedOperation errors.
• It retries those failing API calls based N times based on configuration with back-off period

For example:

• If autoscaler could still remove nodes but not tag them, restarting would lose its entire state and not fix IAM.
• If CNI could still assign some IPs, it helps keep some pods running instead of killing networking completely.

So in case of vpc-cni or autoscaler, pods do not crash if AWS IAM permissions missing. This is intentional: restart wouldn’t help, and partial functionality is better than none.

Delayed troubleshooting: True — relying only on logs can lead to missed signals. But that’s not a fault of the controller, it’s an observability gap.
Retry storms: This is real — infinite retries can spam CloudTrail/CloudWatch. The mitigation is usually backoff with jitter + error metrics + alerts, not crashing the pod.

[ivankatliarchuk]

Example: Restricted Record Types with ExternalDNS

Imagine a Kubernetes platform where the platform alongside with security team owns and operates the ExternalDNS deployment.
As part of their governance, they decide that only certain DNS record types should ever be created — specifically:

• A records (IPv4 addresses)
• CNAME records (aliases)
• TXT records (used for ownership and validation)

To enforce this, they configure IAM policies so that ExternalDNS does not have permissions to create or modify any other record types.

---

Scenario

Now, suppose a malicious actor or simply a well-meaning engineer applies a Kubernetes manifest that requests an unsupported record type, such as an SRV record.

Current behavior (best-effort)

ExternalDNS attempts to create the record, AWS IAM denies the request, and ExternalDNS logs an error like:

• ExternalDNS skips that change, but continues reconciling the rest of the records.
• ✅ The allowed A, CNAME, and TXT records are still updated.
• ✅ DNS resolution for the cluster remains healthy.

New “crash on error” behavior

If ExternalDNS were changed to crash whenever it encounters a permission error, the same denied SRV request would cause the entire ExternalDNS process to stop.

• 🚨 ExternalDNS would no longer manage any records — even the valid A, CNAME, and TXT ones.
• 🚨 This would effectively break DNS automation across the whole cluster due to a single unauthorized record.

---

Takeaway

This scenario demonstrates the risk of fail-fast error handling in long-running controllers like ExternalDNS:

• A single misconfiguration (or malicious resource) could disable critical DNS functionality for the entire system.
• Instead of graceful degradation, the controller becomes a single point of failure.

But again, there are pros/cons

[u-kai]

@ivankatliarchuk

Thanks — I really appreciate the detailed feedback and it helped me rethink this. After considering it again, I agree with your points: I’m comfortable making hard errors for failures that make all subsequent operations impossible (i.e., there is no possibility of partial success). Concretely, this would cover cases such as inability to obtain credentials or a List API failure inside Records — if Records cannot return the initial list, meaningful reconciliation cannot proceed and failing loudly makes sense.

For APIs executed during ApplyChanges (per-resource CREATE/UPDATE/DELETE), I prefer to keep the current best-effort behavior (log, metric, retry/backoff) so that a single bad resource does not stop DNS automation.

What do you think? If you agree, I can prepare a small incremental PR to implement fatal-on-Records-list-failure plus the corresponding metric/event and documentation.