Skip to content

Troubleshooting

Getting ncm-issuer logs

In case of any problem, besides checking the status of created resources, you can also check the ncm-issuer pod logs:

kubectl -n ncm-issuer logs -f `kubectl get pods -A -l app=ncm-issuer -o jsonpath='{.items[0].metadata.name}'`

If you deployed troubleshooting sidecar as well, you can check the ncm-issuer pod logs this way:

kubectl -n ncm-issuer logs -c ncm-issuer -f `kubectl get pods -A -l app=ncm-issuer -o jsonpath='{.items[0].metadata.name}'`

Setting logging verbosity level

In the case of increasing logging verbosity level change the logging.logLevel in values.yaml to wanted value and update your deployment. To get all possible log messages, simply set the logging.logLevel to 3, you can also additionally change the logging.stacktraceLevel to error.

Enabling troubleshooting sidecar

There is also the possibility of using sidecar for debugging purposes - just change the value of sidecar.enabled to true in values.yaml and update your deployment.

From 1.2.1 onwards the sidecar runs under a non-root user (runAsUser: 998) with a narrow capability set: every Linux capability is dropped and only NET_RAW, NET_ADMIN and SYS_PTRACE are added back so tcpdump and strace can be used. The matching file capabilities are baked into the ncm-issuer-utils image (cap_net_raw,cap_net_admin+ep on /usr/sbin/tcpdump and cap_sys_ptrace+ep on /usr/bin/strace), so a kubectl exec into the sidecar lets you run those tools directly without sudo. allowPrivilegeEscalation is intentionally left at true on the sidecar so the kernel honours those file capabilities on execve(); setting it to false would set no_new_privs=1 and silently break both tools.

If your cluster enforces the Pod Security Standards "baseline" or "restricted" profile at the namespace level, the sidecar is rejected at admission. Both profiles reject the added NET_RAW, NET_ADMIN and SYS_PTRACE capabilities. "restricted" also rejects allowPrivilegeEscalation: true. In that case either keep sidecar.enabled: false in those namespaces or label the ncm-issuer namespace with the "privileged" profile:

kubectl label namespace ncm-issuer pod-security.kubernetes.io/enforce=privileged --overwrite

Issuer stuck in Ready=False with NCM API health check failed

From 1.2.1 onwards, the Issuer (and ClusterIssuer) Ready condition is gated on a synchronous health probe of the NCM REST API during reconciliation. If you see something like:

kubectl get ncmissuers -n example-ncm-ns
NAME                 AGE   READY   REASON   MESSAGE
example-ncm-issuer   5s    False   Error    NCM API health check failed: ...

it means the controller could reach the Kubernetes apiserver but the authenticated GET /v1/cas probe against the configured mainAPI (and backupAPI if set) did not return 2xx. Typical causes:

  • wrong mainAPI / backupAPI URL or unreachable NCM host
  • invalid or missing credentials in the auth secret
  • TLS misconfiguration (missing CA bundle when the NCM uses HTTPS)
  • egress/proxy not configured (see the next section)

Check the ncm-issuer pod logs for the exact probe failure. The issuer will be retried by the controller automatically. There is no manual action needed besides fixing the underlying connectivity or configuration.

Using an outbound HTTP(S) proxy

If your cluster has no direct internet/egress connectivity to the NCM instance, configure an HTTP(S) proxy.

If you install using Helm, set standard proxy environment variables via chart values:

proxy:
  httpsProxy: "http://proxy.example:3128"
  httpProxy: "http://proxy.example:3128"
  noProxy: "localhost,127.0.0.1,.cluster.local,10.96.0.0/12"

Under the hood this sets HTTP_PROXY / HTTPS_PROXY / NO_PROXY (and lowercase equivalents) which are honored by Go's HTTP client (including HTTPS via CONNECT).

The variables apply to every connection the controller makes, including its calls to the Kubernetes API server, which it reaches by IP address. Add the cluster's service network to noProxy (10.96.0.0/12 above is the kubeadm default), otherwise those calls go through the proxy as well.