Add explicit exception logging and configuration updates.

[rohit-joy]

• Reduce statement timeout.
• Explicit exception logging when record is not found.
• Reduce puma threads.

[arielr-lt]

@rohit-joy that looks fine to me, just some observations: the depth cap and Puma thread reduction are solid, low risk levers and I'd merge them for immediate relief (prod main-app is sitting at ~6.6 Gi of its 8 Gi limit right now, so both directly attack the OOM we are looking often).

One thing I'd add on the infrastructure side, specifically for the "single IP exhausting us" case: reducing Puma threads does slow abusers, but only as a side effect. When threads saturate, the 503s hit everyone, including legitimate traffic. We can target the abusive IP directly instead, without touching app code.

Our prod ingress is ingress-nginx with externalTrafficPolicy: Local, so the real client IP is already preserved at the ingress. That means we can turn on per IP rate limiting with a few annotations on the app ingress: (for example 👇 )

metadata:
  annotations:
    nginx.ingress.kubernetes.io/limit-rpm: "300"            # per client IP
    nginx.ingress.kubernetes.io/limit-connections: "20"     # concurrent per IP
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3" # short bursts ok
    # optional: exempt trusted/internal consumers
    nginx.ingress.kubernetes.io/limit-whitelist: "10.19.0.0/16"

nginx uses a leaky bucket keyed on the client IP and returns 503 when the IP exceeds the limit. That's exactly the "slow down, retry later" behavior we want, but scoped to the offending IP rather than degrading the whole app. Since today's spike was ~90% one IP and likely a legit bulk pull, throttling (not banning) is the right call. They just get paced, and we can whitelist or raise the limit for known partners.

If we later want to shed volumetric abuse before it reaches the cluster (and get managed bot/rate protections), the next step up is CloudFront plus AWS WAF with a rate based rule in front of the NLB. It's stronger, but a bigger change (distribution, cert, origin config). For now, the ingress-nginx annotations get us most of the protection with a one file change and no new infra, my two cents...

[rohit-joy]

Thank you for the discussion and suggestions @arielr-lt!

I reduced it from 16 to 8 per pod in the PR. This is still good since there are multiple pods running at the same time.

Limit this to 5: nginx.ingress.kubernetes.io/limit-connections: "5"
This one is good: nginx.ingress.kubernetes.io/limit-burst-multiplier: "3" # short bursts ok
RPS (not RPM): nginx.ingress.kubernetes.io/limit-rps: "8" # per client IP

I'll leave the whitelisting one to you.

You can go ahead and send updates to this PR for any changes you are making.

[arielr-lt]

Thanks Rohit. The rate limit annotations are on the PR (rps 8, connections 5, burst 3), with the whitelist left out for now so we protect against everyone first. We will dig into the app logs and monitoring on our side Monday to identify the top source IPs, classify whether it is a legit partner or a scraper, and then decide on any targeted exemptions from there. Have a good weekend.

[rohit-joy]

@arielr-lt Since it’s my PR, I cannot approve it. You or @chuang-CE need to review.

I had already looked at the IPs. It’s not that bad. And they look like legitimate requests based on current guidance. So no need to whitelist IPs at this time.

[arielr-lt]

LGTM!