Caching domain lookups: keys, TTLs and when to refresh

Cache domain lookups on the registrable domain (eTLD+1), not on the raw input, and give each kind of data its own TTL: list-based categories for days, DNS facts for about a week, company profiles for weeks with an explicit refresh path. Keep anything derived from the email's local part out of the domain cache. Most of the bugs we see in lookup caches come from violating that last rule.

This applies to any domain or enrichment API you call, including pingutil. The examples use pingutil's response shape because it makes the split between domain-level and address-level data visible.

Decide what the cache entry actually describes

A lookup response mixes facts about different things. Before choosing a key, sort the fields by what they depend on.

The category TTL follows the source lists. The disposable list pingutil uses is the community disposable-email-domains repo, re-synced weekly, so caching a category for longer than a week means you can miss a newly listed domain for two cycles. Daily is a fine default.

The verdict row is the trap. Two addresses on the same domain can get different answers:

• jane@northwind-example.com might be clean.
• ccjxx90sze@northwind-example.com gets verdict.is_spam: true with reason: "randomized_local_part".

If you store the whole response under northwind-example.com, whichever address arrived first decides the verdict for everyone after it. Worse, pingutil short-circuits a randomized address before enrichment, so that response has company: null. Cache it by domain and you have also cached "no company data" for a real company. Store the domain-level fields under the domain and let the address-level fields live per request.

Build the key properly

A good key has three parts: a namespace with a version, the kind of data, and the normalized domain.

lookup:v2:company:stripe.com
lookup:v2:category:stripe.com
lookup:v2:dns:stripe.com

The version lets you change the stored shape without a migration: bump it and old entries just stop being read. The data kind lets each entry have its own TTL. The domain must be the registrable domain, lowercased and punycoded, or Mail.Stripe.com, stripe.com. and www.stripe.com become three cache misses for one company. Computing that correctly needs the Public Suffix List; eTLD+1 explained covers why splitting on dots fails.

If request options change the response shape, they belong in the key or in separate kinds. A pingutil call with skip_enrichment: true returns no company block for domains it hasn't enriched before. If you cache that under the same key as a full lookup, later readers see company: null and assume the domain has no profile.

Know what the upstream already caches

Your cache sits in front of someone else's. Know theirs so you don't fight it.

pingutil stores results per registrable domain in Cloudflare D1. Company rows don't expire on a timer: once a domain is enriched, later lookups are served from the cache (cached: true) until someone sends refresh: true. DNS signals are cached for 7 days. The category lists are synced weekly. Every response tells you how old the data is:

• cached: whether this response came from the cache
• last_refreshed_at: when the company profile was last produced by a model
• fetched_at: when this response was assembled

So why cache on your side at all? Latency and budget. A cache hit on pingutil is fast, but it is still a network round trip, and the per-key limit is 60 requests a minute. A signup spike from one company, or a backfill job, can burn through that quickly. A local cache in front makes repeated domains free.

Use last_refreshed_at rather than your own insert time when deciding whether a profile is old. If you cached a response today that was enriched eight months ago, it is eight months old.

Stale-while-revalidate and stale-if-error

Two HTTP cache extensions from RFC 5861 are worth borrowing for application caches even when HTTP isn't involved:

• stale-while-revalidate: when an entry has expired, return it immediately and refresh it in the background. The user never waits on the slow path for data you already have.
• stale-if-error: if the refresh fails, keep serving the old entry for a while instead of returning an error.

pingutil does the second one server-side. If enrichment fails and an older company row exists, the API returns it with meta.served_stale: true and meta.enrichment_error set, and still responds 200. If no older row exists, you get a category-only response with company: null and enrichment_error. It never writes a half-filled profile. Your cache should respect that: don't overwrite a good cached profile with a response that has enrichment_error and no company.

Coalesce concurrent misses

When a company rolls out your product, twenty people from the same domain can sign up in the same minute. Without coalescing, all twenty miss the cache and make twenty upstream calls for the same domain. Keep a map of in-flight requests and let late arrivals await the first one. Go calls this pattern singleflight.

A working cache wrapper

This is a compact TypeScript version with per-kind TTLs, stale-while-revalidate, negative caching and coalescing. The Store interface maps onto Redis, Cloudflare KV, or a Map in tests.

import { getDomain } from 'tldts';

interface Store {
  get(key: string): Promise<string | null>;
  set(key: string, value: string, ttlSeconds: number): Promise<void>;
}

interface Entry<T> { value: T | null; storedAt: number; error?: string }

const FRESH_SECONDS = { company: 60 * 24 * 3600, category: 24 * 3600 };
const KEEP_STALE_SECONDS = 180 * 24 * 3600; // how long an expired entry is still usable
const NEGATIVE_SECONDS = 5 * 60;

const inflight = new Map<string, Promise<unknown>>();

export function registrableDomain(input: string): string | null {
  const host = input.includes('@') ? input.slice(input.lastIndexOf('@') + 1) : input;
  return getDomain(host.trim().toLowerCase().replace(/\.$/, '')); // null for IPs, bare suffixes
}

async function lookup(domain: string) {
  const res = await fetch('https://pingutil.com/v1/lookup', {
    method: 'POST',
    headers: {
      authorization: `Bearer ${process.env.PINGUTIL_KEY}`,
      'content-type': 'application/json',
    },
    // Bare domain: we want domain-level facts only. email_signals come back null.
    body: JSON.stringify({ input: domain }),
  });
  if (!res.ok) throw new Error(`lookup_failed_${res.status}`);
  return res.json();
}

export async function getCompany(store: Store, input: string) {
  const domain = registrableDomain(input);
  if (!domain) return null;
  const key = `lookup:v2:company:${domain}`;

  const raw = await store.get(key);
  const entry: Entry<any> | null = raw ? JSON.parse(raw) : null;
  const ageS = entry ? (Date.now() - entry.storedAt) / 1000 : Infinity;

  if (entry?.error && ageS < NEGATIVE_SECONDS) return null;          // recent failure
  if (entry && !entry.error && ageS < FRESH_SECONDS.company) return entry.value;

  const refresh = coalesce(key, () => refill(store, key, domain, entry));
  // Stale but usable: return now, refresh in the background.
  if (entry && !entry.error) { refresh.catch(() => {}); return entry.value; }
  return refresh;
}

function coalesce<T>(key: string, fn: () => Promise<T>): Promise<T> {
  const existing = inflight.get(key) as Promise<T> | undefined;
  if (existing) return existing;
  const p = fn().finally(() => inflight.delete(key));
  inflight.set(key, p);
  return p;
}

async function refill(store: Store, key: string, domain: string, previous: Entry<any> | null) {
  try {
    const data = await lookup(domain);
    if (data.company) {
      await store.set(key, JSON.stringify({ value: data.company, storedAt: Date.now() }), KEEP_STALE_SECONDS);
      return data.company;
    }
    if (data.meta?.enrichment_error) throw new Error(data.meta.enrichment_error);
    // Not a company domain (free, disposable, edu…): cache the absence like a value.
    await store.set(key, JSON.stringify({ value: null, storedAt: Date.now() }), FRESH_SECONDS.category);
    return null;
  } catch (e) {
    // Keep a good previous entry; otherwise remember the failure briefly.
    if (!previous || previous.error) {
      await store.set(key, JSON.stringify({ value: null, storedAt: Date.now(), error: String(e) }), NEGATIVE_SECONDS);
    }
    return previous && !previous.error ? previous.value : null;
  }
}

The inflight map only coalesces within one process. Across many instances you'd need a short lock in the shared store, which is usually not worth it until you can see duplicate upstream calls in your logs. The storedAt here is your insert time; if you need true data age, store the response's last_refreshed_at alongside it.

When to force a refresh

TTL expiry is the floor. Some events justify fetching a fresh profile right now, which on pingutil means refresh: true:

1. A record becomes valuable. A lead turns into an opportunity, or a domain is about to drive routing by company size. Paying for one fresh enrichment is cheap compared to routing a deal on last year's headcount.
2. The profile is weak. meta.low_confidence is set when the model's own confidence was below 0.5. Those are the rows most likely to improve with another attempt, and the ones most likely to still be wrong. How accurate LLM enrichment is goes through what to do with them.
3. Someone reports it. A sales rep says the HQ is wrong. Refresh, then fix by hand if the new answer is still wrong.

Don't refresh in a loop. Each refresh: true on pingutil triggers a new model call, and a second refresh for the same domain within 60 seconds just returns the cached row. Debounce on your side too: one refresh per domain per day is plenty, even for records people care about.

A short checklist

• Key on the registrable domain, versioned, with the data kind in the key.
• Never store verdict or email_signals under a domain key.
• Give categories, DNS facts and company profiles separate TTLs.
• Cache failures for minutes, not hours, and never let a failure overwrite a good entry.
• Serve stale while revalidating; coalesce concurrent misses.
• Judge age by last_refreshed_at, and refresh on business events rather than only on a timer.

If you want to see what the upstream returns before you design your cache, run a few domains through the live demo or read the full response schema in the API reference.