Competitor research from open sources: what shows from the outside
My own tool, not a client project. Starting from a single site address, a working picture comes together in 30-60 minutes: what the product is and on what terms it's sold, what the site is built on, which external services are wired into it, and how all of that changed over the past few years. Free public sources only — not a single paid subscription. Plus a separate rulebook that exists because the methodology caught a mistake of my own.
The site is open to everyone and yet tells you nothing about itself
You open a competitor's site — or the site of a company you're about to work with. It's public, and on its own it answers none of the practical questions. How old the property is and how long it has looked like this: not on the page. What the site is built on and which external services are wired into it — analytics, chat, payments, capture forms: nowhere stated. What sat on the same domain a year ago — which offer, which price, what got removed and what got added: even less so.
The commercial answer is paid subscriptions: technology lookups, domain and certificate history. That's hundreds of dollars a month, and it still covers only part of the question. A service hands you a list of technologies, but it won't tell you which of them actually runs on the page today, nor what changed over the last year.
The second place this work breaks isn't data collection — it's the discipline of the conclusion. Gathering signals is easy. Over-reading them is just as easy: spot a shared third-party service on two different sites and declare them one product. That conclusion is worse than no conclusion — it looks well-founded, and people make decisions on it.
Twelve steps, reproducible collection, and discipline about conclusions
The methodology has two halves, and the second one matters more. First: collection — reproducible, scripted, leaving artifacts behind. Second: rules of interpretation — which signal actually proves something, and which one shows up across hundreds of unrelated sites and proves nothing at all.
WHOIS: registrar, domain creation date, name servers — how old the property really is
DNS records A / NS / SOA / MX / TXT, certificate SANs, crt.sh history
HEAD request: server headers, X-Powered-By, cookie naming patterns
What they sell and on what terms: offer, price, guarantees, capture forms
Public scans, passive DNS, the web archive: how the site looked before
Collection collapsed into one command
The
01-recon.sh
script takes a domain and, in a single pass, runs WHOIS, DNS, certificate parsing, historical
enumeration from certificate transparency, a HEAD request, public scans, passive DNS and the web
archive. Each step lands as its own file in a working directory — an artifact you can return to,
attach to a conclusion, and diff against a re-run of the same domain a month later. An analysis
that exists only in the analyst's terminal isn't an analysis.
The request to the page itself is a separate step — on purpose
The first script never touches the site: WHOIS, DNS, certificates and archives all answer from the
side. The request to the page itself is the only step that leaves a trace in someone else's
analytics, so it lives in its own file
02-live-click.sh
and gets run deliberately rather than by default. What it captures is exactly what the analysis
was for:
- the offer and the price on the date of the analysis — what your own proposition is actually compared against
- capture forms: how many fields, what's required, what they promise after submission
- the external services the page loads: analytics, chat, payments, CDN
Set-Cookieand response headers — they identify the engine more precisely than page content does
The page is saved whole, prices and all, as of the date of the analysis. That matters more than the observation itself: "they were cheaper in spring" is an argument, two dated files are a fact.
The stack is identified by fingerprints, not by what the footer says
A site never states what it's built on. But every platform leaves a fingerprint, and the
fingerprint is more durable than anything printed on the page. The
tracker-signatures.md
reference is a "signal → platform" table that grows with every analysis:
- a service response header unique to one platform — the strongest signal, identification is instant
- cookie naming shape and structure — it reveals both the site engine and the analytics wired in
- asset paths and build file names — they give away the framework and how the frontend is bundled
- the set of external domains the page loads: CDN, analytics, chat, payments
- hostnames in the cloud provider's DNS — they show where and on what everything is hosted
The bundled frontend says more about the product than the marketing page
A dedicated step reads the collected JS and CSS not for logic, but for how the product is built: hard-coded service endpoints show which integrations are wired in; library versions show how fresh the frontend is and how often it gets updated; internal entity names show how their data model is shaped. It's a cheap step, almost always skipped, and it explains more than the rest of the collection combined.
Rules of inference live in their own document, not in a closing paragraph
Behind a single site sits more than one party: hosting, a CDN, a site builder or CMS, the
third-party services wired in. Each of them serves dozens of unrelated clients. So
attribution-rules.md
splits signals into two lists: those that actually prove something, and those that show up
everywhere. The conclusion is drawn from a unique signature, not from shared infrastructure.
Zero paid sources is a constraint, not a saving
The whole collection stands on what's available to anyone: registry whois servers, public DNS resolvers, certificate transparency, public page scans, passive DNS and the web archive. The constraint is methodologically useful: it forces you to rely on signals anyone can independently re-verify, rather than on the verdict of a closed service that can be neither challenged nor reproduced.
Tools that are already on every machine
The whole collection is a plain shell script — no dependencies, no install step
Registrar, domain creation date, name servers — how old the property really is
The DNS map: hosting, mail, third-party service verifications
Certificate SANs — every adjacent domain issued under one certificate
Certificate transparency: historical subdomain and related-name enumeration
Public scans: which assets and external services the page pulls in
Free equivalents of paid DNS-history services
How the offer, prices and site structure looked one and three years ago
JSON response parsing inline in the pipe, no packages to install
Signature and inference references are versioned alongside the scripts
A methodology that catches its own errors
from a site address to a finished profile: offer, stack, wired-in services, change history
no paid subscription for technology lookups, DNS history or certificates
three independent analyses; each one added new signatures to the references
The most valuable result was catching my own mistake
On the third analysis the conclusion was wrong at first. Two different sites converged on the same third-party service, and that looked like proof they were one and the same product — a vivid, reproducible and entirely false signal. The error surfaced one step later: the response headers carried a platform signature, and on the second site it was a different one. All they shared was a third-party service used by dozens of unrelated companies.
The rule that came out of it
Shared tools do not prove a shared team.
That sentence went into
attribution-rules.md
together with an explicit list of signals that prove nothing while looking convincing:
- anycast addresses of a popular CDN — a shared pool across every customer on the tier
- the most popular registrar and its default privacy provider — that's popularity, not a relationship
- a shared third-party widget — chat, form, analytics: dozens of unrelated companies install them
- a boilerplate landing template and the standard cookie set of a popular framework
The resulting algorithm is short: find a unique signature, identify the platform from it, confirm with a second independent signal — and only then claim anything. Overlap in shared infrastructure enters the write-up as supporting evidence, never as the primary claim.
What's left after three analyses
A repository with the playbook, two reusable scripts and two reference documents that grow with each new analysis. Plus a delivery checklist — the fields that must be filled before a conclusion counts as finished: what the product is, what the site is built on, which services are wired in, and exactly what backs each point. The checklist exists for the same reason the rules do: so the analyst's confidence never substitutes for evidence.
The overall takeaway, details aside: a methodology that catches its own errors is worth more than one that is always certain. The document now answers not only "how to find out", but also "when you don't know yet".
Where else the same methodology applies
This case isn't about one particular site. It's the generic task of "assemble a picture of someone else's property from one public address — and don't get the conclusion wrong". It comes up far more often than you'd expect:
- → Counterparty checks before a deal — domain age, what serves the site today, related properties under the same certificate, history in the web archive
- → Technical competitor research — what the site is built on, which analytics and third-party services are wired in, what lived on the domain a year ago
- → Clones and phishing against your brand — a network of lookalike domains usually gives itself away through a shared certificate and shared infrastructure
- → Due diligence before buying a domain or a project — what used to be hosted there and what it's tied to through certificate history
- → Auditing your own perimeter — the same reconnaissance pointed inward: which of your subdomains and internal names are visible from outside for free
- The one-shot recon script: a domain in, a folder of per-step artifacts out
- A separate script for the active request — passive collection never touches the site itself
- A "signal → platform" reference that grows with every new analysis
- Rules of inference: an explicit split between signals that prove and signals that don't
- A delivery checklist — mandatory output fields, so the result doesn't stay in the analyst's head
If the question is "how do they actually do it" — the answer is usually already public
Most of the answer sits in free sources — it just has to be collected in the right order and read correctly. One site takes 30-60 minutes; the methodology, the scripts and the references stay with you and keep working without me.
Related cases
Auto-built weekly reports for clients
One n8n workflow across 8 clients: pulls the numbers from every ad platform, AI commentary, PDF/Notion →…
Astro landing + lead scoring: 4.6% demo conversion
Multi-step form + n8n scoring A/B/C/D + Dadata enrichment by INN + SPARK. Score ≥ B → amoCRM + Slack. Score D…
Apollo + Clay + Instantly pipeline: 50 qualified leads/week
4-stage pipeline: 1.2M ICP base from Apollo → Clay enrichment (SPARK + RSS + BuiltWith) → GPT personalization…
The 5,000 ₽ audit — with a concrete report and quote
I'll tell you what to deploy in your business first, what the payback looks like, and whether you need AI for the task at all (sometimes you don't).
Or just send your question — I reply within 2 hours