What SetRepo checks, and what it doesn't

Last updated

No scanner finds every issue. This page lists what each scan type checks, the categories of rules, what SetRepo doesn't check, and how we test it. Every report also lists what it couldn't check for that app.

What each scan type checks

Quick scan (free): every deterministic check. That's SetRepo's own checks, an access-control analysis built from your app's server routes, and the open-source scanners Gitleaks (secrets, including git history), Opengrep, Bandit (Python), ESLint with security rules (JavaScript and TypeScript), OSV-Scanner (known vulnerabilities in dependencies) and Trivy (infrastructure files). No AI.

Deep scan (paid): everything in a Quick scan, plus an AI review of object-level access (IDOR) and business logic, and a second opinion on critical and high findings from checks that aren't sure of themselves. The review confirms them, or clears them with a reason.

Live site: with a domain, SetRepo also checks the running site: TLS certificate, SPF, DKIM and DMARC records, DNS and domain expiry, security headers, and the JavaScript it serves for keys and exposed Supabase or Firebase config. On a domain you've verified, it also looks for exposed files and Supabase or Firebase data anyone can read without signing in.

Rule categories

  • API security: routes, middleware and server configuration.
  • CORS configuration.
  • Database: access and connection settings.
  • Caching.
  • Tokens and authentication.
  • Scalability.
  • Secrets and dependencies: keys in code, env files and git history, and known-vulnerable packages.
  • Access control: unauthenticated routes that change data, admin routes without a role check, protection that only exists in the browser, IDOR, and Supabase and Firebase rules.
  • IDOR and object access (Deep scans, AI review).
  • Business logic, such as prices trusted from the browser (Deep scans, AI review).
  • Infrastructure hygiene (with a domain): TLS, email records, DNS.
  • Live app (with a domain): the running site and the JavaScript it serves.
  • Accessibility: static checks of markup and CSS.
  • Compliance signals: whether a privacy link and cookie consent are present. Not a legal verdict.
  • SEO: titles, meta tags, headings, canonical links, robots and sitemap.

Opengrep rules cover Python, JavaScript and TypeScript, Go, Ruby, PHP, Java and other JVM languages, C#, Rust and Dart.

What SetRepo doesn't check

SetRepo reads code and looks at your live site from outside. It doesn't run your app, log in to it, or attack it. These are not checked, or only partly:

  • SQL injection, XSS, command injection and unsafe deserialization outside Python and JavaScript/TypeScript, and data flows that cross files.
  • Missing input validation in general, and file uploads beyond a structural check.
  • Token expiry that's set but too long, revocation that exists but isn't enforced, and predictable session IDs.
  • Verbose errors that leak stack traces, and secrets or personal data written to logs.
  • N+1 queries, missing pagination, blocking calls in async handlers, and other performance issues.
  • Cache invalidation when permissions change, and sensitive data kept in localStorage.
  • Git history beyond the default branch's last 100 commits.
  • Without the code (a live URL only): API routes, sign-in, database rules and dependencies.

On a Quick scan, the access-control and business-logic issues that need the AI review aren't checked; the report says so.

How we test

Every change to the scanner runs the full test suite before it can deploy. A failing test stops the release.

  • A precision and recall benchmark: small apps that re-create real findings verified by hand on real repos, and apps with planted bugs. Each case is labeled as an issue the scan must report or a false positive it must not. The benchmark runs the real Quick scan pipeline, offline. A release can't lose a true positive beyond the known misses, and its precision can only go up.
  • Rule tests: every Opengrep rule has fixture code marking a line it must match and a line it must not, and any match that isn't marked fails the test. The built-in checks have their own tests of what they report and what they leave alone.
  • Secret detection is tested on made-up keys generated when the test runs, never real ones, including that the value never appears in a finding.
  • We also run end-to-end tests (Playwright) of the website and dashboard in a real browser against a real backend: sign-in, reports, the Feed, pricing and navigation.