Skip to main content

Fixing 'app-ads.txt Not Found' Errors on AdMob

 There are few administrative tasks in mobile development more frustrating than the app-ads.txt verification cycle. You have created the file, uploaded it to your hosting provider, and verified it opens in your browser. Yet, the AdMob console persistently flags it as "Not Found" or "Not Crawled."

This results in a tangible revenue loss. When AdMob cannot verify your inventory ownership, ad serving is throttled or suspended entirely.

The discrepancy between "I can see the file" and "AdMob cannot see the file" usually stems from strict crawler protocols regarding redirects, content types, or server-side routing logic. This guide dissects the crawling mechanism and provides infrastructure-level fixes for Nginx, Apache, and modern Single Page Applications (Next.js/React).

The Root Cause: How the AdMob Crawler Works

To fix the error, you must understand the crawler's rigid constraints. AdMob does not behave exactly like a standard user browser.

When you list a "Developer Website" on the Google Play Store or Apple App Store (e.g., https://www.example.com/support), the crawler performs specific transformations:

  1. Subdomain Stripping: It strips www., m., and path segments. It attempts to fetch the file from the root apex domain: http://example.com/app-ads.txt and https://example.com/app-ads.txt.
  2. Redirect Sensitivity: While the crawler follows redirects (301/302), it has a limit (usually 5 hops). More importantly, if your root domain redirects to www but fails to pass the specific path (/app-ads.txt) correctly during that redirect, the crawler hits a 404.
  3. MIME Type Enforcement: The server must return a Content-Type: text/plain header. If your server returns text/html (common in React/Angular apps) or application/octet-stream, validation fails.
  4. User-Agent Blocking: If your robots.txt allows standard users but restricts bots, the Google-AdMob-Crawler will be rejected.

Solution 1: Diagnosing with CLI Tools

Do not rely on your browser to test accessibility. Browsers cache heavily and automatically handle complex redirects. Use curl to simulate the crawler's perspective.

Run the following command in your terminal. Replace example.com with your domain (omit www).

# -I: Fetch headers only
# -L: Follow redirects
# -A: Simulate Googlebot
curl -I -L http://example.com/app-ads.txt -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Analyzing the Output

A successful response looks like this:

HTTP/1.1 301 Moved Permanently
Location: https://www.example.com/app-ads.txt

HTTP/2 200
content-type: text/plain; charset=utf-8

Red Flags to look for:

  • HTTP 403 Forbidden: Your WAF or robots.txt is blocking the bot.
  • content-type: text/html: Your SPA is serving a "Not Found" React/Vue page instead of the raw text file.
  • Infinite Loop: The same URL repeats in the output until curl aborts.

Solution 2: Nginx Configuration Fix

If you host your marketing site on a VPS using Nginx, incorrect redirect logic is the most common culprit.

If you force non-www to www, ensure the app-ads.txt path is preserved.

The Problematic Config

Many developers use a blanket redirect that drops the URI: return 301 https://www.example.com; (This redirects example.com/app-ads.txt to the homepage example.com, causing a verification failure).

The Correct Nginx Config

Open your site configuration (usually in /etc/nginx/sites-available/):

server {
    listen 80;
    server_name example.com;

    # Correctly redirects example.com/anything to www.example.com/anything
    return 301 https://www.example.com$request_uri;
}

server {
    listen 443 ssl http2;
    server_name www.example.com;

    root /var/www/html;
    index index.html;

    # Explicitly handle app-ads.txt to ensure correct headers
    location = /app-ads.txt {
        default_type text/plain;
        try_files $uri =404;
        
        # Optional: Allow CORS if you have cross-domain ad verification tools
        add_header Access-Control-Allow-Origin *;
    }

    # Rest of your config...
}

Reload Nginx to apply changes:

sudo nginx -t && sudo systemctl reload nginx

Solution 3: Handling Next.js / React (App Router)

Modern frameworks often hijack routing. If you place app-ads.txt in the root of your source code, the bundler usually ignores it.

Step 1: Correct File Placement

In Next.js (and most webpack-based setups like Create React App), static files must live in the public/ directory.

  • Wrong: src/app/app-ads.txt
  • Correct: public/app-ads.txt

Step 2: Enforcing Headers in next.config.mjs

Even in public/, some hosting providers (like Vercel or Netlify) might default to incorrect content types. You can force the header in your config.

/** @type {import('next').NextConfig} */
const nextConfig = {
  async headers() {
    return [
      {
        source: '/app-ads.txt',
        headers: [
          {
            key: 'Content-Type',
            value: 'text/plain; charset=utf-8',
          },
        ],
      },
    ];
  },
};

export default nextConfig;

This ensures that when the crawler hits the route, it receives an unambiguous signal that this is a plain text file, preventing hydration attempts by the framework.

Solution 4: Robots.txt Permissions

Your site might be preventing the crawler from entering. Check your robots.txt file (usually found at example.com/robots.txt).

If you see this, you are blocking AdMob:

User-agent: *
Disallow: /

You must explicitly allow the ads crawler. Add the following exceptions to your robots.txt:

User-agent: *
Allow: /app-ads.txt

# Specifically allow Google's ad bot
User-agent: AdsBot-Google
Allow: /

User-agent: Googlebot
Allow: /app-ads.txt

Advanced Pitfall: Cloudflare and Caching

If you use Cloudflare, it may serve a cached version of a 404 error or a previous invalid file version.

  1. Log in to the Cloudflare Dashboard.
  2. Navigate to Caching > Configuration.
  3. Click Purge Everything (or Custom Purge for just http://example.com/app-ads.txt).

Furthermore, ensure your "Bot Fight Mode" or WAF rules in Cloudflare are not issuing a "Managed Challenge" to the Google Bot. Check Security > Events and filter by the path /app-ads.txt. If you see "Challenge" or "Block," create a WAF exception rule to "Skip" security checks for this specific URI.

Summary Checklist

If AdMob still reports the file as missing after 24 hours, verify this list:

  1. Root Access: Does example.com/app-ads.txt (no www) resolve successfully via curl?
  2. MIME Type: Is the Content-Type header strictly text/plain?
  3. Encoding: Is the file UTF-8 encoded? (Avoid BOM characters if creating via Notepad).
  4. Robots: Is robots.txt permitting access?

AdMob crawlers can take up to 24 hours to re-crawl. However, once the curl test returns a 200 OK with the correct text body, the issue is resolved on your end; it is simply a waiting game for the propagation.

Labels: