# Scraping

> Configure website scraping behavior and exclusions.

Source: https://docs.graph8.com/settings/scraping/

Control how graph8 scrapes websites and which pages to include or exclude.

## Scraping Settings

graph8 can scrape website data to enrich contact and company records. Configure global scraping preferences for your organization.

### Global Preferences

1. Go to **Settings → Scraping**
2. Configure your scraping behavior:
   - **Enable/Disable** — toggle scraping on or off for your organization
   - **Concurrency** — number of simultaneous scrape requests
   - **Delay** — wait time between requests to the same domain

## Skip Rules

Define which URLs or patterns should be excluded from scraping.

### Adding Skip Rules

1. Click **Add Rule**
2. Choose the rule type:
   - **Exact URL** — skip a specific page
   - **URL Pattern** — skip pages matching a pattern (e.g., `/blog/*`)
   - **Domain** — skip an entire domain
3. Enter the URL or pattern
4. Save

### Common Skip Patterns

- Login and authentication pages
- Terms of service and legal pages
- Internal tool URLs
- Social media profile pages
- Pages with sensitive information

### Managing Rules

- View all active skip rules in the rules list
- Toggle rules on/off without deleting them
- Edit patterns as your needs change
- Delete rules that are no longer needed

## Allowed Domains

Restrict scraping to specific domains for targeted data collection.

### Whitelist Mode

When enabled, graph8 only scrapes domains you've explicitly allowed:

1. Toggle **Whitelist Mode** on
2. Add domains to the allowed list
3. Only these domains are scraped

When disabled, all domains are scraped except those matching skip rules.

## Rate Limits

Control scraping speed to avoid overwhelming target websites.

### Settings

- **Requests per second** — maximum scrape requests per second per domain
- **Concurrent connections** — maximum simultaneous connections
- **Retry attempts** — how many times to retry a failed scrape

graph8 automatically respects `robots.txt` directives on target websites.

## Data Handling

Configure how scraped data maps to your records.

### Field Mapping

Scraped data can populate:

- **Company fields** — website description, industry, technologies, employee count
- **Contact fields** — job title, social profiles, bio

### Data Quality

- graph8 deduplicates scraped data against existing records
- New data only fills empty fields unless you enable overwrite mode
- All scraped data is logged for audit purposes

## Frequently Asked Questions

### Does graph8 respect robots.txt?

Yes. graph8 follows `robots.txt` directives by default. Pages disallowed by robots.txt are not scraped.

### What happens if a scrape fails?

Failed scrapes are retried based on your retry settings. After all retries are exhausted, the URL is logged as failed and skipped until the next scheduled run.

### Can I schedule scraping?

Scraping runs as needed (e.g., when a new company is added or during enrichment). Recurring schedules can be configured for batch operations.

### Will scraping affect my website's performance?

graph8 only scrapes external websites (your prospects' sites), not your own. Rate limits ensure scraping doesn't overload target sites.

---

> **Tip:**
> Start with conservative rate limits and increase gradually based on your needs.