---
title: "Web Crawler_ How to Connect and Collect"
slug: "web-crawler-how-to-connect-and-collect"
tags: ["Web crawler", "Collect", "Connect"]
updated: 2025-02-18T13:43:20Z
published: 2025-02-18T13:43:20Z
canonical: "docs.revealdata.com/web-crawler-how-to-connect-and-collect"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.revealdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawler_ How to Connect and Collect

#### **In this article**:

- Web Crawler Overview
- Web Crawler Requirements
- How to Connect and Collect Using Web Crawler

## **Web Crawler Overview**

Onna's web crawler was created to index web pages.

| **Connector Features** |
| --- |
| **Authorized Connection Required?** No | **Is identity mapping supported?** No |
| **Audit logs available?** Yes | **Admin Access?** No |
| **Supports a full archive?** No | **Custodian based collections?** No |
| **Preserve in place with ILH?** No | **Resumable sync supported?** No |
| **Supports Onna preservation?** No | **Syncs future users automatically?** No |
| **Sync modes supported:** - One-time sync | **Is file versioning supported?** No |
|  |
| **Types of Data Collected** | Metadata Collected |
| - Headings - Subheadings - Paragraphs - Metadata - Pictures - Links - Text links | - There is no Web Crawler specific metadata. |

### **Web Crawler Considerations**

- The web crawler does not currently support password-protected websites or Captcha protected websites.
- You’re not able to collect files in their native format from the links on a web page during collection. Web crawler links are embedded so they will not be able to pull files in their native formats.
- You cannot collect data from a site if a user agent has issued a block request.

## **Web Crawler Requirements**

- When adding a new Web Crawler sync, you have to introduce the URLs with the protocol (http, https)

## **How to Connect and Collect Using Web Crawler**

To create a new Web Crawler collection follow the steps below:

### **Step 1**

Click on ‘Workspaces’ in the main menu (a), then click on the workspace where you’d like to add a new sync (b).

![](https://cdn.us.document360.io/3e21d801-ca9f-4c51-93db-9cbd32741f3d/Images/Documentation/web-crawler_-how-to-connect-and-collect-image-kicutxdw.png)

### **Step 2**

Click on the ‘+’ icon in the upper right corner to add a new source.

![](https://cdn.us.document360.io/3e21d801-ca9f-4c51-93db-9cbd32741f3d/Images/Documentation/web-crawler_-how-to-connect-and-collect-image-x33jos7m.png)

### **Step 3**

Select the Web Crawler connector from your list of available connectors.

![](https://cdn.us.document360.io/3e21d801-ca9f-4c51-93db-9cbd32741f3d/Images/Documentation/web-crawler_-how-to-connect-and-collect-image-1mndtlog.png)

### **Step 4**

To configure your sync start by entering a name for your source in the ‘Name’ field (a). Then, enter the URL you want to collect from (b). Finally, click the blue ‘Done’ button (c).

![](https://cdn.us.document360.io/3e21d801-ca9f-4c51-93db-9cbd32741f3d/Images/Documentation/web-crawler_-how-to-connect-and-collect-image-a3keunnd.png)

### **Step 5**

You’ll now see your new source appear alphabetically in the list of ‘Connected sources’ in your workspace.

![](https://cdn.us.document360.io/3e21d801-ca9f-4c51-93db-9cbd32741f3d/Images/Documentation/web-crawler_-how-to-connect-and-collect-image-1ce2lnu2.png)

##

## Related

- [Getting Started with Sources](/getting-started-with-sources.md)
- [Onna Sync Modes](/onna-sync-modes.md)