Skip to content

Electron based web crawler, can be used to automate several tasks

License

Notifications You must be signed in to change notification settings

NewronAI/cortex

Repository files navigation

Cortex by Newron.ai

Cortex is a powerful and efficient general-purpose web crawler designed and developed by Newron.ai. It aims to simplify the process of collecting data from various web sources, enabling users to extract valuable information with ease.

Table of Contents

Features

  • Customizable and extensible crawling rules
  • Advanced filtering options to target specific data
  • Multithreading support for faster crawling
  • Built-in caching and request throttling to prevent overloading target websites
  • User-friendly Electron app with React and Tailwind CSS for easy configuration and management

Getting Started

These instructions will help you set up Cortex on your local machine for development and testing purposes.

Prerequisites

Before you start, make sure you have the following installed on your system:

  • Node.js (v14.0.0 or later)
  • npm (v7.0.0 or later)

Installation

  1. Clone the repository:

    git clone https://github.com/Newron.ai/Cortex.git
    
  2. Navigate to the project directory:

    cd Cortex
    
  3. Install the dependencies:

    npm install
    
  4. Run the application:

    npm start
    

Usage

Configuration

  1. Open the Cortex application.
  2. Configure the crawling rules, filters, and other options through the user interface.

Running the Crawler

  1. Start the crawler by clicking the "Start Crawler" button.
  2. Monitor the progress and view the collected data in the application.

Contributing

We welcome contributions from the community. If you'd like to contribute to the Cortex project, please follow these steps:

  1. Fork the repository.
  2. Create a new branch for your feature or bug fix.
  3. Make your changes and commit them with a descriptive commit message.
  4. Push your changes to your fork.
  5. Open a pull request and describe the changes you made.

Please make sure to follow our coding standards and guidelines when contributing.

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Acknowledgements