curl-impersonate

What is the project about?

curl-impersonate is a specialized version of the popular curl command-line tool and library. It's designed to mimic the network behavior of major web browsers (Chrome, Edge, Safari, and Firefox) at the TLS and HTTP handshake level. This fork enhances the original with features like Encrypted Client Hello (ECH), ZSTD compression, and more.

What problem does it solve?

Many websites use TLS and HTTP/2 fingerprinting to identify the client accessing them. This can lead to websites delivering different content, blocking access, or otherwise restricting functionality based on the client detected. curl-impersonate bypasses these restrictions by making curl appear identical to a specific browser from a network perspective. It helps maintain a more open and less restrictive web experience.

What are the features of the project?

Browser Impersonation: Performs TLS and HTTP handshakes that are indistinguishable from those of real browsers (Chrome, Edge, Safari, Firefox, including specific versions).
Command-Line Tool: Can be used as a drop-in replacement for the standard curl command.
Library Integration: The libcurl-impersonate library can be used in place of the regular libcurl in applications.
Multiple Browser Versions: Supports impersonating various versions of Chrome, Edge, Safari and Firefox.
Encrypted Client Hello (ECH) Support: Includes support for ECH, a newer TLS feature.
ZSTD Compression Support: Adds support for ZSTD compression.
X25519Kyber768/X25519MLKEM Curves: Supports modern cryptographic curves.
Akamai HTTP/2 Fingerprint Impersonation: Enhanced options for impersonating Akamai fingerprints, especially for Safari.
Upgraded Curl Version: Based on a more recent version of curl (8.7.1 as of April 2024).
Customizable Extensions: Allows modification of TLS extension order and enabling/disabling TLS GREASE.
Single Binary: A single binary supports multiple browsers.
(In progress) HTTP/3 Support: Work is underway to enable HTTP/3.
Python Binding: A user-friendly Python binding called curl_cffi is available.

What are the technologies used in the project?

C: The core language of curl and this project.
BoringSSL: Google's TLS library (used by Chrome and Safari), crucial for accurate impersonation.
libnghttp2: A library for handling HTTP/2.
TLS/SSL: The underlying protocols for secure communication.
HTTP/2: The HTTP protocol version used for fingerprinting.
Patching: Significant modifications to the curl codebase are achieved through patches.
Zig: Used as a toolchain to support more architectures on Linux.
Docker: Used for creating containerized builds.
YAML: Used for the database of known browser signatures.

What are the benefits of the project?

Bypass Fingerprinting: Avoids detection and restrictions based on TLS and HTTP/2 fingerprinting.
Access Restricted Content: Allows access to websites that might otherwise block or alter content for non-browser clients.
Improved Web Scraping: Makes web scraping more reliable by mimicking real browsers.
Testing and Development: Useful for testing website behavior with different browser fingerprints.
Open Web Advocacy: Contributes to a more open and less restrictive web environment.
Cross-compilation: Supports more architectures.

What are the use cases of the project?

Web Scraping: Gathering data from websites that employ anti-bot measures.
Security Research: Analyzing website security and fingerprinting techniques.
Penetration Testing: Simulating browser traffic for security assessments.
Website Testing: Verifying how a website behaves with different browsers.
Accessing APIs: Interacting with APIs that restrict access based on client fingerprint.
Bypassing Geo-Restrictions: In some cases, can help bypass restrictions based on perceived client location (if tied to browser fingerprinting).
Development: Building applications that need to mimic specific browser behavior.