Perplexity caught using stealth, undeclared crawlers to evade website no-crawl directives

staff_writer • August 9, 2025

Imagine a web browser that promises to revolutionize how you explore the internet, powered by cutting-edge AI. Now imagine it’s launched by a company accused of sidestepping the internet’s ethical boundaries to gather the data that fuels it. That’s the story of Perplexity, an AI-powered search startup, which unveiled its Comet browser in July 2025, hot on the heels of allegations that it unlawfully scraped websites using stealth crawlers.


How does a company pivot from controversy to innovation, and what does this mean for the future of web browsing?

Perplexity Launches Comet Web Browser, After Getting Caught Scraping Sites Unlawfully


Perplexity, a rising star in AI-driven search, launched its Comet web browser in July 2025, aiming to redefine how users interact with the internet. Billed as a “thought partner,” Comet integrates Perplexity’s AI to provide real-time answers, contextual insights, and a seamless browsing experience. CEO Aravind Srinivas promised “core browsing improvements that Chrome hasn’t shipped for ages,” positioning Comet as a competitor to established giants.

But the launch comes under a cloud. Just weeks earlier, Cloudflare, a web security titan handling 20% of global web traffic, accused Perplexity of using “stealth, undeclared crawlers” to bypass website no-crawl directives, raising ethical and legal questions. This juxtaposition of innovation and controversy begs the question: Can Perplexity’s ambitious browser succeed while its data practices are under scrutiny?

What is Perplexity's Comet Browser: A New Frontier?

Comet is designed to blend AI-powered search with traditional browsing, offering features like:


The browser targets knowledge seekers, researchers, and professionals, boasting a user base of 15 million for Perplexity’s existing tools. Its launch aligns with a $18 billion valuation surge, fueled by deals like a partnership with Airtel to offer a free year of Perplexity Pro to 360 million customers. Yet, as Perplexity pushes boundaries, its methods for gathering the data that powers Comet have sparked a firestorm.

The Perplexity Scraping Scandal Exposed: What are  Stealth Crawlers? 


In August 2025, Cloudflare dropped a bombshell, alleging Perplexity used undeclared crawlers to evade website no-crawl directives, specifically robots.txt files. These files are the internet’s “do not enter” signs, guiding respectful bots on what content to avoid. Cloudflare’s tests revealed Perplexity’s tactics:

  • Disguised User Agents: When blocked, Perplexity’s bots switched from declared identifiers (PerplexityBot, Perplexity-User) to a generic Chrome-on-macOS user agent, mimicking human traffic.
  • IP Rotation: Bots used unlisted IP addresses and varied Autonomous System Numbers (ASNs) to dodge detection, generating 3-6 million daily stealth requests across tens of thousands of domains.
  • Ignoring Robots.txt: Perplexity often bypassed or failed to fetch robots.txt files, accessing restricted content on test domains created by Cloudflare.


This behavior, Cloudflare argued, violates web norms outlined in RFC 9309, undermining the trust-based internet ecosystem. In contrast, OpenAI’s ChatGPT respected robots.txt and ceased crawling when blocked, earning praise as a model of compliance. Perplexity’s response was defiant, dismissing Cloudflare’s report as a “publicity stunt” and arguing that AI assistants like theirs require real-time data to answer user queries, unlike traditional crawlers building databases. But does this justify bypassing website owners’ explicit wishes?

Ethical and Legal Fallout for Perplexity

The use of stealth crawlers raises thorny ethical and legal issues. Websites rely on robots.txt to protect sensitive data, reduce server strain, and maintain control over their content. Ignoring these directives can:


Perplexity’s practices aren’t new—developer Robb Knight reported similar scraping in 2024, and Amazon is reviewing whether Perplexity violated AWS terms. The controversy highlights a broader tension: AI companies’ hunger for data versus publishers’ rights to control their content.

Website Owners Need to Fight Back

Cloudflare has taken decisive action, delisting Perplexity as a verified bot and deploying heuristics to block its stealth crawlers, protecting over 2.5 million websites. Website owners can adopt strategies to combat unauthorized crawling:

  • Monitor Server Logs: Identify unusual traffic patterns or repeated requests from rotating IPs.
  • Deploy Honeypots: Set traps (e.g., hidden links) to catch crawlers, enabling IP bans.
  • Use Bot Management Tools: Tools like Cloudflare’s analyze request headers and behavior, blocking non-compliant bots.


Innovative defenses like Cloudflare’s “AI Labyrinth,” which traps bots in fake content, and “Pay Per Crawl,” reviving the HTTP 402 code, empower site owners to fight back. Learn about Cloudflare’s defenses here.

The scraping allegations ignited a firestorm online. On X, users like @TechBit lamented Perplexity’s “shady” tactics, while @paigemacp called for stricter.

As the internet evolves, one question remains: Can innovation coexist with respect for digital boundaries? The answer will shape not just Perplexity’s fate, but the web’s future.


Contact Us

Antonio brown football

Antonio Brown Apprehended in Dubai: Will Face Attempted Murder Charge in Miami During Adin Ross Stream

By Max • November 10, 2025
The dramatic fall of former NFL superstar Antonio Brown took a shocking turn on November 6, 2025, when U.S. Marshals apprehended him in Dubai, extraditing the 37-year-old wide receiver to face attempted murder charges in Miami. The allegations stem from a chaotic shooting incident outside a celebrity boxing event hosted by streamer Adin Ross on May 17, 2025, where videos captured Brown allegedly firing shots at a man during a brawl. Now held in Essex County, New Jersey, Brown awaits transfer to Miami-Dade County Jail, where he could face up to 15 years... Antonio Brown Apprehended in Dubai: Details of the Arrest If convicted of second-degree attempted murder with a firearm this high-profile arrest reignites scrutiny over Brown's post-NFL life. What is the Adin Ross connection, and what's next for the once-unstoppable athlete? Miami Police spokesperson Michael Vega confirmed the extradition, praising the "relentless work" of detectives: "This is just another example of the dedication... to identify and arrest those who commit crimes in the City of Miami." lol

PEGATRON CORPORATION'S FIRST U.S. FACILITY: $35M INVESTMENT IN GEORGETOWN, TX

By Max • November 10, 2025
Taiwanese electronics giant Pegatron announces its inaugural U.S. manufacturing site in Georgetown, Texas, with a $35 million capital commitment and hundreds of high-tech jobs. Explore the facility details, economic ripple, and AI focus amid global supply chain shifts.

Does Amazon take EBT ?

By Jack Sterling • November 7, 2025
Well not right now because the government is shut down. The burning question for online shoppers: Does Amazon accept EBT? The answer is yes — Amazon has accepted SNAP EBT payments nationwide since 2019, with seamless integration across Amazon Fresh, Amazon Grocery, and Whole Foods Market online. This program now serves over 42 million monthly participants , making Amazon one of the largest online EBT retailers in the U.S. Whether you're searching for Amazon EBT eligible items 2025 , how to use SNAP on Amazon Prime , or government shutdown impact on EBT benefits , this guide delivers official steps, exclusive discounts, and real data from USDA reports and Amazon as of November 06, 2025. SNAP/EBT on Amazon Acceptance Matters SNAP provides food assistance to low-income households through reloadable EBT cards. In fiscal year 2024, the program distributed $119.4 billion in benefits to an average of 42.1 million people monthly—roughly 12.5% of the U.S. population. Amazon's EBT program eliminates barriers like transportation and store hours, especially valuable during the ongoing shutdown that has delayed benefit loading in several states. Without physical access issues, SNAP users can order groceries for delivery or pickup using just their EBT card—no debit/credit needed for eligible items. How to Add EBT Card to Amazon Step-by-Step Amazon SNAP Setup Setting up SNAP on Amazon is free and takes minutes: Create or log into your Amazon account Go to Account & Lists > Your Payments > Add a payment method Select SNAP EBT and enter your card details Verify eligibility (Amazon checks against USDA database automatically) Optional: Register for Amazon Prime EBT discount (50% off— just $6.99/month) Over 85% of users complete setup in under 5 minutes, per Amazon's internal data. Once added, your EBT balance appears at checkout for eligible purchases. Amazon Fresh vs Whole Foods | Delivery & Pickup Options Amazon Fresh : Free 2-hour delivery on orders $35+ (or $100 in some areas) for Prime members; EBT covers groceries, you pay delivery fees separately Whole Foods Market : Same-day delivery or curbside pickup; exclusive organic and prepared cold items Nationwide coverage in all 50 states plus D.C. Prime members with EBT get unlimited free delivery on Amazon Fresh orders over $35—saving an average $139/year on fees alone.
Fire Chief Robert Tucker Resigns

Zohran's Fire Chief Robert Tucker Resigns Day #1

By Alfanso C. • November 7, 2025
In a shocking turn of events shaking New York City's public safety landscape, FDNY Commissioner Robert Tucker announced his resignation on November 5, 2025 one day after Zohran Mamdani's landslide victory mandate in mayoral election.

2025 Nissan Z vs Toyota GR Supra: Full Comparison, Specs, 0-60, Price & Reviews

By bennett F • November 7, 2025
The world of iconic Japanese sports cars, the 2025 Nissan Z vs Toyota GR Supra rivalry continues to thrill enthusiasts. Both deliver heart-pounding performance, sleek designs, and cutting-edge tech, but which one reigns supreme in 2025? The Nissan Z starts at an affordable $42,970 MSRP , packing 400 horsepower from its twin-turbo V6, while the Toyota GR Supra kicks off at $56,250 MSRP with 382 horsepower from a turbocharged inline-six. This head-to-head breakdown covers engine specs, design, interior, technology, pricing, and real-world driving dynamics to help you decide. Whether you're searching for 2025 Nissan Z review, Toyota Supra 2025 specs, or the best sports car under $60,000, we've got the details. Nissan Z vs Toyota GR Supra: The Ultimate Sports Car Comparison - Specs, Performance, Price & More is right here... In the fiercely competitive world of Japanese sports cars, the 2025 Nissan Z vs Toyota GR Supra battle remains one of the most exciting rivalries for driving enthusiasts. Both icons deliver blistering performance, head-turning style, and modern tech, but they cater to slightly different tastes. The 2 025 Nissa n Z starts at an accessible $42,970 MSRP with 400 horsepower from its twin-turbo V6, while the 2025 Toyota GR Supra commands $56,250 MSRP for 382 horsepower from a turbocharged inline-six. This in-depth comparison covers everything from acceleration times to interior luxury, helping you decide the best sports car for 2025. Nissan Engine Performance VS SUpra Agility Specs Both the 2025 Nissan Z and Toyota GR Supra are built for adrenaline, but their powertrains highlight distinct philosophies. Here’s how they stack up in key performance metrics: Engine : Nissan Z – 3.0L twin-turbo V6; Toyota GR Supra – 3.0L turbocharged inline-6 (BMW-sourced) Horsepower : Nissan Z – 400 hp (420 hp in NISMO trim); Toyota GR Supra – 382 hp Torque : Nissan Z – 350 lb-ft (384 lb-ft in NISMO); Toyota GR Supra – 368 lb-ft 0-60 mph : Nissan Z – ~4.5 seconds (manual Performance model); Toyota GR Supra – ~3.9–4.2 seconds Transmission Options : Nissan Z – 6-speed manual or 9-speed automatic; Toyota GR Supra – 6-speed manual or 8-speed automatic Fuel Economy (combined) : Nissan Z – 20–22 mpg; Toyota GR Supra – 21–26 mpg Drivetrain & Features : Both rear-wheel drive with launch control; Supra is ~300 lbs lighter for sharper handling The Z delivers a muscular V6 roar and explosive low-end torque, perfect for drag-strip launches. The Supra counters with smoother, more linear power delivery and superior track precision—often winning head-to-head races thanks to its lighter curb weight and razor-sharp steering. Design and Aesthetics The 2025 Nissan Z channels retro JDM heritage with sharp lines, iconic rectangular grille, and muscular rear haunches—a modern tribute to the legendary 240Z. New for 2025: heritage colors like Bayside Blue straight from the GT-R lineage. Meanwhile, the Toyota GR Supra embraces a futuristic, aerodynamic silhouette with flowing curves, an aggressive front fascia, and intricate LED lighting that screams high-speed stability. Interior Features and Comfort: Cockpits Built for Driver Inside, driver focus is priority one. The Nissan Z offers a classic analog-feel cockpit with premium leather/Alcantara seats, and intuitive analog gauges complemented by a digital cluster. The GR Supra elevates luxury with supple materials, Alcantara accents, and a more upscale ambiance—though the cabin feels slightly tighter. Both include heated seats, dual-zone climate control, and excellent lateral support for canyon carving or daily commutes. Technology and Connectivity | Nissan vs Supra Modern sports cars demand seamless tech integration. The Nissan Z rocks a 9-inch touchscreen (standard on Performance trim) with wired Apple CarPlay/Android Auto, Bose premium audio, and NissanConnect services including Wi-Fi hotspot and remote start. The Toyota GR Supra answers with an 8.8-inch display, wireless charging, JBL audio, and Supra Connect for over-the-air updates and remote lock/unlock. Safety suites are comprehensive: automatic emergency braking, blind-spot monitoring, and adaptive cruise control come standard on both. 2025 Nissan Z pricing : Starts at $42,970 (Sport), $52,970 (Performance), up to $65,750 (NISMO)—delivering 400+ hp for thousands less 2025 Toyota GR Supra pricing : $56,250 (3.0), $60,495 (3.0 Premium), with A91-MT special edition pushing higher The Z wins on pure value, often available below $44,000 with incentives. The Supra justifies its premium with refined engineering, bulletproof Toyota reliability , and stronger resale value. Prefer laser-sharp handling , efficiency, and everyday refinement? The Toyota GR Supra dominates autocross and grand touring. Test drive both back-to-back—the difference in character is night and day. In 2025, this rivalry has never been hotter.

Zelle p2p payment processor to allow global crypto stablecoin payments

By staff writer • November 2, 2025
In a move that could reshape the $800 billion remittance market, Zelle, America's dominant peer-to-peer payments app, has signaled its intent to go international. On October 24, 2025, announced an initiative to extend the network's "trust, speed, and convenience" beyond U.S. borders using stablecoins. This isn't a full rollout yet, but the groundwork leverages Zelle's massive domestic footprint and newfound regulatory tailwinds. Here's the data-driven breakdown of what's confirmed, what's implied, and what's next... The $1 Trillion Domestic Powerhouse Zelle Stepping in to Stable Coins Zelle isn't starting from scratch. The network, owned by seven major banks including Bank of America, JPMorgan Chase, and Wells Fargo, already processes staggering volumes entirely within the U.S. Key 2024 Zelle metrics: $1 trillion in total payments processed—a 27% year-over-year increase. 3.6 billion transactions, up 25%. 151 million unique enrolled users, encompassing consumers and small businesses. Market dominance is clear; It handles roughly twice Venmo's daily transactions and maintains a 99.95% fraud-free rate . This scale—reaching 80% of U.S. checking accounts via 2,300+ financial institutions —provides the launchpad for global ambitions. Stablecoins as the Bridge To Zelle EWS's press release frames the expansion as an evolution: "Faster and more reliable cross-border money movement" powered by USD-pegged stablecoins. CEO Cameron Fowler emphasized extending domestic perks internationally, with the service open to all Zelle Network participants on equal footing. Full announcement here via PR Newswire; echoed in Payments Dive coverage. Stablecoins enable instant, 24/7 settlement at fractions of traditional costs, sidestepping correspondent banking's multi-day delays and high fees. Timing aligns with the GENIUS Act , signed into law by President Trump in July 2025. This landmark bill establishes the U.S.'s first federal stablecoin regime: 100% reserves in cash or T-bills. Monthly audits and full BSA/AML compliance . Oversight by Fed, OCC, or state regulators —no SEC/CFTC for compliant issuers. Foreign coins only from comparable jurisdictions. Post-enactment, stablecoin supply jumped 20% , signaling confidence. EWS explicitly credits this "improved clarity" for accelerating their push. Timeline and Mechanics for Zelle Stable COin Mechanics Details remain sparse: Rollout date : No firm timeline, expect "weeks to months" for pilots. Stablecoin specifics : Will EWS issue its own, or partner ? Fees : Domestic Zelle is free; international likely low but unspecified. Industry watchers note consortium challenges, citing past efforts like Fnality's delays. Stay Tuned to the Objective Wire for more.

STAY UP TO DATE

GET Objective LATEST