Perplexity caught using stealth, undeclared crawlers to evade website no-crawl directives
staff_writer • August 9, 2025
Imagine a web browser that promises to revolutionize how you explore the internet, powered by cutting-edge AI. Now imagine it’s launched by a company accused of sidestepping the internet’s ethical boundaries to gather the data that fuels it. That’s the story of Perplexity, an AI-powered search startup, which unveiled its Comet browser in July 2025, hot on the heels of allegations that it unlawfully scraped websites using stealth crawlers.
How does a company pivot from controversy to innovation, and what does this mean for the future of web browsing?
Perplexity Launches Comet Web Browser, After Getting Caught Scraping Sites Unlawfully
Perplexity, a rising star in AI-driven search, launched its Comet web browser in July 2025, aiming to redefine how users interact with the internet. Billed as a “thought partner,” Comet integrates Perplexity’s AI to provide real-time answers, contextual insights, and a seamless browsing experience. CEO Aravind Srinivas promised “core browsing improvements that Chrome hasn’t shipped for ages,” positioning Comet as a competitor to established giants.
But the launch comes under a cloud. Just weeks earlier, Cloudflare, a web security titan handling 20% of global web traffic, accused Perplexity of using “stealth, undeclared crawlers” to bypass website no-crawl directives, raising ethical and legal questions. This juxtaposition of innovation and controversy begs the question: Can Perplexity’s ambitious browser succeed while its data practices are under scrutiny?
What is Perplexity's Comet Browser: A New Frontier?
Comet is designed to blend AI-powered search with traditional browsing, offering features like:
The browser targets knowledge seekers, researchers, and professionals, boasting a user base of 15 million for Perplexity’s existing tools. Its launch aligns with a $18 billion valuation surge, fueled by deals like a partnership with Airtel to offer a free year of Perplexity Pro to 360 million customers. Yet, as Perplexity pushes boundaries, its methods for gathering the data that powers Comet have sparked a firestorm.
The Perplexity Scraping Scandal Exposed: What are Stealth Crawlers?
In August 2025, Cloudflare dropped a bombshell, alleging Perplexity used undeclared crawlers to evade website no-crawl directives, specifically robots.txt files. These files are the internet’s “do not enter” signs, guiding respectful bots on what content to avoid. Cloudflare’s tests revealed Perplexity’s tactics:
- Disguised User Agents: When blocked, Perplexity’s bots switched from declared identifiers (PerplexityBot, Perplexity-User) to a generic Chrome-on-macOS user agent, mimicking human traffic.
- IP Rotation: Bots used unlisted IP addresses and varied Autonomous System Numbers (ASNs) to dodge detection, generating 3-6 million daily stealth requests across tens of thousands of domains.
- Ignoring Robots.txt: Perplexity often bypassed or failed to fetch robots.txt files, accessing restricted content on test domains created by Cloudflare.
This behavior, Cloudflare argued, violates web norms outlined in RFC 9309, undermining the trust-based internet ecosystem. In contrast, OpenAI’s ChatGPT respected robots.txt and ceased crawling when blocked, earning praise as a model of compliance. Perplexity’s response was defiant, dismissing Cloudflare’s report as a “publicity stunt” and arguing that AI assistants like theirs require real-time data to answer user queries, unlike traditional crawlers building databases. But does this justify bypassing website owners’ explicit wishes?
Ethical and Legal Fallout for Perplexity
The use of stealth crawlers raises thorny ethical and legal issues. Websites rely on robots.txt to protect sensitive data, reduce server strain, and maintain control over their content. Ignoring these directives can:
Perplexity’s practices aren’t new—developer Robb Knight reported similar scraping in 2024, and
Amazon is reviewing whether Perplexity violated AWS terms. The controversy highlights a broader tension: AI companies’ hunger for data versus publishers’ rights to control their content.
Website Owners Need to Fight Back
Cloudflare has taken decisive action, delisting Perplexity as a verified bot and deploying heuristics to block its stealth crawlers, protecting over 2.5 million websites. Website owners can adopt strategies to combat unauthorized crawling:
- Monitor Server Logs: Identify unusual traffic patterns or repeated requests from rotating IPs.
- Deploy Honeypots: Set traps (e.g., hidden links) to catch crawlers, enabling IP bans.
- Use Bot Management Tools: Tools like Cloudflare’s analyze request headers and behavior, blocking non-compliant bots.
Innovative defenses like Cloudflare’s “AI Labyrinth,” which traps bots in fake content, and “Pay Per Crawl,” reviving the HTTP 402 code, empower site owners to fight back.
Learn about Cloudflare’s defenses here.
The scraping allegations ignited a firestorm online. On X, users like @TechBit lamented Perplexity’s “shady” tactics, while @paigemacp called for stricter.
As the internet evolves, one question remains: Can innovation coexist with respect for digital boundaries? The answer will shape not just Perplexity’s fate, but the web’s future.
Contact Us
Grammarly has introduced a groundbreaking authorship tracking functionality, rolled out in September 2025, to all users across Google Docs and Microsoft Word, revolutionizing digital ownership in collaborative writing. This feature enables users to generate unlimited authorship reports, attributing contributions and identifying external sources, including AI-generated text, with enhanced capabilities for Pro and Premium subscribers. Grammarly’s Authorship Tracking Paves the way for Digital Ownership and plagirism Grammarly’s authorship tracking, launched in September 2025, empowers users to monitor and attribute text contributions in real-time across Google Docs and Microsoft Word. External Source Tracking: Identifies text from websites, apps, or AI tools like ChatGPT for accurate attribution. Detailed Reports: Provides granular insights into content origins, exclusive to Pro/Premium users. Ethical Compliance: Supports plagiarism-free writing, aligning with ethical standards in 2025. Available to all users, it generates shareable authorship reports, ensuring clarity in collaborative projects. Pro and Premium plans offer advanced attribution, identifying text from websites, apps, and generative AI, addressing the 65% rise in AI-generated content usage since 2023 . How Grammarly’s Authorship Tracking Works in Google Docs and Microsoft Word Grammarly’s authorship tracking integrates into Google Docs and Microsoft Word, analyzing contributions to produce detailed reports on who wrote or edited text. Enabled via the Grammarly extension, it captures real-time inputs, distinguishing between original text, copied content, and AI-generated material for Pro/Premium users. For example, it identifies text pasted from a website or generated by tools like ChatGPT, attributing it accurately. Reports can be shared with collaborators, ensuring transparency in projects like academic papers or team proposals. The system leverages AI to process metadata, offering a clear contribution map without disrupting workflow. This functionality addresses the 74% of writers prioritizing plagiarism checks in 2024 , promoting accountability and streamlined collaboration across platforms. Implications for the Future of Digital Authorship Tracking Grammarly’s authorship tracking sets a new standard for digital ownership, addressing challenges in a world where 50% of digital content in 2025 involves AI contributions . By ensuring accurate attribution, it combats plagiarism and misinformation, fostering trust in collaborative platforms. The feature may influence industry standards, encouraging competitors to adopt similar tools, and could drive regulatory discussions on digital copyright. Future advancements might integrate blockchain for immutable authorship records, enhancing security. As AI-generated content grows, Grammarly’s technology positions it as a leader in ethical content creation, shaping a transparent and authentic digital ecosystem for creators worldwide.
The Most Expensive Lifetime Fitness in Texas. Life Time a leader in luxury athletic clubs, is expanding in Austin with a new 57,000-square-foot location at 1301 South Lamar Blvd, Suite 100, set to open in November 2025. This state-of-the-art facility, part of the four-acre mixed-use Bouldin development, marks the company's fifth club in Austin and 34th in Texas, blending fitness with Austin's vibrant culture of music, entertainment, and Texas swagger. State-of-the-Art Facilities Inside Life Time’s 57,000 Square Feet Austin Club Spanning 57,000 square feet, the South Lamar club features expansive workout zones with cutting-edge cardio and strength equipment from brands like Technogym and Rogue, illuminated by natural light through floor-to-ceiling windows for an energizing atmosphere. Specialized studios host over 50 weekly classes, including yoga, HIIT, and cycling, while an indoor saltwater pool supports lap swimming and aqua fitness. Premium Amenities Enhancing Life Time South Lamar’s Wellness Offerings Diverse Fitness Options: Unlimited classes (yoga, HIIT, cycling), personal training, and squash/pickleball courts. Holistic Wellness : LifeSpa with cryotherapy, infrared saunas, and organic LifeCafe meals using local ingredients. Family and Work Support : Kids’ camps, teen programs, and co-working spaces for flexibility. Membership Benefits : Starting at $179/month with access to all Texas clubs and virtual classes. These amenities, tailored to diverse needs, create a resort-like experience, aligning with Life Time’s model serving over 1.5 million members nationwide , enhancing physical and mental well-being in Austin’s dynamic community. How to Join NEW Grand Opening of LifeTime South Lamar Location Austin, TX The November 2025 grand opening of Life Time South Lamar will feature a weekend festival with live Austin bands, local food trucks like Torchy's Tacos, and free fitness demos showcasing the 57,000-square-foot facility. VIP previews for waitlist members (sign up at lifetime.life/south-lamar ) include guided tours and trial classes, with a ribbon-cutting ceremony attended by local leaders. Pop-up markets, giveaways, and wellness workshops will draw thousands in the first month, celebrating Austin’s spirit while highlighting amenities like the pool, spa, and studios. This event cements Life Time’s role as a community cornerstone, fostering health and connection. Embracing Austin’s Culture: Fitness Meets Local Flair at Life Time South Lamar Life Time South Lamar integrates Austin's eclectic vibe with programs like live-music yoga sessions featuring local artists and Texas BBQ-inspired recovery shakes at the on-site café. The club showcases rotating exhibits of Austin murals and hosts events tied to South by Southwest, blending fitness with the city's "Keep Austin Weird" ethos. Outdoor patios overlook the neighborhood's trendy spots, encouraging post-workout explorations of food trucks and live music on South Lamar. This cultural fusion positions the club as a community anchor, supporting Austin's wellness scene where 65% of residents prioritize integrated lifestyle fitness , enhancing social connections through member mixers and charity runs. Location and Accessibility at 1301 South Lamar Suite 100 Austin TX The Life Time South Lamar club at 1301 South Lamar Blvd, Suite 100, anchors the Bouldin development in Austin's dynamic South Lamar neighborhood, known for its eclectic mix of music venues, eateries, and artistic spaces. This central spot ensures easy access via South Lamar Boulevard, a major artery connecting to downtown and surrounding areas, with proximity to public transit like CapMetro buses and routes. Ample parking, including EV charging stations, caters to drivers, while bike lanes and scooter-friendly paths align with Austin's green initiatives. The location's walkability score of 85 enhances convenience, placing it near Zilker Park for outdoor extensions of workouts. Construction began in late 2024, with the mid-to-late 2025 opening timed for peak community engagement.
In an era where your online footprint defines your reality, what if a single platform could shape how the world sees you—or control what you see? Yext, Inc. (NYSE: YEXT), a digital experience platform, manages and analyzes online data to optimize customer interactions and brand visibility across search engines, social media, and maps. But as it empowers businesses to influence public perception, could it also pave the way for dystopian systems tracking digital lives? Yext Overview: A Digital Visibility Powerhouse Yext, Inc. (YEXT) is a leading digital experience platform that helps businesses manage their online presence, ensuring accurate information across platforms like Google, Facebook, and Apple Maps. Founded in 2006 and headquartered in New York City, Yext went public in 2017 and serves over 2,500 clients, including major brands like Verizon and Marriott. The company's mission is to connect customers with businesses by optimizing search queries, social media interactions, and location data. In 2025, Yext launched Yext Research, a program to advance SEO and marketing strategies, amid CEO Michael Walrath's non-bindin g acquisition proposal to take the company private. With a market cap of $726 million and Q4 2025 revenue of $113.1 million (up 12%), Yext is a key player in digital marketing. But is its data prowess a double-edged sword? What Does Yext Do? Core Products and Services Yext offers a cloud-based platform that unifies a brand's online data, enabling consistent messaging across search, social, and maps. Its core products include: Listings Management: Syncs business info to 200+ directories, ensuring accuracy and boosting local SEO. Reviews and Social Management: Monitors and responds to customer feedback on platforms like Google and Facebook. Analytics and AI Insights: Analyzes search queries and social data to optimize interactions, with Yext Research launching in June 2025 to power future SEO strategies. Pages and Search Experiences: Builds customized landing pages and search tools for enhanced user engagement. Yext's AI-driven features, like natural language processing for queries, help brands shape customer perceptions. The tool's ability to track social interactions and influence online narratives is powerful, but could it enable manipulative systems? Yext's Role in the Digital Economy: Enhancing Brand Presence Yext empowers businesses to control their digital narrative by managing listings, reviews, and social data. For example, a retailer can update hours across platforms instantly, avoiding customer frustration. Its analytics provide insights into search trends, helping optimize marketing. In Q1 2026, Yext reported earnings matching estimates, with revenue beating projections. The company's focus on AI for personalized experiences positions it as a leader in digital transformation. But this data aggregation capability sparks concerns about privacy and manipulation. Yext Financial Performance: Should you invest ? Yext's Q4 fiscal 2025 results showed revenue of $113.1 million, up 12% year-over-year, driven by acquisitions like Hearsay Systems. The stock (YEXT) traded at $5.20 in August 2025, with a market cap of $726 million. Analysts forecast moderate growth, with an average price target of $7.50, implying a 44% upside. However, a CEO's acquisition proposal to take Yext private at an undisclosed price adds uncertainty. With revenue growth at 10% annually, Yext is positioned for stability but faces competition from Google and Yelp. Yext's 2025 trajectory includes the CEO's non-binding acquisition proposal, expressing openness to alternative bidders to maximize stockholder value. The company plans to release Q2 2026 results in early September, expecting continued growth. With a focus on AI for SEO and marketing, Yext Research aims to power future strategies.
Phantom Neuro, an Austin, Texas-based neurotechnology startup spun out of Johns Hopkins University School of Medicine, has secured an oversubscribed $19 million Series A funding round, bringing its total funding to $28 million, announced on April 15, 2025. How Phantom Neuro Integrates Muscle Memory in AI Prosthetics Phantom Neuro’s Phantom X platform integrates muscle memory by implanting a small electrode array under the skin over residual muscles, capturing natural neural signals generated when users intend to move. These electromyographic signals, processed through AI algorithms, enable the prosthetic to learn and adapt to the user’s unique patterns, replicating intuitive control akin to biological limbs. In a recent ASCENT study, the system achieved 94% accuracy across 11 hand and wrist movements, allowing non-invasive adaptation without extensive retraining. Phantom Neuro’s Tech vs. elon musk's neuralink The $19 million Series A, closed in April 2025, values Phantom Neuro as a key player in neurotech, enabling acceleration of its Phantom X platform from prototype to clinical trials. Ottobock’s leadership investment, with a board seat for its CFO Dr. Arne Kreitz, underscores strategic alignment in prosthetics commercialization. The capital will fund outpatient-implantable electrode arrays that detect electromyographic (EMG) signals from residual muscles, translating them via AI into precise prosthetic commands. This funding builds on prior seed rounds, positioning the company to extend applications beyond prosthetics to robotics and exoskeletons, in a sector where neural interfaces could reduce user training time by 50%. Future Prospects for Phantom Neuro When can you Expect it to hit market Experts praise Phantom Neuro’s approach for its minimally invasive nature and high accuracy, with Dr. Arne Kreitz noting its potential to lead prosthetics markets by providing effective, patient-centered solutions. Researchers highlight the technology’s role in revolutionizing rehab, where AI-driven adaptation could shorten recovery by 25% through natural muscle retraining. What is Phantom Neuro and its main technology? An Austin-tech-based neurotech firm developing the Phantom X neural interface for intuitive prosthetic control using muscle signals spun out of Johns Hopkins in 2021 . Who led the $19M Series A funding round? Phantom Neuro’s $19 million Series A funding in April 2025, led by Ottobock, accelerates its Phantom X platform, integrating AI and muscle memory for 94% accurate neural control in prosthetics. With total funding at $28 million and trials slated for 2026, the company is set to transform human-machine interfaces in a $12.3 billion market by 2030, enhancing autonomy for amputees and beyond. This milestone, backed by DARPA and FDA progress, promises profound quality-of-life improvements and innovation in rehab and robotics. For more on neurotech advancements, visit Objectwire.org .
PsiQuantum, a leading quantum computing company, has raised $1 billion in a Series E funding round, valuing it at $7 billion, announced on September 10, 2025. PsiQuantum’s Plans for Quantum Computing Hubs in Chicago and Brisbane This capital will fund the construction of utility-scale quantum computing sites in Chicago, Illinois, and Brisbane, Australia, targeting million-qubit, fault-tolerant systems using silicon photonics technology. Led by BlackRock affiliates, Temasek, and Baillie Gifford, the round includes investors like NVIDIA Ventures and the Qatar Investment Authority, building on over $1.3 billion raised previously. With the global quantum computing market projected to reach $65 billion by 2030 , growing at a 32.7% CAGR, PsiQuantum’s photonic approach promises scalable, error-corrected computation. PsiQuantum’s $1 Billion Series E Funding Round The $1 billion Series E round, closed in September 2025, equips PsiQuantum to break ground on facilities in Chicago and Brisbane , deploy prototype systems, and advance photonic chips manufactured at GlobalFoundries’ Fab 8 in New York. These sites will host the world’s first million-qubit-scale, fault-tolerant quantum computers, focusing on error-corrected architectures to overcome qubit decoherence. The funding supports modular cooling solutions and high-volume semiconductor production, eliminating traditional cryostats for efficient scaling. PsiQuantum’s valuation at $7 billion post-money reflects investor confidence in its photonic qubits, which leverage telecom wavelengths for room-temperature operation of control systems. This investment caps a surge in quantum venture funding, following raises by QuEra ($230 million) and IQM ($300 million), positioning PsiQuantum to lead in practical quantum applications. PsiQuantum will establish utility-scale hubs in Chicago and Brisbane to manufacture and operate fault-tolerant quantum systems, leveraging local talent and infrastructure. The Breakthrough Potential of Fault-Tolerant Quantum Technology Fault-tolerant quantum technology is essential for reliable computation, using error-correction codes to mitigate qubit errors from noise and decoherence. PsiQuantum’s silicon photonics enables scalable qubits via high-volume manufacturing, targeting one million physical qubits for logical operations. This breakthrough could solve intractable problems in cryptography, where quantum computers threaten RSA encryption, and pharmaceuticals, simulating molecules for drug discovery at speeds 1,000 times faster than classical supercomputers . By achieving fault tolerance, PsiQuantum aims to deliver utility-scale machines for optimization in finance and logistics, reducing error rates below 10^-15 per gate. Who led PsiQuantum’s funding round? BlackRock affiliates, Temasek, and Baillie Gifford, with participation from NVIDIA Ventures and Qatar Investment Authority . What is the timeline for PsiQuantum’s quantum computers? Prototypes by 2026, full-scale sites operational by early 2030s, targeting commercial utility . PsiQuantum’s Funding Ushers in fault-tolerant quantum computers. With their 1 billion dollar check they will be setting up headquarters in Chicago and Brisbane representing a transformative leap in quantum computing. Valued at $7 billion, this initiative could unlock trillion-dollar impacts across industries, fostering global collaboration and economic growth in a $65 billion market by 2030 . As construction begins, PsiQuantum positions itself at the forefront of the quantum revolution, promising breakthroughs in computation and science. For more on quantum technology trends, sunscribe to the Objectwire.org .
The partnership between Avalanche (Avax) and Toyota Blockchain Lab (the research department of toyota) to build Mobility Orchestration Networkis. MON is reshaping urban mobility by integrating blockchain technology with autonomous vehicle systems. Multi-chain framework: The MON prototype features four custom, Avalanche-based Layer-1 blockchains, each handling specific functions: Issuance of security tokens Management of mobility rights Tracking of service usage Operations for stablecoins Why are Avalanche Blockchain and Toyota Collaborating on Atomized driving Taxis. Announced in September 2025, their collaboration introduces the Mobility Orchestration Network (MON) is a a blockchain-based infrastructure designed to power autonomous robotaxi fleets. Leveraging Avalanche’s high-throughput platform and Toyota’s expertise in autonomous driving, MON aims to create a secure, transparent, and efficient ecosystem for next-generation transportation. The Avalanche-Toyota Blockchain Lab partnership, launched in September 2025, marks a pivotal step in merging blockchain with autonomous mobility. The Mobility Orchestration Network (MON) aims to streamline robotaxi operations, from financing to ride-sharing, using Avalanche’s scalable blockchain. What is the Mobility Orchestration Network (MON)? A Blockchain-Based Mobility Solution by Toyota MON is a blockchain intermediary network designed to “orchestrate trust and unlock mobility’s value” for autonomous robotaxi fleets. Built on Avalanche’s multichain infrastructure and Interchain Messaging (ICM), it enables secure data sharing for vehicle financing, ride-sharing, insurance, and carbon credit tracking. MON’s neutral platform integrates with existing networks, supporting decentralized business models, per toyota-blockchain-lab.org . Key Applications MON supports on-chain operations like leasing, payments, and ownership transfers, creating a seamless ecosystem for robotaxi fleets. Its proof-of-concept, funded with $10.8 million, aims to standardize mobility processes, per en.cryptonomist.ch . Research Insight: MON’s prototype handles 1,000 transactions per second, per Avalanche’s technical specs. Role of Avalanche in Autonomous Robotaxi Infrastructure Why Avalanche’s Blockchain? Avalanche’s platform, known for sub-second finality and high throughput, processes thousands of transactions per second, ideal for real-time robotaxi data demands. Its multichain framework supports dedicated chains for mobility, ensuring scalability and security, per blockonomi.com . Tokenizing Mobility MON introduces VehicleOwnership tokens (ERC-721 NFTs) linked to Mobility Oriented Accounts (MOAs), enabling seamless ownership transfers and financing. This tokenization reduces administrative costs by 50%, per a 2025 Benzinga analysis, allowing investors to fund and track robotaxi fleets transparently. Decentralized Robotaxi Operations MON envisions robotaxis running on decentralized systems, with smart contracts automating ride scheduling, billing, and maintenance. This could reduce operational costs by 20%, per cryptorobotics.ai , enhancing efficiency for operators and passengers. Research Insight: Toyota tested blockchain for vehicle data tracking in 2023, per Toyota Blockchain Lab. Key Features of the Autonomous Toyota Robotaxi Infrastructure MON enables real-time Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication, optimizing routes and reducing congestion. Integration with traffic systems enhances safety. Carbon Credit Tracking MON tracks emissions on an immutable ledger, supporting carbon credit systems. This aligns with sustainability goals, appealing to regulators and eco-conscious consumers, Regulatory Alignment Aligning manufacturers and regulators for blockchain recordkeeping is critical. Roi Hirata of Ava Labs notes challenges in standardizing data formats across jurisdictions, requiring collaboration MON aims to supports smart cities with dynamic pricing and decentralized ownership, reducing car ownership reliance.