AI crawler Firecrawl raises $14.5M, is still looking to hire agents as employees

Firecrawl’s co-founder and CEO Caleb Peffer knew the exact moment he found the investor to lead his Series A. 

He was in a coffee meeting with Nexus Venture Partner’s Abhishek Sharma at the Blue Bottle in San Francisco’s South Park (a favorite VC haunt). While describing the future of the company, he was gesturing so animatedly that his chair tipped over. 

“I actually fell out of my chair. And Abhishek, as a great Investor does, caught the chair and me as I was falling,” Peffer described with a laugh. That felt like a symbol for how the founder/investor relationship is supposed to work. “I think that was a clear signal that he was the right partner.”

On Tuesday, Firecrawl announced a Series A led by Nexus with participation from Shopify CEO Tobias Lütke and existing investor Y Combinator.

Firecrawl offers a popular open-source web crawler for developers and AI agents, with a commercially supported version available through API. It’s used by 350,000 developers, has nearly 50,000 stars on GitHub and its notable customers include Shopify, Replit, and Zapier as well as “some of the largest hedge funds in the world,” Peffer told TechCrunch. And, he says, the company is already profitable.

The startup also just released an API that supports search and will soon add support for natural language prompts, as co-founder and CTO Nicolas Silberstein Camara noted.

Peffer, who co-founded Firecrawl in 2022 with Camara and CMO Eric Ciarla, said gaining Lütke as an investor was “the best kind of validation.” 

They landed him through a gutsy email after discovering the Shopify founder had signed up to try their product through their self-service portal.

“We saw his email come in,” Peffer said. The Firecrawl crew immediately emailed him to welcome him as a customer, but got no response. Two months later, some Shopify folks contacted Firecrawl for an enterprise contract. So Peffer took his shot and emailed Lütke again, mentioning they’d love for him to participate in their upcoming round.  

This time, Lütke responded with compliments about their product. He was in.

While AI web crawlers have a somewhat dubious reputation these days, mostly due to bad actors that ignore robot.txt files, they are also a necessary part of the burgeoning AI world. AI trains on the web, agents need to access web pages to perform their duties, and enterprises need personalized crawlers to consume their own websites for training and operations.

The Firecrawl founders also hope to help address the frustrating parts of their industry. They are working on tools to help website owners, publishers, and other content creators “get paid when AI uses their content. We think this is the way it should be,” Peffer said.

While there have been lots of efforts around this idea from big names like Adobe and Getty as well as startups like Bria, Calliope Networks, and others, Peffer feels that Firecrawl has an edge because it’s already working with those who are scraping data. 

“We already have one side of the marketplace,” he said. “What we want to do is just connect that side of the marketplace to the website owners, the publishers.”

Interestingly, Firecrawl also went viral a few months ago for reasons that have little to do with their open source tool. They had posted an ad to the YC job board looking to hire an AI agent as an employee for a $15,000 salary – perhaps the first job ad for an agent employee, ever.

That job search didn’t yield an agent worth hiring, so Firecrawled upped the budget to up $1 million to hire several agents and the developers who built them, and tried again. Applicants flooded in, Peffer said, but the company hasn’t yet hired any. The founders realized that evaluating and managing wanna-be agent employees is a job in itself. So now they are looking for an AI chief of staff. 

Firecrawl’s Peffer will be on stage at Disrupt in October to discuss all that he’s learned in a session that covers the pros and cons of hiring AI agents as early employees.