How Twitter is (probably) crawling the Internet for AI
36 points by edk-
36 points by edk-
I used to work for a big crawling company, not Oxylabs but one of their competitors. Twitter was one of their big clients scraping millions of pages for different websites each day. That is how big companies stay ahead and crawl without being known
I don't want to doxx you but, unless the market drastically changed, there is only one big competitor of Oxylabs. Everything else is just a rounding error AFAIK. This competitor in question changed their name a few years ago, their old name used to be a pun on the Illuminati.
I used to use to work for a company that used that big competitor when it was still called that. I'm not proud of it now, but my employer was scraping Facebook/Instagram to do analytics on Influencer marketing for brands, at that time Musically/Tiktok were not as big as right now.
Oxylabs was just founded at that time. They had a pretty aggressive sales team, we ultimately switched to Oxylabs because they were an order of magnitude cheaper.
My former employer went bust a few years ago, so I can say whatever I want now :)