A couple years or so ago, I was minding my business scrolling through Quora when I saw a spam post advertising a fake designer handbag site. I did what I normally do, reported it for spam and moved on.
Quora’s feed is shaped by engagement; what you respond to, you see more of. Rather perversely, Quora considers reporting content to be “engagement,” meaning if you report romance scammers you see more romance scammers, if you report trolls you see more trolls, and in this case, if you report more designer handbag spammers you see more designer handbag spammers. So I started seeing more of them. And more. And more after that.
I reported them, and saw more, and reported them, and saw more. “Huh,” I said, “that’s weird. I wonder how many there are?”
I did some digging, deliberately searching out these spam accounts, and by a rough back of the envelope calculation I estimated there were likely somewhere between 6,000 and 8,000 fake Quora profiles spamming these fake designer handbag sites.
I was so very, very wrong. There are so many more than that. The network is huge, and organized, it’s advertising a whole cluster of phony counterfeit handbag sites, and every time I revised my estimate upward I found that I was still lowballing the size of it.
It’s also not advertising to Quora users, oh no. It’s trying to trick AIs like Google Gemini into recommending the fake handbag sites.
The rest of this is a pretty deep dive into how the network works, with some thoughts on AI poisoning as a marketing strategy. Want to know more? Follow the white rabbit 🐇, and I’ll show you how deep the rabbit hole goes.
Okay, so.
As I searched out these accounts spamming fake handbag sites, I noticed something weird. Most social media spammers follow a script. They usually use automated tools to post their spam. These tools create fake social media profiles, send spam from a template, and then create more social media profiles, all automatically.
That means these social media profiles have a certain sameness to them, that makes them stand out to automated anti-spam tools. Once you figure out what to look for, you can spot them easily, so your automated antispam tools can find and delete them all at once. Then the spammers tinker with the profile creation settings or the script, they do a fresh spam run, the antispam software adapts, and round and round it goes.
These spam profiles didn’t have the feel of automated spam tools. In fact, they looked hand-made.
Spammers use shortcuts to try to game social media algorithms: they’ll create a ton of spam profiles all at once and have them follow each other, to signal the social media algorithm that these are profiles worth following. They’ll usually have generic profiles (no profile pics or AI generated profile pics, no work history, no listed interests). They look empty and dead, or if they include profile information it’s bland and generic and it all looks the same—naturally, since it was created by the same software.
This is a pretty typical spam profile on Quora. This profile spamvertises a website that claims to sell prescription drugs without a prescription. No profile photo, no profile description, no credentials, no topics, no history.
This is one of the spam profiles advertising these fake bags:
Night and day. Believable profile pic. Profile description. Credentials. List of topics of interest.
As I dug into this network, all of these spam profiles were this way—and they were all different. They used different styles of names of different ethnicities (American, European, Chinese, Vietnamese, Russian). They all listed credentials, work experience, and/or educational experience. They all listed topics of interest, some quite extensive. All of them have basic profile information filled out, all with different writing styles.
These things make them essentially invisible to moderation software. They don’t look like spam profiles. The payoff is huge: a typical spam profile lasts a couple of months on social media, the median age of the profiles I was discovering was over a year and a half, and the oldest ones I discovered were created in early 2022, more than four years ago!
In other words, these profiles employ considerable defenses against automated moderation.
The spam messages themselves are also interesting. Almost none of them list a URL. Those that do, don’t make it a link. They list the URL as “duppro” or “luxxy” or “regbags,” without the .com. The vast majority, however, only contain the link to the spam site in a photo attached to the spam post.
And again, the spam posts themselves don’t follow a template. They all feel hand-written. These two answers look nothing alike. They both feel written by different people, not a spam template.
And yes, “unidups” and “esluxy” are two sites owned by the same spammer. These spam accounts advertise a network of at least 52 sites. A single spam account will post spam for multiple sites, which is how I know the same spammer owns them all.
I guessed, based on the number of these accounts I was seeing and the number I’d already seen and some quick back-of-the-envelope Bayesian analysis, that there were probably about six or eight thousand of these accounts. I posted about it on Quora, and also emailed my contact in Quora administration “hey, just so you know, you have a problem here, there’s an organized spam ring flooding you with spam for fake handbag sites, don’t just feed these profiles into your moderation bot because they’re employing anti-modbot-defenses so that won’t work.”
Then I dusted off my hands and went on with my life.
That’s when shit got weird.
I got two messages, one from a Quora user who was like “I saw what you wrote about the handbag spam thing, that looks like AI poisoning to me” and another from a guy who claimed to be working for the people responsible, telling me essentially “yeah, you found us, just so you know, we aren’t spamming Quora, we’re tricking Gemini into recommending our sites, and also, there are way more accounts than you think there are.”
So I plunged into the sewer again.
So first, let me digress by talking about spam networks.
Social media spammers have a problem, and that’s visibility. A brand-new account with no followers and no engagement sinks like a stone. The algorithm promotes content that gets engagement, figuring that if this content gets a lot of reactions from a small group of people it’s shown to, it might do well if it’s shown to a larger group.
Spammers try to game this by creating a lot of spam profiles, which is why they use automated tools to do this. You’ll create, say, ten profiles, then you’ll have them all follow each other and upvote each other’s content, to make it look to the algorithm like hey, these profiles are getting attention, maybe they should be distributed more widely.
Say you create eight spam profiles. You make them follow each other (red lines) and upvote each other’s spam posts (blue lines). That tricks the algorithm into showing all eight accounts to more people.
The problem with this is that once someone reports one of the profiles for spam, all eight get banned, because it’s trivial to see that all these accounts are connected to each other.
That’s not what these fake handbag spammers do.
There are thousands of them, arranged in what I call “clusters.” The clusters contain some links to other spam profiles in the same cluster, but there is little or no cross-cluster linking, and they aren’t linked the way most spam profiles are. The linking might look more like this:
Some spam profiles follow other spam profiles. Some spam profiles upvote other spam profiles. But the profiles aren’t highly linked. If you ignore the green circles, you can see that there are groups that aren’t connected to each other.
So what are the green circles?
They’re fake profiles that don’t post spam. All they do is follow or upvote the spam profiles, and also follow and upvote genuine, completely real profiles. But they don’t post spam themselves. They’re designed to look “organic,” not spammy, and they behave like lurker profiles do: posting no content, but following and upvoting other people.
So how do I know they belong to the spammers?
I don’t, not 100%, but it seems likely. This spammer will often create multiple profiles at the same time, and these green profiles have the same creation dates as the profiles they follow and upvote. They do interact with genuine profiles, but most of the profiles they interact with are spam profiles, and they’re often among the first to interact with spam profiles. They pop up again and again and again when you look at spam profiles, even across different groups (in fact, they’re usually how I find disconnected spam groups). They just don’t feel like genuine profiles.
So you see a small number of spam profiles with a large number of follows and likes from other spam profiles, to game the algorithm, but most of the spam profiles aren’t followed or upvoted by anyone, to hide them from link analysis.
The fact that the groups are disconnected from each other means I can’t get a ballpark figure for how many of these spam profiles there are, because the fact that I’ve already seen 3 out of 100 of the profiles in the current group I’m mapping out doesn’t tell me how many groups there are. I thought there were six or eight thousand of these spam profiles. I now believe the number is likely closer to 20,000 to 30,000, and I suspect even that might be a lowball estimate.
The profiles that aren’t connected to each other (0 followers, 0 following) and don’t have their spam posts upvoted, don’t get a lot of attention in the feed. Very few Quora users will ever see them.
So what’s the point?
Ah, now that’s the genius. The spammers don’t care if Quora users see their spam or not. The spam isn’t aimed at Quora users.
Quora trains the major AI systems. The CEO of Quora, Adam d’Angelo, sits on the board of directors of OpenAI. Quora has inked a deal with Google to send the Quora feed directly into the ravening maw of Google Gemini, which is why the AI summary of Google searches so often mentions Quora.
Posting something on Quora dumps it straight into Google, where it immediately is used to train the Google AI. Something on Quora can start showing up in the Google AI Overview in as little as 24 hours, and (this is important)…
…the Google AI can read text that’s placed in images.
The entire history of infosec is you must never ever ever ever ever ever trust user-supplied input because there’s always the chance it can be malicious, and now, with LLMs and GenAI, we’re shoveling unsanitized user input straight into LLMs as fast as they can ingest it. Which is weird and stupid and almost certainly going to make a lot of very large companies very sorry.
The person who messaged me claiming to be one of the people behind these spam accounts showed me screenshots of Google searches that steered people to the spam sites in the AI Overview, which is weird because when I asked Google Gemini directly about these sites, Gemini said they are known counterfeit sites that should be avoided.
File that one under “things that make me go hmm.”
The Quora user I spoke to invited me to a video chat in which he demonstrated an extremely sophisticated toolchain he’s developed to make Google Gemini believe the most absolutely whack, farfetched things you can imagine, like a photo of his pantry being a rare surveillance image from the inside of the Branch Davidian cult headquarters prior to the FBI raid, just by posting things on Quora. He’s not using it for spam, he seems like he’s just exploring the tech to see what he can do, but other people absolutely are using the same techniques for nefarious purposes. (I am not revealing the identities of either of these people because I have agreed not to do so without their permission.)
For all the fact that he’s on the board of OpenAI and has gone all-in on AI, Adam seems to have missed a trick. Part of what makes the Quora moderation software blind to these spam accounts is that Quora’s modbot appears to ignore images, or at least it lacks the ability to parse text in images. It cannot “see” that images like this:
are spam. It doesn’t see the web address in the image. Gemini and other AI systems, however, do. This image trains Gemini to associate the handbag here, which Google is able to identify by make, model, and style, with the website in the image.
The spam group started in at least early 2022, before Google AI Overview and LLMs, and the earliest spam I’ve seen seems directed at trying to get Quora users to visit the spam site. But now? Quora users seem almost irrelevant.
This spam campaign is extremely labor-intensive and has been ongoing for years. I do not believe the profiles are being generated by software; I think they are being generated, and the profile information is being populated, by hand.
How do you make the profiles resistant to moderation bots?
Every profile is unique. Every profile uses a unique userpic, often stolen from websites of high-end consumer goods. Every profile has a different, but natural-seeming, profile description. Every profile has unique work experience and topics of interest. Every profile has unique education credentials.
Some of the profiles appear to be created by someone with a sense of humor. It’s obvious the people running this massive ring of profiles is aware of what they’re doing.
“Worked at Counterfeit Goods.” “Maya Angelou.” 🤣
And perhaps more important if you want to do this at scale, they do not flood. These spam profiles post one or two, or occasionally three, pieces of spam, but rarely more than that. They also post one or two non-spam posts, most often photos of cats or photos of food. They use a different credential on the non-spammy posts.
If a spam post is removed but the account isn’t banned (for example, if users report one particular spam post as spam), they’ll immediately repurpose that account, deleting the other spam posts if there are any and then using that profile to upvote spam posts by other profiles. A profile that has had a post deleted by moderation is burned, and stops posting spam, but is still used to boost other spam posts.
The profiles work together. Some profiles post spam with no links at all, not even in the photo, and then other profiles come in later and post a comment with the link.
This is a highly organized, disciplined, well-resourced spam group in it for the long haul. They have a level of labor, care, attention, and investment I rarely see in spam gangs, which suggests to me they must be seeing considerable return on that investment.
Counterfeit designer handbags are often a moneymaker for large drug cartels, and for organized crime in Asia and Russia. Between that and the level of organization and work I see here, I am reasonably confident this Quora spam group is quite likely associated with organized crime.
I have personally, as of the time of writing this, worked with my contact in Quora administration to shut down 4,298 of these accounts. (Yes, I track every single one I discover and pass along.) I believe this is likely somewhere between 10% and 20% of the number of profiles being used by these spammers.