Dongdong's progress

[saurabh-khanna]

Issue tracking Dongdong's progress towards developing a general approach to detect shadowbans for public/own social media profiles/posts

[CeciliaZhu1997]

• Known shadowbanned hashtags
  https://metahashtags.com/banned-hashtags/
  https://iqhashtags.com/banned-hashtags-checker/

• Detect shadowbanned accounts

  • Test reach: if posts are appearing in hashtag searches (using a few moderately popular hashtags and avoid overly saturated ones) or the Explore page, using another account that doesn’t follow the test account.
  • Track engagement: track likes, comments, and views to see if there is a sudden drop.
  • Comments’ position: shadowbanned accounts’ comments under posts by other users are shown at the bottom of the comment section.

Here is a website that claims that they can detect shadowbanned accounts: https://www.spikerz.com/calculator-shadow-ban

[CeciliaZhu1997]

Hi Saurabh @saurabh-khanna,

I reviewed relevant GitHub projects and listed some available ones here:

Shadowban Scanner on Twitter

[1]
https://github.com/Robot-Inventor/shadowban-scanner/blob/main/README.md
https://github.com/Robot-Inventor/shadowban-scanner/blob/main/doc/en/technical-information.md
This is a Chrome Extension and it uses [React Props] on Twitter Webpage to detect if there are any sensitive values. If yes, the account/post will be regarded as shadow banned.

• Account level: TweetProps.user.possibly_sensitive on an account level. If the value is true, this tool classifies the account into "shadowbanned".
• Profile level: TweetProps.user.profile_interstitial_type：sensitive_media``offensive_profile_content = true
• Posts: sensitive_media：offensive_profile_content = true
• Whether an account is blocked in certain countries: If TweetProps.user.withheld_in_countries is not an empty array, the account is considered to be blocked in those countries.

[2]
https://github.com/Shadowban-Test/X
This program simulates users' behavior to check the visibility of a specific username in search suggestions, search results, and replies, to see if a shadowban in in effect.

Shadow Ban on Reddit

[1]
https://github.com/johannessms/reddit-shadowban-tester/tree/main
This program uses Reddit’s API to determine whether a username is accessible via Reddit’s servers. A False result means the account is either non-existent, or technically exists, but shadow banned.

[2]
https://cable.ayra.ch/reddit/api.php
This is a paid API to check if a Reddit account is restricted to ages 18+/shadowbanned/overloaded etc.

For the next step, I planned to access some research papers to see if there are anything helpful. Let me know if you have any suggestions or questions!
Thank you!

[saurabh-khanna]

hi @CeciliaZhu1997! do let me know if we should speak briefly in coming week with your updates?

[CeciliaZhu1997]

Hi @saurabh-khanna !

Sure, this week I mainly focus on research papers, and most research used qualitative methods to identify the shadow ban. There are not many using quantitative and computational methods. Here are a few papers and takeaways that I think might be helpful:

[1]
Merrer, E. L., Morgan, B., & Trédan, G. (2020). Setting the Record Straighter on Shadow Banning. https://doi.org/10.48550/arxiv.2012.05101

• Extensive data collection and crawl more than 2.5 million profiles: random population, identified bots, celebrities, political representatives in France;
• Use shadown.eu (https://github.com/shadowban-eu) to check for shadowbans: suggestion ban, search ban, ghost ban
• Findings:
  • In such a black-box observation setup, we highlight the salient user profile features that may explain a banning practice (using machine learning predictors).
  • Users often experience a progression: from suggestion ban → search ban → ghost ban.
  • H1: limitations are bugs, as claimed by Twitter
    • result: bans often cluster in social circles
  • H2: shadow banning propagates as an epidemic on user interaction ego-graphs
    • result: The likelihood of being shadow banned increases if one’s connections are already banned.

[2]
Jaidka, K., Mukerjee, S., & Lelkes, Y. (2023). Silenced on social media: the gatekeeping functions of shadowbans in the American Twitterverse. Journal of Communication, 73(2), 163–178. https://doi.org/10.1093/joc/jqac050

Methods

• Collected a stratified sample of U.S. Twitter users based on geolocated tweets from the 1% random sample of the Twitter firehose.
• Then, using the Twitter API and the Botometer API (Yang et al., 2020), we constructed a set of profile features expected to predict the likelihood of shadowbans, such as their verified status on Twitter, the age of their account, and their bot-like behavior.
• We ran periodic checks on whether or not each account was shadowbanned through June–July 2020 and subsequently ran a check in July 2021.
• Next, we queried the Twitter Academic API to collect the tweets by each account in the five time “windows” leading up to each shadowban check. We constructed many features that characterized the account profiles and the content of their tweets using computational linguistic methods and probabilistic topic modeling (Blei, 2012).
• Next, we considered the static and dynamic signals of social influence for each account in terms of their number of social media followers and the number of accounts they follow in turn. Finally, we considered the engagement received by their tweets in terms of the average likes, retweets, replies, and quote tweets gained by their tweets in each tweeting window.
• Regressing these profile, content, and social features against the prevalence of shadowbans helps identify their predictive effects on whether the account gets shadowbanned or not, thereby allowing us to “reverse engineer” Twitter’s algorithm.

Results:

• (a) temporal instability in shadowbans against tweets on political or social issues: algorithms are fluid. As trends in user behavior emerge and ebb, algorithms appear to react to them and revise the platform’s content norms;
• (b) systematic biases by shadowbans against new or low-social influence users.

[3]
Chen, Y.-S., & Zaman, T. (2024). Shaping opinions in social networks with shadow banning. PloS One, 19(3), Article e0299977. https://doi.org/10.1371/journal.pone.0299977

The study treats shadow banning as an optimization problem to influence either mean opinions, polarization, consensus. It does not detect shadow ban, but use models to calculate the impact of shadowban in shaping opinion dynamics. The method is not directly relevant, but might be inspiring:

• Users are nodes, and directional edges represent follower relationships or information flow;
• Opinions evolve through weighted averaging across neighbors;
• A linear programming approach is used to choose which edges to shadow-ban to meet specific opinion manipulation goals;
• Tested the influence on path networks, Stochastic Block Models and Real Twitter subnetworks.

[CeciliaZhu1997]

Let me know if you have time to discuss it this week or our next steps! Thank you!

[saurabh-khanna]

@CeciliaZhu1997 thanks, i have a window to jump on Teams between 3:30-4:30pm today - would that work?

[CeciliaZhu1997]

Hi @saurabh-khanna

Here is the Jupyter notebook to reproduce code from previous studies and projects: https://github.com/theinvisiblelab/shadowbans/blob/main/code/0630_Test.ipynb.
One notebook is based on your Instagram test with minor adjustments, another is for Reddit, and there’s one for X to check if replies to a post were shadowbanned. However, when I tried the last one with my own account, it flagged my account for safety risks and blocked me. I’ll try again later and also work on adapting another project from Java to Python. Thank you and see you soon!

[CeciliaZhu1997]

Hi Saurabh @saurabh-khanna ,

Just a quick update on this week’s progress:

I gave Marimo a try, but I am not super comfortable with its UI yet, and I ran into some issues running the code. So, I’ve gone back to using Jupyter Notebook for now.

For the shadowban project, I focused mainly on Twitter and tried replicating the shadowban-eu project. It’s a bit outdated, but I think the detection approach still makes sense.

Here is how it works:

1. Checks the profile to see if the account exists, is suspended, protected, or restricted.
2. Searches tweets with from:@username using Twitter’s search API. If nothing shows up, it could mean a search ban.
3. Looks for the user in Twitter’s autocomplete (typeahead) suggestions.
4. Checks if replies to the user’s tweets actually appear. If replies are missing, it might be a ghost ban.

Steps 1–4 don’t require logging in. But step 5 does:

5. Checks if the user’s replies show up under the original tweets using a logged-in reference account.

Here is the code link: https://github.com/theinvisiblelab/shadowbans/blob/main/code/0710_Test.ipynb

For Instagram, I still need to log in to automate checks, and I got a final warning about banning my account, so I’ll need to figure out another approach. But there is very little public code available for IG, so please let me know if you have any suggestions.

Thank you and looking forward to meeting you tomorrow!

[CeciliaZhu1997]

Dear Saurabh @saurabh-khanna,

I am writing to follow up on our last meeting. As discussed, I've created a markdown file to document the code I've reproduced so far: https://github.com/theinvisiblelab/shadowbans/blob/main/code/Code%20Organization.md. I hope this helps!

I have to admit that the EU Shadowban project we discussed last week has been a bit challenging for me. The code and its approach to detecting shadowbans seem solid, but I’m still not quite sure how to adapt it to get it working properly.

Anyway, let me know if you have any ideas or suggestions for next steps, or if you’d prefer a better way to organize the code. Thank you!

Best,
Dongdong (from France :)

[CeciliaZhu1997]

Hi Saurabh @saurabh-khanna, I’ve developed a basic method framework for detecting whether a public account might be shadowbanned. Here is what I’m thinking:

Step 1: Account status check

• if the account is suspended, private/protected or restricted;
• if the accounts are accessible via the platform’s API, using username or user ID;
• if the account appears in the platform’s autocomplete/typeahead search;

Step 2: Posts visibility check

• first, check if the account has public posts or replies, if not, exit early;
• if yes, record their post IDs;
• use the platform’s search API to look up those post IDs or known content and confirm whether the posts are searchable and visible to the public;

Step 3: Reply visibility check

• check the users’ replies to other posts appear in the platform’s reply API;
• check if replies are being hidden under "more replies" or similar collapsed sections, which may indicate reply suppression;

Step 4: Engagement check

• check metrics like impressions, likes, replies and reposts for recent posts;
• compare those numbers against the account’s own historical engagement/similar public accounts with comparable follower counts;

My thinking is: if an account triggers at least two red flags in Steps 1–3 (for example, not showing in typeahead and all replies being hidden), it may indicate a higher risk of being shadowbanned.

As for Step 4, engagement patterns could help validate these findings. For instance, if an account consistently shows 50% lower engagement than expected, that might also be a strong signal. However, that approach would require a large and reliable comparison dataset. But comparing the account’s recent data with its historical one would be more feasible.

Let me know what you think, or if there is anything I should improve!

[CeciliaZhu1997]

Hi Saurabh @saurabh-khanna ,

I was testing SNScrape, but since I’m currently back in China, I need to connect to UvA’s VPN to access X. This might be causing an issue, as the VPN seems to have some certificate problems when retrieving content from X.

I tried bypassing the certificate check but wasn't successful. I was wondering if you might have a minute to help me test whether SNScrape itself is working on your end? Below is a simple script, it’s just to confirm whether the issue is with my network setup or with SNScrape itself.
https://github.com/theinvisiblelab/shadowbans/blob/main/code/SNScrape.ipynb

If you are unavailable, no worries at all, I will just ask a friend instead. Sorry for the trouble, and enjoy your family trip!

Best regards,
Dongdong

[saurabh-khanna]

@CeciliaZhu1997 this is the error i get, can we confirm if someone had similar issues in the package's github issues?

Error retrieving https://twitter.com/StateDept: SSLError(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: [/StateDept](https://file+.vscode-resource.vscode-cdn.net/StateDept) (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)')))"))
4 requests to https://twitter.com/StateDept failed, giving up.
Errors: SSLError(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: [/StateDept](https://file+.vscode-resource.vscode-cdn.net/StateDept) (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)')))")), SSLError(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: [/StateDept](https://file+.vscode-resource.vscode-cdn.net/StateDept) (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)')))")), SSLError(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: [/StateDept](https://file+.vscode-resource.vscode-cdn.net/StateDept) (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)')))")), SSLError(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: [/StateDept](https://file+.vscode-resource.vscode-cdn.net/StateDept) (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)')))"))

---------------------------------------------------------------------------
ScraperException                          Traceback (most recent call last)
Cell In[1], [line 9](vscode-notebook-cell:?execution_count=1&line=9)
      5 max_tweets = 100
      7 tweets = []
----> [9](vscode-notebook-cell:?execution_count=1&line=9) for i, tweet in enumerate(sntwitter.TwitterUserScraper(username).get_items()):
     10     if i >= max_tweets:
     11         break

File ~/miniforge3/envs/invisible-languages/lib/python3.10/site-packages/snscrape/modules/twitter.py:1824, in TwitterUserScraper.get_items(self)
   1822 	self._isUserId = False
   1823 	self._query = f'from:{self._user}'
-> [1824](https://file+.vscode-resource.vscode-cdn.net/home/saurabh/ownCloud/GitHub/invisible-languages/code/~/miniforge3/envs/invisible-languages/lib/python3.10/site-packages/snscrape/modules/twitter.py:1824) yield from super().get_items()

File ~/miniforge3/envs/invisible-languages/lib/python3.10/site-packages/snscrape/modules/twitter.py:1763, in TwitterSearchScraper.get_items(self)
   1760 params = {'variables': variables, 'features': features}
   1761 paginationParams = {'variables': paginationVariables, 'features': features}
-> [1763](https://file+.vscode-resource.vscode-cdn.net/home/saurabh/ownCloud/GitHub/invisible-languages/code/~/miniforge3/envs/invisible-languages/lib/python3.10/site-packages/snscrape/modules/twitter.py:1763) for obj in self._iter_api_data('https://twitter.com/i/api/graphql/7jT5GT59P8IFjgxwqnEdQw/SearchTimeline', _TwitterAPIType.GRAPHQL, params, paginationParams, cursor = self._cursor, instructionsPath = ['data', 'search_by_raw_query', 'search_timeline', 'timeline', 'instructions']):
   1764 	yield from self._graphql_timeline_instructions_to_tweets(obj['data']['search_by_raw_query']['search_timeline']['timeline']['instructions'])

File ~/miniforge3/envs/invisible-languages/lib/python3.10/site-packages/snscrape/modules/twitter.py:915, in _TwitterAPIScraper._iter_api_data(self, endpoint, apiType, params, paginationParams, cursor, direction, instructionsPath)
    913 while True:
    914 	_logger.info(f'Retrieving scroll page {cursor}')
--> [915](https://file+.vscode-resource.vscode-cdn.net/home/saurabh/ownCloud/GitHub/invisible-languages/code/~/miniforge3/envs/invisible-languages/lib/python3.10/site-packages/snscrape/modules/twitter.py:915) 	obj = self._get_api_data(endpoint, apiType, reqParams, instructionsPath = instructionsPath)
    916 	yield obj
...
    270 	_logger.fatal(f'Errors: {", ".join(errors)}')
--> [271](https://file+.vscode-resource.vscode-cdn.net/home/saurabh/ownCloud/GitHub/invisible-languages/code/~/miniforge3/envs/invisible-languages/lib/python3.10/site-packages/snscrape/base.py:271) 	raise ScraperException(msg)
    272 raise RuntimeError('Reached unreachable code')

ScraperException: 4 requests to https://twitter.com/StateDept failed, giving up.

[CeciliaZhu1997]

Hi Saurabh @saurabh-khanna ,

Hope you’re enjoying time with your family!

I am just writing to provide a quick update. Over the past few days, I tried a few third-party scraping tools for X and IG without logging in, mainly to see if they could pull profile and post data so we can check account status and post visibility. Some worked, some didn’t. Here are the main takeaways:

(1) Instagram: The big issue is the limit on scraping without login and frequent IP bans from Instagram. Some tools can grab a profile on the first try, but by the second run, they return 401 or 403 Forbidden errors. (Sometimes it works again after I use a new IP address)

(2) X: A lot of packages stopped being maintained since Elon Musk took over, as X now isn’t very developer-friendly. Snscrape (which I tested last week) keeps crashing when scraping X data, and the developers have said they won’t fix it on the Issues Page. I managed to get a few functions working with Twikit, though it is not super stable.

The tools and results are as follows:

SNScrape

• X: Failed.
• IG: Returned a 429 Too Many Requests error, even when scraping just one profile.

Scrape-up

• Replicated the original code, which uses Selenium to scrape IG data. Always returned a login request.

Scrapefly

• Works only for IG. Can retrieve basic profile data (followers, following, post count) but no detailed post data.

Instaloader

• Also IG-focused.
• Failed to retrieve profile or post data due to 403 and 401 errors.
• However, it can check an account’s status (public, private, or nonexistent).

Twikit

• Can scrape users’ recent posts and engagement data.
• When checking account status, it worked yesterday but failed today with a 404 error (using the exact same code 😐.

Here is the link to the code: https://github.com/theinvisiblelab/shadowbans/blob/main/code/Scraping%20Tools%20Test.ipynb

For the next step, I’ll dig deeper into Scrapefly, Instaloader, and Twikit to see if we can get more functionality out of them. Please also let me know if you have any suggestions. Thanks!