Conversation

Notices

1. Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 16:14:54 CEST Evan Prodromou

   in reply to

   • tobias
   • Mirko Adam
   • Codeberg.org

   @tobias @elshid @Codeberg well, that's hardcore, but the problem is that when people can't find your information on their chosen search engine, they don't go find another search engine. They don't even know they're missing your info.

   • Michael Vogel (heluecht@pirati.ca)'s status on Monday, 23-Jun-2025 16:14:54 CEST Michael Vogel

     in reply to

     • tobias
     • Mirko Adam
     • Codeberg.org
     @tobias @elshid @Codeberg @evan Yes. But I think the LLM companies are to blame here and not the ones who run servers that are hit by a near "denial of service" level of requests who also don't follow any rules about content that should be ignored.
   • tobias (tobias@social.diekershoff.de)'s status on Monday, 23-Jun-2025 16:14:55 CEST tobias

     in reply to

     • Mirko Adam
     • Codeberg.org

     @elshid @Codeberg @evan yes, absolutely agree - as long as LLM scraper bots don't follow simple rules (like following robots.txt files) and act not like a destruction swarm on services, I'm not sorry for LLM users not finding the information.

     I've seen LLM-bot swarms hitting forges and the only way to get the forges running for the intended audience was blocking . Sad thing is the scrapers then try to circumvent the blocks aggressively, even dropping their IDs so the blocking gets more complicated again. It's an arms race, and legit users of LLMs are not in the focus of any of the racing parties.

   • Mirko Adam (elshid@social.librem.one)'s status on Monday, 23-Jun-2025 16:15:00 CEST Mirko Adam

     in reply to

     • Codeberg.org

     @evan No, the reason is simply the server load. The AI crawlers have so excessively crawled @Codeberg that their main service, to host a git server, was often very slow.

   • Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 16:15:01 CEST Evan Prodromou

     in reply to

     I understand the goal; many people don't want their code to be used by LLM code generators. But it also means that this document repository isn't visible for people who use LLMs like a search engine. Numbers vary, but afaict somewhere around 10% of people use LLMs as their primary search engine, and about 50% of people use LLMs some of the time for search.

   • Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 16:15:01 CEST Evan Prodromou

     in reply to

     I guess there's maybe some justification like, those people are bad, and they don't deserve nice things like Fediverse Enhancement Proposals? Or, maybe, we have to take a principled stand against LLMs by not providing any training data for them? Such that, perhaps, people disappointed by not having good results in LLMs will return to using traditional search engines like Google or Bing, which are more ethical because reasons.

   • Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 16:15:02 CEST Evan Prodromou

     in reply to

     I was surprised to see that it had really no visibility of the FEPs. After a while, I realized that codeberg.org, the hosting service for FEPs, has ChatGPT blocked.

     https://codeberg.org/robots.txt

   • Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 16:15:03 CEST Evan Prodromou

     in reply to

     I tried it with a document I wrote, FEP 5711. It's an enhancement proposal for ActivityPub, adding some inverse relationships for important properties.

     https://codeberg.org/fediverse/fep/src/branch/main/fep/5711/fep-5711.md

   • Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 16:15:03 CEST Evan Prodromou

     in reply to

     Anyway, I took a paragraph out of the document and asked ChatGPT to identify the URL, publisher, publication date, and title. It failed. You can see the transcript here:

     https://chatgpt.com/share/68573fa9-b340-800f-b9b4-7b74fdf0bf46

   • Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 16:15:04 CEST Evan Prodromou

     I was reading this article about LLMs making bad citations. I found it pretty interesting, so I decided to try to replicate it with ChatGPT.

     https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php