Do you want AI bots crawling through your Substack posts to eat up all your precious writing? Are you happy to let them repurpose your words as they please? Well, it has already started.
There is now a setting to block the AI bots from your Substack Pages. Go to your Dashboard, choose Settings and scroll down to publication details to find the option.
Thanks for the tag. I appreciate your explanation here. AI a good reason to go paid, which will probably be the only way to find quality material on the web moving forward. So much for a democratized internet. I do wish there were more regulations on AI. Its destructive potential goes further than its convenience. I'm on a panel about this topic in a few days, and I'll properly cite this article.
Hey Boodsy, which is so totally different from the "Woodsy" autocorrect wants to call you LOL...
I just saw somewhere on here that you had some questions for me and now I cannot find whence that communication came. But let's do chat. It's so fun to see what you're exploring here. I'm learning a lot. vandayoga @ hotmail.com is my email. Not sure if putting spaces where I did on either side of the "@" will stop robots from finding me, which is why I need you. You know shit. I open the hood of that car and just see dirty pieces of metal and tubes. You have distinctions.
Take it for what it’s worth, but I just wrote a small piece about AI from the Christian perspective. I know that’s broad so maybe just from my Christian perspective. Here’s a link if you’re interested.
If you don’t want to read the whole thing, I basically argue that directly participating in the use of AI goes against working *with God’s creation and indicates we (humans) prefer our own creation, a.k.a. lowercase gods.
But I also acknowledge that to live in the world now means that we will have to live with the fact that AI will be used around us and everyone needs to consider what that will look like and lines that may or may not need to be drawn.
Thank you for this article. I reached out to Substack a few months ago to ask about this very thing but never received an answer. It's alarming that they have a way to stop AI bots from crawling, yet they don't do it. On Fiction Attic, we use paywalls to protect original fiction. Can AI bots get past the paywall?
The ability to stop AI bots is only now being worked out by AI companies. I have also reached out to Substack and will in my next article give an update.
Thanks for tagging me so that I found this article. Very interesting stuff. In a previous life I worked for Microsoft UK as a communications manager so know that they pretty much do what they want regardless! As a writer of fiction too, my novels have been made available on piracy sites for people to download for free. It often seems to me there is no way to stop the stealing of our words.
Echoing Vanda here. I, too, saw a message you sent me but now can’t figure out where it is. And I, too, would love some help understanding stuff underneath a hood I have given little consideration. Reach out by email if you like. Mine’s my name (hollystarley) and then @yahoo.com.
It’s in the Publication Details section of Settings and seems to only be about paywalled content....
This is what it says:
Allow Search Engines to Index Paywalled Content:
Allow Search Engines to see paywalled content, so the full text of your paywalled posts can match search queries. Non-subscribers may see excerpts of paywalled content on the search engine, but they will be shown the paywall when they view the full post. This can increase search traffic to your publication.
If you get new readers from Google Search I would probably not uncheck this. All your free pages by defualt are indexable by Search engines. That is a good thing generally, it makes your work more discoverable.
It's a different class of bots crawling your pages for the AI models. I've not heard anything certain yet, but whispers indicate that new settings or policies are on their way to deter them. Which was the goal of my post really. Most writers seem to say NO to AI, but a few surprisingly don't care.
You writing is so uniquely personal to you, it's as much a privacy question as a copyright one. Would you want an AI to be able to reveal that stuff about you freely? You write for humans not for machines.
Exactly! It strikes me that there should be a “middle ground” of protected content. Paywall (private); free on Substack (protected from crawlers somehow??); the general internet shit pile (fair game)....?
Basically, we (creators and publishers) need to decide: Do we want search engines and AI models to use our work? Or should we block them and keep our content just for our real readers?
Well that setting not to index by search engines only applies to paid posts. You just have to do a search on Google 'lex contrera substack' to see your pages. I found you on Substack itself.
Well, big companies like Google have been scanning physical books for years. We are truly engulfed by big tech. I understand the difficulty of finding an audience. One of the reasons I set up my Substack was to explore exactly what and if Substack can help in that process. For a writer starting out, it seems not much at all. In a world where everything is a commodity we need to set up shop, market, promote and sell every word. For most it is difficult to start out as the app pushes popular writing to the fore leaving the more interesting stuff often hiding in the shadows. But even those shadows are not safe from big tech and their bots!
There is now a setting to block the AI bots from your Substack Pages. Go to your Dashboard, choose Settings and scroll down to publication details to find the option.
The only thing I see related to AI is that you can set it to block AI training on your content. Nothing related to subscribers. What am I missing?
That's the one. They should skip your pages.
Cool. Thanks!
Thanks for the tag. I appreciate your explanation here. AI a good reason to go paid, which will probably be the only way to find quality material on the web moving forward. So much for a democratized internet. I do wish there were more regulations on AI. Its destructive potential goes further than its convenience. I'm on a panel about this topic in a few days, and I'll properly cite this article.
Não quero robozinho plageando meus escriros.
Hey Boodsy, which is so totally different from the "Woodsy" autocorrect wants to call you LOL...
I just saw somewhere on here that you had some questions for me and now I cannot find whence that communication came. But let's do chat. It's so fun to see what you're exploring here. I'm learning a lot. vandayoga @ hotmail.com is my email. Not sure if putting spaces where I did on either side of the "@" will stop robots from finding me, which is why I need you. You know shit. I open the hood of that car and just see dirty pieces of metal and tubes. You have distinctions.
Take it for what it’s worth, but I just wrote a small piece about AI from the Christian perspective. I know that’s broad so maybe just from my Christian perspective. Here’s a link if you’re interested.
https://open.substack.com/pub/derekjpetty/p/thinking-about-ai?r=5z5dg&utm_campaign=post&utm_medium=web
If you don’t want to read the whole thing, I basically argue that directly participating in the use of AI goes against working *with God’s creation and indicates we (humans) prefer our own creation, a.k.a. lowercase gods.
But I also acknowledge that to live in the world now means that we will have to live with the fact that AI will be used around us and everyone needs to consider what that will look like and lines that may or may not need to be drawn.
Thank you for this article. I reached out to Substack a few months ago to ask about this very thing but never received an answer. It's alarming that they have a way to stop AI bots from crawling, yet they don't do it. On Fiction Attic, we use paywalls to protect original fiction. Can AI bots get past the paywall?
The ability to stop AI bots is only now being worked out by AI companies. I have also reached out to Substack and will in my next article give an update.
Thanks for tagging me so that I found this article. Very interesting stuff. In a previous life I worked for Microsoft UK as a communications manager so know that they pretty much do what they want regardless! As a writer of fiction too, my novels have been made available on piracy sites for people to download for free. It often seems to me there is no way to stop the stealing of our words.
Hi Boodsy, thanks for tagging me. I was completely unaware of this issue, and most definitely NO; I do not want AI bots crawling over my writing.
Echoing Vanda here. I, too, saw a message you sent me but now can’t figure out where it is. And I, too, would love some help understanding stuff underneath a hood I have given little consideration. Reach out by email if you like. Mine’s my name (hollystarley) and then @yahoo.com.
And thanks for the kind words about my writing. 😊
And welcome to Substack!
Thanks Holly
There is something in Settings that lets you not be indexed by search engines 🤔 does unchecking this protect against the crawly bots?
Hey Jill, I can't find the settings you mentioned. But the answer would be no for the AI bots.
It’s in the Publication Details section of Settings and seems to only be about paywalled content....
This is what it says:
Allow Search Engines to Index Paywalled Content:
Allow Search Engines to see paywalled content, so the full text of your paywalled posts can match search queries. Non-subscribers may see excerpts of paywalled content on the search engine, but they will be shown the paywall when they view the full post. This can increase search traffic to your publication.
If you get new readers from Google Search I would probably not uncheck this. All your free pages by defualt are indexable by Search engines. That is a good thing generally, it makes your work more discoverable.
It's a different class of bots crawling your pages for the AI models. I've not heard anything certain yet, but whispers indicate that new settings or policies are on their way to deter them. Which was the goal of my post really. Most writers seem to say NO to AI, but a few surprisingly don't care.
You writing is so uniquely personal to you, it's as much a privacy question as a copyright one. Would you want an AI to be able to reveal that stuff about you freely? You write for humans not for machines.
Exactly! It strikes me that there should be a “middle ground” of protected content. Paywall (private); free on Substack (protected from crawlers somehow??); the general internet shit pile (fair game)....?
This is why using paywalls on Substack could be the solution.
I talk more about this issue in my two posts below.
The Rise of Search Engine Parasites:
https://bernhardrieder.substack.com/p/the-rise-of-search-engine-parasites
Block Search Engines Now – Take Control:
https://bernhardrieder.substack.com/p/block-search-engines-now-take-control
Basically, we (creators and publishers) need to decide: Do we want search engines and AI models to use our work? Or should we block them and keep our content just for our real readers?
I'm curious about all your thoughts and ideas!
To every cow it's calf.
Interesting thoughts in the context of the latest NY Times lawsuit against OpenAI https://www.zenontech.co/p/ny-times-vs-openai-court-case-who
Hey Lexi
Well that setting not to index by search engines only applies to paid posts. You just have to do a search on Google 'lex contrera substack' to see your pages. I found you on Substack itself.
Well, big companies like Google have been scanning physical books for years. We are truly engulfed by big tech. I understand the difficulty of finding an audience. One of the reasons I set up my Substack was to explore exactly what and if Substack can help in that process. For a writer starting out, it seems not much at all. In a world where everything is a commodity we need to set up shop, market, promote and sell every word. For most it is difficult to start out as the app pushes popular writing to the fore leaving the more interesting stuff often hiding in the shadows. But even those shadows are not safe from big tech and their bots!