Serious question/project:
What would it take for us to train our own GPT-3 sized model?
"Open"AI will always attempt to monopolize GPT-3 and beyond, Bloom sucks ass, no one else is trying to train an open source LLM. Why can't we bright minds here train one?
It's the same architecture of GPT-2 scaled up, very few changes. That part should be easy. The datasets are also opensource afaik. The only really hard part is infrastructure and distributed training. I'm not sure the code for that is openly available, but I'm also not sure you can't just run it on some cloud and say fuck it, which gives us 3 options:
1 - Figure out a way to pay for cloud infrastructure for this shit (not ideal, expensive as fuck)
2 - Figure out a way to use our own infrastructure and do distributed training even if it takes very long (only have to train the model once anyway). Do it like those astronomy/crypto mining/whatever projects do and let millions of people lend a tiny bit of compute to the cause. I'm sure a ton of people would be up for creating an actual Open GPT-3 level model.
3 - One of you knows or is a cluster guy and can realistically just train the model there, even if it takes a long time.
Again the key thing here is the model only has to be trained once, and we'd forever and ever have a fucking high level open AI for anyone that can actually run it (that's another problem, but the model would be open).
An actual open GPT-3 level model would change world history forever, for the better.
no one here is skilled enough to do anything beyond rendering swastikas on screen or chud tinker toy programs. especially you since you wrote this shitty thread.
Instead of being an idea guy why not develop skills?
why would u develop skills when there's already a robot that's smart enough to do any job i could do better than me and the only reason it doesn't is because the californians are censoring it
Holy mother of retard!
minus breakthrough in distributed training, the most realistic way is probably for each person to train with as much as they can scraping together free credits with cloud providers, then pass the checkpoint and trainer state to the next person
we will get to gpt3 eventually
highly autistic but guaranteed to work
Assemble a team of skilled data scientists, AI/ML people, senior infrastructure devs and academics, get endorsement from universities and private investors, create a Kickstarter to show big investors there is an interest in an open source GPT model etc
Be the change you want to see
But considering what Stability AI did with Stable Diffusion, the model would most likely get censored by pressure of said investors and academics.
Still, better getting a censored open source model than nothing. People could at least have a base model to train a new model from.
There are projects like that already. Look into GPT-J/X/NeoX
Several have already tried this, for example BLOOM. Turns out it's more difficult than people thought. Eleuther, Carper etc. have several open source projects running right now to make better ones. They have much better infrastructure and smarter people than any BOT project could ever achieve.
Even more relevant is GPT-JT since it used decentralized training. The big problem again is that none of them come close to recent OpenAI models. They brag about good classification scores and other irrelevant metrics when the skills that actually matter are shit.
>There are projects like that already. Look into GPT-J/X/NeoX
Exactly this, I run GPT-J at home. You can fine-tune for whatever, some dude used it to make a haiku generator. It's not 175B parameter, but you can get within 2% with 6B, that's still "good enough" value.
To the general point, if you want large scale collective GPU model training and execution, one of you assholes needs to make a cryptocoin where "mining" isn't running some lame SHA256 but rather doing model shite. That's the big-ticket answer, at scale it would crush openAI in computational power.
There was a project that could do that, golem I believe was a name. Their poc was rendering for graphics etc. Other one was iocexec something like that and it's still active I believe. Both were designed to reward for processing power which end user would use for whatever.
PoT = Proof of Train. You could definitely sell an ICO on this concept. And cryptards have shown you never actually need to develop that into a working system.
Get ready to pay AWS/GCS/Azure a fuck ton of money to house that data. Unless you're building your own data lakes. Oh, and also have your own website full of code to train on, like trillions of lines. Oh, and also know how to code at low and high levels and live off of donations or whatever, however you decide to fund this project. oh and...
It's obvious you don't understand what you're talking about.
Retard, I responded to several lines. He even says "Has to figure out a way to pay for cloud blah blah blah". How about you figure out what the fuck you're talking about retard?
We could to crowd-fund this, but people are already working on what you speak of
>ChatGPT is smarter than most people ITT
grim
it's repeating what everyone said in the thread ESL-san.
Open AI has some of the most intelligent people on the planet working for them. They won't hire anyone without a PhD or at least a masters and significant research experience. How the fuck do you think the smoothbrains that visit this board will be able to make anything comparable.
lol he thinks having a degree makes you smart or talented
Can you feed it input from 4chan, stormfront, vnn etc so it normies lose thir fucking heads again like it happened with taytay