FreeGPT

Serious question/project:

What would it take for us to train our own GPT-3 sized model?
"Open"AI will always attempt to monopolize GPT-3 and beyond, Bloom sucks ass, no one else is trying to train an open source LLM. Why can't we bright minds here train one?
It's the same architecture of GPT-2 scaled up, very few changes. That part should be easy. The datasets are also opensource afaik. The only really hard part is infrastructure and distributed training. I'm not sure the code for that is openly available, but I'm also not sure you can't just run it on some cloud and say frick it, which gives us 3 options:

1 - Figure out a way to pay for cloud infrastructure for this shit (not ideal, expensive as frick)
2 - Figure out a way to use our own infrastructure and do distributed training even if it takes very long (only have to train the model once anyway). Do it like those astronomy/crypto mining/whatever projects do and let millions of people lend a tiny bit of compute to the cause. I'm sure a ton of people would be up for creating an actual Open GPT-3 level model.
3 - One of you knows or is a cluster guy and can realistically just train the model there, even if it takes a long time.

Again the key thing here is the model only has to be trained once, and we'd forever and ever have a fricking high level open AI for anyone that can actually run it (that's another problem, but the model would be open).

An actual open GPT-3 level model would change world history forever, for the better.

ChatGPT Wizard Shirt $21.68

Beware Cat Shirt $21.68

ChatGPT Wizard Shirt $21.68

  1. 1 year ago
    Anonymous

    no one here is skilled enough to do anything beyond rendering swastikas on screen or troony tinker toy programs. especially you since you wrote this shitty thread.

    Instead of being an idea guy why not develop skills?

    • 1 year ago
      Anonymous

      why would u develop skills when there's already a robot that's smart enough to do any job i could do better than me and the only reason it doesn't is because the californians are censoring it

      • 1 year ago
        Anonymous

        Holy mother of moron!

  2. 1 year ago
    Anonymous

    minus breakthrough in distributed training, the most realistic way is probably for each person to train with as much as they can scraping together free credits with cloud providers, then pass the checkpoint and trainer state to the next person
    we will get to gpt3 eventually
    highly autistic but guaranteed to work

  3. 1 year ago
    Anonymous

    Assemble a team of skilled data scientists, AI/ML people, senior infrastructure devs and academics, get endorsement from universities and private investors, create a Kickstarter to show big investors there is an interest in an open source GPT model etc

    Be the change you want to see

    • 1 year ago
      Anonymous

      But considering what Stability AI did with Stable Diffusion, the model would most likely get censored by pressure of said investors and academics.
      Still, better getting a censored open source model than nothing. People could at least have a base model to train a new model from.

  4. 1 year ago
    Anonymous

    There are projects like that already. Look into GPT-J/X/NeoX

    • 1 year ago
      Anonymous

      Several have already tried this, for example BLOOM. Turns out it's more difficult than people thought. Eleuther, Carper etc. have several open source projects running right now to make better ones. They have much better infrastructure and smarter people than any bot project could ever achieve.

      Even more relevant is GPT-JT since it used decentralized training. The big problem again is that none of them come close to recent OpenAI models. They brag about good classification scores and other irrelevant metrics when the skills that actually matter are shit.

    • 1 year ago
      Anonymous

      >There are projects like that already. Look into GPT-J/X/NeoX
      Exactly this, I run GPT-J at home. You can fine-tune for whatever, some dude used it to make a haiku generator. It's not 175B parameter, but you can get within 2% with 6B, that's still "good enough" value.

      To the general point, if you want large scale collective GPU model training and execution, one of you buttholes needs to make a cryptocoin where "mining" isn't running some lame SHA256 but rather doing model shite. That's the big-ticket answer, at scale it would crush openAI in computational power.

      • 1 year ago
        Anonymous

        There was a project that could do that, golem I believe was a name. Their poc was rendering for graphics etc. Other one was iocexec something like that and it's still active I believe. Both were designed to reward for processing power which end user would use for whatever.

      • 1 year ago
        Anonymous

        PoT = Proof of Train. You could definitely sell an ICO on this concept. And cryptards have shown you never actually need to develop that into a working system.

  5. 1 year ago
    Anonymous

    Get ready to pay AWS/GCS/Azure a frick ton of money to house that data. Unless you're building your own data lakes. Oh, and also have your own website full of code to train on, like trillions of lines. Oh, and also know how to code at low and high levels and live off of donations or whatever, however you decide to fund this project. oh and...

    • 1 year ago
      Anonymous

      It's obvious you don't understand what you're talking about.

      • 1 year ago
        Anonymous

        moron, I responded to several lines. He even says "Has to figure out a way to pay for cloud blah blah blah". How about you figure out what the frick you're talking about moron?

  6. 1 year ago
    Anonymous

    We could to crowd-fund this, but people are already working on what you speak of

  7. 1 year ago
    Anonymous
    • 1 year ago
      Anonymous

      >ChatGPT is smarter than most people ITT
      grim

      • 1 year ago
        Anonymous

        it's repeating what everyone said in the thread ESL-san.

  8. 1 year ago
    Anonymous

    Open AI has some of the most intelligent people on the planet working for them. They won't hire anyone without a PhD or at least a masters and significant research experience. How the frick do you think the smoothbrains that visit this board will be able to make anything comparable.

    • 1 year ago
      Anonymous

      lol he thinks having a degree makes you smart or talented

  9. 1 year ago
    Anonymous

    Can you feed it input from /misc/, stormfront, vnn etc so it normies lose thir fricking heads again like it happened with taytay

Leave a Reply to Anonymous Cancel reply

Your email address will not be published. Required fields are marked *