>Number 1 limiting factor for local AI is available VRAM. >Biggest consumer cards are only 24GB and cost $2k

>Number 1 limiting factor for local AI is available VRAM
>Biggest consumer cards are only 24GB and cost $2k

The market niche is there. Why has there not been a single company that has just stapled 64GB of VRAM to a mid-tier card and started selling it as a consumer AI accelerator??

Nothing Ever Happens Shirt $21.68

UFOs Are A Psyop Shirt $21.68

Nothing Ever Happens Shirt $21.68

  1. 4 weeks ago
    Anonymous

    >64GB
    Buswidth etc doesn't allow it. Slap 48 or 80GB.

    • 4 weeks ago
      Anonymous

      https://i.imgur.com/XqhVnFS.png

      >Number 1 limiting factor for local AI is available VRAM
      >Biggest consumer cards are only 24GB and cost $2k

      The market niche is there. Why has there not been a single company that has just stapled 64GB of VRAM to a mid-tier card and started selling it as a consumer AI accelerator??

      Because Nvidia decides how much RAM there is, not MSI or PNY or Gigabyte or Asus etc. and Nvidia won't let you get big VRAM for cheap.
      VRAM is cheap to them though, as proven with the story of the 12GB RTX 3060.

      how fast the RAM has to be?

      did you know that in 2013 it was possible to use GPUs that have no RAM at all? instead they would take CPU RAM and use that, of course such GPUs werent meant for heavy duty 3D gaming but they were fine for all other uses including streaming video content

      3D games worked but not at high FPS

      DDR4 RAM for CPU is cheap
      imagine if we could simply give 54GB of the CPUs RAM into GPU usage while GPU has only few gigabytes on its own and then the GPU uses its 512 GPU cores to do the AI calculation while relying on CPU RAM mostly?

      speed of PCI-e links which transfer the different types of RAM from one place to another is quite formidable

      I would say it would be worth a try

  2. 4 weeks ago
    Anonymous

    Because Nvidia decides how much RAM there is, not MSI or PNY or Gigabyte or Asus etc. and Nvidia won't let you get big VRAM for cheap.
    VRAM is cheap to them though, as proven with the story of the 12GB RTX 3060.

    • 4 weeks ago
      Anonymous

      > the story of the 12GB RTX 3060.
      tell us the story, anon

    • 4 weeks ago
      Anonymous

      not AMD or Intel?

  3. 4 weeks ago
    Anonymous

    Because two companies own 99.99% of the market and set whatever price they want

    • 4 weeks ago
      Anonymous

      >Two companies
      Just one. Amd has zero real market share

      • 4 weeks ago
        Anonymous

        Then why doesn't AMD take this opportunity to sell GPUs with a ton of VRAM?
        Wouldn't they essentially corner the local AI market?

        • 4 weeks ago
          Anonymous

          Because AMD's technology is strictly worse, and AMD support for AI is nonexistent. Their GPU division is simply not AMD's primary concern

          • 4 weeks ago
            Anonymous

            Is it feasible for them to make improvements to their tech so that it does support AI?
            Even if it's worse, a massive amount of VRAM could still make it an appealing option, no?

            • 4 weeks ago
              Anonymous

              It's mostly a drivers problem and a community problem. Most AI shit runs on CUDA, which is propertiary Nvidia and AMD has nothing like it. Also most of the people who make AI stuff are basically tech illiterate and can't really make their shit properly utilize your hardware
              AMD could try to release an equivalent product and get people to use it but as I said, they don't seem to be interested in competing with Nvidia in general, they'd rather invest their budget into competing with Intel on the CPU front.

              • 4 weeks ago
                Anonymous

                At best amd sat around and did nothing for over a decade while Nvidia worked on cuda, at worst they actively sabotaged Devs who wanted to use their GPUs for compute
                Amd makes graphics cards for children's videogames

                Could they hypothetically make their own version of CUDA that makes current AIs work on their future cards?
                Or is CUDA-compatibility completely unattainable for them?

                I've seen Silicon Macs run various models too. Are those the same CUDA-compatible models, or are they made specifically for Apple's SoCs?

              • 4 weeks ago
                Anonymous

                They tried and failed. There is zluda, developed by idk who, funded for a while by AMD. They stopped funding it. It's shit from what I hear.

              • 4 weeks ago
                Anonymous

                I see.
                Thanks for the elucidations.

        • 4 weeks ago
          Anonymous

          At best amd sat around and did nothing for over a decade while Nvidia worked on cuda, at worst they actively sabotaged Devs who wanted to use their GPUs for compute
          Amd makes graphics cards for children's videogames

  4. 4 weeks ago
    Anonymous

    Just make a VRAM swap partition. You can even put it on a ramdisk to make it faster.

  5. 4 weeks ago
    Anonymous

    Yeah bro why isn't Nvidia undercutting Nvidia?

  6. 4 weeks ago
    Anonymous

    Chinks are already doing that

  7. 4 weeks ago
    Anonymous

    Nvidia strictly controls what third parties are allowed to do with their cards and won't let anyone put more than 24GB on a 4090. AMD could do it but AI is all optimised for Nvidia so they wouldn't get good performance either way

  8. 4 weeks ago
    Anonymous

    Because an H100 costs them like $2300 to make and they sell it for $30-40K, while an RTX 4080 costs them about $300 to produce and they sell it for $800-900. Which one makes them more profit?

  9. 4 weeks ago
    Anonymous

    Apple markets the M series chips on their Macbook Pros as having 128GB of VRAM due to it being "unified memory".

  10. 4 weeks ago
    Anonymous

    How much VRAM do i need to run 70B LLMs locally at around 6bpw?
    Best card/setup for that?

  11. 4 weeks ago
    Anonymous

    https://wow.groq.com/groqcard-accelerator/
    https://wow.groq.com/groqnode-server/
    Something like these?
    Don't know anything about price/performance though (or how to get them).

Your email address will not be published. Required fields are marked *