Does anyone run their own AI service?

If so,

Which one?

What hardware?

Do you have a way to access it from outside or from a phone?

And, if you’re open to sharing, what do you do with it?

I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.

I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.

  • ProperlyProperTea@lemmy.mlEnglish
    3·
    37 minutes ago

    Last night I managed to run Qwen Flash Next iq2 XS using Strata in docker on a VM

    I passed through my 3090, 64GB of DDR4, 12 vcores of my 5960x,

    Managed 90t/s

    I’ll probably try a larger version soon but QFN already seems a lot more capable than 3.6 35b A3B, and runs better than Qwen 3.8 dense in llama.cpp

  • LiveLM@lemmy.zipEnglish
    2·
    3 hours ago

    I’ve thought about bodging something together with n8n and Tasker so I can just screenshot something or send it a voice note and have it add an event on my calendar.

    That being said my server is an 8th gen i3 w/ 12GB of RAM and I’m not willing to leave my desktop running 24/7 so idk if I can run a model good enough to handle this over there. Even if it is a supposedly menial “OCR and call tool” workflow.

  • trougnouf@lemmy.worldEnglish
    5·
    4 hours ago

    I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.

    • melfie@lemmy.zipEnglish
      1·
      6 minutes ago

      I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.

  • InFerNo@lemmy.mlEnglish
    3·
    5 hours ago

    I run Gemma4 12B on my RX9070 XT. it’s not permanently on, I just turn it on when I want to use it. Access it through a browser. Does what I want out of it. Nothing “agentic”.

  • binaryqueen@programming.devEnglish
    3·
    6 hours ago

    I don’t want to create an extra post, so I ask here.

    Where do you guys get your hardware from, for an affordable price?

    I’d like to run at least a 27b model but my hardware is too weak. I’m on a 16GB unified mem, M4. Tried running Bonsai 27b (it’s Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.

  • rizzothesmall@sh.itjust.worksEnglish
    4·
    7 hours ago

    I self host both at home and at work.

    At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I’m trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone’s life a bit less admin-y without having to hire a full PA role.

    Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC

    • StateMachine@lemmy.zipOPEnglish
      2·
      2 hours ago

      Are you planning on using a specific package for your PA agent? Or is this something you plan on developing in house?

      • rizzothesmall@sh.itjust.worksEnglish
        2·
        39 minutes ago

        Using openclaw for the harness and then attaching office 365 mcp with an Entra ID App token to give it access to email, calendar, and teams. The MCP format means that we can explicitly control what it reads and writes

  • rumba@lemmy.zipEnglish
    3·
    7 hours ago

    Ollama container, 2070 super, 3-7b, meta and qwen

    I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network.

    I also use it for a self hosted mealie to figure out recipe->ingredient.

    I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.

    • StateMachine@lemmy.zipOPEnglish
      2·
      3 hours ago

      I am actually very interested in the news feed thing. How do you find duplicates in the news feeds? Wouldn’t you have to pull the actual articles and then have MB of data to shift through which would exceed the window? Do you use a vector DB? Do you just have it use curl or do you pay for an Internet API service?

  • muusemuuse@sh.itjust.worksEnglish
    4·
    12 hours ago

    I have an LLM running object detection on a b580 that’s also handling camera rotation in frigate. It’s hanging out 95-120 watts

  • aurion@piefed.aurionblack.comEnglish
    51·
    12 hours ago

    Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track

    I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system

    I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.

    Glm flash (4.something?) does pretty good as well.

    I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol

    I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.

    One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.

    Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.

    I think it’s cool as fuck. I geek out every time that it works.

    Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.