Sips' to Selfhosted@lemmy.worldEnglish • 4 months agoCan't relate at all.slrpnk.netimagemessage-square192fedilinkarrow-up10arrow-down10
arrow-up10arrow-down1imageCan't relate at all.slrpnk.netSips' to Selfhosted@lemmy.worldEnglish • 4 months agomessage-square192fedilink
minus-square@Hackworth@lemmy.worldlinkfedilinkEnglish0•4 months agoI’ll try it out! It’s been a hot minute, and it seems like there are new options all the time.
minus-square@brucethemoose@lemmy.worldlinkfedilinkEnglish0•edit-24 months agoTry a new quantization as well! Like an IQ4-M depending on the size of your GPU, or even better, an 4.5bpw exl2 with Q6 cache if you can manage to set up TabbyAPI.
I’ll try it out! It’s been a hot minute, and it seems like there are new options all the time.
Try a new quantization as well! Like an IQ4-M depending on the size of your GPU, or even better, an 4.5bpw exl2 with Q6 cache if you can manage to set up TabbyAPI.