With the recent discussions around replacing Spotify with selfhosted services and the possibilities to obtain the music itself, I’ve been finally setting up Navidrome. I had to do quite a bit of reorganization to do with my existing collection (beets helping a ton) but now it’s in a neatly organized structure and I’m enjoying it everywhere. I get most of my stuff from Bandcamp but I have a big catalog from when I’ve still had a large physical collection.
I’m also still working on my docker quasi gitops stack. I’ve cleaned up my compose files and put the secrets in env files where I hadn’t already, checked them into my new forgejo instance and (mostly) configured renovate. Komodo is about to get productive but I couldn’t find the time yet. Also I need to figure out how to check in secrets in a secure way. I know some but I haven’t tried those with Komodo yet. This close of my fully automated update-on-merge compose stacks!
I’ve also been doing these for quite a while and decided to sometimes post them in !selfhosting@slrpnk.net to possibly help moving a bit from the biggest Lemmy instance, even though this community as it is is perfectly fine as well as it seems.
What’s going on on your servers? Anything you are trying to pursue at the moment?
You must log in or # to comment.
For privacy reasons, I have finally fully disabled dynamic dns updates and closed the last holes in the home firewall, moving to 100% proxying via a VPS for publicly available stuff, and a tailnet (headscale) for everything private. The only real cross-over is Nextcloud - mountains of private data, but I want it publicly available for file shares. Fortunately, Nextcloud has a setting to whitelist IP addresses that allow log-in, so I can restrict that to just the non-VPS tailnet addresses. From the public internet, only public shares are accessible.
I set up a L4 proxy so that the encryption for Nextcloud happens at home and the VPS just passes encrypted packets. Then it occurred to me that a compromised VPS could easily grab a SSL cert for my Nextcloud subdomain via a regular-old http-challenge and MITM access to all my files, defeating the point.
Then I found a neat hack that effectively disables http-challenge certs for subdomains by requiring a wildcard certificate - which can only be created with a dns-challenge. I was able to also disable all other certificate authorities. Obviously, I have /some/ trust in the VPS I administer - it’s on my tailnet network - but no longer have the concern that it could easily MITM Nextcloud. https://www.naut.ca/blog/2019/10/19/mitigating-http-mitm-possibilities-with-lets-encrypt/
I spent some time last week learning both Ansible and Podman Quadlets. They are a powerful duo, especially for self hosting.
Ansible is a desired state system for Linux. Letting you define a list of servers and what their configuration should be, like “have podman installed” and “have this file at this location with this content”.
Podman quadlets is a system for defining podman containers as a service. You define the container, volumes, and networks all in essentially Systemd unit files.
Mixing the two together, I can have my entire podman setup in a format that can be pushed to any server in seconds.
And of course everything is text files that git well.
I was thinking about this for some time now, can you link me to some good tutorials about quadlets in particular? Ansible will have to wait for now.
Unfortunately not. I found documentation largely lacking. I mostly read the docs and searched specific questions that came up(which often just took me back to the docs). I did as a local LLM for help, but found it’s knowledge base lacking. Sometimes it would work for a hint, but it more often than not made up parameters and features.
Which docs did you read in particular? If you mind to share? And so nice, that you setup a local LLM. I sadly don’t have the horsepower for that.
Here is a redhat blog about it: https://www.redhat.com/en/blog/quadlet-podman
This docs page has more details: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/building_running_and_managing_containers/porting-containers-to-systemd-using-podman
And last, this page shows most of the options you’d expect to find: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html
I did the same last week (and am still in the process of setting up more services for my new server). I have a few VMs (running Fedora CoreOS, with podman preinstalled), and I use ansible to push my quadlets, podman secrets, and static configuration files. Persistent data volumes get mounted using virtiofs from the host system, and the VMs are not supposed to contain any state themselves. The VMs are also provisioned using using ansible.
Do you use ansible to automatically restart changed containers after pushing your changes? So far, I just trigger a
systemctl daemon-reload, but trigger restarts manually (which I guess is fine for development).I haven’t gotten too far, but right now I’ve got persistent volumes being pushed by NFS from my NAS. I’m using rocky Linux VMs as my target, but for this use case, Fedora CoreOS should be the same.
I haven’t yet tried using Ansible to create the VMs, but that would be cool. I know teraform is designed for that sort of thing, but if Ansible can do it, all the better. I’d love to get to a point where my entire stack as Ansible.
I don’t yet have Ansible restarting the service, but that should be a simple as adding a few new tasks after the daemon-reload task. What I don’t know how to do is tell it to only restart if there is change to any of the config files uploaded. That would be nice to minimize service restarts.
Oh that’s smart! I just got started with podman and quadlets. Loving how simple it is to setup a systemd service and even organize multi-pod apps
So I am in a vicious cycle. I start doing something, notice there is a better way, change my setup and restart. So from just Ubuntu server, I developed to proxmox. From documenting everything manuall in joplin, i am now using ansible. I started with wireguard, then tailscale with selfhosted headscale. I try to get my setup right on the first try, which i notice is stupid as I am writing. It just hinders me to make progress. I think I should rather try to get it up and running as fast as possible (and securely of cause) to make progress and fail fast maybe? And I like all the changes I made, I think they were the right choice, but its a bit tiering. And I like ansible, I just have the urge to automate absolutely everything, so I can redeploy everything right after I installed proxmox. Which is not necessary at all at this stage, idk :D Maybe someone has some tips how to overcome perfectionism?
For me, tinkering is part of the process and I’m enjoying it. Deciding to do something differently and changing a lot of stuff every now and then is fine. What’s annoying is if you are in the middle of such a process and then run out of (free) time. Next time I look at it I forgot half of it if it’s not finished and documented.
I’m setting up a yunohost machine for my brother as a birthday present. I got him a domain good for 10 years, and installed nextcloud and Jellyfin with some home videos digitized from our parents’ vhs tapes.
What is beets, hobbits?
It’s a tool that checks and corrects metadata for your music collection. You can also import music with it to your collection (it will put everything in the right folders etc).
It does require some manual intervention now and then, though (do you really want to apply this despite some discrepancies? Choose, which of these albums it really is. Etc).
Would you say it’s better than musicbrainz?
I think it uses musicbrainz
Very cool!
I finally set up Jellyfin and Sonarr! I’ve been using Plex and manually managing torrents for a while now, recently found the *arr services and they are very impressive. Got the Jackett - Sonarr - Jellyfin - Nginx stack set up, now working on getting SSL + DynDNS so I can make it available remotely. Also accidentally blasted my ratio downloading a bunch of TV shows all at once so gotta seed up for a bit before i fill it out more. But so far the setup has been pleasantly breezy for how complex a setup it is ❤️
Why not Emby?
Good question; I did not know what Emby is until just now. I will explore it some more, I’m having issues getting the jellyfin ios/android clients to connect consistently to my server so I might ultimately do that instead / in parallel but I’m leery of freemium solutions.
I get that; I’ve tried Jellyfin and it’s just not (IMHO) mature enough as a Plex replacement. Emby comes pretty close.
Note: I’m a plex lifetime subscriber, Emby free user, and Jellyfin user.
Nice, I appreciate the analysis. I’m still early enough on with Jellyfin that I’m still willing to ascribe every issue to user error but I think I see what you mean. But I keep telling myself that I will contribute to a large multi-dev OSS project at some point and still never have; contributing code in public is still kinda nerve-wracking. maybe if I have a selfish enough reason to fix something I’ll finally push through that 😆
I finally got around to setting up my internal services with TLS. It was surprisingly easy with a Caddy docker image supporting Cloudflare DNS challenge.
I did this because various services I use are starting to require https.
Now everything is on a custom domain, https, and I can access it through Tailscale as usual.
I’m trying to find a reasonably priced used rack mount computer to move all my containers to. I have a rack in my house but measuring the depth between posts only gets me around 17.5". Recently deployed paperless-ngx and decided it would be too much to add onto my poor little NAS which hosts everything else so its deployed on my main computer and I want to avoid that strategy.
Challenge is that being new to rack servers and all of this (the NAS was a great intro box) I’ve got a large learning curve ahead of me.
Are you using the rack already? Many people are opting for 10" racks for their homelabs these days. There’s 3D printable enclosures for many thin clients and mini PCs. Minilab is the go-to term. This is mine if you’re interested
Already have the rack and it hosts most of my items (router, switch, raspberry pi rack, NAS, PS3 apple TV). Honestly other than the raspberry pi rack and the router, the rest are just using shelves anyways. I need to find a good, non-bulky way to take the enclosure fan and switch it on when temps get above a set point. What I’ve done in the past with an arduino is way too bulky.
I love your idea and its funny you mention the mini PCs and thin clients. As I look at prices I’m more and more leaning towards just another shelf and using mini PCs.
Just be sure to get a decent KVMoIP (KVM over IP); it’s worth it for the first time you’re not home and a system blue screens.
For what it’s worth, Ikea’s LACK tables make great mini racks
The LACK rack!
I have a Dell R220 and a R240 which I’m looking to offload, free. They’re both specifically for short racks if you happen to be near central NC.
Out of curiosity, what has been your experience on noise and power consumption with your r220 and r240 servers?
They’re pretty decent. Really depends on the temperature.
If the room is cool you hear the hard drives over the chassis fans. Power is totally dependent on the CPU. I put a 4790k in the R220 and it would sit around 45-55w average with two 3.5” drives.
Funny you should say that! If I was still over in that part of the country I’d absolutely take you up. There is a local used IT shop near me and I was really excited to purchase some of those for a good deal. Then found out that they only update their online inventory when you click add to cart. Long story short, the prices were too good to be true and they didn’t have any left in stock.
Considering switching my Forgejo to a Tangled.sh knot. Their easy self hosted CI option is appealing
but mostly itd be easier to collaborate than opening sign ups on my instance of forgejo
Forgejo is developing ActivityPub support, so eventually we’ll be able to collaborate across forgejo instances :) I’ll have a look at tangled as well
oh thats cool thanks for sharing!
That looks super interesting. Just yesterday I was reading on https://radicle.xyz/ for similar reasons.
I have a couple pis that run docker containers including pihole. The containers have their storage on a centralized share drive.
I had a power outage and realized they can’t start if they happen to come up before the share drive PC is back up.
How do people normally do their docker binds? Optimally I guess they would be local but sync/backup to the share drive regularly.
Sort of related question: in docker compose I have restart always and yet if a container exits successfully or seemingly early in it’s process (like pihole) it doesn’t restart. Is there an easy way to still have them restart?
You should be able to modify the docker service to wait until a mount is ready before starting. That would be the standard way to deal with that kind of thing.
What if it’s a network mount inside the container? Doesn’t the mount not happen till the container starts?
What’s the best ACME server?
I am at the very beginning of my journey taking those first baby steps. As I don’t yet understand all the sysadmin stuff, I’m treading rather carefully to avoid making unfuckable mistakes.
I recently switched to Void on my daily driver so it has been a bit of a trial to get used to a new OS and configure it correctly. Nevertheless, it’s been a great learning experience.
Alongside it I’ve downloaded OpenWrt on my router and begun to configure it as well (still need to deal with the Wireguard and Unbound config).
For the actual server I managed to secure an old Dell Optiplex. In the near future, I plan to flash it with Libreboot and then install Debian or FreeBSD (apparently great ZFS support) on it. Though I’ve still no idea whether I should use Proxmox and how I should format my drives (one 500GB SSD and 4TB HDD) for maximum effiency and for the possibility of later easily upgrading my storage capacity.
When I’ve finally past these steps, I plan to selfhost music services, as well as few other basic services. My goal at the moment is to replace Spotify for my whole family. But it’s still a long way to go.
Just got a domain and started exposing my local jellyfin through cloudflare, mostly wanting to listen to my music on my phone when i’m outside too.
I followed some guides that should make it fine with cloudflare’s policy, video doesnt work when i tried it but otherwise its been fun despite me feeling like im walking on eggshells all the time. I guess time will tell if it holds up
Some things which have caused issues for me:
File permissions
Video/audio format (264/aac stereo is best for compatibility)
Oh file permissions are a nightmare to me, I thought I managed to get it sorted but after i installed lidarr, it alone suddenly can’t move files out of the download location anymore. I even tried to chmod 777 the data folders and nothing. I dont think I quite have the grasp on how those work with docker on linux yet, it seems like those arr services also have some internal users too which I dont get why would they.
Wdym with the formats, is this referring to transcoding? I kept those on defaults afaik
In linux user and group names don’t matter. Only the gid and uid matter. Think of user and group names as human names like domains are for IPS.
In docker when you use mounts, all your containers that want to share data must agree on the gid and uids.
In rootless docker and podman things subuids and subgids make it a little more complicated since IDs get mapped between host and container, but its still the IDs that matter.
Could be that lidarr is setting its own permissions for downloaded stuff (look for something like dmask or fmask in the docker config). You might also need to chmod -R so it hits all sub folders. If you have a file or directory mask option, remember that they’re inverse, so instead of 777, you’d do 000 for rwxrwxrwx.
You might be onto something, lidarr does have UMASK=002 setting in the .env file. I think the issue is when sabdnzbd puts the files and then lidarr can’t read them, so what exactly is the expected permission setting then in this case? If I put it to 000 for lidarr, won’t other services then be unable to add the files there?
I always feel so dumb when it comes to these things since in my head it’s something that should be pretty straightforward and simple, why can’t they all just use the same user and share the same permissions within this folder hierarchy…
I feel like my little Pi server is set up nicely now. At least I’m at the point where I’m not concerned about technically maintaining it. It’s as secure as I want it to be and I’ve tweaked my maintenance scripts slightly to avoid any unexpected issues.
I tried installing snikket but I couldn’t figure out how to get it to work with my Caddyfile using my current wildcard domain cert configuration. I’ll try again another time when I’m motivated again. It’s a low priority to me.
The last changes I made were adding logs and making them accessible to myself. So far they are all boring and predictable. Which is good news. It’s also nice to see that I’m the only person accessing it. The bots haven’t found my little corner of the internet yet.
Right now I’m taking a break from self-hosted stuff to work on my gardens and two artsy projects. A wooden carving for a friend’s birthday and an overly complicated shell script that has no real purpose. Although I’ve learned lots from it already so it’s not a complete waste of time.
How did you approach the logging?
Since my logs barely move, I just made aliases to where the logs are so it’s quick display and scan them within the terminal. I’m basically just viewing the system logs, fail2ban log and Caddy’s log so it’s fairly quick and simple for me.
The only change I’d like to do is change the output of Caddy’s log file so it’s not a long single line of information per output. I’ll have to do a bit more reading on that so I know what information I want to keep and how I want to visually organize it. At least for the moment, I am familiarising myself with what I am looking at and am slowly figuring out what information is relevant to me.
I like to keep my systems as simple and lean as possible which seems to strongly reflect my general approach to life. I find that kind of interesting.
If you like, check GoAccess on the Caddy Files. You can watch them through that instead of less/cat/whatever to see a nice Dashboard. It helps getting a better overview IMHO.
Nothing new, just peacefully chugging along hosting my blog, Jellyfin, Radicale for calendar and contacts. Still long-term searching for a photo storing & sharing (gallery) solution, as well as a better music server. Maybe Navidrome is what I’m looking for.
Oh, and I need to renew my SSL certificate soon. I don’t like Letsencrypt. Everything EU-based, I’m not going to start making US-based contracts.
Can recommend Immich for the Photo gallery and sharing option.
Can recommend Navidrome for music.
Did you have a look at Immich yet for photos?
