Have you heard of our load and savior “Automatically renaming and organizing with sonarr and using symlinks to preserve the naming of the torrent downloads so you can seed without using twice as much storage”?
Sonarr makes a symlink from the torrent download folder to a new folder where it renames and reorganizes the file, but the pointer for that file and the file in the downloads folder point to the same file on your hdd so.you have two copies with different names but only one “file”. Now you have a perfectly organized media folder to feed into Plex while all of those files also live in your completed downloads folder with the original naming conventions. And it’s all automagic.
The one main thing that has stopped me from setting up Sonarr is that I want my media server and torrent server to be on two different machines. Can Sonarr handle symlinks or whatever over the network or something?
Currently, I manually add torrents to Qbittorrent on Server A, which downloads the file to the hard drive on Server A. When downloading has completed, I use SFTP to transfer the files to a much larger hard drive pool in Server B, which runs Jellyfin. Then I may use SSH to rename the files to something Jellyfin-friendly, if necessary. I end up with two copies of the files this way, but most likely eventually end up deleting the files from Server A when I need to free up space and decide to no longer seed them.
When I tried to have one server running both programs, having a lot of activity in Qbittorrent made Jellyfin move sluggishly. Running them on different servers like this allows them to not bottleneck each other at all, and they can run at full speed at all times. I could see myself using Sonarr if I can still keep those two main programs segregated to separate machines.
Yes that is absolutely something you can do. All you would need to do is set up a network share on the download box and then mount that share as a drive on the jellyfin box and then you should be able to do anything to it you would be able to do to folder sitting on its own hard drive. However, if you then use that to copy all the data to a second drive on the jellyfin system this leaves you hosting 2 copies of every file you want to see and manually renaming them over SSH or in a file explorer. But we can already access the download systems drives over the network right? So… Cant we just read all the data right off the download system over the network instead of storing two copies of it? You bet your ass we can…
What i would suggest is that you pick a single computer to be your storage system, and store ALL your data there. This will allow you to use sonarr to automatically create symlinks between the downloads and media folders, so that it can rename and organize your stuff for you without you doing anything, while also seeding the torrent in perpetuity and only using the data of a single copy of the file. For the same reason that you could use a network share to copy the data from the torrent box to the media server, you could also just use a network share to read all the data directly off a single storage box. At enterprise scale the storage is often an entirely seprate machine from the machine doing the compute, and the data all flows over network shares, allowing many compute servers to all access data from a single machine to prevent data duplication. I would strongly consider moving to a system where all your mass storage is centralized in one location, removing the need to duplicate production files across multiple machines (outside of the context of backups, which are a good thing). You can hook as many compute nodes as you want up to that single “Network attached storage” device and read the files off it to all those computers, none of which require more than a small ssd to run their os. This will give you twice as much storage on the storage node, becuase your not duplicating it on the jellyfin node.
All that being said, i am also going to assume your operating at a pretty small scale here and likely dont need very much compute, certainly not two computers worth. You should ABSOLUTELY be able to run jellyfin sonarr and qbittorrent off a single box that also acts as your NAS / central data storage location, without any performance issues, unless you have dozens of users or are transcoding like every single file your family is playing. On the jellyfin end, make sure you are set up to be direct playing the files at native resolution and not asking your server to trasncode them to another resolution, this uses lots of resources if you dont have a gpu or a intel cpu with quicksync. On the hardware side, what are you using? I run about 45 docker containers with no performance degradation and stream to up to 4 simultainous plex users WITH transcoding to lower the bitrate cuz my internet sucks and I do all that on a old 8th gen i5 i got out of the garbage. You should be able to get a used bussiness computer off ebay for 100 dollars that will run jellyfin and a sonarr stack with LOADS of overhead left, all on one device so you dont have to worry about network storage and getting bottlenecked by your networking equipment. If you tell me what it is running on i can maybe give you a better idea of if you have some kind of configuration issue, or its just antiquated hardware. But all this shit runs on a toaster. People use raspberry pis. I would imagine there is some kind of config issue leading to performance degradation.
As for performance, I doubt processing power has anything to do with the sluggishness issues I experienced. I figure the bottleneck was the hard drives. With both programs on the same machine, if Qbittorrent is constantly searching around the hard drive for the next bit to seed, when I go to play something with Jellyfin, that hard drive is going to take a longer time to find that file I want. I definitely noticed an increase in performance when I stopped torrenting to the same hard drive that was serving my media.
With that said, I doubt symlinks (even over the network or something) would actually fix that at all. Qbittorrent would still be actually reading off of my media server’s hard drives, limited by their speeds.
That makes sense. Ive got 25Mbps up and an array of 8 drives so i dont ever get anywhere close to being limited by the drive speed when seeding, ive got to set the upload limit well under what a single 7200rpm drive can saturate. I manage to get lots of torrents up to a pretty high ratio and keep a good ratio on all my private trackers with a 1.5 MB/s upload limiter though, so maybe you could just turn your limiter to… lets call it 20MB/s? A 7200 rpm drive should be able to do atleast 80MB/s, so if you just set a reasonable limit you will still be able to seed out more than your fair share of data without messing up your viewing experience. Then you can put all your drives in one pool in the storage server and de-duplicate everything.
Unless you are just REALLY into having the high score on seeding ratio, i am not sure allowing qbittorrent to saturate your hard drive bandwidth is necessary, although as a member of the community i greatly appreciate your efforts to seed so much data!
Yes. Just to add, not everything hard links for me due to files being seeded (and locked) straight after it’s done downloading, so later I go back and fix the bulk by entering the download folder and typing:
To list files that weren’t hard linked:
cd /path/to/downloads/
find . -links 1 -type f | grep ".mp4$\|.mkv$" | sort > ../fixthese.txt
And going through the list and either reimporting manually via *arr>Wanted>Import manually, or
omitting certain shows from the command once you’re certain all the unlinked are unwanted extras with:
Did you set the paths for your mounted folders in the docker appropriately? If you have the download folder and the media library mounted as distinct shares in the docker template it can break the links. I had to put the download and media library folders in the same mounted directory to make my links work.
I had /mnt/user/media/tv mounted to /tv
And /mnt/user/media/downloads mounted to /downloads and had to change to so I just had
/Mnt/user/media mounted to /media, with the download and media library folders both existing as subdirectories inside of /media to make them link properly. There is a faq on the sonarr reddit about this exact issue and it’s a common failure point.
If its running as a service on bare metal then the issue i described should not apply to you, as the files are all on the same harddrive as far as sonarr is concerned. I am by no means an expert on squashing config issues wrt this stuff, but there are people who know considerably more than me hanging out in the sonarr reddit who could probably help. Sorry to point your towards a corpo hell hole, but thats the place that I know where the people that actually know their shit and want to help are hanging out. Just reading the stickied FAQ sorted out all my issues and everything symlinks for me now.
Thank you for this comment - I have been content with just a torrent client and jellyfin for ages. All files are tossed into a television or movies folder across multiple drives, like MoviesDriveA and MoviesDriveB. It works but it’s a real pain sometimes and having everything just go to a simpler torrent downloads would be a great time saver.
I was amazed when I found it this was how you were supposed to do it, and also kinda mad about there being no way for me to go back and seed the thousands of torrents I had renamed to get them to play right with Plex imports. Oh well, you live and you learn. If you have any issues the sonarr subreddit has some great guides in the sticky section and fairly helpful users that should be able to get the symlink think sorted out. Noone tells you that’s how it’s supposed to be when you start, you kinda have to bump into the information when you get mad about this very problem.
Good luck! I bet you’ll be pretty happy with the drastic reduction in labor once you set it up
There’s a silver lining because for some reason qBit updated and I lost hundreds of torrents including for private trackers. So my old workflow of managing via moving torrent locations is broken anyhow. Time to just restart, get new torrents, manage them better, and try to manually reorganize files that don’t have a torrent file anymore. Thank you!!
Sonarr/radar will organize and rename the ones without torrents for you… Don’t do it manually. Just point them (sonarr/radarr) at the downloads folder as an import folder.
Sonarr is for TV and radarr is for movies, if you didn’t know.
Have you heard of our load and savior “Automatically renaming and organizing with sonarr and using symlinks to preserve the naming of the torrent downloads so you can seed without using twice as much storage”?
Sonarr makes a symlink from the torrent download folder to a new folder where it renames and reorganizes the file, but the pointer for that file and the file in the downloads folder point to the same file on your hdd so.you have two copies with different names but only one “file”. Now you have a perfectly organized media folder to feed into Plex while all of those files also live in your completed downloads folder with the original naming conventions. And it’s all automagic.
The one main thing that has stopped me from setting up Sonarr is that I want my media server and torrent server to be on two different machines. Can Sonarr handle symlinks or whatever over the network or something?
Currently, I manually add torrents to Qbittorrent on Server A, which downloads the file to the hard drive on Server A. When downloading has completed, I use SFTP to transfer the files to a much larger hard drive pool in Server B, which runs Jellyfin. Then I may use SSH to rename the files to something Jellyfin-friendly, if necessary. I end up with two copies of the files this way, but most likely eventually end up deleting the files from Server A when I need to free up space and decide to no longer seed them.
When I tried to have one server running both programs, having a lot of activity in Qbittorrent made Jellyfin move sluggishly. Running them on different servers like this allows them to not bottleneck each other at all, and they can run at full speed at all times. I could see myself using Sonarr if I can still keep those two main programs segregated to separate machines.
Yes that is absolutely something you can do. All you would need to do is set up a network share on the download box and then mount that share as a drive on the jellyfin box and then you should be able to do anything to it you would be able to do to folder sitting on its own hard drive. However, if you then use that to copy all the data to a second drive on the jellyfin system this leaves you hosting 2 copies of every file you want to see and manually renaming them over SSH or in a file explorer. But we can already access the download systems drives over the network right? So… Cant we just read all the data right off the download system over the network instead of storing two copies of it? You bet your ass we can…
What i would suggest is that you pick a single computer to be your storage system, and store ALL your data there. This will allow you to use sonarr to automatically create symlinks between the downloads and media folders, so that it can rename and organize your stuff for you without you doing anything, while also seeding the torrent in perpetuity and only using the data of a single copy of the file. For the same reason that you could use a network share to copy the data from the torrent box to the media server, you could also just use a network share to read all the data directly off a single storage box. At enterprise scale the storage is often an entirely seprate machine from the machine doing the compute, and the data all flows over network shares, allowing many compute servers to all access data from a single machine to prevent data duplication. I would strongly consider moving to a system where all your mass storage is centralized in one location, removing the need to duplicate production files across multiple machines (outside of the context of backups, which are a good thing). You can hook as many compute nodes as you want up to that single “Network attached storage” device and read the files off it to all those computers, none of which require more than a small ssd to run their os. This will give you twice as much storage on the storage node, becuase your not duplicating it on the jellyfin node.
All that being said, i am also going to assume your operating at a pretty small scale here and likely dont need very much compute, certainly not two computers worth. You should ABSOLUTELY be able to run jellyfin sonarr and qbittorrent off a single box that also acts as your NAS / central data storage location, without any performance issues, unless you have dozens of users or are transcoding like every single file your family is playing. On the jellyfin end, make sure you are set up to be direct playing the files at native resolution and not asking your server to trasncode them to another resolution, this uses lots of resources if you dont have a gpu or a intel cpu with quicksync. On the hardware side, what are you using? I run about 45 docker containers with no performance degradation and stream to up to 4 simultainous plex users WITH transcoding to lower the bitrate cuz my internet sucks and I do all that on a old 8th gen i5 i got out of the garbage. You should be able to get a used bussiness computer off ebay for 100 dollars that will run jellyfin and a sonarr stack with LOADS of overhead left, all on one device so you dont have to worry about network storage and getting bottlenecked by your networking equipment. If you tell me what it is running on i can maybe give you a better idea of if you have some kind of configuration issue, or its just antiquated hardware. But all this shit runs on a toaster. People use raspberry pis. I would imagine there is some kind of config issue leading to performance degradation.
As for performance, I doubt processing power has anything to do with the sluggishness issues I experienced. I figure the bottleneck was the hard drives. With both programs on the same machine, if Qbittorrent is constantly searching around the hard drive for the next bit to seed, when I go to play something with Jellyfin, that hard drive is going to take a longer time to find that file I want. I definitely noticed an increase in performance when I stopped torrenting to the same hard drive that was serving my media.
With that said, I doubt symlinks (even over the network or something) would actually fix that at all. Qbittorrent would still be actually reading off of my media server’s hard drives, limited by their speeds.
That makes sense. Ive got 25Mbps up and an array of 8 drives so i dont ever get anywhere close to being limited by the drive speed when seeding, ive got to set the upload limit well under what a single 7200rpm drive can saturate. I manage to get lots of torrents up to a pretty high ratio and keep a good ratio on all my private trackers with a 1.5 MB/s upload limiter though, so maybe you could just turn your limiter to… lets call it 20MB/s? A 7200 rpm drive should be able to do atleast 80MB/s, so if you just set a reasonable limit you will still be able to seed out more than your fair share of data without messing up your viewing experience. Then you can put all your drives in one pool in the storage server and de-duplicate everything.
Unless you are just REALLY into having the high score on seeding ratio, i am not sure allowing qbittorrent to saturate your hard drive bandwidth is necessary, although as a member of the community i greatly appreciate your efforts to seed so much data!
Yes. Just to add, not everything hard links for me due to files being seeded (and locked) straight after it’s done downloading, so later I go back and fix the bulk by entering the download folder and typing:
cd /path/to/downloads/ find . -links 1 -type f | grep ".mp4$\|.mkv$" | sort > ../fixthese.txtAnd going through the list and either reimporting manually via *arr>Wanted>Import manually, or
find . -links 1 -type f | grep ".mp4$\|.mkv$" | grep -vE "./(Band.Of.Brothers|The Boys|Westworld|.*\] (Attack On Titan|JoJo)|.*ample.mkv$)" | sort > ../fixthese.txt( “./(|…” covers most shows, “*] (|…” covers files starting with e.g. [Anime Time], and “.*ample.mkv$” covered my ‘sample’ and “Sample” videos)
Did you set the paths for your mounted folders in the docker appropriately? If you have the download folder and the media library mounted as distinct shares in the docker template it can break the links. I had to put the download and media library folders in the same mounted directory to make my links work.
I had /mnt/user/media/tv mounted to /tv And /mnt/user/media/downloads mounted to /downloads and had to change to so I just had /Mnt/user/media mounted to /media, with the download and media library folders both existing as subdirectories inside of /media to make them link properly. There is a faq on the sonarr reddit about this exact issue and it’s a common failure point.
Ah, thank you, but I run it as a service. Sonarr currently has /media/disk2/New for downloads and media/disk2/Shows for main shows.
If its running as a service on bare metal then the issue i described should not apply to you, as the files are all on the same harddrive as far as sonarr is concerned. I am by no means an expert on squashing config issues wrt this stuff, but there are people who know considerably more than me hanging out in the sonarr reddit who could probably help. Sorry to point your towards a corpo hell hole, but thats the place that I know where the people that actually know their shit and want to help are hanging out. Just reading the stickied FAQ sorted out all my issues and everything symlinks for me now.
Thank you for this comment - I have been content with just a torrent client and jellyfin for ages. All files are tossed into a television or movies folder across multiple drives, like MoviesDriveA and MoviesDriveB. It works but it’s a real pain sometimes and having everything just go to a simpler torrent downloads would be a great time saver.
I was amazed when I found it this was how you were supposed to do it, and also kinda mad about there being no way for me to go back and seed the thousands of torrents I had renamed to get them to play right with Plex imports. Oh well, you live and you learn. If you have any issues the sonarr subreddit has some great guides in the sticky section and fairly helpful users that should be able to get the symlink think sorted out. Noone tells you that’s how it’s supposed to be when you start, you kinda have to bump into the information when you get mad about this very problem.
Good luck! I bet you’ll be pretty happy with the drastic reduction in labor once you set it up
There’s a silver lining because for some reason qBit updated and I lost hundreds of torrents including for private trackers. So my old workflow of managing via moving torrent locations is broken anyhow. Time to just restart, get new torrents, manage them better, and try to manually reorganize files that don’t have a torrent file anymore. Thank you!!
Sonarr/radar will organize and rename the ones without torrents for you… Don’t do it manually. Just point them (sonarr/radarr) at the downloads folder as an import folder. Sonarr is for TV and radarr is for movies, if you didn’t know.