+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: Newznab - Anyone using it? (/showthread.php?tid=82119)
RE: Newznab - Anyone using it? - richierea - 2012-12-16
I have the paid for version running on Ubuntu, all seems to be OK but what scripts are people using to pull in the headers using multiple threads?
I can run update_binaries_threaded but which threaded script will get it to loop, say every 15 minutes?
RE: Newznab - Anyone using it? - Harro - 2012-12-16
I opened up the newznab_screen.sh which is found under update_scripts and edited the export NEWZNAB_PATH="path to the update_scripts folder"
Then I puttied into server and cd'ed to the the location of the newznab_screen and opened a new screen and ran newznab_screen start
RE: Newznab - Anyone using it? - richierea - 2012-12-16
I have switched from Windows to Ubuntu to be able to run threaded mode. Kicked off 10 groups with 1 day backfill on 10 threads. It's showing 7 groups complete but seems to be taking ages on a.b.etc (think it is quite a big group), Been running about 5 hours and still going. What sort of times are peoples taking? I know the groups are all very different sizes but if some people could post the groups they have, if threaded or not and the sort of times it's taking I would be interested.
As it stands it almost seems the Windows installation running single thread was quicker. Not sure if somethings wrong or not.
RE: Newznab - Anyone using it? - Dekles1 - 2012-12-16
@wiz561 FYI Giganews does support compressed headers if you use their Giganews Accellerator app. http://www.giganews.com/accelerator.html
RE: Newznab - Anyone using it? - Aenima99x - 2012-12-17
(2012-12-16, 21:31)wiz561 Wrote:That's a bad move unless you're looking forward to lots if incompletes due to their automated DMCA takedowns and increasingly crappy speeds. I had been a happy customer of theirs for about 5 years and had to dump them a couple months ago. I went to UseNetServer and have been pleased.(2012-12-16, 20:59)Dekles1 Wrote: @wiz561 FYI Giganews does support compressed headers if you use their Giganews Accellerator app. http://www.giganews.com/accelerator.html
Yeah, but the downside to that is you *have* to use their app and the bigger problem that I run into is that it only works on windows and not linux. If giganews allows XZVER within the next week and a half, they'll keep me as a customer. If not, I'm jumping over to astraweb.
RE: Newznab - Anyone using it? - scooper750 - 2012-12-17
(2012-12-16, 16:10)Harro Wrote: I have this running on my Unraid box. So far it has been really smooth and updating every 10 minutes.
Can you elaborate on how you got this setup on your unraid server?
RE: Newznab - Anyone using it? - lurkingsheep - 2012-12-17
How often should I should run update_binaries.php & update_releases.php ?
Do I need to set this up in CRON or is this being done automatically?
Thanks
RE: Newznab - Anyone using it? - richierea - 2012-12-17
I seem to have mine working fine now. I had both a windows version and Ubuntu (in a VM virtualBox) running at the same time to see if there was any difference. Both are pulling in headers from 10 groups in about a minute or less, running every 15 minutes.
It took quite a while to backfill 1 day for all the groups, but once that completed it seems very quick on it's updates.
I did try running the Ubuntu version in threaded mode (10 connections) which makes no difference on the 15 min cycle as the amount of headers is so small. It should make a difference if backfilling, but it didn't seem very stable and wasn't sure if it had crashed or not, so gave up on that.
As i'm not too bothered about backfilling i'll forget about the threaded version, and for ease of use I am going to stick with the Windows version.
Thanks to all people that have posted in this thread, there have been some useful comments to get this up and running.
RE: Newznab - Anyone using it? - jatilq - 2012-12-18
From the comment section of the Tiag.me page. 3rd party add ons are working I think.
bcalDecember 17, 2012 at 5:37 am · Reply
Thanks so much for the guide! I hope you get the 3rd party add on guide set up. No matter where I place the 3rd part programs (unrar, mediainfom etc, C drive, F drive etc) I get an “The unrar path does not point to a valid binary” error (for each add on). I use forward slashes and backwards slashes in the path and still no go. Look forward to reading the guides!
jlcDecember 17, 2012 at 6:51 pm · Reply
I don’t know if this helps you but this is my paths which worked for me
C:/Program Files/MediaInfo/MediaInfo.exe
C:/Program Files/WinRAR/winrar.exe
C:/Program Files/ffmpeg/ffmpeg.exe
bcalDecember 17, 2012 at 6:56 pm · Reply
fixed my own problem. For whatever reason, in the site editor, where you input the paths to the 3rd part add ons, it did not like it when I typed in the drive letter and colon. When I copied the “F:/” from the pre-populated Temp Unrar File Path and pasted it into the other, blank file paths and then typed the rest of the file path. it worked fine.
For Example:
1. Highlight the “F:/” in the Temp Unrar File Path
2. Press Ctrl-C
3. Go to the Unrar Path field
4. Press Ctrl-V
5. Then type in xampp/unrar/unrar.exe so the filed now reads F:/xampp/unrar/unrar.exe
6. Save
Now though, when I run the update releases script, I am having problems with mediainfo. When the script uses the mediainfo app, it opens a mediainfo window and wont do anything until I close the window manually.
I personally think I messed up somewhere because its taking forever to go through one cycle. I moved to 1 day from 7 and I'm hoping I dont have to start over. Ive reset/purged all my active groups but still at a loss. Anyone have any simple instructions to clear old data without reinstalling?
after I have run runme.bat I see it just finished with Updating completed in 37.304.07 seconds. I'm not sure if I should run it again or something else to shorten this up
RE: Newznab - Anyone using it? - Harro - 2012-12-18
Just a few things before I comment on the HOW TO.
My unraid box is listed in this forum thread.
I have 16 gigs of mem and a 320 gig cache drive. Mem is overkill but was so cheap I thought WTH. I also am running unRAID Server version 5.0-rc8a AiO, which I think you need inorder for all the additional plugins to run correctly.
I find the cache drive to be a must have for getting the newznab setting up and running, plus a great place to load other plugins. I have Sickbeard, SAB, Couch, Mysql, Plex media server and webserver on mine.
In my unraid shares I have a share named appdata. That share is configured in the share setting as use "cache only". In the folder called "appdata" I have the above folders created for those specific plugins.Having the folder "appdata" using only "cache only" setting will prevent the mover script that I have running everyday from moving anything in the folder called "appdata". Otherwise everything woud be moved to my main array off the cache drive. In order to proceed please create a new share on your unraid server called whatever you want (mine= appdata) but use cache only setting.
Since I am a total noob with linux and the commands for it, I used windows as much as I could. If you are not like me then forgive my stupidity on the subject. So before I get started there are a few programs you will need. Notepad2 or Notepad ++ along with the donated version of Newznab and TortoiseSVN. Last one is Putty.
Packages or plugins needed for newznab to run are as follows, Simple Features and MySql and Newznab plg (For the Newznab.plg right click,save target as all files, rename .xml to .plg)
The free version of newznab is not worth setting up IMO.
Now I am assuming that you are running the unRAID Server version 5.0-rc8a. I also assume you know something about Unraid and how to enable shares to be seen on your network.
1) Unzip Simple Features zip file, find simpleFeatures.core.webGUI-1.0.5-noarch-1.plg and simpleFeatures.web.server-1.0.5-noarch-1.plg along with the Mysql-5.5.22-i486-1gds.plg from the MySql zip folder and open your network in windows, find your unraid box, double click to open all the shares, find the flash (your usb drive) if it is not shared you will need to share it in the unraid webGUI. Where you want to go is like such \\TOWER\flash\config\plugins. Once there drag the file simpleFeatures.core.webGUI-1.0.5-noarch-1.plg, simpleFeatures.web.server-1.0.5-noarch-1.plg, Mysql-5.5.22-i486-1gds.plg and newznab_unplugged.plg into the plugins folder. Exit out
2) go to //tower/ and stop array and then reboot.
3)In my understanding the .plgs you had just dragged into the flash/config/plugins folder should now be working. If not you will need to use putty to install them. Ask for help if the plugins are not working. For right now lets say the plugins are working. go back to //tower/ and under the settings you should now see the new plugins listed on the page. Do nothing yet. Exit out
4)open windows network and go to your unraid cache drive locate your shared folder called appdata, open that folder and now create 2 new folders named "mysql' and "server".
5) Open the server folder and now right click and use tortoisesvn to download or check out the newznab. Once done your folder structure should look like Tower\cache\appdata\server\ with folders.svn,db,docs,misc,nzbfiles,and www.
6) under the Tower/cache/appdata/server/misc/urlrewriting find the lighttpd.txt, copy that to desktop. exit
7) Open up your Unraid webGUI in browser //tower got to settings find mysql and set the data directory to /mnt/cache/appdata/mysql/ and enable to yes, then apply and it should show running. Done
8) back in settings find the webserver set the web root as /mnt/cache/appdata/server/www/ port 81 enable and apply , done
9) back in setting again into web server and disable server.
10)go back into windows network, we are looking for this \\TOWER\flash\config\plugins\simpleFeatures . In the simple features folder is a lighttpd.cfg. On your desktop is a file called lighttpd.txt , open that in notepad2 and copy that into the lighttpd.cfg. Your new lighttpd.cfg should look like this
Code:
# User customizations
# Add your own settings here. These won't get overwritten upon reboot
#
url.rewrite-once = (
"^/.*\.(css|jpg|jpeg|gif|png|js|ico)" => "$0",
"^/(admin|install).*$" => "$0",
"^/([^/\.]+)/?(?:\?(.*))$" => "index.php?page=$1&$2",
"^/([^/\.]+)/?$" => "index.php?page=$1",
"^/([^/\.]+)/([^/]+)/?(?:\?(.*))$" => "index.php?page=$1&id=$2&$3",
"^/([^/\.]+)/([^/]+)/?$" => "index.php?page=$1&id=$2",
"^/([^/\.]+)/([^/]+)/([^/]+)/?$" => "index.php?page=$1&id=$2&subpage=$3"
)11) in the same folder is a php.ini open that up in notepad2 and find the line that says memory_limit = 256 and change that to 512 save and close.
12) While we are in the network folder lets go to \\Tower\cache\appdata\server\misc\update_scripts\nix_scripts and open newznab_screen.sh with notepad2. find the line export NEWZNAB_PATH= and fill in with this "/mnt/cache/appdata/server/misc/update_scripts" save and close.
13) Go back to your browser and into the unraid webGUI, settings enable webserver. Now under the newznab plugin we are ready to configure.
14) under the Newznab configutation set Server Root: * Set in simpleFeatures web server plug-in! to /mnt/cache/appdata/server/ install automatically yes NN Plus setting you can input your username and password in from the email they sent you. Click apply.
15) The plugin says not configure , so click that and if all goes well it should lauch the first step.
16) on step 2 for mysql fill in username root password root then onto step 3 and such if all goes well.
17) Now that newznab is running you need configure the groups list and other items for the webpage under the admin section. Do not grap all the groups at first only a few. Now that you have configured that close it off.
18)Now to get some info into the webpage. Open Putty Host Name or IP is where you would put your internal Unraid ip. Ex.. 192.168.x.x Port 23 and connection type Telnet. save session as tower or whatever. Then open. should be greeted with Tower login: input your password. if you haven't set one it will be root as default.Once logged in type cd /mnt/cache/appdata/server/misc/update_scripts. then type ls -l this will list all the files in that dir. then type php update_binaries.php you should now see all the binaries downloading . Once this is thru in the same dir (/mnt/cache/appdata/server/misc/update_scripts) type php update_releases.php once done close putty. Go to //tower:81 and see the newly created nzbs on your webpage.
19) Well you have everything working but now need to auto mate the updating of those scripts. Open Putty once again and cd /mnt/cache/appdata/server/misc/update_scripts/nix_scripts , then type chmod +x ./newznab_screen.sh This will make the newznab_screen.sh file excutable , I think.
If you get an error permission denied type chmod -R 777 /mnt/cache/appdata/server/misc/update_scripts/nix_scripts/ if it is excutable it will be in green.If not you need to chmod +x ./newznab_screen again.Once it is green type screen. this is like another window in windoz except it is on the same window, but detached.Now type ./newznab_screen.sh start If all is working ok this will start to grap the binaries and the release scripts for you every 10 mins. DO NOT CLOSE PUTTY Instead hit the CTRL and A key and D key all togather. This will detach the screen you are on and bring you back to the other screen. You can now close putty since the newznab_screen.sh will keep running.
Please don't flame me to hard for my lack of commands in linux and probably pretty simple way to install this but this is what worked for me. Took me 2 days to get mine up and running with trial and ALOT of error.
Good Luck to you.
All the credit for all this information should be given to all the people over at Unraid Forum Thse threads were of the most useful This one, This one, This one and The one that started it Thank you guys
RE: Newznab - Anyone using it? - farscapesg01 - 2012-12-18
So... has anyone figured out the average bandwidth used by Newznab? I know it depends on what newsgroups you are pulling headers from, but for my case I'm pulling mainly TV, Movie, and books (ebooks and audiobooks). It's hard to get an accurate count with a couple downloads going on during the day also, but it looks like I'm using about 4GB a day just to download the headers. This seems pretty high to me. Anyone else got some numbers on their bandwidth usage?
RE: Newznab - Anyone using it? - richierea - 2012-12-18
(2012-12-18, 05:59)farscapesg01 Wrote: So... has anyone figured out the average bandwidth used by Newznab? I know it depends on what newsgroups you are pulling headers from, but for my case I'm pulling mainly TV, Movie, and books (ebooks and audiobooks). It's hard to get an accurate count with a couple downloads going on during the day also, but it looks like I'm using about 4GB a day just to download the headers. This seems pretty high to me. Anyone else got some numbers on their bandwidth usage?
To me that sounds a huge amount. I could be wrong as I haven't checked properly but I believe I have downloaded 200MB in the last 24 hours.
That's with 10 groups selected, mainly TV and Movies. Do need to add a couple more though.
RE: Newznab - Anyone using it? - zepfan - 2012-12-18
(2012-12-18, 05:59)farscapesg01 Wrote: So... has anyone figured out the average bandwidth used by Newznab? I know it depends on what newsgroups you are pulling headers from, but for my case I'm pulling mainly TV, Movie, and books (ebooks and audiobooks). It's hard to get an accurate count with a couple downloads going on during the day also, but it looks like I'm using about 4GB a day just to download the headers. This seems pretty high to me. Anyone else got some numbers on their bandwidth usage?
I think i've got ~4 gigs a day with ~2 coming from downloads too. I'll check tomorrow and not download anything just to see how much it's actually using.
RE: Newznab - Anyone using it? - farscapesg01 - 2012-12-18
Luckily there isn't anything airing the next couple days that I "need" to grab... so I'm just going to let it run. My monthly usage is already at 225GB being reported by Comcast, so that gives me a base point to watch over the next couple days (it was 219GB last night, but I did grab about 2 GB of files yesterday so it just depends on when it hits their reporting).
I'm assuming deep-inspection for passworded files would increase the bandwidth, so I could turn that off. I'm pulling headers for about 22 newsgroups, the busiest being a.b.e-books, a.b.multimedia, and a.b.teevee.
RE: Newznab - Anyone using it? - eluSive_NZ - 2012-12-18
(2012-12-17, 20:21)lurkingsheep Wrote: How often should I should run update_binaries.php & update_releases.php ?
Do I need to set this up in CRON or is this being done automatically?
Thanks
Was in the same boat trying to figure this out.
I tried with Byobu and its currently running great! Just running the newznab_screen.sh within this