noisybox.net

nzbperl --
nzb based nntp/usenet downloader in perl

last updated Nov 25th, 2006
overview |  features |  download |  requirements |  usage |  screenshots |  faq |  todo list |  email list |  license

Latest News / Announcements

(changelog)

Novermber 25th, 2006 - Still kicking. Just wanted to let people know that although there hasn't been a release in the better part of a year now, I still consider this project active. I've been silently banging away on the 0.7.x version and hope to have a source repository available in the near future.

December 31st, 2005: Version 0.6.8 released today.

Sneaking out the new version before the turn of the year. This will be the last release in the 0.6.x series, and it now supports disk free space checking and several important bug fixes. As usual, the changelog has gory details, and the summary is:

The next release will be in the 0.7.x series and will be geared at allowing connections to work on the same file (smaller units of parallelization).

November 14th, 2005: Version 0.6.7 released today.

Major changes:

There are some important bug fixes, so upgrade is recommended!

October 16th, 2005: Email group/list now active.

Thanks to google, nzbperl now has an email list. To subscribe, send a message to nzbperl-subscribe@googlegroups.com. The group should now be used for most nzbperl related discussions, bug reports, questions, feature requests, etc.

September 11th, 2005: another broken release. :)

The 0.6.6 version released earlier today had a silly bug. If you downloaded it earlier today and see a problem about --queuedir requiring an ABSOLUTE path, please re-download. It should be fixed now. Sorry for the hassle.

September 11th, 2005: Version 0.6.6 released today.

The major improvements/changes are:

Aug 9th, 2005: Once again, I released a broken version. 0.6.5 had a problem with --config not working, but I've changed it in place...so please re-download 0.6.5 to get the fix. I suppose it's a typical situation where a program's complexity reaches a certain point and the developer can't test it all. So...if anybody wants to write a suite of unit tests, it sure would be nice. :)

{there are older announcements, but I haven't built a nice way to view them yet}

Overview

Run Linux (or OSX or Windows)? Need a multi-connection newsreader utility that can process an nzb file and autodecode and even do bandwidth throttling? You've come to the right place. :)

I looked around for something that I could use for nzb files on my Linux setup, and I soon discovered limited options. There are progs out there that will download nzb files (nzbget and knzb are apparently the most visible ones), but none seemed to have a key feature that I put importance on: bandwidth throttling. So, like many other one-off hacks and open source projects, this little toy was built to scratch an itch.

Features

This has currently been tested and confirmed working with Debian, SuSE, RedHat, NetBSD, OSX, and Windows (via Cygwin). There's no obvious reason this shouldn't work with other platforms or distros. Feel free to send me your success or failure stories.

Download

Requirements / Dependencies

UUDeview - great program, used by nzbperl to do file decoding.

Perl (version likely flexible), and the following modules:

use IO::Socket::INET;
use File::Basename;
use XML::DOM;
use Getopt::Long;
use Time::HiRes;
use Term::ReadKey;
use Term::Cap;
use Cwd;
-------- optional modules --------
use threads; (only if you want threaded decoding)
use Thread::Queue; (only if you want threaded decoding)
use Term::ANSIColor; (only if you want color)
use XML::Simple; (only if you want remote control functions)
use IO::Socket::SSL; (only if you want SSL)
use IO::Socket::Socks; (only if you want to use SOCKS proxy)
use Net::HTTPTunnel; (only if you want to use an HTTP tunnel)
use IO::Socket::INET6; (only if you want IPv6 support)

On most systems, installing Perl modules from CPAN is a snap (as root):
$ perl -MCPAN -e "install XML::DOM"
(as an example)
See the CPAN faq for more information.

If you get a "Malformed UTF-8 character" crash on a Redhat/Fedora system, you may need to preface nzbperl commandline startup with a "LANG=C" like:
LANG=C perl nzbperl.pl <your other options here>

Usage

For now, you just get the --help summary...although most things are pretty self explanatory. Detailed explanations may or may not appear here in the future.


  nzbperl version 0.6.8 -- usage:

  nzbperl <options> <file1.nzb> ... <file.nzb>

  where <options> are:

 --config <file>   : Use <file> for config options (default is ~/.nzbperlrc)
 --server <server> : Usenet server to use (defaults to NNTPSERVER env var)
                   : Port can also be specified with --server <server:port>
 --user <user>     : Username for server (blank of not needed)
 --pw <pass>       : Password for server (blank to prompt if --user given)
 --conn <n>        : Use <n> server connections (default = 2)
 --ssl             : Connect to server using SSL (secure sockets layer).
                   : May be combined with --http_proxy or --socks_server to 
                   : use a proxy server with SSL.
 --socks_server <s>: Connect using <s> as a socks proxy server. Defaults to
                   : port 1080, but can use --socks_server <server:port> to
                   : use an alternative port.
 --http_proxy <s>  : Use <s> as an http proxy server to use.  Defaults
                   : to port 8080, but can use --http_proxy <server:port> to
                   : use an alternative port.
 --proxy_user <u>  : Authenticate to the proxy using <u> as the username
 --proxy_passwd <p>: Use <p> as the proxy user password (otherwise prompted)
 --ipv6            : Use IPv6 sockets for communication
 --keepparts       : Keep all encoded parts files on disk after decoding
 --keepbroken      : Continue downloading files with broken/missing segments
                   : and leave the parts files on disk still encoded.
 --keepbrokenbin   : Decode and keep broken decoded files (binaries) on disk.
 --dlpath <dir>    : Download and decode all files to <dir>  
                   : (default downloads to current dirctory)
 --dlrelative      : Download and decode to the dir that the nzbfiles are in
                   : (default downloads to current directory)
 --dlcreate        : Create download directories per nzb file
 --dlcreategrp     : Create download dirctories with usenet group names
 --queuedir <dir>  : Monitor <dir> for nzb files and queue new ones
 --forever         : Run forever, waiting for new nzbs (requires --queuedir)
 --postdec <prog>  : Run <prog> after each file is decoded, env var params.
 --postnzb <prog>  : Run <prog> after each NZB file is completed.
 --diskfree <perc> : Stop downloading when dir free space above <perc>
 --redo            : Don't skip over existing downloads, do them again
 --insane          : Bypass NZB sanity checks completely
 --dropbad         : Auto-skip files in the NZBs with suspected broken parts
 --skip <n>        : Skip the first <n> files in the nzb (don't process)
 --med <kBps>      : Set "med" bandwidth to kBps (default is 95kBps)
 --low <kBps>      : Set "low" bandwidth to kBps (default is 35kBps)
 --speed <speed>   : Explicitly specify transfer bandwidth in kBps
 --log <file>      : Log status messages into <file> (default = none)
 --decodelog <file>: Append uudeview output into <file> (default = none)
 --dthreadct <ct>  : Use <ct> number of decoder threads.  Set ct = 0 for single
                     threaded perl operation.  (Note: When ct = 0, downloads
                     will be paused during file decoding)
 --daemon          : Run in background as daemon (use log for status)
 --rcport <port>   : Enable remote control functionality on port <port>
 --retrywait <n>   : Wait <n> seconds between reconnect tries (default = 300)
 --nosort          : Don't sort files by name before processing
 --chunksize       : Amount to read on each recv() call (for tweakers only)
                   : Default = 5k, Can specify in bytes or kb (ie. 5120 or 5k)
 --filter <regex>  : Filter NZB contents on <regex> in subject line
 --ifilter <regex> : Inverse filter NZB contents on <regex> in subject line
 --uudeview <app>  : Specify full path to uudeview (default found in $PATH)
 --nocolor         : Don't use color
 --noansi          : Don't use ANSI characters (text only)
 --noupdate        : Don't check for newer versions at startup
 --help            : Show this screen

  During runtime, press 'h' or '?' to see a list of key commands.
  

If all goes well, you'll see the main screen. You can press 'h' or '?' during normal operation to view the help screen which shows available key commands.

Screenshots

screenshot
8 connections maxed out
screenshot
3 connections throtteld down to 25kBps
screenshot
the help screen
screenshot
Running in Windows under Cygwin
screenshot
snap of 80-col text console

User Submitted Screenshots

screenshot
10 connections at 9.3 megabytes per second!
screenshot
Nice looking KDE konsole
screenshot
Running on OSX Tiger
screenshot
Running on OSX 10.4.1 for x86

TODO list

There's always room for improvement. The following is a list of things I may or may not get around to thinking about implementing. Items that have been implemented have been given strikethru and moved lower on the list.

Email List

nzbperl has an email group/list hosted at google groups. The list can be used to ask for support, submit feature requests, provide patches, submit bug reports, and general nzbperl related discussions.

To view the email list on the web, please visit http://groups.google.com/group/nzbperl.
To subscribe, send email to nzbperl-subscribe@googlegroups.com.
To subscribe, send email to nzbperl-unsubscribe@googlegroups.com.

License

This program is distributed under the terms of the GPL.

Copyright (C) 2004 jason plumb

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.