nzbperl

nzb based nntp/usenet downloader in perl

last updated Nov 25th, 2006
overview |  features |  download |  requirements |  usage |  screenshots |  faq |  todo list |  email list |  license

Latest News / Announcements

(changelog)

Novermber 25th, 2006 - Still kicking. Just wanted to let people know that although there hasn't been a release in the better part of a year now, I still consider this project active. I've been silently banging away on the 0.7.x version and hope to have a source repository available in the near future.

December 31st, 2005: Version 0.6.8 released today.

Sneaking out the new version before the turn of the year. This will be the last release in the 0.6.x series, and it now supports disk free space checking and several important bug fixes. As usual, the changelog has gory details, and the summary is:

The next release will be in the 0.7.x series and will be geared at allowing connections to work on the same file (smaller units of parallelization).

November 14th, 2005: Version 0.6.7 released today.

Major changes:

There are some important bug fixes, so upgrade is recommended!

October 16th, 2005: Email group/list now active.

Thanks to google, nzbperl now has an email list. To subscribe, send a message to nzbperl-subscribe@googlegroups.com. The group should now be used for most nzbperl related discussions, bug reports, questions, feature requests, etc.

September 11th, 2005: another broken release. :)

The 0.6.6 version released earlier today had a silly bug. If you downloaded it earlier today and see a problem about --queuedir requiring an ABSOLUTE path, please re-download. It should be fixed now. Sorry for the hassle.

September 11th, 2005: Version 0.6.6 released today.

The major improvements/changes are:

Aug 9th, 2005: Once again, I released a broken version. 0.6.5 had a problem with --config not working, but I've changed it in place...so please re-download 0.6.5 to get the fix. I suppose it's a typical situation where a program's complexity reaches a certain point and the developer can't test it all. So...if anybody wants to write a suite of unit tests, it sure would be nice. :)

{there are older announcements, but I haven't built a nice way to view them yet}

Overview

Run Linux (or OSX or Windows)? Need a multi-connection newsreader utility that can process an nzb file and autodecode and even do bandwidth throttling? You've come to the right place. :)

I looked around for something that I could use for nzb files on my Linux setup, and I soon discovered limited options. There are progs out there that will download nzb files (nzbget and knzb are apparently the most visible ones), but none seemed to have a key feature that I put importance on: bandwidth throttling. So, like many other one-off hacks and open source projects, this little toy was built to scratch an itch.

Features

This has currently been tested and confirmed working with Debian, SuSE, RedHat, NetBSD, OSX, and Windows (via Cygwin). There's no obvious reason this shouldn't work with other platforms or distros. Feel free to send me your success or failure stories.

Download

Requirements / Dependencies

UUDeview - great program, used by nzbperl to do file decoding.

Perl (version likely flexible), and the following modules:

use IO::Socket::INET;
use File::Basename;
use XML::DOM;
use Getopt::Long;
use Time::HiRes;
use Term::ReadKey;
use Term::Cap;
use Cwd;
-------- optional modules --------
use threads; (only if you want threaded decoding)
use Thread::Queue; (only if you want threaded decoding)
use Term::ANSIColor; (only if you want color)
use XML::Simple; (only if you want remote control functions)
use IO::Socket::SSL; (only if you want SSL)
use IO::Socket::Socks; (only if you want to use SOCKS proxy)
use Net::HTTPTunnel; (only if you want to use an HTTP tunnel)
use IO::Socket::INET6; (only if you want IPv6 support)

On most systems, installing Perl modules from CPAN is a snap (as root):
$ perl -MCPAN -e "install XML::DOM"
(as an example)
See the CPAN faq for more information.

If you get a "Malformed UTF-8 character" crash on a Redhat/Fedora system, you may need to preface nzbperl commandline startup with a "LANG=C" like:
LANG=C perl nzbperl.pl

Usage

For now, you just get the --help summary...although most things are pretty self explanatory. Detailed explanations may or may not appear here in the future.

  nzbperl version 0.6.8 -- usage:

  nzbperl <options> <file1.nzb> ... <file.nzb>

  where <options> are:

 --config <file>   : Use <file> for config options (default is ~/.nzbperlrc)
 --server <server> : Usenet server to use (defaults to NNTPSERVER env var)
                   : Port can also be specified with --server <server:port>
 --user <user>     : Username for server (blank of not needed)
 --pw <pass>       : Password for server (blank to prompt if --user given)
 --conn <n>        : Use <n> server connections (default = 2)
 --ssl             : Connect to server using SSL (secure sockets layer).
                   : May be combined with --http_proxy or --socks_server to
                   : use a proxy server with SSL.
 --socks_server <s>: Connect using <s> as a socks proxy server. Defaults to
                   : port 1080, but can use --socks_server <server:port> to
                   : use an alternative port.
 --http_proxy <s>  : Use <s> as an http proxy server to use.  Defaults
                   : to port 8080, but can use --http_proxy <server:port> to
                   : use an alternative port.
 --proxy_user <u>  : Authenticate to the proxy using <u> as the username
 --proxy_passwd <p>: Use <p> as the proxy user password (otherwise prompted)
 --ipv6            : Use IPv6 sockets for communication
 --keepparts       : Keep all encoded parts files on disk after decoding
 --keepbroken      : Continue downloading files with broken/missing segments
                   : and leave the parts files on disk still encoded.
 --keepbrokenbin   : Decode and keep broken decoded files (binaries) on disk.
 --dlpath <dir>    : Download and decode all files to <dir>  
                   : (default downloads to current dirctory)
 --dlrelative      : Download and decode to the dir that the nzbfiles are in
                   : (default downloads to current directory)
 --dlcreate        : Create download directories per nzb file
 --dlcreategrp     : Create download dirctories with usenet group names
 --queuedir <dir>  : Monitor <dir> for nzb files and queue new ones
 --forever         : Run forever, waiting for new nzbs (requires --queuedir)
 --postdec <prog>  : Run <prog> after each file is decoded, env var params.
 --postnzb <prog>  : Run <prog> after each NZB file is completed.
 --diskfree <perc> : Stop downloading when dir free space above <perc>
 --redo            : Don't skip over existing downloads, do them again
 --insane          : Bypass NZB sanity checks completely
 --dropbad         : Auto-skip files in the NZBs with suspected broken parts
 --skip <n>        : Skip the first <n> files in the nzb (don't process)
 --med <kBps>      : Set "med" bandwidth to kBps (default is 95kBps)
 --low <kBps>      : Set "low" bandwidth to kBps (default is 35kBps)
 --speed <speed>   : Explicitly specify transfer bandwidth in kBps
 --log <file>      : Log status messages into <file> (default = none)
 --decodelog <file>: Append uudeview output into <file> (default = none)
 --dthreadct <ct>  : Use <ct> number of decoder threads.  Set ct = 0 for single
                     threaded perl operation.  (Note: When ct = 0, downloads
                     will be paused during file decoding)
 --daemon          : Run in background as daemon (use log for status)
 --rcport <port>   : Enable remote control functionality on port <port>
 --retrywait <n>   : Wait <n> seconds between reconnect tries (default = 300)
 --nosort          : Don't sort files by name before processing
 --chunksize       : Amount to read on each recv() call (for tweakers only)
                   : Default = 5k, Can specify in bytes or kb (ie. 5120 or 5k)
 --filter <regex>  : Filter NZB contents on <regex> in subject line
 --ifilter <regex> : Inverse filter NZB contents on <regex> in subject line
 --uudeview <app>  : Specify full path to uudeview (default found in $PATH)
 --nocolor         : Don't use color
 --noansi          : Don't use ANSI characters (text only)
 --noupdate        : Don't check for newer versions at startup
 --help            : Show this screen

  During runtime, press 'h' or '?' to see a list of key commands.

If all goes well, you'll see the main screen. You can press 'h' or '?' during normal operation to view the help screen which shows available key commands.

Screenshots

8 connections maxed out

3 connections throtteld down to 25kBps

the help screen

Running in Windows under Cygwin

snap of 80-col text console


User Submitted Screenshots

10 connections at 9.3 megabytes per second!

Nice looking KDE konsole

Running on OSX Tiger

Running on OSX 10.4.1 for x86

TODO list

There's always room for improvement. The following is a list of things I may or may not get around to thinking about implementing. Items that have been implemented have been given strikethru and moved lower on the list.

  • Add option (--badNZB or similar) that can call a script when a "bad" NZB is hit (see google groups thread from beheerder@gmail.com).
  • Ensure that GROUP is issued for servers that require it (how did this get dropped?). Maybe make this an option to disable, and maybe have connections track their current group and only issue it when required (ie. hit new file that only exists in groups outside where we are).
  • Quote/unescape $path when doing df for free space checks
  • Fix "freezing bug" when number of connections goes over screen height ($statuslimit goes negative). See "thegeek" messages in email list.
  • Fix bug that causes --postnzb script to be called even when there are files from the nzb still being downloaded. Caused by "last" smaller/faster file finishing before a bigger/slower one.
  • DTD validation (http://www.newzbin.com/DTD/nzb/nzb-1.0.dtd), enable by default(?), option to override.
  • Create "contrib" section for helpful add-ons that people are submitting (ie. repairDir.pl and others)
  • Auth security (and/or ssl?) for the remote control socket to prevent unwanted baddies.
    Would changing the socket to a fifo help for security or just make things more difficult?
    I think HTTP::Daemon::SSL is perhaps the best way to handle security for this?
    Perhaps this blows up into a mini web UI?
  • 'n' (?) key to switch to screen that allows files to be reordered.
  • Color themes. Probably pretty easy to do, but completely unnecessary. ;)
  • Look into reducing memory usage (undef to no longer needed items)
  • Each connection should download a part-at-a-time instead of file-at-a-time. This is a pretty huge change and will be tricky in places, but it's worth doing.
  • Idle connections should help other connections...especially after multiple servers are supported.
  • Pause key (p?) to stop all downloads, close all connections, and wait for an unpause event before continuing (via reconnect).
  • Nice little apt-get -able Debian packages. How to do Perl module deps?
  • Build in a way to prevent auto retry/reconnect logic (--retrywait=0 maybe)
  • Dump missing parts and/or broken files into an output nzb file for later use (like against another server)
  • After decoding a known-broken file and --keepbrokenbin is enabled, rename the binary to show that it's broken (ie. append .broken to filename)
  • Fetch head first to check article existence before fetching body?
    • Once that happens, what about verifying/changing the size from the nzb?
      Perhaps tricky, but worth doing? Have to adjust all sizes...
  • Timeouts on each connection -- if data not seen in N seconds (minutes), reconnect.
  • Finish downloading any parts in progress and cleanly close server connection with QUIT command when user hits 'q' (Default behavior)
  • Implement 'Q' key (capital Q) to force a less graceful but more timely shutdown.
  • Promote the DECODE_DBG_FILE (decoding debug file) into a commandline option. When given, remove the silent option to uudeview.
  • Command line option to sort by reverse date (oldest parts first)
  • Fix bug related to fetch fail from server (response = (400 xxxx.xxxxx.xxxxxxxxxxx.com: Session timeout.))
  • Consider putting tempfile (.parts files) in /tmp or other customize location
  • Support for multiple servers, with priority levels.
  • Support for remote control bandwidth throttling (via cron) -- not sure yet how to do this, but consider it.
  • Keystrokes to dynamically add or subtract connections. Also more complicated with multiple servers, but valuable nontheless.
  • Implement 's'kip key command (to skip current file...hard with multiple connections)
  • Consider using DBI (SQLite/MySQL/?) for queueing. This could really make things pretty simple and could likely give MUCH better control over things (queueing, ordering, expiring, filtering, whatever). Downside: additional dependency.
  • [ Your reasonable feature here ] :)
  • ...show old/completed todo entries...

Email List

nzbperl has an email group/list hosted at google groups. The list can be used to ask for support, submit feature requests, provide patches, submit bug reports, and general nzbperl related discussions.

To view the email list on the web, please visit http://groups.google.com/group/nzbperl.
To subscribe, send email to nzbperl-subscribe@googlegroups.com.
To subscribe, send email to nzbperl-unsubscribe@googlegroups.com.

License

This program is distributed under the terms of the GPL.

Copyright (C) 2004 jason plumb

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.