Usenet Downloading Guide / Fanzub

[Comfun]

I'm using flexget to grab nzb automaticly
If you can add a parameter in the rss url to download nzb name unfiltered this would be very good.
A dump of historic would be cool for repost attempt.

[Comfun]

Hi again, looks like the parser have some problem with long filename as for exemple http://www.fanzub.com/?q=ACX&cat=anime lot of their releases have very long name and sometimes they are not parsed as they should I think.

Resulting a incomplete nzb file.

See you

[Comfun]

Originally Posted by GHDpro

Sorry about that. But yeah, time for some small improvements.

I've now added a bit of spam detection code that will first check what posters have posted more than 10 files in the past day, and then mark any posts by them smaller than 20 MB as spam.

This should not mark any real posts as spam, even if you say post a 8 MB trailer for "Rockman.EXE" as a RAR file (as long as you don't forget to change the default Yenc-PowerPost settings for name/email, something most spammers do seem to do).

It is possible particularly large spam (like 20+ MB posts) will pass through - if that becomes a problem I'll look into what I can do to accurately detect those when it happens.

Also increased default items returned in the RSS feed to 200 (4x more). This should hopefully not cause to many problems.

Hi again, since few days lot of +20MiB .rar spams are going on in aba as you can see on http://www.fanzub.com/?q=.rar&cat=anime

Maybe we should have something like, if there are less than 5 rar without par2 and the size is less than 50MiB and by the same poster in less than an hours, it should consider it as spam.

See you soon.

[GHDpro]

FYI: I've been a bit busy this week and I also don't have much time right now. I'll try to fix the spam problem this weekend.

The issue with very long subject (filenames) should be fixed (although may not work retroactively for existing nzb files). The cause was that the "subject" field for the "posts" table had a limit of 250 characters. I've increased it to 1000 characters which should suffice.

Edit: forgot to fix the "articles" table as well, doing that now...

[cyberbeing]

GHDpro, how hard would it be for you do add multiple pages to search results. For example, making a rule to return either 1000 results or all results from past 30 days, whichever is larger.

Comfun doing mass uploads combined with someone spamming the anime groups has made it so the current 1000 result limit barely even covers the past 24 hours. It nearly makes Fanzub useless, unless you are searching for something very specific.

Bug: Extremely large multi-GB posts usually time-out when requesting an NZB from Fanzub.

[Thoguht]

As an experiment, I switched from torrenting to Usenet for anime a few days ago. It's working extremely well in terms of download speed, but I'm a little concerned about the long-term reliability of the setup. Apart from the current mass uploading activity (thanks btw, lots of good stuff in there), what about the ordinary day-to-day stuff? Is that being uploaded by many different people or just a few dedicated souls?

[Edit] Having spent the morning playing with this, one suggestion for removing a lot of the spam from at least the Anime listings on Fanzub would be to include an option that checks for an opening square or curly bracket in the filename. Virtually all groups these days seem to put their name or the checksum or something else in brackets, but the spam rarely seems to have this feature. I've tried using a bracket as a search term, but that doesn't appear to work.

[GHDpro]

Originally Posted by cyberbeing

GHDpro, how hard would it be for you do add multiple pages to search results. For example, making a rule to return either 1000 results or all results from past 30 days, whichever is larger.

Comfun doing mass uploads combined with someone spamming the anime groups has made it so the current 1000 result limit barely even covers the past 24 hours. It nearly makes Fanzub useless, unless you are searching for something very specific.

Fanzub wasn't originally meant for browsing but for searching (as is explained in the FAQ). I also can't imagine scanning lists of 1000 files is very convenient.

There are over 15000 posts made in the past month (not counting spam, otherwise it would be over 70000) and displaying those all on a single page would probably crash your browser.

Right now I don't have a good solution for how to only browse new files except maybe the AnimeSuki Beta site (but that won't show licensed content for obvious reasons).

The link between the AnimeSuki Beta site and Fanzub wasn't working in the past few days due to some recent outages causing "autossh" to somehow give up an die. I've now restarted autossh and NZB data is being synced again.

Originally Posted by cyberbeing

Bug: Extremely large multi-GB posts usually time-out when requesting an NZB from Fanzub.

Yeah I know. The amount of data that needs to be processed for say, BD ISO files (20+ GB) is quite a lot, so the script times out. One solution would be to cache the files to disc using a cron job. I'm not sure when I'll have time to implement that though.

Originally Posted by Thoguht

As an experiment, I switched from torrenting to Usenet for anime a few days ago. It's working extremely well in terms of download speed, but I'm a little concerned about the long-term reliability of the setup. Apart from the current mass uploading activity (thanks btw, lots of good stuff in there), what about the ordinary day-to-day stuff? Is that being uploaded by many different people or just a few dedicated souls?

The vast majority of files are being posted by comfun and he is using an automated script. But should he ever decide to quit, I'm sure someone else will take his place.

Originally Posted by Thoguht

[Edit] Having spent the morning playing with this, one suggestion for removing a lot of the spam from Fanzub would be to include an option that checks for an opening square or curly bracket in the filename. Virtually all groups these days seem to put their name or the checksum or something else in brackets, but the spam rarely seems to have this feature. I've tried using a bracket as part of the search term, but that doesn't appear to work.

As the scripts are automated I try to make sure there are no false positives.

I've now added an extra rule: any spammer who uploads more than 10 posts in a single day which average less than 10 files per post and contain the strings ".rar", ".exe" or "XXX" will automatically be marked as spam, regardless of the size or even the number of files in any particular post (so if one odd spam post is 12 rar files, it'll mark it as spam too).

comfun is automatically exempted from any spam checking.

[Thoguht]

It's looking a lot better already, hats off to everyone involved in this momentous project

[cyberbeing]

Originally Posted by GHDpro

There are over 15000 posts made in the past month (not counting spam, otherwise it would be over 70000) and displaying those all on a single page would probably crash your browser.

That is why mentioned splitting results onto multiple pages. 1000 results per page is already a lot. It doesn't necessarily need to be 30 days. I do like browsing for interesting things I missed, but checking daily doesn't always happen. Could you just add a second page of 1000 results? That would be better than nothing, hopefully increasing each category to 48+ hours, and expand narrowed down searches into the 30+ day range.

This ultimately is an issue with the inability to filter down search results. If we could filter by Min/Max post size, Poster, Date range, or even if a file had a CRC in the filename or not, that could help make more effective use of the limited results.

[Comfun]

Thanks GHDpro for taking the time to improve and fix the long subject name.

Even if for now I stoped the posting spree due to dcc leehing ( hehe almost 5k post in 2 days, thanks to AnimeCorpX and Arigatou :d )
I think having a option to see only the last show like in animsuki beta would be good, even if non licensed aren't there, since it's normal it could be taken like fanzub faciliting the acces to licenced stuff.

By the way most of crapy spam ( viral exe and rar ) come from alt.binaries.anime which is almost dead and with a lot of german dub/sub.

Not sure it worth it to keep it, since most of interesting anime post are done in alt.binaries.multimedia.anime and alt.binaries.multimedia.anime.highspeed .

But for people using rss, fanzub offer the possibility to create a feed with specific field so it may be easier to run the feed with anim name + team and res rather than runing the feed for cat anime regexing the anime name + team and res because the anime cat feed may be flooded with 50 eps in the next 10min.

Again big up GHDpro, you really rock.

[Comfun]

I thought about an efficient way for easy search the most interesting recent stuff.
The idea would be to add a block between the search box and the result table with the stuff that has been the most clicked the last 48h.

So people who only want to grab recent interesting stuff can click on the links and see what it is without searching.

Or maybe just a button with last 48H Most clicked and the search engine do the search and list them.

If you already have stored in the db a table to count number of click for items it could be done without too much efforts. If not well I guess it's an idea for future improvement.

[cyberbeing]

Once GDHpro eliminated all the spam, things have been a lot better, but when Comfun does a lot of uploads things break-down. Multiple pages and improved filtering options are still on my wish list. Who knows, maybe you can fill that gap with Animesuki-Beta rather than Fanzub directly, if you integrate them well.

Comfun, when you upload entire series, can you do it in a way so they show up as a single NZB entry (collection) on Fanzub? Not the most practical if only looking for a single episode, but since we are result limited, it would probably help. Edit: ...but I guess that means GDHPro would need to fix the large NZB problem first..

[Comfun]

Sorry that's not possible , at last the only way would be to rollback to the whole serie put in folder and rared before post.

As discuted in abmad , looks like most peoples prefer to have episode posted seperatly instead of being packaged in rar.

Regrouping episode with similar filename except ep number can only be done from fanzub side.

[cyberbeing]

Thanks for the info. I was hoping there was a way to keep multiple split files within a single multi-part post with post/subject name trickery, but I guess not.

That makes yet another thing to add to my wishlist for Fanzub, and likely more important to implement than the others.

[GHDpro]

I haven't implemented counting downloads yet, but as each NZB is served by a PHP file (and still will be even after I finally get to implementing caching) it's quite easy to add. Then displaying a page with the most popular downloads should also be easy. But I can't predict if the page will be useful; that kind of depends on whether new and interesting content is really being hit more often (it also kind of depends on how popular Fanzub is - fortunately that is still increasing).

As for whole-series-posting cluttering the page: it might be possible to group several episodes into one "group" of posts for display purposes. However, that will be a lot of work to detect and then properly display I suspect. It's not something I can implement with just 2 lines of code. So I can't promise any ETA.

As for how episodes should be posted on Usenet in general: the best way without a doubt is to post episodes individually with split files (no RAR!) and PAR2 file based on the original file (so that for QuickPar users it will automatically join the split files). There is no a.b.ma.highspeed.repost group is there? Because that's another option - have a switch to hide all posts (cross)posted to a "repost" group. Or maybe a [repost] tag or something?

[Comfun]

A agree with all you said.
About abmah.repost this group doesn't exist but abmar exist and have his own limitation/rules. (limitation on MiB posted per 24h )
Most of posting spree aren't repost but just some post not on the groups yet but still kinda old post.

Taging them with something should be possible too so it could prevent to list them in the main page when you want to list the last posts.

Something like [BATCH] added in subject for whole serie posted from torrent batch file or irc batch get.

Also if you can count the stuff downloaded trought rss it's also possible to generate download for specific groups/serie so it can help them to be in the top list ( in the cases not enough people would download using the website interface to have something interessent/realistic in the top list )

I think the top list is the most easy thing and efficient thing to implement for let people found the recent ep for the most followed serie without having to search or try to figure out where it is in the last 200 items list.

Also an rss of this top list could help those peoples to easy found what they like and it would make the items in the list more downloaded if they are really interesting.

I really start to appreciate the idea of a top list of 100 or 50 items the most downloaded from the last 7 days.

If the rss download are counted too I think it can be very reliable.
Also having a list of the most recents/search during the last 7 dayscould be interesting to see what people are interested for and maybe post them if they are not available.

This could also be used to start to implement auto completion on search .

[Thoguht]

Time for a retrospective After two weeks of Fanzub and SABnzbd, torrenting now just seem like an over-complex and under-powered way of doing things, and that's without even considering its legal status. The only thing lacking now as far as I'm concerned is a way to control the display of batch postings on Fanzub, so I have to admit that I usually look at another torrent site first to see what's new, and then go to Fanzub.

The idea of initially collapsing batches based on some sort of pattern matching into a single line that could then be expanded if desired sounds like an ideal solution from a conceptual point of view, but yes, obviously more than a few lines of code! If I knew PHP I'd gladly offer to help, but I'm mainly a C/C++/C#/.Net guy.

[Comfun]

Also if you follow a serie you can just use the rss to download it automatcly without having to see what's out or not.

I personatly use flexget running with a cronjob for both windows workstation ( with newsleecher ) and linux server ( with sabnzbd ).

Very easy to install and configure. Then if you don't use the search engine you don't even have to connect to fanzub website everyday.

+ fanzub allow you to make our own rss feed so it's very easy to get what you want as soon it's out and available on usenet.
Just go in the sofa watch it already ^^


flexget config exemple for different series/team http://pastebin.com/XkA1RRqD

even more easy to use flexget exemple using fanzub to filter the stuff http://pastebin.com/y1Qr5d7G

Very easy to use. With that since the posting on usenet script may sometimes have 15min to 1 hours delay with the torrents release and fanzub have around 30min delay with the group, you almost always have the new stuff in less than 1 hours after the releases, without doing anything.

Simple as utorrent rss feed option.

[Thoguht]

Did something change with the way .nzb files are being created? SABnzb is now creating folders with exactly the same name as the file, including the extension. This means my script for copying the files to the parent folder is failing because something with the same name already exists there. I can change my script of course, but was this change in the .nzb intentional?

EDIT: Seems OK now. Can't see why anything should have changed on my PC, maybe Comfun fixed it???

[cyberbeing]

Over the past day or so, someone has been spamming the alt.bin.anime with random passworded junk disguised as fansubs complete with CRCs, but with obviously wrong file sizes.

GHDpro, any way for you to filter it out automatically on Fanzub?

Maybe check that file sizes are realistic against the AniDB api? It would still be nice if we could add exclusions ("-excludedtermhere"), add Boolean logic (AND, OR, NOT) to search queries, or search by poster ID. It is currently easy enough to manually filter is out by searching for split files only, but the spammer could mess that up if they wanted to, considering we don't have exclusions.