Usenet Downloading Guide / Fanzub

[GHDpro]

Thanks for the notice.

The following new anti-spam rule is in effect:

Posts made by the default Yenc poster (=what you get when you don't change the settings) will now be considered spam if there are more than 10 posts in 24 hours and the posts do not contain split files (.001, .002 etc).

This user is pretty much only using RAR files, so all posts he made were marked as spam (=hidden).

For legit usenet posters: the easiest way not to get caught by this anti-spam rule is to simply change the default Yenc poster settings. Alternatively post your fansubs and other media as split files, there really isn't an excuse to use RAR files except for large file collections (not relevant for fansubs - always post multiple episodes individually). Extra compression hardly works on AVI/MKV and the maybe 1-2 MB saved does not weigh up to the extra post-processing downloaders need to do with RAR files.

[GHDpro]

A little update about Fanzub.com:

I'm currently working on a major rewrite of the site. Some features I hope to implement are:
- Counting of NZB downloads (preparation for a "Popular Downloads" feature)
- Caching of NZB files (no more time-outs on huge NZB files)
- Allow full browsing of all categories with paginating (so you can view more than just recent entries)
- Ability to download multiple files as one NZB (saves some clicking)

However the majority of the work is actually "under the hood" rather than visible: not just some of the code is getting rewritten, but the entire database is being reconstructed as well. The reason for this is that the current site has poor article matching, causing lots of duplicate articles in the current database. As a result there are lots of duplicate posts as well.

In addition to better article matching I've also written a far more powerful post "scanner" (script that tries to group a number of articles into one "post"). The old one was fairly simple: it simply chopped some parts of the subject line (certain extensions, part indicators etc) and tried to match different article subject lines mostly on filename. This worked okay for fansubs where each post is a single file; but it completely failed for music posts where each article may contain a completely different track number + title for a single album. The new post "scanner" can handle music posts much better. It will also skip most spam by default as each post must be of a certain size to be added (so single file spam will be skipped).

As the database is about 5 gigabytes (with 5+ million articles) this takes quite a long time to process on my home development machine though, especially considering due to a bugs I have had to restart the process several times. I just had one of those moments: it turned out using array_merge() to merge two lists of message-ids was a really bad idea

Anyway, this is just to let you know that I am working on the site. I will hopefully be able to put the reworked site online in a few weeks time.

[cyberbeing]

Originally Posted by GHDpro

- Ability to download multiple files as one NZB (saves some clicking)

Could you also add the ability to select multiple files and download them as multiple NZBs with a single click, possibly zipped? This may sound like an odd request, but this would actually save me time and avoid the Alt.Binz bug which rarely hangs its PAR2 thread when using multi-file NZBs.

[GHDpro]

I'm not 100% sure about that since I don't know how many people (other than you) would appreciate such a feature.

[Sabo]

Originally Posted by GHDpro

I'm not 100% sure about that since I don't know how many people (other than you) would appreciate such a feature.

I found this forum by luck, and seriously? You're the founder of Fanzub? I've emailed you and asked if you could implement the possiblity to download many nzbs as a single one, and you're working on implemting it. Nice.

[GHDpro]

Sorry about it taking so much time though. The reconstruction of the database + rewrite of the post scanner was taking a ton of time.

I'm now at the stage where I got it working pretty much the way I want, but now I still have to implement some of the features I promised. NZB caching is already working (no matter how big the NZB file, it'll download instantly) and paginating (browsing all pages in a category) is too. The others shouldn't take to much time nor should be to hard to implement, so I hope to be able to put the new site online "soon".

Here are some screenshots though:

The "Music category is kind of screwed up" bit is gone from the FAQ, because music posts now simply work:


And a comparison of the DVD category:

BEFORE (current v1):


AFTER (beta v2):


Apart from the missing spam in the beta site (which I didn't remove manually; the new scripts are better at detecting it) you can also see that posts with very short (and undescriptive) filenames now show part of the post description to make them easier to recognize (this also works wonders for the "Raws" category). Last you can see it picks up quite a few posts that the old site completely missed.

[GHDpro]

And it is done!

Finally Fanzub v2 is done. As I mentioned before, the main thing that took a long time to get right was the complete reconstruction of the database. The new header & post matching scripts are much "smarter" than before, which means more posts are detected (completely), more post are complete (especially large ones) and there are no more duplicates.

The immediate effect of this is that the Music category now works as intended; no longer are there any music posts with just PAR files and no tracks The new post "scanner" also allows for a completely new category to be added: Hmanga (downloads & descriptions NSFW), which I didn't bother with because of the problems the old code had with the music category.

There is one downside to this completely reworked database: the post ID number are reset. Before the most recent post ID number was somewhere 300K+, now it is around ~170K partially because the new post "scanner" is more effective at catching spam. That might cause some issues with applications that use download history (of previous NZB files) to determine what they have downloaded and what NZB files are new. Hopefully the problems will not be all too severe. If you only use a browser to access the site (and not say, some kind of RSS reader) you should hardly notice this.

The other new features are (as mostly mentioned before):

• Paginating
  You can now simply browse everything in all categories. Note that browsing search results is limited to 5 pages due to technical reasons (Sphinx Search limitation) but then I'd recommend to simply refine your search query if that isn't enough.
  
  
• Multi-NZB downloads
  You can download multiple NZB files at once by selecting the files you want and clicking the "Get NZB" button at the bottom of the page. To select a file simply click on the row. To select a range, drag your mouse across multiple rows. To select all files, double click the file list. Do all of these actions again to deselect files in the same way. Due to memory limitations it may not be possible to say, download all "BDMV" raws in one go. But then again you'd need a pretty fast connection and huge HDD for that anyway
  
  
• NZB caching
  While the old code for generating NZB's was simply very inefficient (and the rewrite of this speed things up considerably already) I've also added NZB caching. This means that in 99.99% of all cases the NZB file does not need to be assembled from the database and will download instantly.
  
  
• Export feed
  If you want to syndicate Fanzub content, there is a new "export" feed that allows you to retrieve information from the Fanzub database more easily. More information is in the FAQ.
  
  
• Faster updates
  On the old site new headers from newsgroups were downloaded every 20 minutes from 3 different Usenet providers. On the new site new headers are downloaded every 5 (!) minutes from 4 different providers.

All in all the changes might not seem like much, but overall I hope these changes are appreciated.

Update
Oh dear, the new post IDs seem to be causing chaos, at least what I'm seeing in the download counts table and in my own SABnzbd+ client. Because the URLs for matched RSS entries are different SABnzbd sees them all as new downloads, and if I hadn't stopped it myself it would have gone on to download gigabytes of stuff I had already downloaded.

I sincerely hope nobody is to severely impacted by this. If you haven't done so already, it might be best to check your download queue now if you use automated RSS downloading.

[cyberbeing]

Can you make a minor change so Middle Button Mouse Click doesn't select entries? I'm having an issue with it selecting a bunch of random stuff when I'm just trying to scroll the page with Auto-Scroll.

Possibly also add a button at top and bottom for 'Select All' & 'Deselect All' (as an alternative to mouse click dexterity), and 'Invert Selection' (alternative way of selecting things), and add the 'Get NZB' button to the top as well (so you can do selections starting at the bottom of the page to the top).

Also add an advanced search?

Search in specific newsgroups only.
Search by poster.
Search by minimum size.
Search by maximum size.
Search by age.
Search only posts with an NFO (and allow NFO download/view from index?)

Is an Advanced Search functionality like this easily possible with your Fanzub 2.0 search implementation?

[GHDpro]

My main problem with some of your suggestions is that I want to keep the interface as clean and uncluttered as possible. This basically means dumbing it down a bit and not showing too many options that a lot of people might never use.

That said, some of the suggestions you make could be implemented through some kind of "Settings" page (cookie based, so no account registration necessary).

Originally Posted by cyberbeing

Can you make a minor change so Middle Button Mouse Click doesn't select entries? I'm having an issue with it selecting a bunch of random stuff when I'm just trying to scroll the page with Auto-Scroll.

What mouse action is associated with the middle mouse button (see control panel)?

If it is because you are using the touchpad of a laptop with scrolling & tapping enabled I'd recommend getting a real mouse, or disabling tapping (which I personally absolutely hate -- I still use touchpads strictly the classic way).

Originally Posted by cyberbeing

Possibly also add a button at top and bottom for 'Select All' & 'Deselect All' (as an alternative to mouse click dexterity), and 'Invert Selection' (alternative way of selecting things), and add the 'Get NZB' button to the top as well (so you can do selections starting at the bottom of the page to the top).

Select all / Deselect all is done easiest by double-clicking on the list of files.
I'm not sure how useful invert selection is.
"Get NZB" on top could be an option for a "Settings" page.

Originally Posted by cyberbeing

Also add an advanced search?

Search in specific newsgroups only.

Fanzub represents newsgroups as categories and you can already search those. I don't see much point in allowing searching of specific newsgroups while the rest of the site interface tries to hide them.

Originally Posted by cyberbeing

Search by poster.

There is already a hidden feature to do this, though it is not very practical to use because you need to know the "author ID" of a poster and that isn't publicly displayed anywhere.

But for example this page lists all posts made by comfun:
http://www.fanzub.com/?q=&author=2625

Anyway, what would be a practical reason why you would want to do this?

Originally Posted by cyberbeing

Search by minimum size.
Search by maximum size.

I don't really see the point. Again, why would you want to do this?

Originally Posted by cyberbeing

Search by age.

This could be a "Settings" page option.

Originally Posted by cyberbeing

Search only posts with an NFO (and allow NFO download/view from index?)

This I'm afraid is not easy to do. Well searching for posts with NFO is quite possible, but download/viewing would require quite a bit more code. It is however one thing I certainly would be willing to implement in the future, though considering the work required I can't promise any ETA.

Originally Posted by cyberbeing

Is an Advanced Search functionality like this easily possible with your Fanzub 2.0 search implementation?

The search implementation didn't change that much, and some of what you ask is possible, except in some cases I wonder if such options would be used very often.

Anyway, the way I envision the site should be ideally used as is that you start by crafting a search query as specific as possible for what you want. So for example you don't search for "index" and then mark all the files by "Hatsuyuki" in "1280x720" that you want, but you search for "hatsu index 720" instead. Then you double-click to select all files and then individually deselect the files you don't want (for example the v1 for files with a v2 available).

[shinigamidono]

I didn't think my favourite website on the entire internet could get any better... but now it has keep up the good work

[cyberbeing]

Originally Posted by GHDpro

What mouse action is associated with the middle mouse button (see control panel)?

If it is because you are using the touchpad of a laptop with scrolling & tapping enabled I'd recommend getting a real mouse, or disabling tapping (which I personally absolutely hate -- I still use touchpads strictly the classic way).

This is a desktop computer with a real mouse (MX518) using the generic Microsoft HID driver. No button configurations, so the middle mouse button is... the middle mouse button. In Firefox the middle mouse button action is autoscroll and I use it often on long pages instead of the mouse wheel since it's quicker. Please disable selection with the middle mouse button. Holding down the left mouse button and dragging already scrolls the page if you need to make a large selection. When I use the middle mouse button to scroll quickly (instead of using the scroll wheel), I just want to scroll, not select stuff.

Originally Posted by GHDpro

Select all / Deselect all is done easiest by double-clicking on the list of files.

It's not that easy. As it sometime requires double the amount of clicks or more, depending on what it registers your click as. I can't always get it to select/deselect all on the first double-click attempt. Sometimes it requires 3 clicks or more.

Originally Posted by GHDpro

I'm not sure how useful invert selection is.

It can be very useful at times when you want nearly everything on a search result page, but there are a couple items you don't mixed in. Make a smaller selection, hit the Invert Selection button and save a few clicks.

Originally Posted by GHDpro

Fanzub represents newsgroups as categories and you can already search those. I don't see much point in allowing searching of specific newsgroups while the rest of the site interface tries to hide them.

If you know what you want is always posted to a specific newsgroup(s) (which can span multiple categories depending on the groups you pick) and you want to narrow things down more than the given categories or give yourself more relevant search results.

Originally Posted by GHDpro

There is already a hidden feature to do this, though it is not very practical to use because you need to know the "author ID" of a poster and that isn't publicly displayed anywhere.

But for example this page lists all posts made by comfun:
http://www.fanzub.com/?q=&author=2625

Anyway, what would be a practical reason why you would want to do this?

To find items uploaded by the same person because you like their uploads?
To check what (if anything) a person has uploaded in the past when you see a questionable upload?

Originally Posted by GHDpro

I don't really see the point. Again, why would you want to do this?

Again to narrow down results, in combination with more vague keywords (like if something is mislabeled). You know what you want is above a certain size, below a certain size, or within a size range, and that eliminates a bunch of junk from your search results.

Originally Posted by GHDpro

The search implementation didn't change that much, and some of what you ask is possible, except in some cases I wonder if such options would be used very often.

Most of these features I'm requesting are standard options available on other Usenet indexes.

One last thing I've mentioned before but almost forgot. Boolean logic and other similar things:

Keyword exclusions
-unwantedkeyword

Exact phrases
"phrase I want"

Search for two phrases but not both in the same file
keyword1 OR keyword2

I use all of the above daily on Google to prefect my results, don't you? Nowadays I go crazy at the massive efficiency loss when I'm restricted to basic keywords only. Librarians and anybody else who does serious research for their job, would also praise the usefulness of advanced google-fu functionality in search engines.

Originally Posted by GHDpro

Anyway, the way I envision the site should be ideally used as is that you start by crafting a search query as specific as possible for
what you want.

It's extremely rare for someone to approach a search engine or index, always knowing exactly what keywords they need to produce perfect results. Yet most everybody will have at least a basic keyword of what they want and once they see the results, they'll be able to narrow it down. Exclusions help narrow down your results, when the keywords you provided are too broad or unsuitable. Yes you can get by with only keywords, but it can be extremely inefficient, often involving more queries than otherwise necessary.

As long as Fanzub lacks some of the standard features expected on any Usenet index, you are artificially restricting its usefulness. This prevents it from becoming a one-stop Usenet Index, even for the content you specialize in. When I go to Fanzub to look for something, I hate being forced to go back to other indexes to search for the stuff Fanzub specializes, only because I can't make Fanzub do what I want. Fanzub is great at what it does do, but the extremely basic functionality and simple layout is restrictive to more advanced users. There is no reason you can't have both basic features and advanced features for those who want/need them, correct?

[GHDpro]

Originally Posted by cyberbeing

This is a desktop computer with a real mouse (MX518) using the generic Microsoft HID driver. No button configurations, so the middle mouse button is... the middle mouse button. In Firefox the middle mouse button action is autoscroll and I use it often on long pages instead of the mouse wheel since it's quicker. Please disable selection with the middle mouse button. Holding down the left mouse button and dragging already scrolls the page if you need to make a large selection. When I use the middle mouse button to scroll quickly (instead of using the scroll wheel), I just want to scroll, not select stuff.

Okay, I wrote my previous (and current) reply on laptop, which doesn't have a middle button, so I'll have to check my desktop. But programmatically there is no "middle button" -- so I hope the action associated with it isn't something critical.

Originally Posted by cyberbeing

It's not that easy. As it sometime requires double the amount of clicks or more, depending on what it registers your click as. I can't always get it to select/deselect all on the first double-click attempt. Sometimes it requires 3 clicks or more.

While it is possible to accidentally deselect the one row you are doube-clicking I still find that easier than scrolling to a specific part of the page to do the select all/deselect all.

Originally Posted by cyberbeing

It can be very useful at times when you want nearly everything on a search result page, but there are a couple items you don't mixed in. Make a smaller selection, hit the Invert Selection button and save a few clicks.

I'd personally select all first, then deselect what I don't need. Same amount of clicks. But it might be an idea to enable this through a "Settings" page.

Originally Posted by cyberbeing

If you know what you want is always posted to a specific newsgroup(s) (which can span multiple categories depending on the groups you pick) and you want to narrow things down more than the given categories or give yourself more relevant search results.

Mmm. Sorry, but unless there are more people requesting this I don't see the point much.

Originally Posted by cyberbeing

To find items uploaded by the same person because you like their uploads?
To check what (if anything) a person has uploaded in the past when you see a questionable upload?

See above answer mostly. However it would not be to hard to make the username on the "Details" page clickable (and exose the hidden author search feature). This would then allow you to more easily search by poster.

Originally Posted by cyberbeing

One last thing I've mentioned before but almost forgot. Boolean logic and other similar things:

Keyword exclusions
-unwantedkeyword

Works. The only limitation is that a search query can't contain only exclusions. But searching for "{your favorite show} -xvid -480" works.

Originally Posted by cyberbeing

Exact phrases
"phrase I want"

Does not work, I'll have to look into this.

Originally Posted by cyberbeing

Search for two phrases but not both in the same file
keyword1 OR keyword2

Works, but not with "or" but rather with "|" (pipe character). Those familiar with programming will know what I mean. Try searching for "naruto | fairy".

Originally Posted by cyberbeing

It's extremely rare for someone to approach a search engine or index, always knowing exactly what keywords they need to produce perfect results. Yet most everybody will have at least a basic keyword of what they want and once they see the results, they'll be able to narrow it down. Exclusions help narrow down your results, when the keywords you provided are too broad or unsuitable. Yes you can get by with only keywords, but it can be extremely inefficient, often involving more queries than otherwise necessary.

Well I wasn't expecting someone to get the search query right the first time, it takes some experimenting to filter it down to only those things you want.

Originally Posted by cyberbeing

As long as Fanzub lacks some of the standard features expected on any Usenet index, you are artificially restricting its usefulness. This prevents it from becoming a one-stop Usenet Index, even for the content you specialize in. When I go to Fanzub to look for something, I hate being forced to go back to other indexes to search for the stuff Fanzub specializes, only because I can't make Fanzub do what I want. Fanzub is great at what it does do, but the extremely basic functionality and simple layout is restrictive to more advanced users. There is no reason you can't have both basic features and advanced features for those who want/need them, correct?

I still wonder how many people besides you are really missing some of the features you mention. I'd figure 99% of visitors doesn't miss them.

Having said that a slide-out "Advanced Search" like www.nzbindex.nl might be worth implementing (without messing up the "clean look"). However I can't promise any ETA.

[GHDpro]

Originally Posted by cyberbeing

This is a desktop computer with a real mouse (MX518) using the generic Microsoft HID driver. No button configurations, so the middle mouse button is... the middle mouse button. In Firefox the middle mouse button action is autoscroll and I use it often on long pages instead of the mouse wheel since it's quicker. Please disable selection with the middle mouse button. Holding down the left mouse button and dragging already scrolls the page if you need to make a large selection. When I use the middle mouse button to scroll quickly (instead of using the scroll wheel), I just want to scroll, not select stuff.

Posting this from my desktop [and I can replicate your problem; using a Logitech M510 here]

I think it might be fixable:
http://api.jquery.com/mousedown/
(see paragraph with most text in it)

I'll have to do a bit of experimenting though.

[cyberbeing]

Originally Posted by GHDpro

While it is possible to accidentally deselect the one row you are doube-clicking I still find that easier than scrolling to a specific part of the page to do the select all/deselect all.

I still think at least a 'deselect all' button would be useful. Since I am having issues getting it to deselect when clicking. Part of the problem is it always requires you 'Select all' first before it let's you 'Deselect all'. The other part of the problem is your single click will select something else if you don't do the double click on the selection.

Maybe if you made it always default to doing 'Deselect all' first if you have anything selected, but do 'Select all' first if you have nothing selected, it would work better.

Originally Posted by GHDpro

Mmm. Sorry, but unless there are more people requesting this I don't see the point much.

I hear you this, that this would a niche feature. I do use this feature occasionally on general-purpose indexes, but it's probably not as useful on the more narrow categories Fanzub covers. If you ever did implement it, you may as well just add raw newgroup result browsing. Not really a priority feature.

Originally Posted by GHDpro

See above answer mostly. However it would not be to hard to make the username on the "Details" page clickable (and exose the hidden author search feature). This would then allow you to more easily search by poster.

I would really like something like this. As part of it, could you allow author=XXXX in the search query, or is that not possible?

Originally Posted by GHDpro

Works. The only limitation is that a search query can't contain only exclusions. But searching for "{your favorite show} -xvid -480" works.

Could you workaround this by allowing a wildcard * as the search term with exclusions?

Originally Posted by GHDpro

I still wonder how many people besides you are really missing some of the features you mention. I'd figure 99% of visitors doesn't miss them.

Probably, but the same is likely true for Google's hidden and advanced functionality. Even if hardly anybody uses them, it doesn't make such things any less useful. A few of these things would fall under very rarely used, but critical when you do actually have a need for them.

Originally Posted by GHDpro

Having said that a slide-out "Advanced Search" like www.nzbindex.nl might be worth implementing (without messing up the "clean look"). However I can't promise any ETA.

When I was making my "Advanced Search" suggestions, I did have something like that in mind. The two guys that run that site are actually pretty nice and friendly. If you ever get stumped, they may be willing to give you some tips and suggestions.

[GHDpro]

I've implemented the following enhancements/fixes:

• Drag-select now only works with left mouse button
  So no more trouble with middle mouse button (auto scroll).
  
  
• You can search in three specific database fields: @subject, @poster* and @stats @files
  This means basically you can search for specific posters or for files with only specific filetypes (like NFO).
  
  But please note the following: category matching is only done after the search results are retrieved from the search engine (Sphinx Search). This means that if for example you search for "@files nfo" in the "DVD" category, the search engine will return a list of 1000 posts with "nfo" in the "files" field, and that list will only then be reduced to only posts from the "DVD" category. The result is that the returned list is very small (because 99% of all posts with a .nfo file are in the "Anime" category). To bypass this problem a bit, don't make your searches that generic.
  
  
• You can instantly search for posts made by specific posters by clicking their name in the "Details" view
  No need to copy and paste.

*) as a result of this I've actually ripped out the previous hidden "author" option because it is now redundant.

Originally Posted by cyberbeing

Could you workaround this by allowing a wildcard * as the search term with exclusions?

Unfortunately no. A search must have an "inclusion" in order to contain an exclusion, and the "inclusion" keyword cannot be a wildcard.

But the inclusion keyword can be very generic, like for example -xvid @poster comfun

[cyberbeing]

Thank you for fixing the auto-scroll issue.

So @stats is anything after Files: ?

It seems the @poster command is a bit buggy when the poster's name contains an @ sign.
For example, this returns the results of two different posters.

[GHDpro]

Originally Posted by cyberbeing

Thank you for fixing the auto-scroll issue.

So @stats is anything after Files: ?

Yes.

Originally Posted by cyberbeing

It seems the @poster command is a bit buggy when the poster's name contains an @ sign.
For example, this returns the results of two different posters.

I've put a little filter on the poster name. It isn't perfect, but should work.

[GHDpro]

Originally Posted by cyberbeing

Thank you for fixing the auto-scroll issue.

So @stats is anything after Files: ?

Considering your confusion I've decided to rename this field to @files, which makes more sense (the stats name is how the field is named in the database).

As of now this means the old field name @stats no longer works.

[Benni-chan]

wow, thank you for your work. i really like the fanzub site and use it regularly.
(and now i know, why sabnzbd started to download all those files again. first I thought it was some bug in sabnzbd *g*)

btw. i believe a lot of you guy are using sabnzbd to download the fansubs. i wrote a little autoprocess script, which uses the anidb api to rename the files and move them to a target directory.

more infos here: https://github.com/Benni-chan/SABnzb...-for-Anidb.net
(i'll do some updates in the future, especially if there are any wishes )

[usiry]

Hiya. Hope you guys won't mind me asking a few question on usenet. Only took notice of the service today LOL did a bit of read up but was kinda confused i guess. As i just moved countires and am currently in a country with super slow connection speed and very low quota 6-15GB........ usage so was looking to an alternative to BT, sadly enough the speed is so bad that irc also can't work lol. its takes 2-4 hours to dl 100mb using web browser with tons of drops.

Q: Is it a good idea for me to use it? Currently i have 2 internet connection one is 6gb quota with 7mbps speed and the other is 15 gb with speed worst then my 6gb connection and drops alot only good thing is mmo seems to runs almost smoothly

Q: Is it possible to use it over my office network? it won't messed up the other user connection?

Q: alot of the provider seems to mention increased speed? But i assumed your are still kinda locked by your own ISP speed? both my isp will drop the speed to almost 128kbbps once you hit the quota so i was wondering if the increased speed is locked to isp or not

Q: is there any upload hidden or not? since quota = dl +ul

Last Q: Does all usenet provider access the same thing? I found giganews while looking up usenet but kinda confuse does if all usenet connect as i notice that you mention " problem of finding files is the problem that your Usenet server may be missing one or more parts of the file."

sorry for the trouble