Forum - View topicRelease Scraper Completely Broken Unable to Add Releases
|
Goto page 1, 2, 3 Next |
| Author | Message | |||
|---|---|---|---|---|
|
BigOnAnime
Encyclopedia Editor
 Posts: 1263 Location: Minnesota, USA |
|
|||
|
See the following thread for more info on the extent of what's wrong with the release scraper.
animenewsnetwork.com/bbs/phpBB2/viewtopic.php?t=3053645 Simply, the release scraper has been broken for months now for multiple sites (it was a problem before the domain hacking), and now it's to the point where it's impossible to add any releases except manually, which is not able to be done by regular users. Months worth of releases are now currently missing from the encyclopedia as a result of this problem. Is it possible for this to finally be fixed? I sent Dan42 a PM about this issue about a month ago and didn't get a response like I did the last time I brought this issue up via PM in hopes it would finally be addressed. |
||||
|
||||
|
BigOnAnime
Encyclopedia Editor
Posts: 1263 Location: Minnesota, USA |
|
|||
|
Bump, the issue is still here, and the number of releases missing keeps growing.
|
||||
|
||||
|
Calico
Posts: 383 |
|
|||
|
Yeah, I just tried to add another release, and it's still broken. Are there any plans to fix this? And are there any reasons why the release scraper just broke like it did?
|
||||
|
||||
|
BigOnAnime
Encyclopedia Editor
Posts: 1263 Location: Minnesota, USA |
|
|||
|
Bump, it's still broken, and now there's only 3 releases left on the upcoming list.
animenewsnetwork.com/encyclopedia/releases.php?format=video |
||||
|
||||
|
Dan42
Chief Encyclopedist
 Posts: 3793 Location: Montreal |
|
|||
|
BigOnAnime, I want to thank you a lot for the bumps and your politeness and patience. I keep thinking I have to get to this soon and I keep getting buried in other urgent things, and at the same time I sort of remember the last time I looked at the issue and it was ugly, with some sites blocking our scraper and others being javascript-only. I'll try to get to this as soon as I can. Please slap me if I don't.
|
||||
|
||||
|
Shiroi Hane
Encyclopedia Editor
 Posts: 7584 Location: Wales |
|
|||
|
Amazon should be the easiest fix since it is, at least sometimes, pulling data from the API after a release is added manually. The wizard still fails 100% of the time because it can't reach or parse the actually page, but the API data alone ought to be sufficient to at least build a partial entry (my personal API access is broken because they've changed the way partner accounts work, so I can't recall what, if anything, is only available via the web).
|
||||
|
||||
Spoofer
 Posts: 356 Location: NY |
|
|||
|
I finally updated my ANN Collection list with all my purchases throughout the past year. I had to custom add placeholders for almost half of them. Really sad to see this happen to ANN, as I've said before, I used to rely on the ANN encyclopedia's release accuracy when deciding which releases to purchase (and regularly used the affiliate links), as well as to catalog my collection. Obviously this problem just gets worse and worse over time as more and more titles will never be accounted for.
Who's even manually adding titles to begin with? It's admirable that someone's even trying, I suppose, but it's so hit or miss. It would be great if, even if the scraper itself doesn't get fixed any time soon, some alternate solution could be found for the community to chip in and start trying to undo the gap year's damage as far as maintaining a consistent release database goes. Perhaps granting some trustworthy community members with a passion and a vested interest in the release database the ability to manually add entries as well, or, IDK, something so that this aspect of ANN doesn't just crumble away completely. The scraper could never account for releases from projects such as KickStarter or Indiegogo anyway, and I feel the encyclopedia should chronicle those releases as well so that people know what's out there, especially when many of those items are still available via Slacker Backers or company websites. Same deal with rare hybrid items like the Tekken: Blood Vengeance cgi film, where the DVD was easy enough to be added since it contained just the film, but the BD was released as "Tekken Hybrid" and featured both the film and a couple of games on the same disc and couldn't be added via the scraper. |
||||
|
||||
|
Shiroi Hane
Encyclopedia Editor
Posts: 7584 Location: Wales |
|
|||
|
I don't know who else, if anyone, is adding releases manually. I do most of it between calls in work, which means Funimation releases are difficult since the best source, their web store, is inaccessible from the UK without using either a VPN or fiddling around with Google Cache. Prior to the scraper going down I concentrated on releases that couldn't be added with the scraper like TRSI exclusives, although that was less of a problem after their site update. I also moved into ebooks since they were commonly neglected by users when adding the physical books. I also, obviously, concentrate on things that I own or that personally interest me, or just happen to be looking into at the time (I spent days adding Seven Seas releases after adding their latest new title announcements to the system). Yen Press is honestly the best because everything I need to build a release is right there on their business portal. It is also a full time job on its own with both the volume of releases, the fact that they have a habit of changing release dates without notice and the fact that you need to check back occasionally to see if they've added missing cover art, volume titles and descriptions etc.
If there's anything that can't be added with the scraper (when it is working) you can post to the encyclopedia forum and request it. If it is something you have, then providing a barcode (UPC, EAN or ISBN) at least is beneficial (and essential if it is something not sold by regular retailers). If something is Kickstarter-exclusive and doesn't have a barcode then it not possible to add with the current system, at least not without the direct intervention of someone higher up than me. While there is now a workaround for ebooks that don't have barcodes (yay, more work!), we also can't add BookWalker exclusives at present either. If there is anything you want added as a priority, let me know, but try to keep the volume down or it will just become another endless task for me to put off till later... |
||||
|
||||
|
Spoofer
Posts: 356 Location: NY |
|
|||
|
No no, I mean it's not a personal issue for me in terms of wanting it merely so I can easily add items that I already own to a collection list. That would be pretty petty.
What my primary concern is, I'd just love for the database in general to once again become reliable, for me and everyone else. I regularly disappear from the fandom for months or years at a time, then I come back, and often go through my Seen or Want to See lists and click through various titles to see if any new releases have come out that I can pick up during sales. ANN's always delivered in this regard in terms of a high degree of accuracy of what's actually out there. And I spend equal time buying the latest and greatest on RightStuf, and trawling eBay for good deals on old limited edition collector's stuff I again know exist by browsing the release database. Or I can often use an encyclopedia page to compare releases, like for older ADV titles where some collections contain extras and some don't, etc. So it's always been a really handy database for me to base my buying habits on, and as I said I always used the referral links as much as possible, if that actually brought you guys any extra revenue. But this was the first year that I missed out on ordering various stuff during sales because ANN hadn't had stuff listed and I had no idea new releases were out. For example, the Record of Lodoss War BD, which I remember being announced years back as on the horizon, but without seeing it listed under ANN's Encyclopedia when I bought a ton of stuff during a big RS sale this year, I never knew it was out to add it to my cart. etc. I mean now I basically just cross-reference stuff I'm interested in with with RightStuf itself, but that only works with current releases. Or other sites like DVD Aficionado or DVD Profiler. I'd just love for ANN itself to maintain reliability out of principle, is all, or pride that you guys remain at the forefront of accuracy. I basically shun sites like MAL since its database is so piracy-oriented (as are the bulk of its users), so I always loved the contrast that ANN was always so reliable from a US retail standpoint. And I know it's prolly a ton of work without the scraper working as it should, so yeah, obviously I don't expect a single person to do all of it. Sounds like a nightmare adding them all manually as well, so kudos for having done as much as you have. TLDR, again, I just wish there was a way, any way, either via the scraper or some decision to allow the community to chip in to some degree once again, to get it back up and reliable again. I figure it's better to have barebones entries that at least list the title and general date of a release, even if the page it links to has very little manually-entered information at all, compared to having no record of a release at all. Which again, my earlier point was, of the stuff I'm interested in (which is probably more along the mainstream side of things), at least 50% seems to be missing from the database since this has been an issue, and other people like me who discover releases this way might likewise have no idea releases are available. But maybe I'm a rare individual who always relied on ANN for this, IDK, heh.  ;;
|
||||
|
||||
Double Mangekyo
 Posts: 180 |
|
|||
|
Any progress on this issue? Or same as ever?
Adding releases was one of my favourite things to do on here. If it ever gets fixed, might I suggest the wizard let us pick between GN / ebook / novel / artbook / fanbook when we add something? It was pretty common for novels/ebooks to be recognized as GNs from what I remember... |
||||
|
||||
horseradish
 SubscriberPosts: 574 Location: Bay Area |
|
|||
I also rely on ANN to know if a title has had a release in the Anglosphere. Before the scraper broke, there were few gaps in the anime side from what I could tell, with the occasional old DVD and VHS missing. Manga was more noticeable, but a title tends to have several volumes in physical and ebook form. I didn't realize how convenient checking ANN was until I had to search retailers and company websites to make sure.
BDs and BD/DVD combos would be mistaken as DVDs as well. I hope the scraper gets fixed soon or a workaround is given for contributors, since there will be more out-of-print releases that are harder to find as time passes. Last edited by horseradish on Wed Apr 28, 2021 1:26 am; edited 1 time in total |
||||
|
||||
|
Double Mangekyo
Posts: 180 |
|
|||
|
So... any updates on the scraper? Or is it still dead and will stay dead for the foreseeable future?
|
||||
|
||||
|
Dan42
Chief Encyclopedist
Posts: 3793 Location: Montreal |
|
|||
|
Sorry it's taking so long, I just have a lot of urgent tasks right now, but I want to get to the scraper as soon as possible. I really want to.
|
||||
|
||||
|
Double Mangekyo
Posts: 180 |
|
|||
|
This is gonna become a meme for me, I can already tell. But I figured I'd at least bump this topic again... Not that I'm hopeful at this point...
|
||||
|
||||
|
Spoofer
Posts: 356 Location: NY |
|
|||
|
(bump)
|
||||
|
||||
 |
All times are GMT - 5 Hours |
|
|
|
Powered by phpBB © 2001, 2005 phpBB Group