« March 2007 | Main | September 2007 »
June 30, 2007
anri-chan and total independence
it's been over two years since m2k2sda became monetarily self-sufficient.
money changes hands at least once every month, as i have to pay the $99 monthly m2k2sda server bill. when the google ads on the site earn us more than $99 a month, then sda requires no outside investment to continue to operate so long as it is continually updated (it's thought that otherwise, site traffic, and therefore ad revenue, would go down, while the server bill would remain the same, thereby reversing the aforementioned positive trend).
unfortunately, the amount of labor grenola, mike, snow, radix and i pour into sda, if paid, would equate to a sum much, much larger than $99 a month. therefore, when i declared sda monetarily self-sufficient two years ago, i was referring only to actual money earned and spent.
now it's time to take care of the rest.
for years, automating my job was a hopeless fantasy. the number of correct decisions required to encode material for the site was simply too high, and i knew of no way to have the computer make them for me. anyone who has seen the knowledge base and the tech support forum knows that asking inexperienced people to do my job as i do it is a losing proposition.
so i tried instead to make sda's revenue grow - to make the site pay us back for our work on it. i learned many things from my adventures trying to sell sda content on dvd, but probably the most significant was that it doesn't work: sda may never generate enough revenue to pay its workers, and sometimes i think it's better that way.
but the fundamental problem remained: sda needs people to add new content, and if i died today, i thought to myself, would someone be there to replace me? i decided that even though i can't control my mortality, i can control how i will be seen after the fact.
little by little i chipped away at the problem, making generic.avs into a highly concentrated, library-driven block of boolean action. generating the scripts for new projects became a matter of enabling or disabling variables. now one script could do it all.
but there still had to be someone there to make decisions based on the content of the source video. i had automated the ntsc/pal distinction, but the rest of them seemed hopeless: how should i tell the computer to check whether every second or every third frame is the same as the previous? and even if this could somehow be done - what about the games with a varying framerate?
therefore, i created the concepts of d (dimensions), f (framerate) and fdp (framerate decimation paradigm) to quantify the differences between games. if a database of every game's d, f and fdp could be created, i reasoned, then surely some sort of script could be written that would take those values and configure generic.avs, reducing my job to reading numbers out of list and typing them in to a window. even if the game in question weren't yet included in the list, it could be added by any tech support staff member after seeing a brief, automatically-generated sample of the game's video. it would be the next best thing to full automation.
as many of you are already aware, i decided to call this script anri-chan after the anime character who (like mickey and goethe before her) used magic to create numerous inferior copies of herself in a failed attempt to automate her job. thus, i am anri-chan, and anri.bat is an inferior copy of me. (when i began the project, i wasn't even aware that dgindex accepted arguments on the command line, and so i thought that automating the steps involving the user interface might be impossible, thereby dooming it all to failure.)
but these open source tools came through for me in the end, and so i was able to produce a proof-of-concept much more quickly than i had anticipated - and even better - it worked: other people were actually encoding acceptable video using anri-chan!
even vista couldn't stop me.
and as if that weren't enough, the free, open source model impressed me yet again today, as b-man in the sda forum went and wrote the anri-chan encoding settings manager for me. i have always said that it is not only me who can do this (just that i was the first to be chosen), and now i have been proven right. it really is true: if you build it, they will come.
i will finally make sda eternal by taking away the cost of its maintenance (at least as far as my job is concerned).
so now it's just a matter of writing comforting documentation ("when the program says do x, that's when you do x!") and officially releasing the software. with this i will simultaneously increase submissions, decrease my workload and therefore ensure the future of the site.
two years wasn't so long to wait, after all.
Posted by njahnke at 7:12 PM | Comments (0)
June 23, 2007
time for lossy nmf
[20:53:11] <nate> i've decided i'm never using lossless nmf again
[20:58:18] <Radix> why>
[20:58:32] <DJGrenola> you had to ask didn't you
[20:58:33] <DJGrenola> heheh
[21:00:27] <nate> disk read locks
[21:00:31] <nate> i have too much cpu with 2 cores
[21:00:40] <nate> so what happens when i put a 4 core cpu in there
[21:00:46] <nate> i buy a raid? i don't think so
[21:01:00] <nate> more cpu just means less relative cost to decode x264 original
[21:01:10] <nate> can't use it with interlaced stuff but that's not an issue
[21:01:16] <nate> this is post-mvbob stuff
[21:01:39] <nate> basically what happened is the zoid ss flv was cut off
[21:01:49] <nate> because it was one of those times p: filled up while i was in kansas
[21:01:54] <nate> and i only just caught it today
[21:02:01] <nate> so now it's taking ~12 hours to reencode the flv
[21:02:07] <nate> when it's less than half that cpu wise
[21:02:33] <DJGrenola> I'd definitely consider looking into ffvhuff in ffdshow
[21:02:39] <DJGrenola> I use that for everything
[21:02:47] <nate> if it's lossless then it's too big
[21:02:51] <nate> well
[21:02:59] <nate> if it's anything like huffyuv anyway
[21:03:04] <DJGrenola> it's a little smaller than huffyuv is, ~20% or so
[21:03:09] <nate> 3:1 compression for all #000000 is pathetic
[21:03:38] <Radix> have you tried a compressed partition?
[21:04:04] <DJGrenola> :o
[21:04:10] <nate> nope ... but do you really think blind compression i.e. not knowing it's compressing video will do a whole lot
[21:04:19] <DJGrenola> that's a clever idea
[21:04:27] <nate> certainly can't be more than the 3:1 the shit lossless ones get
[21:04:32] <Radix> i dunno, zip compress a lossless avi and compare to huffyuv
[21:04:51] <Radix> huffyuv was built for speed, not compression ratio
[21:06:33] <nate> ok making a test file
[21:06:39] <Radix> also try zip compressiong a huffyuv avi
[21:06:45] <Radix> preferrably the same one i guess :-p
[21:26:26] <nate> ok this is pal d1 f1
[21:26:44] <nate> samus wearing the varia fusion suit kills the sheegoth and gets the wave beam, then leaves c.i.temple
[21:33:30] <nate> clip is about 34 seconds long
1.9G test_rgb24.avi
924M test_rgb24.avi.zip
680M test_huffyuv.avi
611M test_huffyuv.avi.zip
467M test_lagarith.avi
467M test_lagarith.avi.zip
328M test_x264.avi
328M test_x264.avi.zip
[22:02:54] <nate> quantizer 0 x264 avi it is
[22:03:12] <nate> though it seems lagarith would come close
[22:04:32] <nate> xp (10 megabaud) mpeg-2 original from dvd would be just over 42.5 meg
[22:05:10] <DJGrenola> wonder if snow has a lossless mode
[22:19:57] <nate> trying quantizer 1 etc
[22:37:46] <nate> looks like quantizer 1 and quality 1 are similar in size to the quantizer 0
[22:38:01] <nate> so what i'm going to do is no quantizer two pass 10 megabaud and look at the thing closely to see if i can see any artifacts
[...]
[23:13:42] <nate> the difference between the original and 10 megabaud is slight
[23:13:50] <nate> visible ... but not necessarily bad
[23:13:57] <nate> what it did was smooth out some areas
[23:14:03] <nate> typical of h.264 i guess
[23:14:18] <nate> the thing is that a lot of what it smoothed out was actually not supposed to be there in the first place
[23:14:22] <nate> because it was artifacts from mvbob
[23:14:34] <nate> so sometimes the lossy looks better than the lossless (1, 3)
[23:16:54] <nate> it's certainly better than mpeg-2!
[23:17:50] <nate> near key frames they're almost identical (2)
1: original | 10 megabaud 2-pass x264
2: original | 10 megabaud 2-pass x264
3: original | 10 megabaud 2-pass x264
[23:26:31] <nate> so 10 megabaud 2-pass x264 avi is the new standard nmf
[23:26:35] <nate> hurrah
Posted by njahnke at 10:11 PM | Comments (5)