well, dsgamer submitted not only an improvement on his any percent run of prime hunters, but a new 100% run, as well. on top of that, greenalink did a run of mario 64 ds. when all three runs were accepted by verifiers, i knew it was "time to get serious" (as mega man would say) about processing ds footage.

perl has become my new best friend, and i knew it had integration with imagemagick, gd, etc. a few google searches later, i had decided on using imagemagick ("perlmagick"), since it came with a well-tested autocrop function.

originally i was going to try to determine the z-distortion (i don't know the proper term) of the ds screen(s) and correct for that (basically if the top of the screen is further away from the camera than the bottom), but the only implementation i found of that functionality i couldn't get working with any of my test footage (it never changed anything), so i decided to put that off until next time. i felt like i had enough on my plate with making sure the autocrop threshold was correct all the time.

basically, i set a default threshold and try to autocrop the current frame. if the autocrop succeeds, then i note how much was cropped and move on to the next frame. if it doesn't succeed, then i gradually turn up the threshold until it does succeed. if it never succeeds, i don't save the crop data. if it succeeds but it's cropped off way too much as determined by a sanity check, then i don't save the crop data.

i do this for a certain number of frames (i ended up using two seconds' worth for dsgamer's runs), then go back and crop all of those frames by the means of the values i saved for each side. this is much improved from how i was doing it before using avisynth, which could only look at a single frame to determine values. i also don't think i could get the values that the avisynth plugin was using very easily due to how simple the avisynth language is, but i could be wrong on that.

in other words, the old method was very high maintence - i had to go five seconds at a time and manually adjust the threshold. a lot of times bloom from the screen (if the screen was very bright and you could see the ds's body glowing in the dark, for example) would throw off the threshold. i got around that this time by starting with a rather high threshold and then throwing out crop data that deviated too much from an arbitrary norm. (by "deviated too much" i am talking about how if the screen was very dark the high starting threshold would sometimes cause it to crop off too much, for example the area outside the hud in human form.)

a new feature for next time might be to determine the norm on the fly. i guess that would be means for the whole segment. actually i thought i was going to have to do that this time, but dsgamer's runs came out perfect on the first try. it was a good thing, too, because it took the better part of a week for one of the computers in the lab where i work to process them (i think most of it was i/o though since i had to read twice and write once a .png file for every single frame in the runs). i'd also like to learn how to read and write yv12 and do everything through mplayer/mencoder pipes next time for a big speedup. normally i wouldn't care so much about the speed, but i think this may make it into anri someday (even though only two people have ever submitted ds runs to the site), and it's totally unworkable for most people's computers the way it is now.

i did the audio on dsgamer's runs the same way i did it last time - by putting the over-the-air audio on the left channel and the audacity-recorded audio on the right channel, wearing headphones, and adjusting the audacity audio's offset until i couldn't distinguish the right and left channels. (i then cut the over-the-air audio.)

greenalink's run i did totally differently because he recorded it totally differently. he was able to keep his ds much more still than dsgamer was probably because of how the game is controlled differently, so autocrop was not really needed. i just figured out a set of rotation and cropping values for each individual segment and applied those. he also recorded both screens all the time. he and i talked about this quite a bit and we decided that since mplayerc lets you zoom in on one screen in the video, leaving in both screens would be the best option since some people might want to see what's going on on the bottom screen even when he's controlling a character. it also goes along with the site's general ethos, not throwing out potentially valuable data that could be used by someone to improve the run.

greenalink's run didn't need anything done to the audio (except maybe normalization) since he recorded both the video and the audio using a dvd recorder, so it was already synced.

all three runs will be going up on the site soon, and i hope you enjoy them.

here are the tools i made or used to process them this time:
mplayer/mencoder (to dump to pngs/encode video from pngs)
ds.pl (to process the pngs)
avisynth rotate (to rotate greenalink's segments before cropping)

by the way, getting perlmagick to compile under os x was a total pain in the ass. it took me several hours of googling and experimentation. i ended up manually linking the perlmagick sources against the imagemagick libraries delivered by macports.

enjoy.

#!/usr/bin/perl -w
#nathan jahnke 

my $natename = 'rks';
my $thedir = "/p/${natename}";
my %rehash = ( #old, new
	8=>5,
	5=>6,
	6=>7,
	7=>8,
);
my %files;

opendir(P,$thedir);
my @files = readdir(P);
closedir(P);

for (@files) {
	if (-f "${thedir}/${_}") {
		my $newname = &newname;
		system('mv', '-nv', "${thedir}/${_}", "${thedir}/${newname}");
		$files{$newname} = $_;
	}
}

for (keys %files) {
	my $newname;
	if ($files{$_} =~ m/^${natename}-v-(\d+)\.(.+)$/) {
		my $segnum;
		$segnum = $1;
		$segnum = $rehash{$segnum} if $rehash{$segnum};
		$newname = "${natename}-v-${segnum}.${2}";
	} else {
		$newname = "${files{$_}}";
	}
	system('mv', '-nv', "${thedir}/${_}", "${thedir}/${newname}");
}

exit 0;


sub newname {
	my $retvar = '';
	while (-e "${thedir}/".$retvar) { #try until we get a unused name
		$retvar = &random;
	}
	return $retvar;
}

sub random {
	my $retvar = rand(1);
	$retvar =~ s/^0\.//;
	return $retvar;
}

mencoder -dvd-device /p/mm6 dvd://1 -fps 59.94 -vf-add tfields=0 -vf-add scale=320:240 -ofps 59.94 -ovc lavc -lavcopts vcodec=ffvhuff -oac pcm -o nmf.avi

dgindex, because it doesn't drop video frames to compensate for the bad ac3 ones, will cause randomly progressive desync. though i need to remember to use it for timing in cases when the in-game timer isn't used, such as this mm6 run (the release versions of such runs will actually be several seconds shorter than their official times indicate).

after several hours of playing around, my initial guess turned out nearly correct: the "Skipping frame!" messages i was seeing every few hundred frames from mencoder trying to transcode the file were because the audio was bad, not the video. actually, i should have known this from the start - if it had been the video, then the video would have been leading the audio, not vice versa as i initially observed at points. i had also forgotten that the term "frame" also applies to many audio formats. using mencoder's -noskip option together with the -mc 0 option seemed to restore a semblance of audio sync in an nmf - enough to carefully stretch the audio to correct the progressive desync to an acceptable level.

tiki has been advised of the situation. right now we are guessing it was because he used rf (from his nes). pal nes is known to generate a nonstandard signal (enough to cause massive frame droppage with more sensitive capture devices), so maybe the ntsc nes in combination with rf confused his dvd recorder? i'll post again as i learn more.

nas_dstation_settings.png
nas_dstation_tasks.png
nas_network.png
nas_status.png
nas_terminal.png
nas_volume.png
nas_winmac.png

as you can see, i set up the disks in a raid1. this is the new sda project drive ("p"), replacing an old sata 500 gig in v5 (my hp pc). unfortunately the old 500 started locking up a few months ago and after it did it three times last week i decided i had better not wait any longer. i do daily backups of all the in-progress sda stuff but i didn't want to work off of the backup if i could avoid it, which i did. actually even now i'm not sure whether the lockups were caused by the drive, the cable or the controller (on the hp motherboard) but to be honest i don't have the time to be troubleshooting things like that, especially when the lockups are more or less random. also no matter the outcome i would have had to have spent money anyway so i felt this purchase, subsidized 50% with sda funds (even though i have no reason to buy it except sda), was inevitable.

softwarewise:

BusyBox v1.1.0 (2008.06.20-16:34+0000) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

p> uname -a

Linux p 2.6.15 #639 Sat Jun 21 00:30:16 CST 2008 armv5tejl unknown

p> 

i have to say that the built in torrent client (rtorrent with the web gui) isn't bad. the only thing that seems to be missing is down/up rate caps in the web gui (it may be available in the windoze-only standalone app - i haven't checked) but it's not the end of the world. having the torrent client run on the same machine as the project drive is important and this morning i was able to shut down v5 for the first time in years, which was pretty cool (literally). it's by far the biggest power user out of all the v6 machines (by perhaps an order of magnitude - the other three are two laptops and a mac mini) so having it off when i'm not encoding is a boon.

the nas looks like it has a ~2 gig solid state drive for its software, which comes as an image file which is then flashed to it over the network (it doesn't come with any software preloaded, just a disc with an image on it to get you started). upgrades happen this way as well - of course i updated it to the latest after i got it and it worked great from my mac.

p> df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/md0                  2.3G    235.0M      2.1G  10% /
/tmp                     62.0M    108.0k     61.9M   0% /tmp
/dev/md2                684.7G     38.8G    645.9G   6% /volume1
p> 

overall i'm quite pleased with it. it works as advertised.

i'd stayed away from wine previously because i was almost certain nothing as complex as my use of avisynth would run 100% satisfactorily. needless to say, i was quite surprised when it compiled on one of my macs and played an unmodified sda avisynth script in vdub correctly.

that was over a week ago, and i'm still not entirely over the weirdness factor of running the whole anrichan family of tools without running any microsoft code. so i guess i'm not as elated as you might imagine - more like dumbfounded. it's been one of those rare moments when technology catches me off guard.

i've read that wine works even better under linux due to apple's lame x implementation, so i'm now considering what to do with the one remaining machine i own that runs windoze as its primary os ("v5").

it's not perfect by any means - as far as i can tell, pipes aren't supported in batch files, which meant i had to reimplement everything in unix shell language. it hasn't been a big deal except in the case of anri.bat, which will need major reworking, and that's a damn shame, because i thought at first that i had "discovered" a universal anrichan.

i'm also having problems with processes, e.g. avs2wav and bepipe, not being able to close files that are on network volumes. the process will complete successfully, but the close call hangs, meaning i have to hit ctrl-c to continue the encode script - unacceptable. i've actually seen this before under windoze (though not working on sda stuff), so i doubt it's wine's fault per se - just something i'll have to dump more time into.

i also can't seem to figure out how to install codecs that include right-click -> install .inf files. i grabbed the dll call out of a real copy of doze and tried that, and it does run, but doesn't seem to produce any result under wine. i'm lucky the anrichan installer installs lagarith, because i probably wouldn't have access to that codec under wine for doing nmf otherwise.

speaking of nmf, here's my unix nmf command, my v command and my mq commmand as references for how i reimplemented snow's batches in unix shell language. sumichan is essentially unchanged because i'm still using avisynth - the only difference is the batch file writing for calling nmf.bat on each of the slices, which now obviously just writes calls to nmf instead. syntax is like

nmf "P:\\projname\\runname" 1

look for the first runs produced with the help of wine on sda soon.

just set "false" to "true" on line 7 to enable d1 (d4 is default). pal autodetection remains in from the last version, which i can't remember if i posted here or not. change "false" to "true" on line 23 to enable the statid (after you've finished filling out the trim(), for example).

additionally, you'll need to change all occurrences of "dx" to your project name, which should be the same as your index and ac3 basenames. you also may need to change your drive letter (i use p: for my project drive). finally, watch out for the path to the sda logo pngs in the statid subroutines (and, obviously, the LoadPlugin()s at the very top!). all the other options from lines 10-19 depend on the input video source and desired output quality.

enjoy!

the input syntax for this batch varies slightly from the others due to the need to insert LQ in varying places in the filename depending on whether the run is segmented or not. to help me remember it (and to speed up master batch creation), i've also created a generic.bat.