unfortunately, the high quality deinterlacing filter mvbob() is not multithreaded. this means it can only use one core at a time. in the past i have set up multiple instances of vdub with e.g. one fourth of the total segments in a run to each, then started them all at the same time, but of course not all the segments are identical in length, so there would always be inefficiency as some vdubs would finish their deinterlacing before others.

avisynth is a very useful tool, and it was only a matter of time before i used it to automate a process by which any given input could be split up across a given number of threads, then automatically recombined later. i call this process sumi-chan for lack of a more creative descriptor (if the name were to describe the software properly, it would be something like "avisynth filter virtual multithreading system" - so long! so boring!).

usage is simple - i just run sumichan.sh with the avisynth scripts' basename and the desired number of cpu cores across which to split up the work, then run the resulting batch files on each real machine (one batch file for each cpu core). thus, entering 6 for sumi-chan's second argument results in six sets of avisynth scripts and six batch files.

due to rounding errors, if the number of frames in the source material is not divisible by the number of cpu cores specified, a few frames will be "squeezed" to the final work unit. this is unlikely to result in any significant inefficiency, as mvbob() usually processes between 1 and 2 frames per second even in extremely complex scenes. changes in said complexity throughout each work unit will result in much more significant inefficiency (compared to a true multithreading solution whereby all cpu cores would be simultaneously busy or not busy).

sumi-chan will appear as an optional feature in anri-chan 3 (we're getting close to the release of 2 right now). it ought to make encoding d1 material much more realistic for most people.

incidentally, i'm using quantizer 1 vfw xvid (via vdub) for the nmf. i've been extremely pleased with both the cpu and disk costs associated with this codec (they are much smaller than those associated with lagarith - and it crashes less to boot!).

of course, i have to include the obligatory crap quality picture of my new virtual computer, dubbed "v6". cores 1 and 2 are in the macbook on the left, while cores 3, 4, 5 and 6 are monitored on the mac mini's screen, but are in reality split equally between v5 (my pc) and the mini. (the terminal windows lower down on the mini's screen are left over from working on the prime hunters run as discussed in the previous entry.)

however, the original recordings needed to be cropped in order to get rid of their useless black borders. the catch was that the cropping parameters needed to be dynamic - that is, where and how much i need to crop changed every few seconds.

as i'm sure you've figured out by now, i came up with a way to accomplish this in a semi-automated fashion, and the rest of this blog entry will be devoted to this methodology.

the first step was to locate a simple autocropping plug-in for avisynth. this came in the form of the aptly-named autocrop, found at warpenterprises. thankfully, autocrop() offers a great deal of customization of the autocropping algorithm. the user can set not only the crop/don't crop threshold, but the number of sample frames the plug-in takes into consideration within the video, as well.

the crop threshold is probably 8-bit luma, while the sample frames are, according to the plug-in's author, taken equidistantly from within the source video. however, no matter how many samples the plug-in uses, there will always be only one set of cropping settings per video. thus, whenever dsgamer's tired hands moved ever so slightly, the screen would no longer be contained entirely within the initially-calculated cropping values.

the obvious solution was to make many more, smaller videos out of dsgamer's run - enter acrop.

this bash script takes two arguments: the basename of the video and audio originals and the number of frames in the original video. the script then creates a number of avisynth scripts, each of them including a number of source declarations and autocrop() invocations for bits of the original video.

i call each section of video on which autocrop() operates a "window". the window size is not arbitrary, but is determined by how long a sequence of absolute black will be on screen in the source video. if any window consists entirely of black, then autocrop() will naïvely produce 0x0 video, which is a contradiction in terms - avisynth has no ability, from what i have seen, to resize a 0x0 frame. thus, even though it's all black anyway, windows consisting entirely of black must be avoided. the window size, then, should be perhaps one or two seconds longer than the longest stretch of absolute black in the source video.

i found taking the correct number of samples, too, to be crucial: the process will fail (producing 0x0 video) if either too many or too few are taken.

for convenience and efficiency of processing, i included several windows in each new script file. for this speed run i created scripts 1000 frames long, 5 200-frame windows in each. because the source video was exactly 30 fps, a 10-minute segment would correspond to around 20 individual script files.

it was not possible to include every script file in one master script for two reasons. one was the extremely long startup time associated with stacking multiple instances of autocrop() on top of one another. on my core 2 duo 2 ghz, it took between 10 and 20 seconds just to open one script with 5 windows in it. the other was that each window represents one directshowsource() declaration for the source video - in other words, there was one copy of the entire source video in memory for each window in the currently open script(s). avisynth is extremely ram-inefficient in this way, but you can't blame it, actually, because we're talking about windoze here.

of course, my default autocrop() threshold of 100 was not always optimal. in fact, i had to preview every window in vdub to make sure it was not too high or too low. many times i simply replaced every 100 with e.g. 70 because the game environment in that particular segment was darker and autocrop() was getting fooled and cropping off parts of the game screen. it was also occasionally necessary to increase the threshold when e.g. the screen went totally white, resulting in light reflecting off surrounding objects and back onto the area around the ds lite's screens, in turn resulting in autocrop() leaving in the plastic border around said screens.

after i determined the correct autocrop() parameters for each window of a segment, i aligned the audio (recorded separately in audacity by dsgamer via an 1/8" miniplug cable) by placing the over-the-air audio from the original source video on the left channel and the correct, line-in recorded audio on the right channel. i knew that the "new" audio was lined up when i could not distinguish that sound was coming first in either the left or the right channel. i adjusted the correct audio's delay by using delayaudio() in a master avisynth script file which loaded all of the individual pieces of the segment in question (since saved from the original scripts as lagarith-compressed new master files).

i would like to emphasize my success with this relatively novel method of audio realignment. in sharp contrast to the usual method when it comes to audio realignment, i did not even have to look at the video to know the audio was aligned - i literally did the work with my eyes closed.

and so the world's first high quality downloadable nintendo ds speed run was prepared for distribution on sda. i hope that everyone enjoys dsgamer's hard work and my automation-laced lack thereof when his run is posted on sda soon.

i almost forgot - here's the obligatory screenshot of my initial autocrop() parameters discovery process.

in the above picture each | represents one frame. this drawing depicts one complete cycle and only one complete cycle, so the first frame not shown is the first frame of the next cycle. it would be solid blue like the first frame of this cycle.

oot is d4 f3. from the f1 original i was going to f2 first, then to f3 to avoid odd (deflickered) frames (shown above with a bar through them). otherwise i have to use retard bob which does a marginal job of undoing the deflickering and takes exponentially more cpu. more on deflickering in these two blog entries (incidentally this is the same reason i put up this topic).

the frame marked with blue is fine. the problem as i understand it is that sometimes instead of the motion happening in the green frame it happens in the purple or red frame (and gets lost as you can see on the f3 line). in these cases the pattern of motion in f1 is:

K D D K D D or
K D D D K D respectively, instead of
K D K D D D

where K is a motion frame and D is a duplicate (no motion). when i was testing this i saw only sequences of the last type which are ok. but there are often sequences of the first or second type. and once this starts it apparently continues for some time, thus the long, painful stretches of f6 people are seeing.

so basically the game isn't dropping frames at regular intervals. sometimes it will be ok to go -> f2 -> f3 but usually not.

the good news is that i think i've worked out a solution with the help of donald graft's decimate():

changefps(f1/2)
decimate(cycle=3)

this seems to be picking the correct frame (either green or red) from the f2 constellation, probably because it's checking to see that each frame is unique. this means i probably won't have to go back to the horrid retard bob.

total cost was $1,017.28. cost to sda was $339.09. 1/3 of the machine was paid for with sda funds for the following reasons:

1) the recent increase in the number of d1 f1 runs i need to encode for the site meant less and less idle time for my macbook. i was concerned that i would run out of cpu power to meet demand, especially after the warm reception of metroid prime 3 coupled with its length: the current estimate for an optimal any percent run is around three and half hours - not counting cutscenes. for reference, it takes about a week to encode 5 hours of d1 f1 video (deinterlacing to nmf and encoding all seven qualities) using four processor cores (assuming full speed from them all). i had been counting on most runs being d4, but this appears to be coming to an end.

2) related to 1), i had no quick way to take 10-20 gig of content in to campus to upload while the macbook is locked down encoding.

only 1/3 of the cost was covered by sda because i could have gotten a better machine for the money had it not been a mac (i wanted a desktop mac because using the macbook as one was suboptimal). granted, this will help me do non-encoding work for sda more efficiently, but i feel that is a rather weak reason to take more money from our reserves in this crazy time. also, regarding 2), the other half of the reason i have to have the macbook on campus is school, which has yet to demonstrate decisively that it is beneficial to m2k2sda (though it has been making gains of late: some work on anri-chan was actually done as part of my on-campus job!).

i backed down from saving up for the next beast machine (codenamed V6) because it no longer made sense economically. there are many factors involved, but to summarize:

3) i need more power right now (see 1) and 2)). an eight-core machine costs about $4,000. i wouldn't be able to afford that until may at the earliest.
4) there has been no solution found for multithreaded deinterlacing. therefore i will have to write my own (already discussed as a feature for a future version of anri-chan). once i do this, it will no longer matter how many physical computers are performing the deinterlacing - the load will be divided evenly between the available number of cores.
5) following from 4), continuing to pretend otherwise would mean that the macbook and V5 would be considered useless after the purchase of V6. together they have just under 50% of the theoretical power of V6, so this would be a great waste. by buying a weaker machine now and writing software to tie it together with my existing two, i can have 75% of the power of V6 right now for 1/4 of the price.

therefore, i now think of V5, the macbook and the new mac mini together as V6.

obligatory birth sequence documentation:

unfortunately, the amount of labor grenola, mike, snow, radix and i pour into sda, if paid, would equate to a sum much, much larger than $99 a month. therefore, when i declared sda monetarily self-sufficient two years ago, i was referring only to actual money earned and spent.

now it's time to take care of the rest.

for years, automating my job was a hopeless fantasy. the number of correct decisions required to encode material for the site was simply too high, and i knew of no way to have the computer make them for me. anyone who has seen the knowledge base and the tech support forum knows that asking inexperienced people to do my job as i do it is a losing proposition.

so i tried instead to make sda's revenue grow - to make the site pay us back for our work on it. i learned many things from my adventures trying to sell sda content on dvd, but probably the most significant was that it doesn't work: sda may never generate enough revenue to pay its workers, and sometimes i think it's better that way.

but the fundamental problem remained: sda needs people to add new content, and if i died today, i thought to myself, would someone be there to replace me? i decided that even though i can't control my mortality, i can control how i will be seen after the fact.

little by little i chipped away at the problem, making generic.avs into a highly concentrated, library-driven block of boolean action. generating the scripts for new projects became a matter of enabling or disabling variables. now one script could do it all.

but there still had to be someone there to make decisions based on the content of the source video. i had automated the ntsc/pal distinction, but the rest of them seemed hopeless: how should i tell the computer to check whether every second or every third frame is the same as the previous? and even if this could somehow be done - what about the games with a varying framerate?

therefore, i created the concepts of d (dimensions), f (framerate) and fdp (framerate decimation paradigm) to quantify the differences between games. if a database of every game's d, f and fdp could be created, i reasoned, then surely some sort of script could be written that would take those values and configure generic.avs, reducing my job to reading numbers out of list and typing them in to a window. even if the game in question weren't yet included in the list, it could be added by any tech support staff member after seeing a brief, automatically-generated sample of the game's video. it would be the next best thing to full automation.

as many of you are already aware, i decided to call this script anri-chan after the anime character who (like mickey and goethe before her) used magic to create numerous inferior copies of herself in a failed attempt to automate her job. thus, i am anri-chan, and anri.bat is an inferior copy of me. (when i began the project, i wasn't even aware that dgindex accepted arguments on the command line, and so i thought that automating the steps involving the user interface might be impossible, thereby dooming it all to failure.)

but these open source tools came through for me in the end, and so i was able to produce a proof-of-concept much more quickly than i had anticipated - and even better - it worked: other people were actually encoding acceptable video using anri-chan!

even vista couldn't stop me.

and as if that weren't enough, the free, open source model impressed me yet again today, as b-man in the sda forum went and wrote the anri-chan encoding settings manager for me. i have always said that it is not only me who can do this (just that i was the first to be chosen), and now i have been proven right. it really is true: if you build it, they will come.

i will finally make sda eternal by taking away the cost of its maintenance (at least as far as my job is concerned).

so now it's just a matter of writing comforting documentation ("when the program says do x, that's when you do x!") and officially releasing the software. with this i will simultaneously increase submissions, decrease my workload and therefore ensure the future of the site.

two years wasn't so long to wait, after all.

1.9G test_rgb24.avi
924M test_rgb24.avi.zip
680M test_huffyuv.avi
611M test_huffyuv.avi.zip
467M test_lagarith.avi
467M test_lagarith.avi.zip
328M test_x264.avi
328M test_x264.avi.zip

[22:02:54] <nate> quantizer 0 x264 avi it is
[22:03:12] <nate> though it seems lagarith would come close
[22:04:32] <nate> xp (10 megabaud) mpeg-2 original from dvd would be just over 42.5 meg
[22:05:10] <DJGrenola> wonder if snow has a lossless mode
[22:19:57] <nate> trying quantizer 1 etc
[22:37:46] <nate> looks like quantizer 1 and quality 1 are similar in size to the quantizer 0
[22:38:01] <nate> so what i'm going to do is no quantizer two pass 10 megabaud and look at the thing closely to see if i can see any artifacts
[...]
[23:13:42] <nate> the difference between the original and 10 megabaud is slight
[23:13:50] <nate> visible ... but not necessarily bad
[23:13:57] <nate> what it did was smooth out some areas
[23:14:03] <nate> typical of h.264 i guess
[23:14:18] <nate> the thing is that a lot of what it smoothed out was actually not supposed to be there in the first place
[23:14:22] <nate> because it was artifacts from mvbob
[23:14:34] <nate> so sometimes the lossy looks better than the lossless (1, 3)
[23:16:54] <nate> it's certainly better than mpeg-2!
[23:17:50] <nate> near key frames they're almost identical (2)

1: original | 10 megabaud 2-pass x264
2: original | 10 megabaud 2-pass x264
3: original | 10 megabaud 2-pass x264

[23:26:31] <nate> so 10 megabaud 2-pass x264 avi is the new standard nmf
[23:26:35] <nate> hurrah

just set "false" to "true" on line 7 to enable d1 (d4 is default). pal autodetection remains in from the last version, which i can't remember if i posted here or not. change "false" to "true" on line 23 to enable the statid (after you've finished filling out the trim(), for example).

additionally, you'll need to change all occurrences of "dx" to your project name, which should be the same as your index and ac3 basenames. you also may need to change your drive letter (i use p: for my project drive). finally, watch out for the path to the sda logo pngs in the statid subroutines (and, obviously, the LoadPlugin()s at the very top!). all the other options from lines 10-19 depend on the input video source and desired output quality.

enjoy!

i finally uncovered the connection last month during a series of tests i was running for OPERATION ZWEI, an sda-related secret project some members of the sda team are working on right now (expect to see its release sometime this summer). these tests involved extremely memory-intensive avisynth scripts, and i later decided not to accomplish the effect in this way, because the avisynth system is simply broken when it comes to memory allocation (thanks, doom9 guys!). however, this situation afforded me repeated chances to see the exact same "access violation" error pop up -- again, pseudo-randomly -- but this time while watching the task manager intently.

i was able to distinguish two different types of access violation errors -- one thrown when the avisynth mother process reached doze's 2 gb allocation limit and another when the process would attempt to allocate memory beyond the total (3 gb) available to doze. it was at this point that i realized that radix had basically saved m2k2sda from these pesky access violations making their way into final encodes, because under normal conditions i don't use more than 1 gb of ram on v5. surely 2 gb free should be enough for any normal encode to allocate freely without the danger of running over the edge.

and indeed, i haven't seen or heard of a new encode afflicted with the problem since installing the new ram.

the input syntax for this batch varies slightly from the others due to the need to insert LQ in varying places in the filename depending on whether the run is segmented or not. to help me remember it (and to speed up master batch creation), i've also created a generic.bat.

additionally, the new function includes a sharpen() invocation to (at least superficially) recover some of the original quality in the output. this trick works best on very clean (i.e. dvd) source material. when used on material that was recorded using vhs, it tends to only draw out the many problems with such a picture.

inspired by this productive night, i streamlined my generic-v.avs (from which all other avisynth scripts for sda are made), collecting some other blocks of interdependent code into functions i can more easily enable and disable as required.

nate_retard_bob_2() is available as a part of generic-v.avs.

video games suffer from flickering more so than your average tv program because they contain more of the types of material that make flickering obvious, such as straight lines (usually in menus) and rough edges (usually in text). some early attempts at deflickering involved switching the hardware from d4 (half resolution) to d1 (full resolution) when navigating menus. one notable game to do this was metal gear solid. changing to full resolution allows the game console to display different information in each field. flickering is reduced as the jagged edges are reduced. for this reason, some people refer to deflickering as "antialiasing", quite similar in lineage to the methods used to hide "jaggies" from discerning gamers' eyes. (by the way, deflickering can be "experienced" firsthand by entering the super smash bros. melee options and enabling and disabling the option. what looks better on an interlaced scanning display looks worse when it comes to progressive scan.)

there is nothing wrong with games running in full resolution. the missing motion information can be accurately reconstructed with an advanced deinterlacing algorithm such as mvbob(). further, as previously mentioned, "hybrid" games like metal gear solid usually run at f2 (half framerate) outside of menus, so i can simply drop one of the fields to simultaneously deinterlace and de-deflicker in these cases (no mvbob() necessary!).

however, another type of deflickering has recently become much more common with the rise in console emulation (that is, running emulators on consoles). sonic mega collection and the mega man anniversary collection both deflicker their output. they accomplish this by blurring one field a quarter pixel up and the other a quarter pixel down. i have taken pictures of this in action:

the right image occurs one half-frame after the left. that is, each image represents one field. if you turn your attention to the top non-black line in the left picture, you will notice that it is different from the line immediately below it. it is exactly 50% white (or, if you prefer, 50% black!) because it has been blurred with the preceding (black) line. a similar situation can be verified in the second picture by looking at its last non-black line--the only difference is that this time, the picture was blurred down, not up.

material that was once d4 is now effectively d1. as you can imagine, playing back this video without any form of de-deflickering will present an unacceptable level of "bobbing", if you wish to call it that. after pondering this sad situation for many hours, i concluded (with the help of grenola) that the only solution is to blur the down-blurred field up and the up-blurred field down, canceling out the deflickering (thanks go to grenola for this particular method):

converttorgb32
field1=SelectEven.PointResize(640,480).addborders(0,0,0,1).bilinearresize(320,240)
field2=SelectOdd.PointResize(640,480).addborders(0,1,0,0).bilinearresize(320,240)
Interleave(field1,field2)

unfortunately, things are not so simple. upon performing this so-called "retard bob" (so dubbed after "smart bob"), one field will seem clearer when it comes to certain objects, while the other will seem clearer when it comes to other objects. the exact cause of this behavior is not yet known, but one theory involves the quite imperfect horizontal analog representations of luma and chroma changes. that is, different brightness levels and colors at the source will produce different levels of blur after deflickering. the problem now moves beyond my ability to correct.

having said that, "retard bob" has improved things significantly:

mega man's energy bar was used as a reference for attempting blur level equalization. however, rush and the cloud maker towers at the bottom of the frame are clearly more blurry in the left (blur compensated) picture than in the right one. this would seem to dictate less blur compensation. however, applying any less blur compensation results in mega man's energy bar coming out of "blur sync" and "flickering" in the final product. this "neo-flickering" (or, if you prefer, "bobbing") is considerably more noticeable at f3 (third framerate) than at f1 (full framerate) for obvious reasons. unfortunately, the divx medium and low quality as well as the h.264 low quality of all 2d runs are encoded at f3 in order to ensure playability on old hardware (can't be f1) as well as to avoid problems with 60 hz blinking in most games (can't be f2). the ability to encode at f2 would neatly solve this entire situation, as i mentioned earlier when discussing metal gear solid, and this is precisely what i did for the recent sonic 3 & knuckles and sonic 1 runs submitted by mike89 (old school sonic is very special in that it contains no salient 60 hz blinking effects).

using gaussresize() i have settled on a "not-so-happy medium" blur (p) of 40:

converttorgb32
notblurred=SelectEven.PointResize(640,480).addborders(0,0,0,1).gaussresize(320,240,p=40)
blurred=SelectOdd.PointResize(640,480).addborders(0,1,0,0).bilinearresize(320,240)
Interleave(notblurred,blurred)

("notblurred" refers to the field needing blur compensation based on mega man's energy bar.)

however, as shown and explained above, some "neo-flickering" or "bobbing" will be visible no matter the blur compensation. for this reason, i am recommending that emulation be completely avoided when recording speed runs. as previously discussed, there are exceptions, mario 64 and similar 3d games constituting one such class (because the games run at d4 f2, i can simply drop one of the fields to neatly solve the problem).

concluding, i wish to mention that all game boy advance games played using the game boy player suffer from deflickering. i have employed many solutions over the years, including old school "retard bob" in virtualdub (even field quarter scanline up, odd field quarter scanline down in the "field bob" filter) and even mvbob. however, because virtually all gba games run at f1, none of these solutions is optimal for the above explicated reasons. the jury is still out on deflickering.

the new flv creation process recommended by mike hearn is still under evaluation, but i am encouraged by its fruits thus far. the only major sticking point for me has been that it demands that the input video be upside-down (flipped vertically), and so i must create a distinct .avs file for flv encoding now (as opposed to just using the mq mp4's), but this is really quite minor compared to never again getting shot down by one of the macromedia flash video encoder's myriad avs input bugs or having to set the video and audio bitrate every time i reboot the machine or wondering how much better quality/smaller of a file i could make had i multipass encoding--and i could go on.

the divx codec and the macromedia flash video encoder were dragging me down without a doubt, and i am grateful to the community for their continued support of my work.

things have been exciting around here lately. i'm sure y'all have noticed grenola and mike doing updates, and indeed, they are together doing the entire update process now, leaving radix to watch over his creation like a proud father and to occasionally yell, "no, no, you're doing it all wrong!" mike responds to runners, no matter how annoying, with super speed, and grenola has made short work of automating much of the remaining parts of the sda updating process. it's safe to say sda is operating better than never, ever before.

on the home front, i am very pleased with my new dvd recorder, an lg dr1f9h (say that five times fast!). media compatibility is far better than with my old panny, and as an added bonus, the lg writes dvd+rw without the need for a finalization step. that means that i went from using isobuster to raw read the vobs off of every dvd i recorded (because all the media i bought was incompatible with the panny) to not even needing to finalize the disc to take the vobs directly off of it in v5. and on top of all of that, the title corruption bug that precipitated "katamari hell" is nonexistent. good deal for only $200.

the lg has tons of other crazy features i'll probably never use (like dv in and hdmi out), but i think it's a good sign that i'm getting all these earmarks of a high-end dvd recorder at only $200 while over at walmart cheap (but still perfectly good) dvd recorders are down below $100. it's never been clearer to me that my prediction of dvd recorders meeting and then beating vcr prices will happen by this coming summer. that's significant, of course, because i plan to phase out my free vhs capture service at that time. the reason is because i must help the quality of the videos on the site in any way i can, and if that means coercing people into buying dvd recorders, then so be it. at $40 a pop they'll be less expensive than a single next generation game.

i'd like to take this opportunity to thank tom batchelor for his continued monetary support of sda. clearly he's not the only generous one around here, but he is obviously the one with the most money, and i appreciate his confidence in me and my operation. when other people believe in you, you often find a way to make things happen regardless of whether you share their confidence.

i will keep it coming.

this situation (sda annexing the macbook) became necessary when i realized how vastly superior -- and therefore essential -- the mvbob() deinterlacing filter was to its alternatives. with v5 completely tied up making huffyuv new master files of the large amount of d1 f1 runs that have come in recently, i simply needed more cpu power to work on actually encoding those files v5 was creating, lest i find myself still working on the same runs two weeks later, while others continue to flood in.

the setup isn't perfect. for one thing, the parallels "network adapter" seen by windoze xp is only 10 megabaud. this means that the huffyuv compression is just not sufficient to get the d1 f1 file over the network from v5 fast enough to saturate the macbook's cpu. however, i'm seeing about 80% average cpu use on the macbook with d1 encodes, so that's really no big problem. also, because the machine can encode mq and lq (d4) stuff more quickly than the d1 hq and iq, the cpu use goes down for those jobs, the fans spin down, and the machine gets a network-bottleneck-supplied respite from the heat. i believe that this is important, because the model of macbook i own has been shown to have a heat problem coming from a thermal paste application oversight on apple's part. i would fix the paste, but i would void my applecare warranty.

here's a screenshot for your viewing pleasure. note that one of the two "c" drives mounted on the os x desktop is in fact the virtual machine's c: "drive" (as broadcast via smb). transferring files back and forth is made quite easy in this way.

i decided to immediately (on capture) follow the instructions sent by the runners and locate each run on its tape, recording with the dvd recorder only those important parts of each tape. thus, i would use only a handful of dvds to capture the entire project, rather than the matching twenty or thirty discs to twenty or thirty tapes. this would be my first major mistake.

radix came calling a few days later repeating a verifier report of extreme audio desync in the verification copies of the katamari material. i was shocked and appalled, but not nearly as much as i would have been had i not already had a hell of a week trying desperately to keep the requested old material flowing into the flash encoder (but that's another story). one more thing going wrong seemed almost like the natural flow of things.

i quickly determined that the audio started out in sync, but actually became progressively earlier as each disc wore on. i had seen this before when i had tried to index at the same time multiple vobs created at different times, but this couldn't be the case now - the vobs were all part of the one large whole my dvd recorder seems to like to write to the disc (for all discs regardless of length or content, it writes five equal sized vobs together representing all of the material on the disc).

the only other viable explanation meant that i would have to somehow correct audio desync caused by so-called "title" breaks, or breaks inserted when i would press stop and then later record again on the dvd recorder to capture another run.

i began to experiment with various pieces of software (vob2mpg, etc) trying to change the format of the vobs so that dgindex could successfully compensate for the title breaks. none of my experiments were successful, and so i created this thread at doom9. as a side note, my very favorite thing about using open source software is that, in cases of technical questions about the software, the author him- or herself is on hand not only to answer said questions faithfully, but also to recommend different software if the present software cannot do the job.

i followed neuron2's advice and used projectx (an impressive piece of work, by the way) on the katamari vobs. this produced vobs which, when indexed with dgindex, had in-sync audio!

but there was a problem - to make a long story short, it turned out projectx had actually deleted sections of the video to keep the audio in sync. besides being totally unacceptable from a speed running standpoint, this also caused me to have to completely remark the coordinates of all fifty something segments once again in order to get proper verification copies out to the verifiers.

unfortunately, i was not aware of the frame dropping issue introduced by projectx until a verifier reported it to me.

at this point i wondered whether i shouldn't have simply bitten the bullet and used all thirty something dvds to recapture the entire project without using more than one "title" per dvd. i immediately started recapturing the first tape in this way.

then i had an idea: why not get my old dvd player out (bought to test sda dvds before i sent them out way back when) and use it to make an analog copy of the dvds with the corrupted timestamp information? the resulting analog copies would be slightly inferior in terms of video quality (being mpeg-2 compressed twice), but they would each consist of only one title, eliminating all of my problems processing the material. i got out my old dvd player and began the copying process, switching over to a new spindle of dvd-rs.

then i found out that the new dvd-rs are incompatible with my dvd recorder.

at first i didn't want to believe it, because every type of high speed verbatim media i had bought before had been compatible. but all the signs were there: the long finalization process and the single, tiny vob and accompanying table of contents file that resulted.

not wanting to slap down $40 for new media that day, i went to take out my old dvd-rws with poor compatibility i had used with limited success in the early days of the dvd recorder.

but before i could attempt to copy the original katamari dvds again, i again had a thought: why not simply raw read the entire "incompatible" disc and scan the data thereon for vob headers? assuming the dvd recorder wrote the vobs in place (and being write-once media, i could imagine no differing scenario), i knew i would be able to recover the captured video regardless of the mangled table of contents file (created so because the dvd recorder couldn't read properly the media it had just written to). i had already done this once a few months before for kridly using simple unix software written by djgrenola for that sole purpose.

this time, though, i decided to check out ballofsnow's "recovering data from unfinalized dvds" links, and in them i found the wonderful windoze program, isobuster. this program was so impressive to me (it duplicated under doze the functionality and flexibility of grenola's work without any of the manual labor that had been required with the command-line unix version of the recovery process) that i bought it ($30 us) on the spot when it asked for a registration code to complete recovery of the vobs. i figured that it was a good deal, as another spindle of dvds would cost me at least $30, and isobuster would remove in a user-friendly fashion on the encoding machine the need to have compatible media for my dvd recorder.

with a viable capture process notwithstanding the incompatibility of my media i finished making analog copies of the original katamari dvds with corrupt timestamp information.

i next tried for several hours to create a script to "realign" the segment coordinate information from either the original capture attempt or the vobs as "repaired" by projectx. (i was trying to avoid having to mark all of the segment breaks for a third time.) this was my second major mistake, as after several hours of completely fruitless effort, i discovered that the corruption of the timestamps produced desync at every title break that not only was not linear in its growth - it actually went backwards (became more in sync) some of the time, completely destroying the viability of an automated realignment.

so i set out to mark all of the segment breaks a third time. it turned out that the timestamp corruption had also somehow caused entire runs to "slip through the cracks" in the previous encoding attempts - on the second disc alone, i have now marked over twice as many segments as the disc originally appeared to contain. how this is possible i am not sure, as i remember checking the output of the vobs with corrupted timestamps several times, and every time it totaled something reasonable, something just under two hours in length ...

lessons learned from this fiasco are many (i no longer have to have media compatible with my dvd recorder to capture, for one), but i still get the feeling that i in fact traded one lousy situation (capturing with the all-in-wonder) for a much more lousy situation (capturing with the dvd recorder). for example, it will be necessary in the future in the case of a thirty-tape project to use thirty dvds to capture the material.

the quality is better now, and perhaps that should be all that matters in the end - but i find myself time and time again in situations where i have to spend several days "learning my way out of a wet paper bag" as it were. this isn't rocket science, but without the necessary experience, people are just going to be totally screwed when they try to do this stuff themselves - especially if they get a dvd recorder as retarded as mine in this area (and mine is still recommended in the dvd recorder thread in the sda forum!).

one day this community will be delivered from this sorry state of technological affairs.

that day is not today.