onlydarksets:The first thing you should do to get the video the way you want is to delete the "scale" video filter. That should improve things dramatically without doing anything else. You can also try to use a lower quantizer value (essentially a lower compression ratio). If that doesn't give you the control you need, try using the lavc video codec instead of xvid - it seems to give you more nuanced control.
That's exactly what I did. I started off by stripping away pretty much all of the parameters and then added them one by one to achieve the results I liked. In case there is anybody else except me who isn't converting for a mobile device but for his HDTV set, here is the command line I'm using:
mencoder -demuxer 35 -vf scale=944:528 -lavdopts threads=2 -ofps 29.97 -ovc xvid -xvidencopts bitrate=1600:threads=2 -oac mp3lame -lameopts abr:br=160 "InputFile.dvr-ms" -o "OutputFile.avi" > "MEncoder.log" 2> "MEncoderError.log"
The scales I use are values which have proven to get a pretty good balance between resolution, file size, bitrate, and quality. For a 16:9 show I use 944x528, for a 4:3 show it would be 704x528. When keeping the bitrate around 1600, the resulting file will be somewhere between 650 and 700 MB in size for a one hour show. Also, I raised the audio bitrate to 160, even though I openly admit that I probably wouldn't notice the difference to 128 much, it's simply personal preference. You will notice that at the end I'm writing the stdout log and stderr log into separate files, so that I can see what's going on.
I used this on a working show I recorded, and while the Xvid file looks good, I noticed slight AV synch problems. Not everywhere in the file, and not with identical delays, which brings me to the conclusion that it has to do with cutting out the commercials. I am running the conversion on the original file right now that still has the commercials in it to prove my suspicion. I've had problems like this long time ago. Cutting commercials messes with the time stream embedded in the video stream, and after demuxing the audio the time information is lost and the encoder can only put audio and video together at the beginning of both streams without applying any delay. When playing back the original dvr-ms there might be a split-second after a commercials break without audio which it uses to synchs up again, so there you don't have that delay, because the time information is still intact. I know from experience that there is no easy way around this except making sure to keep everything as "clean" as possible (only cut on key frames etc.) when cutting out the commercials.
Once again, I'll post more results as they come in (this is a time-consuming process as jelwood pointed out!).
Update: When playing back the Xvid that I created from the dvr-ms without commercials on my Xbox I did not notice any AV synch problems anymore, which really surprised me. Now I'm testing with different players on my machine to track down what might have caused it. But it seems now that the commercials have indeed been cut out "clean" and that they don't cause problems when reencoding the file!
Update 2: I've been going through both the Xvid with and the one without commercials with different players now, and sometimes it seems that there is some AV synch problem (in both files), but then when I skip back 30 seconds and wait for the same scene it's gone, so I think it's just from jumping around in the file before the synch has been restored by the player. So it might be nothing really. Also, I realized that Survivor is probably the worst show to detect AV synch problems with, because most of the time there are voice overs and people talking that you can't really see, and you have to jump around a lot to find a scene where you can watch somebody's lips move with the audio. I will try out a different show and get back to you on this one. For now disregard any synch problems I've mentioned.