On Wed, 31 Jan 2001 02:04:07 GMT, "Alexander Ibrahim" wrote:

>Sorry if I am barking up the wrong tree David, but it seems to me
>you are espousing the rejection of experimental science when
>testing equipment. We are all trying to get a job done, but the
>reason we come here and go to web sites is an attempt to cut
>through perceptions and marketing hype and get information.
>
>You should use test subjects, but unless you maintain a standard
>test subject you can not ensure repeatability. Maintaining a
>standard test subject means making sure that lighting conditions
>can be replicated easily. It means making sure that the colors on
>your subject don't fade...it isn't easy.
>
>One of the problems I am having right now is that I can't build
>an object that will retain its color over time. Basically I'll
>have to keep my test subject in a box somewhere in a closet with
>controlled humidity.
>
>As a result, test subjects only reveal a portion of the
>performance picture for video equipment. In addition to these you
>have to perform standard tests.
>
>Test charts are designed to expose camera, lens and recorder
>flaws. You may have to use several charts, but they do expose
>flaws in easily repeatable situations. That is key:
>repeatability. The difficulty is that you have to learn how and
>when those tests do apply to the real world. That is why people
>turn to external informations sources, like consumer reports.
>
>Another key is having test equipment whose performance is well
>understood. That means video monitors that are verifiably
>calibrated, and test equipment like a waveform monitor and a
>vectorscope.
>
>Without accurate repeatability there is no way to effectively
>compare different test runs, or different equipment. Thus no
>basis for making authoritative compaarisons.
>
>That is why all my "reviews" are entirely subjective...I do not
>have accurate test equipment. No accurate way to compare with a
>calibrated image. You should not trust my eyes, you should only
>depend on me to give you information with which to judge
>yourself. I may show you results, but it is reasonable for you to
>expect me to also provide you with information on how to
>replicate my tests for yourself.
>
>So, anyone (Tektronix HINT HINT) wanna pitch me a set of test
>tools. (I need a waveform a vectorscope, test signal generator
>and at least one reference monitor. A JND analyzer would be great
>too.)

I think, if a reasonably stable and appropriate test
subject(s) can be used, that more useful information
can be had about real-world performance than by using
test-charts, etc. This sort of "subjective" testing
is best done, though, comparatively and not absolutely,
as in, with two items to be compared tested side-by-side.
In still lens testing, for instance, charts are often used,
but the problem is it is VERY difficult to get exact
alignment and focus, making repetition of the results
difficult. In addition, this method fails to show well
whether edge softness is due to field-curvature, to
field non-parallelism, or to another fault in the lens.
Also, using 3-D test subjects can allows one to more
easily detect local focus anomalies and alignment errors.
And, lens performance can change with focus distance,
and this effect is difficult to see with chart-testing.
With video, the standards for image quality are
far lower, but comparative testing is still useful
for showing the areas of acceptable/unacceptable-level
faults. Objective, "scientific" testing of video gear
would be an exercise in frustration, since the errors
detected would generally be of a high-and-varying level.
I don't bother with this, since I'm most interested not
in the 'scope readouts, but in what "X", "Y", or "Z"
type subject looks like on tape under different
conditions. This tells me more of interest than
"objective" (but often misleading) results. A good
example of this is one you gave: Consumer Reports.
They appear to test consumer goods objectively and
"scientifically", but their conclusions are often
(read, "usually"...;-) at considerable odds with my
own conclusions, having used the same gear...
Do trust your eyes and experience, in the end...
(but do use comparative tests for checking...;-).