|
OpenDRC home
DRCTest is the OpenDRC helper program used to test for difference in the .drc output directories
produced by OpenDRC/DRC1.2/DRC1.2.1. This program, while compairing any two output directories, will find
differences in the corresponding output RT, STATS, Parameter, or ACTS files produced. The RT, STATS and
Parameter files will be tested like the standard Linux diff command.
Each individual ACTS file will be
tested against it's corresponding ACTS file in the other directory. However, the test will look for
structural differences in the file (added or missing lines for example) but will ignore any activation
differences that do not exceed given thresholds. The default thresholds for the individual activation
lines are 0.0000015, and for the activation total lines 0.00000375. The reason for this facility is that
OpenDRC and DRC1.2 calculations suffer from floating point roundoff errors. Since these errors are minimal
they can safely be ignored. However, command line parameters are available for the researcher to adjust
these two thresholds.
DRCTest was used during the development of OpenDRC to verify that OpenDRC conformed to the
specifications of DRC1.2. Differences between the operation of both programs on a particular dataset
will be exposed with DRCTest. This was how the Unsupported Decay Error
was discovered in the Windows version of DRC1.2.
DRCTest produces two output files: DRCTest.txt and DRCTotals.txt (see bottom of page).
DRCTest Examples:
- To run DRCTest from a directory different from the two test directories:
DRCTest -b <new-dir> <old-dir>
- DRCTest can be run with a single directory when testing within the old test directory.
DRCTest <new-directory>
... and the results would look something like this (This example is the top portion of the DRCTest
run for the vocabulary-words batch file run of OpenDRC1.1 vs. DRC1.2.1. NOTE: The Parameter file comparison is
large because OpenDRC outputs a annotated Parameter file and DRC1.2.1 does not, otherwise they are identical):
DRCTest comparison of directories:
Old: 'OfficialTests\DRC1.2USDError_Fix_Check\DRC1.2_allwords.drc'
New: 'allwords.drc'
Error Threshold(A): 0.00000150
Error Totals Multiplier(B): 2.50
Error Totals Threshold(A*B): 0.00000375
Processing RT files:
Comparing:
NEW: allwords.drc\allwords.RT (8031)
OLD: OfficialTests\DRC1.2USDError_Fix_Check\DRC1.2_allwords.drc\allwords.RT (8031)
---
Old: 'pooh phu 101 NAMED CORRECT' (4665)
New: 'pooh phu 100 NAMED CORRECT' (4665)
---
Old: 'scrounged skr6n_ 86 NAMED WRONG' (5465)
New: 'scrounged skr6n_ 86 NAMED CORRECT' (5465)
---
Old: 'straights str1t 79 NAMED WRONG' (6604)
New: 'straights str1t 79 NAMED CORRECT' (6604)
---
Old: 'strengths strENT 80 NAMED WRONG' (6631)
New: 'strengths strENT 80 NAMED CORRECT' (6631)
Processing STATS files:
Comparing:
NEW: allwords.drc\allwords.stats (8)
OLD: OfficialTests\DRC1.2USDError_Fix_Check\DRC1.2_allwords.drc\allwords.stats (7)
---
Old: ' CORRECT WRONG' (1)
Old: ' Named Lowac Total Named Lowac Total' (2)
Old: 'Count: 7999 0 7999 25 7 32' (3)
Old: 'Mean: 75.939 - 75.939 103.840 1000.000 299.875' (4)
Old: 'StdDev: 10.814 - 10.814 10.212 0.000 370.581' (5)
Old: 'Total Count: 8031 Mean: 76.831 StdDev: 29.372' (6)
New: '
' (1)
New: ' CORRECT WRONG' (2)
New: ' Named Lowac Total Named Lowac Total' (3)
New: 'Count: 8002 0 8002 22 7 29' (4)
New: 'Mean: 75.941 - 75.941 106.864 1000.000 322.448' (5)
New: 'StdDev: 10.813 - 10.813 6.405 0.000 382.231' (6)
New: 'Total Count: 8031 Mean: 76.831 StdDev: 29.372' (7)
Processing Parameter files:
Comparing:
NEW: allwords.drc\parameters (51)
OLD: OfficialTests\DRC1.2USDError_Fix_Check\DRC1.2_allwords.drc\parameters (41)
---
Old: '# DRC Parameters File' (0)
Old: '
' (1)
New: '#' (0)
New: '# OpenDRC Parameters File' (1)
New: '#' (2)
New: '# Run Date: Thu Apr 21,2011' (3)
New: '# Run Time: 14:59:09' (4)
New: '#' (5)
---
Old: 'ActivationRate 0.2' (3)
Old: 'FrequencyScale 0.05' (4)
Old: 'MinReadingPhonology 0.4' (5)
Old: '
' (6)
New: 'ActivationRate = 0.200000 # Scaling of input excitation/inhibition/CFS values (Epsilon)' (7)
New: 'FrequencyScale = 0.050000 # Scaling for written and spoken CFS values' (8)
New: 'MinReadingPhonology = 0.400000 # Word recognition threshold value' (9)
New: '#' (10)
---
Old: 'FeatureLetterExcitation 0.005' (8)
Old: 'FeatureLetterInhibition 0.15' (9)
Old: '
' (10)
New: 'FeatureLetterExcitation = 0.005000 # FL->LL Excitation parameter.' (12)
New: 'FeatureLetterInhibition = 0.150000 # FL->LL Inhibition parameter.' (13)
New: '#' (14)
---
Old: 'LetterOrthlexExcitation 0.07' (12)
Old: 'LetterOrthlexInhibition 0.48' (13)
Old: 'LetterLateralInhibition 0' (14)
Old: '
' (15)
New: 'LetterOrthlexExcitation = 0.070000 # LL->OIL Excitation parameter.' (16)
New: 'LetterOrthlexInhibition = 0.480000 # LL->OIL Inhibition parameter.' (17)
New: 'LetterLateralInhibition = 0.000000 # LL->LL Inhibition parameter.' (18)
New: '#' (19)
---
Old: 'OrthlexPhonlexExcitation 0.25' (17)
Old: 'OrthlexPhonlexInhibition 0' (18)
Old: 'OrthlexLetterExcitation 0.3' (19)
Old: 'OrthlexLetterInhibition 0' (20)
Old: 'OrthlexLateralInhibition 0.06' (21)
Old: '
' (22)
New: 'OrthlexPhonlexExcitation = 0.250000 # OIL->POL Excitation parameter.' (21)
New: 'OrthlexPhonlexInhibition = 0.000000 # OIL->POL Inhibition parameter.' (22)
New: 'OrthlexLetterExcitation = 0.300000 # OIL->LL Excitation parameter.' (23)
New: 'OrthlexLetterInhibition = 0.000000 # OIL->LL Inhibition parameter.' (24)
New: 'OrthlexLateralInhibition = 0.060000 # OIL->OIL Inhibition parameter.' (25)
New: '#' (26)
---
Old: 'PhonlexPhonemeExcitation 0.09' (24)
Old: 'PhonlexPhonemeInhibition 0' (25)
Old: 'PhonlexOrthlexExcitation 0.25' (26)
Old: 'PhonlexOrthlexInhibition 0' (27)
Old: 'PhonlexLateralInhibition 0.07' (28)
Old: '
' (29)
New: 'PhonlexPhonemeExcitation = 0.090000 # POL->PB Excitation parameter.' (28)
New: 'PhonlexPhonemeInhibition = 0.000000 # POL->PB Inhibition parameter.' (29)
New: 'PhonlexOrthlexExcitation = 0.250000 # POL->OIL Excitation parameter.' (30)
New: 'PhonlexOrthlexInhibition = 0.000000 # POL->OIL Inhibition parameter.' (31)
New: 'PhonlexLateralInhibition = 0.070000 # POL->POL Inhibition parameter.' (32)
New: '#' (33)
---
Old: 'PhonemePhonlexExcitation 0.04' (31)
Old: 'PhonemePhonlexInhibition 0.16' (32)
Old: 'PhonemeLateralInhibition 0.147' (33)
Old: 'PhonemeUnsupportedDecay 0.05' (34)
Old: '
' (35)
New: 'PhonemePhonlexExcitation = 0.040000 # PB->POL Excitation parameter.' (35)
New: 'PhonemePhonlexInhibition = 0.160000 # PB->POL Inhibition parameter.' (36)
New: 'PhonemeLateralInhibition = 0.147000 # PB->PB Inhibition parameter.' (37)
New: 'PhonemeUnsupportedDecay = 0.050000 # PB activation decay (after activation determined).' (38)
New: '#' (39)
---
Old: 'GPCPhonemeExcitation 0.051' (37)
Old: 'GPCCriticalPhonology 0.05' (38)
Old: 'GPCOnset 26' (39)
Old: '
' (40)
New: 'GPCPhonemeExcitation = 0.051000 # GPC->PB Excitation parameter' (41)
New: 'GPCCriticalPhonology = 0.050000 # GPC LL->GPC Letter Shift, PB Phoneme Activation Threshold' (42)
New: 'GPCOnset = 26 # Cycles before first letter available to GPCRoute' (43)
New: '#' (44)
New: '#' (45)
New: '# Parameters differing from DEFAULT:' (46)
New: 'ReportActGreater = 0.000000 # Report activations greater than this threshold.' (47)
New: 'SaveActivationLvls = true # Save activation levels in ACTS file.' (48)
New: '#' (49)
New: '#' (50)
Processing ACTS files:
Comparing:
NEW: allwords.drc\myrrh.acts (3603)
OLD: OfficialTests\DRC1.2USDError_Fix_Check\DRC1.2_allwords.drc\myrrh.acts (3479)
---
Old: 'Cycle51 GPCRoute myrrh (A)m->m (m)y->I (A)rr->r h->?' (1221)
New: 'Cycle51 GPCRoute myrrh (A)m->m (m)y->I (A)r->r (A)rh->r' (1221)
---
New: 'Cycle51 GPC3 # r' #=0.808745 (1225)
---
New: 'Cycle51 P3 # r' #=0.008249 (1231)
---
Old: 'TGPC Cycle51 #' #=2.426235 (1233)
Old: 'TPh Cycle51 #' #=0.440521 (1234)
New: 'TGPC Cycle51 #' #=3.234980 (1235)
New: 'TPh Cycle51 #' #=0.448770 (1236)
---
Old: 'Cycle52 GPCRoute myrrh+ (A)m->m (m)y->I (A)rr->r h->?' (1253)
New: 'Cycle52 GPCRoute myrrh (A)m->m (m)y->I (A)r->r (A)rh->r' (1255)
---
Old: 'Cycle52 GPC3 # +' #=0.821233 (1257)
New: 'Cycle52 GPC3 # r' #=0.821233 (1259)
---
Old: 'Cycle52 P3 # +' #=0.008377 (1263)
New: 'Cycle52 P3 # r' #=0.016557 (1265)
---
Old: 'TPh Cycle52 #' #=0.483108 (1268)
New: 'TPh Cycle52 #' #=0.491288 (1270)
---
Old: 'Cycle53 GPCRoute myrrh+ (A)m->m (m)y->I (A)rr->r h->?' (1287)
New: 'Cycle53 GPCRoute myrrh (A)m->m (m)y->I (A)r->r (A)rh->r' (1289)
---
Old: 'Cycle53 GPC3 # +' #=0.833062 (1291)
New: 'Cycle53 GPC3 # r' #=0.833062 (1293)
---
Old: 'Cycle53 P3 # +' #=0.016803 (1297)
New: 'Cycle53 P3 # r' #=0.024913 (1299)
---
Old: 'TPh Cycle53 #' #=0.526079 (1302)
New: 'TPh Cycle53 #' #=0.534189 (1304)
---
Old: 'Cycle54 GPCRoute myrrh+ (A)m->m (m)y->I (A)rr->r h->?' (1321)
New: 'Cycle54 GPCRoute myrrh (A)m->m (m)y->I (A)r->r (A)rh->r' (1323)
--
...
OpenDRC help:
Help for DRCTest can be obtained by issuing the command DRCTest -h which will display:
Usage: DRCTest {-acts|-rt|-stats|-{et|em|ml} <num>|-o <ofile>} <NewDir> {<OldDir>}
With respect to the files in the <OldDir> and <NewDir>
<OldDir> and <NewDir> default to the current directory
-acts Toggle the ACTS file testing (Default: On),
-rt Toggle the RT file testing (Default: On),
-stats Toggle the STATS file testing (Default: On),
-param Toggle the Parameter file testing (Default: On),
-et<num> Set the error threshold to <num> (Default: 0.0000015)
-em<num> Set the totals error multiplier (Default: 2.5)
-ml<num> Set maximum lines to compare (Default: 1000)
-o <file> Send output report to <ofile> (Default: DRCTest.txt),
-h This help message
Error Threshold:
The Error Threshold was implemented in DRCTest to deal with the rounding problem in the activation
values of OpenDRC when compared to DRC1.2. On a small number of node activation values for a typical test
case the OpenDRC and DRC1.2 values would differ in the least significant figure (i.e. 0.000001). Since this
difference seemed to be attributable to rounding errors it was considered insignificant. Thus for the DRCTest
report highlighting these rounding errors would be pointless and would tend to obscure the real errors.
However, it was also realized that if these rounding errors became large this would be of concern. Thus a threshold parameter
was implemented to mark when these errors became significant and thus report them. Since the activation for the layer
totals accumulate all the individual activations, and their rounding errors, the thresholding for the layer totals is
somewhat higher and is determined as a multiple of the threshold for the individual activations.
OpenDRC Output Files:
DRCTest will produce two output files for each run: DRCTest.txt and DRCTotals.txt. The purpose
of the DRCTotals.txt file is to allow checking of the final activation totals for each input and layer of the model
from one run to the next. An example of the DRCTotals.txt file (top few lines) for the entire-vocabulary batch file is shown below:
Old Old Old Old Old Old Old New New New New New New New
Filename Cycles TL TO TP TGPC TGPCR TPh Cycles TL TO TP TGPC TGPCR TPh
a.acts 83 7.918721 0.999773 0.941465 1.978057 0.000000 1.198325 83 7.918721 0.999773 0.941465 1.978057 0.000000 1.198325
ace.acts 75 7.774066 0.995438 0.784286 2.908535 0.000000 1.330626 75 7.774065 0.995438 0.784286 2.908535 0.000000 1.330626
ache.acts 104 7.975804 0.999948 0.892192 2.990202 0.000000 1.831712 104 7.975805 0.999948 0.892192 2.990202 0.000000 1.831713
ached.acts 105 7.977427 0.999955 0.903696 3.987812 2.990859 2.590605 105 7.977427 0.999955 0.903696 3.987811 2.990859 2.590605
aches.acts 115 7.989333 0.999989 0.859814 4.992800 0.000000 2.731116 115 7.989332 0.999989 0.859814 4.992799 0.000000 2.731116
act.acts 65 7.589221 0.989569 0.799373 3.778371 0.000000 1.712995 65 7.589221 0.989569 0.799373 3.778371 0.000000 1.712994
acts.acts 67 7.623394 0.990398 0.804004 4.745996 0.000000 2.142757 67 7.623394 0.990398 0.804004 4.745995 0.000000 2.142757
ad.acts 66 7.630669 1.291871 0.805170 2.842752 0.000000 1.277251 66 7.630669 1.291872 0.805170 2.842752 0.000000 1.277251
add.acts 65 7.627223 1.259994 0.804648 2.846117 0.000000 1.276476 65 7.627221 1.259994 0.804648 2.846117 0.000000 1.276476
adds.acts 70 7.687630 0.992794 0.802572 3.831425 0.000000 1.779910 70 7.687630 0.992794 0.802572 3.831425 0.000000 1.779910
ads.acts 71 7.695808 0.992783 0.803371 3.835839 0.000000 1.792317 71 7.695809 0.992783 0.803371 3.835839 0.000000 1.792317
adze.acts 72 7.684597 0.991386 0.799519 3.829806 0.000000 1.803569 72 7.684597 0.991386 0.799519 3.829806 0.000000 1.803569
aft.acts 137 7.997979 1.000000 0.989879 3.998909 0.000000 3.338436 137 7.997980 1.000000 0.989879 3.998909 0.000000 3.338436
age.acts 68 7.676345 0.993319 0.802613 2.868996 0.000000 1.282591 68 7.676346 0.993319 0.802614 2.868997 0.000000 1.282592
aged.acts 72 7.740624 0.994990 0.806583 3.860001 0.000000 1.748347 72 7.740624 0.994990 0.806583 3.860001 0.000000 1.748347
ah.acts 65 7.581070 0.989366 0.802054 1.886989 0.000000 0.851003 65 7.581069 0.989366 0.802054 1.886988 0.000000 0.851003
aid.acts 67 7.675333 1.265515 0.805353 2.862691 0.000000 1.245814 67 7.675333 1.265515 0.805353 2.862691 0.000000 1.245814
aide.acts 68 7.681417 1.294733 0.806951 2.869317 0.000000 1.243943 68 7.681416 1.294733 0.806951 2.869317 0.000000 1.243943
aides.acts 70 7.673992 0.991655 0.787617 3.824080 0.000000 1.676659 70 7.673991 0.991655 0.787617 3.824080 0.000000 1.676659
aids.acts 70 7.677716 0.991929 0.788932 3.826086 0.000000 1.682742 70 7.677715 0.991929 0.788932 3.826086 0.000000 1.682742
ail.acts 71 7.701602 0.992662 0.768043 2.879225 0.000000 1.273058 71 7.701602 0.992662 0.768043 2.879225 0.000000 1.273058
ailed.acts 73 7.711349 0.992166 0.774290 3.844232 0.000000 1.725913 73 7.711347 0.992166 0.774290 3.844232 0.000000 1.725913
ails.acts 74 7.734836 0.993453 0.786890 3.856893 0.000000 1.749260 74 7.734836 0.993453 0.786890 3.856893 0.000000 1.749260
aim.acts 68 7.651315 0.991386 0.780742 2.858886 0.000000 1.255748 68 7.651316 0.991386 0.780742 2.858886 0.000000 1.255748
aimed.acts 70 7.689713 0.992639 0.786255 3.832552 0.000000 1.686841 70 7.689715 0.992639 0.786255 3.832552 0.000000 1.686841
aims.acts 69 7.668838 0.991973 0.790474 3.821295 0.000000 1.679468 69 7.668837 0.991973 0.790474 3.821295 0.000000 1.679468
air.acts 70 7.721375 0.994918 0.804100 1.924806 0.000000 0.854877 70 7.721375 0.994918 0.804099 1.924806 0.000000 0.854877
aired.acts 75 7.763401 0.994730 0.774443 2.904223 2.904223 1.272831 75 7.763400 0.994730 0.774443 2.904223 2.904223 1.272831
airs.acts 75 7.768180 0.995062 0.780058 2.906155 0.000000 1.292588 75 7.768180 0.995062 0.780058 2.906155 0.000000 1.292588
aisle.acts 111 7.986255 0.999987 0.936991 3.992579 0.000000 2.102594 111 7.986256 0.999987 0.936991 3.992579 0.000000 2.102594
aisles.acts 116 7.990242 0.999993 0.938383 5.992097 0.000000 3.239523 116 7.990242 0.999993 0.938383 5.992096 0.000000 3.239523
aitch.acts 75 7.751392 0.994067 0.780474 2.899367 0.000000 1.312191 75 7.751392 0.994067 0.780474 2.899367 0.000000 1.312191
al.acts 67 7.608918 0.989100 0.777202 2.841754 0.000000 1.297282 67 7.608917 0.989100 0.777202 2.841754 0.000000 1.297282
alb.acts 72 7.667901 0.989464 0.766564 3.820823 0.000000 1.727008 72 7.667902 0.989464 0.766564 3.820823 0.000000 1.727008
...
OpenDRC home
|