Statistic	Applies to	Technique	End-to-end code
Total Completion Score (TCS)	Real and calibration system	Adjusted Wald	Yes
Validity via calibration system	Calibration system	Mann-Whitney	Yes
Perfect Ballot Index (PBI)	Real system	Adjusted Wald	Yes
Voter Inclusion Index (VII)	Real system	Capability Index	Yes

The technique column links to the Perl scripts that handle these statistics individually.

All the effectiveness statistics are also supported by end-to-end Perl scripts, one to transcribe the raw ballot data for each system into an election file and a second to tabulate the results based on that file. Here is the recommended process:

1. Gather up all the ballot images (electronic or paper) for a given system, as generated in the "mock" election. Include only those NCAST ballots that have been successfully cast by participants.
2. Transcribe these images, using the transcribe-ballots script. The script allows easy entry of "correct" contest choices and also handles write-ins. This is by far the most labor-intensive step. The result is an "election file" that contains all the ballot information: contests and candidates voted for.
3. The tabulate-election script will score each contest (this can be tricky to get exactly right, if done manually). Maximum score per contest = # candidates you are allowed to vote for. The 20 contests have a total of 28 points available.
4. The tabulate-election script will sum up contests and assign a score of 0-28 to each ballot. The set of ballots in the election therefore determines the raw score distribution - basically an array of 100 or more scores, each from 0 to 28.
5. For the voting system under test (VSUT), the tabulate-election script will do the math to derive the TCS, PBI, and VII and their CIs from the raw score distribution, and report the pass/fail result.
6. For the calibration system, the tabulate-election script will do the math to derive the TCS and Mann-Whitney statistic and their CIs from the raw score distribution, and report the valid/invalid result.
7. The tabulate-election script will also make the necessary allowances for any "invalid ballots" cast by bad-faith participants.

Both scripts (transcribe-ballots and tabulate-election) make use of an auxiliary file named perfect-ballot.txt which describes all the ballot contests and indicates the "correct" choices. In order to run these scripts, you must download perfect-ballot.txt into the same directory as the scripts.

Even though the Perl files are source code, they are named with a ".txt" extension so as to be easily viewable and not trigger execution. If you wish to run these scripts, rename them with your system's extension for Perl, typically ".pl". E.g. rename "MW-pl.txt" as "MW.pl".