21st February 2017 response to some of Perlin’s presentations
10th January 2017 Dr Mark Perlin states at minute 11:45 in https://www.cybgen.com/information/presentations/2016/IPAC/Perlin-DNA-TrueAllele-statistical-analysis-probabilistic-genotyping/page.shtml#Politics: PCAST Report “However using a higher threshold of 50rfu along with careful human data selection…” In the interests of keeping the scientific record correct the 50rfu was the only one used for the Frye and no others were investigated until after the trial. That was selected because that is the AT at NYSP. The epgs were edited, not by me, for spikes and pull-up without reference to Mr Hillary’s profile. This is the normal method. The unedited profiles are attached. unedited-hillary-epgs-at-50-rfu
The peaks used are here: Hillary input data
7th January 2017
Dr Mark Perlin has published a response to a letter from Bill Fitzpatrick but did not post the original. To correct the record here is the original. letter from Bill Fitzpatrick I can personally confirm much of what DA Fitzpatrick states. I repeat my offer to Mark to have a scientific discussion off line on the matters relating to Probabilistic genotyping in People v Hillary.
22nd December 2016: In his YouTube video Dr Mark Perlin implied that STRmix had not honored its disclosure undertakings (see powerpoint slide 33 https://www.cybgen.com/information/presentations/2016/Legal-Aid/Perlin-Fighting-for-DNA-justice-genotyping-software-in-the-Hillary-acquittal/page.shtml). He has provided the retraction shown here: Perlin retraction . We will allow the reader to decide whether this is an honest and candid retraction.
Perlin critises the workflows used by some STRmix users. The workflow he describes requires both the human operator and the software to indicate inclusion before an inclusion is reported. I find it very hard to see any fault with this.
It is becoming a full time job correcting Dr Perlin. May I suggest to him that falsely denigrating STRmix does not make TrueAllele any better.
20th December 2016: Dr Mark Perlin has recently posted an email blast to a wide group and a YouTube video. Below are some corrections that I hope are of use to anyone facing questioning about any such matters in court.
Comments on People v Hillary (New York)
Mark Perlin has made certain comments regarding the results of STRmix analysis on the Hillary profiles in his hands. Specifically he reports an LR of 0 using AT = 30 or 40 rfu. I have not asked for his input data. Recall this instrument is an ABI 3500 machine and 30 or 40 rfu are very low ATs. He states that he has obtained a version of STRmix and ran it, untrained, using the same input settings as used at AT = 50rfu.
Between 30 and 50rfu there are three artifacts. These are two pullups and an area of raised baseline with poor morphology. There is quite a lot of baseline noise with poor morphology just under 30 rfu.
Unlike TrueAllele, STRmix does need some artifacts to be manually removed. It does not model spikes, pullup, baseline, and at this stage it does not model the double back stutters and the exotic stutters (for example 2bp at SE33). Analysts remove these artifacts prior to a STRmix run and this is done without reference to the profile of the POI. This general approach is how DNA analysis has been performed for 20 years and is not new. STRmix deals with baseline noise using the standard method of a laboratory set AT. TrueAllele does model noise and artifacts and this would be a great advance if effective. It does, however, seem to be in this area that TrueAllele has drifted off the proper answer in Hillary. Pattern recognition and image processing is one of the areas where humans still often outperform computers.
I have inferred that Mark Perlin has not removed the artifacts and has not turned on the drop-in function. This has invoked the LR = 0 result. If he performed the analysis in this fashion, it would be misuse of the software. Had he performed one or both of these actions he would have produced a more meaningful LR. Mark Perlin would agree with me that proper training is needed when using software and he has not had this.
I anticipate a cry of subjectivity from Mark Perlin. We do still rely on human judgement for artifact removal and this is correctly described as subjective. STRmix is a step towards objectivity but cannot be run effectively without care and judgement. But perhaps both Mark and myself should make clear that the softwares contain elements of subjectivity programmed into them. I would raise as an example, amongst others, the fact that TrueAllele has a lower limit of LR = 0.01 for any locus. Where did 0.01 come from? I assume Mark’s judgement. It is therefore programmed subjectivity. It would enforce precision but it is still subjective.
Mark Perlin did not claim this Hillary an exclusion when working for the prosecution. His email used the phrase no statistical support, and he recommended further analytical work. The difference is therefore between inconclusive and inclusion. I am not even able to verify from Mark Perlin’s email that we looked at the same set of profiles.
The analysis of the Hillary files reported on the tab “NY v Hillary” was done by a third party who was fully trained by Cybergenetics with a legal copy of TrueAllele. The analysis reported was not by me but from the third party. Thus in the Hillary case the STRmix result is correct, and the TrueAllele result wrong. I repeat my comment that I consider both software to be credible products and this to be one of the rare instances of a difference (recall the actual difference is between inconclusive for TrueAllele and Inclusion for STRmix).
I invite Mark Perlin to an off-line scientific discussion of this case if he desires which we could do by email.
Comments on 100:1 mixtures
100:1 mixtures have a trace that is often barely visible above the AT. STRmix performs as expected at such levels. That is: As the template of the trace is lowered the LR tends, correctly, to 1. Replication or the use of a conditioning profile helps a lot. Mark Perlin has made certain comments about a validation study disclosed in Hillary, which he misascribed to ESR. He appears to be commenting on the single analysis graph. In the Hillary case I used three replicates and the better comparison is the three replicate graph.
Comments on peer review made to PCAST
Mark Perlin and myself were invited (along with Drs Gittelson and Butler) to speak with PCAST 18th November 2016. This was a great opportunity for us to open conversation with this group, an honour, and I found the PCAST representatives very open minded and constructive. We need to recognise that these are very intelligent and respected individuals and when they offer to have a conversation with us we should accept it. In this respect Mark Perlin misused his opportunity, speaking at rather than with the committee and exiting prematurely “I’ve got a plane to catch.” His views about nexus (nexus is a plural noun and is its own plural, in this case I mean plural) should be considered in light of the fact that the one invitation for conversation that I have seen him attend he did not use.
Mark Perlin commented to PCAST and on his website on the short turnaround time for two papers involving me [1, 2] implying that peer review had been absent or light.
With regard to Bieber et al. , the paper was about one year in peer review. The apparent short time frame is because one journal in the BMC group was terminating, and thus, the reviewed and completed manuscript was transferred to another journal in the group.
With regard to Coble et al. , this product is a DNA Commission paper, and they typically are not reviewed. They are, in effect, a position statement. There were five developers of current software on the Commission. Personally, and I can also vouch for Mike Coble, we do not use such committees to advance any one software but rather to advance standards and I am proud of the product. But if Mark Perlin has a criticism perhaps he could take the normal course and write a letter to the editor.
- Bieber, F.R., et al., Evaluation of Forensic DNA Mixture Evidence: Protocol for Evaluation, Interpretation, and Statistical Calculations using the Combined Probability of Inclusio. 2016.
- Coble, M.D., et al., DNA Commission of the International Society for Forensic Genetics: Recommendations on the validation of software programs performing biostatistical calculations for forensic genetics applications. Forensic Science International: Genetics, 2016. 25: 191-197.
Previous postings: Mark Perlin has made certain allegations regarding my actions in NY v Hillary https://www.youtube.com/watch?v=onCRd1ccUMchttps://www.cybgen.com/information/presentations/2016/Legal-Aid/Perlin-Fighting-for-DNA-justice-genotyping-software-in-the-Hillary-acquittal/page.shtml?utm_source=October+2016+Newsletter&utm_campaign=October+2016+Newsletter&utm_medium=email. These allegations cover a range of matters but notable is the suggestion that the AT was selected to maximise the evidence against Mr Hillary. He bolsters this suggestion by quoting the LR at AT from 70 to 30RFU.
The AT was not set by me at all. It is the NYSP AT and was applied by Dr Jo Bright who started preparing the case before I was even involved. She did this by applying the NYSP AT and observing the background noise. At no time prior to the testimony did we ever run at other ATs although we have subsequently run at 30RFU. Contrary to Perlin’s allegation, we get a higher LR at 30RFU. We have never run at 40,60, and 70rfu.
My casework, like most forensic casework, is peer reviewed and then gets an administrative review before being issued. Hence Perlin’s suggestion requires collusion by me and three others. But enough of this. My basic defence is that I did not select the AT to incriminate Mr Hillary and I would not do such a thing. It is not within my belief structure and is not the way a forensic scientist should behave. I know that this is also the way the other three scientists involved in this case feel. Perlin should withdraw his comments or prove them.
Elsewhere Perlin publishes on his page an affidavit from Dominic Saraceno https://www.cybgen.com/information/newsroom/2015/dec/New-York-motion-filed-on-why-STRmix-is-not-TrueAllele.shtml. This asserts, inter alia, that “STRmix is a recent foreign copycat software…” Mr Saraceno confirmed to me by email that the information in this affidavit originates from an interview with Mark Perlin.