Signature Verification in the Vote-By-Mail Era

BRIAN SANTO: I’m Brian Santo, EE Times editor-in-chief. You’re listening to EE Times On Air, and this is your Weekly Briefing for the week ending November 13th.

Election Day in the United States has come and gone, but the election process is dragging on and on, largely because the process of counting votes is dragging on and on – specifically the process of counting ballots that were mailed in.

We are going to devote this whole episode to one topic. We’re going to take a look at voting by mail from two directions: from the technological perspective and from the policy point of view.

Today we speak with Greg Council from Parascript, a company that provides the automated signature verification systems used by several jurisdictions in the United States to assure the integrity of the vote-by-mail process, and we also have a conversation with lawyer and public policy expert Portia Allen-Kyle about making sure that technologies such as Parascript’s work the way they’re supposed to work.

Today, voting in America.

 

PRESIDENT-ELECT JOE BIDEN: The people of this nation have spoken. They have delivered us a clear victory. A convincing victory. A victory for “We the People.” We have won with the most votes ever cast for a presidential ticket in the history of the nation: 74 million. What I must admit is, it surprised me. Tonight, we’re seeing all over this nation, all cities and all parts of the country, indeed across the world, an outpouring of joy, of hope, renewed faith in tomorrow bringing a better day. And I’m humbled by the trust and confidence you’ve placed in me. I pledge to be a President who seeks not to divide, but unify. Who doesn’t see red states and blue states, only sees the United States.

 

BRIAN SANTO: Politics in America have always been contentious. The Founding Fathers were never all in full, harmonious agreement. Politicians in succeeding generations certainly weren’t either.

Divisive political arguments are the norm, not the exception. What is not all that common, however, is when the election process itself is the political issue.

When mechanical and electromechanical voting machines started getting replaced by fully electronic systems, we created a security problem. For at least 30 years, white-hat hackers have been demonstrating how easy it can be to hack some of these machines.

Now, we haven’t heard too much about voting machines getting hacked this election cycle. Instead, the brouhaha has been about voting by mail.

The military first used voting by mail in the Civil War, and resumed the practice during World War II. Starting in the mid-1970s, California, Oregon and Washington began experimenting with voting by mail. These days, in the modern military, voting by mail is now a common option available to members posted away from home. Most of the 50 states now offer voting by mail under at least some circumstances. In 2000, Oregon went completely vote by mail, followed by a few other states, including Washington and Colorado.

Voting by mail works! When there’s been any fraud detected, it has been negligible. One of the reasons we know that there is negligible fraud is because mailed ballots can be tracked, and once received by elections offices, several of the jurisdictions that have implemented voting by mail have adopted signature verification systems as a backstop.

And now along comes 2020, with both an election and an epidemic that makes voting in person a health risk. The obvious remedy was greater adoption of vote by mail.

Long experience with safe and secure voting by mail, coupled with a thorough lack of evidence of fraud, did not and has not stopped the President of the United States from claiming that fraud is rampant in vote-by-mail systems. He pointlessly made voting by mail a political issue, and as the counting of mailed ballots in several states goes on, it remains an issue. And since it remains an issue, it got us to wondering: How do those signature verification systems actually work? And do they work the way they ought to?

To answer the first question, we called up Parascript, which was founded to develop artificial intelligence and machine learning systems to handle handwriting. One of the earlier versions of the company’s technology was used by Apple in its Newton personal assistant. The company has branched out into document processing systems for clients in a variety of industries, including banking and insurance. Voter ballots are just another kind of document, and the company’s long history with handwriting analysis came in handy for performing signature verification for elections boards. It’s used by the State of Oregon and several other states and counties.

Greg Council is vice president of marketing and product management at Parascript. My colleague, Junko Yoshida, and I caught up with him in a videoconference. Here’s Junko:

JUNKO YOSHIDA: Greg, the first question… I think we need to understand the set-up. How is your technology, and this piece of software, being applied in the real setting of signature verification at the scene of, not the crime, but at the scene of where people are matching signatures? Can you explain how it’s applied and how they actually compare signatures on the ballot against the official record? Let’s start from here.

GREG COUNCIL: It’s a good place to start as well, because within the realm of machine learning, there seems to be this fixation on completely autonomous things. How I best characterize automated signature verification, especially within the scope of vote by mail or absentee balloting, is it’s more of an assistive automation, not a complete what we call “straight-through processing,” where there’s no manual intervention. So in the cases where they implement or they utilize signature verification — and there’s something like 33 states now that default to using some sort of signature matching, so to speak, and most of it is manual, but technologies such as ours have gradually caught on — is that they’re using it at a very low level of tolerance. I’ve characterized this before that most election officials are extremely risk-averse, for good measure, because the whole election process has to have zero questions about the “realness,” so to speak, or that it’s completely legitimate.

So what they’ll do is, they implement the signature verification piece of it on the front end, taking in the easiest to process ballots. So the ones that are pretty much a chip shot, so to speak. Allowing for a certain percentage of those ballots to be automaticallyh processed, and then a larger percentage of them will be kicked out for a secondary and tertiary review using the information or the output from signature verification as part of the information, but definitely they will look at it and have one or two reviewers actually look at the signatures as well.

So what it does, ultimately, is it allows the easy stuff, so to speak, to be able to be processed automatically, which reduces the amount of work that they have to do. And as we’ve seen with this recent election, they were still able to get through a large amount of ballots in a relatively short amount of time. But then there’s still a reliance on us humans to also participate in the signature review process. So it’s not a complete robotic type of thing, it’s the humans with the machine learning, or with the robots, working in concert, so to speak.

JUNKO YOSHIDA: Right. So essentially your piece of software is being used as the first pass of this matching? Is that what it is?

GREG COUNCIL: Right. In most cases… We’re not privy to all of the implementation details, because we do work through partners that are the ones, the organizations that help these counties implement the software. But in the ones where they’ve…

You can go and find the processes on line. For instance, Denver here. Colorado is a completely vote-by-mail state. One hundred percent of the ballots are processed in the same way. While we do have the ability to go in and vote in person, a large percentage of the citizens here vote by mail. And they use this same process, where it’s a first pass.

The signature verification software does an automated analysis of every single ballot. But then they have a secondary and even a tertiary process to deal with those hard cases.

The ultimate goal is to not invalidate a ballot. The amount of fraud is incredibly low. So you have to build a process that errs on… to be very cautious about invalidating a ballot too early in the process and cause this really lengthy and time-intensive curing process.

JUNKO YOSHIDA: I assume that each county or each state has different rules, but essentially, what is this software being asked to compare? Signature on ballots against what exactly?

GREG COUNCIL: It’s a really great question because I heard of a story where I think it’s Clark County, Nevada. There was some sort of lawsuit or question regarding the DPI or the quality of the image used to analyze. In all cases of signature verification, we have what we call the “reference signatures,” which are the legitimate or authenticated signatures. And then we use the analysis of those and compare it with what we call a “candidate signature.”

Best practices are really to use more recent signatures. Banks have been using signature verification software for well over 10 years. And what they do is, they typically will recycle signatures using more recent signatures or having a preference to do that. And they will take signatures anywhere they can to ensure that they’ve got a nice range of different types of example signatures for each account.

But when the time comes for the software to analyze the signatures, the best performance is with signatures that use similar DPI. All of our software is trained at a DPI between 200 and 300. It’s not too high in resolution, but it’s not too lossy in terms of resolution. And so best practices are to have candidate signatures scanned at the same DPI level. We do like to have reference signatures that are of matching DPI without a whole lot of data loss.

Now, the question regarding this specific county — and it’s actually something that most counties are dealing with — they often will take signatures for manual review from DMV records. These are typically stored at a lossy 96 DPI.

You can use machine learning to correct for that as well. So what we did earlier this year is have a new release of our software, which will actually enhance the image, but also adjust down and basically take an apples-to-orange issue where we’ve got two images at different DPIs and correct for that so that we can do a like for like. That did have a very positive effect in terms of protecting the reliability of the software.

BRIAN SANTO: That’s really fascinating. I imagine that in most cases you have a drivers license signature, you have a signature from a previous election cycle to compare against. The low-incident cases: a woman who may have just gotten married or anybody who just got married and may have changed their name or either changed it entirely or now hyphenates, or a young person who doesn’t have a drivers license yet and has never voted and has no signature. It’s those low-incident cases that I would guess would trip up the system.

What’s the actual experience? And how do you deal with that?

GREG COUNCIL: In some cases, that’s why humans are involved in this. It’s all about the quality of the input data. We write a lot about access to data and the proper fitting of data to the problem that we’re trying to train our algorithms for. Obviously the whole idea of signature fraud or forgeries is such a low incident. How do you create a quality input data set to train the algorithm so that it can detect forgeries across a range of types of use cases?

One of the use cases that you just provided was where there’s a name change. Which is kind of similar to what we call an “unskilled” or “blind” forgery, where somebody’s just writing in a name; it doesn’t even happen to even be the same name. You can detect that, and it’s not necessarily you’re detecting and trying to recognize the signature and resolve it into an actual name. For instance, Greg Council. But you’re using things like looking at the stroke and of their commonality. So it’s entirely possible for our software to detect that, even if I write Bob Jones, it’s going to figure out that I’m the same signer as when it looks at my signature. It feels really strange, but it can do that.

But that’s not really where we want to train our software for. We want to train our software to be able to allow legitimate signatures to go through, while being very careful about what we have as a positive. Because we don’t want to have false positives. That creates a whole bunch of problems.

In the cases where signatures change… and that’s kind of a story that’s been brought up a lot in the media, which is, “The way my signature was 10 years ago is nowhere near what it is now.” Or maybe somebody of older age has a signature that’s shaking a lot more, it has that type of effect to it. There’s only so much that the signature verification software can do, and that’s why it’s a combination of why we always recommend that you want a consistently refresh your signature, your collection of authentic signatures (basically legitimate signatures) as well as be able to not… You can’t completely rely on the software 100% of the time.

It’s a lot of best practices, it’s a lot of testing to identify the performance of the software on a given election or on test data that they have, and then it’s a good process in terms of just making sure that you’re focusing signature verification on what it does really best and where it performs best, with the goal of reducing the amount of work. But it’s never going to be 100% of the work reduction.

JUNKO YOSHIDA: I think this might be a good segue to ask Greg about the technology building blocks used inside Parascript’s signature verification methods. Can you break it down for us?

GREG COUNCIL: It’s proprietary, so I can’t really divulge all our secrets and everything like that. But the team here utilizes a number of different algorithms, each one keying in on different features of a signature. So it’s based on a lot of good forensic science.

The software itself has won some international awards, scientific awards, where you have a bunch of different organizations, mostly universities, competing to try to solve the problem in the most efficient, most precise way. The only way to really do that is, you can’t rely on one single type of algorithm. So what they is, they blended a number of them. They like to use the word “orthogal.” It’s as if you’re using a number of different experts, each with their own opinions, and then on top of that we have some voting algorithms that will then take all of these independent opinions and identify the most precise one or the most precise answer.

Getting back to it, it is predominantly neural networks. Traditional ones. Increasingly we are using deep learning neural networks, but they have weaknesses as well. But there are other machine learning algorithms that we’re using there on the computer vision side that definitely rely on neural networking.

We’ve also got our own technologies involved with how we deconstruct the shapes of signatures to identify, to break them down into those types of features that are then routed to a different algorithm.

The software’s always changing, so it’s kind of a funny thing. One of the tests that incoming candidates for our R&D team have is, they have to try to create a signature verification algorithm. Occasionally they’ve done some a stellar job that they end up being incorporated into the product. That’s one way they’re always staying fresh on that.

BRIAN SANTO: Do you automatically hire those people?

GREG COUNCIL: Typically I think they’re an automatic hire.

JUNKO YOSHIDA: I read your white paper, and it actually goes into a little bit of detail in terms of what exactly your software is looking at. One of the things that struck me is that algorithms actually do look at the entire signature in a holistic way, just like a human judge would do. But at the same time, you do actually break it down in terms of taking our certain segments of the signature. Is this the descriptive language you talked about? XE or something?

GREG COUNCIL: Yeah. XR elements. I wouldn’t call it a language, but it’s a way that we deconstruct hand-written information. A similar technique is used for our handwriting recognition, where it basically allows the algorithms to look at different types of features. It’s something that the team developed a couple of decades ago to solve handwriting recognition. And it’s been applied to other types of hand-written information as well.

You’re right. It uses a holistic method. It uses fragments of signatures. One of the funny use cases or stories that came up is, our software is used all over the world. In several cases, it’s used in India, where it’s often mistakenly thought of as handwriting-recognition software, but we’re not doing anything like trying to transcribe a signature from John Smith into the words “John Smith” in ASCII or something like that. But people thing of it that way, so they were asking, We’d like to be able to detect when the signature doesn’t have their first and last name. And we have to artfully explain to them, The software can’t do that. It’s not focused on detecting partial signatures. It’s focused on identifying forged signatures from authentic signatures.

In order to do that, we don’t need the entire signature. It’s fine if it’s just a first name. Because that’s all we’re trying to gear on, is the answer to the question, “Is this a fraudulent signature or not?”

JUNKO YOSHIDA: Interesting. What you just said to us… I’m not a lawyer, but if I’m a lawyer and an activist, one of the first things I would flag is… it’s true with any AI. In essence, it is a black box. That’s actually more than a little concerning to the concerned citizens. Can you explain how you actually create transparency to the citizens as well as the municipalities that adopt your technology?

GREG COUNCIL: Explainable AI is a trending interest for that very reason. It’s hard to kind of figure out what’s going on in the mind of a neural network.

In our case, there’s not a whole lot of instrumentation that we can do. Certainly we have instrumentation that allows us to expose what features are being looked at and which potential things are potentially causing the software to focus on one area or not, and do that. Ultimately, though, the real explainable piece of it is, with a lot of testing. So providing test data, trying to be as transparent as possible with the type of input data sets that we train our software on. But ultimately, the counties are satisfied when they test the system on their own data and compare that with the results of their manual processes and do comparisons and things like that.

It’s also a reason why… I started off this conversation calling it an “assistive technology.” There’s not as much emphasis — or any in this case — with trying to get the software and completely rely on it 100%. It is another set of eyes that a reasonable portion of ballots can go through, but it’s by no means the full set of eyes.

JUNKO YOSHIDA: What kind of error rate does your signature software have?

GREG COUNCIL: The error rates themselves are going to be dependent on the amount of automation. We do operate on the concept of a confident score that, over the entire set of representative data, we can plot that. It’s called a reject curve, typically. We plot the reject curve based off of the percentage of accepted ballots (or in this case, accepted signatures versus the ones that are rejected), and so we can plot the error curve.

For instance, within our own data sets, we can achieve about a 70 to 75% accuracy rate, holding at about a 98% accurate rate, or 2% error. That error rate can be halved. Of course, there’s a corresponding reduction in the amount of accuracy. Which is why, again, the counties that are using the software will adjust that error rate very low. So they’re looking at trying to beat the error rate of a human, which is all over the place, basically. But well-trained humans in terms of looking at signatures and things like that. So they’re really focusing on beating that. Which means there’s a downward adjustment on the amount of ballots that can actually be processed completely with the software.

JUNKO YOSHIDA: So positioning this software as more of a human assistant. That is really a much more forgiving environment compared to something that I’m writing about is more of autonomous driving. That’s an entirely different scale.

When you talk about AI, you have to consider that, because AI doesn’t know what it is doing, it actually completely lacks human common sense. In other words, when you do the signature verifications, what do you identify as a potential edge cases or corona cases?

The DPI thing was kind of interesting. I had never thought about that. That’s a good one. But if the signature was stained by something or the ink was… There was a story about Sharpies… Anyway, what do you consider as a potential case that could trip up the signature verification software?

GREG COUNCIL: Certainly the DPI one, from a fundamental perspective on performance, is probably the biggest one. It’s going to be the most common. So if we want to look for outliers and things like that, the software, just to be very honest, doesn’t do as well with highly skilled forgeries. So you can fool the software just like you can a human being, an expert, if you are expert at forging signatures, or you have that ability.

I was just watching the movie, Catch Me If You Can, which is all about check fraud. I was thinking that would be an interesting case to see if you could get some of the artifacts of what he did and see if the software would have been able to pick those up. The level of skill that a forger would have is definitely there.

And that gets back to the input data sets. Having access to the full range of forged signatures and all the different types of use cases and everything like that, we’re just never going to be able to boil that ocean, so to speak. Again, basically the performance of the software is determined by the input data sets and the representativeness of those data sets to be able to do that.

So I would say the highly skilled forger is one area where I think they’re going to find loopholes all day long. In most cases, that’s pretty rare.

JUNKO YOSHIDA: Can you tell us which states are actually already using your piece of software?

GREG COUNCIL: Let’s see. I want to say we’re in 11 states right now. And in about upwards of 70 counties. You can fact check me on that, but it’s somewhere around that number. Obviously the states that are 100% vote by mail like Nevada, like Colorado, I think Oregon is one of them, utilize our software. So we’re obviously in a large number of those counties within those states. And increasingly we’re starting to move into other states that are traditionally kind of just adopting the technology. One county in Florida is using our software as well. Which is a good thing. We don’t want to have that revisit of 2000 where they had the hanging chads. I think they’re big adopters of that.

I can say that the amount of interest ginned up by what happened with the pandemic and everything was just amazing with respect to this type of technology.

BRIAN SANTO: Do you have recommendations for best practices for the municipalities that adopt the techniques in terms of, for instance, using high resolution DPI? What percentage of ballots they should run this on; what to do with the ballots afterwards; how they should properly use this technology most effectively.

GREG COUNCIL: We do offer best practices from a technical level. Those things about DPI and testing data and making sure that they have an adequate amount of sample data that they can use to test it on their own systems and arrive at the proper confidence thresholds that they want to use to control for error. We typically will not be involved in best practices to the overall workflow. And that’s just because we’re not the experts when it comes to vote by mail.

Ultimately, the performance is determined on production data. So we support our partners, these vote-by-mail systems partners, to be able to educate these counties on how ASV works, how you should optimize for it and tune it. It’s ultimately their decision on how much of those ballots are automated or fully automated versus what goes to a manual process. Things like that.

And for that, they look at best practices from the states that have been operating for several years in this method. I think there’s the National Vote at Home Alliance or Coalition that offers a whole lot of really good guidance and things regarding best practices.

JUNKO YOSHIDA: So your company name actually popped up in several articles that I read about this whole vote by mail thing. Are you the only game in town?

GREG COUNCIL: It’s hard to say. We know of some competition out there. Definitely with the advent of machine leaning and cloud-based machine learning and deep learning, it’s something that organizations or those that have the technical wherewithal can solve. If they have the data. Again, it’s all about having the representative data sets and doing that.

There’s obviously the art of knowing what features to focus on and things like that.

Back to the point: There are, at least to my knowledge, two other organizations or companies that offer this software, but we have been in the market the longest, so we’ve been able to foster those relationships with the banks that are using our software, that really have provided that kind of Good Housekeeping Seal of Approval, so to speak.

JUNKO YOSHIDA: Is there any hardware required to run your software?

GREG COUNCIL: Really no. I think increasingly, with computer vision, there’s this desire to move to different types of chipsets and things like that. But we still operate on standard Intel-like CPUs using off-the-shelf hardware. So the requirements really are driven by throughput. So we have minimal requirements, but typically on a two-core, 3ghz machine, you can process signatures in sub-seconds. So it’s a pretty quick process there.

BRIAN SANTO: Again, that was Greg Council, a VP at Parascript, talking about the company’s technology for verifying ballot signatures.

The computer industry knows that results are only as good as the data. The phrase “garbage in, garbage out” has been traced back to at least the 1950s, and if you squint a little, the concept might even go back to Charles Babbage himself.

In more recent years, the garbage in, garbage out phenomenon cropped up again in an unexpected way: as bias in AIs.

Artificial intelligence systems aren’t biased in and of themselves. But an AI can end up exhibiting biases that were inherent in the data they were trained on. It’s rarely a conscious thing; these biases tend to be hidden from recognition even by the people assembling the data.

Famously, some facial recognition systems that originally seemed to be quite successful proved to be notably less effective at correctly identifying certain categories of people – women, and people of color, for example. To put a fine point on it, the system could potentially fail to correctly identify more than half of humanity.

It’s bad enough when your doorbell can’t tell if the person on your doorstep is your brother or just some random Jehovah’s Witness, but you need to get voting right.

In all of our political arguments about what America is all about, this is inarguable: The point of the country, literally the entire reason for the United States of America to exist, is to demonstrate that people can govern themselves. And the way we’ve chosen to do that is by giving everyone a vote. For an American, the opportunity to vote is sacred.

We were pleased when civil rights attorney Portia Allen-Kyle accepted our invitation to talk about voting by mail. She is the advocacy and policy counsel for the ACLU Campaign for Smart Justice, and is the founder of FuturaBold, a consultancy that specializes in the politics of policy.

Once again, both Junko and I participated in the interview.

JUNKO YOSHIDA: Do you have any stats as to how many ballots are actually rejected due to the nonmatching signatures?

PORTIA ALLEN-KYLE: It’s hard because it does vary from state to state, county to county. But there is some evidence out of Florida that in some counties the rejection rate could be under 1%, and in others it can be as high a 3%. And that’s based on studies by the ACLU in partnership with a professor at a state university down there.

It’s not just the rejection rate. It’s also, does everybody who returns a vote-by-mail ballot and signs it, do all voters have a likelihood of being rejected equally? And where there are disproportionate rates of rejection for people of color, for voters who are newer, for voters where English may not be their first language, that’s when we really begin to be concerned with the implementation of automated signature verification.

JUNKO YOSHIDA: I think I read somewhere, the single biggest reason for disqualification of mail ballots in 2016 was nonmatching signatures. Do you agree?

PORTIA ALLEN-KYLE: It depends on the state, but it usually is in the top five, often top two reasons for ballot rejections. If there is one that is more, it’s often ballots that arrive late in states that don’t accept them after election day. So usually there are a lot of those ballots. Then those were there is some type of error, whether it’s a signature or something else, as to why ballots get rejected.

JUNKO YOSHIDA: What specific concerns do you have in regard to the use of AI in signature verification software?

PORTIA ALLEN-KYLE: I’ll give you a little bit of background about myself. Just before I came to the voting space, I did a lot of work on pretrial justice and bail reform. And one of the hot issues there is the use of algorithms in determining risk of people in terms of whether they’re able to make bail, whether they’re able to be released and on what conditions, etc. The concerns there are really around algorithmic discrimination. Because of the bad data of the criminal justice system, which represents years upon years of tilted outcomes with regard to some groups. But all of that gets embedded in the algorithm.

Here in the voting space, it really isn’t that much different. Unfortunately this is an area where usage of algorithms to verify signatures is a little bit ahead of the law that regulates it. There aren’t standard regulations; it’s not anything that’s been captured by the regulations of the Federal Election Commission. States vary. It’s often left up the county to determine whether they’re going to use the software and then how they’re going to use it. So you see a lot of variation, and that I think those concerns in implementation have become even more acute because there are so many more plaintiffs that are using the software.

According to the company that is, as far as we know, the top purveyor of automated verification software, they started out in the banking industry, where the concern is a little bit different. You have a fundamental right to vote; you don’t necessarily have a fundamental right to your debit card not being rejected at the register. So the use of software there, it’s like, Yes, accuracy is helpful. It’s really annoying as a customer when your card gets rejected for no reason, and part of that perhaps is that they think there’s some type of fraudulent activity because your signature is different.

What that means in the voting space is that people’s fundamental right to vote is being blocked from exercise. So that is concerning because of the lack of regulation. The stakes are very high if things go wrong, and reliance on a private company that is not subject to transparency, not subject to public auditing, not subject to other regulations, not required to audit by demographics, to even monitor the demographics of the data that the algorithm is trained on. If we know that the stakes are so high, then it’s really concerning when there are so many protections that people have been pushing for in other advocacy use such as pretrial that just don’t exist in this area.

BRIAN SANTO: Do you make a distinction between the signature verification technology and the user? In my mind, I’m thinking of an analogy. A tool versus how you use it. You wouldn’t hand a chainsaw to somebody without telling them how to use it first. Or making sure that they’re not apt to just like start waving it around.

By the same token, do you separate the concerns between what may be inherent biases built into the tools and inherent biases in the people that are using them?

PORTIA ALLEN-KYLE: I think that those concerns perhaps both exist but are separate. On the one hand, we would want to make sure that we have an algorithm that is 1) transparent, where the data is available, where people can kind of check the math, like look under the hood a little bit and to check to make sure that no groups are most likely to have false negatives, or no groups are more likely to have their signature rejected for whatever reason it is. But also, any kind of robust valuation of the algorithm by its nature needs to understand how people interact with it.

We need to understand that, if a signature has been rejected… Let me back up a little bit. Some election administrators have adopted procedures where, if a signature is rejected, then a human looks to verify it. And what we know about signature matching in general is, it’s not a science. It’s not a science on top of the fact that we’re not sure how people are interacting, like if it gets rejected if people are like, Well, if this real fancy software says it doesn’t match, then I guess it doesn’t match.

We’re not sure if there’s any amount of training that can overcome that, just because we don’t do it. And no one is required to. So both end up being an issue in the scheme of things. It’s for that reason that inserting a human into the process doesn’t solve the potential downfalls of the algorithm.

BRIAN SANTO: Do you have a set of recommendations or safeguards that you would like to see first, as regards the algorithm? And then, as regards using a technological signature verification system?

PORTIA ALLEN-KYLE: Yes. Who knows what a fully robust system would look like in terms of regulation? But I think that we do know from other areas some kind of bare-minimum standards that we should have. The first one is really about transparency.

Insofar as election officials, public government officials, are using the software to facilitate voting, we should know what the algorithm is. We should be able to see what the data is. It should be publicly inspectable, verifiable, and able to be audited. Before anyone is able to adopt it, it should be required that such an algorithm is trained on data that includes demographics so that we can see the potential outcomes on different voters.

Not kind of run-of-the-mill demographics, but demographics that we know to be important in the voting context. So younger, first-time voters or student voters whose signatures may, by definition, change over time as they’re still perfecting their handwriting and getting more accustomed to writing it. Voters who are perhaps new citizens, where English isn’t their first language and perhaps not their written language. Even as technology in other places catches up, are there requirements for there to be a wet signature as opposed to, we’ve all used the pin pads where what you write, no matter your best effort, just doesn’t really look what it does on paper. So when you’re comparing what is on a digital pin pad to what is captured on your ballot, there may be some discrepancies there. There should be processes to update and keep maintained and capture different types of signatures for voters, if that is to be the mechanism to be used.

I kind of want to put a pin in that, in that there has been a trend towards signature matching as the lesser of an evil when compared to voter ID. And we know voter ID creates barriers to access, and as a compromise many places have tried to ensure election security through measures such as signature matching.

But in a world where we know that fraud is virtually nonexistent, signature matching still constructs barriers. I don’t want it to be a given, but we need to match signatures, whether we’re using software or doing it by hand in order for votes to be valid.

BRIAN SANTO: Is it a stark choice between having no safeguard mechanism or using a mixture of safeguards or adopting something else entirely?

PORTIA ALLEN-KYLE: I’m not sure, just because of the variation of voters across the country. Different states have different voting blocs. There won’t be a “one size fits all” solution. But I do think that we should be open to… especially for states where it is already very hard to obtain a vote-by-mail ballot. Where you can’t just sign up. Some states require witnesses; some states require Social Security numbers or drivers’ licenses or whatever it is. It is often the case that this is just another thing, and not the only thing. So we should be really cognizant of, on the whole, how each of these different safeguards that we’re implementing can combine to create real barriers.

For example, there was a woman in California who found out that here vote wasn’t accepted because the signature didn’t match. She was reached out to with regard to the ballot, and what she said was, I have arthritis, so I could not fill out this ballot. My son filled it out and signed my name. So it’s me, but that signature won’t match. So we’ll need to think about really in terms of accommodations for people where writing is… You shouldn’t need to have to write to cast your ballot. There should be ways in which we can do this and you can still assure that your vote has been counted.

We may not know those ways now, but we should be open to it as an electoral system. It’s one way to make it easier to access democracy. And if not, tech should be regulated.

BRIAN SANTO: Yeah! We talked to a company that does the signature verification. They said something interesting. The system improves when it has more data to work from. So in other words, it works well when you already have several signatures to check against.

There are places where you might have to sign up when you first register to vote. He was saying that you might have signed some documents when you got your drivers’ license. But here’s the interesting thing that I wanted to bring up: He said, Drivers’ license bureaus often have really poor technology. When they take a picture of your signature, it’s a low-resolution, not very clear signature. And it makes it difficult… He says that you can compensate for it, but the preference is not to have to compensate.

Listening to you here, it occurs to me that, if the preference is not to have to force a signature in order to allow a vote, but you have a technology that could help but it needs more data, that to me sets up a tension between having more data on record, more signatures on record, and maybe not having the record in the first place. Do you sense that that’s a tension? What do you do with that tension?

PORTIA ALLEN-KYLE: Part of that is one reason why it’s problematic. There is somewhat of a built-in bias against younger or newer voters or people that haven’t been…. In some places the number of black people who don’t, for example, have a government-issued ID is as high as 25%. If you need to have been in another system over a course of years for your signature to be accurate, or more accurate, there’s a built-in bias against people who are newer to the system against people who perhaps have been cut out of the system for a number of different reasons. That inherently is the tension.

And on top of the fact that this is a technology that is procured and implemented by elections officials who have zero control over how the DMV does their business. So in order for it to be accurate on one side, the other side needs the requisite technology in order to capture good quality signatures. You’re kind of setting yourself up for failure.

And without one way that, again, technology getting ahead of regulations, one way to do that is with regulations that outline the quality of signature that should be captured on the licenses. And match that to the quality of signature that should be run through the software.

Arguably, perhaps signatures that are that low quality… I am not sure, but I would be curious to ask some of the proprietors of the algorithms whether those should even be run through the software. Like how much does that contribute to fine-tuning and refining the algorithm to begin with?

JUNKO YOSHIDA: Talking about transparency, it’s easy to say transparency is needed. Every AI discussion ends there. As a lawyer, how would you like to see the transparency implemented or explained to you? I’m not talking about explainable AI. I’m talking about what test results or what validation are you looking for?

PORTIA ALLEN-KYLE: I would love to know the effectiveness of the algorithm, the accuracy by demographic, before it is implemented. Because we can compare that to the voting bloc that is in a particular jurisdiction. It could very much be the case that some algorithms work better on some jurisdictions that are closer to the training data than they do on others. But then knowing that information, making sure there aren’t too many false positives or negatives, comparable over demographics can really give some insightful information.

I would also want to know the composition of the training data. I think that can also give some indication if everyone in your training data is a native English writer, then that is something we probably want to know up front and be able to kind of think through and adjust for and really probably send you back to the baseform to get a bigger sample that includes those who don’t write English.

So starting with transparency by saying, What is the algorithm? Make the data public. Tell us, from your own tests, what your results are in terms of the efficacy here. That would be a good start from a regulatory perspective.

JUNKO YOSHIDA: Or build some kind of benchmark.

PORTIA ALLEN-KYLE: Yeah.

JUNKO YOSHIDA: What really struck me is that just because we have this automated software for signature verification doesn’t mean that it’s a given. It’s not a poll tax, but you’re actually putting up a barrier for people’s ballots to be counted. So you need to be mindful of what you’re doing with your software.

PORTIA ALLEN-KYLE: It’s just another area where there are already valuable civil rights lessons that we’ve learned from AI in other contexts. At this point it would be good to get ahead of it so that we don’t need to experience it. Sometimes people say, Show me the data where these outcome have occurred. In this situation, the more data we have that says signatures have been disproportionately rejected, the more that means that people have literally been disenfranchised and not able to exercise their fundamental right to vote.

So it’s a bit of an area where maybe we shouldn’t wait until we know for certain how discriminatory the algorithm can be and get ahead with some of the protections we know work better in other areas.

JUNKO YOSHIDA: All right. Very good. It was wonderful to talk to you, Portia. Thank you such much for coming to the show.

PORTIA ALLEN-KYLE: Yes! Thank you so much for having me! This has been interesting and fun!

BRIAN SANTO: That was Portia Allen-Kyle, a lawyer and public policy expert. We’d like to thank both Allen-Kyle and Greg Council from Parascript for coming on the podcast.

And that is your Weekly Briefing for the week ending November 13th. Thank you for listening.

The Weekly Briefing is available on all the major podcast platforms, but if you get to us via our web site at www.eetimes.com/podcasts you’ll find a transcript along with links to the stories we mentioned. If you just go to our home page at eetimes.com, click on the button that says “Radio.”

This podcast is Produced by AspenCore Studio. It was Engineered by Taylor Marvin and Greg McRae at Coupe Studios. The Segment Producer was Kaitie Huss.

I’m Brian Santo. See you next week.

 

HIDDEN TRACKS:

BRIAN SANTO: Just a question about the nature of signatures themselves. And I’m going to pick an example that is completely and utterly at random. Donald Trump, for example, signs his name, has a really famous signature. And to me it looks like there are four extra Ms in his last name. What would your system make of somebody who, say, exaggerates their signature?

ALL: (LAUGHTER)

BRIAN SANTO: Come on, Greg. You’re not going to dodge this one, are you?

GREG COUNCIL: No. Hey, our software is completely unbiased.

JUNKO YOSHIDA: Apolitical.

GREG COUNCIL: In fact, for any machine learning you want to remove the bias out of there. So it’s not going to do anything with those additional Ms… provided that they have good, authentic signatures. But it’s going to look at other types of strokes and features and things like that. It’s not going to fixate over a couple of extra Ms or anything like that.

On our handwriting recognition side, oh, yeah, we would puke all over that.

ALL: (LAUGHTER)