Comments on: Data Privacy: The Story of a Paradigm Shift

By: Adversarial Thinking Considered Harmful (Sometimes) « 33 Bits of Entropy

Adversarial Thinking Considered Harmful (Sometimes) « 33 Bits of Entropy — Mon, 08 Nov 2010 16:53:20 +0000

[...] the right paradigm in the privacy context. The theoretical study of database privacy seems to be doing rather well by borrowing methods from cryptography, and I’ve argued in support of adversarial thinking [...]

By: Arvind

Arvind — Thu, 04 Mar 2010 20:50:58 +0000

I think reality has consistently shown that people tend to underestimate privacy risks. It’s hard to measure the harm to individuals, because it is the aggregate of a million tiny harms, so let’s look only at examples where the risk to the data curator did indeed materialize.

1. AOL had to fire top executives after the search data release
2. Netflix got sued and got bad press
3. Decode Genetics had its grand plans in Iceland (i.e., the Health Database Act of 1998) shut down by the Supreme Court, partly because of the re-identification risk. The company later went bankrupt.
4. After the Homer et al. paper, a whole bunch of genetics research databases went behind closed doors and researchers now have to jump through hoops to get them and everyone complains about it.

I could go on. If you look outside re-identification, it gets even worse. Google got a huge amount of bad press due to privacy problems in Buzz even though they fixed it in 4 days.

I’m not arguing for the worst-case model as beneficial the good of humanity. I’m arguing that it is beneficial purely based on self-interest.

By: cowherd

cowherd — Sun, 28 Feb 2010 19:41:32 +0000

“The most common objections to differential privacy come from people who don’t agree that adversaries should be modeled in a worst-case manner.”

People in the “real” world don’t care about security; they care about risk. Risk accounts for the likelihood of a threat (among other factors). Unless the exposure or consequence are much higher than the inverse of the likelihood, in practice the worst case model will continue to be ignored as more cost than it is worth.

By: noamnisan

noamnisan — Fri, 26 Feb 2010 05:32:10 +0000

I like your viewpoint of looking at Differential Privacy as a paradigm. Its effect is noticed even without using any of the field’s technical results, just the general point of view: add some noise and prove that sufficiently little information is leaked and that a sufficiently good answer still emerges. It is the latter they may be first to be picked up by practitioners, it seems.

By: Suresh

Suresh — Fri, 26 Feb 2010 04:28:44 +0000

Too true. people should pay more attention to the history of security breaches. And as more and more data gets into the cloud, this will only get worse and worse. Arguably the event that kicked off the original surge of interest in privacy was the demonstration by Sweeney that the MA governor’s info could be hacked ;)

By: Arvind

Arvind — Fri, 26 Feb 2010 04:25:05 +0000

Yes, absolutely. I was halfway through writing a paragraph on “adversarial thinking” and how differential privacy embodies it because of the crypto background. But then I deleted it because the post was getting too long :) I decided to kinda hint at it instead by talking about upper bounds.

The other reason I didn’t write about it is that people have been asking me for examples of real-life adversaries, and I owe them a whole post, so I thought this would fit in better with that post.

The most common objections to differential privacy come from people who don’t agree that adversaries should be modeled in a worst-case manner. I think we’re gonna need some more high-profile inferential privacy breaches to change their minds :)

By: Suresh

Suresh — Fri, 26 Feb 2010 04:15:43 +0000

I like your analysis of lower bounds vs upper bounds. But there’s plausibly another way of explaining why differential privacy “wins”: it’s because in an adversarial attack model, you have to have clearly modelled adversaries, and differential privacy does that, by using the crypto trick of merely limiting the power of the adversary. Whereas all the anonymization work was a lot fuzzier about the kinds of adversarial models.

Maybe that merely amounts to what you were saying about upper bounds on privacy ?