Big Brother’s Big Data Will Crush You, Puny Human

Last week I told of how deeply personal information about us all dives into the Black Rooms of the US, UK and other Governments like stellar matter spiraling inexoriably into an unquenchable black hole. You cannot hide unless you are truly isolated from Western society, and, if you are reading this, you are not.

So all those satanic governments harvest more and more data about us, but, like harvested crops, data is not much use until it’s cleaned up a bit, prepared and cooked into something digestable. How?

You will have heard of, and perhaps even used, a specialised computer data store called a “database”. Databases are everywhere, nearly every major website has at least one. Every major organisation has loads of them. And lots of databases contain your name. And much more.

A typical database organises data into simple compartments, called tables, with a grid of rows and columns. Then data in those tables is combined to make more useful information. For example, a table might contain a row with your name and address. Another table might contain a row with your name and Facebook “posts” and Twitter “tweets”. So, for example, a government spook can cross reference two such tables to find the address of the person that tweeted about how he hates the government.
In geek speak (which you don’t need to worry about) the government spook will get his answer by making a database “query” that “joins” two “tables” in a “relational database”.

The geeky query computer code (which you don’t need to worry about) might look a little bit like this. Select ADDRESS from Table PERSONAL_DETAILS then Select POSTS and TWEETS from Table SOCIAL_NETWORK then join on NAME.

The world is awash with databases. So useful information can be created by combinging all the tables of all the databases in the world and then querying that data. But there are several problems with that approach.
* It’s difficult to merge different databases because they organise data differently from one another.
* It’s even more difficult to keep that merged data up to date. The computer code becomes too complex.
* This only works when there is not too much data, especially if data is added and updated frequently.

That’s not to say that massive relational databases are not being built that contain loads of stuff about us, they are. But even such nigh omniscient databases are not able to satisfy government spooks, especially for near real-time spying. Such a databases will know whether you’re straight or gay (even if you think it a secret) and it will know your interests and who your friends are, what you do, how much you earn, your credit rating, and hundreds of other details. But it might not be updated fast enough to know where you want last night or about your next holiday plans you just booked.

As mentioned, too much data overwhelms a relational database. When you have MP3 music file sized data (a few megabytes), then (well designed) relational databases work a treat. When you have data the size of a DVD movie (about a “gigabyte”) then relational databases still work great. However when the data size climbs into the equivalent of thousands of DVD movies – we enter the “terabyte” realm, and relational databases can struggle to manage such enormous data volumes. Several terabytes of data is equivalent to storing a million King James Bibles.

Why do relational databases struggle under terabyte loads? I mentioned that, to get useful information, usually tables need to be “joined”. Joining data can take up a lot of computer power and extra data storage. Adding, updating and deleting data to keep it up to date is likewise costly in terms of computing power. This can cause computers to slow down to a crawl or crash. If the data grows faster than you can save it, then you’re busted.

To get around this problem, “Big Data” technology has been invented. The idea is this: Instead of organising data neatly into tables in a database, you take data in relatively chaotic formats, the data might be in normal computer files, and you chuck it onto hard disk storage. Computers can save such raw data very quickly. Thousands of terabytes (petabytes) of data can easily be stored in modern networked computer systems. Then, in geek-speak, what you do is run computer programs called called “Map Reduce” algorithms to sift out the information needles in the data haystack. “Map Reduce” can, for example, grab a subset of the files (the “Map” bit) and then strip out the crap in those files that you don’t want from that data (the ‘Reduce” bit).

Once you have performed that Map Reduce step, then quite often the resultant smaller set of data can then be ploughed into a normal relational database to be analysed. Relational databases are great for analysing data and extracting useful information, as described earlier.

So that’s how governments (and other organisations) can turn immense volumes of data that is mostly of sewerage quality and extract golden nuggets of information, information about you and yours that you might prefer remained private. Worse, if such golden nuggets of information are incorrect, then you may find that the government thinks you’re a terrorist, when you are not, to take a topical example. Can such information be incorrect? Of course it can! The input data may be incorrectly typed, or the input data might be correct but incorrect conclusions drawn from it. This might be by human error, or by computer error (programming logic failure) or by malicious intent. Someone might want to smear you, for example. People have a depressing predilection to conclude that the “accused” must be guilty.

Next week I’ll show how corporations are in bed with governments to analyse you using technology. Even “encrypting” data is no guarantee of evading the spooks and the snoops, I’ll briefly explain what encryption is and why governments can often circuit around it. The Edward Snowden revelations have shown that we geeks have been right all along, it wasn’t paranoia, there really is a conspiracy. There’s a war going on, the Privacy War. Truth is the first casualty and and our freedoms are following it into Hell.

16 comments on “Big Brother’s Big Data Will Crush You, Puny Human

  1. Simon Roberts
    August 30, 2013 at 2:05 pm #

    Looking forward to the next installment. I assume that you’ll be covering VPNs. Can you say a bit about logging and anonymity and even recommend a provider? Ta.

    • James Eadon
      August 30, 2013 at 4:28 pm #

      @Simon – there was an interesting story recently about a geek who suspects the Americans are snooping on his VPN. I’ll mention that.

  2. dr
    August 30, 2013 at 2:22 pm #

    I find this situation with governments spying on their citizens interesting, in the context of the left gaining power (their long march through the institutions), and the predictions of Austrian economics regarding fiat currencies.
    To summarize briefly, Austrian economics predicts that fiat currencies will eventually collapse. They disintegrate in a puff of hyper-inflation.
    We also know that the left are gaining influence due to their control of the MSM and quangos and their insertion of radical activists into government departments. We know that the left favour big government and so support this collection of data and its usage in running the country. We also know that they have been using their influence as a way of moving the political spectrum gradually to the left over the past few decades.
    It seems to me then, at some point Britain will face hyper-inflation due to our economic policies. At the same time, it looks to me like whichever government tries to deal with that inflation will be further to the left than today, it will also have more power over the people than today’s government and more information on individuals due to the increased spying and monitoring that will be possible, the more that we live our lives online.
    So when our currency starts to collapse, the left including the MSM will claim that we are witnessing the fall of capitalism. This to my mind leaves Britain in a dilemma, and this is what I find interesting.
    Either, the people will bring down / vote out the government, and elect a party that can restore liberty to Britain and therefore begin the process of reconstruction after our nation’s bankruptcy.
    Or, the government will be able to use its powers of information gathering to search for opponents in the general population, that is, people who are attempting to destabilize the political process in order to punish the politicians for bankrupting the country. In this scenario, the currency finally collapses, and with it debt is wiped out, the government survives and then, since it has blamed the collapse of the currency on capitalism, needs to create a new currency for Britain together with a new political reality. This could be a form of socialism or communism or dictatorship or an modern environmental political ideology, but it seems reasonable to me that the government could use the crisis of the monetary collapse as an excuse to suspend(maybe permanently) democracy.
    I am aware that many of the right believe that the first of these two scenarios will play out, that is that no government will be able to survive the currency collapse, but I think that this ability to spy on citizens and monitor opposition groups will give the government a better chance of surviving than ever before. It follows then, that the longer we have before hyper-inflation, the more likely it is that the government will have expanded its information gathering to the levels necessary to enable its survival. This would lead to a post hyperinflation nightmare of grinding poverty, next to non-existent liberty and huge government, with most in society enormously dependent on the government machine to survive from day to day.

    • Rocco
      August 30, 2013 at 4:14 pm #

      What a cheery picture you paint! 😀
      But, dr, a flaw in your dystopian scenario (or, a ray of light perhaps) might be this: the origin of government is opinion, and governments exist because people think they’re necessary. What would happen if everyone in Britain decided to ignore the State? Not smash the State, simply ignore it? It would collapse immediately, because it’s power over us is dependent on our belief in it.
      Now, in the event of a currency collapse, the immense size of the State might well prove to be it’s downfall. What doesn’t the State do today, after all? But it couldn’t do anything without money, that is, transferring money from one place to another. And if all those people who rely on the State for one thing or another (not just those who claim benefits, mind), are let down by the State; if they feel that the State is impotent – well, maybe the State would become impotent?
      The bully might be too big, too strong for our punching him on the nose to do any good. In fact, it would certainly make matters much worse. But we don’t have to attempt to fight him, we can send him to Coventry, instead.

      • James Eadon
        August 30, 2013 at 4:50 pm #

        @Rocco – @dr (by the way, for readers, I think MSM means Mainstream Medida) makes two points, firstly that the economic system must collapse and second that the government may use that as an opportunity to become totalitarian.
        In my view, the more information the government has, the more likely it will be to become totalitarian anyway, no matter what the economic situation. It’s an inherently unhealthy shift of power from the people to government.
        A weird situation is that left wing voters (not Labour, mind you) seem to be more uneasy about this spying situation than the right wing. That’s what maddens me. The right wing seem to view spying as somehow a patriotic way we defend ourselves from external threats. That’s bullshit, because it underestimates the threat from within, totalitarian government.
        The left wing are utterly clueless when it comes to economics, taxation and big government, but the right seem relatively unconcerned about the loss of our privacy and freedoms.
        In the States the US government is doing two things simultaneously, it is spying more and it is pushing to disarm the population. Whatever one may think about guns, one can see the direction that they are headed in.
        Notice how pervasively the US “Democrats” use Big Data systems to identify electoral voters and manipulate them. The press seems to think that this is impressive. The Republicans are playing catch-up in this regard, but the real question is, Is this situation healthy? The same rush to spy on us and manipulate us happening worldwide by politicians and by corporations and other orgs.
        We’re in a crisis situation.

    • Rocco
      August 30, 2013 at 4:41 pm #

      On the other hand…
      dr, I don’t believe in this “march through the institutions” conspiracy theory approach to the modern world. I believe that it doesn’t give men enough credit. That is, I don’t believe people need to be tricked into wanting massive government – they want it because they are convinced of it’s goodness.
      So, your ultra-dictatorship may well be the more probable outcome. Although, not quite for the exact reasons you set out above. That is, it will be an ultra-dictatorship based on common consent.

  3. Rocco
    August 30, 2013 at 8:03 pm #

    Mr Eadon, just to clarify, I didn’t assert that the economic system would collapse. I was merely speculating in the possibilities for free market anarchism in the event of a collapse.
    To pick up on your point about growing totalitarian tendencies running parallel with growing technological power, I would agree. But I wouldn’t say this was remarkable, per se.
    That is, all government tends towards totalitarianism, it is simply it’s nature. Totalitarianism is not qualitatively different from small or big government. It is just am extension of them.
    Limited government is an impossibility. One group of people have the monopoly on violence, and the monopoly on deciding whether they’re using that violence justly. What’s more, they have the monopoly power on deciding whether they should become more powerful. Add to this, that the only way for specific individuals within this group to keep their gilded position is to promise to “do more”. Well, it is nonsense to believe such a group would remain limited.

  4. Liz Z
    August 30, 2013 at 8:25 pm #

    Interesting article – definitely food for thought. Looking forward to the next one.

  5. therealguyfaux
    August 31, 2013 at 2:17 am #

    At the point at which the snoopers realise they are spending way too much time on the computer to find out everything they would like to know about you, and not enough time actually acting upon the information, they will, as a practical matter, curtail their surveillance, if all indications are you are not about to act. This, if they operate on other than a terminate-with-extreme-prejudice basis where just proposing something will get you a bullet in the head at close range.

    If they believe you are all bluster, there is no reason for them to go after you and take the chance that some larger fish is getting away, unless they want to publicise your case pour decourager les autres. The question then becomes how important is it for them to be seen doing what they do in terms of punishing Internet sedition and subversion and being cheered, versus going too far and being feared. You can push people to the point that, in order that they do nothing wrong, they do nothing full stop.

    People talk about “going Galt,” i.e., render the least to Caesar conceivably possible by becoming “lazy” and “indolent” and working not anywhere near capacity– a societal work-to-rule slowdown, as it were. What if, to the extent possible, people decided to “go Ludd,” i.e., stay away from the Internet and texting, and paid cash on the nail wherever possible? And took mass transit so the comings and goings of their automobiles could not be tracked and whoever wanted to know where they’d been was reduced to looking at CCTV surveillance aboard vehicles– a time-consuming job and an incomplete one at best. What will they do– make what I’ve said illegal? How do they enforce it?

    My God, how did we ever manage back in the 1950’s and 60’s?

    • James Eadon
      August 31, 2013 at 2:24 pm #

      @therealguyfaux – the trend is towards ever more Orwellian surveillance. Governments can’t get enough of it, it is like crack to them. There is a rationale behind it, which is that knowledge is power, so the surveillance will become increasingly intrusive without end. The only way to curtail all this snooping (at the expense of the tax payer) is to pass laws to outlaw it and encourage the police to prosecute members of the secret services that violate the letter of the law.
      I’d rather the police protect our privacy than waste tax payers hard earned on sadistically bullying 1970’s celebs in the bizarre Yewtree prosecutions.

      • Simon Roberts
        August 31, 2013 at 3:26 pm #

        I’m not sure that the police would be the right people for that role – look at how they behave when they find unacceptable (to them) messages on Twitter.

        No-one gives up power voluntarily, especially government agencies. It’s an unfortunate aspect of human nature.

        There’s no good outcome as long as these agencies exist. Thomas Jefferson knew the score – his wisdom from over two hundred years ago applies perfectly to the situation that we now find ourselves in.

        He knew that advances in technology are unimportant in themselves, it is the power they bestow on governments that’s the problem. That’s why the US Constitution mentions “arms” and isn’t specifc about muskets, cannons etc – the authors knew that human nature was the fundamental problem.

      • Rocco
        August 31, 2013 at 3:44 pm #

        James, as I said above, limited government is impossible.
        The only way to curtail all this is to pass laws against it, you say. But the fellows who you want to pass those laws, are the very same fellows who want to snoop, who benefit from all the snooping. Moreover, the people who’d enforce those laws not only love snooping, but they’re are employed by the would-be snoopers anyway.
        No, man. The State is the problem.

  6. James Eadon
    September 1, 2013 at 10:46 am #

    @rocco – the cops would like to do most illegal things but they tend to also enjoy arresting people for doing the same. They sometimes go after MPs. That’s not an issue. The greater issue is to persuade politicians to pass laws that remove snooper’s rights, a kind of anti-snoopers charter. That is only possible if politicians think it is a vote winner.

    • Rocco
      September 1, 2013 at 1:40 pm #

      But James, man, politicians won’t pass any such laws. The State is not going to limit it’s power like that. Even if they say it in public, maybe with a few high profile show trials, it”ll still happen.
      And anyway, the real point is most people like big government. In fact they love it. ‘The State keeps us safe, keeps us healthy, keeps us warm and fed. The State educates us, protects is from unscrupulous business men, and bearded foreigners on the internet. Without the State none of this would get done.’ This is how most people think. It’s how people who believe in “small government” think, for goodness sake!
      This is the root cause, this is the problem those who believe in liberty should be addressing. Without a massive change in public opinion about the State the chances of even superficial policy reversals on this front are nil.

  7. James Eadon
    September 1, 2013 at 2:38 pm #

    “But James, man, politicians won’t pass any such laws”

    I didn’t say that they will, I said that it’s the only way we can prevent disaster is to persuade them to do so. Ultimately this may be a forlorn hope.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: