There are many good bits (where by “good bits” I mean “bits that make one despair”) but this section caught my eye:

…before full competition was established [in the electricity industry], he, the regulator [Stephen Littlechild], imagined he would act as a surrogate, a kind of State Competitor General, enforcing occasional price cuts to keep the private companies on their toes. In the end, he thought, the need for regulation would largely wither away. What Littlechild, an academic with no business experience, didn’t fully take on board was that the reason private companies compete with each other isn’t that they like competition. They hate it, and will only compete if forced to do so. Rather than competing with a rival on price or product or revenue, they’ll try to eliminate the rival firm and take over its territory by buying it; or reach an unwritten agreement on an oligopolistic cartel of a few big firms, carving up the market between them.

I don’t know why, among all the arguments against privatising certain industries, this hadn’t hit me before: “The reason private companies compete with each other isn’t that they like competition. They hate it.” Of course, yes.

I can’t imagine what it would take. Even with Occupy last year, and the continuing trickle of occasional outrage about bankers’ bonuses, and ministers calling people “plebs”, and all that… on we go. I can’t imagine — from my comfortable, middle class, metropolitan point of view — what would make a mass of people rise up in any meaningful way.

Which is just some background rambling to frame this bit from a review by Ian Jack (subscribers only) of Katherine Boo’s book Behind the Beautiful Forevers: Life, Death, and Hope in a Mumbai Undercity (Amazon UK, US) in the New York Review of 5 April 2012:

What [Boo] discovered was that, though the poor might complain about the greed and self-interest of the rich, they complained about their neighbours much more. Poor individuals blamed other poor individuals for their predicament rather than expressing solidarity and taking their protests to the streets. As group identities based on caste, religion, and language began to wither, “anger and hope was being privatised, like so much else in Mumbai.” And not just in Mumbai, but also in Nairobi, Santiago, Washington, and New York.

In the age of global capitalism, hopes and grievances were narrowly conceived, which blunted a sense of common predicament. Poor people didn’t unite; they competed ferociously amongst themselves for gains as slender as they were provisional. And this under-city strife created only the faintest ripple in the fabric of the society at large. The gates of the rich, occasionally rattled, remained unbreached. The politicians held forth on the middle class. The poor took down one another, and the world’s great, unequal cities soldiered on in relative peace.

Which sums it up.

I guess this is a similar thing to how many people in, say, the US often vote for the Republicans against their own financial self interest — they simply hope to gain a slight advantage over those around them. (I’m simplifying of course; I know there are other reasons people vote a particular way.)

When I read that a couple of months back (I’m catching up) I noted that I should also link to some related article I’d read on the New Statesman’s site. No idea what that was now.

I was particularly taken with his word “midcult”:

The title essay, published in 1960, takes gleeful pleasure in deflating The Old Man and the Sea and other earnest, high-sounding books and plays that served an audience who aspired to be intellectual and avant-garde without being either. Unlike mass-produced popular culture, “midcult” — Macdonald’s Soviet-sounding shorthand for middlebrow culture — “pretends to respect the standards of High Culture while in fact it waters them down and vulgarises them.”

…

Macdonald was writing less about bad art and greedy publishers than about what it means to let oneself disappear into a passive and anonymous public. Reviewers complained that Macdonald didn’t like the masses. What he didn’t like was anyone’s willingness to be submerged into a mass and the culture that profited each time someone sank into it.

A quick Google turned up this 2011 article about Macdonald by Louis Menand at the New Yorker, with plenty more on the theme:

[In the essay ‘Avant-Garde and Kitsch’, by Clement Greenberg, edited by Macdonald] Greenberg explained that both avant-garde art and kitsch (that is, popular, or commercial, culture) were by-products of the industrial revolution. Art became avant-garde when serious artists turned inward, away from a society they felt alienated from. In the case of painting, artists moved from representation to abstraction, from attention to the world to attention to the pain. Kitsch — the word means trash, or, as Greenberg put it in his letter to Macdonald, crap — was a consequence of the fact that the industrial revolution had made universal literacy possible, and the new technology of mechanical reproduction permitted an ersatz culture to be manufacture cheaply for an audience looking for entertainment and diversion. This manufactured culture killed off folk art, which was a genuine popular culture.

And then, later, continuing with this:

“Folk Art was the people’s own institution, their private little garden,” Macdonald argued in the new essay. “But Mass Culture breaks down the wall, integrating the masses into a debased form of High Culture and thus becoming an instrument of political domination.” The most insidious development in this process was what he called l’avant-garde pompier, phony avant-gardism. As he put it, “There is nothing more vulgar than sophisticated kitsch.”

Macdonald eventually categorized this pseudo avant-gardism as the culture of middlebrow aspiration — Midcult. Real kitsch, he decided, could be left to the masses. The true enemy was bourgeois high-mindedness in literature, music, theatre, art, and criticism, and, over the next ten years, he turned much of his critical might to the job of identifying this culture, exposing its calculated banalities, and, often with genuine success, persuading readers of its meretriciousness.

He demolished several books in reviews, such as Colin Wilson’s The Outsider and James Gould Cozzens’s By Love Possessed:

Macdonald did not consider the books he was reviewing to be isolated cases of bad writing or false premises. He regarded them as representative of a formidable and dangerous historical development: the rise of a shallow and pretentious literary and intellectual culture for people who were educated but essentially uncultivated, living, as he put it, beyond their “cultural means.”

The article goes on to the point at which we arrived, the essay ‘Masscult and Midcult’, “a kind of summa of the New York highbrow’s contempt for bourgeois culture”, and a discussion of the torturous logic involved in decided whether the New Yorker itself, for which Macdonald wrote, was middlebrow or not.

In focusing on this one point I’ve made Dwight Macdonald sound rather one note and entirely snobbish, like a mid-century New York Brian Sewell when in fact he sounds more varied and contradictory, just as concerned with morality as with aesthetics.

It took me the best part of a day to piece together the parts of this so I thought they should all be in one place for the next person. I can’t believe I found this so hard. I may have made mistakes, but it seems to work.

I’m going to assume you’ve already got your site set up to store static files on S3 (which is particularly useful if, say, your site is hosted on Heroku). You should already have:

• Installed django-storages.
• Installed boto.
• Set up an AWS account and created a bucket in S3.

Your media and static folders

This isn’t essential for the whole private-files-on-S3 gist of this post, but getting your media and static files to end up on S3 in separate folders is a little non-obvious, but very useful. So, a little aside to cover it… I’ve got this working nicely by using something like this Stackoverflow answer:

In your app create an s3utils.py file and put this in there:

from storages.backends.s3boto import S3BotoStorage

StaticS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaS3BotoStorage = lambda: S3BotoStorage(location='media')

And then, in your settings.py, you’ll need something like this:

DEFAULT_FILE_STORAGE = 'yourproject.yourapp.s3utils.MediaS3BotoStorage' 
STATICFILES_STORAGE = 'yourproject.yourapp.s3utils.StaticS3BotoStorage'

AWS_ACCESS_KEY_ID = 'YOURACCESSKEY'
AWS_SECRET_ACCESS_KEY = 'YOURSECRETACCESSKEY'
AWS_STORAGE_BUCKET_NAME = 'your-bucket-name'

S3_URL = 'http://%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATIC_DIRECTORY = '/static/'
MEDIA_DIRECTORY = '/media/'
STATIC_URL = S3_URL + STATIC_DIRECTORY
MEDIA_URL = S3_URL + MEDIA_DIRECTORY

Those STATIC_DIRECTORY and MEDIA_DIRECTORY settings aren’t standard Django settings, but we need the MEDIA_DIRECTORY value when setting permissions on our private files a little later.

I think you’ll need to manually create the /media/ and /static/ directories in your S3 bucket. Then, if you run the collectstatic Django management command your static files should end up in http://your-bucket-name.s3.amazonaws.com/static/. And any files uploaded through FileField or ImageField attributes on your models should end up in http://your-bucket-name.s3.amazonaws.com/media/ . If those model attributes specify upload_to paths, they will be relative to /media/.

Making files private

By default, those media files are public — if you enter an uploaded file’s URL in your browser, you should be able to access it just fine.

Let’s assume that our model has two kinds of file, one public and one private. So in our models.py we have this:

from django.db import models

class MyModel(models.Model):
    ...
    public_file = models.FileField(blank=True, null=True, upload_to='open/')
    private_file = models.FileField(blank=True, null=True, upload_to='seekrit/')
    ...

Assuming you create the /media/open/ and /media/seekrit/ directories then the files should get uploaded there fine, but all currently publicly-accessible.

For our private files we need to set their permissions after upload to be private. To do this, I’ve ended up with a custom save() method on the model:

from django.conf import settings
from django.db import models

import boto

class MyModel(models.Model):
    ...
    public_file = models.FileField(blank=True, null=True, upload_to='open/')
    private_file = models.FileField(blank=True, null=True, upload_to='seekrit/')
    ...

    def save(self, *args, **kwargs):
        super(MyModel, self).save(*args, **kwargs)
        if self.private_file:
            conn = boto.s3.connection.S3Connection(
                                settings.AWS_ACCESS_KEY_ID,
                                settings.AWS_SECRET_ACCESS_KEY)
            # If the bucket already exists, this finds that, rather than creating.
            bucket = conn.create_bucket(settings.AWS_STORAGE_BUCKET_NAME)
            k = boto.s3.key.Key(bucket)
            k.key = settings.MEDIA_DIRECTORY + self.private_file
            k.set_acl('private')

If you upload a private file with that in place, then you should no longer be able to access it directly. e.g., upload a file called test_file.pdf and visiting http://your-bucket-name.s3.amazonaws.com/media/seekrit/test_file.pdf should get you an XML file containing an AccessDenied error.

Allowing access to certain users

Now that we can upload private files, how do we allow certain users to access them? We need to create temporary signed URLs that let users access the file.

First, in your urls.py add a URL for linking to the files:

from django.conf.urls.defaults import *

from yourproject.yourapp import views

urlpatterns = patterns('',
    ...
    url(r'^(?P<pk>[\d]+)/secretfile/$', views.SecretFileView.as_view(), name='secret_file'),
    ...
)

This will let us link to files something like http://yourdomain.com/42/secretfile/, referring to the secret file on an object with a pk of 42. This is what you should use when linking to the file, e.g., in a template:

<a href="{% url secret_file pk=object.pk %}">Download</a>

Then, in your app’s views.py create the SecretFileView:

from django import http
from django.shortcuts import get_object_or_404
from django.views.generic import RedirectView

from boto.s3.connection import S3Connection

from yourproject.yourapp.models import MyModel

logger = getLogger('django.request')

class SecretFileView(RedirectView):
    permanent = False

    get_redirect_url(self, **kwargs):
        s3 = S3Connection(settings.AWS_ACCESS_KEY_ID,
                            settings.AWS_SECRET_ACCESS_KEY,
                            is_secure=True)
        # Create a URL valid for 60 seconds.
        return s3.generate_url(60, 'GET',
                            bucket=settings.AWS_STORAGE_BUCKET_NAME,
                            key=kwargs['filepath'],
                            force_http=True)

    def get(self, request, *args, **kwargs):
        m = get_object_or_404(MyModel, pk=kwargs['pk'])
        u = request.user

        if u.is_authenticated() and (u.get_profile().is_very_special() or u.is_staff):
            if m.private_file:
                filepath = settings.MEDIA_DIRECTORY + m.private_file
                url = self.get_redirect_url(filepath=filepath)
                # The below is taken straight from RedirectView.
                if url:
                    if self.permanent:
                        return http.HttpResponsePermanentRedirect(url)
                    else:
                        return http.HttpResponseRedirect(url)
                else:
                    logger.warning('Gone: %s', self.request.path,
                                extra={
                                    'status_code': 410,
                                    'request': self.request
                                })
                    return http.HttpResponseGone()
            else:
                raise http.Http404
        else:
            raise http.Http404

What does this all do? First we make sure we get the object from the pk in the URL. Then we want to make sure this user can access the file. The conditions are up to you. Here we’re making sure the user is logged in, and either satisfies some condition set in an is_very_special() method on the user’s UserProfile, or is a staff member.

If that’s OK, and our object actually has a private_file uploaded, then we set the full filepath — this is why we had to create that MEDIA_DIRECTORY setting earlier on, because it needs to be an absolute path, not relative to /media/.

We then generate the signed, temporary URL. Here the URL will be valid for 60 seconds — after that it will no longer function, so it can’t be passed on to anyone else. Once we’ve got this URL, we redirect to it, and the user should be able to access the file — it should appear in the browser, or start downloading, depending on the file type.

Except, that’s not quite all…

Setting the Bucket Policy

Just because we’ve got a signed URL, this doesn’t yet mean the file will download. The private permissions we set on it earlier still apply. We need to specify a policy that will let us bypass this with the signed URLs.

Go to your S3 console, select your Bucket, and right-click to show its Properties. You should see a link saying “Add bucket policy” (or “Edit bucket policy” if you already have one). A window should open, into which you should put something like this:

{
    "Version": "2008-10-17",
    "Id": "My Special Bucket Policy",
    "Statement": [
        {
            "Sid": "Allow Signed Downloads for Private Files",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::12345678901:root"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/media/seekrit/*"
        }
    ]
}

A few points on this…

• No, don’t change the Version date from 2008-10-17.
• I don’t know what the significance of the Id or Sid are.
• In the Principal value, the 12345678901 shown here should be replaced with your AWS Account Number. This should be visible on the AWS Manage Your Account page, currently shown at the top-right. Remove the hyphens and put it in here.
• In Resource put your actual Bucket Name in place of your-bucket-name and set the path to point to the folder your private files are in. The asterisk on the end means this policy applies to all the files in that folder.

This seems to work for me, although I can’t claim to understand it in great depth.

That’s it

And there we go. I found most of that via Googling, but none of it was all in one place and it took way too long to piece together. Hopefully it’ll be useful to others.

Bear in mind that I DON’T REALLY KNOW WHAT I’M DOING and may have got things wrong. If you spot anything that could be improved, please do let me know (email or Twitter).

I didn’t want to read or draw or just play with toys. They weren’t doing. There was a particular itch that needed scratching, and only certain activities could manage that.

Building with Lego did the job. But once I’d built something — usually, unimaginatively, whatever the instruction book told me — I didn’t want to do much with it. Playing with the train, or the town, or the spacemen wasn’t doing.

Similarly with toy soldiers. I could happily set up scores of 35mm soldiers, or hundreds of OO/HO scale soldiers, poised to fight across my bedroom floor, but then… the battle was usually swiftly over because playing at war wasn’t doing.

Constructing, building, configuring hundreds of small parts: those were all doing. I’d have something greater than the sum of those parts to show for the doing.

And I think this is also the itch that programming satisfies. Decades on, if I’m a bit restless, and not in the mood for reading or watching TV or anything so passive, then fiddling around with code feels like doing.

Slowly constructing a thing, trying stuff out, making it work, ending up with a thing that’s so much better than its parts. When my head’s deep in both code and Lego, the time flies by and I’m doing something.

Take books for example. Amazon appears to do a decent job of aggregating different versions of a book into a single page; here’s the Amazon.com page for The City & The City by China Mieville. That’s good — it lists the Kindle, Hardcover, Paperback and Audio versions on one page. It gets more complicated with different publishers’ editions and Amazon Marketplace sellers throwing up slightly different listings. But it’s a good start, maybe it’ll get better, and this makes it easy to link my readers to one page for a book.

Assuming all my readers are in the US that is.

If I also want to link to that book for UK readers, I have to link to a similar page on Amazon.co.uk. For Canadian readers I’d link here. There are also other countries whose inhabitants buy things on Amazon.

This seems clumsy. It results in either a bunch of links to different Amazons or, more usually, a single link to the authors’ local site, leaving foreign readers to manually find their version. Or not bother.

This isn’t good for website authors — making a lot of links is tedious. It’s not good for readers — they’re either confronted with a list of links or with one link that may be no use to them. And it can’t be ideal for Amazon — many readers don’t have a simple, friction-free route toward buying something.

It would be great if I could link to a single page that showed the user a choice of Amazon sites on which to buy the product, maybe with their local one most prominent.

It’s been a long time since permalinks became de rigeur, a necessary part of The Age of Point-at-Things. One page for every thing. As the years go by, it seems increasingly odd to have Amazon’s products exist in separate region-specific silos. There’s no way for me to point at one item on Amazon. Unless I use each of my fingers to point towards a different country.

When I mentioned this on Twitter a while back, a couple of people suggested pointing at The Open Library, where I can point at this page for The City & The City. It’s a great idea but it’s little use for this purpose at the moment. The links are all US-specific and it lacks all the extra information — blurb, reviews, etc — that makes Amazon so useful.

I don’t want to dismiss how difficult this would be for Amazon to achieve. There would be many, many awkward issues to tackle, both technically and organisationally. But I want it to be easy to say to everyone, no matter where they are, “BUY THIS!”, and send them to one place. That’s the kind of thing Amazon is usually very good at.

It might just be coincidence but I’m sure I hear this more and more. Maybe it’s post-millennial disappointment, something Leila Johnston spoke about. Often, as with a couple of speakers at Playful 2011, the discussion is jokily dismissive. They’re all, “Why isn’t the 21st century like everything I was promised as a kid? Why are there no jetpacks / orbiting cities / Mars colonies?”

Well, you know, we don’t have those things because those “promises” were fictions, flights of fancy. And, even if you’re serious, we don’t have those things because it takes longer to develop such technologies and societies than it does to for you to grow up into a neophilic thirty-year-old.

But also, the more usual response, other than rolling one’s eyes and leaving the room, is to point out the many ways that 2012 would seem futuristic to anyone from your childhood.

The text of Warren Ellis’s opening keynote at Improving Reality should be tattooed across the epidermis of anyone who treats the 1979 Usborne Book of the Future as some kind of broken contract. There’s no reason to describe the present as banal and disappointing, compared to past visions.

A writer called Ventakesh Rao recently used the term “manufactured normalcy” to describe this. The idea is that things are designed to activate a psychological predisposition to believe that we’re in a static and dull continuous present.

Later, Warren says:

The theories of atemporality and manufactured normalcy and zero history can be short-circuited by just one thing.

Looking around.

Ballardian banality comes from not getting the future that we were promised, or getting it too late to make the promised difference.

This is because we look at the present day through a rear-view mirror. This is something Marshall McLuhan said back in the Sixties, when the world was in the grip of authentic-seeming future narratives. He said, “We look at the present through a rear-view mirror. We march backwards into the future.”

He went on to say this, in 1969, the year of the crewed Moon landing: “Because of the invisibility of any environment during the period of its innovation, man is only consciously aware of the environment that has preceded it; in other words, an environment becomes fully visible only when it has been superseded by a new environment; thus we are always one step behind in our view of the world. The present is always invisible because it’s environmental and saturates the whole field of attention so overwhelmingly; thus everyone is alive in an earlier day.”

Maybe the problem is that in the past — I don’t know, say, the past of 40+ years ago — (our) society was simpler. Less fractured, more top-down, more homogenous. And so it was easier to create consensus visions of the future. Post-World War II, the world would improve, technology would generate bigger, better, faster stuff, and the only variable would be whether East vs West would destroy us all before we got to Mars.

I don’t quite subscribe to the idea that in the 21st century the world is changing faster than ever before. But (our) society is more varied than ever before, and so maybe the world is changing in more ways than ever before. This perhaps leads to a greater variety of possible visions of the future floating around, all of them seeming less likely than our once shared single jumpsuits-and-space-cities future. Perhaps this is why some people can’t imagine a future — because it’s harder work. There’s no one dominant image of tomorrow.

Which isn’t to say the future is unimaginable. To do so — like one apparently imagination-free guy in the Brighton SF audience who asked a question — is an inability to see the present and to extrapolate in any way from the vast amount of possibilities you see. Warren Ellis again:

To be a futurist, in pursuit of improving reality, is not to have your face continually turned upstream, waiting for the future to come. To improve reality is to clearly see where you are, and then wonder how to make that better.

Act like you live in the Science Fiction Condition. Act like you can do magic and hold séances for the future and build a brightness control for the sky.

Act like you live in a place where you could walk into space if you wanted. Think big. And then make it better.