Hosted Grid - Production

Using the Grid in production

If the Grid feels like a good fit for your organisation, you have many options for how to run it in production.

Install the Grid in your own Amazon Web Services account

The Grid is open source software and the source code can be obtained here.

Your in house developers will need to build the Grid from source and deploy it to AWS.

The Guardian describe some of the AWS dependencies in their setup documentation.

Run the Grid from prebuilt container images

Eel Pie Consulting maintain a container based distribution of the Grid.

A containerised Grid may be easier to install outside of AWS.

Use Hosted Grid as a paid for service

Eel Pie Consulting offer Hosted Grid as a paid for service.

Using Hosted Grid in production

Things to consider when using Hosted Grid in production.

Understanding image retention

Ensure that you understand the Grid's image reaping behaviour and that this is appropriate for your intended use.

Setting up your usage rights

Correctly setting your usage rights before adding images helps to ensure that your Grid is showing correct image usage rights to your editors.

Agency subscriptions

If the Grid knows which agencies you have subscriptions with, it can automatically set the usage rights for uploaded and feed images.

Images from these suppliers will appear as free to use in the Grid:

List the names of supplier you have subscriptions with. freeSuppliers would typically contain suppliers like AP, EPA, Getty Images, PA and Reuters.

If particular collections from a given supplier are paid for images then list them in suppliersCollectionExcl

Identifying your publication photographers

Correctly crediting your in-house photographers helps your editors identify images which they can use in preference to agency images. Been able to identify images your publications created also helps with syndication.

Foreach of your publications, provide these lists of photograph names: externalStaffPhotographers, internalStaffPhotographers, contractedPhotographers, staffIllustrators, contractIllustrators

An image's byline is automatically matched against your photographer lists when it is uploaded. The byline may be populated from one of these metadata fields: dc:creator, IPTC By-line, EXIF Artist

Agency image feeds

Hosted Grid can ingest images from feed suppliers in real time.

The delivery point for agency feeds is a folder in an Amazon Web Services compatible S3 bucket. Feed image upload works the same as bulk archive uploads (above) but images should be sent to the [your grid]/feeds/ folder.

For example, using the AWS command line tool:

aws --endpoint-url http://storage.googleapis.com s3 cp IMG_3365.JPG s3://hostedgrid-ingest/[your grid]/feeds/IMG_3365.JPG

We will supply you with S3 credentials which you can pass to your feed suppliers. It is your responsibility to instruct your feed suppliers to deliver files into this bucket. Hosted Grid's responsibility for ingesting the images begins once the files have been placed in the bucket.

The Endpoint URL parameter is important

Our S3 bucket is hosted on Google Cloud Platform rather than AWS.

This bucket is compatible with S3 clients (including the official AWS ones) but care must be taken to correctly set your S3 client's endpoint URL parameter.

You need to specify an explicit endpoint url rather than an AWS region when using a non AWS S3 provider.

The endpoint-url replaces the default AWS endpoint which would have been calculated from your AWS Region.

AWS describe this in more detail here. They also describe the set endpoint for a single command pattern used in these examples.

Crops and CDN origins

If you are intending to use crops from the Grid online then you will almost certainly want Hosted Grid to publish your crops to your CDN origin.

If you supply Hosted Grid with AWS credentials for one of your S3 buckets, crops can be published directly to that bucket.

Crop publishing bucket

You will need to provide an AWS S3 bucket which the Grid can publish your crops to.

This bucket can be the public access bucket you use as your public doc-root or as your CDN origin.

You will need to provide the Grid with AWS credentials which allow it write, list and delete files within this bucket.

In AWS create a new IAM user and issue an access token and secret for it. Attach an AWS role similar to the one below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid":"ReadWriteS3",
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::amzn-s3-demo-bucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectTagging",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::amzn-s3-demo-bucket/*"
            ]
        }
    ]
}

Migrating an existing Grid to Hosted Grid

How does Hosted Grid differ from the open source Grid?

Hosted Grid is a private fork of the open source Grid. Eel Pie Consulting have modified it to remove Guardian specific features and add enhancements which are useful to non Guardian users.

Hosted Grid is kept up to date with the open source Grid code. Eel Pie Consulting contributes back to the open source Grid where appropriate.

Hosted Grid maintains the same API format as the open source Grid. Data read from Hosted Grid is compatible with the open source Grid.

Usage rights

Your existing Grid will contain an implementation of UsageRightsConfigProvider. This is normally controlled by configuration files.

A copy of your configuration can be loaded into your hosted grid.

Quota configuration

Your existing quota configuration can be copied into your hosted grid.

Bulk archive upload via S3

Image archives can be uploaded via an Amazon Web Services S3 compatible bucket.

We will supply you with S3 credentials (an access token and secret) which allow you to upload your images to our S3 ingest bucket.

The folder to upload to is specific to your Grid.

Examples using the AWS command line tool:

Upload a single image:
aws --endpoint-url http://storage.googleapis.com s3 cp IMG_3365.JPG s3://hostedgrid-ingest/[your grid]/IMG_3365.JPG

Upload a folder of images
aws --endpoint-url http://storage.googleapis.com s3 cp --recursive /your-local-image-folder/ s3://hostedgrid-ingest/[your grid]/

Migrating away from Hosted Grid

Avoiding lock in. Your image data is yours. Hosted Grid stores your data in standard formats. At anytime you are free to download your original data or extract it via the API.

The original Grid is open source software. There is no technical reason why images extracted from Hosted Grid cannot be migrated to a self-hosted Grid or another supplier. Such a migration would be your responsibility and data transfer charges may apply but there are no Hosted Grid restrictions which would prevent you from doing so.

Pricing

Hosted Grid pricing is dependent on your intended usage patterns.

Any pricing discussion will consider your intended use, the volume of images you wish to store and the number of users who require access.

These factors contribute to pricing:

Image storage	Volume of original images. Images are stored on cloud services and the total size of your image files is a direct cost which has to be past on. £0.04 per GB / 30 days.
Upload of images	Volume of uploaded images. Images arriving from agencies or by direct upload incur network traffic charges which contribute to cost. £0.15 per GB.
Download of images	Volume of downloaded images. Original images leaving the system via the API or direct download incur network traffic charges which contribute to cost. £0.15 per GB.
Number of users	Pricing reflects the number of users permitted to use your Grid. £15.00 per user per calendar month.
Support	A flat fee to cover the cost of supporting your Grid.