After my last post on the ONS data structure this post is the first of a few on using that structure and some other public data, mostly UK government data, and mapping it using R. This first post is about getting shapefiles from various locations, loading them into R and plotting them.
Tag: Data
ONS Data Structure
I have been looking through the ONS geographic data on their Geo Portal and there are acronyms and variables everywhere so I thought it best to understand what they all mean. Whenever I refer to the output areas and super output areas I’m referring to the ones as at the 2011 census in England and Wales.
Changing a colour scheme and extracting data from the internet
Sometimes I have a spreadsheet containing lots of spreadsheets of similar formats all using the same colour scheme. And if after a while I decide I don’t like the colours any more then it can be quite annoying to change all of the colours. So I decided to write this short little macro to change the colours.
Mapping UK Cities and Towns in Excel
Continuing my theme of recreating spreadsheets I’ve seen on Clearly and Simply I’ve had a go at making an Excel map of UK cities and towns that can be highlighted to find their combined population.
Looping over a string in SAS
In SAS it is easy to loop a macro between two numbers
%DO I=1 %TO 10; ... %END;
But if you have a list of non-sequential numbers or text you want to run your macro over, e.g. a list of towns, it can be a bit trickier. This SUGI paper gives a macro which lets you do just that. A few years ago I looked at that macro and didn't really understand how it worked and not wanting to use code I didn't understand I wrote my own version.
Summarising data in R
Summarising data
One of the most frequent tasks I do is summarising data using either proc sql or proc means with code like this:
proc means data=inputdata nway missing noprint;
class var1 var2;
var var3 var4;
output out=outputdata (drop = _type_ _freq_) sum=;
run;
Given that I use it in SAS a lot I’m going to assume that I’ll use it in R a lot so it seems like the next sensible thing to learn.
Read More
UK Train Data
One thing I keep on wanting to get around to looking at is what publicly available data is available on the trains in the UK and the answer to a few questions?
- What proportions of the trains in the UK are run by foreign governments?
- At a rough guess it is in the region of 20% based on not very much
- How would this be measured? Passenger miles?
- Why can the French, Dutch and German governments run UK rail franchises but not the UK government? It makes no sense!
- What subsidies are given to the different franchises?
- Are the original tenders released publicly?
- Do the train companies live up to these documents?
First attempt at simple analysis in R - Part 1
It’s time to start some analysis, albeit very basis analysis. I want to look at the interaction between the ONS Rural score and the average Broadband speed. This will be done using the postcode file created in my previous post. I’m assuming that the more rural a place is the slower its broadband will be. Is this actually the case?
The aim of this exercise is to learn so R skills not do some rigorous analysis. This means that some rather broad and potentially foolish assumptions will be made with the data to make some things easier to code given my novice R skills.
Read More
RSS Challenge 2015
I thought I’d take a look at the RSS (Royal Statistical Society ) “Statistical Analytics Challenge” after being sent it at work today. It involves analysing eye movement on 60 pictures.
Whilst I’m not going to enter the competition I am going to have a go and see how far I get. My plan goes something like this:
- Read the image into R
- Split it into a grid
- Initially a large grid and then progressively smaller ones
- Calculate some properties of each of the grid cells
- Check how each of these properties correlate with where the eye movement points are
- Check the properties of the surrounding grid cells relative the current cell
- See how these new properties interact with the eye movements.
- Do each of the above for a number of pictures to come up with a model and then test this on one of the other pictures.
Creating a postcode file in R from different public data sources
Following on from my previous post on creating a postcode file with the 2015 general election results I wanted to create a larger file with more variables. Some from the ONS lookups, others from different public datasets. The one I have added so far is the Office of Communications Broadband Coverage dataset from 2013.
The final dataset will contain for each postcode in the UK:
- The 2015 general election result
- The Westminster Election Constituency
- The Easting and Northing coordinates
- Census lookup areas
- Rural Indicator
- Broadband coverage data
- Which (if any) national park the postcode is in