Category Archives: Data Science

Getting to grips with Probability

14 Monday Sep 2015

Posted by in Data Science

≈ Leave a comment

Tags

,

calcWhen learning Data Science one area to learn is that of probability.

William Chen has created a good 10 page Probability Cheat Sheet to help guide you through. The content is based on “Harvard’s Introduction to Probability”.

The cheatsheet summarizes important probability probability concepts, formulas, and distributions, with figures, examples, and stories.

There are also 16 Data Science books listed on his site. A couple of which cover statistics and probability.

http://www.wzchen.com/data-science-books/

Some other Probability resources to get you started:

Source: http://www.wzchen.com/probability-cheatsheet

Visual Introduction to Machine Learning

03 Monday Aug 2015

Posted by in Data Science, Machine Learning

≈ 1 Comment

Tags

, ,

I came across this “Visual Introduction to Machine Learning” in a forum. This is an experimental site showing statistical thinking with an interactive web page. The page builds as you scroll down and takes you through a journey of Machine Learning.

It provides a high level graphical view of:Machine

  • Nuance
  • Drawing boundaries
  • Machine learning
  • Forks
  • Tradeoffs
  • Best splits
  • Recursion
  • Trees
  • Making predictions

The URL in the Web page indicates that this is part 1 so hopefully there will be more to follow with the first page indicated further posts on “overfitting, and how it relates to a fundamental trade-off in machine learning”

You can follow this project on Twitter @r2d3us

Other posts on Machine Learning

Learning Data Science – Useful References

14 Tuesday Jul 2015

Posted by in Big Data, Data Science, Machine Learning, Open Source

≈ Leave a comment

Tags

, , , ,

Firstly thanks to Tim Osterbuhr who prompteLearningd me to create this list of resources that I have found useful in learning about Data Science after he read my blog post on Learning Data Science. Tim has also provided some of the likes below as well.

Here is the list of Useful References for Learning Data Science. (This list is be no means exhaustive)

From my Blog

From Around the Web

Public Data Sets

Data Science Books

Happy to add other links from readers to this list.

The R Consortium

01 Wednesday Jul 2015

Posted by in Data Science, Open Source, Programming

≈ Leave a comment

Tags

, , ,

RThe R Consortium has been founded and recently launched under an Open Source Governance and Foundation model. This is in response to the growing use of R and its communities.

The initial members of the R Consortium include:

  • Microsoft
  • R Studio
  • TIBC Analytics
  • Alteryx
  • Google
  • HP
  • Ketchum Trading LLC
  • Mango Solutions
  • Oracle

The mission statement is listed as:

The central mission of the R Consortium is to work with and provide support to the R Foundation and to the key organizations developing, maintaining, distributing and using R software through the identification, development and implementation of infrastructure projects.

They have also listed a potential number of projects they will be involved with:

  • strengthening the R Forge infrastructure;
  • assisting the Stanford University group running user!R 2016;
  • developing documentation; and
  • encouraging increased communication and collaboration among users and developers of the R language.

One to watch going forward for influencing the R Community.

Source https://www.r-consortium.org

Getting to Grips with Git

08 Monday Jun 2015

Posted by in Cloud, Data Science, DevOps/OpsDev, Open Source, Programming

≈ Leave a comment

Tags

, , , , , ,

If you are new to GIT or want to refresh your skills/knowledge a good way of learning is through the Learning Git Branching simulator with it taking you through the commands and techniques.

Welcome to Learn Git Branching

Interested in learning Git? Well you’ve come to the right place! “Learn Git Branching” is the most visual and interactive way to learn Git on the web; you’ll be challenged with exciting levels, given step-by-step demonstrations of powerful features, and maybe even have a bit of fun along the way.

After this dialog you’ll see the variety of levels we have to offer. If you’re a beginner, just go ahead and start with the first. If you already know some Git basics, try some of our later more challenging levels.

The simulator covers:

Introduction SequenceGithub
– Introduction to the majority of Git commands

Ramping Up
– Additional Git commands

Moving Work Around
– Modifying the source tree

A Mixed Bag
– Git techniques tricks and tips

Advanced Topics

Open Data Handbook Revised

21 Thursday May 2015

Posted by in Data Science, IoT, Open Source

≈ Leave a comment

Tags

, ,

openOriginally published in 2012 the “Open Data Handbook” has been revamped to inspire Open Data Newcomers.

The new version of the online site allows access to:

  • The Open Data Book
  • Value Stories
  • Resource Library

For those who have not yet been introduced to the Open Data Handbook, the below is a good overview of what its about.

This handbook discusses the legal, social and technical aspects of open data. It can be used by anyone but is especially designed for those seeking to open up data. It discusses the why, what and how of open data – why to go open, what open is, and the how to ‘open’ data.

There are already a lot of “Open” sources of such as Open Source Web Crawlers and Data Sets, which is set to grow in the wake of the Internet of Things (IoT) and other data creation solutions.

The Handbook/Guide provides a good place to start if you are considering making your data Open, however the data created will also be a huge revenue generator for companies who produce the IoT devices.  The amount of data that will made Open remains to be seen more things come online

Open Source Web Crawlers and Data Sets

15 Friday May 2015

Posted by in Big Data, Data Science

≈ 1 Comment

Tags

, ,

webA great list of 50 Open Source Web Crawlers has been produced by Baiju NT on a Big Data Blog

Web Crawlers are useful in gathering data from other sites when performing research, although caution should be used as with today’s levels of protection some sites defenses may consider your data gathering as an attack.

Its probably best to check first if any data sets exist with the data you are looking for.

https://www.quandl.com/ is a search engine for data sets that has listed 12 million data sets.

There are lots of data sets available from governments such as http://data.gov.uk/ in the UK.

If its a smaller list of good data sources is needed have a look at http://www.kdnuggets.com/datasets/index.html

Sources:

Data Mining Courses

28 Tuesday Apr 2015

Posted by in Big Data, Data Science

≈ 1 Comment

Tags

, , ,

mineVia Coursera the University of Illinois at Urbana-Champaign is running a specialisation on Data Mining.  As with all Coursera courses, you don’t have to take the specialisation, but can take the courses individually or one after each other. Taking the courses outside of the specialisation means that you wont get to complete the capstone project and earn your certificate at the end.

This track is made up 5 courses covering:

Pattern Discovery in Data Mining

  • Introduction to data mining
  • Concepts and challenges in pattern discovery and analysis
  • Scalable pattern discovery algorithms
  • Pattern evaluation
  • Mining flexible patterns in multi-dimensional space
  • Mining sequential patterns
  • Mining graph patterns
  • Pattern-based classification
  • Application examples of pattern discovery

Text Retrieval and Search Engines

  • Introduction to text data mining
  • Basic concepts in text retrieval
  • Information retrieval models
  • Implementation of a search engine
  • Evaluation of search engines
  • Advanced search engine technologies

Cluster Analysis in Data Mining

  • Basic concept and introduction
  • Partitioning methods
  • Hierarchical methods
  • Density-based methods
  • Probabilistic models and EM algorithm
  • Spectral clustering
  • Clustering high dimensional data
  • Clustering streaming data
  • Clustering graph data and network data
  • Constraint-based clustering and semi-supervised clustering
  • Application examples of cluster analysis

Text Mining and Analytics

  • Overview of text analytics and applications
  • Extending a search engine to support text analytics (text categorization, text clustering, text summarization)
  • Topic mining and analysis with statistical topic models
  • Opinion mining and summarization
  • Integrative analysis of text and structured data

Data Visualization

  • Visualization Infrastructure (graphics programming and human perception)
  • Basic Visualization (charts, graphs, animation, interactivity)
  • Visualizing Relationships (hierarchies, networks)
  • Visualizing Information (text, databases)

These courses would complement the courses from John Hopkins on Data Science

Source: https://www.coursera.org/specialization/datamining/20

Big Data – 4V’s + Verification

27 Monday Apr 2015

Posted by in Big Data, Data Science, IoT

≈ Leave a comment

Tags

, , , ,

IBM have released an Infographic on the “Four V’s of Big Data” which covers:

  • Volume – Scale of Data
  • Variety – Different forms of Data
  • Velocity – Analysis of Streaming Data
  • Veracity – Uncertainty of Data

4-Vs-of-big-data

There should be another V for “Verification” which covers the questions you ask of the data in order to obtain the results. A check should also be made on the data to look at the inference of the results as different views or questions asked in a slightly different way could produce completely different outcomes in the data.

Having the right data is important and ensuring the data gathered and collected is relevant to the business questions you are asking. Two stats in the infographic stick out for me on this:

  • $3.1 Trillion a year on poor data quality
  • 40 Zetabytes of data created by 2020

Perhaps with the right Verification there may not be so much uncertainty (Veracity) and a huge saving to businesses reducing a high loss in money, time and incorrect data.

R {swirls} – Learning R by doing

16 Thursday Apr 2015

Posted by in Data Science, Programming

≈ 1 Comment

Tags

, , ,

A swirl is an interactive way of learning R by installing a package called {swirl} into R and then installing a course.

I have used swirls in the Data Science Courses on Coursera and found them a useful way of learning and testing your knowledge.

swirl is installed as a package into R using the following command in R (internet connection required).

> install.packages("swirl")

Then launching the swirl library and run it.

> library("swirl")
> swirl()

To locate a swirl course use the following command.

?InstallCourses

Sources: Swirlstats

There are a list of courses available in the swirl repository on GitHub. There are 3 levels of courses available.

Beginner

  • R Programming: The basics of programming in R
  • R Programming Alt: Same as the original, but modified slightly for in-class use
  • Data Analysis: Basic ideas in statistics and data visualization
  • Mathematical Biostatistics Boot Camp: One- and two-sample t-tests, power, and sample size
  • Open Intro: A very basic introduction to statistics, data analysis, and data visualisation

Intermediate

  • Regression Models: The basics of regression modeling in R
  • Getting and Cleaning Data: dplyr, tidyr, lubridate, oh my!

Advanced

  • Statistical Inference: This intermediate to advanced level course closely follows the Statistical Inference course of the Johns Hopkins Data Science Specialization on Coursera.

To install a course you can use the following commands in R

library(swirl)
install_from_swirl("Course Name Here")
swirl()

Datacamp have recently released a free browser based R learning tool. This is a browser based  version to learn R based on a flipcard version of swirl teaching you in bite sized chunks.

Sources:

R