The Hack-R Blog

Posts Tagged ‘big-data’

cbind.fill <- function(…){
nm <- list(…)
nm <- lapply(nm, as.matrix)
n <- max(sapply(nm, nrow))
do.call(cbind, lapply(nm, function (x)
rbind(x, matrix(, n-nrow(x), ncol(x)))))
}

When you generate a model in Ruby on Rails

rails generate model somename

A migration table is also generated.

The Migration is a table that describes a database and the information it holds in columns.

bash >> hive -e ‘select * from some_Table’ | sed ‘s/[\t]/,/g’ > outputfile.txt

Here [\t] means Control+V and then the tab button, i.e.
sed ‘s//,/g’

Example:

[user@server]$ hive -e "use dbname ; select * from tablename" | sed ‘s/ /,/g’ > kpi_event_jan8.csv

Add this to your query:

set hive.cli.print.header=true;

Example:

hive -e "set hive.cli.print.header=true; use a_db; select * from a_table;" > test

Forged by the Gods, the ancient data science venn diagram is the oldest, most sacred representation of the field of data science.

I’ve been in love with this simple diagram since I first began working as a data scientist. I love it because it so clearly and simply represents the unique skillset that makes up data science. I’ll write more on this topic and how my own otherwise eclectic skillset coalesced into the practice of professional data science.

I wish I could take credit for creating this simple-but-totally-unsurpassed graphic. Over the past couple of years I’ve often used it as an avatar and if you look close enough you’ll even find it in the background of my (hacked) WordPress header image. While I like to think that it was immaculately convinced, the word on the street is that it was created for the public domain by Drew Conway, who is the co-author of Machine Learning for Hackers*, a private market intelligence and business consultant, my fellow recovering social scientist,  recent PhD grad from NYU, and a fast-rising name in data science (yea, he wants to be like me).

./bin/hbase shell

Want the power of your own Hadoop cluster but lack the budget, time, human resources, and/or capital? No problem!

Segue makes using Amazon’s Elastic MapReduce in R not only possible, but downright simple. The best thing about this Big Data trick is that is brings you scalability without the out-of-pocket or opportunity costs normally associated with setting up a Hadoop cluster because of the elastic nature of EMR. The term elastic emphasizes that Amazon Web Services (AWS) extended the concept of scalability to their pricing model — clusters of virtual machines can be spun up and down at will, using whatever amount of RAM and compute power that you deem necessary. As soon as you’re finished with your virtual cluster the billing clock stops.

It gets better. AWS provides a free tier of service for the community, so that you can learn what you’re doing and test out EMR on your use case without having to worry about money (last time I checked it was about 750 free machine hours per month; check for changes). I’ve boiled down a 30-45 presentation that I used to give on the topic to the follow 6 simple steps:

R CMD BUILD segue
R CMD INSTALL segue_0_05.tar.gz

library(segue)

setCredentials("user", "pass")

emr.handle <- createCluster(numInstances=6 )