Archive for the ‘Big Data’ Category

R: cbind fill for data.table

cbind.fill <- function(…){
nm <- list(…)
nm <- lapply(nm, as.matrix)
n <- max(sapply(nm, nrow))
do.call(cbind, lapply(nm, function (x)
rbind(x, matrix(, n-nrow(x), ncol(x)))))
}

RStudio-icon

Firebase Quick Tut Transcription

I copied this down from the interactive 5 minute tutorial of Firebase which is useful for storing user-submitted data for mobile apps:

 

Firebase relies on a library that you include in your app. This library gives you easy access to Firebase’s authentication and database features.

To get you started we’ve created an HTML page below. Install the Firebase JavaScript library by adding the following line into the <head> section below:

<script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
1
<html>
2
  <head>
3
    [ADD THE SCRIPT TAG HERE]
4
  </head>
5
  <body>
6
  </body>
7
</html>

2. Accessing your Realtime Database

This tutorial focuses on Firebase’s realtime database. Note that Firebase also has powerful authentication and hosting services.

To access your Firebase database, you’ll first need to create a reference.

References are created using a URL that specifies which data you want to access. We’ve already created a Firebase database specifically for you at this URL: https://u70eg119il2.firebaseio-demo.com/

To complete this step, create a reference to the root of your Firebase database as shown below:

var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
  </head>
5
  <body>
6
    <script>
7
      [ADD NEW FIREBASE CODE HERE]
8
    </script>
9
  </body>
10
</html>

3. Writing Data

Let’s send a chat message

You can use the reference you just created to write data to your Firebase database using the set() function.

To make things easy, we’ve already added input text boxes for the chatter’s name and message as well as a keypress handler that will fire whenever someone tries to send a message.

For this step, write a message to your Firebase database using the set() function as shown:

myDataRef.set('User ' + name + ' says ' + text);
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          [ADD SET() HERE]
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

4. Writing Objects

The set() function can also be used to write objects.

Try changing your code to write an object with text and name properties:

myDataRef.set({name: name, text: text});
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.set('User ' + name + ' says ' + text);
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

5. Writing Lists

The Firebase database supports lists of data.

You’ve already learned how to write data to specific, named locations in the database, but your chat application will require a list of messages. The Firebase database provides a helper function called push() that makes creating lists easy.

Modify your code below to use push() instead of set() so that your chat can support a list of messages (rather than just one):

myDataRef.push({name: name, text: text});
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.set({name: name, text: text});
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

6. Reading Data

Now let’s receive chat messages.

We need to tell the database to notify us when chat messages arrive. We do this by adding a callback to the list of chat messages using the on() method, as shown below:

myDataRef.on('child_added', function(snapshot) {
  //We'll fill this in later.});

This method takes two arguments: the event type and the callback function. We’ll use the ‘child_added’ event so that we are notified of the arrival of individual messages.

1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.push({name: name, text: text});
16
          $('#messageInput').val('');
17
        }
18
      });
19
      [ADD YOUR CALLBACK HERE]
20
    </script>
21
  </body>
22
</html>

7. Using Snapshots

Now we need to display the chat messages on the page.

For each chat message, the database will call your callback with a snapshot containing the message’s data.

Extract the message data from the snapshot by calling the val() function and assign it to a variable. Then, call the displayChatMessage() function to display the message as shown:

var message = snapshot.val();
displayChatMessage(message.name, message.text);
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <div id='messagesDiv'></div>
8
    <input type='text' id='nameInput' placeholder='Name'>
9
    <input type='text' id='messageInput' placeholder='Message'>
10
    <script>
11
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
12
      $('#messageInput').keypress(function (e) {
13
        if (e.keyCode == 13) {
14
          var name = $('#nameInput').val();
15
          var text = $('#messageInput').val();
16
          myDataRef.push({name: name, text: text});
17
          $('#messageInput').val('');
18
        }
19
      });
20
      myDataRef.on('child_added', function(snapshot) {
21
        [MESSAGE CALLBACK CODE GOES HERE]
22
      });
23
      function displayChatMessage(name, text) {
24
        $('<div/>').text(text).prepend($('<em/>').text(name+': ')).appendTo($('#messagesDiv'));
25
        $('#messagesDiv')[0].scrollTop = $('#messagesDiv')[0].scrollHeight;
26
      };
27
    </script>
28
  </body>
29
</html>

Rails: Migration Files

When you generate a model in Ruby on Rails

rails generate model somename

A migration table is also generated.

The Migration is a table that describes a database and the information it holds in columns.
rubylang

Rails: Add data from seeds to db in 1 Line

rake db:seed

Hive: Make CLI output files comma delimited

bash >> hive -e ‘select * from some_Table’ | sed ‘s/[\t]/,/g’ > outputfile.txt

Here [\t] means Control+V and then the tab button, i.e.
sed ‘s//,/g’

Example:

[user@server]$ hive -e "use dbname ; select * from tablename" | sed ‘s/ /,/g’ > kpi_event_jan8.csv

Hive: Get Column Names in CLI Queries

Add this to your query:

set hive.cli.print.header=true;

Example:

hive -e "set hive.cli.print.header=true; use a_db; select * from a_table;" > test

hive_logo_medium

Protected: 2 Ways to Avoid Storing DB Passwords in R Scripts

This content is password protected. To view it please enter your password below:

Linux: How to Install and Configure a Seedbox

#rTorrent for Transferring Free and Open Source files only!
 
mkdir ~/install
mkdir /var/www/files
mkdir /var/www/watch
mkdir /var/www/.temp
chown -R www-data:www-data /var/www
 
apt-get update
apt-get -y upgrade
apt-get -y install apache2 apache2-utils autoconf build-essential ca-certificates comerr-dev libapache2-mod-php5 libcloog-ppl-dev libcppunit-dev libcurl3 libcurl4-openssl-dev libncurses5-dev ncurses-base ncurses-term libterm-readline-gnu-perl libsigc++-2.0-dev libssl-dev libtool libxml2-dev ntp openssl patch libperl-dev php5 php5-cli php5-dev php5-fpm php5-curl php5-geoip php5-mcrypt php5-xmlrpc pkg-config python-scgi dtach ssl-cert subversion zlib1g-dev pkg-config unzip htop irssi curl cfv nano unrar-free mediainfo libapache2-mod-scgi
ln -s /etc/apache2/mods-available/scgi.load /etc/apache2/mods-enabled/scgi.load
 
cd ~/install
svn checkout http://svn.code.sf.net/p/xmlrpc-c/code/stable xmlrpc-c
cd xmlrpc-c
./configure --disable-cplusplus
make
make install
 
cd ~/install
wget http://libtorrent.rakshasa.no/downloads/libtorrent-0.13.2.tar.gz
tar xvf libtorrent-0.13.2.tar.gz
cd libtorrent-0.13.2
./autogen.sh
./configure
make
make install
 
cd ~/install
wget http://libtorrent.rakshasa.no/downloads/libtorrent-0.13.2.tar.gz
tar xvf libtorrent-0.13.2.tar.gz
cd libtorrent-0.13.2
./autogen.sh
./configure
make
make install
 
nano ~/.rtorrent.rc
#PASTE THE FOLLOWING
 
# Configuration file created for www.filesharingguides.com for single user rutorrent seedbox
# Maximum and minimum number of peers to connect to per torrent.
# min_peers = 25
max_peers = 100
# Same as above but for seeding completed torrents (-1 = same as downloading)
min_peers_seed = -1
max_peers_seed = -1
# Maximum number of simultanious uploads per torrent.
max_uploads = 100
# Global upload and download rate in KiB. "0" for unlimited.
download_rate = 0
upload_rate = 0
# Default directory to save the downloaded torrents.
directory = /var/www/files
# Default session directory. Make sure you don't run multiple instance
# of rtorrent using the same session directory. Perhaps using a
# relative path?
session = /var/www/.temp
# Watch a directory for new torrents, and stop those that have been
# deleted.
schedule = watch_directory,5,5,load_start=/var/www/watch/*.torrent
schedule = untied_directory,5,5,stop_untied=
# Close torrents when diskspace is low.
schedule = low_diskspace,5,60,close_low_diskspace=100M
# The ip address reported to the tracker.
#ip = 127.0.0.1
#ip = rakshasa.no
# The ip address the listening socket and outgoing connections is
# bound to.
#bind = 127.0.0.1
#bind = rakshasa.no
# Port range to use for listening.
port_range = 6890-6999
# Start opening ports at a random position within the port range.
#port_random = no
# Check hash for finished torrents. Might be usefull until the bug is
# fixed that causes lack of diskspace not to be properly reported.
#check_hash = no
# Set whetever the client should try to connect to UDP trackers.
#use_udp_trackers = yes
# Alternative calls to bind and ip that should handle dynamic ip's.
#schedule = ip_tick,0,1800,ip=rakshasa
#schedule = bind_tick,0,1800,bind=rakshasa
# Encryption options, set to none (default) or any combination of the following:
# allow_incoming, try_outgoing, require, require_RC4, enable_retry, prefer_plaintext
#
# The example value allows incoming encrypted connections, starts unencrypted
# outgoing connections but retries with encryption if they fail, preferring
# plaintext to RC4 encryption after the encrypted handshake
#
encryption = allow_incoming,enable_retry,prefer_plaintext
# Enable DHT support for trackerless torrents or when all trackers are down.
# May be set to "disable" (completely disable DHT), "off" (do not start DHT),
# "auto" (start and stop DHT as needed), or "on" (start DHT immediately).
# The default is "off". For DHT to work, a session directory must be defined.
#
dht = disable
# UDP port to use for DHT.
#
# dht_port = 6881
# Enable peer exchange (for torrents not marked private)
#
peer_exchange = no
#
# Do not modify the following parameters unless you know what you're doing.
#
# Hash read-ahead controls how many MB to request the kernel to read
# ahead. If the value is too low the disk may not be fully utilized,
# while if too high the kernel might not be able to keep the read
# pages in memory thus end up trashing.
#hash_read_ahead = 10
# Interval between attempts to check the hash, in milliseconds.
#hash_interval = 100
# Number of attempts to check the hash while using the mincore status,
# before forcing. Overworked systems might need lower values to get a
# decent hash checking rate.
#hash_max_tries = 10
scgi_port = 127.0.0.1:5000
 
######################################################
 
To test: 
screen
rtorrent
# NEXT INSTALL ruTorrent UI
cd ~/install
wget http://rutorrent.googlecode.com/files/rutorrent-3.5.tar.gz
tar xvf rutorrent-3.5.tar.gz
mv rutorrent /var/www
wget http://rutorrent.googlecode.com/files/plugins-3.5.tar.gz
tar xvf plugins-3.5.tar.gz
mv plugins /var/www/rutorrent
mv /var/www/rutorrent/* /var/www
chown -R www-data:www-data /var/www/rutorrent
 
#Set up authentication
nano /etc/apache2/sites-available/default
#paste this:
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
 
nano /var/www/.htaccess
#paste this:
AuthType Basic
AuthName "Protected Area"
AuthUserFile /var/passwd/.htpasswd
Require valid-user
 
#change permissions to enable www-data group
chown -R www-data:www-data /var/www/.htaccess
 
 
# create pw file using Apache's htpasswd util
mkdir /var/passwd
htpasswd -c /var/passwd/.htpasswd testuser
chown -R www-data:www-data /var/passwd
 
 
#run on boot
nano /etc/rc.local
# add this before ‘exit 0’:
screen -S rtorrent -d -m rtorrent

Transact-SQL, PL/SQL, PL/pgSQL defined and compared

  • PL/SQL is a proprietary procedural language used by Oracle
  • PL/pgSQL is a proprietary procedural language used by PostgreSQL
  • TSQL is a proprietary procedural language used by Microsoft in SQL Server.

TSQL is a proprietary procedural language used by Microsoft in SQL Server . Procedural languages are designed to extend the SQL ‘s abilities while being able to integrate well with SQL .Several features such as local variables and string/data processing are added.These features make the language Turing complete.

Quick Tip: Create Table as Select (CTaS) Hack in MS SQL Server in 1 Line

select * into target_table from [DB]. [dbo].[table]