Archive for the ‘Substantive Expertise’ Category

R: cbind fill for data.table

cbind.fill <- function(…){
nm <- list(…)
nm <- lapply(nm, as.matrix)
n <- max(sapply(nm, nrow))
do.call(cbind, lapply(nm, function (x)
rbind(x, matrix(, n-nrow(x), ncol(x)))))
}

RStudio-icon

Firebase Quick Tut Transcription

I copied this down from the interactive 5 minute tutorial of Firebase which is useful for storing user-submitted data for mobile apps:

 

Firebase relies on a library that you include in your app. This library gives you easy access to Firebase’s authentication and database features.

To get you started we’ve created an HTML page below. Install the Firebase JavaScript library by adding the following line into the <head> section below:

<script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
1
<html>
2
  <head>
3
    [ADD THE SCRIPT TAG HERE]
4
  </head>
5
  <body>
6
  </body>
7
</html>

2. Accessing your Realtime Database

This tutorial focuses on Firebase’s realtime database. Note that Firebase also has powerful authentication and hosting services.

To access your Firebase database, you’ll first need to create a reference.

References are created using a URL that specifies which data you want to access. We’ve already created a Firebase database specifically for you at this URL: https://u70eg119il2.firebaseio-demo.com/

To complete this step, create a reference to the root of your Firebase database as shown below:

var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
  </head>
5
  <body>
6
    <script>
7
      [ADD NEW FIREBASE CODE HERE]
8
    </script>
9
  </body>
10
</html>

3. Writing Data

Let’s send a chat message

You can use the reference you just created to write data to your Firebase database using the set() function.

To make things easy, we’ve already added input text boxes for the chatter’s name and message as well as a keypress handler that will fire whenever someone tries to send a message.

For this step, write a message to your Firebase database using the set() function as shown:

myDataRef.set('User ' + name + ' says ' + text);
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          [ADD SET() HERE]
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

4. Writing Objects

The set() function can also be used to write objects.

Try changing your code to write an object with text and name properties:

myDataRef.set({name: name, text: text});
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.set('User ' + name + ' says ' + text);
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

5. Writing Lists

The Firebase database supports lists of data.

You’ve already learned how to write data to specific, named locations in the database, but your chat application will require a list of messages. The Firebase database provides a helper function called push() that makes creating lists easy.

Modify your code below to use push() instead of set() so that your chat can support a list of messages (rather than just one):

myDataRef.push({name: name, text: text});
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.set({name: name, text: text});
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

6. Reading Data

Now let’s receive chat messages.

We need to tell the database to notify us when chat messages arrive. We do this by adding a callback to the list of chat messages using the on() method, as shown below:

myDataRef.on('child_added', function(snapshot) {
  //We'll fill this in later.});

This method takes two arguments: the event type and the callback function. We’ll use the ‘child_added’ event so that we are notified of the arrival of individual messages.

1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.push({name: name, text: text});
16
          $('#messageInput').val('');
17
        }
18
      });
19
      [ADD YOUR CALLBACK HERE]
20
    </script>
21
  </body>
22
</html>

7. Using Snapshots

Now we need to display the chat messages on the page.

For each chat message, the database will call your callback with a snapshot containing the message’s data.

Extract the message data from the snapshot by calling the val() function and assign it to a variable. Then, call the displayChatMessage() function to display the message as shown:

var message = snapshot.val();
displayChatMessage(message.name, message.text);
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <div id='messagesDiv'></div>
8
    <input type='text' id='nameInput' placeholder='Name'>
9
    <input type='text' id='messageInput' placeholder='Message'>
10
    <script>
11
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
12
      $('#messageInput').keypress(function (e) {
13
        if (e.keyCode == 13) {
14
          var name = $('#nameInput').val();
15
          var text = $('#messageInput').val();
16
          myDataRef.push({name: name, text: text});
17
          $('#messageInput').val('');
18
        }
19
      });
20
      myDataRef.on('child_added', function(snapshot) {
21
        [MESSAGE CALLBACK CODE GOES HERE]
22
      });
23
      function displayChatMessage(name, text) {
24
        $('<div/>').text(text).prepend($('<em/>').text(name+': ')).appendTo($('#messagesDiv'));
25
        $('#messagesDiv')[0].scrollTop = $('#messagesDiv')[0].scrollHeight;
26
      };
27
    </script>
28
  </body>
29
</html>

How to Conditionally Remove Character of a Vector Element in R

I have (sometimes incomplete) data on addresses that looks like this:

data <- c("1600 Pennsylvania Avenue, Washington DC", 
          ",Siem Reap,FC,", "11 Wall Street, New York, NY", ",Addis Ababa,FC,")  

where I need to remove the first and/or last character if either one of them are a comma.

Avinash Raj was able to help me with this on S.O. and the question turned out to be a popular one, so I’ll show the solution here:

> data <- c("1600 Pennsylvania Avenue, Washington DC", 
+           ",Siem Reap,FC,", "11 Wall Street, New York, NY", ",Addis Ababa,FC,")
> gsub("(?<=^),|,(?=$)", "", data, perl=TRUE)
[1] "1600 Pennsylvania Avenue, Washington DC"
[2] "Siem Reap,FC"                           
[3] "11 Wall Street, New York, NY"           
[4] "Addis Ababa,FC" 

Pattern explanation:

  • (?<=^), In regex (?<=) called positive look-behind. In our case it asserts What precedes the comma must be a line start ^. So it matches the starting comma.
  • | Logical OR operator usually used to combine(ie, ORing) two regexes.
  • ,(?=$) Lookahead aseerts that what follows comma must be a line end $. So it matches the comma present at the line end.

Rlogo

Java: How to import StAX libraries for parsing XML

In short:

 

import javax.xml.stream.*;
import java.io.*;
import java.util.*;//usually, but not always needed

In long:

Here are steps in writing code to parse an XML document with StAX.

  1. Import the following libraries:
     import javax.xml.stream.*;
     import java.io.*;
  1. Create an XMLInputFactory . See the read() method above.
  2. Create an XMLStreamReader and pass a Reader to it such as a FileReader. The XML file is passed as a parameter to FileReader.
  3. We can now iterate through the contents of our XML file using the streamreader’s next() method.
  4. next() returns an event code that indicates which part of the document has been read such as: DTD, START_ELEMENT, CHARACTERS and END_ELEMENT.
  5. If you get the START_ELEMENT event code, you can retrieve the element’s name using the getLocalName() method. To read the attributes, use getAttributeValue() method.
  6. To read the text between the start and end tags, wait until you receive the CHARACTERS event code. Afterwards, you can read the text using getText().

Thanks to my instructor Carl Limsico for the step-by-step!

Software Sec: C / C++ Buffer overflows and Robert Morris

Buffer Overflow = any access of a bugger outside of its allotted bounds
  •      over-read or over-write
  •      could be during iteration (running off the end), or direct access (pointer arithmetic)
  •      this is a general definition; some people use more specific definitions of differing types of buffer overflows

A buffer overflow is a bug that affects low-level code, typically C and C++ with significant sec implications

Normally causes a crash, but can be used to:
  • dump/steal information
  • corrupt information
  • run code (payload)
They also share common features with other bugs.
C and C++ are the most popular languages (behind Java) and therefore buffer overflows are a major vuln. C/C++ are heavily used in:
  •      OS Kernels
  •      embedded systems
  •      HPC servers
 First buffer overflow occurred in 1988 by a student named Robert Morris, as part of a self-propagating computer worm that was an attack against fingerd and VAXes (Morris was caught and punished but is now a MIT professor); this worm affected 10% of the Internet
In 2001, CodeRed exploited a buffer overflow in the MS-IIS server, which infected >300,000 machines in 14 hours
In 2003 SQL Slammer worm infected 75,000 machines in 10 minutes by exploiting a buffer overflow in MS-SQL Server
In 2014 a latent buffer overflow bug was found in X11 that had been present over 23 years.

 

 

Computer History: CDC 6600

The first super-computer was the CD 6600, announced in 1964 by Control Data Corporation with a starting price of $6million

C: Dennis Ritchie, history of C

C was created by Dennis Ritchie of Bell Labs in 1972, while Ritchie worked with Ken Thompson on designing Unix.

C was based on Ken Thompson’s language, B.

 

Game Theory: Nash Equilibrium

In game theory, the Nash equilibrium is a solution concept of a non-cooperative game involving two or more players, in which each player is assumed to know the equilibrium strategies of the other players, and no player has anything to gain by changing only their own strategy.[1] If each player has chosen a strategy and no player can benefit by changing strategies while the other players keep theirs unchanged, then the current set of strategy choices and the corresponding payoffs constitutes a Nash equilibrium.

Stated simply, Amy and Will are in Nash equilibrium if Amy is making the best decision she can, taking into account Will’s decision, and Will is making the best decision he can, taking into account Amy’s decision. Likewise, a group of players are in Nash equilibrium if each one is making the best decision that he or she can, taking into account the decisions of the others in the game.

Macro: Capital Deepening, Capital Widening

Capital deepening is an economic scenario wherein capital per worker is increasing in the economy. This is also referred to as increase in the capital intensity. Capital deepening is often measured by the capital stock per labour hour. Overall, the economy will expand, and productivity per worker will increase. However, according to some economic models, such as the Solow model, economic expansion will not continue indefinitely through capital deepening alone. This is partly due to diminishing returns and wear & tear (depreciation). Investment is also required to increase the amount of capital available to each worker in the system and thus increase the ratio of capital to labor. In other economic models, for example, the AK model or some models in endogenous growth theory, capital deepening can lead to sustained economic growth even without technological progress. Traditionally, in development economics, capital deepening is seen as a necessary but not sufficient condition for economic development of a country.

Capital widening on the other hand is the economic scenario wherein capital stock is increasing at the same rate as the labor force and the depreciation rate, thus the capital per worker ratio remains constant. The economy will expand in terms of aggregate outputbut productivity per worker will remain constant.

Micro: Types of Inefficiency — X, Allocative, Dynamic, Social, Productive, Pareto, and Distributive

‘X’ inefficiency is a concept that was originally associated specifically with management inefficiencies, but can also be applied more widely.

X inefficiency occurs when the output of firms is not the greatest it could be. It is likely to arise when firms operate in highly uncompetitive markets where there is no incentive for managers to maximize output.

Allocative inefficiency occurs when the consumer does not pay aefficient price.

An efficient price is one that just covers the costs of production incurred in supplying the good or service.

Allocative efficiency occurs when the firm’s price, P, equals the extra (marginalcost of supplyMC. This is efficient because the revenue received is just enough to ensure that all the resources used in the making of a product are sufficiently rewarded to encourage them to continue supplying.

Dynamic inefficiency occurs when firms have no incentive to become technologically progressive. This is associated with a lack of innovation, which leads to higher production costs, inferior products, and less choice for consumers.

There are two ways in which firms can innovate:

  1. New production methods, such as when applying new technology to an existing process.
  2. New products, which are a feature of markets with highly competitive firms, such as those in the consumer electronics.

Innovation, researchand development are expensive and risky, so firms will expect a fair level of profits in return. However, because the price mechanism may not generate profits for the supply of public and merit goodsthere is often an absence of dynamic efficiency in these markets.

Social inefficiency occurs when the price mechanism does not take into account all the costs and benefits associated with economic exchange.

The price mechanism will only take into account private costs and benefits arising directly from production and consumption, not the external costs and benefits incurred by thirdparties.

Social costs refer to the total costs borne by society as a result of an economic transaction, and include private costs plus external costs. Social benefits are the private benefits plus external benefits resulting from a transaction.

A transaction is socially efficient if it takes into account costs and benefits associated with the transaction – that is, the social costs and benefits.

Productive inefficiency occurs when a firm is not producing at its lowest unit cost. Unit cost is the average cost of production, which is found by dividing total costs of production by the number of units produced.

It is possible that in markets where there is little competition, the output of firms will be low, and average costs will be relatively high. This is likely to occur if a few firms, or just one, dominate the market, as in the case of oligopoly and monopoly.

Pareto inefficiency is associated with economist Vilfredo Pareto, and occurs when an economy is not operating on the edge of its PPF and is, therefore, not fully exploiting its scarce resources.

This means that the economy is producing less than the maximum possible output of goods and services, from its resources.

Distributive inefficiency occurs when goods and services are not received by those who have the greatest need for them. Abba Lerner first proposed the idea of distributive efficiency in his 1944 book The Economics of Control.