Archive for the ‘Programming’ Category

Java: The Class class

From aau.dk

The java.lang.Class Class
• There is a Class object for each class in your system.
• The way Java type information is represented at run-time.
• Information for Class object is stored in a .class file
• Load when first object is created or static access
• The Class object is used to create all the objects of that class.
• The Java Virual Machine (JVM) finds the appropriate .class file
and loads it as a Class object the first time you need that class.
• Goes through the directories listed in the CLASSPATH
Java

R: Remove constant and identical features programmatically


##### Removing constant features
cat("\n## Removing the constants features.\n")
for (f in names(train)) {
  if (length(unique(train[[f]])) == 1) {
    cat(f, "is constant in train. We delete it.\n")
    train[[f]] <- NULL
    test[[f]] <- NULL
  }
}

##### Removing identical features
features_pair <- combn(names(train), 2, simplify = F)
toRemove <- c()
for(pair in features_pair) {
  f1 <- pair[1]
  f2 <- pair[2]

  if (!(f1 %in% toRemove) & !(f2 %in% toRemove)) {
    if (all(train[[f1]] == train[[f2]])) {
      cat(f1, "and", f2, "are equals.\n")
      toRemove <- c(toRemove, f2)
    }
  }
}

RStudio-icon

Firebase Quick Tut Transcription

I copied this down from the interactive 5 minute tutorial of Firebase which is useful for storing user-submitted data for mobile apps:

 

Firebase relies on a library that you include in your app. This library gives you easy access to Firebase’s authentication and database features.

To get you started we’ve created an HTML page below. Install the Firebase JavaScript library by adding the following line into the <head> section below:

<script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
1
<html>
2
  <head>
3
    [ADD THE SCRIPT TAG HERE]
4
  </head>
5
  <body>
6
  </body>
7
</html>

2. Accessing your Realtime Database

This tutorial focuses on Firebase’s realtime database. Note that Firebase also has powerful authentication and hosting services.

To access your Firebase database, you’ll first need to create a reference.

References are created using a URL that specifies which data you want to access. We’ve already created a Firebase database specifically for you at this URL: https://u70eg119il2.firebaseio-demo.com/

To complete this step, create a reference to the root of your Firebase database as shown below:

var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
  </head>
5
  <body>
6
    <script>
7
      [ADD NEW FIREBASE CODE HERE]
8
    </script>
9
  </body>
10
</html>

3. Writing Data

Let’s send a chat message

You can use the reference you just created to write data to your Firebase database using the set() function.

To make things easy, we’ve already added input text boxes for the chatter’s name and message as well as a keypress handler that will fire whenever someone tries to send a message.

For this step, write a message to your Firebase database using the set() function as shown:

myDataRef.set('User ' + name + ' says ' + text);
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          [ADD SET() HERE]
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

4. Writing Objects

The set() function can also be used to write objects.

Try changing your code to write an object with text and name properties:

myDataRef.set({name: name, text: text});
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.set('User ' + name + ' says ' + text);
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

5. Writing Lists

The Firebase database supports lists of data.

You’ve already learned how to write data to specific, named locations in the database, but your chat application will require a list of messages. The Firebase database provides a helper function called push() that makes creating lists easy.

Modify your code below to use push() instead of set() so that your chat can support a list of messages (rather than just one):

myDataRef.push({name: name, text: text});
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.set({name: name, text: text});
16
          $('#messageInput').val('');
17
        }
18
      });
19
    </script>
20
  </body>
21
</html>

6. Reading Data

Now let’s receive chat messages.

We need to tell the database to notify us when chat messages arrive. We do this by adding a callback to the list of chat messages using the on() method, as shown below:

myDataRef.on('child_added', function(snapshot) {
  //We'll fill this in later.});

This method takes two arguments: the event type and the callback function. We’ll use the ‘child_added’ event so that we are notified of the arrival of individual messages.

1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <input type='text' id='nameInput' placeholder='Name'>
8
    <input type='text' id='messageInput' placeholder='Message'>
9
    <script>
10
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
11
      $('#messageInput').keypress(function (e) {
12
        if (e.keyCode == 13) {
13
          var name = $('#nameInput').val();
14
          var text = $('#messageInput').val();
15
          myDataRef.push({name: name, text: text});
16
          $('#messageInput').val('');
17
        }
18
      });
19
      [ADD YOUR CALLBACK HERE]
20
    </script>
21
  </body>
22
</html>

7. Using Snapshots

Now we need to display the chat messages on the page.

For each chat message, the database will call your callback with a snapshot containing the message’s data.

Extract the message data from the snapshot by calling the val() function and assign it to a variable. Then, call the displayChatMessage() function to display the message as shown:

var message = snapshot.val();
displayChatMessage(message.name, message.text);
1
<html>
2
  <head>
3
    <script src='https://cdn.firebase.com/js/client/2.2.1/firebase.js'></script>
4
    <script src='https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js'></script>
5
  </head>
6
  <body>
7
    <div id='messagesDiv'></div>
8
    <input type='text' id='nameInput' placeholder='Name'>
9
    <input type='text' id='messageInput' placeholder='Message'>
10
    <script>
11
      var myDataRef = new Firebase('https://u70eg119il2.firebaseio-demo.com/');
12
      $('#messageInput').keypress(function (e) {
13
        if (e.keyCode == 13) {
14
          var name = $('#nameInput').val();
15
          var text = $('#messageInput').val();
16
          myDataRef.push({name: name, text: text});
17
          $('#messageInput').val('');
18
        }
19
      });
20
      myDataRef.on('child_added', function(snapshot) {
21
        [MESSAGE CALLBACK CODE GOES HERE]
22
      });
23
      function displayChatMessage(name, text) {
24
        $('<div/>').text(text).prepend($('<em/>').text(name+': ')).appendTo($('#messagesDiv'));
25
        $('#messagesDiv')[0].scrollTop = $('#messagesDiv')[0].scrollHeight;
26
      };
27
    </script>
28
  </body>
29
</html>

R: Setup a grid search for xgboost (!!)

I find this code super useful because R’s implementation of xgboost (and to my knowledge Python’s) otherwise lacks support for a grid search:

# set up the cross-validated hyper-parameter search
xgb_grid_1 = expand.grid(
nrounds = 1000,
eta = c(0.01, 0.001, 0.0001),
max_depth = c(2, 4, 6, 8, 10),
gamma = 1
)

# pack the training control parameters
xgb_trcontrol_1 = trainControl(
method = "cv",
number = 5,
verboseIter = TRUE,
returnData = FALSE,
returnResamp = "all",                                                        # save losses across all models
classProbs = TRUE,                                                           # set to TRUE for AUC to be computed
summaryFunction = twoClassSummary,
allowParallel = TRUE
)

# train the model for each parameter combination in the grid,
#   using CV to evaluate
xgb_train_1 = train(
x = as.matrix(df_train %>%
select(-SeriousDlqin2yrs)),
y = as.factor(df_train$SeriousDlqin2yrs),
trControl = xgb_trcontrol_1,
tuneGrid = xgb_grid_1,
method = "xgbTree"
)

# scatter plot of the AUC against max_depth and eta
ggplot(xgb_train_1$results, aes(x = as.factor(eta), y = max_depth, size = ROC, color = ROC)) +
geom_point() +
theme_bw() +
scale_size_continuous(guide = "none")</code>

Rlogo

T-SQL: Show running queries

SELECT sqltext.TEXT,
req.session_id,
req.status,
req.command,
req.cpu_time,
req.total_elapsed_time
FROM sys.dm_exec_requests req
CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS sqltext;
— Alternate approach
SELECT s.session_id
,r.
STATUS
,r.blocking_session_id 
‘blocked by’
,r.wait_type
,wait_resource
,r.wait_time / (1000.0) 
‘Wait Time (in Sec)’
,r.cpu_time
,r.logical_reads
,r.reads
,r.writes
,r.total_elapsed_time / (1000.0) 
‘Elapsed Time (in Sec)’
,Substring(st.TEXT, (r.statement_start_offset / 2) + 1
, (
(
CASE r.statement_end_offset
WHEN – 1
THEN Datalength(st.TEXT
)
ELSE r.statement_end_offset
END – r.statement_start_offset
) / 2
) + 1) AS statement_text
,Coalesce(Quotename(Db_name(st.dbid)) + N’.’ + Quotename(Object_schema_name(st.objectid, st.dbid)) + N’.’ +
Quotename(Object_name(st.objectid, st.dbid)), ”) AS command_text
,r.command
,s.login_name
,s.
host_name
,s.
program_name
,s.host_process_id
,s.last_request_end_time
,s.login_time
,r.open_transaction_count
FROM sys.dm_exec_sessions AS s
INNER JOIN sys.dm_exec_requests AS r ON r.session_id = s.session_id
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) AS st
WHERE r.session_id != 
@@SPID
ORDER BY r.cpu_time 
DESC
,r.
STATUS
,r.blocking_session_id
,s.session_id

Performance Benchmark — Storm v Spark v Flink (Re-post)

https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at

T-SQL: Select / list all tables of a database

USE YourDBName
GO
SELECT *
FROM sys.Tables
Go
 microsoft-sql-00

SQL Server: Delete all tables in a database (T-SQL)

USE MyDB
GO
EXEC sp_MSforeachtable @command1 = “DROP TABLE ?”
GO
microsoft-sql-00

Mean and Multi-modal Mean Functions (methods) for Java

When it comes to stats Java ain’t no R. Still, we can do anything in one language that we can do in another.

Let’s have a look at some mean functions for Java, to illustrate:

public static double mean(double[] m) {
     double sum = 0;
     for (int i = 0; i < m.length; i++) {
         sum += m[i];
     }
    return sum / m.length;}
For multi-modal:
public static List<Integer> mode(final int[] a) {
     final List<Integer> modes = new ArrayList<Integer>();
     final Map<Integer, Integer> countMap = new HashMap<Integer, Integer>();

     int max = -1;

     for (final int n : numbers) {
         int count = 0;

         if (countMap.containsKey(n)) {
             count = countMap.get(n) + 1;
         } else {
             count = 1;
         }

         countMap.put(n, count);

         if (count > max) {
             max = count;
         }
     }

     for (final Map.Entry<Integer, Integer> tuple : countMap.entrySet()) {
         if (tuple.getValue() == max) {
             modes.add(tuple.getKey());
         }
     }

     return modes;}

Java

Read int from stdin all in just 1 Line (Java 6 or higher)

int n = Integer.parseInt(System.console().readLine());
Note that that under most IDEs System.console will return NULL when the application is invoked via a launcher making it hard to debug.