Archive for the ‘Python’ Category

Example XGboost Grid Search in Python

import sys
import math
import numpy as np
from sklearn.grid_search import GridSearchCV
sys.path.append('xgboost/wrapper/')import xgboost as xgb
class XGBoostClassifier():
    def __init__(self, num_boost_round=10, **params):
        self.clf = None
        self.num_boost_round = num_boost_round
        self.params = params
        self.params.update({'objective': 'multi:softprob'})
    def fit(self, X, y, num_boost_round=None):
        num_boost_round = num_boost_round or self.num_boost_round
        self.label2num = dict((label, i) for i, label in enumerate(sorted(set(y))))
        dtrain = xgb.DMatrix(X, label=[self.label2num[label] for label in y])
        self.clf = xgb.train(params=self.params, dtrain=dtrain, num_boost_round=num_boost_round)
    def predict(self, X):
        num2label = dict((i, label)for label, i in self.label2num.items())
        Y = self.predict_proba(X)
        y = np.argmax(Y, axis=1)
        return np.array([num2label[i] for i in y])
    def predict_proba(self, X):
        dtest = xgb.DMatrix(X)
        return self.clf.predict(dtest)
    def score(self, X, y):
        Y = self.predict_proba(X)
        return 1 / logloss(y, Y)
    def get_params(self, deep=True):
        return self.params
    def set_params(self, **params):
        if 'num_boost_round' in params:
            self.num_boost_round = params.pop('num_boost_round')
        if 'objective' in params:
            del params['objective']
        return self
def logloss(y_true, Y_pred):
    label2num = dict((name, i) for i, name in enumerate(sorted(set(y_true))))
    return -1 * sum(math.log(y[label2num[label]]) if y[label2num[label]] > 0 else -np.inf for y, label in zip(Y_pred, y_true)) / len(Y_pred)

def main():
    clf = XGBoostClassifier(
        eval_metric = 'auc',
        num_class = 2,
        nthread = 4,
        eta = 0.1,
        num_boost_round = 80,
        max_depth = 12,
        subsample = 0.5,
        colsample_bytree = 1.0,
        silent = 1,
    parameters = {
        'num_boost_round': [100, 250, 500],
        'eta': [0.05, 0.1, 0.3],
        'max_depth': [6, 9, 12],
        'subsample': [0.9, 1.0],
        'colsample_bytree': [0.9, 1.0],
    clf = GridSearchCV(clf, parameters, n_jobs=1, cv=2)[[1,2], [3,4], [2,1], [4,3], [1,0], [4,5]], ['a', 'b', 'a', 'b', 'a', 'b'])
    best_parameters, score, _ = max(clf.grid_scores_, key=lambda x: x[1])
    for param_name in sorted(best_parameters.keys()):
        print("%s: %r" % (param_name, best_parameters[param_name]))

if __name__ == '__main__':

Raspberry Pi #antisec LED Alert Script

Just a little Python script I wrote to make an LED blink on a Raspberry Pi and to print a message to the screen when there’s a #antisec tweet:

# Jason D. Miller

from twython import TwythonStreamer
import RPi.GPIO as GPIO
import time

C_KEY =  ""
A_TOKEN = "-"


def blink():
    GPIO.output(18, GPIO.HIGH)
    GPIO.output(18, GPIO.LOW)

class MyStreamer(TwythonStreamer):
     def on_success(self, data):
         if 'text' in data:
              print("Antisec Tweet detected. Call the FBI.")

stream = MyStreamer(C_KEY, C_SECRET, A_TOKEN, A_SECRET)


PyDev (Eclipse) key bindings

press CTRL+ALT+ENTER to send the selected lines to the interactive console
EditRectangular editionShift+Alt+AEditing Text
EditFind NextCtrl+KEditing Text
EditFind and ReplaceCtrl+FIn Windows
EditWord CompletionAlt+/Editing Text
FileCloseCtrl+F4In Windows
FileNewCtrl+NIn Windows
FileSaveCtrl+SIn Windows
NavigateBackward HistoryAlt+LeftIn Windows
NavigateForward HistoryAlt+RightIn Windows
NavigateGo to LineCtrl+LEditing Text
NavigateLast Edit LocationCtrl+QIn Windows
NavigateOpen ResourceCtrl+Shift+RIn Windows
NavigateShow In menuAlt+Shift+WIn Windows
NavigateGo to next problem markerCtrl+.In Windows
PyDev – EditorQuick Fix / Content AssistantsCtrl+1PyDev editor scope
PyDev – EditorCode CompletionCtrl+SpacePyDev editor scope
PyDev – EditorAdd Comment BlockCtrl+4PyDev editor scope
PyDev – EditorAdd Single Comment BlockCtrl+Shift+4PyDev editor scope
PyDev – EditorConvert space-tabs to tabsCtrl+Shift+TabPyDev editor scope
PyDev – EditorConvert tabs to space-tabsCtrl+TabPyDev editor scope
PyDev – EditorNext Method or ClassCtrl+Shift+DownPyDev editor scope
PyDev – EditorPrevious Method or ClassCtrl+Shift+UpPyDev editor scope
PyDev – EditorPython CollapseCtrl+-PyDev editor scope
PyDev – EditorPython Collapse AllCtrl+9PyDev editor scope
PyDev – EditorToggle Mark OccurrencesAlt+Shift+OPyDev editor scope
PyDev – EditorPython Format CodeCtrl+Shift+FPyDev editor scope
PyDev – EditorPython Go To DefinitionF3PyDev editor scope
PyDev – EditorPython Organize ImportsCtrl+Shift+OPyDev editor scope
PyDev – EditorPython Show Quick OutlineCtrl+OPyDev editor scope
PyDev – EditorPython Open Definition Quick OutlineCtrl+Shift+TPyDev editor scope
PyDev – EditorPython Un CollapseCtrl+=PyDev editor scope
PyDev – EditorPython Un Collapse AllCtrl+0PyDev editor scope
PyDev – EditorPython Comment/UncommentCtrl+/ (for both)PyDev editor scope
PyDev – EditorRemove Comment BlockCtrl+5PyDev editor scope
PyDev – EditorStart interactive sessionCtrl+Alt+EnterPyDev editor scope
PyDev – EditorSend line to interactive sessionF2PyDev editor scope
PyDev – EditorList commands binded to Ctrl+2 (only the most important are below)Ctrl+2+helpPyDev editor scope
PyDev – EditorAssigns method parameters to attributes of selfCtrl+2+aPyDev editor scope
PyDev – EditorForces code analysis in the current editorCtrl+2+cPyDev editor scope
PyDev – EditorKills all the python shells spawned by PyDevCtrl+2+killPyDev editor scope
PyDev – EditorRenames local occurrencesCtrl+2+rPyDev editor scope
PyDev – EditorWraps the current selected paragraphCtrl+2+wPyDev editor scope
PyDev – EditorEnables/Disables wrapping in the editorCtrl+2+setwrapPyDev editor scope
PyDev – EditorPasses the passed command to in DjangoCtrl+2+dj commandPyDev editor scope
PyDev – EditorPuts a new line after each comma (split lines)Ctrl+2+slPyDev editor scope
PyDev – EditorTransforms an import into a string (import string)Ctrl+2+isPyDev editor scope
PyDev – EditorSearches the passed (or selected) string in the open editorsCtrl+2+s string_to_findPyDev editor scope
PyDev – SearchFind referencesCtrl+Shift+GPyDev editor scope
PyDev – RefactorPython Extract Local VariableAlt+Shift+LPyDev editor scope
PyDev – RefactorPython Extract MethodAlt+Shift+MPyDev editor scope
PyDev – RefactorPython Inline Local VariableAlt+Shift+IPyDev editor scope
PyDev – RefactorPython RenameAlt+Shift+RPyDev editor scope
Run/DebugRun current editorF9PyDev editor scope
Run/DebugRun current editor unit-testsCtrl+F9PyDev editor scope
Run/DebugDebug Last LaunchedF11In Windows
Run/DebugRun Last LaunchedCtrl+F11In Windows
Run/DebugStep IntoF5Debugging
Run/DebugStep OverF6Debugging
Run/DebugStep ReturnF7Debugging
SearchOpen Search DialogCtrl+HIn Windows
Text EditingCopy LinesCtrl+Alt+DownEditing Text
Text EditingDelete LineCtrl+DEditing Text
Text EditingDelete Next WordCtrl+DeleteEditing Text
Text EditingDuplicate LinesCtrl+Alt+UpEditing Text
Text EditingInsert Line Below Current LineShift+EnterEditing Text
Text EditingMove Lines DownAlt+DownEditing Text
Text EditingMove Lines UpAlt+UpEditing Text
Text EditingScroll Line DownCtrl+DownEditing Text
Text EditingScroll Line UpCtrl+UpEditing Text
Text EditingTo Lower CaseCtrl+Shift+YEditing Text
Text EditingTo Upper CaseCtrl+Shift+XEditing Text
WindowActivate EditorF12In Windows
WindowMaximize Active View or EditorCtrl+MIn Windows
WindowNext EditorCtrl+F6 (LiClipse adds Ctrl+Tab too)In Windows
WindowNext PerspectiveCtrl+F8In Windows
WindowNext ViewCtrl+F7In Windows
WindowOpen Editor Drop DownCtrl+EIn Windows
WindowShow Key AssistCtrl+Shift+LIn Dialogs and Windows
WindowShow Ruler Context MenuCtrl+F10Editing Text
WindowShow System MenuAlt+-In Windows
WindowShow View MenuCtrl+F10In Windows
WindowSwitch to EditorCtrl+Shift+EIn Windows

Python: Install easy_install for Python 3.4 under Windows 7 in 2 Steps

Sometimes your Python installation is a little more bare bones than it should be. To initialize easy_install do the following:

1. Download and run this file:

2. Specify easy_install commands with this syntax:

C:\Python34\python.exe -m easy_install packagename

Python: Print the (Current) Working Directory

It takes 2 lines to do this in Python, which seems excessive to me, but what are you going to do?

>>> import os
>>> print(os.getcwd())

Python: How to Install and Update pip on Ubuntu

Surprisingly, pip doesn’t come pre-installed on Python 2 or 3 in Ubuntu (as of v 14.04). This could be untrue in some flavors of the distro, but mine is pretty inclusive and yet no pip.

To install it:

$ sudo apt-get install python-pip python-dev build-essential
$ sudo pip install –upgrade pip
$ sudo pip install –upgrade virtualenv


Python: Get the Directory of a Given File in 1 Line



Python: Notes on Indentation

In Python the interpreter identifies blocks of code based on indentation.

By way of example, in an if-loop if the next line is indented then it means that this indented block is within the loop. You can use either spaces or tabs, e.g. you can not mix both, to quote PEP 8 – Style Guide for Python Code:

Python 3 disallows mixing the use of tabs and spaces for indentation.
Python 2 code indented with a mixture of tabs and spaces should be converted to using spaces exclusively.

The standard method is to use 4 spaces for indentation, which also happens to be the necessary number of spaces to designate a codeblock on Stackoverflow.

Fortunately, if you’re using the PyDev plugin for Eclipse or many other popular IDE’s, then you can press tab in the text editor and the IDE will automatically translate that to 4 spaces.


CPU Registers: an Overview

Register: a small amount of storage on the CPU; the fastest method for a CPU to access data

In the x86 instruction set, a CPU uses 8 general-purpose registers:
  1. EAX — a.k.a. the accumulator register, used for performing calculations as well as storing return values
  2. EDX — a.k.a. the data register, basically an extension of EAX
  3. ECX — a.k.a. the count register, used for looping, counts DOWNWARD not upward
  4. ESI  — a.k.a. the source index, used for reading, holds the location of the input data stream
  5. EDI  — a.k.a. the destination index, used for writing, points to the location where the result is stored
  6. EBP  — a.k.a. the base pointer, used for managing function calls and stack operations, points to the bottom of the stack unless freed up from this function by the compiler, in which case it would be an extra general purpose register
  7. ESP  — a.k.a. the stack pointer, used for managing function calls and stack operations, points to the very top of the stack
  8. EBX  — an extra register, not designed for anything specific

    Another register worth mentioning is:
  9. EIP — a.k.a. the instruction pointer, points to the current instruction that is being executed; as binary code is being executed by the CPU the EIP is updated to reflect the location where the execution is occurring

White & Black box Debuggers, Intelligent Debugging, and Dynamic Analysis

Debugging is a common task for data scientists, programmers, and security experts alike. In good ole RStudio we have a nice, simple built-in white-box debugger. For many analysis-oriented coders, the basic debugging functionality of an IDE like RStudio is all they know and it may be a surprise that debugging is a bigger, much sexier, topic. Below I define and describe key topics in debugging and dynamic analysis, as well as provide links to the most cutting edge free debuggers I use.

Dynamic Analysis: Runtime tracing of a process, usually performed using a debugger. Dynamic Analysis is critical for exploit development, fuzzer assistance, and malware inspection.

Debugger: a program that is used to test and troubleshoot other programs.Intelligent Debugger: a scriptable debugger that supports extended features such as call hooking, such as Immunity Debugger and PyDbg.

White Box Debugger: Debuggers built into IDEs and other dev platforms, which enable developers to trace through source code with a high degree of control, as to aide in the troubleshooting of functions and other code breakages.
Black Box Debugger: Used by bug hunters and reverse engineers, black box debuggers operate on compiled programs when the source code is not available and the only information is available in a disassembled format. There are two broad subclasses of black box debuggers, which are user mode (i.e. ring 3) and kernel mode (i.e. ring 0).
User mode black box debugger: a processor mode under which your applications run, usually with the least amount of privilege (e.g. double clicking PuTTY.exe launches a user-mode process).
Kernel mode black box debugger: a processor mode where the core of the OS runs using the highest amount of privilege (e.g. capturing packets with a network adapter that is in passive mode).
User-mode Debuggers Commonly used among Reverse Engineers
WinDbg by Microsoft
OllyDbg by Oleh Yuschuk, a F.O.S.S. debugger
GNU Debugger (gdb), a F.O.S.S. Linux debugger by the community