XING is the european pendant to LinkedIn as social network for professionals, so there are bunch of people, who are using these two networks in order to maintain their connections ( datenprodukt.dakoller.net might provide a helper for this challenge ) . Luckily for me, XING started with an API in closed beta state (signup at https://dev.xing.com/).

My setup is a Django-based web application hosted on Heroku, and I am using the last stable version 1.4.1.

This blog is about what I needed to integrate with the app, how I adapted the connection and how I can use it now.

Django-social-auth is  well documented and has already today a broad list of other web applications, with which you can connect quite fast. The really big advantage in terms of software architecture is, that it uses the already good user administration & auth handling, which is already inbuilt in Django and provides a complete abstraction from the provider specific settings, when you are inside your app.

You can now plugin any kind of oAuth enabled web application, without specific auth handling in the application. (However you still have to do the specific API calls after you got the access tokens.)

The needed packages are:

Django==1.4.1
django-social-auth==0.7.6
httplib2==0.7.6
oauth2==1.5.211

Dependencies of these packages are installed using the pip-installer system in Python.

The main piece is a custom module, which talks the oAuth process needed by XING. The code is also available on GitHub.

"""
XING OAuth support

No extra configurations are needed to make this work.
"""
from xml.etree import ElementTree
from xml.parsers.expat import ExpatError

from oauth2 import Token
import oauth2 as oauth

from social_auth.utils import setting
from social_auth.backends import ConsumerBasedOAuth, OAuthBackend, USERNAME
from social_auth.backends.exceptions import AuthCanceled, AuthUnknownError

from pprint import pprint

import settings
import simplejson as json

XING_SERVER = 'xing.com'
XING_REQUEST_TOKEN_URL = 'https://api.%s/v1/request_token' % \
                                    XING_SERVER
XING_ACCESS_TOKEN_URL = 'https://api.%s/v1/access_token' % \
                                    XING_SERVER
XING_AUTHORIZATION_URL = 'https://www.%s/v1/authorize' % \
                                    XING_SERVER
#XING_CHECK_AUTH = 'https://api.%s/v1/users/me.json' % XING_SERVER
XING_CHECK_AUTH = 'https://api.%s/v1/users/me.xml' % XING_SERVER

class XingBackend(OAuthBackend):
    """Xing OAuth authentication backend"""
    name = 'xing'
    EXTRA_DATA = [('id', 'id'),('user_id','user_id')]

    def get_user_details(self, response):
        """Return user details from Xing account"""
        pprint(response)
        first_name, last_name = response['first_name'], response['last_name']
        #first_name='Alfred E.'
        #last_name='Neumann'
        email = response.get('email', '')
        return {USERNAME: first_name + last_name,
                'fullname': first_name + ' ' + last_name,
                'first_name': first_name,
                'last_name': last_name,
                'email': email}

class XingAuth(ConsumerBasedOAuth):
    """Xing OAuth authentication mechanism"""
    AUTHORIZATION_URL = XING_AUTHORIZATION_URL
    REQUEST_TOKEN_URL = XING_REQUEST_TOKEN_URL
    ACCESS_TOKEN_URL = XING_ACCESS_TOKEN_URL
    SERVER_URL = 'api.%s' % XING_SERVER
    AUTH_BACKEND = XingBackend
    SETTINGS_KEY_NAME = 'XING_CONSUMER_KEY'
    SETTINGS_SECRET_NAME = 'XING_CONSUMER_SECRET'
    SCOPE_VAR_NAME=None
    SCOPE_SEPARATOR = '+'

    def user_data(self, access_token, *args, **kwargs):
        """Return user data provided"""
        url = XING_CHECK_AUTH

        consumer = oauth.Consumer(key=settings.XING_CONSUMER_KEY, secret=settings.XING_CONSUMER_SECRET)
        client= oauth.Client(consumer,access_token)

        resp, content = client.request('https://%s%s' % ('api.xing.com','/v1/users/me.json'), "GET")
        profile= json.loads(content)['users'][0]
        #pprint(profile)

        try:
            #return to_dict(ElementTree.fromstring(raw_xml))
            return {'user_id':profile['id'],'id':profile['id'],'first_name': profile['first_name'],'last_name': profile['last_name'],'email': profile['active_email']}
        except (ExpatError, KeyError, IndexError):
            return None

    def auth_complete(self, *args, **kwargs):
        """Complete auth process. Check Xing error response."""
        oauth_problem = self.request.GET.get('oauth_problem')
        if oauth_problem:
            if oauth_problem == 'user_refused':
                raise AuthCanceled(self, '')
            else:
                raise AuthUnknownError(self, 'Xing error was %s' %
                                                    oauth_problem)
        return super(XingAuth, self).auth_complete(*args, **kwargs)

    def get_scope(self):
        """Return list with needed access scope"""
        scope = []
        if self.SCOPE_VAR_NAME:
            scope = setting(self.SCOPE_VAR_NAME, [])
        else:
            scope = []
        return scope

    def unauthorized_token(self):
        """Makes first request to oauth. Returns an unauthorized Token."""
        request_token_url = self.REQUEST_TOKEN_URL
        scope = self.get_scope()
        if scope:
            qs = 'scope=' + self.SCOPE_SEPARATOR.join(scope)
            request_token_url = request_token_url + '?' + qs

        request = self.oauth_request(
            token=None,
            url=request_token_url,
            extra_params=self.request_token_extra_arguments()
        )
        response = self.fetch_response(request)
        return Token.from_string(response)

def to_dict(xml):
    """Convert XML structure to dict recursively, repeated keys entries
    are returned as in list containers."""
    children = xml.getchildren()
    if not children:
        return xml.text
    else:
        out = {}
        for node in xml.getchildren():
            if node.tag in out:
                if not isinstance(out[node.tag], list):
                    out[node.tag] = [out[node.tag]]
                out[node.tag].append(to_dict(node))
            else:
                out[node.tag] = to_dict(node)
        return out

# Backend definition
BACKENDS = {
    'xing': XingAuth,
}

Integration requires to put the oAuth consumer tokens into settings.py:

# get these data from dev.xing.com
XING_CONSUMER_KEY = "xxx"
XING_CONSUMER_SECRET = "yyy"
XING_BASE_URL='https://api.xing.com'

Additionally you need to tell django-social-auth, that you want to use an additional auth module (for Xing) in your application.

AUTHENTICATION_BACKENDS = (
    'nwg.xing.XingBackend',
    'django.contrib.auth.backends.ModelBackend',
)

The effect of this integration is now as follows:

• people can login to the application with ‘/login/xing’,
• existing users of the application can associate their Xing accoutn by calling /associate/xing and
• the application get the oAuth access tokens (for accessing data & initiating actions) by querying the UserSocialAuth objects.

Overall integration dajngo-social-auth with Xing went quite smooth, special kudos goes to the creator of the package, Matias Aguirre, for the effort to setup the package and for the support in my special case.

RapidSlides creates 80% ready business presentations based on visual input. The input comes from photos of whiteboards / flip charts or hand-drawn sketches from your iPad. We use your layout or create one based on your website and deliver our results back to in max. 24  hours.

You want to see how it looks like? Just head over to http://rapidslides.com/de/beispiele , where you can see first examples from customer orders.

You can also participate in the currently running test phase: just sign up at http://rapidslides.com/ .

We (that means  Ralf Westbrock from http://str84wd.com/ and me ) are operating from Munich Germany and follow the Lean Startup methodolody.

My role there is mainly the technical part and i’ll blog here about my lessons learned of the project. Do you have already specific aspects in which you are interested? Just comment here or reach out on Twitter at @dakoller .

1. Data Science Day was a great event … so the next upcoming one is worth a visit too. The focus topic will be Game Analytics.

I heard rumors about an idea to got to Munich for one of the next Data Science Days… do you have any feedback ? Any special focus topics you think of?

You might have seen the nice pic on Facebook, which said
“Please fix my phone or I will tell my X-thousands of followers about your bad performance’”.

This sounds like a joke at the moment, but in terms of customer relationship management you might fear the bad sentiment about your company, which is conveyed to a (potentially) very large group of people by messages like this one. You also might have heard already about the very positive impact on your brand, when you react on customer feedback in social online media in a very constructive, friendly & timely manner.

So not only in case of customer service tasks, but also for customer acquisition tasks you are interested in making influential people to talk about your product or service.

But how would the company know who is important/influential on social networks and in which peer groups their voice is heard best? There is a solution for that!

So the idea is:

Value Proposition:
Enable companies to put  special attention on influential people on social networks (among their customer & prospects).

This enables them to e.g.

• distribute limited special goodies, (such as  beta invites, coupons etc.) to the people influential in their domain,
• prioritize them in customer service applications or
• approach them for feedback on your service.

Solution:
Match their customer information with data sources on influential people, e.g. based on the services of  Klout or PeerIndex.

(yes: there is sometimes bad sentiment about services like Klout, as they reduce a persons relevance to a single number – and they don’t tell you in detail, how they are doing it. This bad sentiment goes as far as motivating people to setup fake services such as flout.com. I think these services learned from this feedback, e.g. to tell you also on which topics someone is influential.)

Many of these service, including the two mentioned ones, make their information available as APIs, so one could link this information with customer information.

This could go as far as: “Distribute 100 books coupons to the most important people in the area of ‘Rosamunde Pilcher’ literature in my customer list.”

Secret sauce: (important to not get immediately a victim of other people copying your service)

Companies wouldn’t want to do this kind of data wrangling from APIs and the merging with other data sources on their own. They likely also don’t want to invest the infrastructure, which is needed the keep this data current and to hint the company to relevant changes over time.

However, there is a caveat: you need a good story (or a list of precautions to show the companies, that you work with their data in a secure way)

(Ein gutes Zeichen ist auch, daß mittlerweile auch deutsche Unternehmen die Relevanz offensichtlich verstanden haben, was man daran sehen kann, dass sie Mitarbeiter dort teilnehmen liessen und das Event auch gesponsert haben, z. B. @Immobilienscout, @Wooga)

Bedeutend ist, daß wir letzte Woche an einem Platz 70 Gäste da hatten, die den ziemlich breiten Raum des Begriffs Data Science ziemlich komplett ausgeleuchtet haben. (Vorher hatte ich hier manchmal das Gefühl, in einem sehr exotischen Feld zu arbeiten)

Die Bandbreite der Themen ging von
- Frameworks und Tools (z.B. Hadoop, Twitter Storm),
- Handling von strukturierter und unstrukturierter Information (z.B. bessere Suche, Inhaltsanalyse und Zusammenfassungen von Text mit den interessanten Übergängen ins Semantic Web) über
- die Relevanz von Open Data (als Basis für öffentliche Informationsangebote und aufrüttelnde Unterstützung für Kampagnen () bis hin zu
- Visualisierungen.(z.B. auf Basis von Processing).

Der Vormittag war gefüllt von gesetzten Präsentationen ( Agenda unter http://de.amiando.com/datascienceday.html)  : gut war, daß Klaas die so kurz halten konnte, daß viel Raum für Diskussionen blieb. Am Nachmittag ging es drei parallel laufenden und kleineren Gruppen um
- sehr technische Fragen, (“was macht man mit Twitter Storm?”),
- visionärere Themen (“was kann man noch mit XYZ machen?”) und
- eine Gruppe zum Thema Texthandling und Open Data-Nutzung.

Aus diesem Tag habe ich folgende Beobachtungen mitgenommen
- die meiste Relevanz hat das Thema Data Science im Moment wohl im Advertising-Markt: da geht es häufig, um sehr schnelle Antworten, die nicht irgendwann zu beantworten sind, sondern genau dann wenn der Webseiten-Besucher auf meiner Seite ist. (Auch wenn es für meinen ‘Traum’ von für mich sehr relevanter, weil auf meine Interessen zugeschnittene, Werbung wohl noch zu früh ist.)
- Kurz danach kommt das Thema von unstrukturierter Information: hier fällt mir auf, daß z.B. im Vergleich zu den USA (wo ich im Februar auf der StrataConf war) das Thema hier wichtiger zu sein scheint.
- Im Moment nutzen hauptsächlich junge und eher kleinere Unternehmen Kompetenz zu dem Thema Big Data/Data Science: in etablierten & größeren Unternehmen, die zwar die Daten hätten (die sich ein Startup ziemlich mühsam sehr beschaffen oder erzeugen muß), ist das Thema wohl noch nicht angekommen.
- Aus der Perspektive eines Startupgründers ist klar, daß aus eigenen Aktionen gewonnene Daten und Analysen essentiell (oder zumindest wichtig) für die Geschäftsentwicklung sind. Ein gutes Beispiel dafür sind die Münchner @10stamps (kurz eine App digitale Stempelkarten), die vermutlich mittlerweile mehr Überblick über meinen Kaffeekonsum als ich selbst haben.
- Das Thema Tools für BigData ist auch noch ein guter Platz, auf dem sich neue Startups ausprobieren können: hier haben für mich Ansätze die Nase vorne, die einem Analysten/Geschäftsentwickler die Tools selbst in die Hand geben, neue Einsichten zu gewinnen. (anstatt für jede neue Fragestellung einen Entwickler suchen zu müssen). Der nächste Entwicklungsschritt sind Tools, die dem Nutzer nicht mehr Zahlenwüsten (z.B. in Powerpoint-Form) vorsetzen, sondern (mutmasslich) neue Einsichten (wie Veränderungen in Mustern & Trends) direkt in die Hand geben.

Zusammengefasst: ein Tag voll von neuem Information und Diskussionen: der nächste Data Science Day findet in 3-4 Monaten statt und wird sich schwerpunktmässig mit dem Thema Analytics in Online-Games befassen.

viaThe R-Podcast Episode 6: Importing Data from External Sources. …is an excellent summary of how to get external data into R.

I recently needed to get JSON-formatted data in R (as people argue that you should use JSON for most of the data transaction needs in the web area, even for e.g. log file content like in http://blog.treasure-data.com/post/21881575472/log-everything-as-json-make-your-life-easier ), where I found http://stackoverflow.com/questions/2617600/importing-data-from-a-json-file-into-r handy.

Machine learning for identification of cars.

Semantic Web: Where are the Semantic web incubators? Any thoughts on building an economic ecosystem for Semantic web to keep momentum enough to attract sizable investment? 1 answer on Quora

Where are the Semantic web incubators? Any thoughts on building an economic ecosystem for Semantic web to keep momentum enough to attract sizable investment?

N-Grams datasets can also be created from your own texts using NTLK functions (see http://nltk.googlecode.com/svn/trunk/doc/howto/collocations.html ): in analytical use cases N-grams give you a better basis to have a machine ‘understand’ the meaning of a text (compared to looking at the words individually.

—Update from 2012-04-10:

Stefan Keller ( http://twitter.com/sfkeller  ) hinted me to a blog entry about how to use n-grams in a PostgreSQL based setting to optimize search functionality.

In Projekten zur Datenauswertung ist es häufig relevant, die Ergebnisse zeitnah ansprechend dokumentieren zu können. Ein gute Weg dabei, den Fortgang der Experimente und deren Ergebnisse in einem dynamisch generierten Dokument zu verfolgen: dabei kann knitR helfen.

knitR bügelt einige Schwächen der Sweave-Lösung (http://www.statistik.lmu.de/~leisch/Sweave/ ): insbesondere können gut aussehende Grafiken besser eingebaut werden.  

Weitere Infos zum Tool finden sich unter http://www.inside-r.org/howto/knitr-elegant-flexible-and-fast-dynamic-report-generation-r , eine Beispielausgabe ist unter http://cloud.github.com/downloads/yihui/knitr/Stat615-Report1-Yihui-Xie.pdf zu sehen.