Hooverphonic (Cover) - Mad about you

Hemos grabado una versión cortita de "Mad about you", de Hooverphonic. Éste es el segundo tema que produzco:




Voz: Xiana Teimoy
Instrumentos y Producción: J Montes

Desde mi perfil en SoundCloud se puede descargar el audio y navegar por los temas.

De paso recuerdo el anterior tema que publiqué: J - Nowadays .

Wikipedia Facts

What can we find if we download and generate some statistics about Wikipedia?

1) Overview: Wikipedia on January 2nd 2013 has 13 057 082 entries (the Encyclopaedia Britannica sums 228 274 entries according to Wikipedia itself). There are almost as many redirections as actual articles:

Wikipedia Articles (blue: articles, green: redirections)

2) Articles: Let's look at the real articles contents only (no redirections). This is more than 7,1 million articles:

The average article belongs to 2.6 categories, links to 2.31 pages outside the site and 33.86 links internal to Wikipedia. Thousands of articles feature thousands links. In total, there are 16 413 888 external links and 240 751 315 internal links. Enough to get lost for a while!

The average size is 4 634 characters (roughly about 110 words per article), but the total size of article text is 32 GB. And this is only raw text, images are not included.

3) Content

This is one of the most striking result of all. I have searched for certain words within the text of the articles, and assigning a score. The follownig diagram shows how many articles are defined by a particular word (the word with most occurrences). This has been shown before, but the results still seem astonishing to me:

Perhaps we should start thinking of how airily we use the term "war".


4) Geography

I can only report for the articles last updated by anonymous users. But for the sake of it, this is how real article updates (by anonymous users) were distributed among the different continents. This population includes 490 080 articles:

 

5) Updates:  This is a result I am pretty surprised of. The following graph shows the year and quarter was the time that articles were last updated (separating redirections, in yellow, from articles, in blue). Apparently, a huge percentage of articles have been updated during the last quarter of 2012, which could mean that Wikipedia is very lively and is being updated frequently, although this value seems to high to me, and so I wonder if this may be some automatic process updating wikipedia articles.

Unfortunately, I can't get the "creation date" of articles as the normal Wikipedia dump doesn't include that information.

6) Titles

The average title entry is 26.9 characters long. There are entries starting with every character you can think of: €, Ɣ, ¢, £, § ... We can also see how entries are distributed along letters. Surprisingly, numbers 1 and 2 have got more entries than letters Q, X, Y or Z. Even also more than U and V.

7) Processing

I have done the analysis and charts with CubesViewer OLAP Data Explorer, an open source data explorer that I published a couple of weeks ago.

Processing the Wikipedia export file (a 39,6 GB XML file) took my computer more than 18 hours, although using 1 core only and I didn't spend any time optimizing this.

source: 8,95GB 18:35:54 [ 140kB/s]
bzcat: 39,6GB 18:35:57 [ 621kB/s] 
articles: 13,1M 18:35:57 [ 195/s]    

New open source application: CubesViewer

I have recently been working on a data exploration and visualization tool, and I am very happy to announce the release of this new project to the public domain.

It is called CubesViewer, and it is an Online Analytical Processing (OLAP) exploration tool. In everyday words, it allows people to design and produce reports and charts about many kinds of data that can be extracted from a database (like contracts, invoices, climate, demography, scientific production, wikipedia articles, public spending, logistics...).

I wanted to use a simple Online Analytical Processing (OLAP) server and I found the Cubes project, a fantastic lightweight OLAP server which includes everything I needed.

It's always nice to publish Open Source software. I hope this is of use to people.

Features:

• User Interface allowing for multiple views on-screen.
• Cube explorer providing drilldown and cut operations.
• Supports dimension hierarchies and date filtering.
• Different types of charts and diagrams.
• View management, sharing and saving.
• Modular and extensible.

Cubesviewer Project: https://github.com/jjmontesl/cubesviewer

"Información" sobre la "censura" de Youtube

Hoy Youtube! me ha escrito. Dicen que uno de mis vídeos quizá contenga contenido "inapropiado". De hecho, ha sido prohibido en Alemania.

Parece que alguien o algo (supongo que más bien algún programa) ha detectado que de fondo una canción comercial.

No entiendo cómo esto puede ser considerado un agravio :(, está totalmente fuera de lugar, como cualquiera puede juzgar por sí mismo viendo el vídeo: "Levitron".

Menos mal que, como me dice Youtube!, gracias al dios esto no penaliza mi cuenta. Cualquier día subo un vídeo y por sabe Dios qué motivo me quedo sin la cuenta de Youtube, y por qué no, sin la de Google, y ya de paso sin acceso a mi propio teléfono. Parece imposible, pero hay a quien Amazon le ha condenado sin acceso a sus libros... 

Reproduzco el texto del email de Youtube (en la página de información sobre copyright dan algún dato más):

Your video "Levitron", may have content that is owned or licensed by UMG, but it’s still available on YouTube! In some cases, it may be blocked, or ads may appear next to it.

This claim is not penalizing your account status. Visit your Copyright Notice page for more details on the policy applied to your video.

Nowadays

I play guitar (self-taught). A few months ago I wrote this song... I later got myself a sound card and tried to record something. This is what I came up with.

Edit: A version with better audio quality and download link can be found at  http://soundcloud.com/jjmontes.

La conjura de los necios

"La conjura de los necios" trata de las peripecias de Ignatius J. Reilly, un ser tan inadaptado e inmaduro como cultivado. El protagonista, mordaz y ultra crítico con la sociedad en la que vive (la Nueva Orleans de mediados del siglo pasado), tiene un discurso irónico y pedante capaz de arrancar carcajadas. Pero lo cierto es que Ignatius es todo lo que ningún hombre querría ser. Los personajes son de auténtica locura.

Esta obra es genial. Se regodea en multitud de aspectos que en España nos resultan más cercanos: los treintaañeros que no se van de casa, la sociedad y tribus urbanas, la sexualidad, el racismo... Algunos pasajes hasta asustan, por ser tan actuales y tan transgresores, a pesar de haber sido escritos hace 50 años. O quizá esto sea un indicio de que los yankis empezaron a degenerar mucho antes que nosotros. Teología y geometría, es la respuesta de Ignatius para los males del mundo. No digo más.

"La conjura de los necios" ("A confederacy of dunces") es la obra maestra de John Kennedy Toole, un profesor de inglés de Nueva Orleans. Escrita en 1962, es una novela controvertida ya desde antes de su publicación. Su autor,  se suicidó a los 31 años tras no conseguir publicarla (siempre he pensado que un buen suicidio a los treinta es una forma genial de potenciar un best-seller). Su madre consiguió publicar la novela mucho después, en 1980. Poco más tarde, Toole recibía un Pulitzer póstumo por ella. Inquietantemente, las similitudes entre la biografía del autor y algunos de sus personajes no son desdeñables.

- Ignatius, contrólate.
- Me niego a "mirar hacia arriba": El optimismo me da náuseas. Es perverso. La posición propia del hombre en el universo, desde la Caída, ha sido la de la miseria y el dolor.
- Yo no me siento mísera.
- Lo eres.
- No, no lo soy.
- Sí, lo eres.
- No lo soy Ignatius. No me siento triste, si me sintiese triste, te lo diría.
- Si yo hubiera demolido propiedad privada en estado de embriaguez y con ello hubiera arrojado a mi hijo a los lobos, estaría dándome golpes de pecho y gimiendo. Estaría arrodillada hasta que me sangradan las rodillas, como penitencia. Por cierto, ¿qué penitencia te puso el sacerdote por tu pecado?
- Tres avemarías y un padrenuestro.
- ¿Nada más? -aulló Ignatius-. ¿Le explicaste lo que hiciste, que interrumpiste una obra crítica de gran importancia?
- Fui a confesarme, Ignatius. Se lo expliqué todo al padre. Y él me dice "No me parece culpa suya, querida. Creo que lo único que pasó fue que el coche patinó un poquito porque la calle estaba mojada". Así que le expliqué lo tuyo. Le dije "Mi hijo dice que soy la que le impide escribir en sus cuadernos. Lleva casi cinco años escribiendo esa historia". Y el padre va y dice "¿Sí? Bueno no me parece tan importante. Dígale que salga de casa y vaya a trabajar".
- Cómo voy a apoyar yo a la iglesia moderna, es imposible -exclamó Ignatius-. Deberían haberte azotado allí mismo, en el confesionario.

• La conjura de los necios
• John Kennedy Toole

(Gracias Pat por dejármela, no te equivocabas :).

A look into various REST APIs

I have recently been working on a web application for data mining. We were initially using a typical stack of server Java technologies. The web MVC part was written on Struts 2.

Our application is pretty UI intensive and we wanted it to have a nice, modern look and feel. As development went on, we soon found ourselves issuing a good amount of AJAX calls to the server. Without careful planning, this can soon become an unmanageable mixture of HTML, JSON and XML response formats, and dozens or hundreds of different GET and POST requests that expect various parameters.

Our architect soon tackled the issue and we decided to define a full featured REST API for the application (incidentally,  we moved to Spring MVC which we found nicer), which we then access from the client side.

REST methods

We have since defined dozens of URLs that handle our application requests. Much has been said about how REST method signatures should look: should resource names be used in singular or plural [1]? is it mandatory to use PUT and DELETE methods and do they need to be idempotent [2]? how do list/search methods look like [3]? Not to forget that there are also some well known implementations that provide default methods for most operations.

Besides, REST is not a protocol, but an application architecture style [4]. Without an strict definition, nowadays there are many REST principles that are sistematically violated in real-world applications: not many REST APIs use hyperlinks as their only method reference (rather, they publish API documentation and expect programmers to directly access their published methods). Not all services use all four HTTP methods (GET, POST, PUT, DELETE) as defined by HTTP. Not all of them have consistent naming rules for resources.

Seeing how loose REST is about API aspect, I have collected a few examples from some of the most famouse services on the Internet, so we can have an oversight of how famous REST APIs look as of today.

Facebook (Graph API)

Facebook publish resources as a graph of social objects [5]. Every object in the social graph has a unique ID. You can access the properties of an object by requesting:

https://graph.facebook.com/ID

For example, the official page for the Facebook Platform has id 19292868552, so you can fetch the object at: https://graph.facebook.com/19292868552.

Responses are JSON:

{
   "name": "Facebook Platform",
   "type": "page",
   "website": "http://developers.facebook.com",
   ...
   "id": 19292868552,
   "category": "Technology"
}

Relationships between objects are called connections and are accessed using the name of the connection:

https://graph.facebook.com/ID/CONNECTION_TYPE
https://graph.facebook.com/me/likes?access_token=...
https://graph.facebook.com/me/friends?access_token=...

And results are also JSON:

{
   "data": [
      {
         "name": "John John",
         "id": "201095685"
      },
      {
         "name": "Andy Andy",
         "id": "202921236"
      },
      ...
   ]
}


You can render the current profile photo for any object by adding the suffix /picture to the object URL. They have consistent support for paging, selecting fields and/or multiple objects in one call, accepting/returning different date formats and fulltext search:

https://graph.facebook.com/bgolub?fields=id,name,picture
https://graph.facebook.com?ids=arjun,vernal
https://graph.facebook.com/search?q=mark&type=user

They also provide support for publishing (via POST) to the appropriate resource, using a few simple parameters. The following URL is used to add a comment to an object:

POST:      https://graph.facebook.com/OBJECT_ID/comments
arguments: message

This API uses only GET and POST. They don't use hyperlinks to reference other objects (for example, lists are composed of object IDs, not URLs).

Twitter

Twitter API [6] is also GET/POST based:

Get user home timeline, users who retweeted a post, followers:

GET statuses/home_timeline
GET statuses/ID/retweeted_by
GET followers/ids

Modifications are done via POST, like the URL used for actually tweeting:

POST statuses/update

These GET and POST methods accept a variety of parameters which are very well documented. Parameters can be used to filter the timespan of the object we want to retrieve, for defining pagination, filtering, whether to include extra metadata about objects returned... Twitter API can also provide a number of response formats: JSON, XML, RSS, Atom.

This is an example of a response list of tweets in JSON format (which is an array of tweet objects):

[
  {
    "coordinates": null,
    "favorited": false,
    "created_at": "Fri Jul 16 16:58:46 +0000 2010",
    "truncated": false,
    "entities": {
      "urls": [
        {
          "expanded_url": null,
          "url": "http://www.flickr.com/photos/cindyli/4799054041/",
          "indices": [
            75,
            123
          ]
        }
      ],
      "hashtags": [
      ],
      "user_mentions": [
    ...

Google Custom Search API

This is a very simple API [7]. You can retrieve results for a particular search by sending an HTTP GET request to its URI. The URI for a search has the following format:

https://www.googleapis.com/customsearch/v1?parameters

Parameters have to include the Google API key, the search query and other optional Google query parameters (response format, pagination, search options, filtering...). For example:

GET https://www.googleapis.com/customsearch/v1?key=INSERT-YOUR-KEY&cx=013036536707430787589:_pqjad5hr1a&q=flowers&alt=json

Response formats can be JSON or Atom.

{
 "kind": "customsearch#search",
 "url": {
  "type": "application/json",
  "template": "https://www.googleapis.com/customsearch/v1?q\u003d{searchTerms}&num\u003d{count?}&start\u003d{startIndex?}&hr\u003d{language?}&safe\u003d{safe?}&cx\u003d{cx?}&cref\u003d{cref?}&sort\u003d{sort?}&alt\u003djson"
 },
 "queries": {
  "nextPage": [
   {
    "title": "Google Custom Search - flowers",
    "totalResults": 10300000,
    "searchTerms": "flowers",
    "count": 10,
    "startIndex": 11,
    "inputEncoding": "utf8",
    ...

Summary

The big players have very different REST method signature approaches, but all of them share the simplicity and handyness of REST APIs. Being public APIs, all of them are also able to provide results in a number of formats, which allow for an easy consumption of these web services from a huge number of technologies.

As the real world rules for REST web services seem so lax, my advice is to prioritize ease of usage when designing REST services, and leverage the support that your tools or framework provide when building these services. The actual aspect of method signatures and result types is not as important as the naming and type consistency across your various method calls. Using HTTP methods properly is not as critical unless full proxy-cache support is a requirement for your system, and in many cases you'll be safe just using POST for requests that change the state of the system.

I hope this helps showing the different approaches used out there.



[1] http://stackoverflow.com/questions/6845772/rest-uri-convention-singular-or-plural-name-of-resource-while-creating-it
[2] http://stackoverflow.com/questions/7016785/is-put-delete-idempotent-with-rest-automatic
[3] http://stackoverflow.com/questions/1418114/questions-on-proper-rest-design

[4] http://en.wikipedia.org/wiki/Representational_state_transfer

[5] http://developers.facebook.com/docs/reference/api/
[6] https://dev.twitter.com/docs/api
[7] http://code.google.com/apis/customsearch/v1/using_rest.html