Migration towards a markdown + Dropbox based system

I am testing a new blogging system purely based on Markdown and Dropbox. The inability of Blogger to correctly show many items (the most important of them is the code snippets that I include in my tips) that I try to include in these pages has worn me out.

In this system, I just write my articles in Markdown based pure text and put them on a dedicated folder in my Dropbox. The blogging engine fetches them and converts them to HTML to show them on the web.

The engine that I am testing currently is Scriptogr.am ( but there are other alternatives like Calepin - that already had some viability problems) and my new blog lives at this address: http://scriptogr.am/myildi/  where you can find some of the articles that have been published here, and new articles that will only published there.

I do not know if Scriptogr.am will be able to survive or not (I hope that it will, since it is very nice), but I consider that Markdown is here to stay and this simple system of automatic conversion of markdown files to HTML, to be published on the web, will persist in some for or another. At worst, I will be able to include a blog section on my web site, using a homemade conversion based on multimarkdown or pandoc.

Using Hazel for the automatization of the conversion: Markdown -> Latex -> PDF

Using Hazel for the automation of the conversion: Markdown -> Latex -> PDF
As I have explained in my last posts, I am trying to craft a writing chain that would allow me to write in Markdown and easily convert this readable text to Latex and PDF. I can even download the final PDF file on my iPAD using a Dropbox folder.
Hazel is a very nice automatization system that I was planning to explore more in details, and it indeed solves a lot of problems in this case.
My chain works as the following (this completes my previous message):

1. I write my text in IAWrite (under OSX or iOS) and they get saved in the iCloud (suppose that the file is called mytext.md).
2. On the desktop, these files appear in the mobile documents folder (~/Library/Mobile Documents/74ZAFF46HB~jp~informationarchitects~Writer/Documents). I use a first Hazel script (see Figure 1), attached to this folder, for copying this file to a Dropbox folder, as soon as a new version is saved. 
   

   

   Fig. 1: Copying the .md file

3. When the .md file is copied, other Hazel scripts, attached to the destination Dropbox folder, start to play.
4. One script converts the file to .html and .opml formats. This script calls multimarkdown for converting the copied markdown file to these formats (see Figures 2–3).

   

   Fig 2: Running the script for converting to HTML and OPML
   

   

   Fig 3: The script for calling multimarkdown

5. Another script (see Figures 4–5) converts the file to Latex format (only if the md file contains the meta tags that are necessary for obtaining a complete Latex file at the end of the conversion – otherwise Multimarkdown converts Latex snippets that cannot be compiled by Latex).

   

   Fig 4: Calling the script for converting to Latex

   

   Fig 5: The script that calls multimarkdown

6. A last script (see Figure 6) converts the Latex file obtained in the previous step to PDF, using latexmake. Latexmake is a Perl script that automatically runs the necessary operations for converting your tex file to PDF (the final format I have chosen here). Moreover, I have discovered that multimarkdown is not able to correctly convert accented characters in the meta tag fields. So, I have problems if the title, for example, is in French. For correctly converting these Unitype characters, one needs to compile the document using xelatex, and not pdflatex. The full Bash script I use can be downloaded below (including some commented instructions that was useful for debugging it, and that could be useful in some other contexts).

   
   

   

   Fig 6: Calling the script that runs latexmake

Download the script
  

Using Marked for previewing and exporting Markdown text

So, I am learning Markdown using iAWriter and I have discovered a very nice tool for previewing my text as it will be typeset, once converted: Marked. Marked uses MathJax for previewing even your equations, written following the standard proposed by Multimarkdown.

Using markdown as a lightweight LateX editing support on iPad and iCloud

I have been looking at the ways of editing LateX files on the iPad. There are already interesting LateX editor (TexTouch, TexWriter etc.), but I find them quite heavy to use yet, and Latex files are not very nice to decode on the small screen.
Markdown is a much simpler and readable tagging language invented by John Gruber (aka Daring Fireball). I have been tempted to play with it several times in the past, without really getting there. My preferred Latex editor is Lyx now, but we are really not ready to get an iPad version of this very nice editor. So Markdown has seemed as an interesting possibility to explore for writing Latex in a nice editor as iAWriter. Since MultimarkDown offers quite efficient translation of Markdown to Latex (including the equations and figures), as week as other formats like HTML and OPML (useful for importing in Freemind).
Here is my automatic setup for this workflow

1. synchronizing iAWriter documents between the iPad and my laptop using iCloud;
2. copying the file from the iCloud folder (~/Library/Mobile Documents/74ZAFF46HB~jp~informationarchitects~Writer/Documents) to a Dropbox folder each time it has been updated (with Hazel);
3. converting the .md file in the Dropbox folder to Latex, HTML and OPML using Multimarkdown in Hazel, a soon as a new version if copied there;
4. converting the Latex to pdf using by running pdflatex+bibtex+pdflatex twice).

For the third step, I had to adapt the Multimarkdown latex support files to create a standard article export, instead of the memoir one used by default.
Now I can edit my documents on the iPad or on the MacBook using iAWriter and they get automatically synched and converted. If my MacBook is on, I can even get back the compiled PDF back on the iPad, using a Dropbox access (through GoodReader, for example).
I can give more details for this process if you ask for it in the comments.

Top 20 journals in economics, counted by Google, following their h-index of the last 5 years

Google scholar also does bibliometrics now, on researchers (you can easily get your personal Google Scholar page, with all your references and citations indexed by Google), but also now on journals, even in several different languages. The following link will, for example give top 20 journals containing the word "econ" in their titles:

Top 20 economics journals in English
and
Top 100 journals in English (all areas)

Using multiple processor cores in R-Projet

My laptop is equipped with a Core i7 processor with 4 cores that can execute in parallel 8 processes. My R-Project computations use only one single core, if I use the default instructions. I have ended up by thinking that it is a pity that other cores are just sitting idle (sort of), instead of contributing to the speed of my computations, even if I do not run yet really heavy ones in my research. As a consequence, I have started to look for an easy way to use all cores in R-project. And, indeed, there is an easy solution to this problem. It uses the doMC library, and the instructions foreach and %dopar%.

For example, for computing linear models with different dependent variables and a given set of exogenous ones, one can do the following computations:

library(doMC) # There are other parallel computing libraries

registerDoMC() # You mud register one of them for foreach

getDoParWorkers() # Indicates you how many cores have been detected by registerDoMC()

Suppose that you have a dataset called mydata, containing the dependent variables y1, y2, y3, and the independent variables, x1,x2,x3.
We can execute in parallel the estimation of linear models of each y on the set of independent variables, by executing the following code:

myVariableList <- c("y1", "y2", "y3")

results <- foreach(i = 1:length(myVariableList),.errorhandling="stop",.inorder=TRUE)

%dopar% {

model <- lm(as.formula(paste(myVariableList[i],"~x1+x2+x3")),data=mydata)

return(model)

}

%dopar% executes these estimations on different cores, in parallel and a list of the estimated models is saved in the variable results.
We can now look at the characteristics of the estimated models, by printing them successively on the output of R:

for (i in 1:length(results)) { print(summary(results[[i]])) }
Voilà!

Of course, this possibility is especially useful for more complex computations, like stepwise regressions with many independent variables, that can take some time, or regression trees with big datasets, etc.

Broken XQuartz applications under OSX Lion

So, my new laptop runs OSX Lion. I have been able to migrate my data from the old laptop (but the automatic process provided by the Migration assistant was a failure, Apple support had to help me to do it manually).

Many of the migrated applications have worked without a hitch as soon as the migration ended, but some other applications, like gretl -the GNU econometric software- and some wine-based windows tools, have refused to start. I was quite annoyed by this, but have not had any time to take care of it until today. I have installed wine using MacPorts (instead of the WineBottler that I normally use), the problem persisted. I have discovered PlayItOnMac (PION) implementation of Wine and installed it, but the installation stalled during the creation of the virtual disk needed by this soft.

Now, the support page of PION tells that you should start XQuartz to complete the installation process if you get stalled the first time. And I have tried to do launch it (it lives in Applications/Utilities), but it crashed by telling that the version I am trying to launch is maybe not compatible with my OS. Actually, that was an important clue. I have directly jumped on the XQuartz website and seen that there was a new version waiting for me. I have installed it and BINGO! All my Wine and X11 softs are working again.

So, if you have difficulties to launch Wine-based or XWindows softs after having upgraded to Lion (or on your new Mac under Lion), you should download and install the most recent version of XQuartz, from their web site.