Patch #1651

Hack to make redmine use pygmentize instead of CodeRay

Added by Youssef Abou-Kewik almost 6 years ago. Updated almost 5 years ago.

Status:NewStart date:2008-07-15
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-

Description

We're going to be using Redmine internally with lots of Python code involved. Thought I'd share this patch with you guys.
I never wrote in ruby before, and I used some of the code mentioned here: http://matt.tarbit.org/2008/02/05/extending-bluecloth-with-pygments

You need to test that pygmentize works on your system first, and if it does, this should work just fine. Unfortunately, syntax highlighting is not a plugin.

patch Magnifier - use pygmentize instead of coderay patch (4.67 KB) Youssef Abou-Kewik, 2008-07-15 20:54

scm_patch_pygmentize.diff Magnifier - patch #2 (4.18 KB) Youssef Abou-Kewik, 2008-07-25 22:17

scm_patch_pygmentize2.diff Magnifier (4.28 KB) Anatolii Kucheruck, 2008-09-19 16:52

scm_patch_pygmentize3.diff Magnifier - cleaner version (against stable 0.8.2, r2614) (5.58 KB) Holger Just, 2009-03-23 00:45

redmine_pygmentize_wiki.diff Magnifier - Pygmentize syntax coloring in redmine wiki (16.7 KB) Sebastien Metrot, 2009-04-08 16:37

syntax_highlight.diff Magnifier - SyntaxHighlighter (http://alexgorbatchev.com/wiki/SyntaxHighlighter) Patch (133 KB) Youssef Abou-Kewik, 2009-05-14 11:24


Related issues

Related to Feature #2985: Make syntax highlighting pluggable Closed 2009-03-16

History

#1 Updated by Calvin Cheng over 5 years ago

Youssef Abou-Kewik wrote:

We're going to be using Redmine internally with lots of Python code involved. Thought I'd share this patch with you guys.
I never wrote in ruby before, and I used some of the code mentioned here: http://matt.tarbit.org/2008/02/05/extending-bluecloth-with-pygments

You need to test that pygmentize works on your system first, and if it does, this should work just fine. Unfortunately, syntax highlighting is not a plugin.

Looks good. It's what I have been looking for!

Coderay has simply been just too limited. Pygments rock in comparison.

Will give this patch a shot. But it looks like I have to maintain this file thereafter since I track the trunk directly.

#2 Updated by Youssef Abou-Kewik over 5 years ago

I'm attaching another update. This one should count fully on pygmentize instead of CodeRay (the previous patch relied on CodeRay to determine file type). However, this one relies on writing a temp file to /tmp where pygmentize can read it and determine its file type on its own. Also, files with UTF8 encoding should work now.

#3 Updated by Anatolii Kucheruck over 5 years ago

Youssef Abou-Kewik wrote:

I'm attaching another update. This one should count fully on pygmentize instead of CodeRay (the previous patch relied on CodeRay to determine file type). However, this one relies on writing a temp file to /tmp where pygmentize can read it and determine its file type on its own. Also, files with UTF8 encoding should work now.

There are one "bug" in this patch: if pygmentize can't recognize file type (e.g. file has no extension or pygmentize has no such type in lexer's list), the error was occurred and empty string was returned, and we have no content.
I'm attaching athother one path with fix this (file type compulsorily set to "text" in described case).

#4 Updated by Kornelius Kalnbach over 5 years ago

It is true that CodeRay's language support is still very limited, but development has started again. We're using Redmine for issue tracking :)

The latest CodeRay version supports some additional languages, but Python is still not included. Python support for CodeRay is scheduled for the next version.

pygments is a very good tool, I can only recommend it - but it would be nice to have a pure-Ruby solution. Ultraviolet is one with a lot of languages, but it has more dependecies and is slower than CodeRay.

#5 Updated by Jean-Philippe Lang over 5 years ago

We're using Redmine for issue tracking :)

Excellent news :)

The latest CodeRay version supports some additional languages

I'll upgrade as soon as possible to this new release. Thanks!

#6 Updated by Eric Davis about 5 years ago

Thank you for the patch. There are a few improvements needed though:

  1. Requires a UNIX environment. From what I non POSIX systems (Windows) don't support output redirection (e.g. The > in `echo #{escape_shell_arg(content)} > /tmp/#{basename}`)
  2. Requires several shell commands (pygmentize, echo, rm)
  3. Shell commands are assumed to be located in the system PATH, which may fail under environments like Apache Passenger
  4. Files are written and deleted without testing if they exist and are valid first. It wouldn't be too difficult to trick the regex in escape_shell_arg to remove a system file
  5. Sending commands to the shell (`command`) are run asynciolsly. The system(command) should be used instead.

#7 Updated by Youssef Abou-Kewik about 5 years ago

Well, the patch was for an immediate need that we had at our office. However, I think something like
http://code.google.com/p/syntaxhighlighter/

or
http://shjs.sourceforge.net/

Are actually better solutions than CodeRay or even my patch. I'll try to work something out using one of the solutions above and update this patch when I get some time.

#8 Updated by Kornelius Kalnbach about 5 years ago

The question is: does it highlight Ruby as good as the current highlighter? Most syntax highlighters have very bad Ruby support. That was the original reason for me to write CodeRay.

#9 Updated by Kornelius Kalnbach about 5 years ago

CodeRay development is slow at the moment. Maybe it would be best to make syntax highlighting a plugin (as the original post suggested), and leave the decision to the users. Ultraviolet for example supports a lot more languages and themes, but requires Oniguruma and other libraries, while solutions like JavaScript syntax highlighting minimize server load (I'm guessing here). But RedMine would still need a default syntax highlighter.

#10 Updated by Casseen Star about 5 years ago

I found another patch to use pygments inside redmine: http://jakewharton.com/2008/11/12/using-pygments-in-redmine/

#11 Updated by Eric Davis about 5 years ago

Kornelius Kalnbach wrote:

CodeRay development is slow at the moment. Maybe it would be best to make syntax highlighting a plugin (as the original post suggested), and leave the decision to the users.

I like that idea. Then alternative highlighers can be used and worked on separately.

#12 Updated by Holger Just about 5 years ago

While the solution of Jake Wharton (see note #10) seems to be the cleanest version of the naive implementation, at least the current stable version of Debian does not include the required pygments patch. Therefore we still need a temporary file with the correct file extension to pass to the pygmentize program.

Because of the outstanding issues on the provided patch, I tried to improve it. My current version addresses some of the issues of Eric (#6). I now use the tempfile package (which needed to be extended for Ruby < 1.9 however). This should resolve most issues with unsafe file access. Some operations with weird filenames are still going to fail. But I think, those filenames should still be safe as tempfile is going to refuse to create certain dangerously named files.

Only pygmentize is required to be in the current path. The current version shipped with Debian Lenny (in package python-pygments) works fine. The patch however will probably still not work on Windows as the rewriting of the error stream of pygmentize will fail on that platform. However, this was not tested.

Note: Previous version of the patch perform unsafe file operations. Using specially crafted filenames in the repository it was possible to read, write and delete any file accessible by the user of the webserver. If you use one of the previously provided patches, you should update your installation as soon as possible!

#13 Updated by Sebastien Metrot about 5 years ago

hi,

I tried the different patches to include Pygment sntax hilighting in redmine but none of them seemed to support it in the wiki. I hacked it a bit and came up with a solution that seems to work. I have attached my patch against revision 2666 of the redmine svn.

It was my first time hacking ruby and the solution seems very dirty to me (particularly the & that I have to transform into & by hand... anyone knows a better solution to unescape the wiki code?).

#14 Updated by Jean-Baptiste Barth about 5 years ago

The ampersands becoming "%x%x" is a redcloth bug ; you could have a look at my second patch in #2985, it solves this problem by simply desactivating it in redcloth (be careful, there are other redcloth changes in this patch). I haven't took a deeper look at it to see why redcloth do that, and how to solve it properly sorry...

#15 Updated by Youssef Abou-Kewik almost 5 years ago

OK, so this patch removes the dependency on pygmentize and no longer writes any temp files. We now use javascript highlighter which takes the load to the client side.

#16 Updated by Jiongliang Zhang almost 5 years ago

There is a new way using pygments as highlight tool for repository files: http://www.redmine.org/boards/3/topics/6890

Also available in: Atom PDF