Wiki External Filter plugin released

Added by Alexander Tsvyashchenko over 7 years ago

Redmine Wiki External Filter plugin allows defining macros that process macro argument using external filter program and render its result in Redmine wiki.

Currently supported macros are plantuml (for UML diagrams drawing), graphviz (for graphs drawing), ritex (for formula drawing) and fortune (simply for fun ;-), but additional ones are easy to add.

plantuml example input:

    {{plantuml(
    Alice -> Bob: Authentication Request
    alt successful case
      Bob -> Alice: Authentication Accepted
    else some kind of failure
      Bob -> Alice: Authentication Failure
      opt
        loop 1000 times
          Alice -> Bob: DNS Attack
        end
      end
    else Another type of failure
      Bob -> Alice: Please repeat
    end
    )}}

and output:

graphviz example input:

    {{graphviz(
    digraph finite_state_machine {
        rankdir=LR;
        size="8,5" 
        node [shape = doublecircle]; LR_0 LR_3 LR_4 LR_8;
        node [shape = circle];
        LR_0 -> LR_2 [ label = "SS(B)" ];
        LR_0 -> LR_1 [ label = "SS(S)" ];
        LR_1 -> LR_3 [ label = "S($end)" ];
        LR_2 -> LR_6 [ label = "SS(b)" ];
        LR_2 -> LR_5 [ label = "SS(a)" ];
        LR_2 -> LR_4 [ label = "S(A)" ];
        LR_5 -> LR_7 [ label = "S(b)" ];
        LR_5 -> LR_5 [ label = "S(a)" ];
        LR_6 -> LR_6 [ label = "S(b)" ];
        LR_6 -> LR_5 [ label = "S(a)" ];
        LR_7 -> LR_8 [ label = "S(b)" ];
        LR_7 -> LR_5 [ label = "S(a)" ];
        LR_8 -> LR_6 [ label = "S(b)" ];
        LR_8 -> LR_5 [ label = "S(a)" ];
    }
    )}}

and output:

ritex example input:

    {{ritex(
    G(y) = \left\{\array{ 1 - e^{-\lambda x} & \text{ if } y \geq 0 \\ 0 & \text{ if } y < 0 }\right.
    )}}

and output:

fortune sample input:

{{fortune}}

and output:

Detailed information is available here: https://www.ndl.kiev.ua/content/redmine-wiki-external-filter-plugin

Replies (80)

RE: Wiki External Filter plugin released - Added by Billy T over 7 years ago

I like this plugin so much.
Thank you.
Could you tell me how to install the patch you provide on your site?

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko over 7 years ago

Just navigate to vendor/plugins/redmine_markdown_extra_formatter of your Redmine installation and execute patch -p1 < redmine_markdown_extra_formatter_fixes.patch

That of course assumes you have redmine_markdown_extra_formatter installed on your site and use it for wiki formatting ;-) If you use any other formatter (including default Redmine one) you should apply the changes implemented in the patch manually.

RE: Wiki External Filter plugin released - Added by Nikolay Kotlyarov over 7 years ago

Can i somehow embed a streaming video or at list *.swf file using your plugin?

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko over 7 years ago

If you want to embed some existing content without any modifications - there's no point in using Wiki External Filter for that as it's main purpose is to embed modified/processed content. If you need to embed modified content - yes, that's possible, here's the example of config file section for macro that treats its argument as URL to the video, downloads it, converts to SWF reducing size/quality using ffmpeg and embeds it to the output page:

flash:
  description: "Converts any given video file to SWF preview" 
  command: "wget -i - -O - | ffmpeg -y -i - -r 20 -s 360x288 -ar 22050 -f swf -" 
  content_type: "application/x-shockwave-flash" 

Usage is as follows:

flash{{(
http://www.example.com/video.avi
)}}

Of course you might need to adjust the exact appearance of embedded content, or even use some third-party library/component to embed/show it such as http://flowplayer.org - for that you need to write appropriate markup in macro_block.html.erb and/or macro_inline.html.erb

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko over 7 years ago

In fact in less than a day after answering your question I had to embed several videos to Redmine myself, so I decided to go ahead and add embedding functionality to wiki external filter, please see new topic about 0.0.2 plugin release.

RE: Wiki External Filter plugin released - Added by Patrick Fratczak about 7 years ago

Your plugin seems really usefull but i have some troubles to make it work !
I use Redmine 0.9 and patches you provide are not for this version ...
I try to embed a video avi file with the video macro but i got the following error :

Error executing the video macro ( ActionView::TemplateError (undefined method `+' for nil:NilClass) on line #5 of vendor/plugins/wiki_external_filter/app/views/wiki_external_filter/macro_flash-video.html.erb: 3: image = begin Magick::Image::from_blob(content[0]).first rescue nil end 4: %> 5: <%= image.rows.to_s %>

It seems like content0 is empty ?

Then, with plantuml.jar

I have installed it to /usr/share/plantuml/lib/plantuml.jar
and i have created /usr/bin/plantuml

#!/bin/bash
/usr/bin/java -Djava.io.tmpdir=/var/tmp -jar /usr/share/plantuml/lib/plantuml.jar

The error :

Error executing the plantuml macro (Error applying external filter: stdout is , stderr is (.:20043): Gtk-WARNING **: cannot open display: )

It seems that plantuml.jar want to open a dialog panel ???

For ritex and graphviz, nothing ... my code is not parsed as a macro ??

Well, is your plugin made for redmine version 0.9 ?

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko about 7 years ago

Hello Patrick,

I use Redmine 0.9 and patches you provide are not for this version ...

To the best of my knowledge there's no such thing as "Redmine 0.9" available, please state the actual version.

Besides that, there's only one patch that needs to be applied to Redmine core itself (for attachments support) - other changes are for wiki formatter, and currently available patches are for Markdown Extra wiki formatter only, you'll have to port them to Textile wiki formatter yourselves if you use it.

So, what wiki formatter you use, what patches you've tried to apply to what components, in what way you've tried to do it and what failed exactly?

Also please do not take offence, but if you've never seen this document I'd really recommend it - it's brilliant.

It seems that plantuml.jar want to open a dialog panel ???

I'd recommend testing diagrams generation with PlantUML from command line first to check it is installed correctly, see their official site, they explain how to do that.

For ritex and graphviz, nothing ... my code is not parsed as a macro ??

Most likely yes, see "Change MACROS_RE" link in the documentation.

Well, is your plugin made for redmine version 0.9 ?

No, see above. However, I successfully use it for Redmine 0.9.3 and attachments patch seems to be applied cleanly to 0.9.4 too (haven't tested it works though).

RE: Wiki External Filter plugin released - Added by Patrick Fratczak about 7 years ago

Sorry, i don't want to take offense ... i just want to know if this plugin works with redmine 0.9...
I have tried to make some changes but nothing works ...
For plantuml, off course, i will read the documentation and if i found a way to make it work, i will post the solution here !
For the attachment patch, it seems that my redmine version is different and the patch cannot apply with the patch command ...
The source code is different.

So, i will try to make it work by myself and will come back here to post the solutions !
Thanks.

RE: Wiki External Filter plugin released - Added by Yuya Nishihara about 7 years ago

For the attachment patch, it seems that my redmine version is different and the patch cannot apply with the patch command ...

What kind of wiki engine are you using?
The patch I found is for markdown. Maybe it'll be helpful to show full output of commands you typed.

I'm trying to patch textile formatter, and it partially works now.
This plugin is awesome. I like it.

RE: Wiki External Filter plugin released - Added by Patrick Fratczak about 7 years ago

Hello,
I have tried to patch by myself the textile formatter but with no luck !
So i have decided to remove my modification to retry later.
You say that your patch partially works.
Could you post here your modifications and perhaps can i help you to make it works totaly ! ;)
I really would like to use this plugin and my server is ready to get i work with graphviz, ritex, plantuml, ffmpeg ...
All are installed but perhaps not totally configured...

RE: Wiki External Filter plugin released - Added by Yuya Nishihara almost 7 years ago

Patrick Fratczak wrote:

You say that your patch partially works.
Could you post here your modifications and perhaps can i help you to make it works totaly ! ;)

Sure, attached. It works but really messy. It tries to unescape HTML tags before passing to macro procedure.
Maybe proper solution is completely different.

RE: Wiki External Filter plugin released - Added by Patrick Fratczak almost 7 years ago

Thank you Yuya ... i will try to apply your patch and perhaps add some modifications to make it works better.

RE: Wiki External Filter plugin released - Added by Denis Bauer almost 7 years ago

HI,
I'm using 0.9.4.devel and an a bit at a loss which patch is necessary or applicable.

Let's start with the attachments patch which you said

seems to be applied cleanly to 0.9.4 too (haven't tested it works though).

I got it from here and tried to apply it in redmine root (the on with the app folder ect.) by doing

patch -p0 -R < patch/redmine-attachments-in-macros.patch

but I get errors:

can't find file to patch at input line 4
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -urN redmine.orig/app/helpers/application_helper.rb redmine/app/helpers/application_helper.rb
|--- redmine.orig/app/helpers/application_helper.rb     2010-03-21 13:07:02.000000000 +0200
|+++ redmine/app/helpers/application_helper.rb  2010-03-25 21:18:36.753773277 +0200
--------------------------
File to patch:  

It's a similar story with the patch for lib/redmine/wiki_formatting/textile/formatter.rb, where the file just does not contain the lines that are replaced by the patch.

Could you spell out again which patches to apply?
Thanks heaps

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko almost 7 years ago

Hi Denis,

Let's start with the attachments patch which you said

[skipped]

I got it from here and tried to apply it in redmine root (the on with the app folder ect.) by doing
patch -p0 -R < patch/redmine-attachments-in-macros.patch
but I get errors:

If running from redmine root directory, you should use it like patch -p1 < redmine-attachments-in-macros.patch (note -p1, not -p0), also note that -R is generally not that good idea at all ;-)

It's a similar story with the patch for lib/redmine/wiki_formatting/textile/formatter.rb, where the file just does not contain the lines that are replaced by the patch.

I do not have the patch for lib/redmine/wiki_formatting/textile/formatter.rb, which of the patches you're trying to apply to it? If it is wiki-external-filter-textile-workaround-2010-06-30.patch mentioned in the posts above - you should ask Yuya Nishihara for details, maybe he developed it for the trunk version?

Could you spell out again which patches to apply?

The approach I use is the following:
  • Patch wiki formatter. Currently I provide the patch for Markdown Extra wiki formatter only! I haven't tested the patch for Textile mentioned above so I cannot say anything about it. In fact for the most recent version of Markdown Exta formatter the older patch will not apply, so you should use forked Markdown Extra formatter version, i've just updated documentation accordingly.
  • Patch redmine core to support attachments in macros using redmine-attachments-in-macros.patch as explained above.

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko almost 7 years ago

2Yuya: Thanks for the patch! Which filters have you tested it with? I've added the link to your patch from documentation page (with disclaimer that it might be incomplete), I hope you do not mind ;-)

RE: Wiki External Filter plugin released - Added by Yuya Nishihara almost 7 years ago

Alexander Tsvyashchenko wrote:

Which filters have you tested it with? I've added the link to your patch from documentation page

Thank you. I run Redmine 0.9.5 with that patch, and it works for plantuml and graphviz.

RE: Wiki External Filter plugin released - Added by Paul paul almost 7 years ago

Hi Yuya,

I followed the instructions in your patch (for Textile), but I get a 404. I'm running 0.9.6.
Running 'dot -Tpng myfile.dot -o myfile.png' in the shell works perfectly (I'm on FreeBSD).
What am I doing wrong? Has anyone else had a 404 problem? Two empty boxes get drawn (one for SVG, one for PNG I presume, according to the wiki_external_filter.yml settings)

Here's the log:

Processing WikiExternalFilterController#filter (for 1.2.3.4 at 2010-07-12 03:03:37) [GET]
Parameters: {"name"=>"8a649c02f62ec93cb245e048be930b2550ac128d6ad522b6a45ab3b940a92faa", "macro"=>"graphviz", "action"=>"filter", "index"=>"0", "controller"=>"wiki_external_filter"}
Rendering template within layouts/base
Rendering common/404 (404)
Completed in 55ms (View: 35, DB: 4) | 404 Not Found [https://myserver/wiki_external_filter/filter?index=0&macro=graphviz&name=8a649c02f62ec93cb245e048be930b2550ac128d6ad522b6a45ab3b940a92faa]

RE: Wiki External Filter plugin released - Added by Yuya Nishihara almost 7 years ago

I followed the instructions in your patch (for Textile), but I get a 404. I'm running 0.9.6.

Thanks for trying it. Do you run Redmine in development mode? I've seen 404 error on development environment, but okay on production env.

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko almost 7 years ago

Paul paul wrote:

I followed the instructions in your patch (for Textile), but I get a 404. I'm running 0.9.6.

Are there any additional log entries related to wiki_external_filter before the one with 404 error?

Is the file mentioned in the error message actually created on disk? The full path should be smth like <your-redmine-installation-path>/cache/views/wiki_external_filter/graphviz/<content-digest> if you've configured caching properly, where <content-digest> is equal to "name" parameter, i.e. long hexadecimal string.

You might also consider either modifying graphviz macro or creating custom one with the command that just dumps passed content to some temporary file on disk - this way you can verify that content is passed properly from macro argument to the filter.

RE: Wiki External Filter plugin released - Added by Paul paul almost 7 years ago

The minute I configure caching, it started working! You might consider clarifying in the instructions that caching is a must.

I have one remaining problem: every graph appears twice.
I have
- command: "/usr/local/bin/dot -Tsvg"
content_type: "image/svg+xml"
- command: "/usr/local/bin/dot -Tpng"
content_type: "image/png"

in my wiki_external_filter.yml. Removing the top or the bottom 2 lines changes nothing.

What should I do to avoid generating the second copy?

Thanks so much!

RE: Wiki External Filter plugin released - Added by Paul paul almost 7 years ago

I just realized that the second copy is only visible in the Preview mode. The minute you save, the second copy disappears.
Is there something I can change for the preview behavior to work properly?

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko almost 7 years ago

Paul paul wrote:

The minute I configure caching, it started working! You might consider clarifying in the instructions that caching is a must.

In fact it isn't really "a must" - if options of the command specify that it does not use caching, it should work fine without it. Actually even if command needs caching (and for the most non-trivial commands caching is on) it should still work if it is not present (albeit very slow as it will have to recalculate things on each access), but it's possible things are just broken in this code path, I will try to have a look at some point at this.

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko almost 7 years ago

Paul paul wrote:

I just realized that the second copy is only visible in the Preview mode. The minute you save, the second copy disappears.
Is there something I can change for the preview behavior to work properly?

I have this behavior too (in preview only) but I did not have time to investigate it as it doesn't really bother me too much. Most likely JavaScript code + CSS attributes used to decide what image to show and what to hide either are not applied at all in preview mode or work incorrectly. If you'd like to investigate it, the JavaScript code is in assets/javascripts/css_browser_selector.js that is included from the app/views/wiki_external_filter/macro_svg.html.erb file and CSS attributes are at the bottom of the macro_svg.html.erb file.

Eventually I might try to have a look at this problem myself, but I cannot promise it will happen soon.

RE: Wiki External Filter plugin released - Added by Paul paul almost 7 years ago

I'm having another problem, this time with the ffmpeg macro. I have not patched Redmine to process attachments, since the videos I plan on linking are external.

Input:

{{video_url(http://www.fileformat.info/format/mpeg/sample/567fd6a0e0da4a8e81bdeb870de3b19c/DELTA.MPG)}}


Preview output:

Error executing the video_url macro (Error applying external filter: stdout is , stderr is -: Invalid URL http://www.fileformat.info/format/mpeg/sample/567fd6a0e0da4a8e81bdeb870de3b19c/DELTA.MPG: Scheme missing No URLs found in -. FFmpeg version 0.5.2, Copyright (c) 2000-2009 Fabrice Bellard, et al. configuration: --prefix=/usr/local --mandir=/usr/local/man --enable-shared --enable-gpl --enable-swscale --enable-postproc --enable-avfilter --enable-avfilter-lavf --enable-pthreads --enable-memalign-hack --cc=cc --extra-cflags=-I/usr/local/include/vorbis -I/usr/local/include --extra-ldflags=-L/usr/local/lib --extra-libs=-pthread --disable-debug --disable-sse --disable-mmx --disable-libamr-nb --disable-libamr-wb --disable-libdirac --disable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --disable-vhook --enable-ipv6 --enable-libmp3lame --enable-libopenjpeg --enable-libschroedinger --disable-ffplay --disable-libspeex --enable-libtheora --enable-libvorbis --disable-x11grab --enable-libx264 --enable-libxvid libavutil 49.15. 0 / 49.15. 0 libavcodec 52.20. 1 / 52.20. 1 libavformat 52.31. 0 / 52.31. 0 libavdevice 52. 1. 0 / 52. 1. 0 libavfilter 0. 4. 0 / 0. 4. 0 libswscale 0. 7. 1 / 0. 7. 1 libpostproc 51. 2. 0 / 51. 2. 0 built on Jul 12 2010 19:16:04, gcc: 4.2.1 20070719 [FreeBSD] pipe:: Unknown format --2010-07-12 19:26:31-- http://%3Ca/ Resolving http://www.fileformat.info/format/mpeg/sample/567fd6a0e0da4a8e81bdeb870de3b19c/DELTA.MPG: Scheme missing. pipe:: Unknown format usage: rm [-f | -i] [-dIPRrvW] file ... unlink file )

production.log output:

Processing WikiController#preview (for 1.2.3.4 at 2010-07-12 19:26:31) [POST] Parameters: {"commit"=>"Save", "extension"=>{"tags"=>{"6"=>"", "11"=>"", "7"=>"", "12"=>"", "8"=>"", "13"=>"", "9"=>"", "14"=>"", "15"=>"", "0"=>"", "16"=>"", "1"=>"", "17"=>"", "2"=>"", "18"=>"", "3"=>"", "19"=>"", "4"=>"", "10"=>"$ executing command: wget -i - -O - | ffmpeg -i - -vframes 1 -f mjpeg - executing command: FN=`cat`; wget $FN -O - | (if [ `echo ${FN##*.} | tr [:upper:] [:lower:]` != 'flv' ]; then TMPVIDEO=`tempfile --directory=/var/tmp`; ffmpeg -y -i - -sameq -ar 44100 -f flv $TMPVIDEO; STATUS=$?; cat $TMPVIDEO; rm $TMPVIDEO; exit $STATUS; else cat; fi) Completed in 250ms (View: 214, DB: 6) | 200 OK [https://myserver/projects/myproject/wiki/Blah/preview]

RE: Wiki External Filter plugin released - Added by Alexander Tsvyashchenko almost 7 years ago

Paul paul wrote:

I'm having another problem, this time with the ffmpeg macro. I have not patched Redmine to process attachments, since the videos I plan on linking are external.

Most likely the URL is distorted when passed as the macro argument. Try putting something like echo \"$FN\" > /tmp/args.txt; after FN=`cat`; in the second command and then examine file /tmp/args.txt to check URL is passed correctly, with no preceding/trailing symbols and without any encoding.

1 2 3 4 (1-25/80)