Feature #1838

Bulk import for LDAP users

Added by paul k almost 9 years ago. Updated 7 months ago.

Status:NewStart date:2008-09-01
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Accounts / authentication
Target version:-
Resolution:

Description

Hi,

if you configure LDAP for authentication, users are not visible unless they have at least logged in once. This is IMO not ideal in terms of usability. You have to tell your users to login but they can't do anything because there are not member of any project (yet) and you can't add them because they don't exist (chicken 'n egg, kinda). The patch implements a new 'import' action for the LDAP auth source. Warning: I'm pretty much a ruby noob so suggestions wrt style and functionality are always welcomed.

Cheers
Paul

bulk_import_ldap.patch Magnifier - bulk import patch (5.93 KB) paul k, 2008-09-01 11:19

bulk_import_ldap2.patch Magnifier - bulk import patch (7.52 KB) paul k, 2008-09-01 11:35

ldapimport.rb Magnifier (2.57 KB) Thimios Dimopulos, 2009-11-10 20:52

import.php Magnifier (2.11 KB) Ritesh Sutaria, 2010-09-06 12:49

ldapimport - just bash script to run .rb. Also used for cron job (161 Bytes) rain man, 2010-09-20 16:36

ldapimport.rb Magnifier - file I modified (3.36 KB) rain man, 2010-09-20 16:36


Related issues

Duplicated by Redmine - Defect #24840: LDAP: Users who didn't login yet, are not accesible (and ... Closed

History

#1 Updated by paul k almost 9 years ago

Grmpf, wrong patch. Please ignore the first one

#2 Updated by Justin Grevich over 8 years ago

Is there any interest in merging this into Redmine? I think it's a great feature. I would like to expand upon it in that the user is given a "bulk import" dialogue box which allows the admin to select which users to import.

Thanks

#3 Updated by A G over 8 years ago

Does not work for me. I receive an error when trying to access an LDAP entry.

#4 Updated by Jens Goldhammer over 8 years ago

+100!

#5 Updated by Thomas D over 8 years ago

A G wrote:

Does not work for me. I receive an error when trying to access an LDAP entry.

Same for me here. I don't know much about Ruby, so unfortunately I can't be of much help

#6 Updated by Thimios Dimopulos almost 8 years ago

We are also very interested in this feature, any chance that it reaches the trunk?

#7 Updated by Thimios Dimopulos almost 8 years ago

I didn't want to patch Redmine in order to get this feature, so based on the patch by paul k, i created the attached script that i call through a cron job using the runner script:

*/5 * * * * /var/www/RedmineInstallations/development/0.8-stable/script/runner /var/www/RedmineInstallations/indice/ldapimport.rb > /var/log/redmine/development/ldapimport.log 2>&1

I wrote a blog entry about it at: http://blog.indice.gr/2009/11/redmine-bulk-import-for-ldap-users.html

regards

#8 Updated by Stanislav German-Evtushenko almost 8 years ago

Greate work man! Thank you!
I made small improvement (login is always to lowcase)

--- ldapimport.rb.orig  2009-11-13 11:31:29.000000000 +0300
+++ ldapimport.rb       2009-11-13 11:42:42.000000000 +0300
@@ -21,7 +21,7 @@
                      :mail => AuthSourceLdap.get_attr(entry, self.attr_mail),
                      :auth_source_id => self.id ]
             #sanity checking (all the above attributes are required)
-            login = AuthSourceLdap.get_attr(entry, self.attr_login)
+            login = AuthSourceLdap.get_attr(entry, self.attr_login).downcase
             catch :SKIP do
               skip = false
               attrs.each { |e| 

#9 Updated by Stanislav German-Evtushenko almost 8 years ago

One else improvement is update email if changed.

--- ldapimport.rb.orig  2009-11-13 11:31:29.000000000 +0300
+++ ldapimport.rb       2009-11-13 13:36:18.000000000 +0300
@@ -37,7 +37,14 @@
               }
             end
             next if skip
-            if User.find(:first, :conditions => ["login=?", login])
+            if u = User.find(:first, :conditions => ["login=?", login])
+             # Update email if changed
+             if u.mail != attrs[0][:mail]
+               u.mail = attrs[0][:mail]
+               if u.save
+                  logger.debug("Email for user #{login} was updated") if logger
+               end
+             end
               logger.debug("User #{login} already there, skipping...") if logger
               skipped.push(login+'(exists)')
               next

#10 Updated by Thimios Dimopulos almost 8 years ago

Thanks for the improvements. In case you also need it, i added a method that removes users that are no longer present in the LDAP database:

def remove_non_existing_users
        ldap_con = initialize_ldap_con(self.account, self.account_password)
        found = 0
        logger.info("Removing users that do not exist in LDAP authentication source: #{self.name}.")
        @users = User.all
        @users.each { |user|

            search_filter = Net::LDAP::Filter.eq("sAMAccountName", user.login)
            entry = ldap_con.search(
                    :base => self.base_dn,
                    :filter => search_filter,
                    :attributes => ['dn', self.attr_firstname, self.attr_lastname, self.attr_mail, self.attr_login]
                )
            # ignore anonymous (empty login name and instance of AnonymousUser) and admin users
            if entry.empty?  && !user.login.empty? && user.class != AnonymousUser && user.login != "admin" 
                logger.info("The user with login name: #{user.login} does not exist any more in LDAP and will be removed")
                      found += 1
                User.delete(user.id)
            end
        }
        logger.info("Removed #{found} users that did not exist in LDAP authentication source: #{self.name}.")
end

#11 Updated by Stanislav German-Evtushenko almost 8 years ago

Thimios Dimopulos wrote:

Thanks for the improvements. In case you also need it, i added a method that removes users that are no longer present in the LDAP database:

[...]

Hm.. I think it's not so good thing, because:
1. Users are using in a history (I mean tasks, projects, wiki edit history, etc).
2. I have one special user (for example "admin") who aren't using LDAP. It used for administration and support (if LDAP DB have down).
3. If LDAP DB will accidentally be corrupted you can remove your active users.

#12 Updated by Thimios Dimopulos almost 8 years ago

You are absolutely right, deleting users just like that would lead to database corruption.

About the admin and anonymous users, i take that into account and do not manipulate them:

if entry.empty?  && !user.login.empty? && user.class != AnonymousUser && user.login != "admin" 

I updated the method so that it just locks users that do not exist in LDAP any more:

def lock_non_existing
        ldap_con = initialize_ldap_con(self.account, self.account_password)
        found = 0
        logger.info("Removing users that do not exist in LDAP authentication source: #{self.name}.")
        @users = User.all
        @users.each { |user| 

            search_filter = Net::LDAP::Filter.eq("sAMAccountName", user.login)
            entry = ldap_con.search(
                    :base => self.base_dn,
                    :filter => search_filter,
                    :attributes => ['dn', self.attr_firstname, self.attr_lastname, self.attr_mail, self.attr_login]
                )
            # ignore anonymous (empty login name and instance of AnonymousUser) and admin users 
            if entry.empty?  && !user.login.empty? && user.class != AnonymousUser && user.login != "admin" 
                logger.info("The user with login name: #{user.login} does not exist any more in LDAP and will be locked")
                      found += 1
                user.status=User::STATUS_LOCKED
                user.save
            end
        }
        logger.info("Locked #{found} users that did not exist in LDAP authentication source: #{self.name}.")
    end

#13 Updated by Stanislav German-Evtushenko almost 8 years ago

Thimios Dimopulos wrote:

You are absolutely right, deleting users just like that would lead to database corruption.

About the admin and anonymous users, i take that into account and do not manipulate them:
[...]

I updated the method so that it just locks users that do not exist in LDAP any more:

[...]

It's better way.
PS: LDAP doesn't have sAMAccountName. sAMAccountName is just Microsoft's thing.

#14 Updated by Igor Kalashnikov almost 8 years ago

Unable to import (undefined method `+' for nil:NilClass)

When I try to "bulk import"

#15 Updated by Ritesh Sutaria almost 7 years ago

PHP script to Bulk Import LDAP users in Redmine.

You may add in crontab to execute every 24 hours.

0 0 * * * * root /user/bin/php /path/phpscript >> /var/log/logfile

#16 Updated by Stanislav German-Evtushenko almost 7 years ago

Ritesh Sutaria wrote:

PHP script to Bulk Import LDAP users in Redmine.

You may add in crontab to execute every 24 hours.

0 0 * * * * root /user/bin/php /path/phpscript >> /var/log/logfile

This script works with PostgreSQL only. I think ldapimport.rb quite better because:
  • it uses own redmine engine for adding accounts
  • it isn't depend of database engine

#17 Updated by Samu Tuomisto almost 7 years ago

Stanislav German-Evtushenko wrote:

Ritesh Sutaria wrote:

PHP script to Bulk Import LDAP users in Redmine.

You may add in crontab to execute every 24 hours.

0 0 * * * * root /user/bin/php /path/phpscript >> /var/log/logfile

This script works with PostgreSQL only. I think ldapimport.rb quite better because:
  • it uses own redmine engine for adding accounts
  • it isn't depend of database engine

Seems to work ok also with redmine 1.01 - thanks!

I wonder why this user population feature is not in redmine core - I don't see any point using LDAP with redmine without populating user base and setting access rights for users before hand.

#18 Updated by rain man almost 7 years ago

Hi, I made some modifications to make this work in Active directory. It handled disabled accounts and additional logic to handle locking of accounts in redmine.

#19 Updated by Samu Tuomisto over 6 years ago

rain man wrote:

Hi, I made some modifications to make this work in Active directory. It handled disabled accounts and additional logic to handle locking of accounts in redmine.

I tested the script but faced some errors. Could you advice me - how to solve following problem. Error messages from ldaplog.txt:

/opt/redmine-1.0.1-0/apps/redmine/script/../config/../vendor/rails/railties/lib/rails/gem_dependency.rb:119:Warning: Gem::Dependency#version_requirements is deprecated and will be removed on or after August 2010.  Use #requirement
/opt/redmine-1.0.1-0/apps/redmine/vendor/rails/railties/lib/commands/runner.rb:48: (eval):1: compile error (SyntaxError)
(eval):1: unknown regexp options - rd
(eval):1: no .<digit> floating literal anymore; put 0 before dot
/opt/redmine-1.0.1-0/apps/redmine/ldapimport.rb
                 ^
(eval):1: syntax error, unexpected tINTEGER
/opt/redmine-1.0.1-0/apps/redmine/ldapimport.rb
                  ^
        from /opt/redmine-1.0.1-0/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `eval'
        from /opt/redmine-1.0.1-0/apps/redmine/vendor/rails/railties/lib/commands/runner.rb:48
        from /opt/redmine-1.0.1-0/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /opt/redmine-1.0.1-0/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /opt/redmine-1.0.1-0/apps/redmine/script/runner:3

My ruby on rails skills are unfortunately very limited.

#20 Updated by Samu Tuomisto over 6 years ago

  • Status changed from New to Resolved

Samu Tuomisto wrote:

rain man wrote:

Hi, I made some modifications to make this work in Active directory. It handled disabled accounts and additional logic to handle locking of accounts in redmine.

I tested the script but faced some errors. Could you advice me - how to solve following problem. Error messages from ldaplog.txt:

[...]

My ruby on rails skills are unfortunately very limited.

This problem resolved - reasons were:
1) problem with gem mysql; solved by: http://oracleabc.com/b/archives/339
2) bitnami stack that I am not familiar with

#21 Updated by Felix Schäfer over 6 years ago

  • Status changed from Resolved to New

Samu Tuomisto wrote:

This problem resolved

Yours yes, but the original request isn't resolved yet, reopening.

#22 Updated by Patrice Bonhomme over 6 years ago

Hi there,

I have tried the ldapimport.rb within a bash file with redmine-1.0.4 and i have got this error :

/services/redmine/script/ldapimport.rb:25:in `import': undefined method `include?' for nil:NilClass (NoMethodError)
        from /services/redmine-trunk/vendor/plugins/ruby-net-ldap-0.0.4/lib/net/ldap.rb:642:in `search'
        from /services/redmine-trunk/vendor/plugins/ruby-net-ldap-0.0.4/lib/net/ldap.rb:1175:in `search'
        from /services/redmine-trunk/vendor/plugins/ruby-net-ldap-0.0.4/lib/net/ldap.rb:1144:in `loop'
        from /services/redmine-trunk/vendor/plugins/ruby-net-ldap-0.0.4/lib/net/ldap.rb:1144:in `search'
        from /services/redmine-trunk/vendor/plugins/ruby-net-ldap-0.0.4/lib/net/ldap.rb:640:in `search'
        from /services/redmine/script/ldapimport.rb:15:in `import'
        from /services/redmine/script/ldapimport.rb:104
        from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `eval'
        from /usr/lib/ruby/gems/1.8/gems/rails-2.3.5/lib/commands/runner.rb:46
        from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /services/redmine/script/runner:3

My knowledge of Ruby language are very limited...

Thanks for any help,
Patrice

#23 Updated by Alexandre Lissy over 6 years ago

I've been able to run this ldapimport.rb script with success on my redmine 1.1.2 instance, only had to change the id of the AuthSourceLdap in find() at the first line, because our LDAP server is id=2.

#24 Updated by Terence Mill about 6 years ago

Since redmine 1.2.1 there is a possibility to delete users, i think also exported to redmine api.
Would be nice to implement optional auto deletion this way.
Also there is a new plugin for ldap sync

#25 Updated by yannick quenec'hdu almost 6 years ago

Terence Mill wrote:

Since redmine 1.2.1 there is a possibility to delete users, i think also exported to redmine api.

How is possible to delete user in Rdmine and LDAP since 1.2.1 ?
I don't see this feature in roadmap. I tested with a redmine 1.2.1, if you delete a user in Redmine, this user isn't delete in LDAP.
If you delete user in LDAP, isn't delete in LDAP.

LDAP sync imposes a particular LDAP structure. It's not very generic.

Yannick

#26 Updated by Terence Mill almost 6 years ago

You can delete user in redmine via gui and i think via api too. I never said that the ldap plugin does sync that back to ldap.
For my understanding of ldap i would be not wished to sync application side user cache back to ldap more the other way round.
LDAP is central user service and the appluication (like redmine) will sync from ldap but never too.
Why don't u use plain redmine user base without ldap then?

#27 Updated by Terence Mill almost 6 years ago

You can delete user in redmine via gui and i think via api too. I never said that the ldap plugin does sync that back to ldap.
For my understanding of ldap it wouldn't be wished to sync application side user cache back to ldap more the other way round.
LDAP is central user service and the appluication (like redmine) will sync from ldap but never to ldap.
Why don't u use plain redmine user base without ldap then?

#28 Updated by yannick quenec'hdu almost 6 years ago

Terence Mill wrote:

You can delete user in redmine via gui and i think via api too. I never said that the ldap plugin does sync that back to ldap.

Because, this topics speak about LDAP sync. The feature "user delete" has existed for a long time, long before version 1.2.1.

For my understanding of ldap i would be not wished to sync application side user cache back to ldap more the other way round.
LDAP is central user service and the appluication (like redmine) will sync from ldap but never too.
Why don't u use plain redmine user base without ldap then?

I use a SSO with LDAP, We have centralized the creation of users with LDAP. When I delete an account, I delete in LDAP.
When I use Redmin, I use authentication with LDAP. Redmine work well to create a new account with LDAP authentication. But, when I delete user in my LDAP, This user is always present in Redmine. I think the feature (ldap authentication) is half-developed

#29 Updated by Jérôme BATAILLE over 5 years ago

You can check :
https://github.com/Utopism/redmine_ldap_sync

a plugin that brings a few of these features.

#30 Updated by Sebastian Paluch almost 2 years ago

+1

#31 Updated by Mischa The Evil 7 months ago

  • Duplicated by Defect #24840: LDAP: Users who didn't login yet, are not accesible (and groups where no user has logged in yet aren't available as well) to assign to a project added

#32 Updated by Svetly Minev 7 months ago

Is this patch working under Redmine 3.3.2 / 3.3.X ???

Or is there any other solution how to import all AD users in Redmine?

I will appreciate every help

Thank you in advance

Also available in: Atom PDF