Defect #21453

LDAP account creation fails when first name/last name contain non ASCII

Added by Haihan Ji almost 3 years ago. Updated over 2 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Jean-Philippe Lang% Done:

0%

Category:LDAP
Target version:3.2.1
Resolution:Fixed Affected version:3.2.0

Description

Maybe, redmine can add an encoding property to AuthSource.

$ ruby bin/about 

sh: svn: command not found
sh: darcs: command not found
sh: hg: command not found
sh: cvs: command not found
sh: bzr: command not found
sh: git: command not found
Environment:
  Redmine version                3.2.0.stable
  Ruby version                   2.2.1-p85 (2015-02-26) [x86_64-linux]
  Rails version                  4.2.5
  Environment                    production
  Database adapter               Mysql2
SCM:
  Filesystem                     
Redmine plugins:
  no plugin installed

LOG

Started GET "/auth_sources/autocomplete_for_new_user?term=xxxx" for 10.10.93.107 at 2015-12-08 10:12:17 +0800
Processing by AuthSourcesController#autocomplete_for_new_user as JSON
  Parameters: {"term"=>"xxxx"}
  SQL (1.7ms)  UPDATE `tokens` SET `tokens`.`updated_on` = '2015-12-08 10:12:17' WHERE `tokens`.`user_id` = 1 AND `tokens`.`value` = '5e869f0905478838585d8022d4347ea0cf68ee9e' AND `tokens`.`action` = 'session'
   (0.5ms)  SELECT MAX(`settings`.`updated_on`) FROM `settings`
  User Load (0.5ms)  SELECT  `users`.* FROM `users` WHERE `users`.`type` IN ('User', 'AnonymousUser') AND `users`.`status` = 1 AND `users`.`id` = 1 LIMIT 1
  Current user: admin (id=1)
  AuthSource Load (0.5ms)  SELECT `auth_sources`.* FROM `auth_sources`
Completed 500 Internal Server Error in 176ms (ActiveRecord: 3.2ms)

Encoding::UndefinedConversionError ("\xE5" from ASCII-8BIT to UTF-8):
  app/controllers/auth_sources_controller.rb:79:in `autocomplete_for_new_user'
  lib/redmine/sudo_mode.rb:63:in `sudo_mode'

  Rendered /usr/local/rvm/gems/ruby-2.2.1/gems/actionpack-4.2.5/lib/action_dispatch/middleware/templates/rescues/_source.erb (3.6ms)
  Rendered /usr/local/rvm/gems/ruby-2.2.1/gems/actionpack-4.2.5/lib/action_dispatch/middleware/templates/rescues/_trace.text.erb (0.8ms)
  Rendered /usr/local/rvm/gems/ruby-2.2.1/gems/actionpack-4.2.5/lib/action_dispatch/middleware/templates/rescues/_request_and_response.text.erb (0.9ms)
  Rendered /usr/local/rvm/gems/ruby-2.2.1/gems/actionpack-4.2.5/lib/action_dispatch/middleware/templates/rescues/diagnostics.text.erb (67.2ms)

The lastname is chinese.

$ rails c
2.2.1 :001 >   AuthSource.search(:xxxx)
  AuthSource Load (0.4ms)  SELECT `auth_sources`.* FROM `auth_sources`
 => [{:dn=>"************", :firstname=>" ", :lastname=>"\xE5\xA7\xAC\xE6\xB5\xB7\xE6\xB6\xB5", :mail=>"****", :auth_source_id=>1, :login=>"xxxx"}] 
2.2.1 :002 > AuthSource.search(:xxxx).to_json
  AuthSource Load (0.4ms)  SELECT `auth_sources`.* FROM `auth_sources`
Encoding::UndefinedConversionError: "\xE5" from ASCII-8BIT to UTF-8
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/core_ext/object/json.rb:34:in `encode'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/core_ext/object/json.rb:34:in `to_json'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/core_ext/object/json.rb:34:in `to_json_with_active_support_encoder'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/json/encoding.rb:57:in `to_json'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/json-1.8.3/lib/json/common.rb:223:in `generate'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/json-1.8.3/lib/json/common.rb:223:in `generate'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/json/encoding.rb:101:in `stringify'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/json/encoding.rb:35:in `encode'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/json/encoding.rb:22:in `encode'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.2.5/lib/active_support/core_ext/object/json.rb:37:in `to_json_with_active_support_encoder'
    from (irb):32
    from /usr/local/rvm/gems/ruby-2.2.1/gems/railties-4.2.5/lib/rails/commands/console.rb:110:in `start'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/railties-4.2.5/lib/rails/commands/console.rb:9:in `start'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:68:in `console'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    from /usr/local/rvm/gems/ruby-2.2.1/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>'
    from bin/rails:4:in `require'
    from bin/rails:4:in `<main>'2.2.1 :003 > 

Related issues

Related to Redmine - Feature #12811: ASCII-8BIT to UTF-8 problems -update net-ldap with net-ld... New
Duplicated by Redmine - Defect #21458: LDAP account creation fails when first name/last name con... Closed
Duplicated by Redmine - Defect #21528: LDAP user autocreation doesn't work with attributes with ... Closed

Associated revisions

Revision 15024
Added by Jean-Philippe Lang over 2 years ago

LDAP account creation fails when first name/last name contain non ASCII (#21453).

Patch by Haihan Ji.

Revision 15025
Added by Jean-Philippe Lang over 2 years ago

Stringify the attribute value before #force_enconding (#21453).

History

#1 Updated by Toshi MARUYAMA almost 3 years ago

  • Related to Feature #12811: ASCII-8BIT to UTF-8 problems -update net-ldap with net-ldap-1 -v0.4.0 added

#2 Updated by Toshi MARUYAMA almost 3 years ago

  • Subject changed from Encoding::UndefinedConversionError: "\xE5" from ASCII-8BIT to UTF-8 to LDAP: Encoding::UndefinedConversionError: "\xE5" from ASCII-8BIT to UTF-8

#3 Updated by Toshi MARUYAMA almost 3 years ago

  • Affected version set to 3.2.0

Try this patch.

diff --git a/app/models/auth_source_ldap.rb b/app/models/auth_source_ldap.rb
--- a/app/models/auth_source_ldap.rb
+++ b/app/models/auth_source_ldap.rb
@@ -84,7 +84,7 @@ class AuthSourceLdap < AuthSource
       attrs[:login] = AuthSourceLdap.get_attr(entry, self.attr_login)
       results << attrs
     end
-    results
+    results.collect {|item| item.force_encoding('UTF-8')}
   rescue Net::LDAP::LdapError => e
     raise AuthSourceException.new(e.message)
   end

#4 Updated by Haihan Ji almost 3 years ago

Toshi MARUYAMA wrote:

Try this patch.
[...]

This patch is not work, the item is a Hash, without 'force_encoding' method.

I think AuthSourceLdap.get_attr is a good place.

def self.get_attr(entry, attr_name)
  if !attr_name.blank?
    value = entry[attr_name].is_a?(Array) ? entry[attr_name].first : entry[attr_name]
    value.force_encoding('UTF-8')
  end
end

#5 Updated by yoshinobu yoshida almost 3 years ago

Toshi MARUYAMA wrote:

Try this patch.

I was patched. But i got an other error.

[root@redmine32 models]# diff auth_source_ldap.rb auth_source_ldap.rb.org
94c94
<     results.collect {|item| item.force_encoding('UTF-8')}
---
>     results

BEFORE

[2015-12-09 01:01:44.417073][15262] Started GET "/redmine/auth_sources/autocomplete_for_new_user?term=**********" for 192.168.16.253 at 2015-12-09 01:01:44 +0900
[2015-12-09 01:01:44.417846][15262] Processing by AuthSourcesController#autocomplete_for_new_user as JSON
[2015-12-09 01:01:44.417899][15262]   Parameters: {"term"=>"**********"}
[2015-12-09 01:01:44.422393][15262]   Current user: admin (id=1)
[2015-12-09 01:01:44.479656][15262] Completed 500 Internal Server Error in 62ms (ActiveRecord: 2.6ms)

[2015-12-09 01:01:44.480361][15262] Encoding::UndefinedConversionError ("\xE7" from ASCII-8BIT to UTF-8):
[2015-12-09 01:01:44.480361][15262]   app/controllers/auth_sources_controller.rb:79:in `autocomplete_for_new_user'
[2015-12-09 01:01:44.480361][15262]   lib/redmine/sudo_mode.rb:63:in `sudo_mode'

AFTER

[2015-12-09 00:56:35.824736][14847] Started GET "/redmine/auth_sources/autocomplete_for_new_user?term=**********" for 192.168.16.253 at 2015-12-09 00:56:35 +0900
[2015-12-09 00:56:35.825657][14847] Processing by AuthSourcesController#autocomplete_for_new_user as JSON
[2015-12-09 00:56:35.825702][14847]   Parameters: {"term"=>"**********"}
[2015-12-09 00:56:35.830259][14847]   Current user: admin (id=1)
[2015-12-09 00:56:35.879744][14847] Completed 500 Internal Server Error in 54ms (ActiveRecord: 2.7ms)

[2015-12-09 00:56:35.880394][14847] NoMethodError (undefined method `force_encoding' for #<Hash:0x007f2fdb8dd4e0>):
[2015-12-09 00:56:35.880394][14847]   app/models/auth_source_ldap.rb:94:in `block in search'
[2015-12-09 00:56:35.880394][14847]   app/models/auth_source_ldap.rb:94:in `collect'
[2015-12-09 00:56:35.880394][14847]   app/models/auth_source_ldap.rb:94:in `search'
[2015-12-09 00:56:35.880394][14847]   app/models/auth_source.rb:61:in `block in search'
[2015-12-09 00:56:35.880394][14847]   app/models/auth_source.rb:58:in `search'
[2015-12-09 00:56:35.880394][14847]   app/controllers/auth_sources_controller.rb:77:in `autocomplete_for_new_user'
[2015-12-09 00:56:35.880394][14847]   lib/redmine/sudo_mode.rb:63:in `sudo_mode'


Haihan Ji wrote:

This patch is not work, the item is a Hash, without 'force_encoding' method.
I think AuthSourceLdap.get_attr is a good place.

I fixed, Thanks.

#6 Updated by Toshi MARUYAMA almost 3 years ago

  • Target version set to 3.2.1

#7 Updated by Toshi MARUYAMA almost 3 years ago

  • Duplicated by Defect #21458: LDAP account creation fails when first name/last name contain umlaut added

#8 Updated by Toshi MARUYAMA almost 3 years ago

  • Subject changed from LDAP: Encoding::UndefinedConversionError: "\xE5" from ASCII-8BIT to UTF-8 to LDAP account creation fails when first name/last name contain non ASCII

#9 Updated by Alexander Ryabinovskiy almost 3 years ago

I confirm this bug, redmine 3.2.0 with net-ldap-0.12.1. If I rollback on net-ldap-0.3.1 - all is ok.
The LDAP server response is identical in both cases.
Haihan Ji wrote:

I think AuthSourceLdap.get_attr is a good place.

Yes, this works, thank you!
UPD:
Maybe more general solution exists? Maybe we need to use other version of net-ldap gem, or fix this gem?
I see the same error in Redmine Ldap Sync plugin:

ActionView::Template::Error (incompatible character encodings: UTF-8 and ASCII-8BIT):
    1: <% @test.users_at_ldap.each do |(login, data)| -%>
    2: <%= "#{l(:label_user)} \"#{login}\"" %>: <%= data == :not_found ?
    3:     "#{l(:label_not_found)}\n" :
    4:     "\n#{user_fields_list data[:fields], data[:groups]}" %>
    5: <% end -%>
    6: <% @test.groups_at_ldap.each do |(name, data)| -%>
    7: <%= "#{l(:label_group)} \"#{name}\"" %>: <%= data == :not_found ?
  lib/redmine/sudo_mode.rb:63:in `sudo_mode'

UPD2:
btw, here is the 4-years old issue related to this problem, in net-ldap github:

https://github.com/ruby-ldap/ruby-net-ldap/issues/4

It will be good, if someone can help to fix this net-ldap bug?

#10 Updated by Toshi MARUYAMA almost 3 years ago

  • Duplicated by Defect #21528: LDAP user autocreation doesn't work with attributes with Cyrillic added

#11 Updated by Aleksandr Yarkov over 2 years ago

Haihan Ji wrote:

I think AuthSourceLdap.get_attr is a good place.
[...]

This Fix don't work to me until i convert result to a string.

value.to_s.force_encoding('UTF-8')

#12 Updated by Jean-Philippe Lang over 2 years ago

  • Status changed from New to Resolved
  • Assignee set to Jean-Philippe Lang
  • Resolution set to Fixed

I've applied the fix from Haihan Ji and the comment above, thanks.

#13 Updated by Johan Eidenvall over 2 years ago

Edit after comment 15: Oops, I missed r15025 somehow... Sorry for taking up your time.

I needed national chars in the Firs/Last Name fields from LDAP, so I cherry picked the changed in r15024 (which is the change that Haihan Ji suggested above).

However this does not play nice with cases when a field is unavailable from the LDAP server.

We don't have the email property set for all LDAP accounts in the AD, but it is set for some of the users so I wanted to use the property when available.

Before this patch: if the email property was set accounts are auto-created without any further questions, and without the email set the user is asked for it when auto-creating his account.

Now, after the patch, if the email property is not set I get the following error message in the log during account auto-creation:

Error during authentication: undefined method `force_encoding' for nil:NilClass

I changed the code to only call the force_encoding method if value is non-nil:

  def self.get_attr(entry, attr_name)
    if !attr_name.blank?
      value = entry[attr_name].is_a?(Array) ? entry[attr_name].first : entry[attr_name]
      value.force_encoding('UTF-8') unless value.nil?
    end
  end

(This is the first line of Ruby I've written and I am not sure I did it right... But i think it solves the problem by not calling force_encoding if value is nil.)

#14 Updated by Toshi MARUYAMA over 2 years ago

  • Status changed from Resolved to New

#15 Updated by Jean-Philippe Lang over 2 years ago

  • Status changed from New to Resolved

You need to apply r15025 as well to prevent this error.

#16 Updated by Jean-Philippe Lang over 2 years ago

  • Status changed from Resolved to Closed

Merged in 3.2-stable.

Also available in: Atom PDF