NFSv4 idmap - in-kernel keyring issues

Recently I encountered with a NFS issue, where some files on NFS mounts displayed with UID/GID of value 4294967294

It was on a RHEL 6.5 client and further investigation lead us to bug 1033708

From RHEL 6.3 onwards, Redhat has dropped rpc.idmapd daemon and instead uses in-kernel keyring for NFSv4 ID mapping. /usr/sbin/nfsidmap program is called for lookups and is configured via  /etc/request-key.d/id_resolver.conf

$ cat /etc/request-key.d/id_resolver.conf
#
# nfsidmap(5) - The NFS idmapper upcall program
# Summary: Used by NFSv4 to map user/group ids into
#          user/group names and names into in ids
# Options:
# -v         Increases the verbosity of the output to syslog
# -t timeout Set the expiration timer, in seconds, on the key
#
create    id_resolver    *         *    /usr/sbin/nfsidmap %k %d

default values for the keyring are very small (200) - so, if your environment has to map more than 200 NFS uid's - you hit the bug and the code just returns -2, which translates to 4294967294

To fix the issue, you need to update your nfs-utils,nfs-utils-lib rpm's and update kernel tunables (shown below) for key-ring values

kernel.keys.maxkeys = 65536
kernel.keys.maxbytes = 4194304
kernel.keys.root_maxkeys = 65536

kernel.keys.root_maxbytes = 4194304

usage of key-ring could be seen via /proc filesystem

$ cat /proc/keys

141e035e I--Q--     6 perm 1f3f0000 994295551 50000 keyring   _ses: 1/4

180127d1 I--Q--     4 perm 1f3f0000 994295551    -1 keyring   _uid.994295551: empty

21a48ca1 I--Q--     2 perm 1f3f0000 994295551 50000 keyring   _ses: 1/4

3765083a I--Q--     1 perm 1f3f0000 994295551    -1 keyring   _uid_ses.994295551: 1/4

$ cat /proc/key-users

    0:    13 12/12 9/65536 259/4194304

12341:     3 3/3 3/65536 83/4194304

994295551:     4 4/4 4/65536 152/4194304

Here I have setup 64k keys