Yarn Capacity Scheduler

1. /etc/hadoop/conf/yarn-site.xml                                                                                                                                                                                    yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

2. Capacity Scheduler settings (/etc/hadoop/conf/capacity-scheduler.xml on resource manager host)

  

   a) Queues are fundamental unit of scheduling in yarn

               yarn.scheduler.capacity.root.queues = support,engineering,marketing     (parent queues)

                       yarn.scheduler.capacity.support.queues=training,services                (leaf queues)

                       yarn.scheduler.capacity.engineering=development,qa                       (leaf queues)

                       yarn.scheduler.capacity.marketing=sales,advertising                         (leaf queues)

 

             b) Capacity Scheduler ACL's

                        

                         Queue users:

                         yarn.scheduler.capacity.root.acl_submit_application=      (asterisk == all users and groups & empty space == deny all users and groups)

                                  yarn.scheduler.capacity.support.acl_submit_applications= sherlock,pacioli,njella hg_group,cd_group

                                  yarn.scheduler.capacity.engineering.acl_submit_applications=user1,user2,user3 group1,group2,group3

                                  yarn.scheduler.capacity.marketing.acl_submit_applications= user4,user2,user5, group4,group5,group6

               Queue Administers

                         yarn.scheduler.capacity.root.acl_administer_queue=

                                  yarn.scheduler.capacity.support.acl_administer_queue= support_admin_group

                                  yarn.scheduler.capacity.engineering.acl_administer_queue=engineering_admin_group

                                  yarn.scheduler.capacity.marketing.acl_administer_queue=engineering_Admin_grp

   c) Queue Mappings:

               

               Setting Up Queue Mappings

                         yarn.scheduler.capacity.queue-mappings=u:njella:engineering,u:chitra:marketing,g:engineering_group:engineering

                         If group name is same as queue name then

                        yarn.scheduler.capacity.queue-mappings=u:%user:%primary_group                       

                        If user name is same as queue name then

                        yarn.scheduler.capacity.queue-mappings=u:%user:%user                       

               Enable queue mapping override:

                            yarn.scheduler.capacity.queue-mappings-override.enable=true

                         Example:                    

                                  <property>

                                          <name>yarn.scheduler.capacity.queue-mappings</name>

                                          <value>u:njella:engineering, g:webadmins:weblog</value> 

                                  </proprty>

                                            

                                            If user "njella" explicitly submits an application to the marketing queue, the default queue assignment of engineering is overriden and the  application is submitted to the marketing queue.

                     

Issues

1. Ambari-agent



Sol : Change verify=platform_default   to verify=disable  in /etc/pythoncert-verification.cfg file

1. sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

-------------------------------------------------------------------------------------