HDFS Commands

[hdfs@sandbox-hdp ~]$ hdfs
Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]

  OPTIONS is none or any of:

--buildpaths                       attempt to add class files from build tree
--config dir                       Hadoop config directory
--daemon (start|status|stop)       operate on a daemon
--debug                            turn on shell script debug mode
--help                             usage information
--hostnames list[,of,host,names]   hosts to use in worker mode
--hosts filename                   list of hosts to use in worker mode
--loglevel level                   set the log4j level for this command
--workers                          turn on worker mode

  SUBCOMMAND is one of:


    Admin Commands:

cacheadmin           configure the HDFS cache
crypto               configure HDFS encryption zones
debug                run a Debug Admin to execute HDFS debug commands
dfsadmin             run a DFS admin client
dfsrouteradmin       manage Router-based federation
ec                   run a HDFS ErasureCoding CLI
fsck                 run a DFS filesystem checking utility
haadmin              run a DFS HA admin client
jmxget               get JMX exported values from NameNode or DataNode.
oev                  apply the offline edits viewer to an edits file
oiv                  apply the offline fsimage viewer to an fsimage
oiv_legacy           apply the offline fsimage viewer to a legacy fsimage
storagepolicies      list/get/set block storage policies

    Client Commands:

classpath            prints the class path needed to get the hadoop jar and the required libraries
dfs                  run a filesystem command on the file system
envvars              display computed Hadoop environment variables
fetchdt              fetch a delegation token from the NameNode
getconf              get config values from configuration
groups               get the groups which users belong to
lsSnapshottableDir   list all snapshottable dirs owned by the current user
snapshotDiff         diff two snapshots of a directory or diff the current directory contents with a snapshot
version              print the version

    Daemon Commands:

balancer             run a cluster balancing utility
datanode             run a DFS datanode
dfsrouter            run the DFS router
diskbalancer         Distributes data evenly among disks on a given node
httpfs               run HttpFS server, the HDFS HTTP Gateway
journalnode          run the DFS journalnode
mover                run a utility to move block replicas across storage types
namenode             run the DFS namenode
nfs3                 run an NFS version 3 gateway
portmap              run a portmap service
secondarynamenode    run the DFS secondary namenode
zkfc                 run the ZK Failover Controller daemon

SUBCOMMAND may print help when invoked w/o parameters or with -h.
[hdfs@sandbox-hdp ~]$



Admin Commands:


Client Commands:

Daemon Commands:

Ambari Commands

Ambari Status	/sbin/service ambari-server status
Start Ambari	/sbin/service ambari-server start
Stop Amabri	/sbin/service ambari-server stop
Reset ambari admin pwd	ambari-admin-password-reset
Ambari ldap sync	ambari-server sync-ldap --users users.txt --groups groups.txt
ambari-server --version	Ambari version
ambari-server --hash	Ambari server hash value
Ambari-server backup	To take backup of ambari settings

Decommission a data node in HDP

From Ambari :

Manually :
[root@namenode1 conf]# pwd
/etc/hadoop/conf
[root@namenode1 conf]# ll | grep dfs.exclude
-rw-r--r--. 1 hdfs hadoop     1 Jan 12 15:32 dfs.exclude
[root@namenode1 conf]# cat dfs.exclude

[root@namenode1 conf]# vi dfs.exclude
[root@namenode1 conf]# cat dfs.exclude
edgenode.hdp.cn
[root@namenode1 conf]# su - hdfs
Last login: Sun Jan 12 15:33:08 EST 2020
[hdfs@namenode1 ~]$ hdfs dfsadmin -refreshNodes
Refresh nodes successful

[hdfs@namenode1 ~]$

Disable Resource Manager HA

1. Stop YARN and Zookeeper services from ambari
2. From Ambari-server :

                                           /var/lib/ambari-server/resources/scripts/configs.py --user=admin --     
                                                             password=admin --host=edgenode.hdp.cn --cluster=hdpdev --
                                                             action=get --config-      type=yarn-site -f yarn-site.json

3. From the below change the first property to value "false" and remove the other properties from yarn-site.json

                                       a) "yarn.resourcemanager.ha.enabled": "false",

                                  b) "yarn.resourcemanager.ha.rm-ids": "rm1,rm2",

                                  c) "yarn.resourcemanager.hostname.rm1": "datanode1.hdp.cn",

                                  d) "yarn.resourcemanager.hostname.rm2": "edgenode.hdp.cn",

                                  e)  "yarn.resourcemanager.webapp.address.rm1": "datanode1.hdp.cn:8088",

                                  f)  "yarn.resourcemanager.webapp.address.rm2": "edgenode.hdp.cn:8088",

                                 g) "yarn.resourcemanager.webapp.https.address.rm1": "datanode1.hdp.cn:8090",

                                 h) "yarn.resourcemanager.webapp.https.address.rm2": "edgenode.hdp.cn:8090",

                                 i) "yarn.resourcemanager.cluster-id": "yarn-cluster",

                                 j) "yarn.resourcemanager.ha.automatic-failover.zk-base-path": "/yarn-leader-election",

4. Set the below properties to existing resource manager

                                   a)  "yarn.resourcemanager.hostname":

                               b) "yarn.resourcemanager.admin.address":

                               c) "yarn.resourcemanager.webapp.address":                                                             

                               d)"yarn.resourcemanager.resource-tracker.address":                 

                               e)"yarn.resourcemanager.scheduler.address":

                                f)"yarn.resourcemanager.webapp.https.address":

                                g)"yarn.timeline-service.webapp.address":

                                h)"yarn.timeline-service.webapp.https.address":

                                 i)"yarn.timeline-service.address":

                                 j)"yarn.log.server.url":

5. Copy the yarn-site.json file back to the ambari-server and run the below command to set the changes made

                                            /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --                                           host=edgenode.hdp.cn --cluster=hdpdev --action=get --config-type=yarn-site

6. Delete the Resource Manager host comonent

                                         curl --user admin:admin -i -H "X-Requested-By: ambari" -X  DELETE  
                                    http://edgenode.hdp.cn:8080/api/v1/clusters/hdpdev/hosts/edgenode.hdp.cn/
                                    host_components/RESOURCEMANAGER
                                                                                                                                                                                                        

7. Start zookeeper service from Ambari.

8. On  zookeeper clients run the below command to change the znode permissions

                                       /usr/hdp/current/zookeeper-client/bin/zkCli.sh getAcl /rmstore/ZKRMStateRoot

                                 /usr/hdp/current/zookeeper-client/bin/zkCli.sh setAcl /rmstore/ZKRMStateRoot                                                       world:anyone:rwcda

9. From Ambari UI, restart zookeeper service and YARN service

Yarn Capacity Scheduler

1. /etc/hadoop/conf/yarn-site.xml                                                                                                                                                                                    yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

2. Capacity Scheduler settings (/etc/hadoop/conf/capacity-scheduler.xml on resource manager host)

  

   a) Queues are fundamental unit of scheduling in yarn

               yarn.scheduler.capacity.root.queues = support,engineering,marketing     (parent queues)

                       yarn.scheduler.capacity.support.queues=training,services                (leaf queues)

                       yarn.scheduler.capacity.engineering=development,qa                       (leaf queues)

                       yarn.scheduler.capacity.marketing=sales,advertising                         (leaf queues)

 

             b) Capacity Scheduler ACL's

                        

                         Queue users:

                         yarn.scheduler.capacity.root.acl_submit_application=      (asterisk == all users and groups & empty space == deny all users and groups)

                                  yarn.scheduler.capacity.support.acl_submit_applications= sherlock,pacioli,njella hg_group,cd_group

                                  yarn.scheduler.capacity.engineering.acl_submit_applications=user1,user2,user3 group1,group2,group3

                                  yarn.scheduler.capacity.marketing.acl_submit_applications= user4,user2,user5, group4,group5,group6

               Queue Administers

                         yarn.scheduler.capacity.root.acl_administer_queue=

                                  yarn.scheduler.capacity.support.acl_administer_queue= support_admin_group

                                  yarn.scheduler.capacity.engineering.acl_administer_queue=engineering_admin_group

                                  yarn.scheduler.capacity.marketing.acl_administer_queue=engineering_Admin_grp

   c) Queue Mappings:

               

               Setting Up Queue Mappings

                         yarn.scheduler.capacity.queue-mappings=u:njella:engineering,u:chitra:marketing,g:engineering_group:engineering

                         If group name is same as queue name then

                        yarn.scheduler.capacity.queue-mappings=u:%user:%primary_group                       

                        If user name is same as queue name then

                        yarn.scheduler.capacity.queue-mappings=u:%user:%user                       

               Enable queue mapping override:

                            yarn.scheduler.capacity.queue-mappings-override.enable=true

                         Example:                    

                                  <property>

                                          <name>yarn.scheduler.capacity.queue-mappings</name>

                                          <value>u:njella:engineering, g:webadmins:weblog</value> 

                                  </proprty>

                                            

                                            If user "njella" explicitly submits an application to the marketing queue, the default queue assignment of engineering is overriden and the  application is submitted to the marketing queue.

                     

Issues

1. Ambari-agent



Sol : Change verify=platform_default   to verify=disable  in /etc/pythoncert-verification.cfg file

1. sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

-------------------------------------------------------------------------------------

How to Synchronize the system clock to Network Time Protocol (NTP) in Linux.

1.  If you do not have the ntpd service enabled use the below command to install and enable..

   a) yum install ntp.
   b) systemctl enable ntpd.

 

2. Modify the  /etc/ntp.conf  file as shown in the below image.  If you are using Cent OS 7, please modify the  cat /etc/ntp/step-tickers.

        You can also find the preferred ones for your location at http://www.pool.ntp.org/

3. Start the ntpd service

    systemctl restart ntpd