i using google cloud storage hadoop 2.3.0 using gcs connector.
i have added gcs.jar lib directory of hadoop installation added path gcs connector in hadoop-env.sh
file as:
export hadoop_classpath=${hadoop_classpath:"/share/hadoop/common/lib/gcs_connector"}
i had made changes core-site.xml
file of hadoop installation as:
<property> <name>fs.defaultfs</name> <value>hdfs://127.0.0.1:9000</value> </property> <property> <name>fs.gs.impl</name> <value>com.google.cloud.hadoop.fs.gcs.googlehadoopfilesystem</value> <description>the filesystem gs: (gcs) uris.</description> </property> <property> <name>fs.abstractfilesystem.gs.impl</name> <value>com.google.cloud.hadoop.fs.gcs.googlehadoopfs</value> <description>the abstractfilesystem gs: (gcs) uris. necessary use hadoop 2. </description> </property> <property> <name>fs.gs.impl</name> <value>com.google.cloud.hadoop.fs.gcs.googlehadoopfilesystem</value> </property> <property> <name>fs.gs.project.id</name> <value>1113</value> </property> <property> <name>fs.gs.system.bucket</name> <value>hadoop1</value> </property> <property> <name>fs.gs.working.dir</name> <value>/</value> </property> <property> <name>fs.gs.auth.service.account.enable</name> <value>true</value> </property> <property> <name>fs.gs.auth.service.account.email</name> <value>7168543aovnjqaf1e7sumil@developer.gserviceaccount.com</value> </property> <property> <name>fs.gs.auth.service.account.keyfile</name> <value>c://hadoop-2.3.0/etc/hadoop/gcskey.p12</value> </property> <property> <name>fs.gs.auth.client.id</name> <value>7168543aovnjqaf1e7sumil.apps.googleusercontent.com</value> </property>
the billing account created project enabled.
i created bucket , contents of buckets visible me using:
hadoop fs -ls gs://hadoop1
i had tried hadoop click-to-deploy option of master , worker nodes hadoop , vm instances created.
i had installed gcloud auth login. git repositories created.
i had followed mapreduce article posted on google, not helpful complete guidance.
question: want run mapreduce programs developed in java using deployed hadoop on cloud? path provide in programs input , output files?
my programs working on hadoop platform on system.
Comments
Post a Comment