hadoop - want help for running MapReduce programs on Google Cloud storage -


i using google cloud storage hadoop 2.3.0 using gcs connector.

i have added gcs.jar lib directory of hadoop installation added path gcs connector in hadoop-env.sh file as:

export hadoop_classpath=${hadoop_classpath:"/share/hadoop/common/lib/gcs_connector"}  

i had made changes core-site.xml file of hadoop installation as:

   <property>    <name>fs.defaultfs</name>    <value>hdfs://127.0.0.1:9000</value>    </property>    <property>    <name>fs.gs.impl</name>    <value>com.google.cloud.hadoop.fs.gcs.googlehadoopfilesystem</value>    <description>the filesystem gs: (gcs) uris.</description>    </property>    <property>    <name>fs.abstractfilesystem.gs.impl</name>    <value>com.google.cloud.hadoop.fs.gcs.googlehadoopfs</value>    <description>the abstractfilesystem gs: (gcs) uris. necessary use hadoop 2.   </description>   </property>   <property>   <name>fs.gs.impl</name>   <value>com.google.cloud.hadoop.fs.gcs.googlehadoopfilesystem</value>   </property>   <property>   <name>fs.gs.project.id</name>   <value>1113</value>   </property>   <property>   <name>fs.gs.system.bucket</name>   <value>hadoop1</value>   </property>   <property>   <name>fs.gs.working.dir</name>   <value>/</value>   </property>   <property>   <name>fs.gs.auth.service.account.enable</name>   <value>true</value>   </property>   <property>   <name>fs.gs.auth.service.account.email</name>   <value>7168543aovnjqaf1e7sumil@developer.gserviceaccount.com</value>   </property>   <property>   <name>fs.gs.auth.service.account.keyfile</name>   <value>c://hadoop-2.3.0/etc/hadoop/gcskey.p12</value>   </property>   <property>   <name>fs.gs.auth.client.id</name>   <value>7168543aovnjqaf1e7sumil.apps.googleusercontent.com</value>    </property> 

the billing account created project enabled.

i created bucket , contents of buckets visible me using:

hadoop fs -ls gs://hadoop1  

i had tried hadoop click-to-deploy option of master , worker nodes hadoop , vm instances created.

i had installed gcloud auth login. git repositories created.

i had followed mapreduce article posted on google, not helpful complete guidance.

question: want run mapreduce programs developed in java using deployed hadoop on cloud? path provide in programs input , output files?

my programs working on hadoop platform on system.


Comments