本文介紹了如何為org.apache.parquet.avro.AvroParquetReader配置S3訪問(wèn)？的處理方法，對(duì)大家解決問(wèn)題具有一定的參考價(jià)值，需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)吧！
問(wèn)題描述
我在這個(gè)問(wèn)題上掙扎了一段時(shí)間，想分享我的解決方案。AvroParquetReader是一個(gè)很好的閱讀Parquet的工具，但它對(duì)S3訪問(wèn)的默認(rèn)設(shè)置很弱：
java.io.InterruptedIOException: doesBucketExist on MY_BUCKET: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: Unable to load credentials from service endpoint
我想使用類似于com.amazonaws.auth.profile.ProfileCredentialsProvider，使用的憑據(jù)提供程序，它用于訪問(wèn)我的S3存儲(chǔ)桶，但從AvroParquetReader的類定義或文檔中不清楚我將如何實(shí)現(xiàn)這一點(diǎn)。
推薦答案
此代碼適用于我。它允許AvroParquetReader使用ProfileCredentialsProvider訪問(wèn)S3。
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import org.apache.parquet.avro.AvroParquetReader;
import org.apache.parquet.hadoop.ParquetReader;
import org.apache.hadoop.fs.Path;
import org.apache.avro.generic.GenericRecord;
import org.apache.hadoop.conf.Configuration;
...
final String path = "s3a://"+bucketName+"/"+pathName;
final Configuration configuration = new Configuration();
configuration.setClass("fs.s3a.aws.credentials.provider", ProfileCredentialsProvider.class,
AWSCredentialsProvider.class);
ParquetReader<GenericRecord> parquetReader =
AvroParquetReader.<GenericRecord>builder(new Path(path)).withConf(configuration).build();
這篇關(guān)于如何為org.apache.parquet.avro.AvroParquetReader配置S3訪問(wèn)？的文章就介紹到這了，希望我們推薦的答案對(duì)大家有所幫助，
