本文介紹了要緩沖或字節流的拼圖編寫器的處理方法,對大家解決問題具有一定的參考價值,需要的朋友們下面隨著小編來一起學習吧!
問題描述
我有一個Java應用程序,可以將JSON消息轉換為PARQUET格式。Java中有沒有可以寫入緩沖區或字節流的拼圖寫入器?在大多數示例中,我都看到過寫入文件。
推薦答案
TLDR;您需要實現OutputFile
,例如類似于:
的內容
import org.apache.parquet.io.OutputFile;
import org.apache.parquet.io.PositionOutputStream;
import java.io.BufferedOutputStream;
import java.io.IOException;
public class ParquetBufferedWriter implements OutputFile {
private final BufferedOutputStream out;
public ParquetBufferedWriter(BufferedOutputStream out) {
this.out = out;
}
@Override
public PositionOutputStream create(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
private PositionOutputStream createPositionOutputstream() {
return new PositionOutputStream() {
@Override
public long getPos() throws IOException {
return 0;
}
@Override
public void write(int b) throws IOException {
out.write(b);
}
};
}
@Override
public PositionOutputStream createOrOverwrite(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
@Override
public boolean supportsBlockSize() {
return false;
}
@Override
public long defaultBlockSize() {
return 0;
}
}
您的作者應該是這樣的:
ParquetBufferedWriter out = new ParquetBufferedWriter();
try (ParquetWriter<Record> writer = AvroParquetWriter.
<Record>builder(out)
.withRowGroupSize(DEFAULT_BLOCK_SIZE)
.withPageSize(DEFAULT_PAGE_SIZE)
.withSchema(SCHEMA)
.build()) {
for (Record record : records) {
writer.write(record);
}
} catch (IOException e) {
throw new IllegalStateException(e);
}
這篇關于要緩沖或字節流的拼圖編寫器的文章就介紹到這了,希望我們推薦的答案對大家有所幫助,