• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

Java PartitionChunk类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中io.druid.timeline.partition.PartitionChunk的典型用法代码示例。如果您正苦于以下问题:Java PartitionChunk类的具体用法?Java PartitionChunk怎么用?Java PartitionChunk使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



PartitionChunk类属于io.druid.timeline.partition包,在下文中一共展示了PartitionChunk类的4个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: getShardSpec

import io.druid.timeline.partition.PartitionChunk; //导入依赖的package包/类
private static ShardSpec getShardSpec(final int partitionNum)
{
  return new ShardSpec()
  {
    @Override
    public <T> PartitionChunk<T> createChunk(T obj)
    {
      return null;
    }

    @Override
    public boolean isInChunk(long timestamp, InputRow inputRow)
    {
      return false;
    }

    @Override
    public int getPartitionNum()
    {
      return partitionNum;
    }

    @Override
    public ShardSpecLookup getLookup(List<ShardSpec> shardSpecs)
    {
      return null;
    }
  };
}
 
开发者ID:druid-io,项目名称:druid-api,代码行数:30,代码来源:DataSegmentTest.java


示例2: getSplits

import io.druid.timeline.partition.PartitionChunk; //导入依赖的package包/类
@Override
public List<InputSplit> getSplits(final JobContext jobContext) throws IOException, InterruptedException
{
  final Configuration conf = jobContext.getConfiguration();
  final String dataSource = getDataSource(conf);
  final List<Interval> intervals = getIntervals(conf);
  final List<DataSegment> segments;
  try (final HttpClientHolder httpClient = HttpClientHolder.create()) {
    segments = new DruidMetadataClient(
        httpClient.get(),
        objectMapper(),
        getCoordinatorHost(conf)
    ).usedSegments(dataSource, intervals);
  }
  catch (Exception e) {
    throw Throwables.propagate(e);
  }

  log.info(
      "Got %,d used segments for dataSource[%s], intervals[%s] from coordinator.",
      segments.size(),
      dataSource,
      Joiner.on(", ").join(intervals)
  );

  // Window the DataSegments by putting them in a timeline.
  final VersionedIntervalTimeline<String, DataSegment> timeline = new VersionedIntervalTimeline<>(Ordering.natural());
  for (DataSegment segment : segments) {
    timeline.add(segment.getInterval(), segment.getVersion(), segment.getShardSpec().createChunk(segment));
  }

  final List<InputSplit> splits = Lists.newArrayList();

  for (Interval interval : intervals) {
    final List<TimelineObjectHolder<String, DataSegment>> lookup = timeline.lookup(interval);
    for (final TimelineObjectHolder<String, DataSegment> holder : lookup) {
      for (final PartitionChunk<DataSegment> chunk : holder.getObject()) {
        final WindowedDataSegment windowedDataSegment = new WindowedDataSegment(
            chunk.getObject(),
            holder.getInterval()
        );
        splits.add(DruidInputSplit.create(windowedDataSegment));
      }
    }
  }

  log.info(
      "Found %,d splits for dataSource[%s], intervals[%s].",
      splits.size(),
      dataSource,
      Joiner.on(", ").join(intervals)
  );

  return splits;
}
 
开发者ID:implydata,项目名称:druid-hadoop-inputformat,代码行数:56,代码来源:DruidInputFormat.java


示例3: createChunk

import io.druid.timeline.partition.PartitionChunk; //导入依赖的package包/类
@Override
public <T> PartitionChunk<T> createChunk(T obj) {
    throw new NotImplementedException("createChunk method is not implemented");
}
 
开发者ID:yahoo,项目名称:fili,代码行数:5,代码来源:NumberedShardSpec.java


示例4: getSplits

import io.druid.timeline.partition.PartitionChunk; //导入依赖的package包/类
@Override
public List<InputSplit> getSplits(JobContext context) throws IOException, InterruptedException
{
  Configuration conf = context.getConfiguration();

  String overlordUrl = conf.get(CONF_DRUID_OVERLORD_HOSTPORT);
  Preconditions.checkArgument(
      overlordUrl != null && !overlordUrl.isEmpty(),
      CONF_DRUID_OVERLORD_HOSTPORT + " not defined"
  );
  logger.info("druid overlord url = " + overlordUrl);

  String schemaStr = conf.get(CONF_DRUID_SCHEMA);
  Preconditions.checkArgument(
      schemaStr != null && !schemaStr.isEmpty(),
      "schema undefined,  provide " + CONF_DRUID_SCHEMA
  );
  logger.info("schema = " + schemaStr);

  DatasourceIngestionSpec ingestionSpec = HadoopDruidIndexerConfig.JSON_MAPPER.readValue(
      schemaStr,
      DatasourceIngestionSpec.class
  );
  String segmentsStr = getSegmentsToLoad(
      ingestionSpec.getDataSource(),
      Iterables.getOnlyElement(ingestionSpec.getIntervals()),
      overlordUrl
  );
  logger.info(String.format("segments list received from overlord = [%s]", segmentsStr));

  List<DataSegment> segmentsList = (HadoopDruidIndexerConfig.JSON_MAPPER.readValue(
      segmentsStr,
      Segments.class
  )).getSegments();
  VersionedIntervalTimeline<String, DataSegment> timeline = new VersionedIntervalTimeline<>(Ordering.natural());
  for (DataSegment segment : segmentsList) {
    timeline.add(segment.getInterval(), segment.getVersion(), segment.getShardSpec().createChunk(segment));
  }
  final List<TimelineObjectHolder<String, DataSegment>> timeLineSegments = timeline.lookup(Iterables.getOnlyElement(
                                                                                               ingestionSpec.getIntervals()
                                                                                           ));
  final List<WindowedDataSegment> windowedSegments = new ArrayList<>();
  for (TimelineObjectHolder<String, DataSegment> holder : timeLineSegments) {
    for (PartitionChunk<DataSegment> chunk : holder.getObject()) {
      windowedSegments.add(new WindowedDataSegment(chunk.getObject(), holder.getInterval()));
    }
  }

  conf.set(CONF_INPUT_SEGMENTS, HadoopDruidIndexerConfig.JSON_MAPPER.writeValueAsString(windowedSegments));

  return super.getSplits(context);
}
 
开发者ID:himanshug,项目名称:druid-hadoop-utils,代码行数:53,代码来源:DruidInputFormat.java



注:本文中的io.druid.timeline.partition.PartitionChunk类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java BeamChannel类代码示例发布时间:2022-05-15
下一篇:
Java TaxonomyTypeCollection类代码示例发布时间:2022-05-15
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap