Configure ItemWriter, ItemReader, Step and Job dynamically in Spring Batch

Question

I'm working on process which uses Spring Integration and Spring Batch

1)Using Spring integration I will poll remote sftp dir to get different csv files as Message

2)Message which carries csv file as payload is sent downstream to Transformer which will transform Message to JobLaunchRequest

3)Spring batch reads csv files and dumps into DB

Question: For each csv file I need to configure (ItemReader, ItemWriter, Step, Job) So with that into consideration if I have to deal with 10 different csv files do I have to configure all 4 beans listed above for each csv? CSVs differs in HeaderNames and HeaderCount and each csv has different JPA Entity

Eventually I will have 40 @Bean Configurations which ideally I think is bad Can anyone suggest me if this is how spring batch is made to work or there is other way to make it one common dynamic bean for different CSVs

Here is code:

IntegartionFlow:

 @Bean
public IntegrationFlow integrationFlow(JobLaunchingGateway jobLaunchingGateway) {
    return IntegrationFlows.from(Sftp.inboundAdapter(sftpSessionFactory)
                    .remoteDirectory("/uploads")
                    .localDirectory(new File("C:\Users\DELL\Desktop\local"))
                    .patternFilter("*.csv")
                    .autoCreateLocalDirectory(true)
            , c -> c.poller(Pollers.fixedRate(1000).taskExecutor(taskExecutor()).maxMessagesPerPoll(1)))
            .transform(fileMessageToJobRequest())
            .handle(jobLaunchingGateway)
            .log(LoggingHandler.Level.WARN, "headers.id + ': ' + payload")
            .route(JobExecution.class, j -> j.getStatus().isUnsuccessful() ? "jobFailedChannel" : "jobSuccessfulChannel")
            .get();
}

Transformer:

 @Transformer
public JobLaunchRequest toRequest(Message<File> message) {
    JobParametersBuilder jobParametersBuilder =
            new JobParametersBuilder();

    jobParametersBuilder.addString(fileParameterName,
            message.getPayload().getAbsolutePath());
    jobParametersBuilder.addLong("key.id", System.currentTimeMillis());

    return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}

Batch Job:

 @Bean
public Job vendorMasterBatchJob(Step vendorMasterStep) {
    return jobBuilderFactory.get("vendorMasterBatchJob")
            .incrementer(new RunIdIncrementer())
            .start(vendorMasterStep)
            .listener(deleteInputFileJobListener)
            .build();
}

Batch Step:

  @Bean
public Step vendorMasterStep(FlatFileItemReader<ERPVendorMaster> vendorMasterReader,
                             JpaItemWriter<ERPVendorMaster> vendorMasterWriter) {
    return stepBuilderFactory.get("vendorMasterStep")
            .<ERPVendorMaster, ERPVendorMaster>chunk(chunkSize)
            .reader(vendorMasterReader)
            .writer(vendorMasterWriter)
            .faultTolerant()
            .skipLimit(Integer.MAX_VALUE)
            .skip(RuntimeException.class)
            .listener(skipListener)
            .build();
}

ItemWriter:

 @Bean
public JpaItemWriter<ERPVendorMaster> vendorMasterWriter() {
    return new JpaItemWriterBuilder<ERPVendorMaster>()
            .entityManagerFactory(entityManagerFactory)
            .build();
}

ItemReader:

 @Bean
@StepScope
public FlatFileItemReader<ERPVendorMaster> vendorMasterReader(@Value("#{jobParameters['input.file.name']}") String fileName) {
    return new FlatFileItemReaderBuilder<ERPVendorMaster>()
            .name("vendorMasterItemReader")
            .resource(new FileSystemResource(fileName))
            .linesToSkip(1)
            .delimited()
            .names(commaSeparatedVendorMasterHeaderValues.split(","))
            .fieldSetMapper(new BeanWrapperFieldSetMapper<ERPVendorMaster>() {{
                setConversionService(stringToDateConversionService());
                setTargetType(ERPVendorMaster.class);
            }})
            .build();
}

I'm very new to Spring boot any help will be appreciated

Thanks

question from:https://stackoverflow.com/questions/65626409/configure-itemwriter-itemreader-step-and-job-dynamically-in-spring-batch

Answer 1

Waitting for answers